The Berlin Rent Freeze: How many illegal overpriced offers can I find online?

This post presents an analysis of Berlin online real estate listings, investigating a controversial law capping rents in the state, which went into effect on February 23. Are current landlords already respecting the new rent cap?

By Felix Vemmer, Data Analyst at N26


Photo by Jonas Tebbe on Unsplash


The current housing situation in Berlin

Having moved to Berlin, finding a reasonably priced flat was definitely one of the hardest parts. To get an idea of how competitive it can be, in November 1,749 flat-hunters queued outside to visit a vacant apartment in the popular Schöneberg district. The flat was advertised online just 12 hours before.


To stop the ever increasing costs of housing, the Berlin state government passed a controversial law that caps the rent. In one week, on February 23, the new law will come into effect.


Are current online flat offerings following the rent cap?

As my next data science side project, I decided to analyse current online listings on ImmobilienScout24, to see whether current landlords already respect the new rent cap. Not following the law imposes quite a high fine:

A fine of up to EUR 500,000 may be imposed for violations of the rent cap.

I split this blogpost into three parts, to help you find and read what is most relevant to you:

  • Part1: A few technical details and how I got the dataset
  • Part 2: Learn more about the rent cap and how it is calculated
  • Part 3: Go straight to the results and see what I found out.


Part 1: Downloading 4040 listings in under 4 minutes

With the amazing scrapy package, I was able to web scrape a total of 4,040 listings in two sessions under two minutes. Of course, setting up the spider took quite a bit longer.

Next, all the data was loaded into a pandas dataframe and cleaned up. I will not go too much into detail here but it was quite some cleaning.


Part 2: Batch calculating the new rental cap

In order to calculate the new rental cap, I had to engineer some features for the dataset. First, however, let’s have a look at how the rental cap is calculated.


Rental Cap Calculation 101

In short, the new rent cap is calculated from the following four components:

1. A base price: The following rent table determines how high the net cold rent may be depending on the year of construction, the existence of a central heating system (Sammelheizung) and/or bathroom (Bad).

Berlin Rent table, source: Stadtentwicklung Berlin

2. Modern equipment factor: For apartments with modern furnishing the price increases by 1.00 Euro, if at least three of the following features are present:

  • an elevator that can be reached without thresholds
  • fitted kitchen
  • high-quality sanitary equipment
  • high-quality flooring in the vast majority of living spaces
  • energy consumption characteristic value of less than 120 kWh/(m² a)

3. A location premium/deduction: For flats in a …

  • … simple residential area, a deduction of 0.28 euros from the relevant rent in the rent table is taken into account when calculating the upper rent limit.
  •  medium residential area 0.09 euros are deducted
  • … good residential area a surcharge of 0.74 euros is to be taken into account.

Quite ironically, the location classification has still not been published yet.

4. Single or double family house: If the living space is located in buildings with no more then two apartments, the rent ceiling is increased by a surcharge of ten percent.

If you sum up all those four factors you get the rent cap. However, a rent is only excessive if the following criteria is met:

A rent is excessive if it is more than 20 percent above the relevant upper rent limit in the rent table, taking into account the four factors above.

If you are lost at this point, live in Berlin and rather care about if you are currently paying too much for your flat in Berlin, try out this cool calculator (in German) and see whether you understand the calculation better.


Extracting the location rating from 256 PDF pages

By far the most challenging part was to find a proxy for the not yet published location classification. In the online calculator, I saw that the Berlin rent index of 2019 was used as a proxy, so I followed the same approach.

The biggest challenge of this project, however, was that this info was published in a PDF document far from the format I actually needed. With the help of the tabula package, I was able to turn 256 PDF pages into 270,068 rows of location ratings for every street in Berlin.

Finally, merging the slightly differently spelled street names from the two sources, became a lot simpler with the fuzzywuzzy package. Although quite computationally expensive, a lot easier than every imaginable regex pattern.


My conservative assumptions for the rent cap calculation

In my calculations, I took the following conservative assumptions to calculate the price rental ceiling per square meter:

  • the maximum price from the base table (1) for the respective year of construction, assuming that all the flats have central heating and a bathroom
  • all the flats fulfill at least three criteria (2) of modern furnishing, assuming that all of them have modern equipment
  • the rent index PDF document published in 2019 is a good approximation for the location rating (3)
  • all the flats are non single family houses, given that I only scraped apartments (4)


Disclaimer for the correctness and completeness of the information and analysis provided below: Although I carried out this project with great care, I do not assume any liability for the correctness nor completeness of the results provided below.



Part 3: Analysing the results

From around 4k scraped listings, 1687 listings (41.6%) have the year of construction before 2014 and address info so that the rent cap could be calculated.

Quite shockingly, three out of four analysed listings from the sample of 1687 listings had a higher price than the allowed rent cap.

Three out of four analysed rental listings on ImmobilienScout 24 were over the rental cap.

If all of those flats would be rented out at the current illegal price, all tenants would pay 463,059.79€ per month more than they had to under the new law:

Rent composition of all illegally priced listings in dataset.

In the overall dataset (listings with sufficient information) the monthly average cold rent is at 15.12€/m². Under the new law, this average would decrease by 39% to an average of 9.21€/m².

Having a closer look at how much the cold rent/m² is exceeding the new rent cap reveals that for example 386 out of the 1272 listings are charging between 0.50€ and 2.49€ too much.

Having a closer look at how this changes across the different districts, shows that central districts, such as Mitte or Tiergarten, see on average a bigger decrease in the price, compared to more remote districts, such as Hellersdorf or Marzahn.

Rent comparison between current average price and new rent cap.

Percentage change from current cold rent/m² to newly caculated rent cap/m²

Finally, I had a look at who posted the most listings on ImmibolienScout24 in my small sample of 1272 overpriced flats. As expected, many bigger real estate companies made it to the top of the list. Then I calculated the average of their current illegal price/m² to and compare it with my calculated average rent cap to find out how much on average they charge in excess for all of their listings. The results are shown below, and Berlinhaus Verwaltung GmbH with 29 listings made it to the top of the list with charging on average 10.60€/m² more than they will be allowed under the new rental cap.

Comparison of all real estate companies and how much they charge in excess under the new rental cap.

Finally, I wanted to showcase one specific example and bring us back to the beginning of this blogpost where we started with an overcrowded apartment viewing in Schöneberg. Filtering my dataset for Schöneberg with the highest rent excess price, revealed that in the future under the new law one can save 406.93€ per month reducing the rent from 650.00€ to 243.07€.

Example rent composition of flat in Schöneberg

To double check my results one more time, I plugged the data into the online calculator and arrived at the same result:


I hope you enjoyed this blog post. If you are currently living in Berlin and wonder if you are paying too much for your house, check out this neat calculator for free.

If I find more time I might build a small web app myself where anyone can similarly post a link to an online listing and have an automatic calculation whether the rent is exceeding the law.

Bio: Felix Vemmer is a Data Analyst at N26 focusing on creating interesting data sets and projects through web scraping and machine learning.

Original. Reposted with permission.