How to Process Large JSON Files in Python With ijson

JSON is one of the most common formats for storing and exchanging data in Python applications. For small files, Python's built-in json module works perfectly well:

import json

with open("data.json") as f:
    data = json.load(f)

The problem starts when the file becomes large.

json.load() parses the entire JSON document and creates the corresponding Python objects in memory before you can start processing the data. A file that occupies a few hundred megabytes on disk can therefore require much more memory once converted into Python dictionaries, lists, strings, and numbers.

A better option for this type of workload is ijson, an iterative JSON parser that lets you process data incrementally. In this tutorial, we will use a large JSON file containing San Francisco land-parcel data. We will first process it with json.load(), measure what happens to memory, and then solve the same task with ijson.

Step 1: Downloading a Large JSON Dataset

For this experiment, we will use the San Francisco City Lots dataset. I am using Google Colab for this article. Download it using:

curl -L \
https://raw.githubusercontent.com/zemirco/sf-city-lots-json/master/citylots.json \
-o citylots.json

You can check its size with:

ls -lh citylots.json

The dataset is 181 MB in size, making it large enough to demonstrate the difference between loading and streaming JSON. The file follows the standard GeoJSON FeatureCollection structure. You can check it using this code:

import json

with open("citylots.json", "r") as f:
    data = json.load(f)

print(data.keys())
data["features"][0]

Output:

dict_keys(['type', 'features'])

{'type': 'Feature',
 'properties': {'MAPBLKLOT': '0001001',
  'BLKLOT': '0001001',
  'BLOCK_NUM': '0001',
  'LOT_NUM': '001',
  'FROM_ST': '0',
  'TO_ST': '0',
  'STREET': 'UNKNOWN',
  'ST_TYPE': None,
  'ODD_EVEN': 'E'},
 'geometry': {'type': 'Polygon',
  'coordinates': [[[-122.42200352825247, 37.80848009696725, 0.0],
    [-122.42207601332528, 37.808835019815085, 0.0],
    [-122.42110217434863, 37.808803534992904, 0.0],
    [-122.42106256906727, 37.80860105681815, 0.0],
    [-122.42200352825247, 37.80848009696725, 0.0]]]}}

The important part for us is the features array.

Each feature represents one parcel and contains two main sections:

feature
├── properties
│   ├── BLKLOT
│   ├── BLOCK_NUM
│   ├── LOT_NUM
│   └── STREET
│
└── geometry
    └── coordinates

Our analysis only needs a few values under properties. However, if we use the standard JSON parser, Python still has to load all of the geometry coordinates as well.

Step 2: Processing the File With json.load()

Let's first use the approach most Python developers would naturally try. Create a file called standard_json.py:

import json
import time

start = time.perf_counter()

with open("citylots.json", "r") as f:
    data = json.load(f)

features = data["features"]

market_count = 0
first_five = []

for feature in features:

    properties = feature["properties"]

    if properties.get("STREET") == "MARKET":

        market_count += 1

        if len(first_five) < 5:
            first_five.append(
                properties.get("BLKLOT")
            )

elapsed = time.perf_counter() - start

print(f"Total parcels: {len(features):,}")
print(f"Market Street parcels: {market_count:,}")
print(f"First five parcel IDs: {first_five}")
print(f"Time: {elapsed:.2f} seconds")

The code is simple and perfectly valid.

The problem is this line:

data = json.load(f)

Before our loop can process even one parcel, Python has to:

Read the complete file
        ↓
Parse every JSON object
        ↓
Create all Python dictionaries
        ↓
Create all lists and coordinates
        ↓
Store everything in memory
        ↓
Start our analysis

This means that even though we only need STREET and BLKLOT, we also load every polygon and coordinate from the dataset. Here's the output that I got:

Total parcels: 206,560
Market Street parcels: 808
First five parcel IDs: ['0265003', '0265004', '0265005', '0290012', '0291001']
Time: 4.01 seconds

Step 3: Measuring the Memory Usage

Instead of simply saying that json.load() consumes a lot of memory, let's measure it.

One easy option is memory-profiler.

Install it:

pip install memory-profiler

Now replace the code inside it with this:

from memory_profiler import profile
import json

@profile
def process_file():

    with open("citylots.json", "r") as f:
        data = json.load(f)

    features = data["features"]

    market_count = 0

    for feature in features:

        if (
            feature["properties"]
            .get("STREET") == "MARKET"
        ):
            market_count += 1

process_file()

Run:

python -m memory_profiler standard_json.py

The key line to watch is data = json.load(f):

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     5     43.4 MiB     43.4 MiB           1   @profile
     6                                         def process_file():
     7                                         
     8    808.9 MiB      0.0 MiB           2       with open("citylots.json", "r") as f:
     9    808.9 MiB    765.5 MiB           1           data = json.load(f)
    10                                         
    11    808.9 MiB      0.0 MiB           1       features = data["features"]
    12                                         
    13    808.9 MiB      0.0 MiB           1       market_count = 0
    14                                         
    15    808.9 MiB      0.0 MiB      206561       for feature in features:
    16                                         
    17                                                 if (
    18    808.9 MiB      0.0 MiB      413120               feature["properties"]
    19    808.9 MiB      0.0 MiB      206560               .get("STREET") == "MARKET"
    20                                                 ):
    21    808.9 MiB      0.0 MiB         808               market_count += 1
    22                                         
    23    808.9 MiB      0.0 MiB           1       print("Total parcels:", len(features))
    24    808.9 MiB      0.0 MiB           1       print("Market Street parcels:", market_count)

You can see a noticeable increase in memory consumption at this point.

Why can memory usage be much larger than the original JSON file?

Because JSON such as:

{
  "STREET": "MARKET"
}

becomes several Python objects:

dictionary
   ↓
Python string key
   ↓
Python string value

The same happens for nested geometry arrays and numeric coordinates.

The file size on disk is therefore not the same as its memory footprint once parsed.

This gives us a baseline to compare against ijson.

Step 4: Streaming the JSON File With ijson

Now let's solve the same problem without loading the complete document.

Install ijson:

pip install ijson

The key function we will use is:

ijson.items()

Our dataset has this hierarchy:

root
│
├── type
│
└── features
     ├── item
     ├── item
     ├── item
     └── ...

Therefore, we can ask ijson to give us each feature individually:

import ijson

with open("citylots.json", "rb") as f:

    for feature in ijson.items(
        f,
        "features.item"
    ):

        print(feature)
        break

The prefix:

features.item

means:

Enter the features array and return each item inside it.

Instead of creating one enormous list of parcels, ijson gives us records one at a time.

Conceptually:

Feature 1
   ↓
process
   ↓
discard

Feature 2
   ↓
process
   ↓
discard

Feature 3
   ↓
process
   ↓
discard

This changes the memory requirements dramatically.

Step 5: Performing the Same Analysis With ijson

Now let's repeat exactly the same task.

Create stream_json.py:

import ijson
import time

start = time.perf_counter()

total = 0
market_count = 0
first_five = []

with open("citylots.json", "rb") as f:

    for feature in ijson.items(
        f,
        "features.item"
    ):

        total += 1

        properties = feature["properties"]

        if properties.get("STREET") == "MARKET":

            market_count += 1

            if len(first_five) < 5:
                first_five.append(
                    properties.get("BLKLOT")
                )

elapsed = time.perf_counter() - start

print(f"Total parcels: {total:,}")
print(f"Market Street parcels: {market_count:,}")
print(f"First five parcel IDs: {first_five}")
print(f"Time: {elapsed:.2f} seconds")

Output:

Total parcels: 206,560
Market Street parcels: 808
First five parcel IDs: ['0265003', '0265004', '0265005', '0290012', '0291001']
Time: 4.58 seconds

We no longer have:

data = json.load(f)

or:

features = data["features"]

Instead, the program processes each feature directly from the parser:

for feature in ijson.items(
    f,
    "features.item"
):

At any moment, we mainly need memory for:

current feature
+
ijson's parser buffer
+
small counters
+
five saved IDs

We do not need memory for the complete dataset.

Run the same memory test:

Output:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     5     43.8 MiB     43.8 MiB           1   @profile
     6                                         def process_file():
     7                                         
     8     43.8 MiB      0.0 MiB           1       total = 0
     9     43.8 MiB      0.0 MiB           1       market_count = 0
    10                                         
    11     46.1 MiB      0.0 MiB           2       with open("citylots.json", "rb") as f:
    12                                         
    13     46.1 MiB      2.2 MiB      206562           for feature in ijson.items(
    14     43.8 MiB      0.0 MiB           1               f,
    15     43.8 MiB      0.0 MiB           1               "features.item"
    16                                                 ):
    17                                         
    18     46.1 MiB      0.0 MiB      206560               total += 1
    19                                         
    20                                                     if (
    21     46.1 MiB      0.1 MiB      413120                   feature["properties"]
    22     46.1 MiB      0.0 MiB      206560                   .get("STREET") == "MARKET"
    23                                                     ):
    24     45.8 MiB      0.0 MiB         808                   market_count += 1
    25                                         
    26                                         
    27     46.1 MiB      0.0 MiB           1       print("Total parcels:", total)
    28     46.1 MiB      0.0 MiB           1       print("Market Street parcels:", market_count)

Now compare the peak memory usage with the result from json.load().

Approach	Input File	Peak Memory	Processing Model
json.load()	181 MB	808.9 MiB	Load everything
ijson.items()	181 MB	46.1 MiB	Stream records
Difference	Same file	~762.8 MiB less memory	More efficient processing

The important behavior should remain the same: ijson is primarily useful because of this more stable memory footprint.

This does not necessarily mean ijson will always be faster. If enough RAM is available, json.load() may sometimes parse a document very quickly. The main advantage of ijson is that memory consumption no longer needs to grow with the size of the complete JSON document.

Wrapping Up

Large JSON files do not necessarily require large amounts of memory.

The problem is often not the JSON format itself but how we choose to process it.

In this tutorial, we used a real San Francisco land-parcel dataset of roughly 181 MB.

With Python's standard:

json.load(f)

the complete dataset had to be parsed into Python objects before our analysis could begin.

That included thousands of nested dictionaries, lists, property values, polygons, and coordinates that our particular task did not even need.

Using:

ijson.items(
    f,
    "features.item"
)

changed the processing model completely.

Instead of:

load everything → process

we used:

read → process → discard

The key takeaway is simple:

If your task only requires one record at a time, there is usually no reason to load every record at once.

 
 

Kanwal Mehreen is a machine learning engineer and a technical writer with a profound passion for data science and the intersection of AI with medicine. She co-authored the ebook "Maximizing Productivity with ChatGPT". As a Google Generation Scholar 2022 for APAC, she champions diversity and academic excellence. She's also recognized as a Teradata Diversity in Tech Scholar, Mitacs Globalink Research Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having founded FEMCodes to empower women in STEM fields.