3 Polars Tricks for High-Performance Data Manipulation

Polars gets its speed from two places: its expression engine written and executing in Rust across every core at its disposal, and its query optimizer that rewrites your work before any of it runs. Almost every slow Polars script lacks in terms of one of those two. The awkward part is that the fast version and the slow version end up looking nearly identical on the page. Here are three places that happens.

The examples run against a month of NYC yellow taxi trips, published by the TLC as Parquet. Download it first:

curl -O https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2026-01.parquet

Everything below was checked against Polars 1.44.2.

Trick 1: Scanning a File Instead of Reading It

pl.read_parquet pulls an entire file's contents into memory and then lets you filter it. pl.scan_parquet hands back a LazyFrame instead; this records what you asked for without doing any of it. That gap is where the optimizer earns its keep: it pushes your filter and your column list down to the scan itself, so the narrowing happens as the file is read rather than after. The rows you threw away were never decoded, and thus computation was never wasted upon them.

import polars as pl

q = (
    pl.scan_parquet("yellow_tripdata_2026-01.parquet")
    .filter(pl.col("fare_amount") > 50)
    .select("PULocationID", "tip_amount")
    .group_by("PULocationID")
    .agg(pl.col("tip_amount").mean())
)

# This explains the plan, not the data
print(q.explain())
df = q.collect()

Nothing becomes permanent until collect(). explain() prints the plan the optimizer has built, and you can read the pushed-down predicate and the trimmed column list:

AGGREGATE[maintain_order: false]
  [col("tip_amount").mean()] BY [col("PULocationID")]
  FROM
  simple π 2/2 ["PULocationID", "tip_amount"]
    Parquet SCAN [yellow_tripdata_2026-01.parquet]
    PROJECT 3/20 COLUMNS
    SELECTION: col("fare_amount") > 50.0
    ESTIMATED ROWS: 3724889

The habit you want to break is calling collect() partway through a chain out of nervousness. Every collect() is a wall the optimizer cannot see past.

Trick 2: Per-Group Values Without the Group-By Round Trip

Wanting a group-level number back on every row is a common thing you will want to do to a dataframe. This is usually turned into a group_by and an agg followed by a join back onto the frame you already had. This is two full passes, a materialized intermediate, and a join key to fuss with. .over() does the same work within a single expression, in one pass, with row order preserved. Its default mapping strategy, group_to_rows, maps each aggregate back to the rows it came from.

Four rows, two zones, each fare as a share of its own zone's total:

df = pl.DataFrame({
    "pickup_zone": ["A", "A", "B", "B"],
    "fare_amount": [30.0, 70.0, 25.0, 75.0],
})

out = df.with_columns(
    (pl.col("fare_amount") / pl.col("fare_amount").sum().over("pickup_zone"))
    .alias("share_of_zone")
)
print(out)

Output:

shape: (4, 3)
┌─────────────┬─────────────┬───────────────┐
│ pickup_zone ┆ fare_amount ┆ share_of_zone │
│ ---         ┆ ---         ┆ ---           │
│ str         ┆ f64         ┆ f64           │
╞═════════════╪═════════════╪═══════════════╡
│ A           ┆ 30.0        ┆ 0.3           │
│ A           ┆ 70.0        ┆ 0.7           │
│ B           ┆ 25.0        ┆ 0.25          │
│ B           ┆ 75.0        ┆ 0.75          │
└─────────────┴─────────────┴───────────────┘

Both zones total 100.0, so zone A gets 0.3 and 0.7, zone B 0.25 and 0.75. over also takes an order_by, which is what turns a running total or a per-group lag into one line, as opposed to a sort followed by a join. Switch to explode only when you want the frame to change shape. The default is right almost every time.

Trick 3: Getting Python Out of the Loop

map_elements hands every value in a column to a Python callable, one at a time, and the documentation is blunt about the cost: "much slower than the native expressions API." Polars even raises a PolarsInefficientMapWarning when it spots a map it thinks it can replace. Most uses are a conditional, and when/then/otherwise handles those natively.

Banding a numeric column, with no Python in sight:

banded = df.with_columns(
    pl.when(pl.col("fare_amount") > 50)
    .then(pl.lit("high"))
    .when(pl.col("fare_amount") > 20)
    .then(pl.lit("medium"))
    .otherwise(pl.lit("low"))
    .alias("fare_band")
)
print(banded)

Output:

shape: (4, 3)
┌─────────────┬─────────────┬───────────┐
│ pickup_zone ┆ fare_amount ┆ fare_band │
│ ---         ┆ ---         ┆ ---       │
│ str         ┆ f64         ┆ str       │
╞═════════════╪═════════════╪═══════════╡
│ A           ┆ 30.0        ┆ medium    │
│ A           ┆ 70.0        ┆ high      │
│ B           ┆ 25.0        ┆ medium    │
│ B           ┆ 75.0        ┆ high      │
└─────────────┴─────────────┴───────────┘

The result is identical to a map_elements version, with the difference being in what the CPU did. One note: Polars computes every branch of a when/then chain in parallel and filters afterwards, so each branch has to be valid on its own.

Wrapping Up

The three tricks are really one larger trick — which is, in and of itself, a trick. In each case the fast version keeps the work inside the engine, while the slow version hands it back to Python or to memory. When a script is slower than it should be, the first question isn't which flag to tune; it's which of those two boundaries you crossed. Scan instead of read, and express instead of loop.
 
 

Matthew Mayo (@mattmayo13) holds a master's degree in computer science and a graduate diploma in data mining. As managing editor of KDnuggets & Statology, and contributing editor at Machine Learning Mastery, Matthew aims to make complex data science concepts accessible. His professional interests include natural language processing, language models, machine learning algorithms, and exploring emerging AI. He is driven by a mission to democratize knowledge in the data science community. Matthew has been coding since he was 6 years old.