Integrating Rust and Python for Data Science

Image by Author

# Introduction

 
Python is the default language of data science for good reasons. It has a mature ecosystem, a low barrier to entry, and libraries that let you move from idea to result very quickly. NumPy, pandas, scikit-learn, PyTorch, and Jupyter Notebook form a workflow that is hard to beat for exploration, modeling, and communication. For most data scientists, Python is not just a tool; it is the environment where thinking happens.

But Python also has its own limits. As datasets grow, pipelines become more complex, and performance expectations rise, teams start to notice friction. Some operations feel slower than they should on a normal day, and memory usage becomes unpredictable. At a certain point, the question stops being “can Python do this?” and becomes “should Python do all of this?”

This is where Rust comes into play. Not as a replacement for Python, nor as a language that suddenly requires data scientists to rewrite everything, but as a supporting layer. Rust is increasingly used underneath Python tools, handling the parts of the workload where performance, memory safety, and concurrency matter most. Many people already benefit from Rust without realizing it, through libraries like Polars or through Rust-backed components hidden behind Python application programming interfaces (APIs).

This article is about that middle ground. It does not argue that Rust is better than Python for data science. It demonstrates how the two can work together in a way that preserves Python’s productivity while addressing its weaknesses. We will look at where Python struggles, how Rust fits into modern data stacks, and what the integration actually looks like in practice.

# Identifying Where Python Struggles in Data Science Workloads

 
Python’s biggest strength is also its biggest limitation. The language is optimized for developer productivity, not raw execution speed. For many data science tasks, this is fine because the heavy lifting happens in optimized native libraries. When you write df.mean() in pandas or np.dot() in NumPy, you are not really running Python in a loop; you are calling compiled code.

Problems arise when your workload does not align cleanly with those primitives. Once you are looping in Python, performance drops quickly. Even well-written code can become a bottleneck when applied to tens or hundreds of millions of records.

Memory is another pressure point. Python objects carry significant overhead, and data pipelines often involve repeated serialization and deserialization steps. Similarly, when moving data between pandas, NumPy, and external systems, it can create copies that are difficult to detect and even harder to control. In large pipelines, memory usage often becomes the primary reason jobs slow down or fail, rather than central processing unit (CPU) usage.

Concurrency is where things get especially tricky. Python’s global interpreter lock (GIL) simplifies many things, but it limits true parallel execution for CPU-bound work. There are ways to circumvent this, such as using multiprocessing, native extensions, or distributed systems, but each approach comes with its own complexity.

# Using Python for Orchestration and Rust for Execution

 
The most practical way to think about Rust and Python together is the division of responsibility. Python remains in charge of orchestration, handling tasks such as loading data, defining workflows, expressing intent, and connecting systems. Rust takes over where execution details matter, such as tight loops, heavy transformations, memory management, and parallel work.

If we are to follow this model, Python remains the language you write and read most of the time. It is where you shape analyses, prototype ideas, and glue components together. Rust code sits behind clear boundaries. It implements specific operations that are expensive, repeated often, or hard to express efficiently in Python. This boundary is explicit and intentional.

One of the most stressful tasks is deciding what belongs where; it ultimately comes down to a few key questions. If the code changes often, depends heavily on experimentation, or benefits from Python’s expressiveness, it probably belongs in Python. However, if the code is stable and performance-critical, Rust is a better fit. Data parsing, custom aggregations, feature engineering kernels, and validation logic are common examples that lend themselves well to Rust.

This pattern already exists across modern data tooling, even when users are not aware of it. Polars uses Rust for its execution engine while exposing a Python API. Parts of Apache Arrow are implemented in Rust and consumed by Python. Even pandas increasingly rely on Arrow-backed and native components for performance-sensitive paths. The ecosystem is quietly converging on the same idea: Python as the interface, Rust as the engine.

The key benefit of this approach is that it preserves productivity. You do not lose Python’s ecosystem or readability. You gain performance where it actually matters, without turning your data science codebase into a systems programming project. When done well, most users interact with a clean Python API and never need to care that Rust is involved at all.

# Understanding How Rust and Python Actually Integrate

 
In practice, Rust and Python integration is more straightforward than it sounds, as long as you avoid unnecessary abstraction. The most common approach today is to use PyO3. PyO3 is a Rust library that enables writing native Python extensions in Rust. You write Rust functions and structs, annotate them, and expose them as Python-callable objects. From the Python side, they behave like regular modules, with normal imports and docstrings.

A typical setup looks like this: Rust code implements a function that operates on arrays or Arrow buffers, handles the heavy computation, and returns results in a Python-friendly format. PyO3 handles reference counting, error translation, and type conversion. Tools like maturin or setuptools-rust then package the extension so it can be installed with pip, just like any other dependency.

Distribution plays a crucial role in the story. Building Rust-backed Python packages used to be difficult, but the tooling has greatly improved. Prebuilt wheels for major platforms are now common, and continuous integration (CI) pipelines can produce them automatically. For most users, installation is no different from installing a pure Python library.

Crossing the Python and Rust boundary incurs a cost, both in terms of runtime overhead and maintenance. This is where technical debt can creep in — if Rust code starts leaking Python-specific assumptions, or if the interface becomes too granular, the complexity outweighs the gains. This is why most successful projects maintain a stable boundary.

# Speeding Up a Data Operation with Rust

 
To illustrate this, consider a situation that most data scientists often find themselves in. You have a large in-memory dataset, tens of millions of rows, and you need to apply a custom transformation that is not vectorizable with NumPy or pandas. It is not a built-in aggregation. It is domain-specific logic that runs row by row and becomes the dominant cost in the pipeline.

Imagine a simple case: computing a rolling score with conditional logic across a large array. In pandas, this often results in a loop or an apply, both of which become slow once the data no longer fits neatly into vectorized operations.

// Example 1: The Python Baseline

def score_series(values):
    out = []
    prev = 0.0
    for v in values:
        if v > prev:
            prev = prev * 0.9 + v
        else:
            prev = prev * 0.5
        out.append(prev)
    return out

This code is readable, but it is CPU-bound and single-threaded. On large arrays, it becomes painfully slow. The same logic in Rust is straightforward and, more importantly, fast. Rust’s tight loops, predictable memory access, and easy parallelism make a big difference here.

// Example 2: Implementing with PyO3

use pyo3::prelude::*;

#[pyfunction]
fn score_series(values: Vec) -> Vec {
    let mut out = Vec::with_capacity(values.len());
    let mut prev = 0.0;

    for v in values {
        if v > prev {
            prev = prev * 0.9 + v;
        } else {
            prev = prev * 0.5;
        }
        out.push(prev);
    }

    out
}

#[pymodule]
fn fast_scores(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(score_series, m)?)?;
    Ok(())
}

Exposed through PyO3, this function can be imported and called from Python like any other module.

from fast_scores import score_series
result = score_series(values)

In benchmarks, the improvement is often dramatic. What took seconds or minutes in Python drops to milliseconds or seconds in Rust. The raw execution time improved significantly. CPU utilization increased, and the code performed better on larger inputs. Memory usage became more predictable, resulting in fewer surprises under load.

What did not improve was the overall complexity of the system; you now have two languages and a packaging pipeline to manage. When something goes wrong, the issue might reside in Rust rather than Python.

// Example 3: Custom Aggregation Logic

You have a large numeric dataset and need a custom aggregation that does not vectorize cleanly in pandas or NumPy. This often occurs with domain-specific scoring, rule engines, or feature engineering logic.

Here is the Python version:

def score(values):
    total = 0.0
    for v in values:
        if v > 0:
            total += v ** 1.5
    return total

This is readable, but it is CPU-bound and single-threaded. Let’s take a look at the Rust implementation. We move the loop into Rust and expose it to Python using PyO3.

Cargo.toml file

[lib]
name = "fastscore"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.21", features = ["extension-module"] }

src/lib.rs

use pyo3::prelude::*;

#[pyfunction]
fn score(values: Vec) -> f64 {
    values
        .iter()
        .filter(|v| **v > 0.0)
        .map(|v| v.powf(1.5))
        .sum()
}

#[pymodule]
fn fastscore(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(score, m)?)?;
    Ok(())
}

Now let’s use it from Python:

import fastscore

data = [1.2, -0.5, 3.1, 4.0]
result = fastscore.score(data)

But why does this work? Python still controls the workflow. Rust handles only the tight loop. There is no business logic split across languages; instead, execution occurs where it matters.

// Example 4: Sharing Memory with Apache Arrow

You want to move large tabular data between Python and Rust without serialization overhead. Converting DataFrames back and forth can significantly impact performance and memory. The solution is to use Arrow, which provides a shared memory format that both ecosystems understand.

Here is the Python code to create the Arrow data:

import pyarrow as pa
import pandas as pd

df = pd.DataFrame({
    "a": [1, 2, 3, 4],
    "b": [10.0, 20.0, 30.0, 40.0],
})

table = pa.Table.from_pandas(df)

At this point, data is stored in Arrow’s columnar format. Let’s write the Rust code to consume the Arrow data, using the Arrow crate in Rust:

use arrow::array::{Float64Array, Int64Array};
use arrow::record_batch::RecordBatch;

fn process(batch: &RecordBatch) -> f64 {
    let a = batch
        .column(0)
        .as_any()
        .downcast_ref::()
        .unwrap();

    let b = batch
        .column(1)
        .as_any()
        .downcast_ref::()
        .unwrap();

    let mut sum = 0.0;
    for i in 0..batch.num_rows() {
        sum += a.value(i) as f64 * b.value(i);
    }
    sum
}

# Rust Tools That Matter for Data Scientists

 
Rust’s role in data science is not limited to custom extensions. A growing number of core tools are already written in Rust and quietly powering Python workflows. Polars is the most visible example. It offers a DataFrame API similar to pandas but is built on a Rust execution engine.

Apache Arrow plays a different but equally important role. It defines a columnar memory format that both Python and Rust understand natively. Arrow enables the transfer of large datasets between systems without requiring copying or serialization. This is often where the biggest performance wins come from — not from rewriting algorithms but from avoiding unnecessary data movement.

# Determining When You Should Not Reach for Rust

 
At this point, we have shown that Rust is powerful, but it is not a default upgrade for every data problem. In many cases, Python remains the right tool.

If your workload is mostly I/O-bound, orchestrating APIs, running structured query language (SQL), or gluing together existing libraries, Rust will not buy you much. Most of the heavy lifting in common data science workflows already happens inside optimized C, C++, or Rust extensions. Wrapping more code in Rust on top of that often adds complexity without real gains.

Another thing is that your team's skill matters more than benchmarks. Introducing Rust means introducing a new language, a new build toolchain, and a stricter programming model. If only one person understands the Rust layer, that code becomes a maintenance risk. Debugging cross-language issues can also be slower than fixing pure Python problems.

There is also the risk of premature optimization. It is easy to spot a slow Python loop and assume Rust is the answer. Often, the real fix is vectorization, better use of existing libraries, or a different algorithm. Moving to Rust too early can lock you into a more complex design before you fully understand the problem.

A simple decision checklist helps:

• Is the code CPU-bound and already well-structured?
• Does profiling show a clear hotspot that Python cannot reasonably optimize?
• Will the Rust component be reused enough to justify its cost?

If the answer to these questions is not a clear "yes," staying with Python is usually the better choice.

# Conclusion

 
Python remains at the forefront of data science; it is still very popular and useful to date. You can perform several activities ranging from exploration to model integration and much more. Rust, on the other hand, strengthens the foundation underneath. It becomes necessary where performance, memory control, and predictability become important. Used selectively, it allows you to push past Python’s limits without sacrificing the ecosystem that enables data scientists to work efficiently and iterate quickly.

The most effective approach is to start small by identifying one bottleneck, then replacing it with a Rust-backed component. After this, you have to measure the result. If it helps, expand carefully; if it does not, simply roll it back.
 
 

Shittu Olumide is a software engineer and technical writer passionate about leveraging cutting-edge technologies to craft compelling narratives, with a keen eye for detail and a knack for simplifying complex concepts. You can also find Shittu on Twitter.