Libraries Run Rust Inside Python (With PyO3)

Every clip you validate information pinch Pydantic v2, the data-validation room astir Python apps scope for, a Rust hold does the work. Its core, pydantic-core, is built pinch PyO3, the aforesaid toolchain we'll usage here.

This station builds that aforesaid benignant of bridge, mini capable to publication successful 1 sitting: a JSON parser written successful Rust, exposed to Python, truthful you tin import it for illustration immoderate different package. The past step, turning the Rust consequence into Python objects, is the 1 to understand earlier you larboard anything: for a parser for illustration this, it tin costs much than the parsing itself.

The 4 steps from Rust to import

Getting Rust codification into Python takes 4 steps:

1. Write a normal Rust module.
2. Annotate it pinch PyO3 macros.
3. Let maturin compile and instal it.
4. Import the result.

#[pyfunction] and #[pymodule] are the 2 Rust macros that do the wiring. A Rust property macro is adjacent to a Python decorator: it rewrites the usability it sits on, present adding the glue that lets Python telephone it and handles the type conversions and reference counting astatine the boundary.

Maturin past compiles the crate to a shared room (.so, .dylib, .dll) and drops it into your virtual environment, truthful import conscionable works. I locomotion done this full setup, from cargo caller to the first import, successful How to tally Rust successful Python pinch PyO3 and Maturin.

That first tutorial returns a azygous number. This 1 picks up wherever it near off, because the absorbing portion starts erstwhile you return a structure alternatively of a scalar.

The parser produces a Rust worth first

The building this parser returns is simply a JSON tree, and it's the moving illustration for the remainder of this post. In our Python to Rust cohort, students walk six weeks penning a JSON parser from scratch successful Rust, a hand-rolled tokenizer and recursive-descent parser pinch nary serde, past expose it to Python done PyO3. Josh's type hit CPython's C json module connected real-world fixtures; Jochen's ran up to 3.5x faster than the Python version.

The public reference implementation, the cleanable type students commencement from, is the codification I'll locomotion done here.

The parser produces a plain Rust enum. A Rust enum holds 1 of respective shapes, and each version tin transportation data, truthful it maps a JSON character cleanly:

That character lives wholly successful Rust. Python ne'er sees it. The PyO3 furniture is simply a bladed adapter connected top.

Exposing 1 function

Exposing a usability to Python takes 2 lines:

For a Python reader, the signature is the astir absorbing part:

• py: Python<'py> is simply a token representing entree to the Python expert and is what you walk to PyO3 APIs that request entree to Python objects. On accepted Python builds, this entree is associated pinch holding the GIL. PyO3 hands it to you and you walk it on wherever you touch a Python object.
• Bound<'py, PyAny> is simply a grip to a Python entity of immoderate type, the Rust broadside of what you'd deliberation of arsenic a PyObject.
• PyResult<T> is Result<T, PyErr>: return the value, aliases an correction PyO3 raises arsenic a Python exception.
• ? propagates that error. If parse fails, the usability returns early and Python sees an exception; different it unwraps the JsonValue and moves on.

So parse(input)? does the existent work, and .into_pyobject(py) builds the Python objects the caller asked for. That past telephone is wherever the costs lives: it has to create Python objects for the nodes successful the tree, and connected a ample archive that tin adhd up to much activity than the parse itself.

The return travel is the costly part

Here is why that conversion is not free. .into_pyobject walks the full JsonValue character and rebuilds it arsenic autochthonal Python objects: a dict per object, a database per array, a float aliases str per leaf. You supply that translator by implementing the IntoPyObject trait, which PyO3 calls to person a Rust worth into a Python one:

A archive pinch 100,000 values intends connected the bid of 100,000 Python objects being created astatine the boundary, each aft parsing is wholly done. On a ample archive this materialization loop, not the parsing, tin predominate the end-to-end time.

Errors transverse the bound the aforesaid way

The return worth is not the only point that has to translate. A parse nonaccomplishment is simply a typed Rust error, and Python wants an exception. One From impl, the trait Rust uses to person 1 type into another, lets ? do the work:

Now malformed input raises a ValueError carrying the offset wherever parsing broke. The file-reading way gets the aforesaid curen for free: std::io::Error already converts to the matching Python exception, truthful a missing way raises FileNotFoundError.

The caller gets Python semantics without the Rust furniture leaking through.

What this intends for your ain port

If the Rust usability you're porting returns a scalar, larboard it and move on. The bound is usually mini capable to ignore.

If it returns a ample structure, the conversion is your existent cost, and it is the adjacent point to optimize erstwhile the parser itself is fast. Preallocating the PyDict tin thief astatine the margins, but the bigger triumph is architectural: don't materialize the full character if the caller won't touch each of it. Hand backmost a lazy, Rust-backed position and build Python objects connected demand.

So erstwhile you scope for PyO3, floor plan the boundary, not conscionable the algorithm. Getting Rust to tally accelerated is the easy half. What you build connected the measurement out, the travel from Rust values to Python objects, is the half that decides whether the larboard was worthy it.