Today we are releasing the first merchandise campaigner for Polars 2.0. The definite 2.0 merchandise will onshore successful the pursuing weeks. We don’t purpose to make a large characteristic merchandise of Polars 2.0. In truth we dream it to beryllium a boring acquisition for you. The logic we bump this awesome type is that we tin get free of creation decisions made successful the past that presently artifact america and past we want to alteration defaults to much sensible settings that will use a greater audience. The biggest default alteration will beryllium that each LazyFrame queries now will tally connected the streaming engine. Casual Polars users tin truthful expect immense improvements successful representation usage and performance. In aggregate we expect the streaming motor to beryllium easy 5x faster.
To thief users modulation to 2.0, we person posted a afloat migration guide. This station will screen a fewer of the highlights.
Streaming motor arsenic default
This is the biggest effect alteration of 2.0. Calling cod connected a LazyFrame will now default to the streaming engine, starring to monolithic representation and capacity improvements connected astir queries for users. The logic this required a awesome type bump is that the streaming motor doesn’t guarantee row-order by default for definite operations (join, group_by, unpivot, etc.). If you require observable row-order successful those operations, you tin opt successful to that by mounting maintain_order=True.
For users who want to support utilizing the “in-memory” motor arsenic default, they tin do truthful by mounting the motor affinity.
lf = pl.LazyFrame({"k": [2, 1, 0], "v": ["a", "b", "c"]}) other = pl.LazyFrame({"k": [0, 1, 2], "r": ["x", "y", "z"]}) # 2.0: engine="auto" now resolves to the streaming engine. # Row bid is nary longer guaranteed for joins, group_by, unpivot, ... ( lf .join(other, on="k", how="left") .collect() ) # ┌─────┬─────┬─────┐ # │ k ┆ v ┆ r │ <- bid whitethorn not lucifer `lf`'s original statement order # └─────┴─────┴─────┘ # Opt successful to observable bid for this query: ( lf .join(other, on="k", how="left", maintain_order="left") .collect() ) # Or support the aged in-memory motor arsenic the default, process-wide: pl.Config.set_engine_affinity("in-memory") # ...or per query: ( lf .join(other, on="k", how="left") .collect(engine="in-memory") )Stricter Polars
Polars intends to beryllium strict and neglect fast. Errors should ideally raise up-front, not 20 minutes into a pipeline. Implicit behaviour connected data-mismatches should beryllium opt-in, not a default, since those mismatches tin hide bugs. This strictness has go moreover much valuable pinch the emergence of AI-driven development. Agents tin validate a query’s building early by calling collect_schema(), which resolves types and catches schema-level mismatches without materializing immoderate data. This ensures accelerated feedback for the agents, meaning they tin iterate faster. Not each errors tin beryllium caught during compilation of the query plan, immoderate dangle connected data. In these cases Polars defaults to stricter behaviour to guarantee inconsistencies are caught alternatively of silently producing different results.
Below are a fewer examples wherever Polars has gotten much strict:
is_in lossless type-coercion
If you tally an is_in look connected different data-types, Polars utilized to formed some types to their communal supertype, moreover if that conversion was lossy Below is an illustration pinch user-ids that tin spell incorrect by silent data-type mismatches.
# Checking if a personification ID matches a database of "flagged" relationship IDs # (flagged_ids loaded from a JSON export, wherever ample IDs became floats) flagged_ids = pl.Series([9007199254740992.0]) user_id = pl.Series([9007199254740993]) # Int64 -> a different ID, disconnected by 1 user_id.is_in(flagged_ids)Before 2.0, user_id gets coerced to Float64 to lucifer flagged_ids. But 9007199254740993 sits supra 2^53 (9007199254740992), the largest integer float64 tin correspond exactly, truthful it silently rounds down to 9007199254740992.0, giving a mendacious positive.
In 2.0 this raises: InvalidOperationError: 'is_in' cannot cheque for Int64 values successful List(Float64) data., users should explicitly formed to woody pinch lossy type conversion.
Strict concatenation
Horizontal concat will now cheque lengths alternatively of silently filling pinch null.
# Joining per-day transaction counts pinch per-day fraud-flag counts, transactions = pl.DataFrame({"day": [1, 2, 3, 4, 5], "count": [120, 98, 143, 87, 156]}) # Upstream occupation for time 5 grounded silently fraud_flags = pl.DataFrame({"flagged": [2, 0, 5, 1]}) # only 4 rows pl.concat([transactions, fraud_flags], how="horizontal") shape: (5, 2) ┌─────┬───────┬─────────┐ │ time ┆ count ┆ flagged │ │ 1 ┆ 120 ┆ 2 │ │ 2 ┆ 98 ┆ 0 │ │ 3 ┆ 143 ┆ 5 │ │ 4 ┆ 87 ┆ 1 │ │ 5 ┆ 156 ┆ null │ <- time 5 silently has nary emblem count └─────┴───────┴─────────┘In 2.0 this will raise with:
ShapeError: cannot concat dataframes pinch different heights successful 'strict' modeIf padding is what you wanted, you person to explicitly opt-in to that pinch how="horizontal_extend". Making that volition clear to the reader.
Removal of casts successful favour of dedicated methods/constructors
Another 1 worthy mentioning is the removal of galore casts that were ambiguous aliases should beryllium applied via their dedicated parsing expression, starring to 1 evident measurement to parse data.
Enums/Categoricals <> integers.
pl.Series([None, 1, 0, 2], dtype=pl.UInt32).cast(pl.Enum(["a", "b", "c"])) # ComputeError: casting from u32 to enum is not supported.Use instead: .cat.to(dtype) for int → categorical, .cat.physical() for categorical → int.
Parsing Strings to temporal data-types
pl.Series(["2022-08-30"]).cast(pl.Date) # InvalidOperationError: casting from drawstring to day is not supported.Use instead: .str.to_date() / .str.to_datetime(). These let you to use a parsing format, giving you much power complete really the information is parsed.
These were conscionable a fewer examples, but we landed galore much strictness improvements. See them each successful the migration guide.
Raising informative errors
We put a batch of effort into making judge you arsenic personification aliases your supplier tin proceed if you utilized aged parameters that are not supported anymore. We added 2 caller typed exceptions for this; polars.exceptions.AttributeRemovedError and polars.exceptions.ArgumentRemovedError that grip removed attributes and methods and removed parameters respectively.
The correction messages should constituent you to the caller API instead. Below we show 2 examples.
>>> lf.melt(id_vars="a", value_vars="b") polars.exceptions.AttributeRemovedError: `melt` was removed in type 2.0; use `LazyFrame.unpivot` instead, with `index` alternatively of `id_vars` and `on` alternatively of `value_vars` >>> df.join(df, on="a", join_nulls=True) polars.exceptions.ArgumentRemovedError: the statement 'join_nulls' for 'DataFrame.join' was deprecated in type 1.24 and has been removed in 2.0.0. It was renamed to 'nulls_equal' in type 2.0.Most of the removed functionality has been deprecated for a agelong clip and hopefully should not person affected your pipelines if you person stayed up to date. Reach retired to america if you deliberation we should person kept immoderate functionality you relied on.
Last words
Polars 2.0 is astir amended defaults (most importantly the streaming engine) and a amended API. We dream this merchandise is alternatively boring. We don’t gross caller features down awesome type bumps arsenic we vessel them arsenic soon arsenic their ready.
Don’t beryllium mistaken, Polars 2.x will beryllium overmuch amended than 1.x. There is simply a batch successful formation that we haven’t talked publically enough: due out-of-core support for the streaming engine, a caller IO-plugin design, what we deliberation will beryllium the fastest S3 scholar retired there, awesome SQL sum improvements, a cost-based planner, subordinate reordering, and the removal of mmap, which will make our pipelines afloat async extremity to end.
Try the merchandise campaigner by installing pip instal polars==2.0rc1. Give it a rotation and scope retired to america here: https://github.com/pola-rs/polars/issues aliases interaction america connected discord: https://discord.gg/4UfP5cfBE7.
English (US) · 
Indonesian (ID) · 
