TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Polars has released version 2.0, making its streaming engine the default for LazyFrame collection and enabling initial spill-to-disk support. The release also expands SQL support and adds a Map dtype; performance comparisons cited by the project come from its own benchmarks and have stated limits.
Polars has released version 2.0, making its streaming engine the default when users collect a LazyFrame and enabling an initial form of spill-to-disk processing. The changes affect how queries run and handle memory, while the release also expands SQL support and adds a native Map data type.
Polars says the default streaming engine can reduce memory use and improve performance for many queries. The change also affects results that depend on row order: the project warns that operations such as joins, group-bys and unpivots do not guarantee observable row order by default under streaming execution. Users who need to preserve order can set maintain_order=True for supported operations.
Version 2.0 enables out-of-core processing, which can spill data to disk as a query runs. According to Polars, spilling begins at about 80% of available RAM, a threshold the project says may need tuning, and the default disk budget is 64 GB. At release, supported operations include sorts, window functions and many expressions; joins and group-bys are not yet supported for spill-to-disk execution.
The release also adds first-class SQL support and a Map dtype compatible with Arrow MapType. Polars describes its SQL improvements as including join reordering, common-subplan elimination and dynamic predicates or bloom filters. The Map type represents key-value data in a form the project says can be accessed with operations such as key lookups, checking for a key, and retrieving keys or values.
Streaming Changes Query Execution
Making streaming the default may change the memory profile of everyday Polars queries without users explicitly selecting a different engine. For analysts working with data larger than available memory, the initial disk-spilling feature offers a way for some supported operations to continue by using storage rather than relying only on RAM. Its current limits matter: this is not yet out-of-core support for every query type, and the release notes identify joins and group-bys as future work.
The row-order change is a practical migration concern. A query that previously appeared to preserve order may not do so under the new default for certain operations. Users whose downstream steps rely on a particular order may need to request it explicitly and check results after upgrading. SQL support, meanwhile, gives teams another way to use the engine, but the performance evidence released with the update is based on tests run by Polars rather than an independent evaluation.
As an affiliate, we earn on qualifying purchases.
Why Version 2.0 Changes Defaults
Polars says it had previously explained the rationale for moving to version 2.0 and uses the release post to describe the features included. The project characterizes the update as not intended to be a major feature release, while emphasizing the scope of changes to the engine and SQL capabilities. The major-version bump is tied in part to the streaming engine’s different row-order guarantees.
For its SQL performance comparison, Polars tested data derived from TPC-H and TPC-DS against DuckDB 1.5.6, DuckDB 2.0 alpha and DataFusion 54.0.0. Tests ran on two AWS machine configurations, used five hot runs per query, and reported the best run. Polars says it compared totals and geometric means; results exclude specified DataFusion timeouts and an out-of-memory query. The project says its default configuration was fastest in all but one benchmark, while also reporting overhead on the 192-core machine for smaller queries. It has published a repository for reproducing the benchmark.
Limits and Results Still Need Testing
The release post does not give an independent verification of its benchmark comparisons, and the results depend on the listed hardware, data, query generation, cache conditions and scoring method. Polars invites readers to reproduce the tests; the source does not report results from outside teams. It also says performance on the 192-thread machine has a constant overhead that hurts small queries and that a fix may come in a later release.
The initial spill-to-disk feature does not cover every operation. Polars says joins and group-bys are planned for future out-of-core support, but the release post does not provide a delivery date. The source material also does not specify the release post’s publication date, compatibility details for every API change, or whether all users will see the same performance effects on their own workloads.
More Out-of-Core Work Planned
Polars says it plans to add out-of-core support for joins and group-bys, extending spill-to-disk beyond the operations supported at launch. The project also says it has diagnosed the large-machine overhead seen in its SQL tests and hopes to address it in the next release; that is a stated intention, not a confirmed release date or fix.
In the meantime, users can review the release notes, test queries that rely on row order, and compare memory use and runtime on their own workloads. Readers seeking to check the SQL claims can consult the benchmark repository linked by Polars, while keeping the project’s test conditions and excluded results in view.
Key Questions
What is the main change in Polars 2.0?
Collecting a LazyFrame now defaults to the streaming engine. The release also enables initial spill-to-disk support and adds SQL and data-type improvements.
Does streaming preserve row order?
Not for every operation by default. Polars names joins, group-bys and unpivots as examples where row order is not guaranteed; users can set maintain_order=True when they need observable order for supported operations.
Which operations can spill data to disk?
At launch, Polars lists sorts, window functions and many expressions. It says joins and group-bys are not yet supported for out-of-core execution, with support planned for later.
Are Polars 2.0’s SQL performance results independently verified?
The release post presents benchmarks run by Polars and provides a repository for replication. The source does not include independent verification, so its performance conclusions should be understood as the project’s reported results.
Source: hn
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
