Apache DataFusion 55.0.0 Released

Posted on: Tue 25 August 2026 by pmc

We are proud to announce the release of DataFusion 55.0.0. This post highlights some of the many improvements since DataFusion 54.0.0, such as significant performance increases, range partitioning, MERGE INTO support, and runtime row-group pruning. The complete list of changes is available in the changelog. This release represents roughly 10 weeks of development and 877 commits. Thanks to the 175 contributors (a new record!) for making it possible.

Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0.

Figure 1: Development activity over the last three DataFusion releases: total commits, commits per day, and unique contributors, computed from each release's changelog and release dates.

Performance Improvements 🚀

In this release, we focused our optimizations on making DataFusion faster across the board rather than further optimizing our very satisfying ClickBench numbers (DataFusion is already the fastest in some cases — see the appendix), as ClickBench represents only a tiny fraction of what our actual users do (e.g. its files have no page index and contain only integer and string columns).

Here is a representative sample of the performance improvements in this release; see the full list in the appendix.

Improvement Representative Result Area
Runtime row-group pruning for TopK 4.2x faster topk_tpch Q8 Sort / TopK
Faster IN list evaluation up to 14.7x faster for small primitive lists, 9.7x faster for UInt8 Expressions
Prune unread Parquet leaves for nested columns Reduces reads from 1.35 TB to 30.9 GB in a production Comet query Scan / IO
Fewer object store requests for CSV 70% faster TPC-H CSV with simulated latency Scan / IO
Faster SortPreservingMerge tie-breaker 8% faster sort_tpch Q6 Sorting
Native GROUP BY on FixedSizeBinary (e.g. UUIDs) ~5% faster grouping 200M UUIDs, much less memory Aggregation

Sort Pushdown + TopK Pruning

The multi-release Sort Pushdown effort continues to optimize ORDER BY and ORDER BY ... LIMIT (TopK) queries. In DataFusion 55, as a dynamic filter threshold tightens, the Parquet reader re-evaluates the threshold against the remaining row groups and drops those that can no longer contribute (#22450), and compound ORDER BY queries are now supported. Together these reduce the total topk_tpch suite runtime by ~43%; see our Optimizing for Almost Sorted Data blog post for more details. Thanks to @zhuqi-lucas for driving this work, with reviews from @adriangb.

Aggregation Improvements

Complete Multi-Column GROUP BY Type Coverage: DataFusion's column-wise GROUP BY storage (GroupValuesColumn) has type-specific fast paths, but previously any unsupported column type forced the entire grouping onto a slower row-encoded fallback. For example, this query to deduplicate a table of UUIDs used to hit the slow path:

SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);

DataFusion 55 completes the type coverage (#22715), so the query above now runs about 5% faster on 200M UUIDs, and uses much less memory (see #23645). Thanks to @zhuqi-lucas, @tohuya6, and @maxburke for this work.

Faster Functions

DataFusion ships hundreds of built-in functions, so speeding them up improves performance for many workloads. This release optimizes dozens of functions — up to 24x faster for find_in_set and 100x for approx_distinct with low-cardinality inputs and many groups (#22768). It also includes dictionary-encoding preservation for many string functions (#23743, #23930, #24100) and new IN list specializations, such as bitmap filters for small integer types (#19241). See the full list in the appendix. Thanks to the many contributors who drove this work, especially @andygrove, @geoffreyclaude, @neilconway, @lyne7-sc, @theirix, and @haohuaijin.

Planner Improvements

Unified Distribution and Sorting Enforcement: The EnforceDistribution and EnforceSorting physical optimizer passes are now merged into a single EnsureRequirements pass with idempotent sort pushdown (#21976), fixing longstanding ordering issues between the two passes and enabling the sort pushdown work described above. Thanks to @zhuqi-lucas for this work, with reviews from @2010YOUY01 and @alamb.

Smarter Join Planning: DataFusion 55 now converts inner joins to more efficient semi joins when equivalent (#22652), eliminates LEFT/RIGHT joins with redundant sides (#23566), handles intermediate projections in outer join elimination (#22534), and reorders predicates in conjunctions using a cost heuristic (#22343). Thanks to @neilconway and @simonvandel for driving this work.

Better Scalar UDF Metadata APIs: Scalar UDFs can now declare that they are strict (they return NULL when any input is NULL) (#23148), letting the optimizer eliminate outer joins for queries that filter on a function result, and strictly order preserving (sorted input yields identically sorted output) (#23807), letting the optimizer eliminate redundant sorts on expressions such as custom casts. Thanks to @lyne7-sc and @rluvaton for this work, with reviews from @alamb, @kosiew, and @getChan.

Faster Optimizer: The optimizer continues to get faster, with improvements such as selective subquery traversal and in-place rewrites (#22298), collapsing chained projections (#22389), avoiding re-inlining expensive common subexpressions (#23459), and a faster PushDownFilter rule that modifies plans in place rather than copying them (#20002, #21668). Thanks to @adriangb, @Dandandan, @fordN, and @joroKr21 for this work.

Scan Improvements

Pruning Unread Parquet Leaves for Nested Columns:

Systems that embed DataFusion — such as DataFusion Comet, delta-rs, and Iceberg integrations — often hand DataFusion a table schema that includes only the nested subfields the query needs. For example, given a file whose events column physically holds four subfields, a table might declare only two of them:

-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
-- Table definition only refers to the first two subfields, id and name
CREATE EXTERNAL TABLE events (
  events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
)
STORED AS PARQUET LOCATION 'events.parquet';

DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all four leaves were read from the file and decoded, including the large payload and trace subfields, which were then thrown away. The Comet project reported a production query where this extra decoding caused 1.35 TB of reads, whereas plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes that gap by not reading the undeclared payload and trace leaves from the file at all (#24090). Thanks to @mbutrovich for this work, with reviews from @adriangb.

Other Scan Improvements: DataFusion 55 also skips loading the page index (and an expensive ParquetMetaData clone) when a file has no page index (#24150), supports file-level Parquet row selections (#22940), and lowers the default repartition_file_min_size from 10 MiB to 1 MiB for better parallelism on small files (#22439). Thanks to @alamb, @haohuaijin, and @adriangb.

Stability Improvements 🛡️

The community also improved DataFusion's handling of larger-than-memory aggregate workloads (e.g. #23657, #23965, #24061), building on a refactoring of the aggregation path into dedicated streams (epic #22710). Sorts under memory pressure are more resilient: when a spill merge cannot reserve enough memory, DataFusion now re-spills the largest stream in smaller batches rather than failing (#22945), and caps the merge fan-in to bound memory use (#23066). Thanks to @2010YOUY01, @EmilyMatt, @yinli-systems, @Rachelint, and @pepijnve (who fixed a subtle lost-wakeup bug in the spill pool, #23522) for this work.

New Features ✨

file_row_index() and input_file_name()

DataFusion 55 adds file_row_index (#22604) and input_file_name (#22978) functions to expose Parquet virtual columns:

> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
+---------+-------------------+------------------+
| column1 | input_file_name() | file_row_index() |
+---------+-------------------+------------------+
| 100     | tmp/foo.parquet   | 0                |
| 200     | tmp/foo.parquet   | 1                |
+---------+-------------------+------------------+

Such functions are useful for change data capture, debugging, and Spark-compatible workloads. Thanks to @mbutrovich and @AdamGS for this work (reviving earlier work from @jkylling), with reviews from @adriangb, @comphead, and @niebayes.

Range Partitioning

DataFusion 55 adds native range partitioning support, which maps rows to partitions by key ranges (rather than hash values). Query inputs are often range partitioned in real-world scenarios, such as time-series data written as one file per day or hour. DataFusion uses range partitioning information to avoid expensive repartitioning operations and push more specific dynamic filters to scans.

Data that is range partitioned declares an ordering and a list of split points. Partition i holds the keys that fall between split point i-1 and split point i:

ordering     = [date ASC NULLS LAST]
split_points = [(2022-01-01), (2023-01-01)]

partition 0: date < 2022-01-01
partition 1: 2022-01-01 <= date < 2023-01-01
partition 2: date >= 2023-01-01

For more details, please see the documentation for Partitioning::Range, the planning epic (#22395), and the design discussion (#21992). Thanks to @gene-bordegaray, @saadtajwar, @peterxcli, @stuhood, @gmhelmold, @mattp5657, @mithuncy, @JSOD11, @EdsonPetry, @Rich-T-kid, and @blinding-pixels for driving this substantial community effort.

MERGE INTO Planner Support

MERGE INTO (SQL:2003) is a widely used DML statement for upsert and conditional update workloads, and a key building block for table formats such as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types (#20763) along with SQL planner and physical planner support, and a new TableProvider::merge_into hook (#22988) so table implementations can execute merge operations:

MERGE INTO target t
USING source s
ON t.id = s.id
WHEN MATCHED AND s.deleted THEN DELETE
WHEN MATCHED THEN UPDATE SET name = s.name
WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);

Built-in table providers do not yet implement the hook, but custom TableProvider implementations (such as lakehouse table formats) can now plug in their own merge execution. More MERGE INTO improvements are planned for upcoming releases; see #20746 for details. Thanks to @wirybeaver for implementing this feature, with reviews from @alamb and @kosiew.

Pluggable Spill Backends

DataFusion spills to disk when a query exceeds its memory budget, but the spill infrastructure was previously hardwired to OS-level temporary files. DataFusion 55 introduces a pluggable SpillFile trait and TempFileFactory (#21882, #22230) so hosts can route spill data through their own storage layers — for example, extensions like ParadeDB can now integrate spilling into the Postgres buffer pool. Implement TempFileFactory and install it on the RuntimeEnv:

let runtime = RuntimeEnvBuilder::new()
    .with_disk_manager_builder(
        // register a custom TempFileFactory
        DiskManagerBuilder::default()
            .with_temp_file_factory(Arc::new(MyTempFileFactory::new())),
    )
    .build_arc()?;
let ctx = SessionContext::new_with_config_rt(SessionConfig::new(), runtime);

See the object_store_spill.rs example for a complete implementation that spills to an ObjectStore such as S3. Thanks to @pantShrey for this work, with reviews from @alamb.

Extensibility for Distributed Engines

Several new APIs make it easier to build distributed systems such as datafusion-distributed, DataFusion Ballista, and DataFusion Python on top of DataFusion:

EXPLAIN Improvements

DataFusion 55 adds a Postgres-style EXPLAIN (...) option list (#21768) and a pgjson output format for EXPLAIN ANALYZE (#21767), making plan output easier to consume with existing Postgres tooling such as plan visualizers.

EXPLAIN ANALYZE can produce many metrics, and you can narrow them down explicitly with the METRICS option. For example, to see only row counts for each plan node, use this:

> EXPLAIN (ANALYZE, METRICS 'rows', LEVEL summary)
  SELECT region_id, sum(amount) FROM orders GROUP BY region_id;

AggregateExec: mode=Single, gby=[region_id@0 as region_id], aggr=[sum(orders.amount)], metrics=[output_rows=5]
  DataSourceExec: file_groups={1 group: [[orders.parquet]]}, projection=[region_id, amount], file_type=parquet, metrics=[output_rows=100.0 K, row_groups_pruned_statistics=1 total → 1 matched, scan_efficiency_ratio=7.83% (28.30 K/361.5 K), ...]

Adding FORMAT pgjson renders the same physical plan and its live metrics as Postgres-compatible JSON, which can be pasted straight into plan visualizers such as Dalibo:

> EXPLAIN (ANALYZE, FORMAT pgjson, METRICS 'rows', LEVEL summary)
  SELECT region_id, sum(amount) FROM orders GROUP BY region_id;

[
  {
    "Plan": {
      "Node Type": "AggregateExec",
      "Actual Rows": 5,
      "Plans": [
        { "Node Type": "DataSourceExec", "Actual Rows": 100000, ... }
      ]
    }
  }
]

See the EXPLAIN usage guide for the full option list. Thanks to @adriangb for this work.

New Functions

SQL and Scalar Functions: DataFusion 55 adds new array math functions array_scale, array_add, array_subtract, array_sum, and array_avg, plus the higher-order array_first function. It also includes the new unnest_outer function (#22100), which preserves empty inputs as NULLs rather than dropping them, and approx_distinct now supports more types. Thanks to @crm26, @SubhamSinghal, @EdsonPetry, @athlcode, and @mkleen for these contributions.

Spark-Compatible Functions: The datafusion-spark crate gains new or improved Spark-compatible functions, including hypot, atan2, weekday, monthname, and concat_ws with array support, plus a new Spark SQL parser dialect config (#22529). Thanks to the contributors who drove this work, especially @KarpagamKarthikeyan, @sjhddh, @JeelRajodiya, @davidlghellin, and @kumarUjjawal.

Upgrade Guide and Changelog 📖

Upgrading to 55.0.0 should be straightforward for most users, though there are some breaking changes. See the Upgrade Guide for details and migration snippets, and the changelog for the full list of changes.

About DataFusion

Apache DataFusion is an extensible query engine, written in Rust, that uses Apache Arrow as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries, and machine learning and streaming applications. While DataFusion's primary design goal is to accelerate the creation of other data-centric systems, it provides a reasonable experience directly out of the box as a dataframe library, Python library, and command-line SQL tool.

DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone. Without DataFusion, highly performant vectorized query engines would remain the domain of a few large companies and world-class research institutions. With DataFusion, we can all build on top of a shared foundation and focus on what makes our projects unique.

How to Get Involved

DataFusion is not a project built or driven by a single person, company, or foundation. Rather, our community of users and contributors works together to build a shared technology that none of us could have built alone.

If you are interested in joining us, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code. A list of open issues suitable for beginners is here, and you can find out how to reach us on the communication doc.

Appendix: ClickBench Results

We try not to get too excited by benchmarks, though it is hard not to get caught up in Benchmaxxing. As noted above, ClickBench covers only a tiny fraction of what our users actually do, and reads local files rather than object storage. Even so, DataFusion now sits at the top of the ClickBench leaderboard for processing partitioned Parquet files, as measured by ClickBench's combined metric. Results vary slightly by engine and instance type, but DataFusion matches other state-of-the-art engines on local files and often significantly exceeds them on object storage. Plenty left to optimize, of course.

DataFusion ClickBench performance on c7a.metal-48xlarge Figure 2: ClickBench results for c7a.metal-48xlarge as of 2026-08-24; DataFusion is the fastest engine for processing partitioned Parquet files, by ClickBench's combined metric. See the ClickBench results page for the latest results.

DataFusion ClickBench performance on c6a.4xlarge Figure 3: ClickBench results for c6a.4xlarge as of 2026-08-24; DataFusion is the second fastest engine for processing partitioned Parquet files (combined metric) on this VM type. See the ClickBench results page for the latest results.

DataFusion Performance over time

Figure 4: Average and median normalized execution times for DataFusion 55.0.0 on ClickBench queries, compared to previous releases. Query times are normalized using the ClickBench definition. See the DataFusion Benchmarking Page for more details.

Appendix: Full List of Performance Improvements

The tables below list the performance improvements in this release along with a representative measurement for each. Results marked (micro) come from Criterion microbenchmarks and are not expected to translate directly into end-to-end query speedups. Speedups are reported as ratios of old to new runtime: "n% faster" means the old runtime was (100+n)% of the new, and speedups of 2x or more are reported as a multiple.

Sort / TopK

Improvement Issue / PR Representative Result
Runtime row-group pruning for TopK #23036, #22450 4.2x faster on topk_tpch Q8; 5 of 11 queries 3.6-4.2x faster
Faster SortPreservingMerge tie-breaker #23107 8% faster on sort_tpch Q6

Window Functions

Improvement Issue / PR Representative Result
LEAD / LAG with IGNORE NULLS #23711 21.7x faster for List, 10.5x for Utf8View (micro)
Sliding-window MIN / MAX monotonic deques #23827 up to 3.5x faster (strings; ~2x for numerics) (micro)
Skip fully calculated window partitions #24127 49% faster with 32,768 sparse partitions: 161ms → 108ms
Skip re-slicing quiet window partitions #24047 26% faster with 32,768 sparse partitions: 209ms → 166ms

Aggregation

Improvement Issue / PR Representative Result
approx_distinct with many groups #22768 101x faster: 1723ms → 17ms (Int64, 50K groups)
array_agg(DISTINCT ...) #23716 4.0x faster at high cardinality (micro)
percentile_cont / median #23954 2.7x faster: 247µs → 92µs (median, window=256) (micro)
Multi-column GROUP BY type coverage #22715, #23523 46% less memory for mixed-schema keys: 1096KB → 594KB
Native GROUP BY on FixedSizeBinary (e.g. UUIDs) #23645, #23646 ~5% faster: 1.13s → 1.07s grouping 200M UUIDs, addressing a reported out-of-memory crash
Semi / anti join index alignment #22794 TPC-DS Q15 16% faster

Expressions and Functions

Improvement Issue / PR Representative Result
Faster IN list evaluation #19241, #23014 up to 14.7x faster for small primitive lists (micro)
Faster IN list (UInt8 bitmap) #23011 9.7x faster: 38.4µs → 4.0µs (micro)
Faster IN list (Int8 / Int16) #23299 4.4x faster for i16, 4-element list (micro)
Preserve dictionary encoding for string functions #23930 360x faster initcap on cardinality-10 dictionaries (micro)
Preserve dictionary encoding for trim family #24100 151x faster ltrim on cardinality-10 dictionaries (micro)
Preserve dictionary encoding for (ascii, bit_length, octet_length) #23743 28.7x faster ascii (micro)
find_in_set #23460 24x faster: 1.25ms → 52µs (micro)
array_has with array needle #23337 15.1x faster; join query 0.95s → 0.059s
trunc with scalar precision #23593 12.9x faster: 9.5µs → 735ns (micro)
encode / hex #23456 5.1x faster: 200µs → 39.5µs (micro)
arrays_zip perfect-list fast path #22285 4.2x faster (micro)
overlay #22182 5.3x faster on high-null input: 401µs → 75µs (micro)
translate #22171 4.4x faster: 265µs → 60µs (micro)
date_trunc #23542 4.2x faster for week: 23.3µs → 5.6µs (micro)
replace #23589 2.5x faster: 228µs → 91µs (micro)
regexp_instr #23540 90% faster: 49.2µs → 26.0µs (micro)
left / right #23762 81% faster on string_view long results: 94µs → 52µs (micro)
round #23471 79% faster: 1486ns → 831ns (micro)
regexp_match with literal pattern #23547 59% faster with literal pattern and flags: 280µs → 176µs (micro)
date_part isodow #23491 reported 38% improvement (micro)

Planning

Improvement Issue / PR Representative Result
Collapse chained projections #22389 4.0x faster planning: 623ms → 155ms
Skip subquery traversal, rewrite in place #22298 29% faster TPC-DS optimization: 220ms → 170ms
Don't re-inline CSE'd expensive expressions #23459 67% faster on repeated power(a, 2)
Skip ensure_distribution rebuild for unchanged children #22521 2.9x faster per call: 171µs → 59µs
Unified EnsureRequirements pass #21976 TPC-H: 8 queries faster, 0 slower
Predicate reordering heuristic #22343 ClickBench Q21 10-13% faster

Scan / IO

Improvement Issue / PR Representative Result
Prune unread Parquet leaves for nested columns #24090 Reduces reads from 1.35 TB to 30.9 GB in a production Comet query
Fewer object store requests for CSV #22962 70% faster on TPC-H CSV with simulated latency
Lower repartition_file_min_size to 1 MiB #22439 TPC-H Q22 68% faster
Skip page index load when the file has none #24149, #24150 ClickBench (single file) Q1 38% faster

Comments