Apache DataFusion 56.0.0 Released

Posted on: Fri 20 November 2026 by pmc

We are proud to announce the release of DataFusion 56.0.0, which includes many new features such as ASOF joins, SQL registration of external catalogs, Avro and ALP write support, better support for extension types, and many performance and stability improvements. The complete list of changes is available in the changelog. This release represents roughly 12 weeks of development and 1200 commits. Thanks to the 247 contributors who made it possible.

New Features ✨

ASOF Joins

DataFusion 56 adds ASOF JOIN, a long-standing feature request (#318) that matches each row on the left with the closest row on the right rather than an exact match. Such joins are common in time-series workloads such as finding the most recent quote for each trade, or aligning two sensors that sample at different rates.

For example, given tables of temperature and humidity readings from two sensors that report at different times:

id location ts temp
T1 roof 2026-11-20T09:30:05 11.5
T2 roof 2026-11-20T09:30:12 11.7
T3 lobby 2026-11-20T09:30:07 20.1
id location ts humidity
H1 roof 2026-11-20T09:30:00 71.0
H2 roof 2026-11-20T09:30:10 70.5
H3 lobby 2026-11-20T09:30:10 45.2

Since no temperature reading has the exact same timestamp as any humidity reading, a regular equality join on location and ts would match nothing. Instead, an ASOF join efficiently pairs each temperature reading with the most recent humidity reading at or before it:

SELECT t.id AS temp_id, h.id AS humidity_id, t.location, t.ts, t.temp, h.humidity
FROM temperature AS t ASOF JOIN humidity AS h
MATCH_CONDITION (t.ts >= h.ts)
ON t.location = h.location
ORDER BY t.id;
+---------+-------------+----------+---------------------+------+----------+
| temp_id | humidity_id | location | ts                  | temp | humidity |
+---------+-------------+----------+---------------------+------+----------+
| T1      | H1          | roof     | 2026-11-20T09:30:05 | 11.5 | 71.0     |
| T2      | H2          | roof     | 2026-11-20T09:30:12 | 11.7 | 70.5     |
| T3      | NULL        | lobby    | 2026-11-20T09:30:07 | 20.1 | NULL     |
+---------+-------------+----------+---------------------+------+----------+
Full Example
CREATE TABLE temperature (id VARCHAR, location VARCHAR, ts TIMESTAMP, temp DOUBLE) AS VALUES
  ('T1', 'roof',  '2026-11-20T09:30:05', 11.5),
  ('T2', 'roof',  '2026-11-20T09:30:12', 11.7),
  ('T3', 'lobby', '2026-11-20T09:30:07', 20.1);

CREATE TABLE humidity (id VARCHAR, location VARCHAR, ts TIMESTAMP, humidity DOUBLE) AS VALUES
  ('H1', 'roof',  '2026-11-20T09:30:00', 71.0),
  ('H2', 'roof',  '2026-11-20T09:30:10', 70.5),
  ('H3', 'lobby', '2026-11-20T09:30:10', 45.2);

SELECT t.id AS temp_id, h.id AS humidity_id, t.location, t.ts, t.temp, h.humidity
FROM temperature AS t ASOF JOIN humidity AS h
MATCH_CONDITION (t.ts >= h.ts)
ON t.location = h.location
ORDER BY t.id;

DataFusion supports the Snowflake syntax; please see the documentation for more details. Thanks to @Xuanwo for contributing this feature and to @jayzhan211, @Kristofferjoha, @2010YOUY01, @xudong963, @alamb, and @jonathanc-n for reviews and improvements.

External Catalogs in SQL

Connecting DataFusion to an external catalog, such as an Iceberg REST catalog, previously required writing Rust code. DataFusion 56 adds support for CREATE EXTERNAL CATALOG and DROP CATALOG statements (#25112) to attach catalogs with SQL in the same way as external tables:

CREATE EXTERNAL CATALOG my_catalog
STORED AS ICEBERG
LOCATION 's3://bucket/warehouse'
OPTIONS ('catalog.uri' 'http://rest-catalog:8181', 'warehouse' 'my_catalog');

SELECT * FROM my_catalog.sales.orders LIMIT 10;

Note that DataFusion does not include any catalog implementations itself. Instead, users register a CatalogProviderFactory for each STORED AS type, and then attach catalogs with SQL; see the CREATE EXTERNAL CATALOG documentation for more details.

Thanks to @pepijnve and @TTyChud for this work.

File Format Improvements

Writing Avro: DataFusion has been able to read Apache Avro files for a long time, and DataFusion 56 adds support for writing them:

COPY orders TO 'orders.avro' STORED AS AVRO;
Full Example
CREATE TABLE orders (id INT, customer VARCHAR, total DOUBLE) AS VALUES
  (1, 'alice', 42.50),
  (2, 'bob', 17.25),
  (3, 'carol', 99.99);

-- Write the table to an Avro file
COPY orders TO 'orders.avro' STORED AS AVRO;

-- Read it back
SELECT * FROM 'orders.avro';
+----+----------+-------+
| id | customer | total |
+----+----------+-------+
| 1  | alice    | 42.5  |
| 2  | bob      | 17.25 |
| 3  | carol    | 99.99 |
+----+----------+-------+

Parquet ALP Encoding: ALP (Adaptive Lossless floating-Point) is a new Parquet encoding that often stores FLOAT and DOUBLE columns much more compactly. DataFusion 56 can read and write ALP-encoded files:

COPY measurements TO 'measurements.parquet' STORED AS PARQUET
OPTIONS (
  'format.encoding::price' 'alp',
  'format.dictionary_enabled::price' 'false'
);
Full Example
-- One million prices with two decimal places
CREATE TABLE measurements AS
SELECT i AS id, round(100000 * random(), 2) AS price
FROM generate_series(1, 1000000) AS t(i);

-- Write with the default settings, and with ALP for the price column
COPY measurements TO 'default.parquet' STORED AS PARQUET;
COPY measurements TO 'alp.parquet' STORED AS PARQUET
OPTIONS (
  'format.encoding::price' 'alp',
  'format.dictionary_enabled::price' 'false'
);

-- Compare the size of the price column in the two files.
-- Note that the ALP-encoded column is about a third smaller than with
-- the default settings.
SELECT 'default' AS file, encodings, total_compressed_size
FROM parquet_metadata('default.parquet') WHERE path_in_schema = '"price"'
UNION ALL
SELECT 'alp', encodings, total_compressed_size
FROM parquet_metadata('alp.parquet') WHERE path_in_schema = '"price"';
+---------+------------------------------+-----------------------+
| file    | encodings                    | total_compressed_size |
+---------+------------------------------+-----------------------+
| default | [PLAIN, RLE, RLE_DICTIONARY] | 4481181               |
| alp     | [RLE, ALP]                   | 3018657               |
+---------+------------------------------+-----------------------+

Thanks to @pepijnve and @Surajsuthar for this work.

More SQL Support

MATERIALIZED CTEs: By default, DataFusion inlines Common Table Expressions (CTEs), so a CTE that is referenced multiple times is evaluated multiple times. DataFusion 56 supports the Postgres WITH ... AS [NOT] MATERIALIZED syntax to evaluate a CTE once and reuse the result (#25580). The syntax is currently only recognized by the postgres SQL dialect.

Example
SET datafusion.sql_parser.dialect = 'postgres';

-- The CTE runs once, so both references see the same random value
WITH r AS MATERIALIZED (SELECT random() AS v)
SELECT a.v = b.v AS same FROM r AS a, r AS b;
+------+
| same |
+------+
| true |
+------+

-- Without MATERIALIZED, each reference evaluates random() separately
WITH r AS (SELECT random() AS v)
SELECT a.v = b.v AS same FROM r AS a, r AS b;
+-------+
| same  |
+-------+
| false |
+-------+

Thanks to @adriangb for this work.

datafusion-cli Improvements

datafusion-cli adds parquet_file_metadata and parquet_page_index table functions for inspecting Parquet files. Thanks to @jay-dee7 for this work, with reviews from @kumarUjjawal.

Versioned Documentation

The DataFusion documentation site previously always reflected the latest main branch, making it hard to tell which features were available in what version. The site now has a version picker and publishes a snapshot of the documentation for each release. Thanks to @yiaany for this work and @Omega359 for the original request.

Screenshot of the DataFusion documentation site showing the new version picker, a drop-down menu listing Development (main) and releases 55.2.0, 54.1.0, 53.0.0, 52.0.0, 51.0.0, and 50.0.0.

Figure 1: The new version picker on the DataFusion documentation site.

Improved Support for Extension Types

As described in the User Defined Types blog, DataFusion's type system can be extended using Arrow extension types to attach a logical type to an Arrow physical type. This technique allows the DataFusion ecosystem to add entirely new data types without changes to the DataFusion core (see Ecosystem Highlights below). DataFusion 56 improves this support in several ways:

  • The SQL types UUID and JSON now map to the Arrow canonical extension types arrow.uuid and arrow.json.
  • DESCRIBE, SHOW COLUMNS, and information_schema.columns show the extension type name rather than the storage type.
  • Extension types are preserved when reading Parquet files with the default settings.
  • Extension types are preserved through more operators such as casts, min / max, scalar subqueries, and INSERT.

For example, a column declared with the SQL JSON type is stored as a standard Arrow string and tagged with the arrow.json extension type:

CREATE TABLE events (doc JSON);
INSERT INTO events VALUES ('{"a":1}');
SELECT doc,
       arrow_typeof(doc)                           AS doc_type,
       arrow_metadata(doc)['ARROW:extension:name'] AS doc_extension
FROM events;
+---------+----------+---------------+
| doc     | doc_type | doc_extension |
+---------+----------+---------------+
| {"a":1} | Utf8     | arrow.json    |
+---------+----------+---------------+

Thanks to @cblp, @cyberbeam524, @paleolimbot, @M-Tesla, and @osipovartem for this work.

Ecosystem Highlights 🌱

DataFusion is designed to be extended, and this section highlights two projects in the DataFusion ecosystem that are made possible by the extension points in DataFusion 56.

Semi-Structured Data (e.g. JSON)

Parquet Variant is a type for storing semi-structured data, similar to JSON but with an efficient binary encoding, that was recently added to Apache Parquet and Apache Iceberg (see the blog on the Parquet site for more details). The datafusion-variant crate extends DataFusion to add the VARIANT SQL type and Spark-compatible functions. Register it with a single call to register_all on a SessionContext:

let mut ctx = SessionContext::new();
datafusion_variant::register_all(&mut ctx)?;

Then you can load, query, and store semi-structured data using SQL. For example, given an events table with a VARIANT column:

SELECT * FROM events;
+------------------------------------------------------------------+
| event                                                            |
+------------------------------------------------------------------+
| {"latency_ms":12,"service":"api","status":200,"tags":["cached"]} |
| {"error":null,"latency_ms":85,"service":"api","status":500}      |
| {"error":"Somethings not working","service":"api","status":504}  |
+------------------------------------------------------------------+

you can extract and filter on individual fields:

-- Extract fields with the Spark-style `:` operator
SELECT event:service::VARCHAR, event:latency_ms::BIGINT
FROM events
WHERE event:status::BIGINT >= 400;
+----------------------+-------------------------+
| events.event:service | events.event:latency_ms |
+----------------------+-------------------------+
| api                  | 85                      |
| api                  | NULL                    |
+----------------------+-------------------------+
Full Example
-- Load JSON data into a VARIANT column
CREATE TABLE events AS
SELECT parse_json(json_data) AS event
FROM (VALUES
    ('{"service":"api","status":200,"latency_ms":12,"tags":["cached"]}'),
    ('{"service":"api","status":500,"latency_ms":85,"error":null}'),
    ('{"service":"api","status":504,"error":"Somethings not working"}')
) AS input(json_data);

-- Show the contents of the table
SELECT * FROM events;

-- Extract fields with the Spark-style `:` operator
SELECT event:service::VARCHAR, event:latency_ms::BIGINT
FROM events
WHERE event:status::BIGINT >= 400;

Variant columns can be written to Parquet files with COPY statements, optionally shredding commonly used fields into typed Parquet columns for faster queries:

COPY events TO 'shredded_events.parquet' STORED AS PARQUET
OPTIONS ('variant.shredding' '{"event": "STRUCT<service VARCHAR, status BIGINT>"}');

The datafusion-variant crate is under active development; refer to the project documentation for more details. Thanks to @sdf-jkl, @friendlymatthew, @alamb, @raghuvanshraj, @getChan, and the other contributors for making it possible.

Geospatial Data

geodatafusion provides spatial extensions for DataFusion, built on the GeoArrow extension types and modeled after the PostGIS SQL API. It includes around 90 spatial functions such as ST_Distance and ST_Intersects, support for reading GeoParquet, reading and writing FlatGeobuf, and writing GeoJSON files, and Python bindings.

Install it with a single call:

let ctx = SessionContext::new();
geodatafusion::register(&ctx);

And then you can query geometries using familiar PostGIS-style SQL:

SELECT ST_Distance(ST_Point(0, 0), ST_Point(3, 4)) AS dist;
+------+
| dist |
+------+
| 5.0  |
+------+

SELECT ST_Area(ST_GeomFromText('POLYGON((0 0, 4 0, 4 4, 0 4, 0 0))')) AS area;
+------+
| area |
+------+
| 16.0 |
+------+
Full Example
-- A table of named places
CREATE TABLE places (name VARCHAR, x DOUBLE, y DOUBLE) AS VALUES
  ('cafe',    1.0, 1.0),
  ('library', 3.0, 4.0),
  ('airport', 9.0, 9.0);

-- The distance of each place from the origin, and whether it is inside
-- the 5 x 5 "downtown" square
SELECT name,
       ST_AsText(ST_Point(x, y)) AS location,
       round(ST_Distance(ST_Point(x, y), ST_Point(0, 0)), 2) AS distance,
       ST_Intersects(
         ST_Point(x, y),
         ST_GeomFromText('POLYGON((0 0, 5 0, 5 5, 0 5, 0 0))')
       ) AS downtown
FROM places
ORDER BY distance;
+---------+------------+----------+----------+
| name    | location   | distance | downtown |
+---------+------------+----------+----------+
| cafe    | POINT(1 1) | 1.41     | true     |
| library | POINT(3 4) | 5.0      | true     |
| airport | POINT(9 9) | 12.73    | false    |
+---------+------------+----------+----------+

Thanks to @kylebarron for creating geodatafusion, and to @Xuanwo, @gadomski, @ianthetechie, @sharkinsspatial, and the other contributors for their work on it.

Performance Improvements 🚀

DataFusion 56 continues to get faster across the board. Here is a representative sample of the improvements in this release.

Improvement Representative Result Area
IN / NOT IN subqueries in the SELECT list 533 ms → 2 ms Subqueries
Ordered ARRAY_AGG and STRING_AGG 10.5x faster, 59% less memory Aggregation
GROUP BY on fully sorted keys 2.8x - 5x faster (micro) Aggregation
array_agg(DISTINCT ...) with many groups up to 36x faster (micro) Aggregation
Dynamic filters across join keys up to 2.3x faster on Join Order Benchmark queries Joins
Sort-merge joins with many matches per key ~35% faster (micro) Joins
Nested loop join memory 2x lower peak memory Joins
LAST_VALUE ... IGNORE NULLS 13.15 ms → 111 µs (micro) Window
Planning SELECTs with many aggregates 5.9x faster for 1000 aggregates Planning
Debug build of datafusion from scratch ~3x faster end to end, some crates up to 10x Build

Faster Builds

A debug build of the datafusion crate and its dependencies from scratch is now about 3x faster end to end, and some individual crates, including the core datafusion crate, compile up to 10x faster (see #24329, #24325, #24330, #24326). Thanks to @Dandandan.

Faster TopK and LIMIT Queries

Queries that return only a few rows keep getting faster. DataFusion 56 can evaluate ORDER BY ... LIMIT before an outer join when the sort columns come from the preserved side, so far fewer rows are joined (#21621). For example, this query now joins only three orders rather than every order:

SELECT o.id, o.total, c.name
FROM orders AS o LEFT JOIN customers AS c ON o.customer_id = c.id
ORDER BY o.total DESC
LIMIT 3;

Parquet scans now stop after reading only the pages that are guaranteed to satisfy a LIMIT query's filter (#25000), and custom table providers can implement OFFSET pushdown, which is useful for paging through results (#25404). Thanks to @SubhamSinghal, @peterxcli, and @martin-g.

Faster Joins

Dynamic filters created by a join are now applied to other tables that are joined on the same key, and pushed through nested loop joins, skipping more data at the scan (#25255, #25030). Other join improvements include faster sort-merge joins (#26277), a nested loop join that needs half the memory (#25371), faster range joins (#24579), and rewriting joins against a single constant row into filters (#26404). Thanks to @jayzhan211, @yashrb24, @andygrove, @2010YOUY01, and @xudong963.

Faster Aggregations and Subqueries

Aggregations over already sorted inputs now avoid hashing entirely (#25150, #26370), and ordered ARRAY_AGG and STRING_AGG are over 10x faster while using less than half the memory (#24392). IN and NOT IN subqueries used as expressions (for example in the SELECT list) are now planned as a single join, which makes some queries hundreds of times faster (#25338). Thanks to @QinXi-ai, @geoffreyclaude, @lyne7-sc, @zhuqi-lucas, and @adriangb.

Faster Functions and Expressions

As in previous releases, many functions are faster, including upper, lower, translate, replace, left, right, and levenshtein, along with IN list evaluation. Thanks to @neilconway, @geoffreyclaude, @andygrove, @Rich-T-kid, and @comphead for making this happen.

Stability Improvements 🛡️

Correctness: As a foundational piece of technology, it is important that DataFusion's results can be relied on. During this release cycle the community systematically focused on finding and fixing bugs that produced incorrect results. The tracking epic (#26672) lists more than 60 such bugs that are fixed in DataFusion 56, along with the ones we are still working on. Thanks to @haohuaijin, @adriangb, @AdamGS, @matthewmturner, @viirya, @mbutrovich, and the many others who reported, fixed, and reviewed these issues.

More Robust Larger-Than-Memory Queries: Queries that spill to disk when memory runs out are more robust. Aggregations with very large groups (such as array_agg over a few keys) no longer fail after spilling (#25853), spill files for string and dictionary data are smaller (#23565, #27015), and memory accounting is more accurate (#24744, #25188, #26074). Thanks to @andygrove, @cetra3, @brancz, and @kosiew.

Upgrade Guide and Changelog 📖

DataFusion 56 upgrades to Arrow and Parquet 60 and object_store 0.14, and the minimum supported Rust version is now 1.95.0. Upgrading should be straightforward for most users. See the Upgrade Guide for details and migration snippets, and the changelog for the full list of changes.

About DataFusion

Apache DataFusion is an extensible query engine, written in Rust, that uses Apache Arrow as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries, and machine learning and streaming applications. While DataFusion's primary design goal is to accelerate the creation of other data-centric systems, it provides a reasonable experience directly out of the box as a dataframe library, Python library, and command-line SQL tool.

DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone. Without DataFusion, highly performant vectorized query engines would remain the domain of a few large companies and world-class research institutions. With DataFusion, we can all build on top of a shared foundation and focus on what makes our projects unique.

How to Get Involved

DataFusion is not a project built or driven by a single person, company, or foundation. Rather, our community of users and contributors works together to build a shared technology that none of us could have built alone.

If you are interested in joining us, we would love to have you. You can try out DataFusion on some of your own projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code. Please don't dump unreviewed AI output on us 🙏. A list of open issues suitable for beginners is here, and you can find out how to reach us on the communication doc.

If you want to know what we are planning to work on next, or to suggest what you think is important, please join the 2026 Q4 Roadmap discussion.


Comments