Apache DataFusion 56.0.0 Released
Posted on: Fri 20 November 2026 by pmc
We are proud to announce the release of DataFusion 56.0.0, which includes many
new features such as ASOF joins, SQL registration of external catalogs, Avro
and ALP write support, better support for extension types, and many performance
and stability improvements. The complete list of changes is available in the
changelog. This release represents roughly 12 weeks of development and 1200
commits. Thanks to the 247 contributors who made it possible.
New Features ✨¶
ASOF Joins¶
DataFusion 56 adds ASOF JOIN, a long-standing feature request (#318) that
matches each row on the left with the closest row on the right rather than an
exact match. Such joins are common in time-series workloads such as finding the
most recent quote for each trade, or aligning two sensors that sample at
different rates.
For example, given tables of temperature and humidity readings from two
sensors that report at different times:
| id | location | ts | temp |
|---|---|---|---|
| T1 | roof | 2026-11-20T09:30:05 | 11.5 |
| T2 | roof | 2026-11-20T09:30:12 | 11.7 |
| T3 | lobby | 2026-11-20T09:30:07 | 20.1 |
| id | location | ts | humidity |
|---|---|---|---|
| H1 | roof | 2026-11-20T09:30:00 | 71.0 |
| H2 | roof | 2026-11-20T09:30:10 | 70.5 |
| H3 | lobby | 2026-11-20T09:30:10 | 45.2 |
Since no temperature reading has the exact same timestamp as any humidity
reading, a regular equality join on location and ts would match nothing.
Instead, an ASOF join efficiently pairs each temperature reading with the most
recent humidity reading at or before it:
SELECT t.id AS temp_id, h.id AS humidity_id, t.location, t.ts, t.temp, h.humidity
FROM temperature AS t ASOF JOIN humidity AS h
MATCH_CONDITION (t.ts >= h.ts)
ON t.location = h.location
ORDER BY t.id;
+---------+-------------+----------+---------------------+------+----------+
| temp_id | humidity_id | location | ts | temp | humidity |
+---------+-------------+----------+---------------------+------+----------+
| T1 | H1 | roof | 2026-11-20T09:30:05 | 11.5 | 71.0 |
| T2 | H2 | roof | 2026-11-20T09:30:12 | 11.7 | 70.5 |
| T3 | NULL | lobby | 2026-11-20T09:30:07 | 20.1 | NULL |
+---------+-------------+----------+---------------------+------+----------+
Full Example
CREATE TABLE temperature (id VARCHAR, location VARCHAR, ts TIMESTAMP, temp DOUBLE) AS VALUES
('T1', 'roof', '2026-11-20T09:30:05', 11.5),
('T2', 'roof', '2026-11-20T09:30:12', 11.7),
('T3', 'lobby', '2026-11-20T09:30:07', 20.1);
CREATE TABLE humidity (id VARCHAR, location VARCHAR, ts TIMESTAMP, humidity DOUBLE) AS VALUES
('H1', 'roof', '2026-11-20T09:30:00', 71.0),
('H2', 'roof', '2026-11-20T09:30:10', 70.5),
('H3', 'lobby', '2026-11-20T09:30:10', 45.2);
SELECT t.id AS temp_id, h.id AS humidity_id, t.location, t.ts, t.temp, h.humidity
FROM temperature AS t ASOF JOIN humidity AS h
MATCH_CONDITION (t.ts >= h.ts)
ON t.location = h.location
ORDER BY t.id;
DataFusion supports the Snowflake syntax; please see the documentation for more details. Thanks to @Xuanwo for contributing this feature and to @jayzhan211, @Kristofferjoha, @2010YOUY01, @xudong963, @alamb, and @jonathanc-n for reviews and improvements.
External Catalogs in SQL¶
Connecting DataFusion to an external catalog, such as an Iceberg REST catalog,
previously required writing Rust code. DataFusion 56 adds support for
CREATE EXTERNAL CATALOG and DROP CATALOG statements (#25112) to attach
catalogs with SQL in the same way as external tables:
CREATE EXTERNAL CATALOG my_catalog
STORED AS ICEBERG
LOCATION 's3://bucket/warehouse'
OPTIONS ('catalog.uri' 'http://rest-catalog:8181', 'warehouse' 'my_catalog');
SELECT * FROM my_catalog.sales.orders LIMIT 10;
Note that DataFusion does not include any catalog implementations itself.
Instead, users register a CatalogProviderFactory
for each STORED AS type, and then attach catalogs with SQL; see the
CREATE EXTERNAL CATALOG documentation
for more details.
Thanks to @pepijnve and @TTyChud for this work.
File Format Improvements¶
Writing Avro: DataFusion has been able to read Apache Avro files for a long time, and DataFusion 56 adds support for writing them:
COPY orders TO 'orders.avro' STORED AS AVRO;
Full Example
CREATE TABLE orders (id INT, customer VARCHAR, total DOUBLE) AS VALUES
(1, 'alice', 42.50),
(2, 'bob', 17.25),
(3, 'carol', 99.99);
-- Write the table to an Avro file
COPY orders TO 'orders.avro' STORED AS AVRO;
-- Read it back
SELECT * FROM 'orders.avro';
+----+----------+-------+
| id | customer | total |
+----+----------+-------+
| 1 | alice | 42.5 |
| 2 | bob | 17.25 |
| 3 | carol | 99.99 |
+----+----------+-------+
Parquet ALP Encoding: ALP (Adaptive Lossless floating-Point) is a new
Parquet encoding that often stores FLOAT and DOUBLE columns much more
compactly. DataFusion 56 can read and write ALP-encoded files:
COPY measurements TO 'measurements.parquet' STORED AS PARQUET
OPTIONS (
'format.encoding::price' 'alp',
'format.dictionary_enabled::price' 'false'
);
Full Example
-- One million prices with two decimal places
CREATE TABLE measurements AS
SELECT i AS id, round(100000 * random(), 2) AS price
FROM generate_series(1, 1000000) AS t(i);
-- Write with the default settings, and with ALP for the price column
COPY measurements TO 'default.parquet' STORED AS PARQUET;
COPY measurements TO 'alp.parquet' STORED AS PARQUET
OPTIONS (
'format.encoding::price' 'alp',
'format.dictionary_enabled::price' 'false'
);
-- Compare the size of the price column in the two files.
-- Note that the ALP-encoded column is about a third smaller than with
-- the default settings.
SELECT 'default' AS file, encodings, total_compressed_size
FROM parquet_metadata('default.parquet') WHERE path_in_schema = '"price"'
UNION ALL
SELECT 'alp', encodings, total_compressed_size
FROM parquet_metadata('alp.parquet') WHERE path_in_schema = '"price"';
+---------+------------------------------+-----------------------+
| file | encodings | total_compressed_size |
+---------+------------------------------+-----------------------+
| default | [PLAIN, RLE, RLE_DICTIONARY] | 4481181 |
| alp | [RLE, ALP] | 3018657 |
+---------+------------------------------+-----------------------+
Thanks to @pepijnve and @Surajsuthar for this work.
More SQL Support¶
MATERIALIZED CTEs: By default, DataFusion inlines Common Table Expressions
(CTEs), so a CTE that is referenced multiple times is evaluated multiple times.
DataFusion 56 supports the Postgres WITH ... AS [NOT] MATERIALIZED syntax to
evaluate a CTE once and reuse the result (#25580). The syntax is currently
only recognized by the postgres SQL dialect.
Example
SET datafusion.sql_parser.dialect = 'postgres';
-- The CTE runs once, so both references see the same random value
WITH r AS MATERIALIZED (SELECT random() AS v)
SELECT a.v = b.v AS same FROM r AS a, r AS b;
+------+
| same |
+------+
| true |
+------+
-- Without MATERIALIZED, each reference evaluates random() separately
WITH r AS (SELECT random() AS v)
SELECT a.v = b.v AS same FROM r AS a, r AS b;
+-------+
| same |
+-------+
| false |
+-------+
Thanks to @adriangb for this work.
datafusion-cli Improvements¶
datafusion-cli adds parquet_file_metadata and
parquet_page_index table functions for inspecting
Parquet files. Thanks to @jay-dee7 for this work, with reviews from
@kumarUjjawal.
Versioned Documentation¶
The DataFusion documentation site previously always reflected the latest
main branch, making it hard to tell which features were available in what
version. The site now has a version picker and publishes a snapshot of the
documentation for each release. Thanks to @yiaany for this work and
@Omega359 for the original request.

Figure 1: The new version picker on the DataFusion documentation site.
Improved Support for Extension Types¶
As described in the User Defined Types blog, DataFusion's type system can be extended using Arrow extension types to attach a logical type to an Arrow physical type. This technique allows the DataFusion ecosystem to add entirely new data types without changes to the DataFusion core (see Ecosystem Highlights below). DataFusion 56 improves this support in several ways:
- The SQL types
UUIDandJSONnow map to the Arrow canonical extension typesarrow.uuidandarrow.json. DESCRIBE,SHOW COLUMNS, andinformation_schema.columnsshow the extension type name rather than the storage type.- Extension types are preserved when reading Parquet files with the default settings.
- Extension types are preserved through more operators such as casts,
min/max, scalar subqueries, andINSERT.
For example, a column declared with the SQL JSON type is stored as a standard
Arrow string and tagged with the arrow.json extension type:
CREATE TABLE events (doc JSON);
INSERT INTO events VALUES ('{"a":1}');
SELECT doc,
arrow_typeof(doc) AS doc_type,
arrow_metadata(doc)['ARROW:extension:name'] AS doc_extension
FROM events;
+---------+----------+---------------+
| doc | doc_type | doc_extension |
+---------+----------+---------------+
| {"a":1} | Utf8 | arrow.json |
+---------+----------+---------------+
Thanks to @cblp, @cyberbeam524, @paleolimbot, @M-Tesla, and @osipovartem for this work.
Ecosystem Highlights 🌱¶
DataFusion is designed to be extended, and this section highlights two projects in the DataFusion ecosystem that are made possible by the extension points in DataFusion 56.
Semi-Structured Data (e.g. JSON)¶
Parquet Variant is a type for storing semi-structured data, similar to JSON
but with an efficient binary encoding, that was recently added to Apache Parquet
and Apache Iceberg (see the blog on the Parquet site for more details). The
datafusion-variant crate extends DataFusion to add the VARIANT SQL type and
Spark-compatible functions. Register it with a single call to register_all on
a SessionContext:
let mut ctx = SessionContext::new();
datafusion_variant::register_all(&mut ctx)?;
Then you can load, query, and store semi-structured data using SQL. For
example, given an events table with a VARIANT column:
SELECT * FROM events;
+------------------------------------------------------------------+
| event |
+------------------------------------------------------------------+
| {"latency_ms":12,"service":"api","status":200,"tags":["cached"]} |
| {"error":null,"latency_ms":85,"service":"api","status":500} |
| {"error":"Somethings not working","service":"api","status":504} |
+------------------------------------------------------------------+
you can extract and filter on individual fields:
-- Extract fields with the Spark-style `:` operator
SELECT event:service::VARCHAR, event:latency_ms::BIGINT
FROM events
WHERE event:status::BIGINT >= 400;
+----------------------+-------------------------+
| events.event:service | events.event:latency_ms |
+----------------------+-------------------------+
| api | 85 |
| api | NULL |
+----------------------+-------------------------+
Full Example
-- Load JSON data into a VARIANT column
CREATE TABLE events AS
SELECT parse_json(json_data) AS event
FROM (VALUES
('{"service":"api","status":200,"latency_ms":12,"tags":["cached"]}'),
('{"service":"api","status":500,"latency_ms":85,"error":null}'),
('{"service":"api","status":504,"error":"Somethings not working"}')
) AS input(json_data);
-- Show the contents of the table
SELECT * FROM events;
-- Extract fields with the Spark-style `:` operator
SELECT event:service::VARCHAR, event:latency_ms::BIGINT
FROM events
WHERE event:status::BIGINT >= 400;
Variant columns can be written to Parquet files with COPY statements,
optionally shredding commonly used fields into typed
Parquet columns for faster queries:
COPY events TO 'shredded_events.parquet' STORED AS PARQUET
OPTIONS ('variant.shredding' '{"event": "STRUCT<service VARCHAR, status BIGINT>"}');
The datafusion-variant crate is under active development; refer to the
project documentation for more details. Thanks to
@sdf-jkl, @friendlymatthew, @alamb, @raghuvanshraj, @getChan, and the
other contributors for making it possible.
Geospatial Data¶
geodatafusion provides spatial extensions for DataFusion, built on the
GeoArrow extension types and modeled after the PostGIS SQL API. It includes
around 90 spatial functions such as ST_Distance and
ST_Intersects, support for reading GeoParquet, reading and
writing FlatGeobuf, and writing GeoJSON files, and Python bindings.
Install it with a single call:
let ctx = SessionContext::new();
geodatafusion::register(&ctx);
And then you can query geometries using familiar PostGIS-style SQL:
SELECT ST_Distance(ST_Point(0, 0), ST_Point(3, 4)) AS dist;
+------+
| dist |
+------+
| 5.0 |
+------+
SELECT ST_Area(ST_GeomFromText('POLYGON((0 0, 4 0, 4 4, 0 4, 0 0))')) AS area;
+------+
| area |
+------+
| 16.0 |
+------+
Full Example
-- A table of named places
CREATE TABLE places (name VARCHAR, x DOUBLE, y DOUBLE) AS VALUES
('cafe', 1.0, 1.0),
('library', 3.0, 4.0),
('airport', 9.0, 9.0);
-- The distance of each place from the origin, and whether it is inside
-- the 5 x 5 "downtown" square
SELECT name,
ST_AsText(ST_Point(x, y)) AS location,
round(ST_Distance(ST_Point(x, y), ST_Point(0, 0)), 2) AS distance,
ST_Intersects(
ST_Point(x, y),
ST_GeomFromText('POLYGON((0 0, 5 0, 5 5, 0 5, 0 0))')
) AS downtown
FROM places
ORDER BY distance;
+---------+------------+----------+----------+
| name | location | distance | downtown |
+---------+------------+----------+----------+
| cafe | POINT(1 1) | 1.41 | true |
| library | POINT(3 4) | 5.0 | true |
| airport | POINT(9 9) | 12.73 | false |
+---------+------------+----------+----------+
Thanks to @kylebarron for creating geodatafusion, and to @Xuanwo,
@gadomski, @ianthetechie, @sharkinsspatial, and the other contributors
for their work on it.
Performance Improvements 🚀¶
DataFusion 56 continues to get faster across the board. Here is a representative sample of the improvements in this release.
| Improvement | Representative Result | Area |
|---|---|---|
IN / NOT IN subqueries in the SELECT list |
533 ms → 2 ms | Subqueries |
Ordered ARRAY_AGG and STRING_AGG |
10.5x faster, 59% less memory | Aggregation |
GROUP BY on fully sorted keys |
2.8x - 5x faster (micro) | Aggregation |
array_agg(DISTINCT ...) with many groups |
up to 36x faster (micro) | Aggregation |
| Dynamic filters across join keys | up to 2.3x faster on Join Order Benchmark queries | Joins |
| Sort-merge joins with many matches per key | ~35% faster (micro) | Joins |
| Nested loop join memory | 2x lower peak memory | Joins |
LAST_VALUE ... IGNORE NULLS |
13.15 ms → 111 µs (micro) | Window |
Planning SELECTs with many aggregates |
5.9x faster for 1000 aggregates | Planning |
Debug build of datafusion from scratch |
~3x faster end to end, some crates up to 10x | Build |
Faster Builds¶
A debug build of the datafusion crate and its dependencies from scratch is
now about 3x faster end to end, and some individual crates, including the core
datafusion crate, compile up to 10x faster (see #24329, #24325, #24330,
#24326). Thanks to @Dandandan.
Faster TopK and LIMIT Queries¶
Queries that return only a few rows keep getting faster. DataFusion 56 can
evaluate ORDER BY ... LIMIT before an outer join when the sort columns come
from the preserved side, so far fewer rows are joined (#21621). For example,
this query now joins only three orders rather than every order:
SELECT o.id, o.total, c.name
FROM orders AS o LEFT JOIN customers AS c ON o.customer_id = c.id
ORDER BY o.total DESC
LIMIT 3;
Parquet scans now stop after reading only the pages that are guaranteed to
satisfy a LIMIT query's filter (#25000), and custom table providers can
implement OFFSET pushdown, which is useful for paging through results
(#25404). Thanks to @SubhamSinghal, @peterxcli, and @martin-g.
Faster Joins¶
Dynamic filters created by a join are now applied to other tables that are joined on the same key, and pushed through nested loop joins, skipping more data at the scan (#25255, #25030). Other join improvements include faster sort-merge joins (#26277), a nested loop join that needs half the memory (#25371), faster range joins (#24579), and rewriting joins against a single constant row into filters (#26404). Thanks to @jayzhan211, @yashrb24, @andygrove, @2010YOUY01, and @xudong963.
Faster Aggregations and Subqueries¶
Aggregations over already sorted inputs now avoid hashing entirely
(#25150, #26370), and ordered ARRAY_AGG and STRING_AGG are over 10x
faster while using less than half the memory (#24392). IN and NOT IN
subqueries used as expressions (for example in the SELECT list) are now
planned as a single join, which makes some queries hundreds of times faster
(#25338). Thanks to @QinXi-ai, @geoffreyclaude, @lyne7-sc,
@zhuqi-lucas, and @adriangb.
Faster Functions and Expressions¶
As in previous releases, many functions are faster, including upper, lower,
translate, replace, left, right, and levenshtein, along with IN
list evaluation. Thanks to @neilconway, @geoffreyclaude, @andygrove,
@Rich-T-kid, and @comphead for making this happen.
Stability Improvements 🛡️¶
Correctness: As a foundational piece of technology, it is important that DataFusion's results can be relied on. During this release cycle the community systematically focused on finding and fixing bugs that produced incorrect results. The tracking epic (#26672) lists more than 60 such bugs that are fixed in DataFusion 56, along with the ones we are still working on. Thanks to @haohuaijin, @adriangb, @AdamGS, @matthewmturner, @viirya, @mbutrovich, and the many others who reported, fixed, and reviewed these issues.
More Robust Larger-Than-Memory Queries: Queries that spill to disk when
memory runs out are more robust. Aggregations with very large groups (such as
array_agg over a few keys) no longer fail after spilling (#25853), spill
files for string and dictionary data are smaller (#23565, #27015), and
memory accounting is more accurate (#24744, #25188, #26074). Thanks to
@andygrove, @cetra3, @brancz, and @kosiew.
Upgrade Guide and Changelog 📖¶
DataFusion 56 upgrades to Arrow and Parquet 60 and
object_store 0.14, and the minimum supported Rust
version is now 1.95.0. Upgrading should be straightforward for most users. See
the Upgrade Guide for details and migration snippets, and the changelog for
the full list of changes.
About DataFusion¶
Apache DataFusion is an extensible query engine, written in Rust, that uses Apache Arrow as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries, and machine learning and streaming applications. While DataFusion's primary design goal is to accelerate the creation of other data-centric systems, it provides a reasonable experience directly out of the box as a dataframe library, Python library, and command-line SQL tool.
DataFusion's core thesis is that, as a community, together we can build much more advanced technology than any of us as individuals or companies could build alone. Without DataFusion, highly performant vectorized query engines would remain the domain of a few large companies and world-class research institutions. With DataFusion, we can all build on top of a shared foundation and focus on what makes our projects unique.
How to Get Involved¶
DataFusion is not a project built or driven by a single person, company, or foundation. Rather, our community of users and contributors works together to build a shared technology that none of us could have built alone.
If you are interested in joining us, we would love to have you. You can try out DataFusion on some of your own projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code. Please don't dump unreviewed AI output on us 🙏. A list of open issues suitable for beginners is here, and you can find out how to reach us on the communication doc.
If you want to know what we are planning to work on next, or to suggest what you think is important, please join the 2026 Q4 Roadmap discussion.