# Upgrade Guides ## DataFusion 56.0.0 **Note:** DataFusion `56.0.0` has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version. ### Session time zone now applies to mixed timestamp operations When `datafusion.execution.time_zone` is set, DataFusion now uses it to interpret timezone-naive timestamps in comparisons and subtraction with timezone-aware timestamps. This applies to binary comparisons, `IN`, `BETWEEN`, `CASE expr WHEN`, `IN` subqueries, and `ANY` or `ALL` comparisons. Previously, mixed timestamp operations did not consistently use the configured session timezone. They could use the timezone-aware operand's timezone or subtract raw timestamp values. The new behavior matches PostgreSQL and DuckDB. The default remains `None`. Without a session timezone, comparisons and subtraction use the timezone-aware operand's timezone. Applications that set this option need to review queries that mix timezone-aware and timezone-naive timestamps. ### `get_coerce_type_for_list` and `get_coerce_type_for_case_when` are deprecated `IN` and `CASE expr WHEN` are defined in terms of `=`, and `=` now reads a timezone-naive timestamp in `datafusion.execution.time_zone` when it is compared against a timezone-aware one. These two functions have nowhere to take that setting from, so the planner no longer calls them and they now return a coercion that disagrees with the comparison operator they are defined in terms of. Both remain in place and behave exactly as they did in 55.0.0. To follow the planner, fold `comparison_coercion_with_session_timezone` over the values yourself: ```rust // 55.0.0 let coerced = get_coerce_type_for_list(&expr_type, &list_types); // 56.0.0 let coerced = list_types.iter().try_fold(expr_type, |left, right| { comparison_coercion_with_session_timezone(&left, right, session_time_zone) }); ``` Pass `options.execution.time_zone.as_ref().map(|tz| tz.as_str())` as `session_time_zone`, or `None` where no session is in scope. `None` reproduces the 55.0.0 behavior, except that `NULL` against a dictionary-encoded type coerces to the dictionary type rather than to its value type, which is what `=` has always done. ### `MSRV` updated to 1.95.0 The Minimum Supported Rust Version (MSRV) has been updated to [`1.95.0`]. [`1.95.0`]: https://releases.rs/docs/1.95.0/ ### Aggregate group-clustering APIs `AggregateExec::input_order_mode()` has been replaced by `AggregateExec::group_clustering_mode()`. It returns a `GroupClusteringMode`: `None`, `Partial(indices)`, or `Full`, corresponding to the previous aggregate modes `Linear`, `PartiallySorted(indices)`, and `Sorted`. `AggregateExec::compute_properties` now accepts `&GroupClusteringMode` in place of `&InputOrderMode`. `GroupClusteringMode` describes the input's group-clustering guarantees, which determine when groups can be emitted during aggregation. In `datafusion_physical_plan::aggregates::order`, `GroupOrdering`, `GroupOrderingPartial`, and `GroupOrderingFull` have been renamed to `GroupClustering`, `GroupClusteringPartial`, and `GroupClusteringFull`. `GroupClustering::try_new` accepts `&GroupClusteringMode`. Aggregate plans now display `group_clustering_mode=Full` or `group_clustering_mode=Partial(indices)` instead of `ordering_mode=Sorted` or `ordering_mode=PartiallySorted(indices)`. ### Upgrade arrow/parquet to 60.0.0 and object_store to 0.14.2 DataFusion 56.0.0 uses `arrow` and `parquet` 60.0.0, and `object_store` 0.14.2. This may require updates to your Cargo.toml if you have direct dependencies on these crates. See the [Arrow 60.0.0 release notes] and the [object_store 0.14.2 upgrade guide] for details on breaking changes in those versions. [arrow 60.0.0 release notes]: https://github.com/apache/arrow-rs/releases/tag/60.0.0 [object_store 0.14.2 upgrade guide]: https://github.com/apache/arrow-rs-object-store/blob/v0.14.2/CHANGELOG.md #### Field and schema metadata now use arrow's `Metadata` type Arrow 60 introduced a dedicated [`Metadata`] type for field and schema metadata in place of `HashMap`. Following that change, several DataFusion APIs now use `Metadata` as well, such as `DFSchema::metadata` and `ExprSchema::metadata` return `&Metadata`. `CastExpr::target_metadata` and `TryCastExpr::target_metadata` return `Option<&Metadata>`. **Migration guide:** ```rust,ignore // Before let meta: &HashMap = schema.metadata(); let field = Field::new("a", DataType::Int64, true) .with_metadata([("k".to_string(), "v".to_string())].into()); // After let meta: &Metadata = schema.metadata(); let field = Field::new("a", DataType::Int64, true) .with_metadata(Metadata::new().with("k", "v")); ``` `Metadata` supports `.get()`, `.iter()`, `.is_empty()` and `.extend()` like `HashMap`, and converts to and from `HashMap` and `BTreeMap` via `From`, so most call sites need only a type change. Use `FieldMetadata::to_hashmap` where a `HashMap` is still required. [`metadata`]: https://docs.rs/arrow/60.0.0/arrow/datatypes/struct.Metadata.html ### `CsvSource::newlines_in_values` is deprecated `CsvSource::newlines_in_values()` is deprecated starting in 56.0.0 but remains callable with unchanged behavior. There is no replacement getter, and `CsvSource` does not expose a `csv_options()` method. **Migration guide:** Retain the `CsvOptions` used to configure the source before passing them to `CsvSource::with_csv_options`. Read `options.newlines_in_values.unwrap_or(false)` to obtain the same boolean value: `None` and `Some(false)` mean false, while `Some(true)` means true. CSV reading, repartitioning, and serialization behavior are unchanged. ### `ParquetAccessPlan::into_overall_row_selection` is deprecated `ParquetAccessPlan::into_overall_row_selection` is deprecated starting in 56.0.0. DataFusion now keeps selections local to each row group during Parquet scanning. The method remains available with its existing conversion and selection-length validation behavior, so direct callers continue to work but receive a deprecation warning. Users who attach a `ParquetAccessPlan` to a `PartitionedFile` for DataFusion to read need no changes. For custom readers, use `ParquetAccessPlan::into_inner()` to obtain the `RowGroupAccess` entries. Their positions are the original file row-group indexes, and each selection uses coordinates within its own row group. When migrating away from the deprecated method, callers must handle conversion and selection-length validation themselves. Parquet's push decoder and async reader accept `RowGroupSelection` values through `with_row_group_selections`; the synchronous reader still requires `with_row_groups` and an overall `with_row_selection`. ### `RowGroupAccessPlanFilter::prune_by_range` takes a `RowGroupRangeAssignment` The new `datafusion.execution.parquet.row_group_range_assignment` option decides which byte range of a split Parquet file reads each row group. The default, `start_offset`, keeps the previous behavior. `midpoint` matches Apache Spark. `RowGroupAccessPlanFilter::prune_by_range` takes the assignment as a new argument. Pass `RowGroupRangeAssignment::StartOffset` for the previous behavior: ```rust,ignore // Before filter.prune_by_range(groups, &range); // After filter.prune_by_range(groups, &range, RowGroupRangeAssignment::StartOffset); ``` `ParquetOptions` and its generated protobuf struct gained a `row_group_range_assignment` field. Exhaustive struct literals must set it or use `..Default::default()`. ### Missing Parquet null counts are treated as unknown DataFusion now preserves an omitted Parquet `null_count` statistic as unknown. Previously, it assumed zero nulls, which could discard matching NULL rows in `IS NULL` filters and `ORDER BY ... NULLS FIRST LIMIT` queries, or produce an incorrectly high `COUNT(column)` from metadata. Queries over affected files may read or sort more data because these optimizations require a known null count: - Pruning row groups for `IS NULL` filters. - Computing `COUNT(column)` using only file metadata. - Pruning row groups with a dynamic `NULLS FIRST` TopK filter. - Eliminating a sort over nullable columns in sorted, non-overlapping files. Such queries may now retain a full `SortExec`, even when the data has no NULLs. parquet-rs versions before 53.1.0 omitted zero null counts. This includes files produced with the older parquet-rs dependency used by DataFusion releases before 42.1.0. [Arrow #6490](https://github.com/apache/arrow-rs/pull/6490) changed the writer to record known zero counts. Files with explicit counts retain their existing behavior. To identify column chunks with bounds but no null count, run this query in the DataFusion CLI: ```sql SELECT row_group_id, path_in_schema FROM parquet_metadata('data.parquet') WHERE (stats_min IS NOT NULL OR stats_max IS NOT NULL) AND stats_null_count IS NULL; ``` Rewriting affected files with a current DataFusion version, for example using `COPY ... TO` with Parquet statistics enabled, records the missing counts and restores optimizations that depend on them. Preserve the required data ordering and ordering metadata when rewriting sorted files. ### `Timestamp(u, Some(tz)) - Timestamp(u, None)` reads the naive operand in `tz` Subtracting a timezone-naive timestamp from a timezone-aware one _at the same time unit_ did not coerce at all in `55.0.0`: the naive operand was read as UTC, which differed both from `=` on the same pair of values and from every other pairing of time units. It now coerces like the rest, so the naive operand is read in the aware operand's time zone: ```sql -- ts_tz is 2024-11-01T00:00:00-04:00 (America/New_York), ts is the naive -- 2024-11-01T00:00:00; ts_tz = ts is true in both versions -- 55.0.0: 0 days 4 hours 0 mins 0.000000000 secs -- 56.0.0: 0 days 0 hours 0 mins 0.000000000 secs SELECT ts_tz - ts FROM timestamps; ``` ### `ForeignSession::create_physical_plan` is unsupported `ForeignSession::create_physical_plan` no longer forwards to the library that owns the session. It now returns a `NotImplemented` error because forwarding can re-enter an installed foreign planner, and the execution-plan handle returned by the old callback cannot restore local Rust type identities for downcasting. The original `FFI_SessionRef` callback slot remains in place for ABI compatibility with DataFusion 55 consumers, but calling that callback returns the same error. The session-owning library should instead export its original planner as a `datafusion_ffi::query_planner::FFI_QueryPlanner` before installing a foreign planner. The foreign planner can retain and invoke that handle to receive a serialized physical plan reconstructed with local type identities. See the `datafusion_ffi::query_planner` module documentation for the complete delegation pattern. `ForeignSession::query_planner`, `optimize`, and `physical_optimizers` continue to forward to the owning session across the FFI boundary. ### Range partitioning supports retained samples and scaling `RangePartitioning::new` is deprecated. Prefer the validated `RangePartitioning::try_new_with_samples(ordering, samples, partition_count)`. For exact boundaries, pass `split_points.len() + 1` as the count, or keep using the validated `try_new` constructor. Retain additional samples when future scaling above that count is required; scaling does not create new sample values. `scale` returns `Option`. It returns `None` when the target partition count is zero or exceeds the retained sample capacity; callers can then retain the existing layout or choose another partitioning. Equality includes retained samples; use `has_same_layout` to compare the current key ordering and effective boundaries. `FFI_RangePartitioning` has a new layout; rebuild FFI consumers against the compatible release. The generated `PhysicalRangePartitioning` protobuf struct also gains fields, affecting exhaustive struct literals. Older protobuf payloads remain readable, and effective split points remain available to older readers. ### `GroupColumn` now requires `values_preserving` Custom implementations of the public `GroupColumn` trait must implement `values_preserving`. This method returns selected rows without changing the stored values or their group indices. It preserves the requested order and supports repeated indices. **Migration guide:** Implement `values_preserving`, call `selection.validate_num_groups(self.len())?` before reading, and return rows in `selection.iter()` order without changing the builder state. ### `datafusion-proto`: common option and constraint conversions are fallible Protobuf conversions for `CsvOptions`, `JsonOptions`, `ParquetCdcOptions`, `Constraint`, and `Constraints` now reject integer values that do not fit `usize`. Their infallible `From` implementations have been replaced with `TryFrom`. The existing `TryFrom` conversions for `ParquetOptions` and `TableParquetOptions` now also validate all `usize`-backed fields. A constraint without a `constraint_mode` returns an error instead of panicking. **Migration guide:** ```rust,ignore // Before let csv = CsvOptions::from(&proto_csv); let cdc = ParquetCdcOptions::from(proto_cdc); let constraints: Constraints = proto_constraints.into(); // After let csv = CsvOptions::try_from(&proto_csv)?; let cdc = ParquetCdcOptions::try_from(proto_cdc)?; let constraints = Constraints::try_from(proto_constraints)?; ``` See [issue #24170](https://github.com/apache/datafusion/issues/24170) for details. ### `CastExpr::cast_options` now returns `CastOptions` by value `CastExpr` now owns format strings decoded from protobuf, so `cast_options()` returns a borrowing `CastOptions<'_>` value instead of `&CastOptions<'static>`. Code that only reads the options generally needs no change. To replace only the child, use `PhysicalExpr::with_new_children` to preserve the options and whether the target is type-only or explicit. Use `CastExpr::with_new_expr_and_target_field` when replacing both the child and an explicit target field. Neither method requires a `'static` borrow of the options. ### `ExecutionOptions` has a new `enable_nlj_coordinated_fallback` field `ExecutionOptions` gained a public `enable_nlj_coordinated_fallback: bool` field (default `true`). It controls whether the memory-limited `NestedLoopJoinExec` fallback shares per-chunk build-side state across probe partitions, which is what lets `LEFT`, `LEFT SEMI`, `LEFT ANTI`, `LEFT MARK` and `FULL` joins spill instead of failing with `ResourcesExhausted` when the right side has several partitions. **Who is affected:** - Users constructing `ExecutionOptions` (or `ConfigOptions`) with an exhaustive struct literal. Reading or `SET`-ing configuration is unaffected. - Distributed engines that run each output partition as an independent task. The coordination assumes all probe partitions run in one process; with one coordinator per task the shared probe-thread counter never reaches zero and the fallback would stall. Such engines should set the flag to `false`, which keeps the previous fail-fast behaviour for the affected join types. **Migration guide:** Set the new field, or fill it from `Default`: ```rust,ignore // Before ExecutionOptions { batch_size: 8192, // ... every other field ... } // After: set it explicitly ExecutionOptions { batch_size: 8192, enable_nlj_coordinated_fallback: true, // ... every other field ... } // After: or let the remaining fields come from Default, which also keeps // future field additions from breaking the literal ExecutionOptions { batch_size: 8192, ..Default::default() } ``` Distributed engines opting out: ```rust,ignore let mut config = SessionConfig::new(); config.options_mut().execution.enable_nlj_coordinated_fallback = false; ``` ### `ExecutionOptions::time_zone` is now a validated `ConfigTimeZone` `ExecutionOptions::time_zone` changed from `Option` to `Option`. The value is now checked when it is set, so `SET TIME ZONE = 'Asia/Taipei2'` fails immediately instead of on a later query. The string is kept exactly as written (`+08` stays `+08`). **Who is affected:** - Code that reads or assigns `ExecutionOptions::time_zone` directly. - Code that sets the value through SQL `SET` or `ConfigOptions::set` needs no code changes, but invalid time zones are now rejected immediately. **Migration guide:** ```rust,ignore // Before options.execution.time_zone = Some("America/Denver".to_string()); let tz: Option = options.execution.time_zone.clone(); // After options.execution.time_zone = Some("America/Denver".parse()?); let tz: Option<&str> = options.execution.time_zone.as_ref().map(|tz| tz.as_str()); ``` ### `ParquetOptions::coerce_int96_tz` is now a validated `ConfigTimeZone` `ParquetOptions::coerce_int96_tz` changed from `Option` to `Option`, the same type as `ExecutionOptions::time_zone`. An invalid time zone is now rejected by `SET`, by table options, and by protobuf decoding instead of on the first Parquet read. This applies even when `coerce_int96` is not set, where the time zone has no effect. Valid values and the protobuf wire format are unchanged. **Who is affected:** - Code that reads or assigns `ParquetOptions::coerce_int96_tz` directly. - Code that sets the value through SQL or `ConfigOptions::set` needs no code changes, but invalid time zones are now rejected immediately. **Migration guide:** ```rust,ignore // Before options.execution.parquet.coerce_int96_tz = Some("UTC".to_string()); // After options.execution.parquet.coerce_int96_tz = Some("UTC".parse()?); ``` ### `ExecutionOptions` has a new `soft_max_bytes_per_output_file` field `ExecutionOptions` gained a public `soft_max_bytes_per_output_file: ConfigNonZeroUsize` field (default 4 GiB minus one byte, or 4,294,967,295 bytes). When writing multiple output files, DataFusion starts a new file once a writer reports that the encoded size has reached this target or the existing `soft_max_rows_per_output_file` target has been reached. The byte limit is soft: buffered batches, in-flight batch serialization, individual batch sizes, and file metadata can make a completed file larger than the target. It does not apply to single-file output. **Who is affected:** - Users constructing `ExecutionOptions` (or `ConfigOptions`) with an exhaustive struct literal must provide the new field. - Users writing multiple output files can set a smaller byte target to rotate files before they reach the default size. **Migration guide:** Set the field explicitly, or fill it from `Default` as shown above. For example, to target 128 MiB output files in Rust: ```rust,ignore let mut config = SessionConfig::new(); config.options_mut().execution.soft_max_bytes_per_output_file = ConfigNonZeroUsize::try_new(128 * 1024 * 1024)?; ``` The same setting can be changed with SQL: ```sql SET datafusion.execution.soft_max_bytes_per_output_file = 134217728; ``` ### New field `enable_rle_to_dictionary` added to `ParquetOptions` `ParquetOptions` gained a new field `enable_rle_to_dictionary: bool` (default `false`). It controls whether top-level string and binary Parquet columns with dictionary pages are read as `Dictionary` / `Dictionary` instead of their plain type. `ParquetOptions` is a public struct without `#[non_exhaustive]`, so downstream crates that construct it with an exhaustive struct literal will fail to compile with `E0063: missing field 'enable_rle_to_dictionary'`. **Migration guide:** Add `enable_rle_to_dictionary: false` to any exhaustive `ParquetOptions { .. }` literal, or use `..Default::default()` to future-proof against further additions: ```rust,ignore // Before ParquetOptions { enable_page_index: true, // ... every other field ... } // After: set it explicitly ParquetOptions { enable_page_index: true, enable_rle_to_dictionary: false, // ... every other field ... } // After: or let remaining fields come from Default ParquetOptions { enable_page_index: true, ..Default::default() } ``` ### `datafusion.optimizer.use_statistics_registry` is deprecated and ignored The `datafusion.optimizer.use_statistics_registry` config flag is deprecated and now ignored (setting it emits a deprecation warning). The pluggable `StatisticsRegistry` is always consulted during the single statistics walk: providers registered on the session take effect directly. With no providers registered the registry is a no-op, so default behavior is unchanged from the previous default (`use_statistics_registry = false`). **Who is affected:** - Anyone who set `datafusion.optimizer.use_statistics_registry`. The setting is still accepted (with a deprecation warning) but has no effect, and will be removed in a future release. **Migration guide:** - Drop any `use_statistics_registry` setting. - To use statistics providers, register them on the session with `SessionStateBuilder::with_statistics_registry(...)`. - To opt out, register nothing (the default). ### Generated protobuf `JsonWriterOptions` gained a `compression_level` field The generated public protobuf `JsonWriterOptions` struct now carries an optional `compression_level` so JSON sink plans preserve explicitly configured compression levels across serialization. **Who is affected:** - Users constructing generated `JsonWriterOptions` values with an exhaustive struct literal. **Migration guide:** Set `compression_level` explicitly, or fill it from `Default`: ```rust,ignore // Before JsonWriterOptions { compression } // After JsonWriterOptions { compression, compression_level: None, } // or JsonWriterOptions { compression, ..Default::default() } ``` The protobuf wire format remains backward compatible. See [PR #24945](https://github.com/apache/datafusion/pull/24945) for details. ### Generated protobuf Parquet structs gained state fields The generated public protobuf `ParquetScanExecNode` struct now carries an optional `metadata_size_hint`, and `ParquetSink` now carries optional `sorting_columns`. These fields preserve the corresponding Parquet source and sink settings across physical-plan serialization. **Who is affected:** - Users constructing either generated struct with an exhaustive struct literal. **Migration guide:** Add `metadata_size_hint: None` to `ParquetScanExecNode` literals and `sorting_columns: None` to `ParquetSink` literals to retain the previous behavior, or use `..Default::default()` for unspecified fields. The protobuf wire format remains backward compatible. See [PR #25057](https://github.com/apache/datafusion/pull/25057) for details. ### Generated protobuf `CsvWriterOptions` changed The generated `CsvWriterOptions` now includes `compression_level`, `timestamp_tz_format`, and `terminator`. Its four existing format fields changed from `String` to `Option` to distinguish unset from explicitly empty formats. This affects exhaustive struct literals and direct format-field access. Wrap existing formats in `Some` or use `None`, and initialize the new fields (explicitly or through `Default`): ```rust,ignore // Before (unchanged fields omitted) CsvWriterOptions { date_format: "%Y-%m-%d".to_string(), datetime_format: String::new(), timestamp_format: String::new(), time_format: String::new(), // ... } // After CsvWriterOptions { date_format: Some("%Y-%m-%d".to_string()), datetime_format: None, timestamp_format: None, time_format: None, compression_level: None, timestamp_tz_format: None, terminator: Vec::new(), // ... } ``` The protobuf wire format remains backward compatible. See [PR #25058](https://github.com/apache/datafusion/pull/25058) for details. ### `DefaultStatisticsProvider` is deprecated `datafusion_physical_plan::operator_statistics::DefaultStatisticsProvider` is deprecated. It is redundant: the statistics walk (`StatisticsContext`) falls back to each operator's `statistics_from_inputs` when the provider chain delegates or is empty, so a terminal "default" provider is no longer needed. It is no longer part of `StatisticsRegistry::default_with_builtin_providers()`. **Migration guide:** - Remove `DefaultStatisticsProvider` from any custom provider chain; register no terminal provider instead (the walk falls back on its own). ### The bundled statistics providers are deprecated `StatisticsRegistry::default_with_builtin_providers()` and all the statistics providers bundled in `datafusion_physical_plan::operator_statistics` (`FilterStatisticsProvider`, `ProjectionStatisticsProvider`, `PassthroughStatisticsProvider`, `AggregateStatisticsProvider`, `JoinStatisticsProvider`, `LimitStatisticsProvider` and `UnionStatisticsProvider`) are deprecated. They duplicate or replace the estimation each operator already does in `statistics_from_inputs`, which is the default. `StatisticsRegistry` remains the extension point for user-defined providers. **Migration guide:** - Stop registering the bundled providers; with no matching provider, the statistics walk uses the operator's own estimate. - To keep a bundled estimation technique, implement it in a user-defined `StatisticsProvider`. ### `StatisticsRegistry::compute` and `compute_base` are deprecated Use the walk instead: ```rust,ignore StatisticsContext::new_with_registry(registry) .compute_extended(plan, &StatisticsArgs::new())?; // or .compute(...) for core Statistics ``` ### API change for `floor` and `ceil` UDF The output type of the `floor` and `ceil` UDFs has been changed from the exact input type to a rescaled type with the same bit width. For example, for input `Decimal32(7,2)` `floor` now returns `Decimal32(6,0)`, where the new precision is `p - s + 1`, matching Spark's behaviour. See [#24703] for more details. **Who is affected:** - Users storing query result with these UDFs in a fixed schema - Users relying on `arrow_typeof` for these UDFs **Migration guide:** Change the expected type or wrap the expression in `CAST`. It's recommended to avoid relying on decimal's exact precision and scale. ### `map_extract` / `element_at` return an empty list for absent keys `map_extract` (and its alias `element_at`) previously returned a single-element list containing `NULL` when the key was not present in the map. It now returns an empty list, matching the documented behavior and DuckDB. Two related cases changed at the same time, also matching DuckDB: - A `NULL` map input now yields `NULL` instead of `[NULL]`. - A `NULL` lookup key now yields `[]` instead of `[NULL]`. A key that is present with a `NULL` value still returns `[NULL]`, so absent keys and `NULL` values are now distinguishable. **Migration guide:** ```sql -- Before SELECT map_extract(MAP {'a': 1}, 'missing'); -- [NULL] -- After SELECT map_extract(MAP {'a': 1}, 'missing'); -- [] ``` Expressions that assumed the result always has exactly one element, for example by checking its length, unnesting it, or comparing it to `[NULL]`, should treat an empty list as the absent-key case instead. See [issue #24981](https://github.com/apache/datafusion/issues/24981) and [issue #24983](https://github.com/apache/datafusion/issues/24983) for details. ### `date_bin` rejects strides that are not positive `date_bin` now returns an error when its stride is not greater than zero, matching [PostgreSQL] for fixed strides. PostgreSQL does not support month strides, and DataFusion applies the same rule to them. Negative strides were previously accepted but did not produce well-defined bins; use the absolute value of the stride instead. See [issue #25856] for details. [postgresql]: https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-BIN [issue #25856]: https://github.com/apache/datafusion/issues/25856 ### `DataFrame::from_columns` accepts `IntoIterator` `DataFrame::from_columns` now accepts any `IntoIterator` instead of specifically accepting a `Vec<(&str, ArrayRef)>`. ```rust,ignore // Existing Vec usage continues to work let df = DataFrame::from_columns(vec![ ("id", id), ("name", name), ])?; // Arrays can now be used directly let df = DataFrame::from_columns([ ("id", id), ("name", name), ])?; ``` Most existing call sites using `Vec` require no changes. Code that relies on the exact non-generic function signature of `DataFrame::from_columns` may need to be updated to account for the new generic API. ### Aggregate ordering state field names are now namespaced Ordering expressions exposed as top-level aggregate state fields now use names derived from the aggregate name and their ordering position. For example: ```text first_value(value)[ordering_0] ``` is now used instead of an unqualified ordering field name such as: ```text timestamp@0 ``` This guarantees unique aggregate state field names and allows `DFSchema::check_names()` to be enforced. **Who is affected:** Users or integrations that inspect aggregate state field names directly, including custom UDAFs and FFI integrations. ### Physical filter pushdown resolves columns by position `datafusion_physical_plan::filter_pushdown::ChildFilterDescription::from_child` and `FilterDescription::from_children` now resolve filter columns by position instead of looking them up by name. This prevents incorrect results when a child schema contains duplicate column names. `from_child` requires the child field at each referenced position to have the same name as the filter column. `ChildFilterDescription::from_child_with_allowed_indices` is deprecated but preserves its previous name-based mapping to the first matching child field. Migrate to `from_child_with_column_mapping` because name resolution is ambiguous when the child schema contains duplicate field names. **Migration guide:** Use `from_child` (or `from_children` for multiple children) when the parent and child schemas have matching column positions and names. When a node projects, reorders, or renames columns, use `from_child_with_column_mapping` with an explicit map from parent output indices to child input indices. Columns absent from the mapping cannot be pushed down. For example, if the parent outputs `[a, b]` and the child outputs `[b, a]`, a filter on `a@0` must map to child column `a@1`: ```rust,ignore use std::collections::{HashMap, HashSet}; use datafusion_physical_plan::filter_pushdown::ChildFilterDescription; // Before: allow parent column 0 and resolve "a" by name in the child. let description = ChildFilterDescription::from_child_with_allowed_indices( &parent_filters, HashSet::from([0]), &child, )?; // After: explicitly map parent column 0 to child column 1. let description = ChildFilterDescription::from_child_with_column_mapping( &parent_filters, HashMap::from([(0, 1)]), &child, )?; ``` ### Default `datafusion.sql_parser.recursion_limit` raised from 50 to 51 DataFusion now uses `sqlparser` 0.63.0, which adds recursion guards to more parse functions (for example, data type and `INTERVAL` parsing). As a result, the same SQL statement consumes slightly more of the parser recursion budget than it did with previous `sqlparser` versions. In particular, every expression now needs one additional level of depth. To avoid rejecting queries that previously parsed successfully, the default value of the `datafusion.sql_parser.recursion_limit` configuration option, and of `DFParserBuilder::with_recursion_limit`, has been raised from `50` to `51`. If you set `datafusion.sql_parser.recursion_limit` explicitly to a value close to the depth of your queries, you may see `RecursionLimitExceeded` errors after upgrading and should raise the limit accordingly. See [PR #25278](https://github.com/apache/datafusion/pull/25278) for details. ### `ensure_distribution` is deprecated in favour of `ensure_distribution_with_stats` `datafusion_physical_optimizer::enforce_distribution::ensure_distribution` now takes its `StatisticsContext` from the caller, so one context (and its memoization cache) can be shared across a whole bottom-up traversal. The old two-argument form is kept as a deprecated wrapper that allocates a fresh context per call, which recomputes each shared subtree's statistics once per ancestor. ```rust,ignore // Before let plan = DistributionContext::new_default(plan) .transform_up(|ctx| ensure_distribution(ctx, config)) .data()?; // After: one context for the whole walk. `StatsCache` is keyed by raw plan-node // pointers, so reset it whenever a node's plan pointer actually changed. let stats_ctx = StatisticsContext::new(); let plan = DistributionContext::new_default(plan) .transform_up(|ctx| { let before = Arc::clone(&ctx.plan); let result = ensure_distribution_with_stats(ctx, config, &stats_ctx)?; if !Arc::ptr_eq(&before, &result.data.plan) { stats_ctx.reset_cache(); } Ok(result) }) .data()?; ``` Callers that only need the previous behaviour can keep using the deprecated form, or pass a freshly constructed `StatisticsContext` per call. ### Physical optimizer rules share one `StatisticsContext` per query `PhysicalOptimizerContext` has a new `statistics_context` method. The physical planner returns one `StatisticsContext` for all the rules of one `optimize_physical_plan` call, so statistics computed by one rule are reused by later rules. The method returns `None` by default, so existing implementations are not affected. Custom rules that compute statistics should call the new `PhysicalOptimizerContext::compute_statistics` method in `optimize_with_context`. It uses the shared context, or, if there is none, a new context built from the statistics registry: ```rust,ignore let statistics = context.compute_statistics(&plan, &StatisticsArgs::new())?; ``` To pass one `StatisticsContext` down a traversal, use `datafusion_session::with_statistics_context`, which applies the same choice. `AggregateStatistics`, `LimitPushdown` and the co-partitioning choice in `EnsureRequirements` now also consult the session's statistics providers, as `JoinSelection` already did. Sessions without registered providers (the default) are not affected. As for operators, `Exact` statistics from a provider must be guaranteed values: `AggregateStatistics` answers `COUNT(*)`, `MIN` and `MAX` from them, and `LimitPushdown` removes a limit over an `Exact(0)` input. For the same reason, `datafusion_physical_optimizer::limit_pushdown::pushdown_limit_helper` is deprecated in favour of `pushdown_limit_helper_with_stats`, which takes the `StatisticsContext` from the caller. The deprecated form creates a new context on every call, without statistics providers. When providers are registered, switching to `pushdown_limit_helper_with_stats` with a context built from the statistics registry can change the result, because the providers are then consulted. ### `MergeIntoOp` requires the SQL-visible target qualifier `MergeIntoOp` now stores the SQL-visible target qualifier separately from the target provider identity. This prevents an aliased target from being confused with a source relation that uses the target table's name. The struct is now non-exhaustive. Replace 55.0 struct literals with `MergeIntoOp::new`: ```rust,ignore // DataFusion 55.0 let op = MergeIntoOp { on, clauses }; // DataFusion 56.0 let op = MergeIntoOp::new(target_qualifier, on, clauses); ``` `target_qualifier` must be the relation name referenced by the MERGE expressions: the target alias when one is present, otherwise the target table reference. `DmlStatement::table_name` remains the provider/catalog identity. ### `MergeIntoOpNode` protobuf literals require `target_qualifier` The generated `MergeIntoOpNode` type has a new optional field. Code constructing the generated type with a struct literal must provide it: ```rust,ignore // DataFusion 55.0 let node = protobuf::MergeIntoOpNode { on, clauses }; // DataFusion 56.0 let node = protobuf::MergeIntoOpNode { on, clauses, target_qualifier: Some(protobuf::TableReference::from(target_qualifier)), }; ``` Wire compatibility is directional: - A 56.0 reader accepts a 55.0 payload. When `target_qualifier` is absent, it falls back to `DmlNode.table_name`, matching the 55.0 representation. - A 55.0 reader must not consume an alias-preserving 56.0 MERGE payload. It ignores the unknown field but cannot preserve the qualifier required by the expressions, which can cause resolution failure or incorrect rebinding. ### Null-aware joins record their `NOT IN` value key count Multi-column `(a, b) NOT IN (SELECT x, y ...)` is now planned as a null-aware join. Its first equi-join keys are the tuple elements, followed by any correlation keys, so a null-aware join now records how many of its leading keys are `NOT IN` value keys: - `datafusion_expr::logical_plan::Join` has a new `null_aware_value_keys: usize` field. - `HashJoinExec` has a new `null_aware_value_keys: usize` field, set with `HashJoinExecBuilder::with_null_aware_value_keys`. The count is only read when `null_aware` is set. `Join::try_new`, the `LogicalPlanBuilder` join methods and `HashJoinExec::try_new` set it to `1`, the scalar `NOT IN` of earlier releases. Code constructing `Join` with a struct literal must initialize the field: ```rust,ignore let join = Join { // ... null_aware, null_aware_value_keys: 1, }; ``` The generated protobuf `JoinNode` and `HashJoinExecNode` types have a new `null_aware_value_keys` field, which struct literals must provide too. A 56.0 reader decodes a payload without the field (value `0`) as a scalar `NOT IN`. A 55.0 reader must not consume a 56.0 plan holding a multi-column `NOT IN`: it ignores the unknown field and would read the extra tuple elements as correlation keys, returning wrong results. ### `FunctionalDependence` has a new `null_equality` field `FunctionalDependence` now records the NULL semantics its dependency holds under. Build it with `FunctionalDependence::new(..)` and, when needed, `.with_null_equality(NullEquality::NullEqualsNull)` instead of a struct literal. **Migration guide:** ```rust,ignore // Before let dep = FunctionalDependence { source_indices, target_indices, nullable, mode: Dependency::Single, }; // After let dep = FunctionalDependence::new(source_indices, target_indices, nullable) .with_mode(Dependency::Single) .with_null_equality(NullEquality::NullEqualsNull); ``` ### Nullable `UNIQUE` constraints no longer add or remove `GROUP BY` expressions `get_target_functional_dependencies` and `get_required_group_by_exprs_indices` now ignore a functional dependency that does not hold across NULL determinant values (see `FunctionalDependence::is_valid_across_nulls`), such as one derived from a nullable `UNIQUE` constraint, because `GROUP BY` places all NULL keys in a single group. As a result, a column that is only determined by a nullable `UNIQUE` key can no longer be selected without appearing in the `GROUP BY`. This matches PostgreSQL, which accepts such a query only for a `PRIMARY KEY`: ```sql CREATE TABLE t (x INT UNIQUE, y INT); -- Now fails: Column in SELECT must be in GROUP BY or an aggregate function SELECT x, y FROM t GROUP BY x; -- Works: add `y` to the GROUP BY, aggregate it, or declare `x` NOT NULL SELECT x, y FROM t GROUP BY x, y; ``` ### Functional dependency helpers take field indices instead of names `aggregate_functional_dependencies`, `get_target_functional_dependencies`, `get_required_group_by_exprs_indices` and `get_required_sort_exprs_indices` in `datafusion_common` now take, for each GROUP BY or ORDER BY expression, the index of the input field it references (`Option`) instead of its name. Matching by name gave incorrect results, because different expressions can have the same name: `CAST(t.pk AS INT)` is named `t.pk`, but it does not determine the columns that `t.pk` does. Queries are affected too: only a column reference to the key carries its dependencies, so a query that selects other columns while grouping by an expression over the key is now rejected, as in PostgreSQL and DuckDB. Group by the key column itself instead: ```sql -- `id` is the INT PRIMARY KEY of `t` SELECT name FROM t GROUP BY CAST(id AS BIGINT); -- now an error SELECT name FROM t GROUP BY id; -- ok ``` **Migration guide:** Pass `None` for an expression that computes a new value. For a column reference, `datafusion_expr::utils::passthrough_field_index` returns the index: ```rust,ignore // Before let names: Vec = exprs.iter().map(|e| e.schema_name().to_string()).collect(); let target = get_target_functional_dependencies(schema, &names); // After let indices: Vec> = exprs .iter() .map(|e| passthrough_field_index(e, schema)) .collect(); let target = get_target_functional_dependencies(schema, &indices); ``` ### `FileSource::exact_filter` controls scan equivalence properties `FileScanConfig` no longer derives equivalence properties (constant columns and equal columns) from `FileSource::filter`. It uses the new `FileSource::exact_filter` method instead. This method returns the part of the filter that every output row satisfies. A filter that the source uses only for pruning (for example, Parquet row group statistics when `datafusion.execution.parquet.pushdown_filters` is `false`) lets non-matching rows through, and equivalences derived from it gave wrong query results. `exact_filter` returns `None` by default. `ParquetSource` returns the pushed-down conjuncts when `pushdown_filters` is enabled. **Migration guide:** If your custom `FileSource` applies its filter to every row, implement `exact_filter` to keep the equivalence properties of the scan: ```rust,ignore impl FileSource for MySource { fn exact_filter(&self) -> Option> { self.filter() } // ... } ``` ### `TableProvider::scan_with_args` supports skip pushdown The `ScanArgs` struct now carries a new `skip` (number of rows to skip) field alongside `limit`. `TableProvider::scan_with_args()` by default forwards to `scan()`, so existing `TableProvider` implementations keep compiling and working unchanged; implementations that want to take advantage of `skip` (or future `ScanArgs` fields) can override `scan_with_args` in addition to `scan`. A new `TableProvider::supports_skip_pushdown() -> bool` (default `false`) lets a provider guarantee it honors `ScanArgs::skip` exactly — that a scan built with `skip` omits _exactly_ the first `skip` rows it would otherwise have produced, not just approximately. `ListingTable` overrides this to `true` and honors `skip`. DataFusion's SQL/DataFrame optimizer (`push_down_limit`) now populates `LogicalPlan::TableScan::skip` from a query's `LIMIT ... OFFSET ...` clause, but only when the scan's source reports `supports_skip_pushdown() == true`. In that case the skip is moved entirely into the scan and the remaining `Limit` above it no longer re-applies it; every other provider keeps today's behavior unchanged, with the skip always enforced above the scan. `skip` is also reachable directly, independent of the optimizer, by constructing a `TableScan` (for example via `LogicalPlanBuilder::scan_with_filters_fetch_skip`) or by calling `scan_with_args` with `ScanArgs::with_skip`. This pushdown is also preserved end-to-end through plan serialization: `datafusion-substrait`'s producer folds a scan's `skip` into the `FetchRel` it emits for the wrapping `Limit` (Substrait's `ReadRel` has no field of its own for it), and `datafusion-proto`'s `ListingTableScanNode`, `ViewTableScanNode`, and `CustomTableScanNode` messages gained optional `fetch`/`skip` fields so a `TableScan`'s own `fetch`/`skip` — previously silently dropped — now round-trips exactly. #### Who is affected - Custom `TableProvider` implementations that want skip pushdown: override `scan_with_args` to read `ScanArgs::skip()` and honor it exactly, and override `supports_skip_pushdown` to return `true`. Once you do, `LIMIT ... OFFSET ...` queries against that provider will have their skip pushed into the scan automatically. - Callers that already build a scan through `ScanArgs`/`scan_with_args` (for example custom query planners) can now also set `skip`. #### API breaks The _public_ `TableScan` struct has one new _public_ field - `skip` and one field whose type is changed - `statistics_requests`. The addition of `skip` field increases the size of `TableScan` and thus the size of `LogicalPlan` enum. To keep the size of `LogicalPlan` reasonable the type of the `TableScan::statistics_requests` field is changed from `BTreeSet` (24 bytes) to `Box>` (8 bytes). The users' applications are recommended to use `TableScanBuilder` to construct new instances of `TableScan` instead of instantiating directly `TableScan { ... }` to avoid the API breaks. #### Example ```rust,ignore let plan = provider .scan_with_args( &state, ScanArgs::default() .with_projection(projection) .with_filters(Some(&filters)) .with_limit(limit) .with_skip(skip), ) .await? .into_inner(); ``` ### Extension `ExecutionPlan`s can decode without a `PhysicalExtensionCodec` `ExecutionPlan::try_to_proto` let a plan serialize itself, but there was no way back: decoding an extension node always routed through `PhysicalExtensionCodec::try_decode`. `PhysicalExtensionNode` carried no type discriminator, so the codec _was_ the discriminator. `ComposedPhysicalExtensionCodec` works around that by writing the position of the encoding codec into the payload, which means the decoding side must register the same codecs in the same order, and a name collision between two independent crates is undetectable. `PhysicalExtensionNode` now has an optional `plan_name`, and a session can map that name to a decoder. An extension plan implements `ExtensionPlanFromProto`, which pairs the name it is dispatched by with the constructor that rebuilds it, and is registered on the session that will decode it: ```rust,ignore use datafusion::physical_plan::proto::{ ExecutionPlanDecodeCtx, ExecutionPlanEncodeCtx, ExtensionPlanFromProto, }; impl ExecutionPlan for MyExec { // ... fn try_to_proto( &self, ctx: &ExecutionPlanEncodeCtx<'_>, ) -> Result> { // `extension_node` stamps `Self::NAME` on the node, so the name on the // wire and the registry key cannot drift apart. Ok(Some(ctx.extension_node::( my_payload_bytes(self)?, self.children(), )?)) } } impl ExtensionPlanFromProto for MyExec { // Namespace the name so a collision with another crate is an error at // registration rather than a wrong decode. const NAME: &'static str = "my-crate.MyExec"; fn try_from_proto( node: &PhysicalPlanNode, ctx: &ExecutionPlanDecodeCtx<'_>, ) -> Result> { let extension = expect_plan_variant!(node, PhysicalPlanType::Extension, "Extension"); let children = ctx.decode_children(&extension.inputs)?; // `ctx.task_ctx()` is available here, so a plan that rebuilds session // state at decode time (a connection pool, say) can do so. my_plan_from_bytes(&extension.node, children) } } // On every session that decodes the plan — for a distributed engine, the // workers as well as the coordinator. let mut config = SessionConfig::new(); config.update_extension(MyExec::register_proto)?; ``` Only extension plans implement `ExtensionPlanFromProto`, and registration is the trait's provided `register_proto` method. A built-in plan has a wire variant of its own and keeps its inherent `try_from_proto`, so `FilterExec::register_proto(&mut registry)` does not compile. `ProtoDecoderRegistry` is shared by every kind of extension DataFusion can decode, not only plans. Its key is the trait plus the name, so a plan and an expression may use the same name, and one session carries one registry rather than one per kind. A session carries one registry. Each library adds its decoders to it with `SessionConfig::update_extension`, which creates the registry if there is none. Do not attach a registry with `with_extension`: it replaces the registry, so it discards the decoders that other libraries added. ```rust,ignore // In each library that ships extension types: pub fn register(config: &mut SessionConfig) -> Result<()> { config.update_extension(MyExec::register_proto) } // In the application that owns the session: let mut config = SessionConfig::new(); shuffle_lib::register(&mut config)?; sampling_lib::register(&mut config)?; ``` The libraries need to know nothing about each other, and the order they are called in does not matter. Dispatch is by name, and a real name collision is an error from `register` rather than a wrong decode later. A registered plan needs no `PhysicalExtensionCodec` at all, and two crates' plans coexist on one session without a `ComposedPhysicalExtensionCodec`. **Who is affected:** - Nobody is required to change anything. This is opt-in per plan type: a `PhysicalExtensionNode` with no name, or with a name the decoding session does not know, takes the `PhysicalExtensionCodec` chain exactly as before. On the binary wire format the new field is additive: old writers omit it and old readers skip it. - The generated JSON codec is stricter than the binary one. `pbjson` rejects a field it does not know, so a named node written by a new writer cannot be read as JSON by a reader on an older DataFusion. A plan that does not opt in writes no name and is unaffected. If you serialize plans as JSON across versions, upgrade the readers before you opt a plan in. - Users constructing `PhysicalExtensionNode` with an exhaustive struct literal must set the new `plan_name` field (`None` reproduces the previous behavior). - Replacing a `PhysicalExtensionCodec` with a registration changes only _how_ the payload is dispatched, not what the payload is. Readers running an older DataFusion ignore `plan_name` and hand the payload to their codec, so a mixed fleet keeps working only while the bytes your `try_to_proto` writes stay decodable by the codec you removed. `PhysicalExtensionCodec` is not deprecated: extension `PhysicalExpr`s and UDF payloads still need it, and it remains the fallback for unmigrated plans. See [issue #24625](https://github.com/apache/datafusion/issues/24625) for details. ### `LogicalPlan` has the new variant `MaterializedCte` `WITH x AS MATERIALIZED (...)` is planned with the new variant `LogicalPlan::MaterializedCte(MaterializedCtePlan)`. `MaterializedCtePlan` has two cases: - `MaterializedCtePlan::Cte` has two inputs: the body of the CTE, which is computed once, and the continuation (the query that declared the CTE). Its output is the output of the continuation. - `MaterializedCtePlan::Scan` is a leaf. It reads the buffered output of the `MaterializedCtePlan::Cte` that has the same id. #### Who is affected - Code that has an exhaustive `match` on `LogicalPlan` must add an arm for the new variant. - A custom optimizer rule that moves a filter, limit or projection below a node must not move it into the body of a `MaterializedCte`, because all the scans of the CTE share the body. The continuation has no such limit. `MaterializedCteScan::find_in` returns the scans of one CTE in a plan, also in subquery expressions. ### Extension `PhysicalExpr`s can decode by name instead of through a codec `PhysicalExtensionExprNode` has a new optional `expr_name` field. An extension expression that sets it is decoded by a decoder registered on the session, not by a `PhysicalExtensionCodec`. This is the same mechanism extension `ExecutionPlan`s use (see above), and it uses the same `ProtoDecoderRegistry`. **Who is affected:** nobody, unless they opt in. A node with no name, or with a name the session has no decoder for, decodes through `PhysicalExtensionCodec::try_decode_expr` as before. Binary protobuf readers skip the new field. Readers that use the optional JSON serde reject unknown fields, so they must be built after this change to read a node that has `exprName`. **Opting in:** ```rust,ignore use datafusion::execution::TaskContext; use datafusion::physical_plan::proto::{ ExtensionExprFromProto, PhysicalExprDecodeCtx, PhysicalExprEncodeCtx, extension_expr_node, }; impl PhysicalExpr for MyExpr { // ... fn try_to_proto( &self, ctx: &PhysicalExprEncodeCtx<'_>, ) -> Result> { // Writes `Self::NAME` on the node. Ok(Some(extension_expr_node::( ctx, my_payload_bytes(self)?, self.children(), )?)) } } impl ExtensionExprFromProto for MyExpr { // Put the crate name in the name, so that a collision is an error at registration. const NAME: &'static str = "my_crate.MyExpr"; fn try_from_proto( node: &PhysicalExprNode, ctx: &PhysicalExprDecodeCtx<'_>, ) -> Result> { // The session, for example to find a UDF by name. let task_ctx = ctx.session::(); // ... } } let mut config = SessionConfig::new(); config.update_extension(MyExpr::register_proto)?; ``` Only extension expressions can be registered. `Column::register_proto(..)` does not compile. If a registered decoder fails, the decode fails. It does not fall back to the codec, because the codec could decode a payload that it did not write. Changing an expression to write its own payload is therefore a wire-format change for that expression: register the type on every reader before writers start to use it. `PhysicalExprDecodeCtx::session::()` is new. It gives the session the decode runs under. `datafusion-proto` supplies its `TaskContext`. A custom `PhysicalExprDecode` implementation returns `None` by default. See [issue #24626](https://github.com/apache/datafusion/issues/24626) for details. ### Order-preservation hooks for limit pushdown are deprecated A file scan that advertises an output ordering now always reads its files in order, so `ORDER BY ... LIMIT` queries no longer depend on `LimitPushdown` telling the scan to keep its order. Before this change, pushing the fetch of a `SortPreservingMergeExec` into an ordered Parquet scan could return the wrong rows, because the scan then shared files between partitions or skipped row groups based on its limit. The plumbing that carried this request is deprecated and no longer used: - `ExecutionPlan::with_preserve_order` and `DataSource::with_preserve_order` are never called. Built-in operators no longer override them. Overrides in custom `ExecutionPlan` or `DataSource` implementations are now dead code, and the compiler does not warn about them, because `#[deprecated]` on a trait method only warns callers. - A custom scan that advertises an output ordering must keep that order by itself, also when a limit is pushed into it with `with_fetch`. DataFusion no longer tells it whether the limit is order-sensitive. - `GlobalLimitExec::required_ordering` and `LocalLimitExec::required_ordering` always return `None`, and their `set_required_ordering` methods do nothing. The `required_ordering` protobuf fields are no longer written and are ignored when read. A plain `LIMIT` without `ORDER BY` over a table with a declared ordering now also reads files in order, so it no longer uses file work stealing or Parquet limit pruning. Other queries over such tables already read files in order. If the file groups are uneven, for example a few large files next to many small ones, a selective `WHERE ... LIMIT` can be slower than before. **Migration guide:** To keep a file scan's order, give it an output ordering (`FileScanConfigBuilder::with_output_ordering`). To read files in group order without declaring one, use `FileScanConfigBuilder::with_preserve_order(true)`. Code that reads `FileScanConfig::preserve_order` to decide whether files may be reordered should call `FileScanConfig::preserves_file_order()` instead, which also accounts for the output ordering.