ALP: Adaptive Lossless Floating-Point Encoding in Apache Parquet

A technical overview of ALP’s design, performance, and adoption across the Apache Parquet ecosystem.
By Kosta Tarasov, Andrew Lamb, Prateek Gaur |
Categories:

Apache Parquet has added the Adaptive Lossless floating-Point (ALP) Encoding – a new lightweight floating-point encoding with compression ratios similar to zstd, much faster decompression, random-access support, and GPU- and SIMD-friendly decoding.

ALP works best for decimal values that are stored as floating-point types (32-bit FLOAT and 64-bit DOUBLE), such as

  • Monetary values (exchange rates, public funds, stocks, prices, etc.) – e.g., 1.2345 or 22.03
  • Geographic coordinates (longitude/latitude) – e.g., 42.3584, -71.0598
  • Scientific measurements (temperature, pressure, speed, degrees, etc.) – e.g., -273.15, 9.81, 3.14159

ALP is not suitable for data that uses a wide range of exponents or a large number of significant digits, such as vector embeddings which typically span the full floating-point range. Such use cases can continue to use existing Parquet features such as PLAIN or BYTE_STREAM_SPLIT encoding followed by general-purpose compression such as ZSTD.

Decimal values can be stored with Parquet’s DECIMAL logical type, but it requires the precision and scale to be known and declared up front and cannot store values outside of that range. For this reason, systems commonly store decimal values as FLOAT or DOUBLE when the exact shape of their data is not known beforehand. For example, JavaScript’s only* number type is DOUBLE, common data science tools such as pandas infer float64 for decimal-looking values, and NumPy has no decimal dtype at all.

* JavaScript also has BigInt, but it can only represent integers.

Why ALP?

Encoding floating-point data is a complicated engineering problem due to the nature of floating-point values. They do not exactly represent most real values. This leads to rounding errors that prevent using existing lightweight encodings like Delta and Frame of Reference (FOR).

Prior to ALP, BYTE_STREAM_SPLIT was the only non-dictionary alternative to PLAIN for FLOAT/DOUBLE values in Parquet. It does not reduce the size of data but can improve the compression ratio and speed when a heavyweight compressor is used afterwards.

Heavyweight compression buys that ratio at three costs:

  • Decode speed – decompression runs well below what a scan can consume.
  • Random access – reading one value means decoding the whole page.
  • Data dependence – variable-length compression means that decoding a value requires decoding previous values, making it hard to parallelize with modern hardware such as SIMD instructions and GPUs.

ALP is designed to solve all three of these problems for common data patterns, while achieving a similar compression ratio.

ALP Performance

Parquet applies an encoding first, then an optional compression codec as a separate step. The charts below compare the PLAIN and BYTE_STREAM_SPLIT encodings followed by ZSTD compression and the ALP encoding with no additional compression. Users can expect ALP to decode 10x faster and retrieve individual values thousands of times faster, with a slightly lower compression ratio and slightly faster compression.

The code and instructions to reproduce these results and try ALP with your own Parquet datasets can be found in the alp_benchmark repository, with the Rust Parquet implementation included.

Average compression ratio benchmark
Average compression speed benchmark
Average decompression speed benchmark
Average random-access benchmark
Figure 1: Average compression ratio, compression speed, and decompression speed of PLAIN+ZSTD and BYTE_STREAM_SPLIT+ZSTD (each encoding followed by per-page ZSTD compression) and ALP (no compression codec) across 30 datasets on three machines. Higher is better. Random-access speed is measured by decoding 100 deterministic, uniformly distributed rows from city_temperature_f.

Note that the numbers reported are for the pre-release Rust implementation of ALP. We expect the performance of ALP encoders to improve as the implementations are optimized and tuned. The current implementations are already faster than zstd in many cases, even though most zstd implementations have already been heavily optimized.

Technical Overview

ALP was developed by Azim Afroozeh, Leonardo Kuffó, and Peter Boncz from the Database Architectures Group at CWI and published in a SIGMOD 2024 paper. ALP takes advantage of a common pattern: many values stored as FLOAT or DOUBLE originated as decimal numbers with relatively few digits, such as prices or measurements. This section explains the intuition behind ALP. The following sections then explain the encoding and decoding pipeline in more detail.

ALP encodes floating-point values in batches called “vectors” of between 8 and 32K values (e.g., 1024). Each value in a vector is encoded as an integer, and the vector as a whole stores two more integers shared by all its values: an “exponent” (e) and a “factor” (f). Each vector can use a different exponent and factor, and how they are chosen is explained below. The original value is recovered by computing

value = encoded × 10f × 10-e

The calculation above uses floating-point arithmetic, which may round the result to the nearest representable value instead of reproducing the original value. When the decoding process does not reproduce the original value exactly, ALP instead stores the original full-precision value separately as an “exception”, keeping the encoding lossless. Special values such as NaN, ±Infinity, and -0.0 are also stored as exceptions.

Within each vector, the encoded values are stored by subtracting the lowest value (frame of reference) and then bit-packing to a fixed width. Exceptions are stored directly after the encoded array. The layout of each ALP vector is shown below.

ALP serialized vector layout
Figure 2: Layout of a serialized ALP vector: a fixed-size Vector Header followed by a variable-size Data Section.

Since each value is stored as a bit-packed integer of a fixed width, locating an arbitrary row requires computing the offset of the encoded bits. To decode the value, the frame of reference, exponent, and factor are applied to the encoded value to recover the original floating-point value. Finally, the exception indices are checked for the target row, and if an exception is present, its value is returned instead.

Picking a good exponent and factor is key to good ALP performance. Each Parquet writer is free to choose the exponent and factor for each vector using any algorithm. The Parquet specification provides an example sampling-based algorithm that aims to minimize the encoded size. Typically, the exponent is chosen to capture most decimal digits in the vector while minimizing exceptions, and the factor is chosen to remove as many trailing zeros as possible.

Finally, to minimize the number of bits needed to store the encoded values, ALP uses the minimum value as a frame of reference and subtracts it from each encoded value before bit-packing.

ALP Encoding and Decoding

The encoding pipeline is straightforward, as shown in the following example of encoding a vector of values:

ALP encoding pipeline example
Figure 3: Encoding a vector of 1024 64-bit floating-point values using ALP.

To encode this vector, first the parameters e = 4 and f = 3 are chosen. Then the values are transformed to integers using the formula encoded = round(value × 104 × 10-3). Each integer is checked by reversing the transformation with decoded = encoded × 103 × 10-4. Values that do not reproduce the original value, such as 8.0605123 (which decodes to 8.1), are stored in the exception array. The minimum value across the vector, 3335, becomes the frame of reference and is subtracted from each integer, and the resulting deltas are bit-packed using 15 bits. In this example, ALP uses 1920 bytes for the bit-packed deltas, plus a 13-byte vector header and space for exceptions. PLAIN uses 8192 bytes for the same 1024 values. This comparison excludes page-level metadata for both encodings. See the ALP Encoding specification for more details on how the parameters are chosen and how rounding and exception handling work.

Decoding a vector requires similar steps, but in reverse, as shown below.

ALP decoding pipeline example
Figure 4: Decoding a vector of 1024 values back to floating-point values using ALP.

First, the bit-packed deltas are unpacked and the original values are computed by original = (3335 + delta) × 103 × 10-4. Then any exceptions are “patched” by overwriting the output array at the exception positions with the exception values.

Ecosystem Adoption

The encoding was officially accepted into Parquet in July 2026, and we expect several major open source Parquet implementations to add ALP support in the next few months. Work is already in progress in the following:

You can also try it today on your own datasets using the tool in the ALP benchmark repository.

Conclusion

ALP brings fast, parallelizable decoding and practical random access to floating-point data in a standard form that any Parquet implementation can read after adding support for the encoding. Its addition is one more example of Apache Parquet evolving to meet the needs of modern data systems.

As with all additions to Parquet, this was a community endeavor with contributions from many individuals and vendors working together to agree on a common standard. Together, we created a well-documented specification and reference implementations in several languages, and we expect ALP to be widely adopted in the Parquet ecosystem over the coming years.

Resources

Last modified August 26, 2026: Add ALP implementation in go (8b978fd)