alamb commented on code in PR #195: URL: https://github.com/apache/parquet-site/pull/195#discussion_r3864664824
########## content/en/blog/features/alp_encoding.md: ########## @@ -0,0 +1,291 @@ +--- +title: "ALP: Adaptive Lossless Floating-Point Encoding in Apache Parquet" +date: 2026-08-14 +description: "A technical overview of ALP's design, performance, and adoption across the Apache Parquet ecosystem." +author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb), [Prateek Gaur](https://github.com/prtkgaur)" +categories: ["features"] +--- + +Apache Parquet has added the [Adaptive Lossless floating-Point (ALP) Encoding] -- a new lightweight floating-point encoding with compression ratios similar to [`zstd`], much faster decompression, random-access support, and GPU- and SIMD-friendly decoding. + +---- +[`zstd`]: https://github.com/facebook/zstd +<!-- Note ALP is not yet published to the Parquet website, so guess what the link will be--> +[Adaptive Lossless floating-Point (ALP) Encoding]: https://parquet.apache.org/docs/file-format/data-pages/encodings/#ALP + +ALP works best for decimal values that are stored as floating-point types (32-bit `FLOAT` and 64-bit `DOUBLE`), such as + +- Monetary values (exchange rates, public funds, stocks, prices, etc.) -- e.g., `1.2345` or `22.03` +- Geographic coordinates (longitude/latitude) -- e.g., `42.3584`, `-71.0598` +- Scientific measurements (temperature, pressure, speed, degrees, etc.) -- e.g., `-273.15`, `9.81`, `3.14159` + +ALP is not suitable for data that uses a wide range of exponents or a large +number of significant digits, such as vector embeddings which typically span the +full floating-point range. Such use cases can continue to use existing Parquet +features such as `PLAIN` or [`BYTE_STREAM_SPLIT`] encoding followed by `ZSTD` or +`SNAPPY` general-purpose compression. + +Decimal values can be stored with Parquet's `DECIMAL` logical type, but it +requires the precision and scale to be known and declared up front and cannot +store values outside of that +range. For this reason, systems commonly store decimal values as `FLOAT` or +`DOUBLE` when the exact shape of their data is not known beforehand. For example, +JavaScript's only* [number type is `DOUBLE`], common data science tools such as +pandas [infer `float64` for decimal-looking values], and NumPy has [no decimal dtype at all]. + +[number type is `DOUBLE`]: https://tc39.es/ecma262/#sec-ecmascript-language-types-number-type +[infer `float64` for decimal-looking values]: https://pandas.pydata.org/docs/reference/api/pandas.to_numeric.html +[no decimal dtype at all]: https://numpy.org/doc/stable/reference/arrays.dtypes.html +[`BigInt`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt + +<small>\* JavaScript also has [`BigInt`], but it can only represent integers.</small> + +## Why ALP? + +Encoding floating-point data is a complicated engineering problem due to the nature of floating-point values. They do not exactly represent most real values. This leads to rounding errors that prevent using existing lightweight encodings like Delta and Frame of Reference (FOR). + +Prior to ALP, `BYTE_STREAM_SPLIT` was the only non-dictionary alternative to +`PLAIN` for `FLOAT`/`DOUBLE` values in Parquet. It does not reduce the size of +data but *can* improve the compression ratio and speed when a heavyweight +compressor is used afterwards. + +Heavyweight compression buys that ratio at three costs: + - Decode speed -- decompression runs well below what a scan can consume. + - Random access -- reading one value means decoding the whole page. + - Data dependence -- variable-length compression means that decoding a value requires decoding previous values, making it hard to parallelize with modern hardware such as [SIMD instructions] and [GPU]s. + +[SIMD instructions]: https://en.wikipedia.org/wiki/SIMD +[GPU]: https://en.wikipedia.org/wiki/Graphics_processing_unit +[`BYTE_STREAM_SPLIT`]: https://parquet.apache.org/docs/file-format/data-pages/encodings/#BYTESTREAMSPLIT + +ALP is designed to solve all three of these problems for common data patterns, while achieving a similar compression ratio. + +### ALP Performance + +As shown in the charts below, users can expect ALP to be `10x` faster to decode +than `zstd` and thousands of times faster to retrieve individual values, with +similar compression ratios and comparable compression speed. + +The code and instructions to reproduce these results and try ALP with your own +Parquet datasets can be found in the [alp_benchmark](https://github.com/alamb/alp_benchmark) repository, with the Rust Parquet implementation included. + +<div class="row g-3 td-max-width-on-larger-screens"> + <div class="col-12 col-md-6"> + <img src="/blog/alp/avg_compression_ratio.png" alt="Average compression ratio benchmark" class="img-fluid"> + </div> + <div class="col-12 col-md-6"> + <img src="/blog/alp/avg_compression_speed.png" alt="Average compression speed benchmark" class="img-fluid"> + </div> + <div class="col-12 col-md-6"> + <img src="/blog/alp/avg_decompression_speed.png" alt="Average decompression speed benchmark" class="img-fluid"> + </div> + <div class="col-12 col-md-6"> + <img src="/blog/alp/avg_random_access.png" alt="Average random-access benchmark" class="img-fluid"> + </div> + <div> + <b>Figure 1</b>: Average compression ratio, compression speed, and decompression speed of <code>PLAIN+ZSTD</code> (per-page <code>zstd</code> compression) and <code>ALP</code> across <code>30</code> datasets on three machines. Higher is better. Review Comment: -- I tried to clarify in https://github.com/apache/parquet-site/pull/195/commits/500bf9b717c0d006089cc609bde37f15737a2296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
