sdf-jkl commented on code in PR #195: URL: https://github.com/apache/parquet-site/pull/195#discussion_r3834884145
########## content/en/blog/features/alp_encoding.md: ########## @@ -0,0 +1,284 @@ +--- +title: "ALP: Adaptive Lossless Floating-point Encoding in Apache Parquet" +date: 2026-08-14 +description: "Fast, random access, GPU and SIMD-friendly compression and decompression; similar in size to zstd but much faster to decode." +author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb), [Prateek Gaur](https://github.com/prtkgaur)" +categories: ["features"] +--- + +Apache Parquet has added the [Adaptive Lossless floating-Point (ALP) Encoding] -- a new lightweight floating-point encoding with similar compression to [`zstd`], much faster decompression speed, random access support, and SIMD and GPU friendly decoding. + +---- +[`zstd`]: https://github.com/facebook/zstd +<!-- Note ALP is not yet published to the Parquet website, so guess what the link will be--> +[Adaptive Lossless floating-Point (ALP) Encoding]: https://parquet.apache.org/docs/file-format/data-pages/encodings/#ALP + +ALP works best for decimal values that are stored as floating-point types (32-bit `FLOAT` and 64-bit `DOUBLE`), such as +- Monetary values (exchange rates, public funds, stocks, prices, etc.) - e.g. `1.2345` or `22.03` +- Geographic coordinates (longitude/latitude) - `42.3584`, `-71.0598` +- Scientific measures (temperature, pressure, speed, degrees, etc.) - e.g. `-273.15`, `9.81`, `3.14159` + +ALP is not suitable for data that uses a wide range of exponents or a large +number of significant digits, such as vector embeddings which typically span the +full floating-point range. Such use cases can continue to use existing Parquet +features such as `PLAIN` or [`BYTE_STREAM_SPLIT`] encoding followed by `ZSTD` or +`SNAPPY` general purpose compression. + +Decimal values can be stored with Parquet's `DECIMAL` logical type, but it +requires the precision and scale to be known and declared up front and can not +store values outside of that +range. For this reason, it is common for systems where the exact shape +of their data is not known beforehand, to store decimal values as `FLOAT` or `DOUBLE`. For example, +JavaScript's only* [number type is `DOUBLE`], common data science tools such as +pandas [infer `FLOAT` for decimal-looking values], and NumPy has [no decimal dtype at all]. + +[number type is `DOUBLE`]: https://tc39.es/ecma262/#sec-ecmascript-language-types-number-type +[infer `FLOAT` for decimal-looking values]: https://pandas.pydata.org/docs/reference/api/pandas.to_numeric.html +[no decimal dtype at all]: https://numpy.org/doc/stable/reference/arrays.dtypes.html +[`BigInt`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt + +<small>\* JavaScript also has [`BigInt`], but it can only represent integers.</small> + +## Why ALP? + +Encoding floating-point data is a complicated engineering problem due to the nature of floating-point values. They do not exactly represent most real values. This leads to rounding errors that prevent using existing lightweight encodings like Delta and Frame of Reference (FOR). + +Prior to ALP the only `FLOAT`/`DOUBLE` encoding in Parquet (other than `PLAIN`) was [`BYTE_STREAM_SPLIT`](https://parquet.apache.org/docs/file-format/data-pages/encodings/#BYTESTREAMSPLIT). `BYTE_STREAM_SPLIT` does not reduce the size of data but *can* make the compression ratio and speed better when a heavyweight compressor is used afterwards. + +Heavyweight compression buys that ratio at three costs: + - Decode speed -- decompression runs well below what a scan can consume. + - Random access -- reading one value means decoding the whole page. + - Data dependence -- variable-length compression means that decoding a value requires decoding previous values, making it hard to parallelize with modern hardware such as [SIMD instructions] and [GPU]s. + +[SIMD instructions]: https://en.wikipedia.org/wiki/SIMD +[GPU]: https://en.wikipedia.org/wiki/Graphics_processing_unit +[`BYTE_STREAM_SPLIT`]: https://parquet.apache.org/docs/file-format/data-pages/encodings/#BYTESTREAMSPLIT + +ALP is designed to solve all three of these problems for common data patterns, while achieving a similar compression ratio. + +### ALP Performance + +As shown in the charts below, users can expect ALP to be `10x` faster to decode +than `zstd`, and thousands of times faster to retrieve individual values, with +similar compression ratios and comparable compression speed. + +The code and instructions to reproduce these results and try ALP with your own +Parquet datasets can be found in the [alp_benchmark](https://github.com/alamb/alp_benchmark) repository, with the Rust Parquet implementation included. + +<div class="row g-3 td-max-width-on-larger-screens"> + <div class="col-12 col-md-6"> + <img src="/blog/alp/avg_compression_ratio.png" alt="Average compression ratio benchmark" class="img-fluid"> + </div> + <div class="col-12 col-md-6"> + <img src="/blog/alp/avg_compression_speed.png" alt="Average compression speed benchmark" class="img-fluid"> + </div> + <div class="col-12 col-md-6"> + <img src="/blog/alp/avg_decompression_speed.png" alt="Average decompression speed benchmark" class="img-fluid"> + </div> + <div class="col-12 col-md-6"> + <img src="/blog/alp/avg_random_access.png" alt="Average random access benchmark" class="img-fluid"> + </div> + <div> + <b>Figure 1</b>: Average compression ratio, compression speed, decompression speed, and random access performance of <code>PLAIN+ZSTD</code> (per-page <code>zstd</code> compression) and <code>ALP</code> across <code>30</code> datasets on three machines. Higher is better. + Random access speed is measured by decoding <code>100</code> deterministic, uniformly distributed rows from <code>city_temperature_f</code>. + </div> + <p/> +</div> + +The numbers reported are for the pre-release Rust implementation of ALP. We expect +the performance of ALP encoders to improve as the implementations are optimized and +tuned. The current implementations are already faster than `zstd` in many cases, +even though most `zstd` implementations have already been heavily optimized. + +## Technical overview + +ALP was developed by the [Database Architectures Group at CWI](https://www.cwi.nl/en/research/database-architectures/) Review Comment: Added here - https://github.com/apache/parquet-site/pull/195/commits/8010bd2388dade2d011a14446f33704e7a8732b4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
