alamb commented on code in PR #195: URL: https://github.com/apache/parquet-site/pull/195#discussion_r3795978722
########## content/en/blog/features/alp_encoding.md: ########## @@ -0,0 +1,363 @@ +--- +title: "ALP lightweight floating-point encoding in Apache Parquet" +date: 2026-08-14 +description: "Fast, random access, GPU and SIMD-friendly compression and decompression; similar in size to ZSTD" +author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb)" +categories: ["features"] +--- + +Apache Parquet adopts ALP (Adaptive Lossless floating-Point) -- a new lightweight floating-point encoding that shows similar compression ratio to heavyweight compressors like zstd and **much** faster decompression speed + "random access" support. + +"Random access" is a key feature of ALP -- it allows retrieving a single value without decoding the entire page. This property is becoming increasingly important for workloads such as point lookups with text and vector indexes. + +---- + +ALP was developed by the [Database Architectures Group at CWI](https://www.cwi.nl/en/research/database-architectures/) and uses smart tricks to represent floating-point data as integers. + +ALP shines at data that was originally decimal and stored as FLOAT/DOUBLE: +- Monetary values (exchange rates, public funds, stocks, prices, etc.) +- Geographic coordinates (longitude/latitude) +- Scientific measures (temperature, pressure, speed, degrees, etc.) +- Timestamps stored as floating-point + +## Why ALP? + +Encoding floating point data is a complicated engineering problem. Prior to ALP the only FLOAT/DOUBLE encoding in Parquet (other than Plain) was [Byte Stream Split](https://parquet.apache.org/docs/file-format/data-pages/encodings/#BYTESTREAMSPLIT). Byte Stream Split does not reduce the size of data but *can* make the compression ratio and speed better when a heavyweight compressor is used afterwards. + +Using heavyweight compression, however, introduces two problems: +- Slow decompression speed +- Requires decoding the entire page to retrieve a single value (the "random access" problem) + +ALP succeeds at solving both of these problems without losing compression ratio. Review Comment: ```suggestion ALP is designed to solve both of these problems for common data patterns, while achieving a similar compression ratio. ``` ########## content/en/blog/features/alp_encoding.md: ########## @@ -0,0 +1,363 @@ +--- +title: "ALP lightweight floating-point encoding in Apache Parquet" +date: 2026-08-14 +description: "Fast, random access, GPU and SIMD-friendly compression and decompression; similar in size to ZSTD" +author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb)" Review Comment: ```suggestion author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb), [Prateek Gaur](https://github.com/prtkgaur)" ``` ########## content/en/blog/features/alp_encoding.md: ########## @@ -0,0 +1,363 @@ +--- +title: "ALP lightweight floating-point encoding in Apache Parquet" +date: 2026-08-14 +description: "Fast, random access, GPU and SIMD-friendly compression and decompression; similar in size to ZSTD" +author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb)" +categories: ["features"] +--- + +Apache Parquet adopts ALP (Adaptive Lossless floating-Point) -- a new lightweight floating-point encoding that shows similar compression ratio to heavyweight compressors like zstd and **much** faster decompression speed + "random access" support. + +"Random access" is a key feature of ALP -- it allows retrieving a single value without decoding the entire page. This property is becoming increasingly important for workloads such as point lookups with text and vector indexes. + +---- + +ALP was developed by the [Database Architectures Group at CWI](https://www.cwi.nl/en/research/database-architectures/) and uses smart tricks to represent floating-point data as integers. + +ALP shines at data that was originally decimal and stored as FLOAT/DOUBLE: +- Monetary values (exchange rates, public funds, stocks, prices, etc.) +- Geographic coordinates (longitude/latitude) +- Scientific measures (temperature, pressure, speed, degrees, etc.) +- Timestamps stored as floating-point + +## Why ALP? + +Encoding floating point data is a complicated engineering problem. Prior to ALP the only FLOAT/DOUBLE encoding in Parquet (other than Plain) was [Byte Stream Split](https://parquet.apache.org/docs/file-format/data-pages/encodings/#BYTESTREAMSPLIT). Byte Stream Split does not reduce the size of data but *can* make the compression ratio and speed better when a heavyweight compressor is used afterwards. Review Comment: It would be good to say why "it is a complicated engineering problem" -- for example that floating point data can vary widely between datasets and can have wide ranges of distributions. You could then also make the point that many datasets don't actually use the full range of floating point numbers and that ALP exploits that observation ########## content/en/blog/features/alp_encoding.md: ########## @@ -0,0 +1,363 @@ +--- +title: "ALP lightweight floating-point encoding in Apache Parquet" +date: 2026-08-14 +description: "Fast, random access, GPU and SIMD-friendly compression and decompression; similar in size to ZSTD" +author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb)" +categories: ["features"] +--- + +Apache Parquet adopts ALP (Adaptive Lossless floating-Point) -- a new lightweight floating-point encoding that shows similar compression ratio to heavyweight compressors like zstd and **much** faster decompression speed + "random access" support. + +"Random access" is a key feature of ALP -- it allows retrieving a single value without decoding the entire page. This property is becoming increasingly important for workloads such as point lookups with text and vector indexes. Review Comment: I think we should also mention the friendliness to modern hardware (SIMD and GPU as well). Maybe something like ```suggestion Apache Parquet has added ALP (Adaptive Lossless floating-Point) -- a new lightweight floating-point encoding with similar compression to heavyweight compressors like zstd and **much** faster decompression speed, supports random access, and is SIMD and GPU friendly. ``` Then we can fill out the claims in the sections of the article ########## content/en/blog/features/alp_encoding.md: ########## @@ -0,0 +1,363 @@ +--- +title: "ALP lightweight floating-point encoding in Apache Parquet" +date: 2026-08-14 +description: "Fast, random access, GPU and SIMD-friendly compression and decompression; similar in size to ZSTD" +author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb)" +categories: ["features"] +--- + +Apache Parquet adopts ALP (Adaptive Lossless floating-Point) -- a new lightweight floating-point encoding that shows similar compression ratio to heavyweight compressors like zstd and **much** faster decompression speed + "random access" support. + +"Random access" is a key feature of ALP -- it allows retrieving a single value without decoding the entire page. This property is becoming increasingly important for workloads such as point lookups with text and vector indexes. + +---- + +ALP was developed by the [Database Architectures Group at CWI](https://www.cwi.nl/en/research/database-architectures/) and uses smart tricks to represent floating-point data as integers. + +ALP shines at data that was originally decimal and stored as FLOAT/DOUBLE: +- Monetary values (exchange rates, public funds, stocks, prices, etc.) +- Geographic coordinates (longitude/latitude) +- Scientific measures (temperature, pressure, speed, degrees, etc.) +- Timestamps stored as floating-point + +## Why ALP? + +Encoding floating point data is a complicated engineering problem. Prior to ALP the only FLOAT/DOUBLE encoding in Parquet (other than Plain) was [Byte Stream Split](https://parquet.apache.org/docs/file-format/data-pages/encodings/#BYTESTREAMSPLIT). Byte Stream Split does not reduce the size of data but *can* make the compression ratio and speed better when a heavyweight compressor is used afterwards. + +Using heavyweight compression, however, introduces two problems: +- Slow decompression speed +- Requires decoding the entire page to retrieve a single value (the "random access" problem) + +ALP succeeds at solving both of these problems without losing compression ratio. + +### Benchmark + +To demonstrate ALP's performance we used the same datasets used in the paper. + +- The first 13 are time series data, mostly scientific measurements +- 17 datasets are more representative of +doubles stored in classical database workloads, mostly monetary data + +<details><summary>Full list of datasets </summary> + +| Name | Semantics | Source | N° of Values | +|---|---|---|---:| +| **Time series** | | | | +| Air-Pressure | Barometric Pressure (kPa) | NEON | 137,721,453 | +| Basel-temp | Temperature (C°) | meteoblue | 123,480 | +| Basel-wind | Wind Speed (Km/h) | meteoblue | 123,480 | +| Bird-migration | Coordinates (lat, lon) | InfluxDB | 17,964 | +| Bitcoin-price | Exchange Rate (BTC-USD) | InfluxDB | 2,686 | +| City-Temp | Temperature (F°) | Udayton | 2,905,887 | +| Dew-Point-Temp | Temperature (C°) | NEON | 5,413,914 | +| IR-bio-temp | Temperature (C°) | NEON | 380,817,839 | +| PM10-dust | Dust content in air (mg/m3) | NEON | 221,568 | +| Stocks-DE | Monetary (Stocks) | INFORE | 43,565,658 | +| Stocks-UK | Monetary (Stocks) | INFORE | 59,305,326 | +| Stocks-USA | Monetary (Stocks) | INFORE | 282,076,179 | +| Wind-dir | Angle Degree (0°-360°) | NEON | 198,898,762 | +| **Non Time series** | | | | +| Arade/4 | Energy | PBI Bench. | 9,888,775 | +| Blockchain-tr | Monetary (BTC) | Blockchain | 231,031 | +| CMS/1 | Monetary Avg. (USD) | PBI Bench. | 18,575,752 | +| CMS/25 | Monetary Std. Dev. (USD) | PBI Bench. | 18,575,752 | +| CMS/9 | Discrete Count | PBI Bench. | 18,575,752 | +| Food-prices | Monetary (USD) | WFP | 2,050,638 | +| Gov/10 | Monetary (USD) | PBI Bench. | 141,123,827 | +| Gov/26 | Monetary (USD) | PBI Bench. | 141,123,827 | +| Gov/30 | Monetary (USD) | PBI Bench. | 141,123,827 | +| Gov/31 | Monetary (USD) | PBI Bench. | 141,123,827 | +| Gov/40 | Monetary (USD) | PBI Bench. | 141,123,827 | +| Medicare/1 | Monetary Avg. (USD) | PBI Bench. | 9,287,876 | +| Medicare/9 | Discrete Count | PBI Bench. | 9,287,876 | +| NYC/29 | Coordinates (lon) | PBI Bench. | 17,446,346 | +| POI-lat | Coordinates (lat, in radians) | Kaggle | 424,205 | +| POI-lon | Coordinates (lon, in radians) | Kaggle | 424,205 | +| SD-bench | Storage Capacity (GB) | Kaggle | 8,927 | + +</details> + +<br> + +We compare compression ratio and [de]compression speed between: +- Plain +- Plain + ZSTD +- Byte Stream Split + ZSTD +- ALP + +### Benchmark results + +| Encoding scheme | Compression (GB/s) | Decompression (GB/s) | Compressed size (bits/value) | +|---|---:|---:|---:| +| **PLAIN** | **65.547** | **76.644** | **64.01** | +| **PLAIN + ZSTD** | **1.401** | **3.236** | **22.75** | +| **BYTE_STREAM_SPLIT + ZSTD** | **1.786** | **5.132** | **32.76** | +| **ALP** | **1.273** | **33.805** | **24.27** | + +<details><summary>Full benchmark results </summary> + +| Dataset | Parquet choice | Compression (GB/s) | Decompression (GB/s) | Compressed size (bits/value) | +|---|---|---:|---:|---:| +| arade4 | PLAIN | 63.775 | 68.205 | 64.01 | +| arade4 | PLAIN + ZSTD | 0.574 | 1.499 | 37.39 | +| arade4 | BYTE_STREAM_SPLIT + ZSTD | 1.875 | 4.980 | 54.58 | +| arade4 | ALP | 1.698 | 31.860 | 24.99 | +| basel_temp_f | PLAIN | 33.559 | 57.828 | 64.01 | +| basel_temp_f | PLAIN + ZSTD | 0.458 | 1.656 | 23.07 | +| basel_temp_f | BYTE_STREAM_SPLIT + ZSTD | 1.181 | 2.080 | 54.59 | +| basel_temp_f | ALP | 0.534 | 28.058 | 29.23 | +| basel_wind_f | PLAIN | 52.641 | 76.402 | 64.01 | +| basel_wind_f | PLAIN + ZSTD | 0.591 | 1.733 | 18.53 | +| basel_wind_f | BYTE_STREAM_SPLIT + ZSTD | 1.405 | 2.191 | 54.12 | +| basel_wind_f | ALP | 0.576 | 29.778 | 29.87 | +| bird_migration_f | PLAIN | 65.238 | 93.070 | 64.01 | +| bird_migration_f | PLAIN + ZSTD | 0.420 | 1.778 | 23.49 | +| bird_migration_f | BYTE_STREAM_SPLIT + ZSTD | 1.210 | 10.407 | 45.82 | +| bird_migration_f | ALP | 0.164 | 27.212 | 20.24 | +| bitcoin_f | PLAIN | 91.371 | 222.346 | 64.07 | +| bitcoin_f | PLAIN + ZSTD | 0.585 | 1.660 | 50.01 | +| bitcoin_f | BYTE_STREAM_SPLIT + ZSTD | 1.345 | 19.512 | 48.79 | +| bitcoin_f | ALP | 0.047 | 30.608 | 27.18 | +| bitcoin_transactions_f | PLAIN | 61.862 | 76.608 | 64.01 | +| bitcoin_transactions_f | PLAIN + ZSTD | 1.081 | 1.985 | 47.96 | +| bitcoin_transactions_f | BYTE_STREAM_SPLIT + ZSTD | 1.412 | 2.580 | 56.65 | +| bitcoin_transactions_f | ALP | 0.572 | 21.192 | 41.27 | +| city_temperature_f | PLAIN | 74.404 | 75.291 | 64.01 | +| city_temperature_f | PLAIN + ZSTD | 0.561 | 1.368 | 17.67 | +| city_temperature_f | BYTE_STREAM_SPLIT + ZSTD | 0.962 | 3.569 | 16.64 | +| city_temperature_f | ALP | 1.972 | 37.725 | 10.80 | +| cms1 | PLAIN | 64.257 | 62.655 | 64.01 | +| cms1 | PLAIN + ZSTD | 0.627 | 1.600 | 26.84 | +| cms1 | BYTE_STREAM_SPLIT + ZSTD | 0.670 | 1.651 | 38.64 | +| cms1 | ALP | 1.061 | 18.514 | 35.19 | +| cms25 | PLAIN | 70.317 | 71.847 | 64.01 | +| cms25 | PLAIN + ZSTD | 0.798 | 1.829 | 58.11 | +| cms25 | BYTE_STREAM_SPLIT + ZSTD | 1.287 | 4.345 | 56.72 | +| cms25 | ALP | 1.516 | 21.901 | 41.17 | +| cms9 | PLAIN | 65.334 | 67.735 | 64.01 | +| cms9 | PLAIN + ZSTD | 0.668 | 1.436 | 11.71 | +| cms9 | BYTE_STREAM_SPLIT + ZSTD | 2.151 | 5.502 | 10.07 | +| cms9 | ALP | 1.932 | 34.393 | 12.16 | +| food_prices | PLAIN | 71.786 | 75.631 | 64.01 | +| food_prices | PLAIN + ZSTD | 0.574 | 1.356 | 18.13 | +| food_prices | BYTE_STREAM_SPLIT + ZSTD | 0.744 | 1.865 | 25.47 | +| food_prices | ALP | 0.887 | 20.747 | 23.20 | +| gov10 | PLAIN | 67.983 | 70.573 | 64.01 | +| gov10 | PLAIN + ZSTD | 0.486 | 1.259 | 29.12 | +| gov10 | BYTE_STREAM_SPLIT + ZSTD | 0.648 | 1.740 | 37.31 | +| gov10 | ALP | 1.119 | 24.621 | 29.88 | +| gov26 | PLAIN | 66.105 | 67.432 | 64.01 | +| gov26 | PLAIN + ZSTD | 10.325 | 23.010 | 0.20 | +| gov26 | BYTE_STREAM_SPLIT + ZSTD | 8.112 | 18.208 | 0.24 | +| gov26 | ALP | 1.876 | 84.789 | 1.40 | +| gov30 | PLAIN | 61.089 | 64.308 | 64.01 | +| gov30 | PLAIN + ZSTD | 2.038 | 5.186 | 4.52 | +| gov30 | BYTE_STREAM_SPLIT + ZSTD | 1.695 | 4.155 | 6.14 | +| gov30 | ALP | 1.020 | 33.276 | 17.88 | +| gov31 | PLAIN | 70.238 | 71.923 | 64.01 | +| gov31 | PLAIN + ZSTD | 3.821 | 8.994 | 1.65 | +| gov31 | BYTE_STREAM_SPLIT + ZSTD | 3.780 | 9.643 | 2.47 | +| gov31 | ALP | 1.659 | 50.153 | 6.77 | +| gov40 | PLAIN | 74.751 | 75.933 | 64.01 | +| gov40 | PLAIN + ZSTD | 9.232 | 17.740 | 0.43 | +| gov40 | BYTE_STREAM_SPLIT + ZSTD | 6.477 | 13.890 | 0.62 | +| gov40 | ALP | 1.879 | 77.866 | 2.59 | +| medicare1 | PLAIN | 73.791 | 65.523 | 64.01 | +| medicare1 | PLAIN + ZSTD | 0.552 | 1.508 | 31.68 | +| medicare1 | BYTE_STREAM_SPLIT + ZSTD | 0.800 | 2.166 | 45.27 | +| medicare1 | ALP | 0.958 | 19.660 | 40.46 | +| medicare9 | PLAIN | 73.908 | 75.228 | 64.01 | +| medicare9 | PLAIN + ZSTD | 0.706 | 1.471 | 11.86 | +| medicare9 | BYTE_STREAM_SPLIT + ZSTD | 2.126 | 5.717 | 10.19 | +| medicare9 | ALP | 1.938 | 36.311 | 12.82 | +| neon_air_pressure | PLAIN | 70.685 | 73.563 | 64.01 | +| neon_air_pressure | PLAIN + ZSTD | 0.789 | 2.051 | 11.85 | +| neon_air_pressure | BYTE_STREAM_SPLIT + ZSTD | 0.782 | 2.268 | 28.51 | +| neon_air_pressure | ALP | 1.876 | 36.770 | 16.48 | +| neon_bio_temp_c | PLAIN | 63.818 | 69.314 | 64.01 | +| neon_bio_temp_c | PLAIN + ZSTD | 0.522 | 1.513 | 16.84 | +| neon_bio_temp_c | BYTE_STREAM_SPLIT + ZSTD | 1.240 | 2.885 | 35.40 | +| neon_bio_temp_c | ALP | 1.941 | 35.578 | 10.81 | +| neon_dew_point_temp | PLAIN | 72.097 | 73.614 | 64.01 | +| neon_dew_point_temp | PLAIN + ZSTD | 0.465 | 1.638 | 23.73 | +| neon_dew_point_temp | BYTE_STREAM_SPLIT + ZSTD | 1.503 | 2.370 | 48.00 | +| neon_dew_point_temp | ALP | 1.931 | 32.863 | 13.63 | +| neon_pm10_dust | PLAIN | 51.533 | 70.722 | 64.01 | +| neon_pm10_dust | PLAIN + ZSTD | 0.848 | 1.689 | 7.79 | +| neon_pm10_dust | BYTE_STREAM_SPLIT + ZSTD | 0.634 | 1.682 | 22.21 | +| neon_pm10_dust | ALP | 0.927 | 38.427 | 8.41 | +| neon_wind_dir | PLAIN | 54.631 | 60.583 | 64.01 | +| neon_wind_dir | PLAIN + ZSTD | 0.432 | 1.312 | 24.41 | +| neon_wind_dir | BYTE_STREAM_SPLIT + ZSTD | 1.387 | 3.518 | 42.31 | +| neon_wind_dir | ALP | 1.883 | 47.114 | 15.94 | +| nyc29 | PLAIN | 71.342 | 71.984 | 64.01 | +| nyc29 | PLAIN + ZSTD | 0.611 | 1.539 | 24.67 | +| nyc29 | BYTE_STREAM_SPLIT + ZSTD | 0.939 | 3.768 | 36.91 | +| nyc29 | ALP | 1.679 | 24.137 | 40.43 | +| poi_lat | PLAIN | 58.440 | 50.294 | 64.01 | +| poi_lat | PLAIN + ZSTD | 0.635 | 1.793 | 57.78 | +| poi_lat | BYTE_STREAM_SPLIT + ZSTD | 2.711 | 5.929 | 55.30 | +| poi_lat | ALP | 1.052 | 11.874 | 88.19 | +| poi_lon | PLAIN | 61.546 | 73.497 | 64.01 | +| poi_lon | PLAIN + ZSTD | 0.850 | 1.945 | 60.44 | +| poi_lon | BYTE_STREAM_SPLIT + ZSTD | 2.467 | 5.826 | 57.24 | +| poi_lon | ALP | 1.180 | 14.614 | 79.12 | +| ssd_hdd_benchmarks_f | PLAIN | 66.698 | 112.905 | 64.02 | +| ssd_hdd_benchmarks_f | PLAIN + ZSTD | 0.813 | 1.802 | 12.98 | +| ssd_hdd_benchmarks_f | BYTE_STREAM_SPLIT + ZSTD | 1.265 | 2.850 | 17.42 | +| ssd_hdd_benchmarks_f | ALP | 0.114 | 35.975 | 16.04 | +| stocks_de | PLAIN | 68.397 | 71.924 | 64.01 | +| stocks_de | PLAIN + ZSTD | 0.664 | 1.689 | 10.07 | +| stocks_de | BYTE_STREAM_SPLIT + ZSTD | 0.881 | 2.278 | 33.46 | +| stocks_de | ALP | 1.224 | 35.921 | 11.20 | +| stocks_uk | PLAIN | 60.600 | 64.531 | 64.01 | +| stocks_uk | PLAIN + ZSTD | 0.608 | 1.413 | 11.29 | +| stocks_uk | BYTE_STREAM_SPLIT + ZSTD | 1.180 | 4.006 | 14.89 | +| stocks_uk | ALP | 0.948 | 32.869 | 12.75 | +| stocks_usa_c | PLAIN | 64.214 | 67.851 | 64.01 | +| stocks_usa_c | PLAIN + ZSTD | 0.692 | 1.634 | 8.24 | +| stocks_usa_c | BYTE_STREAM_SPLIT + ZSTD | 0.712 | 2.390 | 26.89 | +| stocks_usa_c | ALP | 2.028 | 39.329 | 7.95 | +| **ALL AVG.** | **PLAIN** | **65.547** | **76.644** | **64.01** | +| **ALL AVG.** | **PLAIN + ZSTD** | **1.401** | **3.236** | **22.75** | +| **ALL AVG.** | **BYTE_STREAM_SPLIT + ZSTD** | **1.786** | **5.132** | **32.76** | +| **ALL AVG.** | **ALP** | **1.273** | **33.805** | **24.27** | + +</details> + +### Random access benchmark + +We also measured random access time for the encodings above. Review Comment: I will also try and make this into a chart ########## content/en/blog/features/alp_encoding.md: ########## @@ -0,0 +1,363 @@ +--- +title: "ALP lightweight floating-point encoding in Apache Parquet" +date: 2026-08-14 +description: "Fast, random access, GPU and SIMD-friendly compression and decompression; similar in size to ZSTD" +author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb)" +categories: ["features"] +--- + +Apache Parquet adopts ALP (Adaptive Lossless floating-Point) -- a new lightweight floating-point encoding that shows similar compression ratio to heavyweight compressors like zstd and **much** faster decompression speed + "random access" support. + +"Random access" is a key feature of ALP -- it allows retrieving a single value without decoding the entire page. This property is becoming increasingly important for workloads such as point lookups with text and vector indexes. + +---- + +ALP was developed by the [Database Architectures Group at CWI](https://www.cwi.nl/en/research/database-architectures/) and uses smart tricks to represent floating-point data as integers. + +ALP shines at data that was originally decimal and stored as FLOAT/DOUBLE: Review Comment: Maybe also worth mentioning this double modeling is very common in real world workloads (for example, Numpy and pandas don't include a decimal type defauly -- e.g. https://numpy.org/devdocs/user/basics.types.html) ########## content/en/blog/features/alp_encoding.md: ########## @@ -0,0 +1,363 @@ +--- +title: "ALP lightweight floating-point encoding in Apache Parquet" +date: 2026-08-14 +description: "Fast, random access, GPU and SIMD-friendly compression and decompression; similar in size to ZSTD" +author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb)" +categories: ["features"] +--- + +Apache Parquet adopts ALP (Adaptive Lossless floating-Point) -- a new lightweight floating-point encoding that shows similar compression ratio to heavyweight compressors like zstd and **much** faster decompression speed + "random access" support. + +"Random access" is a key feature of ALP -- it allows retrieving a single value without decoding the entire page. This property is becoming increasingly important for workloads such as point lookups with text and vector indexes. + +---- + +ALP was developed by the [Database Architectures Group at CWI](https://www.cwi.nl/en/research/database-architectures/) and uses smart tricks to represent floating-point data as integers. + +ALP shines at data that was originally decimal and stored as FLOAT/DOUBLE: +- Monetary values (exchange rates, public funds, stocks, prices, etc.) +- Geographic coordinates (longitude/latitude) +- Scientific measures (temperature, pressure, speed, degrees, etc.) +- Timestamps stored as floating-point + +## Why ALP? + +Encoding floating point data is a complicated engineering problem. Prior to ALP the only FLOAT/DOUBLE encoding in Parquet (other than Plain) was [Byte Stream Split](https://parquet.apache.org/docs/file-format/data-pages/encodings/#BYTESTREAMSPLIT). Byte Stream Split does not reduce the size of data but *can* make the compression ratio and speed better when a heavyweight compressor is used afterwards. + +Using heavyweight compression, however, introduces two problems: +- Slow decompression speed +- Requires decoding the entire page to retrieve a single value (the "random access" problem) + +ALP succeeds at solving both of these problems without losing compression ratio. + +### Benchmark + +To demonstrate ALP's performance we used the same datasets used in the paper. + +- The first 13 are time series data, mostly scientific measurements +- 17 datasets are more representative of +doubles stored in classical database workloads, mostly monetary data + +<details><summary>Full list of datasets </summary> Review Comment: I made some example summary diagrams for consideration: <img width="1080" height="920" alt="Image" src="https://github.com/user-attachments/assets/01a3250d-4bbc-4125-82eb-e558b7d519c1"; /> <img width="1080" height="840" alt="Image" src="https://github.com/user-attachments/assets/d81a7926-855c-48ba-ab8c-e616c730367e"; /> <img width="1080" height="840" alt="Image" src="https://github.com/user-attachments/assets/50f2fcc7-3c32-4879-bcba-fdf413769da6"; /> ########## content/en/blog/features/alp_encoding.md: ########## @@ -0,0 +1,363 @@ +--- +title: "ALP lightweight floating-point encoding in Apache Parquet" +date: 2026-08-14 +description: "Fast, random access, GPU and SIMD-friendly compression and decompression; similar in size to ZSTD" +author: "[Kosta Tarasov](https://github.com/sdf-jkl), [Andrew Lamb](https://github.com/alamb)" +categories: ["features"] +--- + +Apache Parquet adopts ALP (Adaptive Lossless floating-Point) -- a new lightweight floating-point encoding that shows similar compression ratio to heavyweight compressors like zstd and **much** faster decompression speed + "random access" support. + +"Random access" is a key feature of ALP -- it allows retrieving a single value without decoding the entire page. This property is becoming increasingly important for workloads such as point lookups with text and vector indexes. + +---- + +ALP was developed by the [Database Architectures Group at CWI](https://www.cwi.nl/en/research/database-architectures/) and uses smart tricks to represent floating-point data as integers. + +ALP shines at data that was originally decimal and stored as FLOAT/DOUBLE: +- Monetary values (exchange rates, public funds, stocks, prices, etc.) +- Geographic coordinates (longitude/latitude) +- Scientific measures (temperature, pressure, speed, degrees, etc.) +- Timestamps stored as floating-point + +## Why ALP? + +Encoding floating point data is a complicated engineering problem. Prior to ALP the only FLOAT/DOUBLE encoding in Parquet (other than Plain) was [Byte Stream Split](https://parquet.apache.org/docs/file-format/data-pages/encodings/#BYTESTREAMSPLIT). Byte Stream Split does not reduce the size of data but *can* make the compression ratio and speed better when a heavyweight compressor is used afterwards. + +Using heavyweight compression, however, introduces two problems: +- Slow decompression speed +- Requires decoding the entire page to retrieve a single value (the "random access" problem) + +ALP succeeds at solving both of these problems without losing compression ratio. + +### Benchmark + +To demonstrate ALP's performance we used the same datasets used in the paper. + +- The first 13 are time series data, mostly scientific measurements +- 17 datasets are more representative of +doubles stored in classical database workloads, mostly monetary data + +<details><summary>Full list of datasets </summary> Review Comment: I feel like this detail on experimental setup and detailed results is obscuring the main narrative. I recommend we 1. Add a summary chart with results 1. move all these details to an Appendix at end of the blog 2. Make sure we link somewhere so that anyone who cares can reproduce these numbers In terms of reproducer, i recommend we create a repository with the benchmark runner script pulled otu (rather than a branch in arrow-rs) and we can link there. I can do this if it would help -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
