Hi Azim, Thank you for the valuable feedback. The rationale behind `BYTE_STREAM_SPLIT_ZSTD` fallback looks quite solid.
Looking at the POI dataset and the compression achieved with BYTE_STREAM_SPLIT + ZSTD I see ~ 57 and 59 bits per values, so around ~15% compression ratio. Which can be totally achieved with the first 2 bytes (2/8 ~ 25% of the bytes of data) if they exhibit low cardinality in them. Will give this a try after the ALP pseudo decimal impl in arrow. Best Prateek On Mon, Nov 24, 2025 at 10:08 AM Azim Afroozeh <[email protected]> wrote: > Hi everyone, > > Azim here, first author of the ALP paper. > > Great to see the ALP moving into Parquet. I wanted to share one > recommendation that may help when using BYTE_STREAM_SPLIT with ZSTD for > real double-precision data, based on what we learned while designing > ALP_RD. > > Recommendation: > When using BYTE_STREAM_SPLIT as the fallback for real double floating-point > columns, after applying BYTE_STREAM_SPLIT and obtaining the byte streams, > consider a design where only the first two byte streams are compressed with > ZSTD, rather than applying ZSTD to all byte streams. > > Rationale: > During the design of ALP_RD we found that only the high-order bytes contain > meaningful, repeatable patterns. ALP_RD therefore focuses on the first 16 > bits (2bytes) and applies a specialized dictionary-style encoding that > captures redundancy in the sign, exponent, and upper mantissa bits. These > are the parts of floating-point numbers where we consistently observed > structure that is compressible. > > From this experience, I would expect that applying ZSTD only to the first > two BYTE_STREAM_SPLIT streams would achieve similar (or sometimes better) > compression ratios than ALP_RD, while avoiding compression of the remaining > byte streams (the lower mantissa bytes), which are effectively high-entropy > noise. ZSTD generally cannot compress these streams, and in some cases > compressing them actually increases the encoded size. Leaving those byte > streams uncompressed also improves decompression speed. > > By focusing compression only on the first two byte streams, you retain > almost all of the benefit that ALP_RD provided while keeping the fallback > much simpler and avoiding negative compression on noisy byte streams. > > Technical note: > Since ZSTD is a page-level compression codec and BYTE_STREAM_SPLIT is an > encoding, this selective approach cannot be expressed with the current > layering. However, if you consider introducing a new encoding (for example, > something like BYTE_STREAM_SPLIT_ZSTD), that encoding could internally > apply ZSTD only to the first two byte streams and leave the remaining > streams uncompressed. > > Happy to share more details if useful. > > Best, > Azim > > On 2025/11/21 01:21:26 Prateek Gaur via dev wrote: > > Hi team, > > > > *ALP ---> ALP PeudoDecimal* > > > > As is visible from the numbers above and as stated in the paper too for > > real double values, i.e the values with high precision points, it is very > > difficult to get a good compression ratio. > > > > This combined with the fact that we want to keep the spec/implementation > > simpler, stating Antoine directly here > > > > `*2. Do not include the ALPrd fallback which is a homegrown dictionary* > > > > *encoding without dictionary reuse accross pages, and instead rely on > > awell-known Parquet encoding (such as BYTE_STREAM_SPLIT?)*` > > > > Also based on some discussion I had with Julien in person and the > biweekly > > meeting with a number of you. > > > > We'll be going with ALPpd (pseudo decimal) as the first > > implementation relying on the query engine based on its own heuristics to > > decide on the right fallback to BYTE_STREAM_SPLIT of ZSTD. > > > > Best > > Prateek > > > > > > > > On Thu, Nov 20, 2025 at 5:09 PM Prateek Gaur <[email protected]> > > wrote: > > > > > Sheet with numbers > > > < > > https://docs.google.com/spreadsheets/d/1NmCg0WZKeZUc6vNXXD8M3GIyNqF_H3goj6mVbT8at7A/edit?gid=1351944517#gid=1351944517 > > > > > . > > > > > > On Thu, Nov 20, 2025 at 5:09 PM PRATEEK GAUR <[email protected]> wrote: > > > > > >> Hi team, > > >> > > >> There was a request from a few folks, Antoine Pitrou and Adam Reeve if > I > > >> remember correctly, to perform the experiment on some of the papers > that > > >> talked about BYTE_STREAM_SPLIT for completeness. > > >> I wanted to share the numbers for the same in this sheet. At this > point > > >> we have numbers on a wide variety of data. > > >> (Will have to share the sheet from my snowflake account as our laptops > > >> have fair bit of restriction with respect to copy paste permissions :) > ) > > >> > > >> Best > > >> Prateek > > >> > > >> On Thu, Nov 20, 2025 at 2:25 PM PRATEEK GAUR <[email protected]> wrote: > > >> > > >>> Hi Julien, > > >>> > > >>> Yes based on > > >>> > > >>> - Numbers presented > > >>> - Discussions over the doc and > > >>> - Multiple discussions in the biweekly meeting > > >>> > > >>> We are in a stage where we agree this is the right encoding to add > and > > >>> we can move to the DRAFT/POC stage from DISCUSS stage. > > >>> Will start working on the PR for the same. > > >>> > > >>> Thanks for bringing this up. > > >>> Prateek > > >>> > > >>> On Thu, Nov 20, 2025 at 8:16 AM Julien Le Dem <[email protected]> > wrote: > > >>> > > >>>> @PRATEEK GAUR <[email protected]> : Would you agree that we are past > > >>>> the DISCUSS step and into the DRAFT/POC phase according to the > proposals > > >>>> process < > https://github.com/apache/parquet-format/tree/master/proposals > > >>>> >? > > >>>> If yes, could you open a PR on this page to add this proposal to the > > >>>> list? > > >>>> https://github.com/apache/parquet-format/tree/master/proposals > > >>>> Thank you! > > >>>> > > >>>> > > >>>> On Thu, Oct 30, 2025 at 2:38 PM Andrew Lamb <[email protected]> > > >>>> wrote: > > >>>> > > >>>> > I have filed a ticket[1] in arrow-rs to track prototyping ALP in > the > > >>>> Rust > > >>>> > Parquet reader if anyone is interested > > >>>> > > > >>>> > Andrew > > >>>> > > > >>>> > [1]: https://github.com/apache/arrow-rs/issues/8748 > > >>>> > > > >>>> > On Wed, Oct 22, 2025 at 1:33 PM Micah Kornfield < > > >>>> [email protected]> > > >>>> > wrote: > > >>>> > > > >>>> > > > > > >>>> > > > C++, Java and Rust support them for sure. I feel like we > should > > >>>> > > > probably default to V2 at some point. > > >>>> > > > > >>>> > > > > >>>> > > I seem to recall, some of the vectorized java readers (Iceberg, > > >>>> Spark) > > >>>> > > might not support V2 data pages (but I might be confusing this > with > > >>>> > > encodings). But this is only a vague recollection. > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > On Wed, Oct 22, 2025 at 6:38 AM Andrew Lamb < > [email protected] > > >>>> > > > >>>> > > wrote: > > >>>> > > > > >>>> > > > > Someone has to add V2 data pages to > > >>>> > > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > https://github.com/apache/parquet-site/blob/production/content/en/docs/File%20Format/implementationstatus.md > > >>>> > > > > :) > > >>>> > > > > > >>>> > > > Your wish is my command: > > >>>> > https://github.com/apache/parquet-site/pull/124 > > >>>> > > > > > >>>> > > > As the format grows in popularity and momentum builds to > evolve, > > >>>> I feel > > >>>> > > the > > >>>> > > > content on the parquet.apache.org site could use refreshing / > > >>>> > updating. > > >>>> > > > So, while I had the site open, I made some other PRs to > scratch > > >>>> various > > >>>> > > > itches > > >>>> > > > > > >>>> > > > (I am absolutely 🎣 for someone to please review 🙏): > > >>>> > > > > > >>>> > > > 1. Add Variant/Geometry/Geography types to implementation > status > > >>>> > matrix: > > >>>> > > > https://github.com/apache/parquet-site/pull/123 > > >>>> > > > 2. Improve introduction / overview, add more links to spec and > > >>>> > > > implementation status: > > >>>> https://github.com/apache/parquet-site/pull/125 > > >>>> > > > > > >>>> > > > > > >>>> > > > Thanks, > > >>>> > > > Andrew > > >>>> > > > > > >>>> > > > On Wed, Oct 22, 2025 at 4:09 AM Antoine Pitrou < > > >>>> [email protected]> > > >>>> > > wrote: > > >>>> > > > > > >>>> > > > > > > >>>> > > > > Hi Julien, hi all, > > >>>> > > > > > > >>>> > > > > On Mon, 20 Oct 2025 15:14:58 -0700 > > >>>> > > > > Julien Le Dem <[email protected]> wrote: > > >>>> > > > > > > > >>>> > > > > > Another question from me: > > >>>> > > > > > > > >>>> > > > > > Since the goal is to not use compression at all in this > case > > >>>> (no > > >>>> > > ZSTD) > > >>>> > > > > > I'm assuming we would be using either: > > >>>> > > > > > - the Data Page V1 with UNCOMPRESSED in the > > >>>> ColumnMetadata.column > > >>>> > > > > > < > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > https://github.com/apache/parquet-format/blob/786142e26740487930ddc3ec5e39d780bd930907/src/main/thrift/parquet.thrift#L887 > > >>>> > > > > > > > >>>> > > > > > field. > > >>>> > > > > > - the Data Page V2 with false in the > > >>>> DataPageHeaderV2.is_compressed > > >>>> > > > > > < > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > https://github.com/apache/parquet-format/blob/786142e26740487930ddc3ec5e39d780bd930907/src/main/thrift/parquet.thrift#L746 > > >>>> > > > > > > > >>>> > > > > > field > > >>>> > > > > > The second helping decide if we can selectively compress > some > > >>>> pages > > >>>> > > if > > >>>> > > > > they > > >>>> > > > > > are less compressed by the > > >>>> > > > > > A few years ago there was a question on the support of the > > >>>> > > DATA_PAGE_V2 > > >>>> > > > > and > > >>>> > > > > > I was curious to hear a refresh on how that's generally > > >>>> supported > > >>>> > in > > >>>> > > > > > Parquet implementations. The is_compressed field was > exactly > > >>>> > intended > > >>>> > > > to > > >>>> > > > > > avoid block compression when the encoding itself is good > > >>>> enough. > > >>>> > > > > > > >>>> > > > > Someone has to add V2 data pages to > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > https://github.com/apache/parquet-site/blob/production/content/en/docs/File%20Format/implementationstatus.md > > >>>> > > > > :) > > >>>> > > > > > > >>>> > > > > C++, Java and Rust support them for sure. I feel like we > should > > >>>> > > > > probably default to V2 at some point. > > >>>> > > > > > > >>>> > > > > Also see https://github.com/apache/parquet-java/issues/3344 > for > > >>>> > Java. > > >>>> > > > > > > >>>> > > > > Regards > > >>>> > > > > > > >>>> > > > > Antoine. > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > > >>>> > > > > > Julien > > >>>> > > > > > > > >>>> > > > > > On Mon, Oct 20, 2025 at 11:57 AM Andrew Lamb > > >>>> > > > > <[email protected]> wrote: > > >>>> > > > > > > > >>>> > > > > > > Thanks again Prateek and co for pushing this along! > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > 1. Design and write our own Parquet-ALP spec so that > > >>>> > > > implementations > > >>>> > > > > > > > know exactly how to encode and represent data > > >>>> > > > > > > > > >>>> > > > > > > 100% agree with this (similar to what was done for > > >>>> > ParquetVariant) > > >>>> > > > > > > > > >>>> > > > > > > > 2. I may be missing something, but the paper doesn't > seem > > >>>> to > > >>>> > > > > mention > > >>>> > > > > > > non-finite values (such as +/-Inf and NaNs). > > >>>> > > > > > > > > >>>> > > > > > > I think they are handled via the "Exception" mechanism. > > >>>> Vortex's > > >>>> > > ALP > > >>>> > > > > > > implementation (below) does appear to handle finite > > >>>> numbers[2] > > >>>> > > > > > > > > >>>> > > > > > > > 3. It seems there is a single implementation, which is > > >>>> the one > > >>>> > > > > published > > >>>> > > > > > > > together with the paper. It is not obvious that it > will be > > >>>> > > > > > > > maintained in the future, and reusing it is probably > not > > >>>> an > > >>>> > > option > > >>>> > > > > for > > >>>> > > > > > > > non-C++ Parquet implementations > > >>>> > > > > > > > > >>>> > > > > > > My understanding from the call was that Prateek and team > > >>>> > > > re-implemented > > >>>> > > > > > > ALP (did not use the implementation from CWI[3]) but > that > > >>>> would > > >>>> > be > > >>>> > > > > good to > > >>>> > > > > > > confirm. > > >>>> > > > > > > > > >>>> > > > > > > There is also a Rust implementation of ALP[1] that is > part > > >>>> of the > > >>>> > > > > Vortex > > >>>> > > > > > > file format implementation. I have not reviewed it to > see > > >>>> if it > > >>>> > > > > deviates > > >>>> > > > > > > from the algorithm presented in the paper. > > >>>> > > > > > > > > >>>> > > > > > > Andrew > > >>>> > > > > > > > > >>>> > > > > > > [1]: > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > https://github.com/vortex-data/vortex/blob/534821969201b91985a8735b23fc0c415a425a56/encodings/alp/src/lib.rs > > >>>> > > > > > > [2]: > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > https://github.com/vortex-data/vortex/blob/534821969201b91985a8735b23fc0c415a425a56/encodings/alp/src/alp/compress.rs#L266-L281 > > >>>> > > > > > > [3]: https://github.com/cwida/ALP > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > On Mon, Oct 20, 2025 at 4:47 AM Antoine Pitrou > > >>>> > > > > <[email protected]> wrote: > > >>>> > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > Hello, > > >>>> > > > > > > > > > >>>> > > > > > > > Thanks for doing this and I agree the numbers look > > >>>> impressive. > > >>>> > > > > > > > > > >>>> > > > > > > > I would ask if possible for more data points: > > >>>> > > > > > > > > > >>>> > > > > > > > 1. More datasets: you could for example look at the > > >>>> datasets > > >>>> > that > > >>>> > > > > were > > >>>> > > > > > > > used to originally evalute BYTE_STREAM_SPLIT (see > > >>>> > > > > > > > https://issues.apache.org/jira/browse/PARQUET-1622 > and > > >>>> > > > specifically > > >>>> > > > > > > > the Google Doc linked there) > > >>>> > > > > > > > > > >>>> > > > > > > > 2. Comparison to BYTE_STREAM_SPLIT + LZ4 and > > >>>> BYTE_STREAM_SPLIT > > >>>> > + > > >>>> > > > ZSTD > > >>>> > > > > > > > > > >>>> > > > > > > > 3. Optionally, some perf numbers on x86 too, but I > expect > > >>>> that > > >>>> > > ALP > > >>>> > > > > will > > >>>> > > > > > > > remain very good there as well > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > I also have the following reservations towards ALP: > > >>>> > > > > > > > > > >>>> > > > > > > > 1. There is no published official spec AFAICT, just a > > >>>> research > > >>>> > > > paper. > > >>>> > > > > > > > > > >>>> > > > > > > > 2. I may be missing something, but the paper doesn't > seem > > >>>> to > > >>>> > > > mention > > >>>> > > > > > > > non-finite values (such as +/-Inf and NaNs). > > >>>> > > > > > > > > > >>>> > > > > > > > 3. It seems there is a single implementation, which is > > >>>> the one > > >>>> > > > > published > > >>>> > > > > > > > together with the paper. It is not obvious that it > will be > > >>>> > > > > > > > maintained in the future, and reusing it is probably > not > > >>>> an > > >>>> > > option > > >>>> > > > > for > > >>>> > > > > > > > non-C++ Parquet implementations > > >>>> > > > > > > > > > >>>> > > > > > > > 4. The encoding itself is complex, since it involves a > > >>>> fallback > > >>>> > > on > > >>>> > > > > > > > another encoding if the primary encoding (which > > >>>> constitutes the > > >>>> > > > real > > >>>> > > > > > > > innovation) doesn't work out on a piece of data. > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > Based on this, I would say that if we think ALP is > > >>>> attractive > > >>>> > for > > >>>> > > > us, > > >>>> > > > > > > > we may want to incorporate our own version of ALP with > the > > >>>> > > > following > > >>>> > > > > > > > changes: > > >>>> > > > > > > > > > >>>> > > > > > > > 1. Design and write our own Parquet-ALP spec so that > > >>>> > > > implementations > > >>>> > > > > > > > know exactly how to encode and represent data > > >>>> > > > > > > > > > >>>> > > > > > > > 2. Do not include the ALPrd fallback which is a > homegrown > > >>>> > > > dictionary > > >>>> > > > > > > > encoding without dictionary reuse accross pages, and > > >>>> instead > > >>>> > rely > > >>>> > > > on > > >>>> > > > > a > > >>>> > > > > > > > well-known Parquet encoding (such as > BYTE_STREAM_SPLIT?) > > >>>> > > > > > > > > > >>>> > > > > > > > 3. Replace the FOR encoding inside ALP, which aims at > > >>>> > compressing > > >>>> > > > > > > > integers efficiently, with our own DELTA_BINARY_PACKED > > >>>> (which > > >>>> > has > > >>>> > > > the > > >>>> > > > > > > > same qualities and is already available in Parquet > > >>>> > > implementations) > > >>>> > > > > > > > > > >>>> > > > > > > > Regards > > >>>> > > > > > > > > > >>>> > > > > > > > Antoine. > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > On Thu, 16 Oct 2025 14:47:33 -0700 > > >>>> > > > > > > > PRATEEK GAUR <[email protected]> wrote: > > >>>> > > > > > > > > Hi team, > > >>>> > > > > > > > > > > >>>> > > > > > > > > We spent some time evaluating ALP compression and > > >>>> > decompression > > >>>> > > > > > > compared > > >>>> > > > > > > > to > > >>>> > > > > > > > > other encoding alternatives like CHIMP/GORILLA and > > >>>> > compression > > >>>> > > > > > > techniques > > >>>> > > > > > > > > like SNAPPY/LZ4/ZSTD. We presented these numbers to > the > > >>>> > > community > > >>>> > > > > > > members > > >>>> > > > > > > > > on October 15th in the biweekly parquet meeting. ( I > > >>>> can't > > >>>> > seem > > >>>> > > > > to > > >>>> > > > > > > access > > >>>> > > > > > > > > the recording, so please let me know what access > rules > > >>>> I need > > >>>> > > to > > >>>> > > > > get to > > >>>> > > > > > > > be > > >>>> > > > > > > > > able to view it ) > > >>>> > > > > > > > > > > >>>> > > > > > > > > We did this evaluation over some datasets pointed by > > >>>> the ALP > > >>>> > > > paper > > >>>> > > > > and > > >>>> > > > > > > > some > > >>>> > > > > > > > > pointed by the parquet community. > > >>>> > > > > > > > > > > >>>> > > > > > > > > The results are available in the following document > > >>>> > > > > > > > > < > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > https://docs.google.com/document/d/1PlyUSfqCqPVwNt8XA-CfRqsbc0NKRG0Kk1FigEm3JOg/edit?tab=t.0 > > >>>> > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > : > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > https://docs.google.com/document/d/1PlyUSfqCqPVwNt8XA-CfRqsbc0NKRG0Kk1FigEm3JOg > > >>>> > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > Based on the numbers we see > > >>>> > > > > > > > > > > >>>> > > > > > > > > - ALP is comparable to ZSTD(level=1) in terms of > > >>>> > > compression > > >>>> > > > > ratio > > >>>> > > > > > > > and > > >>>> > > > > > > > > much better compared to other schemes. (numbers > in > > >>>> the > > >>>> > sheet > > >>>> > > > > are > > >>>> > > > > > > bytes > > >>>> > > > > > > > > needed to encode each value ) > > >>>> > > > > > > > > - ALP going quite well in terms of decompression > > >>>> speed > > >>>> > > > (numbers > > >>>> > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > sheet are bytes decompressed per second) > > >>>> > > > > > > > > > > >>>> > > > [message truncated...] >
