Hi all, Since the earlier PFOR thread I have kept measuring and improved decoding speed significantly, added a delta mode, and evaluated the encoding on real dictionary index runs. The detailed write-up for all of this has been added to the original doc itself : [1] .
First, decode. With code changes and with no encoded byte changing, made PFOR about 40% faster to decode at -O3. Second, added delta mode. Certain data distributions definitely benefit from delta encoding and wanted to explore this capability on top of PFOR's blocking + patching scheme and results looking quite promising for the data distributions that can take advantage of it. Here I added it as a per block decision mode which selects delta mode if the values are close together. I see a comparable compression ratio to delta bit pack hybrid (same idea). And pretty good decompression speed. Added details to the same document. Code for the same is in [2]. Third, PFOR for dictionary index runs. Based on the characteristics of PFOR and the dictionary index runs I was hopeful that I'll get good results with PFOR on dictionary indices. So I did an evaluation of PFOR for dictionary index runs and added the results on the same to the document. I do see value in using PFOR for dictionary index encoding. Looking for feedback from the team. Best Prateek [1] https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/ [2] https://github.com/apache/arrow/pull/50088 c++ (pfor+frame only) [3] https://github.com/apache/arrow/pull/51150 c++ (pfor + dynamic frame or delta) [4] https://github.com/apache/arrow-rs/pull/10977 rust On Mon, Jul 13, 2026 at 8:31 AM PRATEEK GAUR <[email protected]> wrote: > Hi team, > > Just wanted to resurface the doc in your email folders, looking forward to > some suggestions. > I'll work towards adding a few more datasets to the comparison in the > coming days. > > Best > Prateek > > On Wed, Jul 1, 2026 at 9:01 AM PRATEEK GAUR <[email protected]> wrote: > >> Hi Antoine, >> >> Apologies for the delay. Got some time to work on it and updated the doc >> with the requested numbers. >> >> - Obtained the numbers for ZSTD again. I agree they were incorrect. >> - Fixed the algorithm to use decompressed size/data >> - Added BSS + ZSTD to the evaluation metrics >> - Added BSS + Lz4 to the evaluation metrics >> >> >> Updated doc : >> https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/edit?tab=t.0#heading=h.uzgoevv9ajp >> Will be pushing the branch/code for the same today. >> >> Best >> Prateek >> >> On Wed, Dec 10, 2025 at 11:30 AM Antoine Pitrou <[email protected]> >> wrote: >> >>> >>> Hi, >>> >>> I looked at the doc and the stated decompression speeds for ZSTD look >>> highly irrealistic. >>> >>> I think what happens is that you are computing decompression speed as: >>> >>> size of compressed data / time to decompress >>> >>> while you should really compute it as: >>> >>> size of uncompressed data / time to decompress >>> >>> Otherwise you're simply making ZSTD look miserable because it compresses >>> so well. >>> >>> Also, I think you should also add BYTE_STREAM_SPLIT + ZSTD into the mix >>> (and possible BYTE_STREAM_SPLIT + LZ4 if you're going to evaluate LZ4). >>> >>> Regards >>> >>> Antoine. >>> >>> >>> Le 06/12/2025 à 23:24, PRATEEK GAUR a écrit : >>> > Hi team, >>> > >>> > We wanted to share performance numbers for one of the candidate >>> encodings >>> > we have been discussing for parquet, for numeric compression. >>> > >>> > Following doc, PFOR : Encoding >>> > < >>> https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/edit?tab=t.0 >>> >, >>> > talks through numbers on compression speed, compression ratio and >>> > decompression speed for columns values from the clickbench data. Based >>> on >>> > the numbers PFOR gives superior decompression speed compared to >>> > DELTABITPACK and RLEBITPACKHYBRID and fares better on average in >>> > compression ratio. Comparison has also been made with ZSTD compression >>> in >>> > the above doc. >>> > >>> > We plan to expand to a few more datasets and work on a prototype in >>> arrow >>> > cpp code in early Jan. >>> > Looking forward to the feedback from the group. >>> > >>> > Best >>> > Prateek >>> > >>> >>> >>>
