Sorry for my ignorance -- what does DBP stand for? On Tue, Sep 22, 2026 at 1:24 PM Alkis Evlogimenos via dev < [email protected]> wrote:
> I left a comment in the doc. Posting here as well: > > 1. Can we compare PFOR-delta vs DBP? > 2. Is there a design where we can embed DBP blocks between PFOR blocks and > get the same benefits without opening the writer/encoder to more choices? > > > On Tue, Sep 8, 2026 at 5:07 PM PRATEEK GAUR <[email protected]> wrote: > > > Hi all, > > > > Since the earlier PFOR thread I have kept measuring and improved decoding > > speed > > significantly, added a delta mode, and evaluated the encoding on real > > dictionary index runs. > > The detailed write-up for all of this has been added to the original doc > > itself : [1] . > > > > First, decode. > > With code changes and with no encoded byte changing, made PFOR > > about 40% faster to decode at -O3. > > > > Second, added delta mode. > > Certain data distributions definitely benefit from delta encoding and > > wanted to explore this > > capability on top of PFOR's blocking + patching scheme and results > looking > > quite promising > > for the data distributions that can take advantage of it. Here I added it > > as a per block decision > > mode which selects delta mode if the values are close together. I see a > > comparable compression > > ratio to delta bit pack hybrid (same idea). And pretty good decompression > > speed. Added details to > > the same document. Code for the same is in [2]. > > > > Third, PFOR for dictionary index runs. > > Based on the characteristics of PFOR and the dictionary index runs I was > > hopeful > > that I'll get good results with PFOR on dictionary indices. So I did an > > evaluation of PFOR for dictionary > > index runs and added the results on the same to the document. I do see > > value in using PFOR for > > dictionary index encoding. > > > > Looking for feedback from the team. > > > > Best > > Prateek > > > > [1] > > > > > https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/ > > [2] https://github.com/apache/arrow/pull/50088 c++ (pfor+frame only) > > [3] https://github.com/apache/arrow/pull/51150 c++ (pfor + dynamic frame > > or > > delta) > > [4] https://github.com/apache/arrow-rs/pull/10977 rust > > > > On Mon, Jul 13, 2026 at 8:31 AM PRATEEK GAUR <[email protected]> wrote: > > > > > Hi team, > > > > > > Just wanted to resurface the doc in your email folders, looking forward > > to > > > some suggestions. > > > I'll work towards adding a few more datasets to the comparison in the > > > coming days. > > > > > > Best > > > Prateek > > > > > > On Wed, Jul 1, 2026 at 9:01 AM PRATEEK GAUR <[email protected]> > wrote: > > > > > >> Hi Antoine, > > >> > > >> Apologies for the delay. Got some time to work on it and updated the > doc > > >> with the requested numbers. > > >> > > >> - Obtained the numbers for ZSTD again. I agree they were incorrect. > > >> - Fixed the algorithm to use decompressed size/data > > >> - Added BSS + ZSTD to the evaluation metrics > > >> - Added BSS + Lz4 to the evaluation metrics > > >> > > >> > > >> Updated doc : > > >> > > > https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/edit?tab=t.0#heading=h.uzgoevv9ajp > > >> Will be pushing the branch/code for the same today. > > >> > > >> Best > > >> Prateek > > >> > > >> On Wed, Dec 10, 2025 at 11:30 AM Antoine Pitrou <[email protected]> > > >> wrote: > > >> > > >>> > > >>> Hi, > > >>> > > >>> I looked at the doc and the stated decompression speeds for ZSTD look > > >>> highly irrealistic. > > >>> > > >>> I think what happens is that you are computing decompression speed > as: > > >>> > > >>> size of compressed data / time to decompress > > >>> > > >>> while you should really compute it as: > > >>> > > >>> size of uncompressed data / time to decompress > > >>> > > >>> Otherwise you're simply making ZSTD look miserable because it > > compresses > > >>> so well. > > >>> > > >>> Also, I think you should also add BYTE_STREAM_SPLIT + ZSTD into the > mix > > >>> (and possible BYTE_STREAM_SPLIT + LZ4 if you're going to evaluate > LZ4). > > >>> > > >>> Regards > > >>> > > >>> Antoine. > > >>> > > >>> > > >>> Le 06/12/2025 à 23:24, PRATEEK GAUR a écrit : > > >>> > Hi team, > > >>> > > > >>> > We wanted to share performance numbers for one of the candidate > > >>> encodings > > >>> > we have been discussing for parquet, for numeric compression. > > >>> > > > >>> > Following doc, PFOR : Encoding > > >>> > < > > >>> > > > https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/edit?tab=t.0 > > >>> >, > > >>> > talks through numbers on compression speed, compression ratio and > > >>> > decompression speed for columns values from the clickbench data. > > Based > > >>> on > > >>> > the numbers PFOR gives superior decompression speed compared to > > >>> > DELTABITPACK and RLEBITPACKHYBRID and fares better on average in > > >>> > compression ratio. Comparison has also been made with ZSTD > > compression > > >>> in > > >>> > the above doc. > > >>> > > > >>> > We plan to expand to a few more datasets and work on a prototype in > > >>> arrow > > >>> > cpp code in early Jan. > > >>> > Looking forward to the feedback from the group. > > >>> > > > >>> > Best > > >>> > Prateek > > >>> > > > >>> > > >>> > > >>> > > >
