Hi team, Thanks Micah for multiple rounds of review and thanks Gang for taking a look at the PR and giving feedback. I've addressed all comments in the PR. Looking forward to the next round of feedback (or approval). Would like to get it in soon.
https://github.com/apache/arrow/pull/48345/changes With this done and after talking to Arnav, I plan to spend some time on the ALP implementation in the arrow-go repository helping spread the support to more languages. Best Prateek On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <[email protected]> wrote: > Thanks Divjot for bringing it up. > > I was under the same impression as Russell and Andrew that we'd have it > flag gated. > With the new versioning proposal 'preview' might be a good tag for it. > > Best > Prateek > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <[email protected]> > wrote: > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment in >> the >> spec section). >> >> Russel: That is my understanding as well. >> >> Hopefully we are able to formalize the versioning process shortly so we >> can >> formalize the notion of "preview" >> >> Andrew >> >> >> [1]: https://github.com/apache/parquet-format/pull/604 >> >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer < >> [email protected]> >> wrote: >> >> > I thought we had agreed it would be merged, and features like this would >> > be feature flag gated. So a V-Old writer could always opt in to using >> this >> > encoding but that must be a conscious effort if you are not using VNext >> of >> > Parquet. If you are on V-Next the writer can choose to use ALP whenever >> it >> > likes without user intervention. >> > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev < >> > [email protected]> wrote: >> > >> >> Yes, that’s my understanding as well. However, as you noted, the >> >> versioning >> >> discussion is still in early stages and hasn’t prescribed concrete >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no >> >> indication that it’s in preview or that writers shouldn’t use it by >> >> default >> >> yet. Given that we’ve held off on merging other incompatible changes >> (e.g >> >> removal of path_in_schema) until the versioning discussion closes, my >> >> preference would be to leave ALP un-merged for the time being as well. >> >> >> >> — Div >> >> >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <[email protected]> >> wrote: >> >> >> >> > After re-reading Julien's Versioning Proposal, my understanding is >> that >> >> if >> >> > we adopt that system, we will somehow mark ALP in "Preview" in >> >> > parquet-format for release Version 3. >> >> > >> >> > The versioning document doesn't seem to proscribe the particular >> method >> >> > for annotating "preview" features -- I think clearly marking them as >> >> such >> >> > in parquet.thrift as well as reflecting that on the >> >> > https://parquet.apache.org/docs/file-format/versions/ page somehow >> >> would >> >> > be adequate. >> >> > >> >> > Andrew >> >> > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <[email protected] >> > >> >> > wrote: >> >> > >> >> >> In my mind ALP is in the same position as other encodings that were >> >> added >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types) >> >> >> >> >> >> That is to say it will be available in the latest open source >> writers >> >> and >> >> >> readers, but realistically not widely used in the ecosystem where >> >> >> compatibility is a concern (a large part) >> >> >> >> >> >> I view Julian's work[2][3] to gather consensus on how to name >> >> >> combinations of features (e.g. "versions") as the mechanism by >> which we >> >> >> will accelerate the adoption of this change (along with other >> forward >> >> >> incompatible changes) across the ecosystem. >> >> >> >> >> >> > Will we exclude this change when cutting parquet-format v2.14.0? >> >> >> >> >> >> No I don't think so -- I would expect it to go out in the next >> >> >> parquet-format version as do other changes in the spec. >> >> >> >> >> >> Andrew >> >> >> >> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/ >> >> >> [2]: >> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss >> >> >> [3]: >> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj >> >> >> >> >> >> >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev < >> >> >> [email protected]> wrote: >> >> >> >> >> >>> Hi Andrew, >> >> >>> >> >> >>> How does this interact with the ongoing versioning discussion? >> Given >> >> that >> >> >>> new encodings are not forward compatible, I would assume ALP is >> >> included >> >> >>> in >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that works if >> it's >> >> >>> merged to the parquet-format master branch now. Will we exclude >> this >> >> >>> change >> >> >>> when cutting parquet-format v2.14.0? >> >> >>> >> >> >>> -- Div >> >> >>> >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb < >> [email protected]> >> >> >>> wrote: >> >> >>> >> >> >>> > Hello, >> >> >>> > >> >> >>> > As an update, thanks to the epic work from many contributors, >> >> >>> especially >> >> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We >> >> plan to >> >> >>> > make additional editorial The remaining items I know of are >> merging >> >> the >> >> >>> > examples from parquet-testing[2], and then of course the various >> >> >>> > implementations >> >> >>> > >> >> >>> > Thanks again everyone. This is a great step forward >> >> >>> > Andrew >> >> >>> > >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557 >> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100 >> >> >>> > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <[email protected] >> > >> >> >>> wrote: >> >> >>> > >> >> >>> > > It was also in my spam folder... :( >> >> >>> > > I've seen a bunch of apache list emails go to my spam recently. >> >> >>> > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer < >> >> >>> > > [email protected]> >> >> >>> > > wrote: >> >> >>> > > >> >> >>> > > > Yes, my hope was that my vote would kick it out of spam but >> we >> >> >>> > definitely >> >> >>> > > > had two other votes that also went to spam for me. >> >> >>> > > > >> >> >>> > > > For anyone looking the thread is here >> >> >>> > > > >> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt >> >> >>> > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield < >> >> >>> > [email protected]> >> >> >>> > > > wrote: >> >> >>> > > > >> >> >>> > > > > I've seen other's votes not go to spam, so hopefully people >> >> will >> >> >>> see >> >> >>> > it >> >> >>> > > > in >> >> >>> > > > > there inboxes (or see this thread and look in there spam). >> >> >>> > > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR < >> >> >>> [email protected]> >> >> >>> > > > wrote: >> >> >>> > > > > >> >> >>> > > > > > Okay, >> >> >>> > > > > > >> >> >>> > > > > > I sent out the voting email yesterday and so far I've >> heard >> >> >>> from 3 >> >> >>> > > > people >> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be >> >> blamed >> >> >>> for >> >> >>> > > it?. >> >> >>> > > > > > >> >> >>> > > > > > What's the protocol here? >> >> >>> > > > > > I let people hunt for the email in their spam folder or >> send >> >> >>> out >> >> >>> > > > another >> >> >>> > > > > > one? >> >> >>> > > > > > >> >> >>> > > > > > Best >> >> >>> > > > > > Prateek >> >> >>> > > > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR < >> >> >>> [email protected]> >> >> >>> > > > > wrote: >> >> >>> > > > > > >> >> >>> > > > > > > Hi team, >> >> >>> > > > > > > >> >> >>> > > > > > > Vinoo has added the tests which were requested in the >> last >> >> >>> round >> >> >>> > of >> >> >>> > > > > > review. >> >> >>> > > > > > > In his own words. >> >> >>> > > > > > > >> >> >>> > > > > > > " >> >> >>> > > > > > > The extreme value tests Micah wanted are done and >> pushed. >> >> I >> >> >>> added >> >> >>> > > > > > coverage >> >> >>> > > > > > > for values that need the full FOR bit width after the >> >> frame >> >> >>> of >> >> >>> > > > > reference >> >> >>> > > > > > is >> >> >>> > > > > > > applied: 64-bit (the case where the signed max minus >> min >> >> >>> > > subtraction >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit >> for >> >> >>> > floats. >> >> >>> > > > All >> >> >>> > > > > > > lossless with zero exceptions. I also verified them >> >> through >> >> >>> Arrow >> >> >>> > > C++ >> >> >>> > > > > ALP >> >> >>> > > > > > > reader (the local build), so the Java-written extreme >> >> columns >> >> >>> > > decode >> >> >>> > > > > > > correctly in C++, bit-exact against the expected >> values." >> >> >>> > > > > > > >> >> >>> > > > > > > State of the PR (#3397): >> >> >>> > > > > > > >> >> >>> > > > > > > - Merges cleanly with master (I pulled master in and >> >> >>> resolved >> >> >>> > > the >> >> >>> > > > > > > conflicts). >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining >> are >> >> >>> > > > intentional: >> >> >>> > > > > > > one deferred perf nit, one that stays open until >> >> >>> > parquet-format >> >> >>> > > > > ships >> >> >>> > > > > > ALP, >> >> >>> > > > > > > >> >> >>> > > > > > > " >> >> >>> > > > > > > >> >> >>> > > > > > > With this both the >> >> >>> > > > > > > 1) C++ PR : >> >> >>> https://github.com/apache/arrow/pull/48345/changes >> >> >>> > > > > > > 2) Java PR : >> >> >>> https://github.com/apache/parquet-java/pull/3397 >> >> >>> > > > > > > >> >> >>> > > > > > > Are cross language tested and range of test scenarios. >> >> >>> > > > > > > >> >> >>> > > > > > > Best >> >> >>> > > > > > > Prateek >> >> >>> > > > > > > >> >> >>> > > > > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR < >> >> >>> [email protected] >> >> >>> > > >> >> >>> > > > > wrote: >> >> >>> > > > > > > >> >> >>> > > > > > >> Thanks Matt, >> >> >>> > > > > > >> >> >> >>> > > > > > >> I'll be on it this week. >> >> >>> > > > > > >> >> >> >>> > > > > > >> Best >> >> >>> > > > > > >> Prateek >> >> >>> > > > > > >> >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol < >> >> >>> > > [email protected]> >> >> >>> > > > > > >> wrote: >> >> >>> > > > > > >> >> >> >>> > > > > > >>> There is also the Go implementation submitted by >> Arnav ( >> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which >> is >> >> >>> waiting >> >> >>> > > for >> >> >>> > > > > > >>> updates. >> >> >>> > > > > > >>> >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost here. >> >> >>> > > > > > >>> >> >> >>> > > > > > >>> --Matt >> >> >>> > > > > > >>> >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR < >> >> >>> > > [email protected]> >> >> >>> > > > > > >>> wrote: >> >> >>> > > > > > >>> >> >> >>> > > > > > >>> > Hi all, >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > I'd like to share a status update on the ALP >> encoding >> >> >>> effort >> >> >>> > > and >> >> >>> > > > > get >> >> >>> > > > > > a >> >> >>> > > > > > >>> feel >> >> >>> > > > > > >>> > from the community before starting a formal vote. >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > We now have the specification plus implementations >> in >> >> >>> both >> >> >>> > > > > languages: >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > - Spec: apache/parquet-format#557 — >> GH-533 >> >> >>> Add >> >> >>> > ALP >> >> >>> > > > > > encoding >> >> >>> > > > > > >>> > specification >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 >> >> >>> > > [C++][Parquet] >> >> >>> > > > > Add >> >> >>> > > > > > >>> ALPpd >> >> >>> > > > > > >>> > encoding >> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397 — >> Parquet >> >> >>> Java >> >> >>> > ALP >> >> >>> > > > > > >>> > Implementation (by Vinoo) >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > Current state: >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > - The C++ implementation has been through 3-4 >> >> rounds >> >> >>> of >> >> >>> > > > review. >> >> >>> > > > > > >>> > - The Java implementation has been tracking >> well >> >> and >> >> >>> has >> >> >>> > > had >> >> >>> > > > > > >>> initial >> >> >>> > > > > > >>> > reviews from contributors, with no major >> >> remaining >> >> >>> > > issues. >> >> >>> > > > > > >>> > - Cross-language compatibility tests are >> passing: >> >> the >> >> >>> > Arrow >> >> >>> > > > C++ >> >> >>> > > > > > >>> decoder >> >> >>> > > > > > >>> > reads Java-written data bit-exactly across >> >> ~1.56M >> >> >>> > values >> >> >>> > > > and >> >> >>> > > > > 18 >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple >> >> vector >> >> >>> > > sizes, >> >> >>> > > > > and >> >> >>> > > > > > >>> > several >> >> >>> > > > > > >>> > real datasets — zero mismatches. >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > As raised in the java-pr we will be adding >> coverage >> >> for >> >> >>> > > extreme >> >> >>> > > > > > values >> >> >>> > > > > > >>> > (those requiring >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a >> >> vote. >> >> >>> > We'll >> >> >>> > > > aim >> >> >>> > > > > > to >> >> >>> > > > > > >>> get >> >> >>> > > > > > >>> > that >> >> >>> > > > > > >>> > done in parallel. >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > Plan: unless there are objections, I intend to >> start a >> >> >>> vote >> >> >>> > at >> >> >>> > > > the >> >> >>> > > > > > end >> >> >>> > > > > > >>> of >> >> >>> > > > > > >>> > this week/early >> >> >>> > > > > > >>> > next week. >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > Thanks >> >> >>> > > > > > >>> > Prateek >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR < >> >> >>> > > [email protected]> >> >> >>> > > > > > >>> wrote: >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > > Hi Team, >> >> >>> > > > > > >>> > > >> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP. >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for >> the >> >> c++ >> >> >>> PR : >> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345 >> >> >>> > > > > > >>> > > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been >> working >> >> on >> >> >>> > > following >> >> >>> > > > > PR >> >> >>> > > > > > : >> >> >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397 >> >> >>> > > > > > >>> > > >> >> >>> > > > > > >>> > > Best >> >> >>> > > > > > >>> > > Prateek >> >> >>> > > > > > >>> > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield < >> >> >>> > > > > > >>> [email protected]> >> >> >>> > > > > > >>> > > wrote: >> >> >>> > > > > > >>> > > >> >> >>> > > > > > >>> > >> Hi Antoine, >> >> >>> > > > > > >>> > >> >> >> >>> > > > > > >>> > >> > Apologies if the question was already asked, >> but >> >> >>> should >> >> >>> > we >> >> >>> > > > > care >> >> >>> > > > > > >>> about >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more >> >> efficient >> >> >>> than >> >> >>> > > > > FLOAT16 >> >> >>> > > > > > + >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example? >> >> >>> > > > > > >>> > >> >> >> >>> > > > > > >>> > >> It was. We thought we could defer it for the >> >> >>> following >> >> >>> > > > reasons: >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy >> >> reference >> >> >>> > > datasets >> >> >>> > > > to >> >> >>> > > > > > >>> test >> >> >>> > > > > > >>> > its >> >> >>> > > > > > >>> > >> effectiveness. It does look like there might be >> >> one >> >> >>> or >> >> >>> > two >> >> >>> > > on >> >> >>> > > > > > >>> > huggingface >> >> >>> > > > > > >>> > >> (e.g. >> >> https://huggingface.co/datasets/kikitora/curdie >> >> >>> ). >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more >> likely >> >> >>> used >> >> >>> > for >> >> >>> > > > > values >> >> >>> > > > > > >>> that >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal values. >> >> >>> > > > > > >>> > >> 3. It could be added as an extension later if >> >> needed. >> >> >>> > > > > > >>> > >> >> >> >>> > > > > > >>> > >> Cheers, >> >> >>> > > > > > >>> > >> Micah >> >> >>> > > > > > >>> > >> >> >> >>> > > > > > >>> > >> >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou < >> >> >>> > > > > [email protected] >> >> >>> > > > > > > >> >> >>> > > > > > >>> > wrote: >> >> >>> > > > > > >>> > >> >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > Hello, >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > Apologies if the question was already asked, >> but >> >> >>> should >> >> >>> > we >> >> >>> > > > > care >> >> >>> > > > > > >>> about >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more >> >> efficient >> >> >>> than >> >> >>> > > > > FLOAT16 >> >> >>> > > > > > + >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example? >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > Regards >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > Antoine. >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit : >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback >> on >> >> >>> the two >> >> >>> > > > PR's >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo) >> >> >>> > > > > > >>> > >> >> https://github.com/apache/arrow/pull/48345/changes >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo) >> >> >>> > > > > > >>> > >> > >> >> https://github.com/apache/parquet-format/pull/557 >> >> >>> > > > > > >>> > >> > > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed >> >> something) >> >> >>> > > comments >> >> >>> > > > on >> >> >>> > > > > > >>> the two >> >> >>> > > > > > >>> > >> > PR's. >> >> >>> > > > > > >>> > >> > > >> >> >>> > > > > > >>> > >> > > Best >> >> >>> > > > > > >>> > >> > > Prateek >> >> >>> > > > > > >>> > >> > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK >> GAUR < >> >> >>> > > > > > >>> [email protected]> >> >> >>> > > > > > >>> > >> wrote: >> >> >>> > > > > > >>> > >> > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah. >> >> >>> > > > > > >>> > >> > >> >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the >> >> >>> > > implementations` >> >> >>> > > > > > >>> > >> > >> For the c++ I have already started >> addressing >> >> the >> >> >>> > > > > feedback, I >> >> >>> > > > > > >>> > should >> >> >>> > > > > > >>> > >> be >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday. >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good >> >> progress >> >> >>> on >> >> >>> > the >> >> >>> > > > Java >> >> >>> > > > > > >>> > >> > implementation. >> >> >>> > > > > > >>> > >> > >> >> >> >>> > > > > > >>> > >> > >> Best >> >> >>> > > > > > >>> > >> > >> Prateek >> >> >>> > > > > > >>> > >> > >> >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew >> Lamb < >> >> >>> > > > > > >>> > [email protected] >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > >> wrote: >> >> >>> > > > > > >>> > >> > >> >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification >> -- I >> >> >>> will >> >> >>> > try >> >> >>> > > > and >> >> >>> > > > > > look >> >> >>> > > > > > >>> > into >> >> >>> > > > > > >>> > >> the >> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in >> this >> >> next >> >> >>> > week >> >> >>> > > > > > >>> > >> > >>> >> >> >>> > > > > > >>> > >> > >>> [1]: >> >> >>> https://github.com/apache/arrow-rs/pull/9372 >> >> >>> > > > > > >>> > >> > >>> >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah >> >> >>> Kornfield < >> >> >>> > > > > > >>> > >> > [email protected]> >> >> >>> > > > > > >>> > >> > >>> wrote: >> >> >>> > > > > > >>> > >> > >>> >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew, >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of >> feedback >> >> on >> >> >>> at >> >> >>> > > least >> >> >>> > > > > the >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think we've >> >> waited >> >> >>> > till >> >> >>> > > > they >> >> >>> > > > > > are >> >> >>> > > > > > >>> > >> close to >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote. >> Otherwise I >> >> >>> agree >> >> >>> > we >> >> >>> > > > are >> >> >>> > > > > > very >> >> >>> > > > > > >>> > >> close. >> >> >>> > > > > > >>> > >> > >>>> >> >> >>> > > > > > >>> > >> > >>>> -Micah >> >> >>> > > > > > >>> > >> > >>>> >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb >> < >> >> >>> > > > > > >>> [email protected] >> >> >>> > > > > > >>> > > >> >> >>> > > > > > >>> > >> > wrote: >> >> >>> > > > > > >>> > >> > >>>> >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek, >> >> >>> > > > > > >>> > >> > >>>>> >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me >> >> like >> >> >>> we >> >> >>> > are >> >> >>> > > > > ready >> >> >>> > > > > > >>> to >> >> >>> > > > > > >>> > >> start a >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into >> Parquet >> >> >>> > > > > > >>> > >> > >>>>> >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should >> >> >>> postpone it >> >> >>> > > for >> >> >>> > > > > > >>> longer? >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to >> >> >>> review? >> >> >>> > > > > > >>> > >> > >>>>> >> >> >>> > > > > > >>> > >> > >>>>> Andrew >> >> >>> > > > > > >>> > >> > >>>>> >> >> >>> > > > > > >>> > >> > >>>>> >> >> >>> > > > > > >>> > >> > >>>>> >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK >> >> GAUR < >> >> >>> > > > > > >>> > [email protected]> >> >> >>> > > > > > >>> > >> > >>>>> wrote: >> >> >>> > > > > > >>> > >> > >>>>> >> >> >>> > > > > > >>> > >> > >>>>>> Hi team, >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a >> >> chance >> >> >>> to >> >> >>> > work >> >> >>> > > > > > >>> through all >> >> >>> > > > > > >>> > >> the >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec >> >> doc. >> >> >>> Here >> >> >>> > > are >> >> >>> > > > > the >> >> >>> > > > > > >>> new >> >> >>> > > > > > >>> > >> > artifacts >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document : >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> >> >> >>> > > > > > >> >> >>> > > > > >> >> >>> > > > >> >> >>> > > >> >> >>> > >> >> >>> >> >> >> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format >> repo : >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > https://github.com/apache/parquet-format/pull/557 >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ >> repo : >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > https://github.com/apache/arrow/pull/48345/changes >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java >> >> repo : >> >> >>> Work >> >> >>> > > for >> >> >>> > > > > > >>> Vinoo and >> >> >>> > > > > > >>> > >> > Julien >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > https://github.com/apache/parquet-java/pull/3397 >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking >> >> artifacts in >> >> >>> > > > > > >>> parquet-testing >> >> >>> > > > > > >>> > >> repo : >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > https://github.com/apache/parquet-testing/pull/100 >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> And >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in >> >> >>> progress >> >> >>> > > > > > >>> implementation >> >> >>> > > > > > >>> > in >> >> >>> > > > > > >>> > >> Go. >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> https://github.com/apache/arrow-go/pull/704 >> >> >>> > (I >> >> >>> > > > > > haven't >> >> >>> > > > > > >>> > >> started >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet) >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew >> mentioned >> >> that >> >> >>> > this >> >> >>> > > > work >> >> >>> > > > > > is >> >> >>> > > > > > >>> also >> >> >>> > > > > > >>> > >> in >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!) >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation * >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by >> >> Antoine to >> >> >>> > > report >> >> >>> > > > > the >> >> >>> > > > > > >>> > numbers >> >> >>> > > > > > >>> > >> as >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin >> have >> >> >>> given 1 >> >> >>> > > > round >> >> >>> > > > > of >> >> >>> > > > > > >>> > >> feedback >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please >> note >> >> >>> that >> >> >>> > the >> >> >>> > > > > > default >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 >> and >> >> not >> >> >>> > Q3. I >> >> >>> > > > got >> >> >>> > > > > > >>> around >> >> >>> > > > > > >>> > >> 70% >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding >> >> speed >> >> >>> > when >> >> >>> > > > > using >> >> >>> > > > > > >>> the O3 >> >> >>> > > > > > >>> > >> > flag. >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working >> >> with >> >> >>> > Vinoo >> >> >>> > > > and >> >> >>> > > > > > >>> Julien) >> >> >>> > > > > > >>> > >> and >> >> >>> > > > > > >>> > >> > **Cross >> >> >>> > > > > > >>> > >> > >>>>>> Language testing* >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any >> questions >> >> or >> >> >>> > > > feedback. >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — >> >> Spotify >> >> >>> > Columns >> >> >>> > > > > > >>> (Graviton >> >> >>> > > > > > >>> > 3, >> >> >>> > > > > > >>> > >> ARM >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1) >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> >> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐ >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ >> >> -O3 >> >> >>> > > (MB/s) │ >> >> >>> > > > > > >>> Speedup │ >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> >> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤ >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │ >> >> >>> 5,523 >> >> >>> > > │ >> >> >>> > > > > > >>> 1.75x │ >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │ >> >> >>> 5,685 >> >> >>> > > │ >> >> >>> > > > > > >>> 1.76x │ >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │ >> >> >>> 5,652 >> >> >>> > > │ >> >> >>> > > > > > >>> 1.77x │ >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │ >> >> >>> 5,473 >> >> >>> > > │ >> >> >>> > > > > > >>> 1.72x │ >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> >> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘ >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK >> >> GAUR >> >> >>> < >> >> >>> > > > > > >>> > [email protected] >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > >>>>>> wrote: >> >> >>> > > > > > >>> > >> > >>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield < >> [email protected]> >> >> : >> >> >>> Got >> >> >>> > > it. >> >> >>> > > > > > >>> > >> > >>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <[email protected]> >> >> >>> > > > > > >>> > >> > >>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to >> start >> >> >>> moving >> >> >>> > the >> >> >>> > > > > spec >> >> >>> > > > > > >>> > >> development >> >> >>> > > > > > >>> > >> > >>>>>>>> into >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for >> >> >>> finalizing >> >> >>> > > it? >> >> >>> > > > > > >>> > >> > >>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some >> of >> >> the >> >> >>> > > > examples I >> >> >>> > > > > > >>> have in >> >> >>> > > > > > >>> > >> the >> >> >>> > > > > > >>> > >> > >>>>>>> spec based >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we >> >> should >> >> >>> be >> >> >>> > good >> >> >>> > > > to >> >> >>> > > > > go >> >> >>> > > > > > >>> for >> >> >>> > > > > > >>> > the >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format. >> >> >>> > > > > > >>> > >> > >>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone! >> >> >>> > > > > > >>> > >> > >>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew >> >> >>> > > > > > >>> > >> > >>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM >> PRATEEK >> >> >>> GAUR < >> >> >>> > > > > > >>> > >> [email protected]> >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote: >> >> >>> > > > > > >>> > >> > >>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team, >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test >> files. >> >> >>> My PR >> >> >>> > > did >> >> >>> > > > > add >> >> >>> > > > > > >>> all >> >> >>> > > > > > >>> > the >> >> >>> > > > > > >>> > >> > test >> >> >>> > > > > > >>> > >> > >>>>>>>> files I >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. >> >> Maybe >> >> >>> we >> >> >>> > can >> >> >>> > > > club >> >> >>> > > > > > it >> >> >>> > > > > > >>> > >> together. >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust >> >> >>> > > > implementation. >> >> >>> > > > > > >>> This is >> >> >>> > > > > > >>> > >> > great. >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the >> >> >>> numbers >> >> >>> > on >> >> >>> > > > AMD. >> >> >>> > > > > > >>> Looks >> >> >>> > > > > > >>> > >> like >> >> >>> > > > > > >>> > >> > you >> >> >>> > > > > > >>> > >> > >>>>>>>> are >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding >> >> performance of >> >> >>> > BSS. >> >> >>> > > > This >> >> >>> > > > > > is >> >> >>> > > > > > >>> > >> > amazing!!. >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the >> >> >>> sampling >> >> >>> > > > design. >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on >> Fastlanes. In >> >> >>> some >> >> >>> > > crude >> >> >>> > > > > > >>> > >> experiments I >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on >> >> >>> Graviton3 >> >> >>> > > (but >> >> >>> > > > > > maybe >> >> >>> > > > > > >>> > there >> >> >>> > > > > > >>> > >> was >> >> >>> > > > > > >>> > >> > >>>>>>>> something >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation). >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception >> >> >>> encoding for >> >> >>> > > the >> >> >>> > > > > > spec >> >> >>> > > > > > >>> in >> >> >>> > > > > > >>> > >> this >> >> >>> > > > > > >>> > >> > >>>>>>>> case. >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved >> >> for >> >> >>> all >> >> >>> > > open >> >> >>> > > > > > >>> questions >> >> >>> > > > > > >>> > >> minus >> >> >>> > > > > > >>> > >> > >>>>>>>> the >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back >> on >> >> >>> this >> >> >>> > > soon) >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec >> does >> >> >>> allow >> >> >>> > > for >> >> >>> > > > > > using >> >> >>> > > > > > >>> > >> FastLane >> >> >>> > > > > > >>> > >> > >>>>>>>> with the >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for >> >> layout. We >> >> >>> > > should >> >> >>> > > > be >> >> >>> > > > > > >>> able to >> >> >>> > > > > > >>> > >> > inject >> >> >>> > > > > > >>> > >> > >>>>>>>> any >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design. >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining >> open >> >> >>> > comments >> >> >>> > > > on >> >> >>> > > > > > the >> >> >>> > > > > > >>> spec >> >> >>> > > > > > >>> > >> this >> >> >>> > > > > > >>> > >> > >>>>>>>> week. >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM >> Steve >> >> >>> > Loughran < >> >> >>> > > > > > >>> > >> > >>>>>>>> [email protected]> >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote: >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah >> >> >>> Kornfield >> >> >>> > < >> >> >>> > > > > > >>> > >> > >>>>>>>> [email protected]> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote: >> >> >>> > > > > > >>> > >> > >>>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue >> >> >>> described for >> >> >>> > > ORC >> >> >>> > > > > in >> >> >>> > > > > > >>> the >> >> >>> > > > > > >>> > >> paper >> >> >>> > > > > > >>> > >> > >>>>>>>> is that >> >> >>> > > > > > >>> > >> > >>>>>>>>> it >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a >> >> batch. >> >> >>> This >> >> >>> > > is >> >> >>> > > > > > >>> different >> >> >>> > > > > > >>> > >> then >> >> >>> > > > > > >>> > >> > >>>>>>>> the >> >> >>> > > > > > >>> > >> > >>>>>>>>> design >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still >> >> fixed >> >> >>> > > > encoding >> >> >>> > > > > > per >> >> >>> > > > > > >>> page >> >> >>> > > > > > >>> > >> in >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet. >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I >> don't >> >> >>> think >> >> >>> > > > branch >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I >> agree >> >> >>> that >> >> >>> > we >> >> >>> > > > > should >> >> >>> > > > > > >>> be >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings. >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1 >> >> >>> > > > > > >>> > >> > >>>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>>> >> >> >>> > > > > > >>> > >> > >>>>>>> >> >> >>> > > > > > >>> > >> > > >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> >> >> >>> > > > > > >>> > > >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> >> >> >>> > > > > > >> >> >> >>> > > > > > >> >> >>> > > > > >> >> >>> > > > >> >> >>> > > >> >> >>> > >> >> >>> >> >> >> >> >> >> > >> >
