After re-reading Julien's Versioning Proposal, my understanding is that if we adopt that system, we will somehow mark ALP in "Preview" in parquet-format for release Version 3.
The versioning document doesn't seem to proscribe the particular method for annotating "preview" features -- I think clearly marking them as such in parquet.thrift as well as reflecting that on the https://parquet.apache.org/docs/file-format/versions/ page somehow would be adequate. Andrew On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <[email protected]> wrote: > In my mind ALP is in the same position as other encodings that were added > recently[1] (such as BYTE_STREAM_SPLIT for non floating point types) > > That is to say it will be available in the latest open source writers and > readers, but realistically not widely used in the ecosystem where > compatibility is a concern (a large part) > > I view Julian's work[2][3] to gather consensus on how to name combinations > of features (e.g. "versions") as the mechanism by which we will accelerate > the adoption of this change (along with other forward incompatible changes) > across the ecosystem. > > > Will we exclude this change when cutting parquet-format v2.14.0? > > No I don't think so -- I would expect it to go out in the next > parquet-format version as do other changes in the spec. > > Andrew > > [1]: https://parquet.apache.org/docs/file-format/versions/ > [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss > [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj > > > On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev < > [email protected]> wrote: > >> Hi Andrew, >> >> How does this interact with the ongoing versioning discussion? Given that >> new encodings are not forward compatible, I would assume ALP is included >> in >> Parquet V<next> (V3?), but it's not clear to me how that works if it's >> merged to the parquet-format master branch now. Will we exclude this >> change >> when cutting parquet-format v2.14.0? >> >> -- Div >> >> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <[email protected]> >> wrote: >> >> > Hello, >> > >> > As an update, thanks to the epic work from many contributors, especially >> > Prateek, I just merged the parquet-format change for ALP[1]. We plan to >> > make additional editorial The remaining items I know of are merging the >> > examples from parquet-testing[2], and then of course the various >> > implementations >> > >> > Thanks again everyone. This is a great step forward >> > Andrew >> > >> > [1]: https://github.com/apache/parquet-format/pull/557 >> > [2]: https://github.com/apache/parquet-testing/pull/100 >> > >> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <[email protected]> >> wrote: >> > >> > > It was also in my spam folder... :( >> > > I've seen a bunch of apache list emails go to my spam recently. >> > > >> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer < >> > > [email protected]> >> > > wrote: >> > > >> > > > Yes, my hope was that my vote would kick it out of spam but we >> > definitely >> > > > had two other votes that also went to spam for me. >> > > > >> > > > For anyone looking the thread is here >> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt >> > > > >> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield < >> > [email protected]> >> > > > wrote: >> > > > >> > > > > I've seen other's votes not go to spam, so hopefully people will >> see >> > it >> > > > in >> > > > > there inboxes (or see this thread and look in there spam). >> > > > > >> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <[email protected] >> > >> > > > wrote: >> > > > > >> > > > > > Okay, >> > > > > > >> > > > > > I sent out the voting email yesterday and so far I've heard >> from 3 >> > > > people >> > > > > > that it ended up in their "spam folder". Dev VPN to be blamed >> for >> > > it?. >> > > > > > >> > > > > > What's the protocol here? >> > > > > > I let people hunt for the email in their spam folder or send out >> > > > another >> > > > > > one? >> > > > > > >> > > > > > Best >> > > > > > Prateek >> > > > > > >> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR < >> [email protected]> >> > > > > wrote: >> > > > > > >> > > > > > > Hi team, >> > > > > > > >> > > > > > > Vinoo has added the tests which were requested in the last >> round >> > of >> > > > > > review. >> > > > > > > In his own words. >> > > > > > > >> > > > > > > " >> > > > > > > The extreme value tests Micah wanted are done and pushed. I >> added >> > > > > > coverage >> > > > > > > for values that need the full FOR bit width after the frame of >> > > > > reference >> > > > > > is >> > > > > > > applied: 64-bit (the case where the signed max minus min >> > > subtraction >> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for >> > floats. >> > > > All >> > > > > > > lossless with zero exceptions. I also verified them through >> Arrow >> > > C++ >> > > > > ALP >> > > > > > > reader (the local build), so the Java-written extreme columns >> > > decode >> > > > > > > correctly in C++, bit-exact against the expected values." >> > > > > > > >> > > > > > > State of the PR (#3397): >> > > > > > > >> > > > > > > - Merges cleanly with master (I pulled master in and >> resolved >> > > the >> > > > > > > conflicts). >> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are >> > > > intentional: >> > > > > > > one deferred perf nit, one that stays open until >> > parquet-format >> > > > > ships >> > > > > > ALP, >> > > > > > > >> > > > > > > " >> > > > > > > >> > > > > > > With this both the >> > > > > > > 1) C++ PR : >> https://github.com/apache/arrow/pull/48345/changes >> > > > > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397 >> > > > > > > >> > > > > > > Are cross language tested and range of test scenarios. >> > > > > > > >> > > > > > > Best >> > > > > > > Prateek >> > > > > > > >> > > > > > > >> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR < >> [email protected] >> > > >> > > > > wrote: >> > > > > > > >> > > > > > >> Thanks Matt, >> > > > > > >> >> > > > > > >> I'll be on it this week. >> > > > > > >> >> > > > > > >> Best >> > > > > > >> Prateek >> > > > > > >> >> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol < >> > > [email protected]> >> > > > > > >> wrote: >> > > > > > >> >> > > > > > >>> There is also the Go implementation submitted by Arnav ( >> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is >> waiting >> > > for >> > > > > > >>> updates. >> > > > > > >>> >> > > > > > >>> Just wanted to make sure it didn't get lost here. >> > > > > > >>> >> > > > > > >>> --Matt >> > > > > > >>> >> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR < >> > > [email protected]> >> > > > > > >>> wrote: >> > > > > > >>> >> > > > > > >>> > Hi all, >> > > > > > >>> > >> > > > > > >>> > I'd like to share a status update on the ALP encoding >> effort >> > > and >> > > > > get >> > > > > > a >> > > > > > >>> feel >> > > > > > >>> > from the community before starting a formal vote. >> > > > > > >>> > >> > > > > > >>> > We now have the specification plus implementations in both >> > > > > languages: >> > > > > > >>> > >> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add >> > ALP >> > > > > > encoding >> > > > > > >>> > specification >> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 >> > > [C++][Parquet] >> > > > > Add >> > > > > > >>> ALPpd >> > > > > > >>> > encoding >> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet >> Java >> > ALP >> > > > > > >>> > Implementation (by Vinoo) >> > > > > > >>> > >> > > > > > >>> > Current state: >> > > > > > >>> > >> > > > > > >>> > - The C++ implementation has been through 3-4 rounds >> of >> > > > review. >> > > > > > >>> > - The Java implementation has been tracking well and >> has >> > > had >> > > > > > >>> initial >> > > > > > >>> > reviews from contributors, with no major remaining >> > > issues. >> > > > > > >>> > - Cross-language compatibility tests are passing: the >> > Arrow >> > > > C++ >> > > > > > >>> decoder >> > > > > > >>> > reads Java-written data bit-exactly across ~1.56M >> > values >> > > > and >> > > > > 18 >> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple vector >> > > sizes, >> > > > > and >> > > > > > >>> > several >> > > > > > >>> > real datasets — zero mismatches. >> > > > > > >>> > >> > > > > > >>> > As raised in the java-pr we will be adding coverage for >> > > extreme >> > > > > > values >> > > > > > >>> > (those requiring >> > > > > > >>> > 63-64 bits after FOR is applied) before we close a vote. >> > We'll >> > > > aim >> > > > > > to >> > > > > > >>> get >> > > > > > >>> > that >> > > > > > >>> > done in parallel. >> > > > > > >>> > >> > > > > > >>> > Plan: unless there are objections, I intend to start a >> vote >> > at >> > > > the >> > > > > > end >> > > > > > >>> of >> > > > > > >>> > this week/early >> > > > > > >>> > next week. >> > > > > > >>> > >> > > > > > >>> > Thanks >> > > > > > >>> > Prateek >> > > > > > >>> > >> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR < >> > > [email protected]> >> > > > > > >>> wrote: >> > > > > > >>> > >> > > > > > >>> > > Hi Team, >> > > > > > >>> > > >> > > > > > >>> > > Just wanted to provide some updates on ALP. >> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the c++ >> PR : >> > > > > > >>> > > https://github.com/apache/arrow/pull/48345 >> > > > > > >>> > > >> > > > > > >>> > > For the Java implementation Vinoo has been working on >> > > following >> > > > > PR >> > > > > > : >> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397 >> > > > > > >>> > > >> > > > > > >>> > > Best >> > > > > > >>> > > Prateek >> > > > > > >>> > > >> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield < >> > > > > > >>> [email protected]> >> > > > > > >>> > > wrote: >> > > > > > >>> > > >> > > > > > >>> > >> Hi Antoine, >> > > > > > >>> > >> >> > > > > > >>> > >> > Apologies if the question was already asked, but >> should >> > we >> > > > > care >> > > > > > >>> about >> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient >> than >> > > > > FLOAT16 >> > > > > > + >> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example? >> > > > > > >>> > >> >> > > > > > >>> > >> It was. We thought we could defer it for the following >> > > > reasons: >> > > > > > >>> > >> 1. It's not clear there are a lot of easy reference >> > > datasets >> > > > to >> > > > > > >>> test >> > > > > > >>> > its >> > > > > > >>> > >> effectiveness. It does look like there might be one or >> > two >> > > on >> > > > > > >>> > huggingface >> > > > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie >> ). >> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely used >> > for >> > > > > values >> > > > > > >>> that >> > > > > > >>> > >> were less likely to reduce to decimal values. >> > > > > > >>> > >> 3. It could be added as an extension later if needed. >> > > > > > >>> > >> >> > > > > > >>> > >> Cheers, >> > > > > > >>> > >> Micah >> > > > > > >>> > >> >> > > > > > >>> > >> >> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou < >> > > > > [email protected] >> > > > > > > >> > > > > > >>> > wrote: >> > > > > > >>> > >> >> > > > > > >>> > >> > >> > > > > > >>> > >> > Hello, >> > > > > > >>> > >> > >> > > > > > >>> > >> > Apologies if the question was already asked, but >> should >> > we >> > > > > care >> > > > > > >>> about >> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient >> than >> > > > > FLOAT16 >> > > > > > + >> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example? >> > > > > > >>> > >> > >> > > > > > >>> > >> > Regards >> > > > > > >>> > >> > >> > > > > > >>> > >> > Antoine. >> > > > > > >>> > >> > >> > > > > > >>> > >> > >> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit : >> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on the >> two >> > > > PR's >> > > > > > >>> > >> > > 1) (c++ arrow repo) >> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes >> > > > > > >>> > >> > > 2) (parquet-format repo) >> > > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557 >> > > > > > >>> > >> > > >> > > > > > >>> > >> > > I have addressed all (unless I missed something) >> > > comments >> > > > on >> > > > > > >>> the two >> > > > > > >>> > >> > PR's. >> > > > > > >>> > >> > > >> > > > > > >>> > >> > > Best >> > > > > > >>> > >> > > Prateek >> > > > > > >>> > >> > > >> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR < >> > > > > > >>> [email protected]> >> > > > > > >>> > >> wrote: >> > > > > > >>> > >> > > >> > > > > > >>> > >> > >> Thanks Andrew and Micah. >> > > > > > >>> > >> > >> >> > > > > > >>> > >> > >> `fair amount of feedback on at least the >> > > implementations` >> > > > > > >>> > >> > >> For the c++ I have already started addressing the >> > > > > feedback, I >> > > > > > >>> > should >> > > > > > >>> > >> be >> > > > > > >>> > >> > >> done with that Monday/Tuesday. >> > > > > > >>> > >> > >> I think Vinoo too has been making good progress on >> > the >> > > > Java >> > > > > > >>> > >> > implementation. >> > > > > > >>> > >> > >> >> > > > > > >>> > >> > >> Best >> > > > > > >>> > >> > >> Prateek >> > > > > > >>> > >> > >> >> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb < >> > > > > > >>> > [email protected] >> > > > > > >>> > >> > >> > > > > > >>> > >> > >> wrote: >> > > > > > >>> > >> > >> >> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I will >> > try >> > > > and >> > > > > > look >> > > > > > >>> > into >> > > > > > >>> > >> the >> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this next >> > week >> > > > > > >>> > >> > >>> >> > > > > > >>> > >> > >>> [1]: >> https://github.com/apache/arrow-rs/pull/9372 >> > > > > > >>> > >> > >>> >> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield >> < >> > > > > > >>> > >> > [email protected]> >> > > > > > >>> > >> > >>> wrote: >> > > > > > >>> > >> > >>> >> > > > > > >>> > >> > >>>> Hi Andrew, >> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback on at >> > > least >> > > > > the >> > > > > > >>> > >> > >>>> implementations, typically I think we've waited >> > till >> > > > they >> > > > > > are >> > > > > > >>> > >> close to >> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I >> agree >> > we >> > > > are >> > > > > > very >> > > > > > >>> > >> close. >> > > > > > >>> > >> > >>>> >> > > > > > >>> > >> > >>>> -Micah >> > > > > > >>> > >> > >>>> >> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb < >> > > > > > >>> [email protected] >> > > > > > >>> > > >> > > > > > >>> > >> > wrote: >> > > > > > >>> > >> > >>>> >> > > > > > >>> > >> > >>>>> Thanks Prateek, >> > > > > > >>> > >> > >>>>> >> > > > > > >>> > >> > >>>>> I think from this content it looks to me like >> we >> > are >> > > > > ready >> > > > > > >>> to >> > > > > > >>> > >> start a >> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet >> > > > > > >>> > >> > >>>>> >> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should >> postpone it >> > > for >> > > > > > >>> longer? >> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to review? >> > > > > > >>> > >> > >>>>> >> > > > > > >>> > >> > >>>>> Andrew >> > > > > > >>> > >> > >>>>> >> > > > > > >>> > >> > >>>>> >> > > > > > >>> > >> > >>>>> >> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR < >> > > > > > >>> > [email protected]> >> > > > > > >>> > >> > >>>>> wrote: >> > > > > > >>> > >> > >>>>> >> > > > > > >>> > >> > >>>>>> Hi team, >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to >> > work >> > > > > > >>> through all >> > > > > > >>> > >> the >> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc. >> Here >> > > are >> > > > > the >> > > > > > >>> new >> > > > > > >>> > >> > artifacts >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> 1) Spec document : >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >> > > > > > >>> > >> >> > > > > > >>> > >> > > > > > >>> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo : >> > > > > > >>> > >> > >>>>>> >> > https://github.com/apache/parquet-format/pull/557 >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo : >> > > > > > >>> > >> > >>>>>> >> > https://github.com/apache/arrow/pull/48345/changes >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : >> Work >> > > for >> > > > > > >>> Vinoo and >> > > > > > >>> > >> > Julien >> > > > > > >>> > >> > >>>>>> >> > https://github.com/apache/parquet-java/pull/3397 >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in >> > > > > > >>> parquet-testing >> > > > > > >>> > >> repo : >> > > > > > >>> > >> > >>>>>> >> > https://github.com/apache/parquet-testing/pull/100 >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> And >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress >> > > > > > >>> implementation >> > > > > > >>> > in >> > > > > > >>> > >> Go. >> > > > > > >>> > >> > >>>>>> >> https://github.com/apache/arrow-go/pull/704 >> > (I >> > > > > > haven't >> > > > > > >>> > >> started >> > > > > > >>> > >> > >>>>>> looking at it yet) >> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that >> > this >> > > > work >> > > > > > is >> > > > > > >>> also >> > > > > > >>> > >> in >> > > > > > >>> > >> > >>>>>> progress (So 4 languages!) >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation * >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to >> > > report >> > > > > the >> > > > > > >>> > numbers >> > > > > > >>> > >> as >> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have >> given 1 >> > > > round >> > > > > of >> > > > > > >>> > >> feedback >> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note >> that >> > the >> > > > > > default >> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not >> > Q3. I >> > > > got >> > > > > > >>> around >> > > > > > >>> > >> 70% >> > > > > > >>> > >> > >>>>>> performance improvement in the decoding speed >> > when >> > > > > using >> > > > > > >>> the O3 >> > > > > > >>> > >> > flag. >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with >> > Vinoo >> > > > and >> > > > > > >>> Julien) >> > > > > > >>> > >> and >> > > > > > >>> > >> > **Cross >> > > > > > >>> > >> > >>>>>> Language testing* >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> Let me know if you have any questions or >> > > > feedback. >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify >> > Columns >> > > > > > >>> (Graviton >> > > > > > >>> > 3, >> > > > > > >>> > >> ARM >> > > > > > >>> > >> > >>>>>> Neoverse V1) >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐ >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 >> > > (MB/s) │ >> > > > > > >>> Speedup │ >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤ >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │ >> 5,523 >> > > │ >> > > > > > >>> 1.75x │ >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │ >> 5,685 >> > > │ >> > > > > > >>> 1.76x │ >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │ >> 5,652 >> > > │ >> > > > > > >>> 1.77x │ >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │ >> 5,473 >> > > │ >> > > > > > >>> 1.72x │ >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘ >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR < >> > > > > > >>> > [email protected] >> > > > > > >>> > >> > >> > > > > > >>> > >> > >>>>>> wrote: >> > > > > > >>> > >> > >>>>>> >> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <[email protected]> : >> Got >> > > it. >> > > > > > >>> > >> > >>>>>>> >> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <[email protected]> >> > > > > > >>> > >> > >>>>>>> >> > > > > > >>> > >> > >>>>>>> >> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start >> moving >> > the >> > > > > spec >> > > > > > >>> > >> development >> > > > > > >>> > >> > >>>>>>>> into >> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for >> finalizing >> > > it? >> > > > > > >>> > >> > >>>>>>>> >> > > > > > >>> > >> > >>>>>>> >> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the >> > > > examples I >> > > > > > >>> have in >> > > > > > >>> > >> the >> > > > > > >>> > >> > >>>>>>> spec based >> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we should be >> > good >> > > > to >> > > > > go >> > > > > > >>> for >> > > > > > >>> > the >> > > > > > >>> > >> > >>>>>>> markdown format. >> > > > > > >>> > >> > >>>>>>> >> > > > > > >>> > >> > >>>>>>> Thanks everyone! >> > > > > > >>> > >> > >>>>>>> >> > > > > > >>> > >> > >>>>>>> >> > > > > > >>> > >> > >>>>>>>> >> > > > > > >>> > >> > >>>>>>>> Andrew >> > > > > > >>> > >> > >>>>>>>> >> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK >> GAUR < >> > > > > > >>> > >> [email protected]> >> > > > > > >>> > >> > >>>>>>>> wrote: >> > > > > > >>> > >> > >>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> Hi team, >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> 1) Andrew >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My >> PR >> > > did >> > > > > add >> > > > > > >>> all >> > > > > > >>> > the >> > > > > > >>> > >> > test >> > > > > > >>> > >> > >>>>>>>> files I >> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we >> > can >> > > > club >> > > > > > it >> > > > > > >>> > >> together. >> > > > > > >>> > >> > >>>>>>>> WIll also >> > > > > > >>> > >> > >>>>>>>>> aid >> > > > > > >>> > >> > >>>>>>>>> cross language testing >> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust >> > > > implementation. >> > > > > > >>> This is >> > > > > > >>> > >> > great. >> > > > > > >>> > >> > >>>>>>>> Thanks >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> 2) Antoine >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the >> numbers >> > on >> > > > AMD. >> > > > > > >>> Looks >> > > > > > >>> > >> like >> > > > > > >>> > >> > you >> > > > > > >>> > >> > >>>>>>>> are >> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of >> > BSS. >> > > > This >> > > > > > is >> > > > > > >>> > >> > amazing!!. >> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling >> > > > design. >> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In >> some >> > > crude >> > > > > > >>> > >> experiments I >> > > > > > >>> > >> > >>>>>>>> didn't get >> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on >> Graviton3 >> > > (but >> > > > > > maybe >> > > > > > >>> > there >> > > > > > >>> > >> was >> > > > > > >>> > >> > >>>>>>>> something >> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation). >> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding >> for >> > > the >> > > > > > spec >> > > > > > >>> in >> > > > > > >>> > >> this >> > > > > > >>> > >> > >>>>>>>> case. >> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for >> all >> > > open >> > > > > > >>> questions >> > > > > > >>> > >> minus >> > > > > > >>> > >> > >>>>>>>> the >> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on this >> > > soon) >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> 3) Micah >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does >> allow >> > > for >> > > > > > using >> > > > > > >>> > >> FastLane >> > > > > > >>> > >> > >>>>>>>> with the >> > > > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We >> > > should >> > > > be >> > > > > > >>> able to >> > > > > > >>> > >> > inject >> > > > > > >>> > >> > >>>>>>>> any >> > > > > > >>> > >> > >>>>>>>>> layout >> > > > > > >>> > >> > >>>>>>>>> in the current design. >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open >> > comments >> > > > on >> > > > > > the >> > > > > > >>> spec >> > > > > > >>> > >> this >> > > > > > >>> > >> > >>>>>>>> week. >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> Best >> > > > > > >>> > >> > >>>>>>>>> Prateek >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve >> > Loughran < >> > > > > > >>> > >> > >>>>>>>> [email protected]> >> > > > > > >>> > >> > >>>>>>>>> wrote: >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah >> Kornfield >> > < >> > > > > > >>> > >> > >>>>>>>> [email protected]> >> > > > > > >>> > >> > >>>>>>>>>> wrote: >> > > > > > >>> > >> > >>>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described >> for >> > > ORC >> > > > > in >> > > > > > >>> the >> > > > > > >>> > >> paper >> > > > > > >>> > >> > >>>>>>>> is that >> > > > > > >>> > >> > >>>>>>>>> it >> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. >> This >> > > is >> > > > > > >>> different >> > > > > > >>> > >> then >> > > > > > >>> > >> > >>>>>>>> the >> > > > > > >>> > >> > >>>>>>>>> design >> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed >> > > > encoding >> > > > > > per >> > > > > > >>> page >> > > > > > >>> > >> in >> > > > > > >>> > >> > >>>>>>>> parquet. >> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't >> think >> > > > branch >> > > > > > >>> > >> > >>>>>>>> misprediction should >> > > > > > >>> > >> > >>>>>>>>>> be a >> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree >> that >> > we >> > > > > should >> > > > > > >>> be >> > > > > > >>> > >> > >>>>>>>> conservative in >> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings. >> > > > > > >>> > >> > >>>>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>>> +1 >> > > > > > >>> > >> > >>>>>>>>>> >> > > > > > >>> > >> > >>>>>>>>> >> > > > > > >>> > >> > >>>>>>>> >> > > > > > >>> > >> > >>>>>>> >> > > > > > >>> > >> > > >> > > > > > >>> > >> > >> > > > > > >>> > >> > >> > > > > > >>> > >> > >> > > > > > >>> > >> >> > > > > > >>> > > >> > > > > > >>> > >> > > > > > >>> >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> >
