Hi Andrew, I agree the read path for Option C can be optimized in several ways to approach Option B and Option A. Similarly, this might be almost true for the writing path, again at the cost of implementation complexity, that some parquet implementations might not decide to take on.
Could we - for the sake of this discussion - evaluate the cost of the new repetition tradeoff? Rok On Mon, Jul 6, 2026 at 11:09 PM Andrew McCormick via dev < [email protected]> wrote: > Hi Rok, > > Looking over your gist and the linked PRs I could find, as far as I can > tell option C is still using the baseline full dremel machinery readback > without any optimizations, and indeed that is quite slow compared with A/B. > With the read optimization of simply ignoring the rep levels when the hint > is present implemented for option C, we get the same performance you got > for B, because in both cases we're skipping any rep/def machinery. > > On the write side I can see a few options. One option is that your writer > can simply have an optimized fast path for the runtime-determined case when > the array lengths are all the same; that's relatively easy to add, and > saves most of the time. Another option is for your engine to incorporate > the concept of VECTOR, allowing it to exercise a statically determined fast > path for writing the array, and thus save even the small amount of time > spent checking for they are all the same length. Engines can have the > concept of vector even if the format does not. > > Overall, I still think that option C is the clear winner, because as far as > I can tell you can get essentially the same speed as B, but without the > backward compatibility concerns. > > On Mon, Jul 6, 2026 at 9:06 AM Rok Mihevc <[email protected]> wrote: > > > I ran benchmarks [1] on a couple of draft implementations on top of > > arrow-cpp ([A], [B], [C]) parquet reader/writer using non-sythetic data. > > Results seem close to the Photon engine results Alkis quotes. > > See the read / write charts below: > > > > [image: image.png] > > Figure 1: (Second plot in the notebook [1]) Please note how random reads > > on non-nullable (req) column do much better on Option B than Option C. I > > believe this is due to reading of rep/def levels which should still be > > optimiable in favour of Option C. In the nullable case they perform > almost > > on-par, but Option B still does better. > > Given that the length of values and definition levels for LIST is > > data-dependant I would expect random reads on Option B to perform better > > than Figure 1 shows compared to Option C. > > > > [image: image.png] > > Figure 2: (Third plot in the notebook [1]) In the write benchmark for > > non-nullable - Option B does not have to write rep/def and does 4-5x > > better. In nullable case Option B is closer to Option C but still does 2x > > better. > > > > We've discussed reading performance so far, what are our expectations for > > write performance? > > While having the VECTOR physically be LIST is good for backwards > > compatibility we'd be leaving redundant information and contract in the > > format. > > > > > > [A] https://github.com/rok/arrow/pull/53 > > [B] https://github.com/rok/arrow/pull/51 > > [C] https://github.com/rok/arrow/pull/52 > > [1] https://gist.github.com/rok/e95c9877ce1dc895f3846809ffa6a9ee - plots > > all the way down > > > > > > Rok > > > > On Mon, Jul 6, 2026 at 5:12 PM Antoine Pitrou <[email protected]> > wrote: > > > >> > >> Le 06/07/2026 à 17:03, Alkis Evlogimenos via dev a écrit : > >> > Here's why Option C is the superior option: > >> > > >> > 1. The logical type makes the implementation that assumes all arrays > are > >> > fixed length, on par in performance with Option B > >> > 2. Without the logical type (and with a little bit extra complexity) a > >> > smart enough *reader* can walk the def/rep levels before decoding, > infer > >> > (1) - the writer wrote fixed len arrays - and call the implementation > in > >> > (1). Compared to (1) this is 1.5x slower. > >> > >> At the cost of higher implementation complexity and maintenance cost. > >> Does any mainstream open source implementation of Parquet do this? > >> > >> > 3. Without the logical type a naive reader - that is any reader of > >> today - > >> > can decode the fixed len arrays. Compared to (1) this is 5x slower. > >> > > >> > The above are prototyped and benchmarked on Databricks Photon engine > >> (C++). > >> > >> While the numbers you give are believable, I would personally be more > >> confident with benchmarks obtained on a mainstream OSS implementation > >> (also with the patch / PR published somewhere to look at). > >> > >> Regards > >> > >> Antoine. > >> > >> > >> >
