I ran benchmarks [1] on a couple of draft implementations on top of arrow-cpp ([A], [B], [C]) parquet reader/writer using non-sythetic data. Results seem close to the Photon engine results Alkis quotes. See the read / write charts below:
[image: image.png] Figure 1: (Second plot in the notebook [1]) Please note how random reads on non-nullable (req) column do much better on Option B than Option C. I believe this is due to reading of rep/def levels which should still be optimiable in favour of Option C. In the nullable case they perform almost on-par, but Option B still does better. Given that the length of values and definition levels for LIST is data-dependant I would expect random reads on Option B to perform better than Figure 1 shows compared to Option C. [image: image.png] Figure 2: (Third plot in the notebook [1]) In the write benchmark for non-nullable - Option B does not have to write rep/def and does 4-5x better. In nullable case Option B is closer to Option C but still does 2x better. We've discussed reading performance so far, what are our expectations for write performance? While having the VECTOR physically be LIST is good for backwards compatibility we'd be leaving redundant information and contract in the format. [A] https://github.com/rok/arrow/pull/53 [B] https://github.com/rok/arrow/pull/51 [C] https://github.com/rok/arrow/pull/52 [1] https://gist.github.com/rok/e95c9877ce1dc895f3846809ffa6a9ee - plots all the way down Rok On Mon, Jul 6, 2026 at 5:12 PM Antoine Pitrou <[email protected]> wrote: > > Le 06/07/2026 à 17:03, Alkis Evlogimenos via dev a écrit : > > Here's why Option C is the superior option: > > > > 1. The logical type makes the implementation that assumes all arrays are > > fixed length, on par in performance with Option B > > 2. Without the logical type (and with a little bit extra complexity) a > > smart enough *reader* can walk the def/rep levels before decoding, infer > > (1) - the writer wrote fixed len arrays - and call the implementation in > > (1). Compared to (1) this is 1.5x slower. > > At the cost of higher implementation complexity and maintenance cost. > Does any mainstream open source implementation of Parquet do this? > > > 3. Without the logical type a naive reader - that is any reader of today > - > > can decode the fixed len arrays. Compared to (1) this is 5x slower. > > > > The above are prototyped and benchmarked on Databricks Photon engine > (C++). > > While the numbers you give are believable, I would personally be more > confident with benchmarks obtained on a mainstream OSS implementation > (also with the patch / PR published somewhere to look at). > > Regards > > Antoine. > > >
