> > 2. Without the logical type (and with a little bit extra complexity) a > > smart enough *reader* can walk the def/rep levels before decoding, infer
> At the cost of higher implementation complexity and maintenance cost. > Does any mainstream open source implementation of Parquet do this? Triggered by the conversation on the call last week, I implemented pretty much this in Hardwood [1]. For definition levels, we're applying the common optimization to not materialize them if the def levels of a page are a single RLE run. For repetition levels, we apply a tiled matcher which detects the periodic pattern for both bit-packed and RLE-encoded values. The results look quite promising; apart from lists with size 1, this is always faster than the baseline. I'm working on some systematic benchmarking right now [2] and plan to share the findings in a quick blog post later this week or next. --Gunnar [1] https://github.com/hardwood-hq/hardwood/pull/741 [2] https://github.com/hardwood-hq/hardwood-benchmarks/pull/5 On Mon, 6 Jul 2026 at 17:13, Antoine Pitrou <[email protected]> wrote: > > > Le 06/07/2026 à 17:03, Alkis Evlogimenos via dev a écrit : > > Here's why Option C is the superior option: > > > > 1. The logical type makes the implementation that assumes all arrays are > > fixed length, on par in performance with Option B > > 2. Without the logical type (and with a little bit extra complexity) a > > smart enough *reader* can walk the def/rep levels before decoding, infer > > (1) - the writer wrote fixed len arrays - and call the implementation in > > (1). Compared to (1) this is 1.5x slower. > > At the cost of higher implementation complexity and maintenance cost. > Does any mainstream open source implementation of Parquet do this? > > > 3. Without the logical type a naive reader - that is any reader of today - > > can decode the fixed len arrays. Compared to (1) this is 5x slower. > > > > The above are prototyped and benchmarked on Databricks Photon engine (C++). > > While the numbers you give are believable, I would personally be more > confident with benchmarks obtained on a mainstream OSS implementation > (also with the patch / PR published somewhere to look at). > > Regards > > Antoine. > >
