Le 06/07/2026 à 17:03, Alkis Evlogimenos via dev a écrit :
Here's why Option C is the superior option: 1. The logical type makes the implementation that assumes all arrays are fixed length, on par in performance with Option B 2. Without the logical type (and with a little bit extra complexity) a smart enough *reader* can walk the def/rep levels before decoding, infer (1) - the writer wrote fixed len arrays - and call the implementation in (1). Compared to (1) this is 1.5x slower.
At the cost of higher implementation complexity and maintenance cost. Does any mainstream open source implementation of Parquet do this?
3. Without the logical type a naive reader - that is any reader of today - can decode the fixed len arrays. Compared to (1) this is 5x slower. The above are prototyped and benchmarked on Databricks Photon engine (C++).
While the numbers you give are believable, I would personally be more confident with benchmarks obtained on a mainstream OSS implementation (also with the patch / PR published somewhere to look at).
Regards Antoine.
