Hi everyone, I'd like to revisit Parquet Page Index pruning in Iceberg's custom row-based Parquet reader.
This has a fairly long history in the project. Page skipping was originally discussed in #193, and #1566 implemented an earlier version. More recent work and questions in #9479 and #14865 also touched on pushing filtering further into Iceberg's Parquet readers. I've been experimenting with the current Iceberg main branch and parquet-java 1.17.1 to understand whether the current APIs make this easier to integrate safely. I opened: issue: #17596 https://github.com/apache/iceberg/issues/17596 draft POC: #17597 https://github.com/apache/iceberg/pull/17597 The POC is intentionally limited to the custom row-based reader and uses Parquet Page Index only for conservative page-level I/O pruning. Residual predicate evaluation remains unchanged. I also ran a small local I/O experiment. The input contained 500,000 rows in one row group, approximately 1,000 rows per page, and a predicate matching a 100-row range. For data sorted by the predicate column: candidate rows: 500,000 -> 1,000 bytes read: 131.054 MiB -> 0.288 MiB median time: 67.79 ms -> 8.16 ms For randomly distributed data, no pages could be eliminated: candidate rows: 500,000 -> 500,000 bytes read: 131.054 MiB -> 131.079 MiB median time: 66.56 ms -> 69.72 ms A file without a predicate-column ColumnIndex also fell back conservatively to the complete row group. These are POC measurements rather than formal JMH benchmark results. I mainly used CountingInputFile to verify that selecting fewer pages actually resulted in less physical input being read. My current preference is to keep the first production scope narrow: row-based reader only, conservative pruning only, conservative fallback for missing/unsupported Page Index information, and no new public configuration until the reader architecture is agreed upon. Thanks, Shangqing
