yangshangqing95 opened a new pull request, #17597: URL: https://github.com/apache/iceberg/pull/17597
## Context Related to #17596. This is a draft POC that about Parquet page-level pruning in Iceberg's custom row-based Parquet reader. The purpose of this PR is to validate behaviors against the current Iceberg reader and parquet-java APIs before proposing a production-ready implementation. This PR is not intended to merge in its current form. ## What this POC does The POC: - enables parquet-java ColumnIndex filtering; - propagates filtered `PageReadStore` row indexes into Iceberg's position reader; - preserves physical `_pos` for non-contiguous page selections; - preserves fallback `_row_id` semantics derived from `_pos`. The Page Index is used only for conservative I/O pruning. It does not perform final record-level filtering, and existing residual evaluation is still required. ## Tests The included tests verify: - disjoint candidate ranges; - `_pos` correctness; - multiple row groups; - fallback `_row_id`; - zero-candidate row groups; - unchanged sequential `_pos` behavior without Page Index filtering. ## Initial measurements A local POC microbenchmark used: ```text 500,000 rows 1 row group ~1,000 rows/page uncompressed projection: id, payload predicate: id >= 250000 AND id < 250100 ``` Results: | Dataset | Page Index | Candidate Rows | Bytes Read | Median Time | | --- | ---: | ---: | ---: | ---: | | Sorted | OFF | 500,000 | 131.054 MiB | 67.79 ms | | Sorted | ON | 1,000 | 0.288 MiB | 8.16 ms | | Random | OFF | 500,000 | 131.054 MiB | 66.56 ms | | Random | ON | 500,000 | 131.079 MiB | 69.72 ms | | Sorted / no index | OFF | 500,000 | 131.054 MiB | 66.02 ms | | Sorted / no index | ON | 500,000 | 131.070 MiB | 69.15 ms | I would prefer to agree on the reader architecture and correctness constraints first. Then I can proceed to the next step of the work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
