dwsmith1983 opened a new pull request, #6040: URL: https://github.com/apache/datafusion-comet/pull/6040
## Which issue does this PR close? Closes #6039. ## Rationale for this change The native Iceberg scan skips Parquet pages through iceberg-rust's row selection, but no test notices if that stops. The behaviour depends on one builder flag in `IcebergScanExec` and on iceberg-rust internals that change with every pin bump, and the scan node reports no page-index counters, so a regression would only show up as slower queries. ## What changes are included in this PR? One test in `CometIcebergNativeSuite`, `native scan skips pages within a single row group`. It writes 300,000 sorted rows into a hadoop-catalog table with a 512 MB row group, 16 KB pages and a 2000-row page limit, so the table is one data file with one row group and many pages. It reads the data file path from the `.files` metadata table and checks the footer: one row group, and more than 50 pages in the `id` column's offset index, so the fixture cannot degrade into a layout where file-level or row-group pruning would explain the savings. It then runs a full read and `WHERE id BETWEEN 1000 AND 1100` through `CometIcebergNativeScanExec` and asserts exact sums, 101 output rows, one split each, and that the range read's `bytes_scanned` is under a fifth of the full read's. With one row group, only page skipping can produce that. ## How are these changes tested? This is the test. Measured locally on Spark 3.5 with Iceberg 1.8.1: 158 `id` pages, full read 20,887,912 bytes, range read 549,074 bytes, so the bound has a wide margin in both directions. The test runs in about 2.5 seconds inside the suite. `CometIcebergNativeSuite` passes with 98 succeeded and 8 cancelled (the cancelled ones need a REST catalog), and spotless and the semantic scalafix check are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
