andygrove opened a new pull request, #7233: URL: https://github.com/apache/datafusion-comet/pull/7233
## Which issue does this PR close? Part of #7138. Based on #7156, which should merge first. Until then, the diff includes its changes. The change specific to this PR is the last commit. ## Rationale for this change A data file can store a narrower type than its table column, such as `INT` data in a `BIGINT` column or `TINYINT` data in an `INT` column. iceberg-rust skips all pages of such a file for range and `IN` predicates, and its row filter narrows the literal to the file's type, so an out-of-range value becomes NULL. The filter above the scan cannot restore the dropped rows. #7156 guards the promotions that the schema history records. Files imported by `add_files`, `migrate` or `snapshot` are not checked against the schema, so the history records nothing for them and they still lose rows. ## What changes are included in this PR? Those three procedures set `schema.name-mapping.default`. For a table that has one, treat every `INT`, `BIGINT` and `DOUBLE` column as promoted. Only null checks and equality outside `NOT` are pushed for those columns, which is the rule #7156 applies to recorded promotions. Range and `IN` predicates stay with the filter above the scan. Tables with a name mapping and no imported files lose range pruning on those columns. That is the cost of not dropping rows silently. Still open under #7138: promotions made before Iceberg 0.12, and files imported into a table without a name mapping. The Iceberg guide documents both. ## How are these changes tested? New test `residuals on tables with a name mapping keep rows from imported files` in `CometIcebergNativeSuite`. It appends a Parquet file that stores `INT`, `TINYINT` and `FLOAT` to a table with `BIGINT`, `INT` and `DOUBLE` columns, adds a name mapping the way `add_files` does, and compares ten range, `IN` and equality predicates with Spark. It also checks that `id = 2 AND t = 2` still pushes. The test fails with the guard disabled. Spark 4.1 / JDK 17: `CometIcebergNativeSuite`, `CometIcebergNativeScanSuite` and `CometFuzzIcebergSuite` passed (181 passed, 2 version cancellations). Other Spark profiles were not run locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
