rich7420 opened a new pull request, #7156: URL: https://github.com/apache/datafusion-comet/pull/7156
## Which issue does this PR close? Closes #7136. Closes #7137. Part of #7138. ## Rationale for this change The native Iceberg scan pushes each task's residual into iceberg-rust, which uses it to prune row groups using statistics, select rows using the page index and filter rows during decode. The filter above the scan cannot restore rows dropped there. On `main` and `branch-1.1`, three kinds of residual cause the native scan to drop rows that Spark returns: - `FLOAT`/`DOUBLE` predicates: Spark treats `-0.0` as equal to `0.0` and sorts every NaN above all numbers, while iceberg-rust compares values using the IEEE 754 total order, and its row-group pruning trusts statistics that can omit NaN (apache/iceberg-rust#3494). `WHERE f = 0.0` drops `-0.0` rows. - `NOT (id <=> 1)`: Iceberg passes `<=>` as `=`, and the native reader drops NULL rows that Spark keeps. - Columns whose type was promoted, such as `INT` to `BIGINT`: older files keep the narrower type, and iceberg-rust compares values from those files incorrectly (apache/iceberg-rust#3122). These issues were found while reviewing #6847. This PR splits out their fixes so they can be backported and listed as fixes. ## What changes are included in this PR? The residual serde now leaves these predicates to the filter above the scan: - On `FLOAT`/`DOUBLE` columns: range comparisons, comparisons with `0.0`, `-0.0` or NaN, and `FLOAT` literals that do not narrow exactly. Equality, `!=` and `IN` with other literals are still pushed. - Equality under `NOT` on a nullable column. - On columns whose type changed in the schema history: everything except null checks and `=` outside `NOT`. If snapshot expiration has removed older schemas, every `BIGINT`, `DOUBLE` and `DECIMAL` column counts as promoted. This removes some pushdown that `main` has today. Files imported by `add_files`, `migrate` or `snapshot`, and promotions made before Iceberg 0.12, leave no schema history, so they remain a known issue under #7138. The Iceberg guide lists the predicates excluded from pushdown and that known issue. ## How are these changes tested? Serializer tests cover each excluded predicate form, including under `AND`, `OR` and nested `NOT`, and the schema-history detection. Native tests compare against Spark for signed zeros and NaN, negated `<=>`, and promoted columns before and after snapshot expiration. A Rust test covers the bind-failure fallback, which the float tests no longer reach. The comparison and logical pushdown fixtures move from `DOUBLE` to `BIGINT` so their predicates are still pushed. Local (Spark 4.1 / Scala 2.13 / JDK 21): `CometIcebergNativeScanSuite` and `CometIcebergNativeSuite` passed (166 passed, 2 canceled by version assumptions), and so did the Rust test. `-Pspark-3.5,delta -Pstrict-warnings` test-compile passed. Other Spark profiles and the Iceberg Spark SQL suites were not run locally; `run-iceberg-tests` is needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
