andygrove opened a new pull request, #7313: URL: https://github.com/apache/datafusion-comet/pull/7313
## Which issue does this PR close? Backport of #7085 to `branch-1.2`. #7085 closes #7030 on `main`. ## Rationale for this change When every requested nested leaf is missing from a Parquet file, the native scan can return non-null containers where Spark returns NULL. That changes `IS NULL`, filters and `count` results, silently, on the default configuration (#7030, priority:critical, correctness). `branch-1.2` has the bug, so it needs the fix before 1.2.0 ships. #7085 also makes native scans of name-mapped nested Iceberg tables fall back to Spark. `branch-1.2` is the newest release branch, so no newer branch needs a backport. #7085 carries no `backport-1.1` label, so no `branch-1.1` backport is part of this. ## What changes are included in this PR? `git cherry-pick -x c4a0b1f2bc` of the squash commit of #7085, with no adaptations. It applied without conflicts. The only change outside tests, docs and the scan code is a comment in `operator.proto`; no proto field is added or renumbered. See #7085 for the full description. ## How are these changes tested? On this branch, with a `libcomet` built from the pick: - `cargo check --all-targets`, `cargo fmt --check`, and the 351 `parquet` unit tests in `datafusion-comet` pass. - `./mvnw test-compile` passes. - `CometNativeReaderSuite`, `IcebergReflectionSuite` and `CometIcebergNativeSuite` pass: 255 succeeded, 0 failed, 3 canceled. The canceled tests are version-conditional (`issue #4136`, an unknown partition transform, and an output-ordering fallback); the new tests all ran. - With `parquet_support.rs` reverted to `branch-1.2`, `issue #7030: missing nested fields - legacy true` in `CometNativeReaderSuite` fails, so the branch has the bug and the test catches it. I did not run the Spark SQL test suites or the Iceberg suites across versions that #7085 ran in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
