ErikBPF commented on PR #5732: URL: https://github.com/apache/datafusion-comet/pull/5732#issuecomment-5655310392
Updated the title and description to match the final behavior: list, map, and struct null checks retain native scanning, while complex-column residuals are omitted and the post-scan filters enforce correctness. Removed the stale `CometScanRuleSuite` claim and the unused `IcebergReflection.getFilterExpressions` helper. The end-to-end list/map/struct null-check test now checks `getResidualPoolCount == 0` on supported Iceberg versions, in addition to Spark equivalence, explicit expected rows, and one native scan. A separate sweep exercises null checks over the generated complex columns, including arrays of structs and structs of arrays. Orion verification on Spark 4.1.3 / Iceberg 1.11.0 passed: both targeted regressions, followed by the full `CometIcebergNativeSuite` and `CometFuzzIcebergSuite` run with 114 succeeded, 0 failed, and 1 existing cancellation for SPARK-55626. The native build and formatting checks passed. The current GitHub workflow runs are waiting for maintainer approval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
