szehon-ho opened a new pull request, #17956: URL: https://github.com/apache/iceberg/pull/17956
## Description This adds support for Spark 4.2's new opaque `PartitionPredicate` API on top of #14984. Iceberg now: - accumulates data and partition predicates across iterative pushdown calls - evaluates opaque Spark partition predicates against planned task partition keys - prunes tasks that do not match evaluable predicates - matches partition fields by field ID and transform for each scanned spec - keeps tasks when a predicate cannot be evaluated against an evolved spec All partition predicates are currently returned to Spark for post-scan evaluation. This provides a correctness backstop for partition evolution. A future optimization can return only predicates that cannot be evaluated against every scanned spec. The tests cover direct predicate evaluation, iterative pushdown, partition evolution, unsupported V2 filters, UDFs, non-first and multiple partition fields, and nested identity partition fields. ## Testing - `TestFilteredScan`: 85 tests, 9 skipped, 0 failures - `TestFilterPushDown`: 38 tests, 0 failures - focused `endsWith` partition predicate and partition-evolution tests pass - Spark 4.2 Java formatting check passes The Spark 4.2 tests currently require a temporary JDK queue substitution for two stale Guava `Queues` usages introduced by #14984. That unrelated workaround is not included in this PR. --- **AI Disclosure** - Model: GPT-5 - Platform/Tool: Codex - Human Oversight: partially reviewed - Prompt Summary: Add Spark 4.2 PartitionPredicate support to Iceberg scan planning, including partition evolution handling and integration tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
