joyhaldar opened a new pull request, #15064: URL: https://github.com/apache/iceberg/pull/15064
## Summary This PR adds manifest pruning optimization for `NOT IN` and `!=` predicates when a manifest contains a single distinct partition value (i.e., when `lower == upper`). ## Problem Currently, [ManifestEvaluator](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java) cannot prune manifests for `NOT IN` and `!=` predicates, even when the manifest provably contains no matching partitions. ## Solution When `lower == upper` and the manifest has no nulls or NaNs, we can safely prune if: * For `NOT IN`: the single partition value is in the exclusion list * For `!=`: the single partition value equals the literal This mirrors the optimization added in #14593 for [InclusiveMetricsEvaluator](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java), but applied at the manifest level for partition pruning. ## Testing * Added unit tests for both `notIn` and `notEq` optimizations * Verified correct behavior with nulls (must scan) and without nulls (can prune) Fixes #15063 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
