tdcmeehan opened a new pull request, #58705: URL: https://github.com/apache/spark/pull/58705
### What changes were proposed in this pull request? Add a Catalyst optimizer rule that derives a data-source-pushable predicate from comparisons over ANSI integral addition and subtraction by literals. For example, `i - 10 > 100` gains the conjunct `i > 110 OR i < INT_MIN + 10`. The second range retains inputs for which the original arithmetic must raise an overflow error. The original comparison remains as a residual predicate and continues to define exact evaluation semantics. The rule supports byte, short, integer, and long arithmetic with literal constants and ordered comparisons or equality. It does not change legacy arithmetic, floating-point arithmetic, multiplication, or arithmetic with a nonliteral operand. ### Why are the changes needed? Arithmetic around a column currently prevents Parquet and other data sources from using their min/max statistics for predicate pruning. The derived predicate exposes safe ranges over the underlying column while preserving ANSI overflow behavior. ### Does this PR introduce _any_ user-facing change? Yes. In ANSI mode, eligible arithmetic filters can now be pushed into data sources for pruning. Query results and overflow errors are unchanged because the original predicate is retained. ### How was this patch tested? Added Catalyst optimizer coverage for all ordered comparisons, equality, reversed operands, both overflow directions, and unsupported cases. Added a Parquet V2 test that verifies the derived `Or` predicate is pushed into the scan and executes the query. Ran: ``` sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 'catalyst/testOnly *BinaryComparisonSimplificationSuite' sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 'sql/testOnly *ParquetV2FilterSuite -- -z "push down derived predicate"' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex 0.147.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
