david-mollitor-db opened a new pull request, #58830: URL: https://github.com/apache/spark/pull/58830
### What changes were proposed in this pull request? A new logical optimizer rule `CombineDisjunctiveInPredicates` that coalesces OR-connected `EqualTo` and `In` disjuncts on the same deterministic, non-foldable subject into a single `In`, letting the adjacent `OptimizeIn` finish the job (dedup, single-element -> `EqualTo`, large list -> `InSet`). `EqualTo(x, c)` is treated as a one-element membership `In(x, [c])`. Examples: - `x = 1 OR x = 2 OR x = 3` -> `x IN (1, 2, 3)` - `x IN (1, 2) OR x IN (3, 4)` -> `x IN (1, 2, 3, 4)` - `x = 1 OR x IN (2, 3)` -> `x IN (1, 2, 3)` The rule runs immediately before `OptimizeIn` and is gated by an internal flag `spark.sql.optimizer.combineDisjunctiveInPredicates.enabled` (default true). Other engines already do this: MySQL/MariaDB's range optimizer treats OR-of-equals as equivalent to `IN`, and PostgreSQL 17 added an explicit OR -> `= ANY` transform. ### Why are the changes needed? A single `In`/`InSet` is a more optimized representation than the equivalent OR chain: - **Single subject evaluation** — `In.eval` evaluates the subject expression once; an `Or` of N `EqualTo`/`In` nodes re-evaluates it N times. - **Set dispatch** — once merged and above the conversion threshold, `OptimizeIn` turns the `In` into `InSet`, which uses a `switch` (compact ints) or a sorted-array binary search; OR-connected nodes never get this. - **Avoids the whole-stage-codegen method-size cliff** — a very large OR chain overflows the generated method-size limit and falls back to interpreted execution; the merged `InSet` stays compact. **Dependency on SPARK-59539 (important).** This rule's payoff depends on `InSet` being fast. On top of the boxing-free sorted-array binary-search `InSet` (SPARK-59539) it is a clean win; but with the current **boxed `Set[Any]` `InSet` it regresses at moderate N** — `Set.contains(Object)` autoboxes the subject on every row and is slower than the inline primitive OR chain it replaced. This PR should therefore land on top of, or together with, SPARK-59539. Microbenchmark, `Long` column, 10M rows, non-matching values (worst case), rule off (raw OR) vs rule on (merged), measured on top of the SPARK-59539 binary-search `InSet` (local, indicative; committable numbers to be regenerated on the CI runners): | N | raw OR (ms) | merged IN/InSet (ms) | |---|---|---| | 20 | 39 | 40 | | 100 | 148 | 54 | | 500 | 65142 (interpreted fallback) | 82 | For contrast, with the boxed `Set` `InSet` the N=20 merged case is ~107 ms — ~2.7x **slower** than the raw OR — which is exactly why the SPARK-59539 dependency matters. Note: merging a large (> `spark.sql.optimizer.inSetConversionThreshold`) OR-of-equalities into `IN` -> `InSet` can make Parquet row-group pruning coarser (a min/max envelope over the set vs per-value `FilterApi.or(eq, ...)`); the internal flag allows disabling the rule where that matters. ### Does this PR introduce _any_ user-facing change? No. Query results are unchanged; this only rewrites equivalent predicate shapes. The new internal flag defaults to on. ### How was this patch tested? New `CombineDisjunctiveInPredicatesSuite` (`PlanTest`), covering: OR-chains of equalities -> `In`; conversion to `InSet` above the threshold; commuted equality; OR'd `IN` lists merged; mixed equality + `IN`; three `IN` lists; partial OR keeping non-membership disjuncts; distinct subjects merged independently; non-literal members kept as `In`; single-membership and lone-`IN` no-ops; non-deterministic subject not merged; foldable-subject not merged; and config-off no-op. All 14 pass; existing `OptimizeInSuite` continues to pass. An exploratory `OrMembershipToInBenchmark` is included for the numbers above; committable results to be regenerated on the CI runners. ### Was this patch authored or co-authored using generative AI tooling? This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
