holdenk opened a new pull request, #58403:
URL: https://github.com/apache/spark/pull/58403
### What changes were proposed in this pull request?
Follow up to SPARK-47672. That change stopped `PushPredicateThroughNonJoin`
from pushing a filter below a `Project` when the filter references expensive
projected aliases, because pushing caused them to be evaluated twice, and
parked those conditions above the projection.
This splits the projection into a stack of `Project`/`Filter` layers
instead, so that the expensive expressions a condition does not need are only
evaluated for the rows that condition kept:
Filter f AND g Filter g
Project a, rlike(e,'magic') AS f, Project a, f,
rlike(e,'other') AS g
rlike(e,'other') AS g --> Filter f
child Project a, e,
rlike(e,'magic') AS f
child
Conditions are grouped by the aliases they reference, and the group needing
the fewest not-yet-computed aliases is split off first, breaking ties towards
the condition written first. Two restrictions keep the rule from costing more
than it saves:
- A projection is only split off while doing so leaves an expensive alias
for a later layer. An extra operator has a per-row cost of its own, so it has
to buy a real deferral.
- Aliases sharing an expensive sub-expression are treated as one indivisible
unit. Subexpression elimination works within a single projection and cannot
reach across a filter, so splitting such aliases apart would evaluate the
shared part once for every row below the filter and again for every row that
survived it. A struct-returning UDF read field by field is the common shape.
Gated on the new `spark.sql.optimizer.splitProjectionForExpensiveFilters`
(default true), which also requires `spark.sql.optimizer.avoidDoubleFilterEval`.
### Why are the changes needed?
SPARK-47672 stops the double evaluation but leaves the projection computing
every expensive element for every row reaching it, even when a filter above it
is about to discard most of them. Whole stage codegen already defers a
projected expression past a filter that does not need it
(`CodegenSupport.evaluateRequiredVariables`), so what this buys is the same
saving on the paths codegen does not cover: interpreted projections, operators
it bails out of, and Python UDF evaluation.
### Does this PR introduce _any_ user-facing change?
Plans for queries filtering on expensive projected expressions gain
projection/filter layers. Results, schemas and column ordering are unchanged.
### How was this patch tested?
Thirteen new plan tests in `FilterPushdownSuite` covering splitting, the
ordering heuristic, the no-split cases (a condition needing every expensive
element, only cheap elements left to defer, shared expensive work), column
ordering, and the config off. Three new end-to-end tests in `DataFrameSuite`
count UDF invocations to show the saving is real (30 -> 15), that results match
with splitting and codegen each on and off, that the split follows the filter's
order rather than the projection's, and that shared expensive work is not split
apart.
`PlanStabilitySuite` confirms the TPCDS golden plans are unaffected.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5 using my previous work in the original PR as a
starting point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
