viirya opened a new pull request, #60010:
URL: https://github.com/apache/spark/pull/60010
### What changes were proposed in this pull request?
This is a backport of #59944 to `branch-4.x`.
`PushPredicateThroughNonJoin` no longer pushes a throwable conjunct
(`raise_error`, `assert_true`) below an operator ahead of a conjunct written
before it that stays above, in the generic `UnaryNode` case and the
`Aggregate`, `Window`, `Union` and `EventTimeWatermark` cases. It also never
pushes a throwable conjunct through a `Generate` without `outer` or a
`ScriptTransformation`, and in the `Aggregate` case a conjunct over a throwable
alias counts as throwable. See #59944 for the full description.
Differences from the `master` commit:
- `branch-4.x` does not have SPARK-60150 (`pushBelowAggregate`), so the
`Aggregate` case applies `splitPushable` in place, as #59944 did before it was
merged with SPARK-60150.
### Why are the changes needed?
A throwable conjunct can run on rows that the `AND` short-circuit would have
kept from it, so a query that should succeed fails with
`USER_RAISED_EXCEPTION`. For example:
```scala
spark.range(10).select($"id", explode(array($"id")) as "x")
.where($"x" > 5 && assert_true($"id" > 5).isNull) // Generate
spark.range(10).groupBy($"id").agg(max($"id") as "m")
.where($"m" > 5 && assert_true($"id" > 5).isNull) // Aggregate
```
The same happens below a Python UDF evaluation (`BatchEvalPython` /
`ArrowEvalPython`), a `Window` and a `Union`.
### Does this PR introduce _any_ user-facing change?
Yes. Queries like the ones above failed with `USER_RAISED_EXCEPTION` and now
succeed. Filters without throwable conjuncts are pushed as before, except that
conjuncts left above the operator in the generic and `Window` cases keep their
written order, instead of the non-deterministic ones moving to the end.
### How was this patch tested?
The tests from #59944. `catalyst/testOnly
org.apache.spark.sql.catalyst.optimizer.*` and `sql/testOnly` of
`DataFrameSuite`, `PythonUDFSuite`, `*FilterPushdown*`,
`GeneratorFunctionSuite` and `DataFrameAggregateSuite` on JDK 17, and
scalastyle.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
