Kimahriman commented on code in PR #34558:
URL: https://github.com/apache/spark/pull/34558#discussion_r3293466169
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala:
##########
@@ -146,9 +146,13 @@ class EquivalentExpressions(
// There are some special expressions that we should not recurse into all of
its children.
// 1. CodegenFallback: it's children will not be used to generate code
(call eval() instead)
// 2. ConditionalExpression: use its children that will always be
evaluated.
+ // 3. HigherOrderFunction: lambda functions operate in the context of
local lambdas and can't
+ // be called outside of that scope, only the arguments can be
evaluated ahead of
+ // time.
private def childrenToRecurse(expr: Expression): Seq[Expression] = expr
match {
case _: CodegenFallback => Nil
case c: ConditionalExpression =>
c.alwaysEvaluatedInputs.map(skipForShortcut)
+ case h: HigherOrderFunction => h.arguments
Review Comment:
Fixed in `eecc672cb44`.
I changed HOF CSE recursion to use an `alwaysEvaluatedArguments` hook. By
default it returns `arguments`, but `ArrayAggregate` overrides it to return
only the input array argument, so `zero` is no longer made CSE-visible before
the null-array guard.
I added the ANSI/CSE regression from this comment and verified it fails with
the old `h.arguments` recursion, then passes with the fix.
Verified with:
```bash
build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z
"aggregate function - null array does not evaluate zero expression through CSE"'
```
Result: passed (`1` test, `0` failures).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
