Github user rednaxelafx commented on a diff in the pull request:
https://github.com/apache/spark/pull/23079#discussion_r234508561
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
---
@@ -767,6 +767,15 @@ object ReplaceNullWithFalse extends Rule[LogicalPlan] {
replaceNullWithFalse(cond) -> value
}
cw.copy(branches = newBranches)
+ case af @ ArrayFilter(_, lf @ LambdaFunction(func, _, _)) =>
--- End diff --
I'm not sure if that's useful or not. First of all, the
`replaceNullWithFalse` handling doesn't apply to all higher-order functions. In
fact it only applies to a very narrow set, ones where a lambda function returns
`BooleanType` and is immediately used as a predicate. So having a generic
utility can certainly help make this PR slightly simpler, but I don't know how
useful it is for other cases.
I'd prefer waiting for more such transformation cases to introduce a new
utility for the pattern. WDYT?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
