cloud-fan commented on code in PR #56227:
URL: https://github.com/apache/spark/pull/56227#discussion_r3364187776
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala:
##########
@@ -208,6 +213,58 @@ object SchemaPruning extends SQLConfHelper {
}
}
+ private def getArrayReturningHigherOrderFunctionRootFields(expr:
Expression): Seq[RootField] = {
+ expr match {
+ case ArrayFilter(argument, lambda: LambdaFunction) =>
+ getArrayReturningHigherOrderFunctionRootFields(argument, lambda,
numElementVariables = 1)
+ case ArraySort(argument, lambda: LambdaFunction,
allowNullComparisonResult)
+ if !allowNullComparisonResult && lambda.function.nullable =>
+ // The strict null-comparator error includes the compared values, so
pruning their
+ // element fields would change the observable error parameters.
+ getRootFields(argument) ++ getRootFields(lambda.function)
+ case ArraySort(argument, lambda: LambdaFunction, _) =>
+ getArrayReturningHigherOrderFunctionRootFields(argument, lambda,
numElementVariables = 2)
+ case _ =>
+ expr.children.flatMap(getArrayReturningHigherOrderFunctionRootFields)
+ }
+ }
+
+ private def getArrayReturningHigherOrderFunctionRootFields(
+ argument: Expression,
+ lambda: LambdaFunction,
+ numElementVariables: Int): Seq[RootField] = {
Review Comment:
These two helpers (`getArrayReturningHigherOrderFunctionRootFields` here and
the pre-existing `getArrayHigherOrderFunctionRootField` above) share the same
collect -> merge -> `SelectedField.withDataType` core, differing only in
element-variable arity and the no-fields fallback (`Seq.empty` vs `None`).
Optional: factoring out the shared core would keep the two field-collection
paths from silently diverging if one is edited later. Non-blocking — the
current split is intentional and correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
