L. C. Hsieh created SPARK-61667:
-----------------------------------
Summary: Use Expression.isTotal to fold the size of CreateArray in
ConstantFolding
Key: SPARK-61667
URL: https://issues.apache.org/jira/browse/SPARK-61667
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
ConstantFolding folds size(array(e1, ..., en)) to n without evaluating the
elements when none of them can fail. It decides that with its own check,
hasNoSideEffect, which above attributes, literals and some casts only accepts
CreateArray and CreateNamedStruct.
Since SPARK-61663, every expression declares whether it is total with
Expression.isTotal, and the ones that are always total mix in NoThrow
(comparisons, And/Or/Not, IsNull, Coalesce, Length, Substring, ...).
SPARK-61663 kept ConstantFolding as it was by naming CreateArray and
CreateNamedStruct there, so e.g. size(array(i > 1)) is still evaluated although
its element cannot fail.
This is to revisit hasNoSideEffect so that it relies on the declaration
(NoThrow, or Expression.isTotal for the expressions that are total in some eval
modes only) instead of the two names. It changes which plans are folded, so it
needs its own review: the rule skips the evaluation of the elements, and a
wrong declaration would hide an error (see SPARK-60820 and SPARK-60868).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]