[
https://issues.apache.org/jira/browse/SPARK-49396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876818#comment-17876818
]
Avery Qi commented on SPARK-49396:
----------------------------------
Assign to Avery, WIP
> PLAN_VALIDATION_FAILED_RULE_IN_BATCH in SimplifyConditionals rule
> -----------------------------------------------------------------
>
> Key: SPARK-49396
> URL: https://issues.apache.org/jira/browse/SPARK-49396
> Project: Spark
> Issue Type: Task
> Components: Optimizer
> Affects Versions: 4.0.0
> Reporter: Avery Qi
> Priority: Major
>
> *_{{SimplifyConditionals}}_* has a simplification code where if any of the
> (not first) branches in _*{{CaseWhen}}*_ is *_{{TrueLiteral}}_* we remove
> all the remaining branches including the *_{{{}elseValue{}}}._*
> {code:java}
> case CaseWhen(branches, ) if branches.exists(._1 == TrueLiteral) => // a
> branch with a true condition eliminates all following branches,
> // these branches can be pruned away
> val (h, t) = branches.span(_._1 != TrueLiteral)
> CaseWhen( h :+ t.head, None)}}{code}
> Now, the nullability check of *_{{CaseWhen}}_* checks that
> (1) either of the branches including *_{{elseValue}}_* is {{nullable}} or
> (2) *_{{elseValue}}_* is {{{}None{}}}.
>
> The above simplification makes the *_{{elseValue}}_* as {{{}None{}}}.
> Combined with this nullability check makes the *_{{CaseWhen}}_* switch from
> {{non-nullable}} to {{{}nullable{}}}.
> This ticket aims to fix this issue and remain the nullability of the
> expression by replacing the *_elsevalue_* to the value of the TrueLiteral
> expression.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
