[ 
https://issues.apache.org/jira/browse/SPARK-33035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204679#comment-17204679
 ] 

Apache Spark commented on SPARK-33035:
--------------------------------------

User 'maropu' has created a pull request for this issue:
https://github.com/apache/spark/pull/29911

> Updates the obsoleted entries of attribute mapping in 
> QueryPlan#transformUpWithNewOutput
> ----------------------------------------------------------------------------------------
>
>                 Key: SPARK-33035
>                 URL: https://issues.apache.org/jira/browse/SPARK-33035
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.0.1, 3.1.0
>            Reporter: Takeshi Yamamuro
>            Priority: Major
>
> This ticket aims at fixing corner-case bugs in the 
> `QueryPlan#transformUpWithNewOutput` that is used to propagate updated 
> `ExprId`s in a bottom-up way. Let's say we have a rule to simply assign new 
> `ExprId`s in a projection list like this;
> {code}
> case class TestRule extends Rule[LogicalPlan] {
>   override def apply(plan: LogicalPlan): LogicalPlan = 
> plan.transformUpWithNewOutput {
>     case p @ Project(projList, _) =>
>       val newPlan = p.copy(projectList = projList.map { _.transform {
>         // Assigns a new `ExprId` for references
>         case a: AttributeReference => Alias(a, a.name)()
>       }}.asInstanceOf[Seq[NamedExpression]])
>       val attrMapping = p.output.zip(newPlan.output)
>       newPlan -> attrMapping
>   }
> }
> {code}
> Then, this rule is applied into a plan below;
> {code}
> (3) Project [a#5, b#6]
> +- (2) Project [a#5, b#6]
>    +- (1) Project [a#5, b#6]
>       +- LocalRelation <empty>, [a#5, b#6]
> {code}
> In the first transformation, the rule assigns new `ExprId`s in `(1) Project` 
> (e.g., a#5 AS a#7, b#6 AS b#8). In the second transformation, the rule 
> corrects the input references of `(2) Project`  first by using attribute 
> mapping given from `(1) Project` (a#5->a#7 and b#6->b#8) and then assigns new 
> `ExprId`s (e.g., a#7 AS a#9, b#8 AS b#10). But, in the third transformation, 
> the rule fails because it tries to correct the references of `(3) Project` by 
> using incorrect attribute mapping (a#7->a#9 and b#8->b#10) even though the 
> correct one is a#5->a#9 and b#6->b#10. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to