gaoyajun02 created SPARK-36815:
----------------------------------

             Summary: Found duplicate rewrite attributes
                 Key: SPARK-36815
                 URL: https://issues.apache.org/jira/browse/SPARK-36815
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.0.2
            Reporter: gaoyajun02
             Fix For: 3.0.2


We are using Spark version 3.0.2 in production and some ETLs contain 
multi-level CETs and the following error occurs when we join them.
{code:java}
java.lang.AssertionError: assertion failed: Found duplicate rewrite attributes 
at scala.Predef$.assert(Predef.scala:223) at 
org.apache.spark.sql.catalyst.plans.QueryPlan.rewrite$1(QueryPlan.scala:207) at 
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$transformUpWithNewOutput$1(QueryPlan.scala:193)
 at 
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:405)
 at 
org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:243)
 at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:403)
{code}
I reproduced the problem with a simplified SQL as follows:
{code:java}
-- SQL
with
a as ( select name, get_json_object(json, '$.id') id, n from (
    select get_json_object(json, '$.name') name, json from values 
('{"name":"a", "id": 1}' ) people(json)
    ) LATERAL VIEW explode(array(1, 1, 2)) num as n ),
b as ( select a1.name, a1.id, a1.n from a a1 left join (select name, count(1) c 
from a group by name) a2 on a1.name = a2.name)
select b1.name, b1.n, b1.id from b b1 join b b2 on b1.name = b2.name;{code}
In debugging I found that a reference to the root Project existed in both 
subqueries, and when `ResolveReferences` resolved the conflict, `rewrite` 
occurred in both subqueries, containing two new attrMapping, and they were both 
eventually passed to the root Project, leading to this error

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to