gaoyajun02 created SPARK-36815:
----------------------------------
Summary: Found duplicate rewrite attributes
Key: SPARK-36815
URL: https://issues.apache.org/jira/browse/SPARK-36815
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.2
Reporter: gaoyajun02
Fix For: 3.0.2
We are using Spark version 3.0.2 in production and some ETLs contain
multi-level CETs and the following error occurs when we join them.
{code:java}
java.lang.AssertionError: assertion failed: Found duplicate rewrite attributes
at scala.Predef$.assert(Predef.scala:223) at
org.apache.spark.sql.catalyst.plans.QueryPlan.rewrite$1(QueryPlan.scala:207) at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$transformUpWithNewOutput$1(QueryPlan.scala:193)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:405)
at
org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:243)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:403)
{code}
I reproduced the problem with a simplified SQL as follows:
{code:java}
-- SQL
with
a as ( select name, get_json_object(json, '$.id') id, n from (
select get_json_object(json, '$.name') name, json from values
('{"name":"a", "id": 1}' ) people(json)
) LATERAL VIEW explode(array(1, 1, 2)) num as n ),
b as ( select a1.name, a1.id, a1.n from a a1 left join (select name, count(1) c
from a group by name) a2 on a1.name = a2.name)
select b1.name, b1.n, b1.id from b b1 join b b2 on b1.name = b2.name;{code}
In debugging I found that a reference to the root Project existed in both
subqueries, and when `ResolveReferences` resolved the conflict, `rewrite`
occurred in both subqueries, containing two new attrMapping, and they were both
eventually passed to the root Project, leading to this error
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
