[ https://issues.apache.org/jira/browse/SPARK-44934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Toth reassigned SPARK-44934: ---------------------------------- Assignee: Wen Yuen Pang > PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called > over CTE with duplicate attributes > -------------------------------------------------------------------------------------------------------------- > > Key: SPARK-44934 > URL: https://issues.apache.org/jira/browse/SPARK-44934 > Project: Spark > Issue Type: Bug > Components: Optimizer > Affects Versions: 3.3.3, 3.4.1 > Reporter: Wen Yuen Pang > Assignee: Wen Yuen Pang > Priority: Minor > Fix For: 3.4.2, 3.5.0, 4.0.0, 3.3.4 > > > When running the query > {code:java} > with cte as ( > select c1, c1, c2, c3 from t where random() > 0 > ) > select cte.c1, cte2.c1, cte.c2, cte2.c3 from > (select c1, c2 from cte) cte > inner join > (select c1, c3 from cte) cte2 > on cte.c1 = cte2.c1 {code} > > The query fails with the error > {code:java} > org.apache.spark.scheduler.DAGScheduler: Failed to update accumulator 9523 > (Unknown class) for task 1 > org.apache.spark.SparkException: attempted to access non-existent accumulator > 9523{code} > Further investigation shows that the rule > PushdownPredicatesAndPruneColumnsForCTEDef creates an invalid plan when the > output of a CTE contains duplicate expression IDs. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org