Rui Wang created SPARK-44082: -------------------------------- Summary: Generate operator does not update reference set properly Key: SPARK-44082 URL: https://issues.apache.org/jira/browse/SPARK-44082 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.5.0 Reporter: Rui Wang Assignee: Rui Wang
Before ``` == Optimized Logical Plan == Project [col1#2, col2#19] +- Generate replicaterows(sum#17L, col1#2, col2#3), [2], false, [col1#2, col2#3] +- Filter (isnotnull(sum#17L) AND (sum#17L > 0)) +- Aggregate [col1#2, col2#19], [col1#2, col2#19, sum(vcol#14L) AS sum#17L] +- Union false, false :- Aggregate [col1#2], [1 AS vcol#14L, col1#2, first(col2#3, false) AS col2#19] : +- LogicalRDD [col1#2, col2#3], false +- Project [-1 AS vcol#15L, col1#8, col2#9] +- LogicalRDD [col1#8, col2#9], false ``` Couldn't find col2#3 in [col1#2,col2#19,sum#17L] after ``` == Optimized Logical Plan == Project [col1#2, col2#19] +- Generate replicaterows(sum#17L, col1#2, col2#19), [2], false, [col1#2, col2#19] +- Filter (isnotnull(sum#17L) AND (sum#17L > 0)) +- Aggregate [col1#2, col2#19], [col1#2, col2#19, sum(vcol#14L) AS sum#17L] +- Union false, false :- Aggregate [col1#2], [1 AS vcol#14L, col1#2, first(col2#3, false) AS col2#19] : +- LogicalRDD [col1#2, col2#3], false +- Project [-1 AS vcol#15L, col1#8, col2#9] +- LogicalRDD [col1#8, col2#9], false ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org