[ https://issues.apache.org/jira/browse/SPARK-33260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-33260. ----------------------------------- Fix Version/s: 3.0.2 3.1.0 Resolution: Fixed Issue resolved by pull request 30160 [https://github.com/apache/spark/pull/30160] > SortExec produces incorrect results if sortOrder is a Stream > ------------------------------------------------------------ > > Key: SPARK-33260 > URL: https://issues.apache.org/jira/browse/SPARK-33260 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.0.0, 3.0.1 > Reporter: Ankur Dave > Assignee: Ankur Dave > Priority: Major > Fix For: 3.1.0, 3.0.2 > > > The following query produces incorrect results. The query has two essential > features: (1) it contains a string aggregate, resulting in a {{SortExec}} > node, and (2) it contains a duplicate grouping key, causing > {{RemoveRepetitionFromGroupExpressions}} to produce a sort order stored as a > Stream. > SELECT bigint_col_1, bigint_col_9, MAX(CAST(bigint_col_1 AS string)) > FROM table_4 > GROUP BY bigint_col_1, bigint_col_9, bigint_col_9 > When the sort order is stored as a {{Stream}}, the line > {{ordering.map(_.child.genCode(ctx))}} in > {{GenerateOrdering#createOrderKeys()}} produces unpredictable side effects to > {{ctx}}. This is because {{genCode(ctx)}} modifies {{ctx}}. When {{ordering}} > is a {{Stream}}, the modifications will not happen immediately as intended, > but will instead occur lazily when the returned {{Stream}} is used later. > Similar bugs have occurred at least three times in the past: > https://issues.apache.org/jira/browse/SPARK-24500, > https://issues.apache.org/jira/browse/SPARK-25767, > https://issues.apache.org/jira/browse/SPARK-26680. > The fix is to check if {{ordering}} is a {{Stream}} and force the > modifications to happen immediately if so. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org