[ https://issues.apache.org/jira/browse/SPARK-28845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuanjian Li resolved SPARK-28845. --------------------------------- Resolution: Won't Do After further investigation, I found the objective of performing sort only for the retried indeterminate stage is unable to achieve. That will break our assumption for the `outputDeterministicLevel` for each RDD, which should be defined when the job submitted. While here we expected the output deterministic level depends on the stage attempt number. As the SPARK-25341 completed, the current behavior depends on the config `spark.sql.execution.sortBeforeRepartition`. Spark will sort before repartition, and rerun the failed tasks when setting the config to true. On the contrary, the whole stage will rerun. > Enable spark.sql.execution.sortBeforeRepartition only for retried stages > ------------------------------------------------------------------------ > > Key: SPARK-28845 > URL: https://issues.apache.org/jira/browse/SPARK-28845 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL > Affects Versions: 3.0.0 > Reporter: Yuanjian Li > Priority: Major > > For fixing the correctness bug of SPARK-28699, we disable radix sortĀ for the > scenario of repartition in Spark SQL. This will cause a performance > regression. > So for limiting the performance overhead, we'll do the optimizing work by > only enable sort for the repartition operation while stage retries happening. > This work depends on SPARK-25341. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org