[GitHub] [spark] ulysses-you commented on pull request #32468: [SPARK-35335][SQL] Improve CoalesceShufflePartitions to avoid generating small files

ulysses-you commented on pull request #32468: URL: https://github.com/apache/spark/pull/32468#issuecomment-840233602 Some random thoughts. We considered about supporting stage level config completly that means for every query stage we can use it's own config. Some options: *

[GitHub] [spark] ulysses-you commented on pull request #32468: [SPARK-35335][SQL] Improve CoalesceShufflePartitions to avoid generating small files

ulysses-you commented on pull request #32468: URL: https://github.com/apache/spark/pull/32468#issuecomment-840227088 Thank you for introducing this idea. But that's not only about `spark.sql.adaptive.coalescePartitions.minPartitionNum ` but also other adaptive configs, such as