HeartSaVioR commented on pull request #31355: URL: https://github.com/apache/spark/pull/31355#issuecomment-768801016
> AQE won't kick in if users specify num partitions, e.g. df.repartition(5), I think the same applies here if the sink requires a certain num partitions. The case I imagine is like this, `df.repartition(5).write.(blabla).save()`, with the condition sink requires a specific distribution but doesn't require specific number of partitions. Now Spark would repartition to default number of shuffle partitions, and I'm unsure changing it to None (SPARK-34230) would keep the user intention. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
