jerrypeng commented on code in PR #57286:
URL: https://github.com/apache/spark/pull/57286#discussion_r3591322401
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala:
##########
@@ -300,7 +300,10 @@ object ShuffleExchangeExec {
// corner-cases where a partitioner constructed with `numPartitions`
partitions may output
// fewer partitions (like RangePartitioner, for example).
val conf = SparkEnv.get.conf
- val shuffleManager = SparkEnv.get.shuffleManager
+ // This decision concerns the default (regular) shuffle path only. A
pipelined shuffle is served
+ // by a separate incremental manager (see SparkEnv.shuffleManagerFor) and
does not go through
+ // here, so inspect the default manager's type directly.
+ val shuffleManager = SparkEnv.get.defaultShuffleManager
Review Comment:
Why does the defaultShuffleManager need to be used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
