cloud-fan commented on a change in pull request #26434: [SPARK-29544] [SQL]
optimize skewed partition based on data size
URL: https://github.com/apache/spark/pull/26434#discussion_r364211160
##########
File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala
##########
@@ -343,15 +343,17 @@ private[spark] abstract class MapOutputTracker(conf:
SparkConf) extends Logging
/**
* Called from executors to get the server URIs and output sizes for each
shuffle block that
* needs to be read from a given range of map output partitions
(startPartition is included but
- * endPartition is excluded from the range) and is produced by a specific
mapper.
+ * endPartition is excluded from the range) and is produced by
+ * a range of mappers (startMapId, endMapId, startMapId is included and the
endMapId is excluded).
Review comment:
let's use mapIndex consistently. There are still many mapId in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
