otterc commented on a change in pull request #32140:
URL: https://github.com/apache/spark/pull/32140#discussion_r640312469
##########
File path:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##########
@@ -108,13 +115,6 @@ final class ShuffleBlockFetcherIterator(
private[this] val startTimeNs = System.nanoTime()
- /** Local blocks to fetch, excluding zero-sized blocks. */
- private[this] val localBlocks =
scala.collection.mutable.LinkedHashSet[(BlockId, Int)]()
-
- /** Host local blockIds to fetch by executors, excluding zero-sized blocks.
*/
- private[this] val hostLocalBlocksByExecutor =
- LinkedHashMap[BlockManagerId, Seq[(BlockId, Long, Int)]]()
-
Review comment:
Note to reviewers: Both of these are created locally in the `initialize`
and `fetchFallbackBlocks`. With push-based shuffle,
`partitionBlocksByFetchMode` can be called multiple times because failure to
fetch merged shuffle blocks/chunks (fallback) finds original blocks that made
up the failed merged block/chunk and then we need to create new FetchRequests
from these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
