ivoson commented on code in PR #39459: URL: https://github.com/apache/spark/pull/39459#discussion_r1118013777
########## core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala: ########## @@ -728,7 +800,22 @@ class BlockManagerMasterEndpoint( } if (storageLevel.isValid) { + val firstBlock = locations.isEmpty locations.add(blockManagerId) + + blockId.asRDDId.foreach { rddBlockId => + (trackingCacheVisibility, firstBlock) match { + case (true, true) => + // Mark as invisible for the first block. + invisibleRDDBlocks.add(rddBlockId) + case (true, false) if !invisibleRDDBlocks.contains(rddBlockId) => + // If the rdd block is already visible, ask storage manager to update the visibility + // status. + blockManagerInfo(blockManagerId).storageEndpoint + .ask[Unit](MarkRDDBlockAsVisible(rddBlockId)) Review Comment: > So ensure that the replica is also marked as visible, right ? Yes, this is one scenario. Another scenario is that once a task failed fetching the cached block from remote executor, it'll compute and cache the block again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org