Ngone51 commented on a change in pull request #32007: URL: https://github.com/apache/spark/pull/32007#discussion_r625477570
########## File path: core/src/main/scala/org/apache/spark/storage/BlockId.scala ########## @@ -87,6 +87,29 @@ case class ShufflePushBlockId(shuffleId: Int, mapIndex: Int, reduceId: Int) exte override def name: String = "shufflePush_" + shuffleId + "_" + mapIndex + "_" + reduceId } +@DeveloperApi +case class ShuffleMergedBlockId(appId: String, shuffleId: Int, reduceId: Int) extends BlockId { + override def name: String = "mergedShuffle_" + appId + "_" + shuffleId + "_" + reduceId + ".data" +} + +@DeveloperApi +case class ShuffleMergedIndexBlockId( + appId: String, + shuffleId: Int, + reduceId: Int) extends BlockId { + override def name: String = + "mergedShuffle_" + appId + "_" + shuffleId + "_" + reduceId + ".index" Review comment: > If we are moving the delete to the executor, then we could formulate it to make the change minimal right ? Yes, and thanks for bringing the detailed steps, which looks good. BTW, there's another tricky way to ship the merge directory that is appending the merge directory into `ExecutorShuffleInfo.localDirs`. Also, adding a new RPC doesn't seem to help a lot here according to the discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org