mridulm commented on pull request #35076: URL: https://github.com/apache/spark/pull/35076#issuecomment-1008181443
One possibility I was thinking of was if we calling `finalizePartition` before any chunks have been fully written out for a reducer. Can you test with this change ? ``` diff --git a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java index d0eb4aed65..0d3a3c7448 100644 --- a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java +++ b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java @@ -575,17 +575,23 @@ public class RemoteBlockPushResolver implements MergedShuffleFileManager { List<Long> sizes = new ArrayList<>(shuffleMergePartitions.size()); for (AppShufflePartitionInfo partition: shuffleMergePartitions.values()) { synchronized (partition) { + boolean shouldDelete = false; try { // This can throw IOException which will marks this shuffle partition as not merged. partition.finalizePartition(); - bitmaps.add(partition.mapTracker); - reduceIds.add(partition.reduceId); - sizes.add(partition.getLastChunkOffset()); + long size = partition.getLastChunkOffset(); + if (size > 0) { + bitmaps.add(partition.mapTracker); + reduceIds.add(partition.reduceId); + sizes.add(size); + } else { + shouldDelete = true; + } } catch (IOException ioe) { logger.warn("Exception while finalizing shuffle partition {}_{} {} {}", msg.appId, msg.appAttemptId, msg.shuffleId, partition.reduceId, ioe); } finally { - partition.closeAllFilesAndDeleteIfNeeded(false); + partition.closeAllFilesAndDeleteIfNeeded(shouldDelete); } } } ``` +CC @otterc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org