mridulm commented on pull request #35076:
URL: https://github.com/apache/spark/pull/35076#issuecomment-1008181443
One possibility I was thinking of was if we calling `finalizePartition`
before any chunks have been fully written out for a reducer.
Can you test with this change ?
```
diff --git
a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
index d0eb4aed65..0d3a3c7448 100644
---
a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
+++
b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
@@ -575,17 +575,23 @@ public class RemoteBlockPushResolver implements
MergedShuffleFileManager {
List<Long> sizes = new ArrayList<>(shuffleMergePartitions.size());
for (AppShufflePartitionInfo partition:
shuffleMergePartitions.values()) {
synchronized (partition) {
+ boolean shouldDelete = false;
try {
// This can throw IOException which will marks this shuffle
partition as not merged.
partition.finalizePartition();
- bitmaps.add(partition.mapTracker);
- reduceIds.add(partition.reduceId);
- sizes.add(partition.getLastChunkOffset());
+ long size = partition.getLastChunkOffset();
+ if (size > 0) {
+ bitmaps.add(partition.mapTracker);
+ reduceIds.add(partition.reduceId);
+ sizes.add(size);
+ } else {
+ shouldDelete = true;
+ }
} catch (IOException ioe) {
logger.warn("Exception while finalizing shuffle partition {}_{}
{} {}", msg.appId,
msg.appAttemptId, msg.shuffleId, partition.reduceId, ioe);
} finally {
- partition.closeAllFilesAndDeleteIfNeeded(false);
+ partition.closeAllFilesAndDeleteIfNeeded(shouldDelete);
}
}
}
```
+CC @otterc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
