FMX commented on code in PR #3132:
URL: https://github.com/apache/celeborn/pull/3132#discussion_r1993048246
##########
client/src/main/java/org/apache/celeborn/client/read/PartitionReader.java:
##########
@@ -31,4 +33,8 @@ public interface PartitionReader {
void close();
PartitionLocation getLocation();
+
+ Optional<PartitionReaderCheckpointMetadata>
getPartitionReaderCheckpointMetadata();
Review Comment:
You can add a chunk index in the method of `next` to make sure that you can
record the chunk that has been fully consumed.
##########
client/src/main/java/org/apache/celeborn/client/read/DfsPartitionReader.java:
##########
@@ -238,7 +257,7 @@ public ByteBuf next() throws IOException,
InterruptedException {
break;
}
}
- results.put(Unpooled.wrappedBuffer(buffer));
+ results.put(Pair.of(currentChunkIndex,
Unpooled.wrappedBuffer(buffer)));
Review Comment:
A chunk that has been returned does not mean that the chunk is read fully.
IMO, I think you should return the chunk index with the chunk and record the
chunk index as fully read after the celeborn input stream consumed the chunk
fully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
