zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r399038839
##########
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTaskNetworkInput.java
##########
@@ -197,6 +205,29 @@ public int getInputIndex() {
return checkpointedInputGate.getAvailableFuture();
}
+ @Override
+ public CompletableFuture<?> prepareSnapshot(long checkpointId) throws
IOException {
+ // Note that if considering recovery in future, we should
guarantee that the spilled buffers in one channel
+ // should be close together because one record might span
multiple buffers.
+ for (int channelIndex = 0; channelIndex <
recordDeserializers.length; channelIndex++) {
+ final InputChannel channel =
checkpointedInputGate.getChannel(channelIndex);
+
+
recordDeserializers[channelIndex].getUnconsumedBuffer().ifPresent(buffer ->
Review comment:
The partial buffer in `RecordDeserializer` should be treated same as the
inflight buffers in `RemoteInputChannel` queue. When the given checkpoint is
triggered, this partial buffer is also overtaken actually. If there are
multiple checkpoints concurrent, this partial buffer might belong to multiple
checkpoint states, then that brings the concern of duplicated persistence like
incremental checkpoint.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
