AHeise commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r408104278
##########
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/CheckpointedInputGate.java
##########
@@ -171,6 +176,18 @@ else if (bufferOrEvent.getEvent().getClass() ==
CancelCheckpointMarker.class) {
}
}
+ public List<Buffer> requestInflightBuffers(long checkpointId, int
channelIndex) throws IOException {
+ if (((CheckpointBarrierUnaligner)
barrierHandler).hasInflightData(checkpointId, channelIndex)) {
+ return
inputGate.getChannel(channelIndex).requestInflightBuffers(checkpointId);
Review comment:
`CheckpointBarrierUnaligner#hasInflightData` is only updated in task thread.
So while calling `requestInflightBuffers` (on gate/channel) from task thread,
the state cannot change.
At this point, `receivedBuffers` of input channel can also not be drained by
task thread, so we are not losing any buffer. However, new buffers could be
added by netty thread causing duplicate buffers.
To guard against this case, `RemoteInputChannel#requestInflightBuffers`
first acquires lock and copies the buffers. It then updates
`lastRequestedCheckpointId`, so that all subsequently added `receivedBuffers`
should go through `BufferReceivedListener`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
