Hi Arvid, Very thanks for bring up the discussion! From our side unable to finish the checkpoint is commonly met for online jobs, therefore +1 from my side to implement this. A tiny issue of the FLIP is that the Discussion Thread URL attached seems to be not right.
Best, Yun ------------------------------------------------------------------ From:Arvid Heise <ar...@ververica.com> Send Time:2019 Sep. 30 (Mon.) 20:31 To:dev <dev@flink.apache.org> Subject:[DISCUSS] FLIP-76: Unaligned checkpoints Hi Devs, I would like to start the formal discussion about FLIP-76 [1], which improves the checkpoint latency in systems under backpressure, where a checkpoint can take hours to complete in the worst case. I recommend the thread "checkpointing under backpressure" [2] to get a good idea why users are not satisfied with the current behavior. The key points: - Since the checkpoint barrier flows much slower through the back-pressured channels, the other channels and their upstream operators are effectively blocked during checkpointing. - The checkpoint barrier takes a long time to reach the sinks causing long checkpointing times. A longer checkpointing time in turn means that the checkpoint will be fairly outdated once done. Since a heavily utilized pipeline is inherently more fragile, we may run into a vicious cycle of late checkpoints, crash, recovery to a rather outdated checkpoint, more back pressure, and even later checkpoints, which would result in little to no progress in the application. The FLIP proposes "unaligned checkpoints" which improves the current state, such that - Upstream processes can continue to produce data, even if some operator still waits on a checkpoint barrier on a specific input channel. - Checkpointing times are heavily reduced across the execution graph, even for operators with a single input channel. - End-users will see more progress even in unstable environments as more up-to-date checkpoints will avoid too many recomputations. - Facilitate faster rescaling. The key idea is to allow checkpoint barriers to be forwarded to downstream tasks before the synchronous part of the checkpointing has been conducted (see Fig. 1). To that end, we need to store in-flight data as part of the checkpoint as described in greater details in this FLIP. Although the basic idea was already sketched in [2], we would like get broader feedback in this dedicated mail thread. Best, Arvid [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Checkpointing-under-backpressure-td31616.html