Zakelly Lan created FLINK-36118:
-----------------------------------
Summary: FLIP-455: Declare async state processing and checkpoint
the in-flight requests
Key: FLINK-36118
URL: https://issues.apache.org/jira/browse/FLINK-36118
Project: Flink
Issue Type: Improvement
Components: Runtime / Checkpointing, Runtime / State Backends
Reporter: Zakelly Lan
Assignee: Zakelly Lan
The FLIP: https://cwiki.apache.org/confluence/x/C4owEg
FLIP-423 introduced the disaggregated state management and the FLIP-425
introduced the new execution model of asynchronous state access in an
event-driven way. This model has the potential to significantly boost
performance by leveraging parallel I/O operations. However, it does lead to
increased draining times during checkpoints, presenting a trade-off between
system throughput and checkpoint synchronization delay. This balance can be
calibrated through adjusting the buffer size. As a follow-up FLIP for FLIP-425,
this FLIP proposes a faster way of checkpoint by snapshot state requests that
are waiting in the buffer of "Asynchronous Execution Controller (AEC)" as part
of the checkpoint. By this approach, we expect only a great optimization for
the draining time overhead compared with the original plan in FLIP-425,
especially under a high back-pressure scenario. To achieve the snapshot of
state requests, the callbacks from user should be persisted across job
attempts. This FLIP introduces a novel approach for declaring element
processing where all callbacks are re-declared and bound to the corresponding
previous state requests during the operator's initialization phase. This
ensures that the entire pipeline can be accurately restored and operations can
resume smoothly after a job restart.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
