Hi, Zakelly.
This FLIP LGTM overall, and I think it makes sense to speed up checkpointing.
After reading the entire FLIP, I have the following questions.
1. Since declareProcess is introduced in `Input` and `TwoInputStreamOperator`
to
align with `processElement`, do we also need to consider modifying other
clazzes
such as `KeyedProcessFunction` to introduce similar methods?
2. Besides `processElement`, there are other methods that access state, such as
`snapshotState`. Are these methods also not applicable under this FLIP?
3. For chain-style declarations, can they be nested? For example, can another
finished
declared chain1 be used within a declared chain2?
--
Best！
Xuyang
At 2024-07-30 14:06:08, "Zakelly Lan" <[email protected]> wrote:
>Hi devs,
>
>I would like to initiate a discussion about FLIP-455: Declare async state
>processing and checkpoint the in-flight requests[1].
>
>FLIP-423[2] and the related sub-FLIPs introduced the disaggregated state
>and async accessing model of state. However, the in-flight state requests
>(or records) should be drained at the checkpoint, which leads to an
>increased checkpoint synchronization delay. The main goal of this FLIP is
>to snapshot the in-flight requests as part of a checkpoint. This will
>accelerate the draining process and decouple the mutual impact between
>checkpoints and data processing.
>
>For your ease of understanding, the FLIP covers three key aspects:
>1. (Public API) Additional APIs for conditional branchings under
>`StateFuture`
>2. (Internal API) An internal API set for declaring and defining the record
>processing
>3. (Implementation) Method to snapshot the in-flight state requests
>For the first two parts, there is also a PoC branch[3] provided.
>
>Looking forward to hearing from you.
>
>[1] https://cwiki.apache.org/confluence/x/C4owEg
>[2] https://cwiki.apache.org/confluence/x/R4p3EQ
>[3] https://github.com/apache/flink/pull/24719
>
>
>Best,
>Zakelly
