Sxnan commented on PR #138:
URL: https://github.com/apache/flink-agents/pull/138#issuecomment-3310183353
@letaoj
> How's the filtering logic would work in this case? The comment
>
> ```
> // Subsequent steps {@link #tryResumeProcessActionTasks} will then
filter out keys that do
> // not belong to the key range of current task.
> ```
>
> indicate that the tryResumeProcessActionTasks will do the filtering but i
did not see any explicit filtering happens in that method. Can you help me
understand a bit more over there? If we are using the unionlist and access all
key range, then during recovery, we could potentially let all consumer to read
from all the partition which could incur read amplification.
You are right that it is not explicitly filtered. But in the
`ActionExecutionOperator#processActionTaskForKey` method, it will try to pull
from the `actionTasksKState`, which is a keyed state. If it is not found, then
it means that the current subtask is not responsible for this key, so it will
just skip processing that key.
> Since I'm new to the flink state management. During crash recovery, do we
still do checkpoint during crash recover or we only start new checkpoint once
the recover completed?
From the Flink perspective, recovery is completed after all the states in
the checkpoint have been restored, and then it starts processing the incoming
data as if it were arriving for the first time. A checkpoint can happen anytime
during data processing, depending on the checkpoint interval. Therefore, a
checkpoint can definitely happen during recovery from the action state store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
