[
https://issues.apache.org/jira/browse/FLINK-25322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17733588#comment-17733588
]
Feifan Wang commented on FLINK-25322:
-------------------------------------
Hi [~ym] , [~ym] , I saw that this issue has not been progressed for a long
time, I think it is very important to support the no-claim mode, and no-claim
mode is the default mode is a proof. If you don't mind, can you assign this
ticket to me ?
As for the implementation, I have some different ideas. The current
implementation method of no-claim mode is to make full snapshot in the first
checkpoint after the job recovers from the retained snapshot, but there will be
some problems when the Log based checkpoint make full snapshot:
# State changelog data cannot be re-uploaded to durable storage, so if we want
to make a full snapshot, we must either copy the changelog file in the retained
checkpoint, or force a full materialization at the first checkpoint. It’s okay
if the storage of changelog supports fast copy, but if it doesn’t, the time
overhead of copying these changelog files will be high, causing checkpoint
timeout.
# If we choose to force a full materialization at the first checkpoint, we
will need to upload a large amount of data, which will also easily cause the
checkpoint timeout. At the same time, the materialization operations that were
originally executed on a staggered peak on each subtask will be executed
simultaneously, which will increase the pressure of the checkpoint storage and
lead to a longer materialization time.
The no-claim mode has two requirements: First, the new job cannot delete any
files in the restored snapshot, otherwise cannot start multiple jobs from the
same snapshot. Second, the new job's checkpoint cannot reference the file of
the restored checkpoint, because in this mode The ownership of the restored
snapshot belongs to the user, and the user may delete the snapshot later. We
can call the ability of new jobs to run completely free from restored snapshots
as "{*}state self-sustained{*}". When we implement the no-claim mode by forcing
the first checkpoint to be a full checkpoint, the job enters the
state-sustained state after the first checkpoint is completed. Please note that
new job are not state-sustained as soon as they enter the running state. *I
think that in the no-claim mode, it is not necessary to enter the
state-sustained state at the first checkpoint, but to enter the state-sustained
state as soon as possible.*
*Proposal：*
Based on the above explanation, I have the following proposal:
*1. ChangelogStateBackend only implements no-claim mode by forcing the first
materialization to be a full materialization*
# Since the changelog data before the materialization trigger is not required
in the checkpoint after the materialization is completed, once all keyed state
backends completed a fully materialization, the job no longer depends on the
restored snapshot file.
# Materialization is still performed asynchronously, so the longer
materialization time caused by full upload will not cause checkpoint timeout.
# In general, when log based checkpoint is enabled, the materialization
interval is roughly the same as the checkpoint interval when log based
checkpoint is not enabled. Therefore, the time for log based checkpoint to
enter the state self-sustained state in this way will not be longer than that
without state changelog.
*2. Show whether job is state-sustained in flink UI and provide a rest api to
query for query that.*
Before this, the user needs to check whether the job has completed at least one
checkpoint to determine whether the restored checkpoint can be deleted, which
is not straightforward enough. I think a flag in flink UI and a dedicated rest
api is better. SharedStateRegistry can be used to track whether the checkpoint
refers to the restored no-claim snapshot, and then know whether the job is
state-sustained.
WDYT [~dwysakowicz], [~ym], [~pnowojski], [~roman] ?
> Support no-claim mode in changelog state backend
> ------------------------------------------------
>
> Key: FLINK-25322
> URL: https://issues.apache.org/jira/browse/FLINK-25322
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing, Runtime / State Backends
> Reporter: Dawid Wysakowicz
> Assignee: Yuan Mei
> Priority: Major
> Fix For: 1.18.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
