Kaivalya Apte created FLINK-41442:
-------------------------------------
Summary: Unaligned checkpoint trigger overtakes queued
AddSplitEvent, splits lost on restore
Key: FLINK-41442
URL: https://issues.apache.org/jira/browse/FLINK-41442
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 2.1.3, 2.2.1, 2.3.0, 2.1.2, 2.1.1, 2.2.0, 2.1.0
Environment: Unaligned checkpoints enabled
Reporter: Kaivalya Apte
h3. *Problem*
Since FLINK-35051, \{{StreamTask#triggerCheckpointAsync}} sends an unaligned
checkpoint trigger as an *urgent* mail. Urgent mails run before all queued
normal mails. Operator events such as \{{AddSplitEvent}} are normal mails
(\{{StreamTask#dispatchOperatorEvent}}).
{\{OperatorCoordinatorHolder#completeCheckpointOnceEventsAreDone}} sends the
trigger only after the task acknowledges all earlier events. The task
acknowledges an event when the event is queued, not when it is processed. So
the trigger can run before a queued \{{AddSplitEvent}}:
* The coordinator checkpoint records the split as assigned.
* The reader checkpoint does not contain the split.
After a restore from that checkpoint, no reader owns the split. The job stays
RUNNING with no error and never reads the split again. Later checkpoints keep
the same state, so the loss is permanent.
h3. *Conditions*
* Unaligned checkpoints, for example
\{{execution.checkpointing.unaligned.enabled=true}} with
\{{aligned-checkpoint-timeout=0}}.
* A checkpoint triggers while split assignments are queued on a busy task,
usually the first checkpoint after job start.
* The job restores from exactly that checkpoint (failover or rescale).
We observed this on several jobs. Logs show "Assigning splits" before
"Triggering checkpoint 1", and "Adding split(s) to reader" before "Completed
checkpoint 1". After the restore, the lost sources restore 0 splits.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
