[
https://issues.apache.org/jira/browse/FLINK-37368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu reassigned FLINK-37368:
----------------------------------
Assignee: yux
> MySQL Pipeline job failed to checkpoint after closing idle readers
> ------------------------------------------------------------------
>
> Key: FLINK-37368
> URL: https://issues.apache.org/jira/browse/FLINK-37368
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.3.0
> Reporter: yux
> Assignee: yux
> Priority: Major
>
> Currently, PreTransformOperator uses `getUnionListState` to store upstream
> schema maps. However, with MySQL `scan.incremental.close-idle-reader.enabled`
> option, most subTasks will be closed (goes into "finished" state) after
> snapshot stage finishes, which would block all later checkpoints:
> {code:java}
> /**
> * If a job vertex using {@code UnionListState} has part of tasks FINISHED
> where others are
> * still in RUNNING state, the checkpoint would be aborted since it might
> cause incomplete
> * {@code UnionListState}.
> */
> private void checkNoPartlyFinishedVertexUsedUnionListState(
> Map<JobVertexID, ExecutionJobVertex> partlyFinishedVertex,
> Map<OperatorID, OperatorState> operatorStates) {
> for (ExecutionJobVertex vertex : partlyFinishedVertex.values()) {
> if (hasUsedUnionListState(vertex, operatorStates)) {
> throw new PartialFinishingNotSupportedByStateException(
> String.format(
> "The vertex %s (id = %s) has used"
> + " UnionListState, but part of its tasks
> are FINISHED.",
> vertex.getName(), vertex.getJobVertexId()));
> }
> }
> } {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
