[
https://issues.apache.org/jira/browse/FLINK-40656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Fang updated FLINK-40656:
----------------------------------
Description:
h2. Description
A failed schema change can leave the distributed `SchemaCoordinator` in
`EVOLVING`. During global failover, the coordinator instance is reused, but
`resetToCheckpoint()` restores only the schema manager. Replayed schema
requests then fail with `Unexpected evolving status: EVOLVING`, exhausting the
restart budget even after the sink becomes available again.
h2. Steps to reproduce
# 1. Start a Fluss-to-Fluss CDC pipeline and complete a checkpoint.
# 2. Stop the sink Fluss Coordinator, add `extra_v2 STRING` to the source
table, and write rows using the new column.
# 3. Let the failed sink schema update trigger failover, then restart the sink
Fluss Coordinator.
# 4. Schema requests are replayed, but the CDC job repeatedly fails with
`Unexpected evolving status: EVOLVING`; the target column and subsequent data
remain unsynchronized.
h2. Expected behavior
Checkpoint restore should discard in-flight schema evolution state and allow
schema requests to be replayed successfully after the sink recovers. The
previous schema-change worker must exit before restored state is installed or a
new metadata-apply call starts.
h2. Proposed fix
- Quiesce previous coordinator work at the shared reset boundary, then restore
durable schema state and reinitialize transient protocol state in both
coordinator implementations.
- Treat null checkpoint state as an empty coordinator state, and reject
requests belonging to a reset attempt.
The same-instance coordinator regression reproduces the exception on main
commit `b11d6fdf6346520e720396f25e5b5162ce5435b9`. This is distinct from
FLINK-37578 (successful evolution response ordering) and FLINK-37713
(preserving restored schemas during initial startup).
was:
## Description
A failed schema change can leave the distributed `SchemaCoordinator` in
`EVOLVING`. During global failover, the coordinator instance is reused, but
`resetToCheckpoint()` restores only the schema manager. Replayed schema
requests then fail with `Unexpected evolving status: EVOLVING`, exhausting the
restart budget even after the sink becomes available again.
## Steps to reproduce
1. Start a Fluss-to-Fluss CDC pipeline and complete a checkpoint.
2. Stop the sink Fluss Coordinator, add `extra_v2 STRING` to the source table,
and write rows using the new column.
3. Let the failed sink schema update trigger failover, then restart the sink
Fluss Coordinator.
4. Schema requests are replayed, but the CDC job repeatedly fails with
`Unexpected evolving status: EVOLVING`; the target column and subsequent data
remain unsynchronized.
## Expected behavior
Checkpoint restore should discard in-flight schema evolution state and allow
schema requests to be replayed successfully after the sink recovers. The
previous schema-change worker must exit before restored state is installed or a
new metadata-apply call starts.
## Proposed fix
- Quiesce previous coordinator work at the shared reset boundary, then restore
durable schema state and reinitialize transient protocol state in both
coordinator implementations.
- Treat null checkpoint state as an empty coordinator state, and reject
requests belonging to a reset attempt.
The same-instance coordinator regression reproduces the exception on main
commit `b11d6fdf6346520e720396f25e5b5162ce5435b9`. This is distinct from
FLINK-37578 (successful evolution response ordering) and FLINK-37713
(preserving restored schemas during initial startup).
> Distributed schema coordinator can remain EVOLVING after checkpoint restore
> ---------------------------------------------------------------------------
>
> Key: FLINK-40656
> URL: https://issues.apache.org/jira/browse/FLINK-40656
> Project: Flink
> Issue Type: Bug
> Reporter: Xiaobing Fang
> Priority: Major
>
> h2. Description
> A failed schema change can leave the distributed `SchemaCoordinator` in
> `EVOLVING`. During global failover, the coordinator instance is reused, but
> `resetToCheckpoint()` restores only the schema manager. Replayed schema
> requests then fail with `Unexpected evolving status: EVOLVING`, exhausting
> the restart budget even after the sink becomes available again.
> h2. Steps to reproduce
> # 1. Start a Fluss-to-Fluss CDC pipeline and complete a checkpoint.
> # 2. Stop the sink Fluss Coordinator, add `extra_v2 STRING` to the source
> table, and write rows using the new column.
> # 3. Let the failed sink schema update trigger failover, then restart the
> sink Fluss Coordinator.
> # 4. Schema requests are replayed, but the CDC job repeatedly fails with
> `Unexpected evolving status: EVOLVING`; the target column and subsequent data
> remain unsynchronized.
> h2. Expected behavior
> Checkpoint restore should discard in-flight schema evolution state and allow
> schema requests to be replayed successfully after the sink recovers. The
> previous schema-change worker must exit before restored state is installed or
> a new metadata-apply call starts.
> h2. Proposed fix
> - Quiesce previous coordinator work at the shared reset boundary, then
> restore durable schema state and reinitialize transient protocol state in
> both coordinator implementations.
> - Treat null checkpoint state as an empty coordinator state, and reject
> requests belonging to a reset attempt.
> The same-instance coordinator regression reproduces the exception on main
> commit `b11d6fdf6346520e720396f25e5b5162ce5435b9`. This is distinct from
> FLINK-37578 (successful evolution response ordering) and FLINK-37713
> (preserving restored schemas during initial startup).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
