davidzollo opened a new issue, #11354: URL: https://github.com/apache/seatunnel/issues/11354
## Background SeaTunnel CDC already handles multiple source families and multiple offset models, such as LSN, SCN, resolved timestamps, commit timestamps, and connector-specific split/restore state. However, SeaTunnel still does not provide a unified user-facing **CDC lag observability view**. That gap makes CDC diagnosis unnecessarily difficult. Users often need to reason from connector-specific logs or internal offsets instead of a consistent product-level lag model. This becomes especially painful when diagnosing restore/resume correctness, transaction-ordering problems, or uneven source catch-up behavior. ## Problem to solve Today, different CDC connectors expose different notions of progress, and SeaTunnel lacks a standard way to answer questions like: - how far behind is this CDC job right now - is the source currently blocked in snapshot or incremental phase - is lag growing because upstream commit progress is faster than SeaTunnel consumption - did restore resume from the expected logical position after upgrade or failover - are multiple tables or splits progressing unevenly Bugs such as the TiDB CDC resolved-ts problem show that these questions are not theoretical; they directly affect correctness and diagnosis quality. ## Proposed scope Add a unified CDC lag observability feature across connectors. Suggested first-delivery scope: - define a normalized CDC lag/progress model that can represent connector-specific offsets without hiding important raw details - expose both normalized progress and connector-native raw position fields where useful - distinguish snapshot progress from incremental lag - surface split/table-level skew when it is operationally meaningful - provide a user-facing view and/or API that lets operators compare current progress, recent movement, and stalled state - document which connectors fully support the model and which fields are best-effort initially ## Design challenges This is a cross-cutting feature. It touches: - connector-specific offset/state models - restore/resume semantics - public observability/API contracts - UI presentation of mixed raw and normalized progress If this is designed loosely, SeaTunnel risks exposing a misleading lag number that looks universal but means different things per connector. ## STIP requirement before implementation Because this feature affects CDC correctness diagnostics and introduces a new cross-connector public contract, **the claimant should submit a STIP design first and get maintainer agreement before starting implementation**. The STIP should explicitly define: - the normalized CDC lag/progress model - required vs optional fields for each connector family - how snapshot progress and incremental lag are represented separately - how restore/resume progress is validated after restart or upgrade - how UI and docs should explain connector-specific limitations without implying a false universal precision ## Acceptance criteria - SeaTunnel defines a documented CDC lag/progress model instead of leaving each connector to expose unrelated signals. - At least a representative subset of CDC connectors implements the first version. - Users can tell whether a job is in snapshot, catch-up, or steady incremental consumption. - Connector limitations are documented explicitly. - English and Chinese docs are updated. ## Non-goals for the first version - forcing every connector to expose identical raw offset fields - pretending all databases support the same exact lag semantics - replacing connector logs for deep debugging in the first delivery ## Related work - Issue #11013 - PR #11113 - PR #10780 - PR #10799 - Issue #11239 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
