SEZ9 commented on PR #13239: URL: https://github.com/apache/seatunnel/pull/13239#issuecomment-6842832328
Thanks for the work so far. I agree with the direction @nzw921rx laid out; let me consolidate the asks so there is a single clear set before further implementation work. **Scope and naming.** The progress contract should not be limited to CDC. Batch and streaming sources own the same kinds of facts (split discovery, creation, assignment and known completion on the enumerator side; actual read progress, current processing state and consumed positions on the reader side), so let's shape this as a general Source-level contract around two abstractions: `SourceEnumeratorMetrics` and `SourceReaderMetrics`. CDC snapshot/incremental details should be concrete implementations of those, not the basis of the public API. **Surface area.** The current adoption path (`SupportCdcProgress`, `CdcProgressProvider<R>`, the empty `CdcProgressReport` interface, plus several supporting model types) is too fragmented for a first contract. Concretely: - Reconsider the empty `CdcProgressReport` interface: please explain what it enables beyond acting as a generic bound, and whether it is necessary. - Keep `SupportCdcProgress` only if the engine needs to identify this capability during planning, before Readers/Enumerators exist. If it is needed, document how it differs from the runtime provider. - Keep the two report types, each containing only facts owned by that component (the enumerator should not infer reader state and vice versa). - Keep the initial set of wrappers and states minimal, each tied to a concrete reporting requirement. More can be added once later implementations show the need. **What I'd like to see next, as a proposal before more code:** 1. A simplified API sketch using `SourceEnumeratorMetrics` / `SourceReaderMetrics`. 2. One Batch example and one CDC example showing what a connector author implements. 3. The corresponding engine-side collection example showing what the engine consumes and how it does so without blocking processing. 4. A short note on how unsupported or temporarily unavailable information is represented, and which progress semantics are common vs. implementation-specific. Exact method signatures and type parameters can fall out of those examples; the priority is settling responsibilities and extension points first. Once we agree on that smaller surface, the connector and engine implementations can be built on top of it. Happy to iterate here in the thread. <!-- streview-comment:2314 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
