SEZ9 commented on PR #13239:
URL: https://github.com/apache/seatunnel/pull/13239#issuecomment-6842832328

   Thanks for the work so far. I agree with the direction @nzw921rx laid out; 
let me consolidate the asks so there is a single clear set before further 
implementation work.
   
   **Scope and naming.** The progress contract should not be limited to CDC. 
Batch and streaming sources own the same kinds of facts (split discovery, 
creation, assignment and known completion on the enumerator side; actual read 
progress, current processing state and consumed positions on the reader side), 
so let's shape this as a general Source-level contract around two abstractions: 
`SourceEnumeratorMetrics` and `SourceReaderMetrics`. CDC snapshot/incremental 
details should be concrete implementations of those, not the basis of the 
public API.
   
   **Surface area.** The current adoption path (`SupportCdcProgress`, 
`CdcProgressProvider<R>`, the empty `CdcProgressReport` interface, plus several 
supporting model types) is too fragmented for a first contract. Concretely:
   
   - Reconsider the empty `CdcProgressReport` interface: please explain what it 
enables beyond acting as a generic bound, and whether it is necessary.
   - Keep `SupportCdcProgress` only if the engine needs to identify this 
capability during planning, before Readers/Enumerators exist. If it is needed, 
document how it differs from the runtime provider.
   - Keep the two report types, each containing only facts owned by that 
component (the enumerator should not infer reader state and vice versa).
   - Keep the initial set of wrappers and states minimal, each tied to a 
concrete reporting requirement. More can be added once later implementations 
show the need.
   
   **What I'd like to see next, as a proposal before more code:**
   
   1. A simplified API sketch using `SourceEnumeratorMetrics` / 
`SourceReaderMetrics`.
   2. One Batch example and one CDC example showing what a connector author 
implements.
   3. The corresponding engine-side collection example showing what the engine 
consumes and how it does so without blocking processing.
   4. A short note on how unsupported or temporarily unavailable information is 
represented, and which progress semantics are common vs. 
implementation-specific.
   
   Exact method signatures and type parameters can fall out of those examples; 
the priority is settling responsibilities and extension points first. Once we 
agree on that smaller surface, the connector and engine implementations can be 
built on top of it. Happy to iterate here in the thread.
   
   <!-- streview-comment:2314 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to