SEZ9 commented on PR #13239:
URL: https://github.com/apache/seatunnel/pull/13239#issuecomment-7024379726

   @goutamadwant thanks for the detailed walk-through. The four points line up 
with what was asked on the `SourceEnumeratorMetrics` / `SourceReaderMetrics` 
split.
   
   - **Completion:** agreed. A reader is `COMPLETED` only after the 
no-more-splits signal and a full drain, and enumerator `COMPLETED` only covers 
discovery and assignment.
   - **Counts:** `inFlightSplitCount` as assigned minus completed is clearer. 
The `addSplitsBack()` behaviour keyed by split id works, and so does the 
incremental-only CDC correction, with only `completed` as `UNSUPPORTED`.
   - **Read path:** one immutable snapshot behind a volatile field is good, and 
+1 to publishing the enumerator snapshot on its assignment path.
   
   One inconsistency is left in the position semantics, as already noted in 
this thread. You define `activeSplitId` as the split that produced the 
position, with the two always changing together. But row 3 of the file example 
switches it to `b.csv` while still showing `a.csv`'s position. Please pick one 
of these:
   
   1. Keep `a.csv` as the owning split id until the first record from `b.csv` 
is emitted.
   2. Clear both the position and its owner on the switch.
   
   Either is fine, as long as a mismatched owner/position pair is never 
exposed. This is a clarification of the proposal, not a new implementation 
blocker.
   
   Remaining asks:
   
   - A completion or idle publication should publish the latest state without 
producing a new `lastPositionChangeAt`.
   - Please keep the reassignment example as the basis for the future engine 
stale-attempt test.
   - Please push the changes you described: the `inFlightSplitCount` rename, 
exact `pending` / `assigned` / `inFlight` for incremental-only CDC, the file 
example using the same sets as `currentUnassignedSplitSize()`, and 
`lastSourceEventAt` being null when the record has no source time.
   
   I'll take another look once those are up.
   
   <!-- streview-comment:2465 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to