SEZ9 commented on PR #13239: URL: https://github.com/apache/seatunnel/pull/13239#issuecomment-7024379726
@goutamadwant thanks for the detailed walk-through. The four points line up with what was asked on the `SourceEnumeratorMetrics` / `SourceReaderMetrics` split. - **Completion:** agreed. A reader is `COMPLETED` only after the no-more-splits signal and a full drain, and enumerator `COMPLETED` only covers discovery and assignment. - **Counts:** `inFlightSplitCount` as assigned minus completed is clearer. The `addSplitsBack()` behaviour keyed by split id works, and so does the incremental-only CDC correction, with only `completed` as `UNSUPPORTED`. - **Read path:** one immutable snapshot behind a volatile field is good, and +1 to publishing the enumerator snapshot on its assignment path. One inconsistency is left in the position semantics, as already noted in this thread. You define `activeSplitId` as the split that produced the position, with the two always changing together. But row 3 of the file example switches it to `b.csv` while still showing `a.csv`'s position. Please pick one of these: 1. Keep `a.csv` as the owning split id until the first record from `b.csv` is emitted. 2. Clear both the position and its owner on the switch. Either is fine, as long as a mismatched owner/position pair is never exposed. This is a clarification of the proposal, not a new implementation blocker. Remaining asks: - A completion or idle publication should publish the latest state without producing a new `lastPositionChangeAt`. - Please keep the reassignment example as the basis for the future engine stale-attempt test. - Please push the changes you described: the `inFlightSplitCount` rename, exact `pending` / `assigned` / `inFlight` for incremental-only CDC, the file example using the same sets as `currentUnassignedSplitSize()`, and `lastSourceEventAt` being null when the record has no source time. I'll take another look once those are up. <!-- streview-comment:2465 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
