nzw921rx commented on PR #13239: URL: https://github.com/apache/seatunnel/pull/13239#issuecomment-6977672333
> [@nzw921rx](https://github.com/nzw921rx) here is the proposal, using `SourceReaderMetrics` and `SourceEnumeratorMetrics`, with a batch example and a CDC example. No implementation changes until this is settled; I have converted the PR to draft. > > ### Summary > A Source-level progress contract with two report types and no new interfaces. A reader or enumerator that can describe its progress returns an immutable snapshot from one default method; everything else keeps returning `null`. Batch and CDC sources use the same two shapes; what differs is the values they put in them. CDC is the first implementation; one batch source is used below to show the shape is not CDC-specific. > > Public surface (all `@Experimental`, package `org.apache.seatunnel.api.source`): > > Type Kind Responsibility > `SourceReader#getReaderMetrics()` default method, returns `null` Reader-owned facts, pulled by the engine > `SourceSplitEnumerator#getEnumeratorMetrics()` default method, returns `null` Enumerator-owned facts, pulled by the engine > `SourceReaderMetrics` final immutable class Phase, active split, consumed position, timestamps, bounded details > `SourceEnumeratorMetrics` final immutable class Phase, split counts, bounded details > `MetricValue<T>` + `MetricAccuracy` final class + enum A value and how much it can be trusted > `SourcePosition` final immutable class Connector-native coordinates under a named family > Five types and two default methods. Compared with [#13239](https://github.com/apache/seatunnel/pull/13239) as pushed: no `SupportCdcProgress` marker, no `CdcProgressProvider`, no empty `CdcProgressReport`, no per-split detail list, no CDC-specific enums in the common contract. > > ### 1. What each component maintains and exposes > ### Reader > public final class SourceReaderMetrics { > String phase; // see §2 > String activeSplitId; // nullable before the first split > MetricValue<SourcePosition> position; // latest consumed position > long lastPositionChangeAt; // epoch millis, 0 if never > Long lastSourceEventAt; // source event time when known, else null > Map<String, String> details; // connector-specific, bounded, opaque to the engine > } > The reader reports what it has observed, not what has been checkpointed. The engine adds task identity, attempt, sequence and observation time when it collects. > > ### Enumerator > public final class SourceEnumeratorMetrics { > String phase; // see §2 > MetricValue<Integer> pendingSplitCount; // created, not yet assigned > MetricValue<Integer> assignedSplitCount; // handed to a reader (running + completed) > MetricValue<Integer> runningSplitCount; // assigned, completion not yet known > MetricValue<Integer> completedSplitCount; // assigned and reported complete > MetricValue<Integer> unsplitUnitCount; // discovered tables/files/partitions not yet split > Map<String, String> details; > } > The enumerator reports only what it knows. Many batch enumerators never learn that a split finished; they report `completedSplitCount` and `runningSplitCount` as `UNSUPPORTED` instead of guessing. > > Both classes validate in the constructor: present counts are non-negative; `assigned == completed + running` is enforced only when all three are `EXACT`; `details` is copied and capped (proposed: 32 entries, 256 characters per key and value); `phase` and `position` are non-null. > > ### 2. Common semantics versus implementation-specific > Common (fixed by the contract) Implementation-specific (carried as values) > The two shapes above and their field meanings Which `phase` strings a source uses beyond the common ones > `MetricAccuracy` and the rule that `EXACT` is only claimed when runtime state proves it The `SourcePosition` family and its keys (`MYSQL_BINLOG`, `POSTGRES_LSN`, `FILE`, ...) > Common phases: `UNKNOWN` (no split yet), `READING`, `COMPLETED` (no more splits and no more elements) for readers; `DISCOVERING`, `ASSIGNING`, `COMPLETED`, `NOT_APPLICABLE` for enumerators Finer reader phases: CDC uses `SNAPSHOT`, `CATCH_UP`, `INCREMENTAL` in place of `READING` > Non-blocking, thread-safe, immutable, truthful provider guarantees (§3, §4) Anything in `details` > Phases are strings with the common values as constants on the two classes. A consumer that does not know a connector-specific phase treats it as `READING`. An enum would be safer for consumers but would put CDC-specific values into a Source-level type or force a second enum per connector family; the string keeps the common type small. If the committers prefer an enum for the common phases plus a free-form `detailPhase`, that is a mechanical change. > > Position families are strings too, for the same reason. The contract fixes only that a position carries a family, a schema version and string coordinates, that payloads never contain credentials or connection details, and that consumers compare positions within one family only. > > ### 3. Unsupported or temporarily unavailable information > Every count and the position are a `MetricValue<T>`: > > Accuracy Payload Meaning > `EXACT` non-null Current runtime state proves the value > `BEST_EFFORT` non-null Useful for diagnosis, precision not guaranteed > `UNSUPPORTED` null This source cannot provide the value > `UNAVAILABLE` null Supported, but not available for this observation > A missing value is never replaced by zero or a guess. Nullable scalars (`activeSplitId`, `lastSourceEventAt`) are plain nullables because "unknown" is their only non-value. Returning `null` from the default method means "no report yet"; the engine does not record anything. > > ### 4. How the engine collects without blocking processing > Unchanged from [#11512](https://github.com/apache/seatunnel/pull/11512), with names updated: > > * Both methods return a snapshot the component already maintains. They never perform source, > network, checkpoint or other blocking I/O and never take a lock that record processing or split > assignment holds. Implementations publish a new immutable instance on change (volatile field); > the engine reads the field. > * On each worker, `TaskExecutionService` samples `getReaderMetrics()` on the existing > `print-execution-info-interval` cadence, wraps non-null results in an envelope (task location, > source vertex, execution attempt, attempt-local sequence, observed-at) and sends one batch per > worker to the coordinator. A throwing implementation is isolated per task and logged by class > name only. > * The coordinator asks each worker that hosts a split enumerator for `getEnumeratorMetrics()` on > the same cadence, at most one request per worker in flight. Enumerators returning `null` add > nothing to the batch. > * The coordinator keeps the latest report per owner and task; lower attempts and non-increasing > sequences are ignored; reports are removed with their pipeline. > * Wire format: one fixed schema per report type, enums and phases by name, bounded entry and > field counts. Because the shapes are fixed, the engine never deserializes connector classes. > > Nothing is exposed through REST, metrics or the CLI in this phase. > > ### 5. Batch example: Local file source > Enumerator (`FileSourceSplitEnumerator` knows the file list and the assigned set; it is not told when a reader finishes a file): > > @Override > public SourceEnumeratorMetrics getEnumeratorMetrics() { > int total = filePaths.size(); > int assigned = assignedSplit.size(); > return new SourceEnumeratorMetrics( > assigned == total ? SourceEnumeratorMetrics.PHASE_COMPLETED > : SourceEnumeratorMetrics.PHASE_ASSIGNING, > MetricValue.exact(total - assigned), // pending > MetricValue.exact(assigned), // assigned > MetricValue.unsupported(), // running: enumerator is not told > MetricValue.unsupported(), // completed: enumerator is not told > MetricValue.exact(0), // unsplit units: files are the splits > Collections.emptyMap()); > } > Reader (one file at a time; rows emitted since the file was opened are exact): > > @Override > public SourceReaderMetrics getReaderMetrics() { > return latest; // volatile; rebuilt when a file is opened, every N rows, and on completion > } > > private void refreshMetrics(String path, long rowsRead, boolean done) { > Map<String, String> coords = new LinkedHashMap<>(); > coords.put("path", path); > coords.put("rows", Long.toString(rowsRead)); > latest = new SourceReaderMetrics( > done ? SourceReaderMetrics.PHASE_COMPLETED : SourceReaderMetrics.PHASE_READING, > path, > MetricValue.exact(new SourcePosition("FILE", 1, coords)), > System.currentTimeMillis(), > null, > Collections.emptyMap()); > } > What this proves: the shapes hold without any CDC concept; a batch enumerator can be honest about not knowing completion; a batch reader's position is a split-local offset rather than a stream offset, and `SourcePosition` carries both. > > ### 6. CDC example: `connector-cdc-base` > Enumerator (`SnapshotSplitAssigner` already keeps an immutable count snapshot; the incremental assigner has no snapshot phase): > > Startup mode Enumerator phase pending / assigned / running / completed / unsplit > `initial`, `snapshot-only` `DISCOVERING` → `ASSIGNING` → `COMPLETED` all `EXACT` while counts are consistent, `BEST_EFFORT` otherwise; `unsplit` = tables not yet chunked > `earliest`, `latest`, `specific`, `timestamp`, `committed-offset` `NOT_APPLICABLE` all `UNSUPPORTED` > Reader (`CdcReaderProgressTracker` derives the phase from the split it is processing): > > Startup mode Reader phase sequence position before → after the first record > `initial` `UNKNOWN` → `SNAPSHOT` → `CATCH_UP` → `INCREMENTAL`; idle readers → `COMPLETED` `UNAVAILABLE` → `BEST_EFFORT` (min snapshot watermark) → `EXACT` > `snapshot-only` `UNKNOWN` → `SNAPSHOT` → `COMPLETED` `UNAVAILABLE` throughout > incremental-only modes `UNKNOWN` → `INCREMENTAL` `BEST_EFFORT` (configured start) → `EXACT` > `SNAPSHOT`, `CATCH_UP` and `INCREMENTAL` are CDC's refinement of `READING`. Position families: `MYSQL_BINLOG` (file, pos, gtids), others use the plugin name until a family is defined. The `details` map stays empty in v1. > > ### 7. Scope and compatibility > * Default methods: no existing connector or engine changes behaviour. A connector that does not > override them is simply not observed. > * CDC (`connector-cdc-base`, MySQL family) is the only implementation in this STIP. The file > example above is a design proof; implementing batch connectors is separate work, if wanted. > * The contract stays `@Experimental` until at least one more source family (batch or a second CDC > dialect such as PostgreSQL) has adopted it without changing the shapes. > * Naming: the types carry structured progress, not numeric counters, so they do not go through > `MetricsContext`; the doc will state that distinction. If `SourceReaderProgress` / > `SourceEnumeratorProgress` reads clearer alongside `MetricsContext`, the rename is cheap now and > expensive later. > * STIP-30 ([[STIP-30][Feature][CDC] : Unified CDC lag and progress observability model #11364](https://github.com/apache/seatunnel/issues/11364)) becomes "CDC as the first implementation of the Source progress contract"; > its startup-mode semantics, accuracy rules and engine collection design are unchanged. > > ### 8. Changes to the open PRs if this is accepted > * [[Feature][API] Add experimental CDC progress contract #13239](https://github.com/apache/seatunnel/pull/13239) (API): rename and move the types, drop the two provider interfaces in favour of the > default methods, replace the CDC lifecycle and assignment enums with phase constants, add the > bounded `details` map, add the batch example to the doc. > * [[Feature][CDC] Report reader and enumerator progress from connector-cdc-base #13240](https://github.com/apache/seatunnel/pull/13240) (connector): adopt the default methods; phase strings instead of enums. > * [[Feature][Zeta] Collect CDC progress reports on the coordinator #11512](https://github.com/apache/seatunnel/pull/11512) (engine): rename, codec gains the `details` block, otherwise unchanged. > * [[Test][E2E] Validate MySQL CDC progress reporting #12137](https://github.com/apache/seatunnel/pull/12137) (MySQL E2E) follows after the three merge. I support keeping `SourceReaderMetrics` and `SourceEnumeratorMetrics` as two experimental contracts. Before implementing them, I’d like to clarify four behaviors that matter for both File and CDC sources in SeaTunnel. 1. **Define completion from each component’s point of view.** A File reader can finish one file and move on to the next. It should report `COMPLETED` only after it has received the no-more-splits signal and drained its assigned splits. The enumerator can finish discovery and assignment while readers are still working. CDC phases such as `SNAPSHOT`, `CATCH_UP`, and `INCREMENTAL` can provide additional detail about what a reader is doing. 2. **Count splits using the state the enumerator owns.** The File example should use the same split set as `currentUnassignedSplitSize()`. Counting `filePaths` can overstate the total when paths produce duplicate split IDs. Please also define how `addSplitsBack()` affects each count so that reassigning a split does not count it twice. If the enumerator only knows that a split was assigned and has no completion report, `inFlightSplitCount` describes that observation more precisely than `runningSplitCount`. Incremental-only CDC still assigns splits, so its pending and in-flight counts can be meaningful too. 3. **Keep each position tied to a specific split.** The proposed “latest consumed position” is useful. For a reader handling multiple splits, `activeSplitId` should identify the split that produced the reported position. Please clarify whether consuming a heartbeat or a filtered event advances it. Zeta can expose the native coordinates as supplied by the connector. Checkpoint state defines the recovery position. 4. **Keep progress reporting cheap on the read path.** A connector should be able to update lightweight local state and publish a bounded snapshot periodically. The sampling thread can then read it without locking the reader or traversing the enumerator’s mutable collections. `lastPositionChangeAt` should advance only when the position actually changes. This lets us distinguish an idle source from a freshly sampled one. Could you update the File example to show one reader consuming two splits, and the MySQL CDC example to show an incremental split being returned and reassigned? Those two cases would make the phase, count, and position semantics concrete enough to review before adding the Zeta integration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
