li3zhi4 opened a new pull request, #11618:
URL: https://github.com/apache/seatunnel/pull/11618
## Why
`stop.mode = "specific"` (and by the same code path `"latest"`) is
documented but never works for MySQL CDC:
the incremental split is read by `MySqlBinlogFetchTask`, which creates a
plain Debezium
`MySqlStreamingChangeEventSource` with **no stopOffset check**, so the job
runs forever even after
the configured stop position has been passed. Previously reported in #6789
and #7400; both were
auto-closed by the stale bot without a fix.
This PR implements bounded read for the MySQL incremental split so the job
terminates at the exact
configured binlog offset, plus two related correctness fixes found while
debugging it.
Closes #11617
## What changed
### 1. Bounded read for incremental split (`stop.mode = "specific"`)
- `connector-cdc-base/.../source/offset/Offset.java`
- Add `isNeverStop()` default implementation (subclasses override with
their unbounded sentinel).
-
`connector-cdc-mysql/.../source/reader/fetch/binlog/MySqlBinlogFetchTask.java`
- Add `BoundedMySqlStreamingChangeEventSource` inner class: checks the
split's stop offset on
every event, ends the binlog read when the target position is reached,
and logs binlog progress
periodically. The unbounded (`stop.mode = "never"`) path is unchanged.
- `connector-cdc-base/.../source/reader/IncrementalSourceReader.java`
- Allow incremental splits to finish normally (previously only snapshot
splits could finish).
-
`connector-cdc-base/.../source/reader/external/IncrementalSourceStreamFetcher.java`
- `pollSplitRecords()` returns `null` to signal split completion **only
for bounded reads**
(`!stopOffset.isNeverStop()`), so unbounded jobs can never be falsely
marked FINISHED.
### 2. Fix `BinlogOffset.compareTo()` when GTID sets are mixed
- `connector-cdc-mysql/.../source/offset/BinlogOffset.java`
- Correct the GTID fallback comparison so offsets with mixed/partial GTID
sets compare correctly
(needed by both start-offset and stop-offset comparisons).
### 3. Fix `taskStarted` race in `IncrementalSourceStreamFetcher`
- Set `taskStarted`/`executing` synchronously when submitting the task,
closing a race window that
could make a bounded-read job finish prematurely with silently missing
data (affects
`stop.mode = "specific"` and `"latest"`, not `"never"`).
### 4. Minor: correct `stop.mode` descriptions
- Oracle / SQL Server CDC option descriptions updated to match the actual
supported value `"never"`.
## Verification
- Manual test with `stop.mode = "specific"` (start `mysql-bin.059734:4`,
stop `mysql-bin.059818:4`)
on a production-like table: job now terminates with `FINISHED` exactly at
the stop position
(previously ran forever).
- `stop.mode = "never"` (default) regression: unbounded job keeps running;
split-completion `null`
path is not triggered.
- `mvn install -pl connector-cdc/connector-cdc-mysql -am -DskipTests` passes.
## Checklist
- [x] Code changes complete and verified (ported from 2.3.13 custom branch,
rebased onto dev layout)
- [x] Related issue referenced: #11617
- [x] Compiles with `mvn install -DskipTests` for the CDC module chain
- [ ] `spotless:check` (full CI will verify)
- [ ] Unit test for `BinlogOffset.compareTo()` mixed-GTID cases (follow-up)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
