goutamadwant opened a new pull request, #11123:
URL: https://github.com/apache/seatunnel/pull/11123
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code
changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB
issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the
pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix
typo in README.md doc`.
-->
### Purpose of this pull request
Fixes #11037.
This PR adds GTID based specific startup support for MySQL CDC.
Before this change, `startup.mode = specific` could only start from a binlog
file and position. This change also allows MySQL CDC to start from a configured
GTID set, with optional skip controls for resuming inside a transaction.
Added options:
- `startup.specific-offset.gtid-set`
- `startup.specific-offset.skip-events`
- `startup.specific-offset.skip-rows`
Existing file/position specific startup is still supported.
`startup.specific-offset.gtid-set` is validated separately and cannot be used
together with `startup.specific-offset.file` or `startup.specific-offset.pos`.
The skip event/row values are kept in the same `BinlogOffset` map used by
the checkpoint and restore path, so replay can resume from a more exact
position.
### Does this PR introduce _any_ user-facing change?
Yes.
MySQL CDC users can now configure `startup.mode = specific` with a GTID set
and optional skip controls.
Example:
- `startup.mode = "specific"`
- `startup.specific-offset.gtid-set =
"3E11FA47-71CA-11E1-9E33-C80AA9429562:1-10"`
- `startup.specific-offset.skip-events = 0`
- `startup.specific-offset.skip-rows = 1`
Existing file/position startup config is still supported.
### How was this patch tested?
Added unit coverage for:
- accepting GTID based specific startup config
- preserving existing file/position specific startup behavior
- rejecting mixed GTID and file/position config
- rejecting missing specific offset config
- rejecting malformed GTID values
- rejecting negative skip values
- preserving GTID and skip fields through `IncrementalSplitState`
- comparing GTID offsets by skip events and skip rows
Added one MySQL CDC e2e regression case for GTID startup with `skip-rows`.
Verified locally with:
- `JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH"
./mvnw -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base -DskipITs
-DfailIfNoTests=false test`
- `JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH"
./mvnw -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql -DskipITs
-DfailIfNoTests=false package`
- `JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH"
./mvnw -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e
-DskipTests -DskipITs -DfailIfNoTests=false test-compile`
- `git diff --check`
The Docker-backed e2e method was not run locally because Testcontainers
could not find a valid Docker environment on my machine. The new e2e test
compiles locally and should run in CI.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
