goutamadwant opened a new pull request, #11512:
URL: https://github.com/apache/seatunnel/pull/11512
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code
changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB
issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the
pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix
typo in README.md doc`.
-->
### Purpose of this pull request
Implements the first runtime reporting slice from STIP-30 in #11364 and
addresses the lifecycle concerns raised in #11395.
This change:
- adds experimental reader and enumerator CDC progress report contracts
- tracks the current consumed position from the record emission path
- keeps current, completed-checkpoint, and restored positions as separate
values
- reports checkpoint and restore positions as unsupported until their
lifecycle callbacks are wired
- associates snapshot splits and watermarks using the same split ID
- distinguishes assigned, completed, running, prepared, and not-yet-chunked
snapshot work
- collects reader and enumerator progress through the engine runtime
- retains only the latest report for each source task and execution attempt
- rejects stale sequences and reports from superseded execution attempts
- removes stored progress when a pipeline is cleaned up
- adds Hazelcast serialization support for the progress reporting operation
This PR does not expose CDC progress through REST or change checkpoint and
restore behavior. Those remain separate implementation slices under STIP-30.
### Does this PR introduce _any_ user-facing change?
Yes.
It introduces experimental CDC progress provider and report contracts and
wires their collection into the runtime. The progress is not exposed through
REST or the CLI in this PR, so existing job configuration and output remain
unchanged.
### How was this patch tested?
Added focused coverage for:
- progress model validation and immutable position values
- current offset updates during record emission
- the `CATCH_UP` to `INCREMENTAL` lifecycle transition
- snapshot progress with multiple splits and lazy chunking
- split ID and watermark association
- stale sequence and execution-attempt rejection
- source and task isolation
- pipeline cleanup
- reader and enumerator operation serialization
Verified the focused tests with:
```shell
./mvnw -q \
-pl
seatunnel-api,connector-cdc-base,connector-cdc-mysql,seatunnel-engine-server \
-Dskip.ui=true \
-DfailIfNoTests=false \
-Dtest=CdcProgressModelTest,CdcReaderProgressTrackerTest,SnapshotSplitAssignerTest,CdcProgressServiceTest,ReportCdcProgressOperationSerializationTest
\
test
```
Verified the affected API and CDC modules with:
```shell
./mvnw -q \
-pl seatunnel-api,connector-cdc-base,connector-cdc-mysql \
-Dskip.ui=true \
-DfailIfNoTests=false \
package
```
Verified the full Java build with:
```shell
./mvnw -q -DskipTests -Dskip.ui=true verify
```
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
No new connector, binary dependency, plugin mapping, or incompatible
configuration change is introduced, so the connector packaging items above are
not applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
