DanielLeens opened a new pull request, #12257:
URL: https://github.com/apache/seatunnel/pull/12257
### Purpose of this pull request
Closes #12256.
`SinkAggregatedCommitterTask` creates one entry in
`checkpointBarrierCounter` for every checkpoint barrier. When a sink writer
returns `Optional.empty()` from `prepareCommit(checkpointId)`, no writer commit
information is delivered and therefore no entry is added to
`checkpointCommitInfoMap`.
Before this change, `notifyCheckpointComplete()` only removed barrier
counters while iterating `checkpointCommitInfoMap`. A successful empty
checkpoint therefore left its barrier counter retained for the lifetime of the
task. At a five-second checkpoint interval, one affected task retained 17,280
entries per day.
This is the empty-commit residual path not covered by #10189: that fix
removes counters associated with entries present in `checkpointCommitInfoMap`,
while this path has no such entry.
### Changes
* Remove all completed checkpoint IDs from `checkpointBarrierCounter`
independently of commit-info presence.
* Add a regression test covering 10,000 successful checkpoints with no
commit information.
The cleanup uses the same cumulative `<= checkpointId` boundary already used
for commit-info state, so future checkpoint counters remain untouched.
### Verification
Reproduced on `bigdata3` using current `dev` commit
`cf67b549a7a6c35fa0beb12d83c62892427ea919` and OpenJDK 8:
```bash
./mvnw -nsu -Dmaven.gitcommitid.skip=true \
-pl seatunnel-engine/seatunnel-engine-server \
-DskipITs -DskipIT=true \
-Dtest=SinkAggregatedCommitterTaskTest#testCheckpointBarrierCountersAreCleanedWithoutCommitInfo
\
test
```
Before the fix: `Tests run: 1, Failures: 1` because all 10,000 barrier
counters remained.
After the fix: `Tests run: 1, Failures: 0, Errors: 0, Skipped: 0` and `BUILD
SUCCESS`.
Local formatting also passed:
```bash
./mvnw -nsu -Dmaven.gitcommitid.skip=true \
-pl seatunnel-engine/seatunnel-engine-server \
-DskipTests spotless:apply
git diff --check
```
No configuration, dependency, public API, or serialization changes are
included.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
