aho135 opened a new pull request, #20387:
URL: https://github.com/apache/druid/pull/20387
### Description
In **bounded (backfill) mode** with `replicas > 1`, the supervisor spins up
an endless stream of tasks that complete instantly for partitions/groups that
have already finished their backfill range.
#### Root cause
`SeekableStreamSupervisor.createNewTasks()` has two places that create tasks:
1. **Task-group recreation** (for groups no longer in
`activelyReadingTaskGroups`) — this path is already guarded with
`hasTaskGroupReachedBoundedEnd(groupId)` and correctly skips completed bounded
groups.
2. **Replica top-up** ("make sure each task group has the desired number of
replica tasks") — this path had **no** bounded-completion guard.
With `replicas > 1`, once a group's committed offsets have reached the
configured bounded end, replicas that finish and exit drop the group below the
configured replica count. The top-up loop then keeps creating replacement
replicas whose start offset is already at the bounded end (`start == end`), so
each one reads nothing, completes immediately, drops the group below the
replica count again, and re-triggers the top-up — churning tasks forever.
Meanwhile genuinely lagging partitions continue reading normally.
Observed as a flood of `index_kafka_*` tasks whose `startSequenceNumbers ==
endSequenceNumbers` for an already-completed partition.
#### Fix
Guard the replica top-up loop with the same
`hasTaskGroupReachedBoundedEnd()` check already used by the recreation path, so
completed bounded groups are not topped up.
This only affects bounded mode (`ioConfig.isBounded()`); streaming
supervisors are unchanged.
### Testing
Added
`KafkaSupervisorTest#testBoundedStreamConfig_completedGroup_doesNotTopUpReplicas`:
a bounded supervisor with `replicas = 2` and a group whose metadata offsets
have reached the bounded end must not create replacement replica tasks.
> Note: I was unable to compile/run the test locally — `master` now targets
Java 25 and my environment only has JDK 17/21. Relying on CI to validate.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors. (n/a —
internal behavior fix)
- [x] a release note entry in the PR description. (n/a — bugfix)
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
