amaechler opened a new pull request, #20651:
URL: https://github.com/apache/druid/pull/20651
Fixes #20604.
### Description
Since #20505, a streaming task's segment publish action waits on the
Overlord until earlier task groups on the same partitions finish. When an
earlier pending-completion group hits its completion timeout,
`checkPendingCompletionTasks` kills it and stops the later groups on the same
partitions, then calls `TaskGroup.onCompleted()` on all of them. That fires the
later task's publish wait with `true`, so the Overlord commits its segments and
offsets while the task is being stopped. The task's publish thread is
interrupted, it never sees the publish succeed, and it does not emit
`ingest/rows/published`.
This is what makes
`KafkaIndexFaultToleranceTest.test_supervisorRecovers_afterCoordinatorRestart`
flaky. The Coordinator restart stalls handoff past the test's 5s completion
timeout. From the server logs of a failing run:
1. Task A (partition 1, offsets 500->712) has published and is waiting for
handoff. Task B (712->1213) is blocked: "needs to wait before publishing as [1]
other taskGroups are currently publishing".
2. `17:46:36,956` A times out and is killed; B is stopped ("one of earlier
groups ... timed out before publishing segments").
3. `17:46:36,963` Overlord: "Updated metadata for supervisor ... from
{p1=712} to {p1=1213}".
4. `17:46:36,966` B: "Failed publish (java.lang.RuntimeException:
java.lang.InterruptedException)".
5. The supervisor resumes from 1213. All rows are ingested, but
`ingest/rows/published` stays 500 short and `verifyAndTearDown` times out after
240s.
Passing runs hit the same completion timeout; they just have no publish in
flight when it fires.
#### Fix
`killTask` and `stopTask(id, false)` now complete the task's pending publish
wait with `false` before shutting it down. The later `set(true)` from
`onCompleted()` is a no-op. `false` maps to the existing
`SegmentPublishResult.retryableFailure("Task is not ready to publish yet")`;
the stop request reaches the task well within the first retry backoff (~1s).
An alternative is a release-time check in `TaskWaitingToPublish` that the
waiting task is still tracked by a task group. I went with the kill/stop hook
because the supervisor already routes every shutdown through these two methods,
and it leaves the existing readiness tests unchanged.
<hr>
##### Key changed/added classes in this PR
* `SeekableStreamSupervisor`
* `SeekableStreamSupervisorStateTest`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
- [x] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
