Anubhav-Roy opened a new pull request, #20299:
URL: https://github.com/apache/druid/pull/20299
Fixes #20295.
### Description
Under a large pending-task backlog with saturated workers, the Overlord's
`httpRemote`
task runner stalls: one pending-task-runner thread holds `statusLock` while
deep in
worker-snapshot reconstruction, blocking new task submission, status
updates, and
worker-sync operations cluster-wide. Restarting doesn't help because the
active task set is
reloaded from metadata, the backlog reappears, and the loop re-enters the
same
lock-holding scan.
#### Fixed the Overlord stall on a large pending-task backlog
In `HttpRemoteTaskRunner.pendingTasksExecutionLoop()`, the loop holds the
single
`statusLock` while iterating every pending task, and for each task calls
`findWorkerToRunTask(Task)`, which rebuilds a full immutable snapshot of all
workers via
`getWorkersEligibleToRunTasks()`. This makes the loop cost
`O(pendingTasks × workers × tasksAnnouncedPerWorker)`, while it holds the
lock.
This change computes the snapshot **once per pass**,
inside `synchronized (statusLock)` before iterating, and passes it into a
new overload
`findWorkerToRunTask(Task, ImmutableMap<String, ImmutableWorkerInfo>
eligibleWorkers)`.
The existing `findWorkerToRunTask(Task)` is retained and now delegates to
the overload,
so no other call site changes behavior.
This reduces per-pass cost to `O(workers × tasksPerWorker)` with no
behavioral change:
worker selection within a pass is identical because the input snapshot is
identical to
what each per-task call would have recomputed.
#### Release note
Fixed an issue where the Overlord using the `httpRemote` task runner could
stall for
extended periods (holding `statusLock`) when a large backlog of pending tasks
accumulated while workers were saturated, blocking task submission and
status updates.
<hr>
##### Key changed/added classes in this PR
* `HttpRemoteTaskRunner`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md).
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
