Anubhav-Roy opened a new issue, #20295:
URL: https://github.com/apache/druid/issues/20295
### Description
`HttpRemoteTaskRunner.pendingTasksExecutionLoop()` holds the single
`statusLock` monitor while iterating `pendingTaskIds`, and for **every**
pending task it calls `findWorkerToRunTask(Task)`, which rebuilds a full
immutable snapshot of all workers:
```java
// findWorkerToRunTask(Task)
return strategy.findWorkerForTask(
config,
ImmutableMap.copyOf(getWorkersEligibleToRunTasks()), // rebuilt on
every call
task
);
```
`getWorkersEligibleToRunTasks()` filters and transforms the whole `workers`
map, and each `WorkerHolder.toImmutable()` reconstructs that worker's
announced-task set via `ImmutableWorkerInfo.fromWorkerAnnouncements(...)`.
```
O(pendingTasks × workers × tasksAnnouncedPerWorker)
```
…and the entire pass is executed while holding `statusLock`.
At small backlogs this is invisible. Under a large pending backlog with the
cluster at/near capacity, a single loop pass holds `statusLock` for many
seconds to minutes. Because `statusLock` is also required by `run()` (new task
submission), `taskComplete()` / status updates, and the worker-sync path, the
Overlord effectively freezes.
Restarting the Overlord does not recover it: the active task set is
persisted in metadata and reloaded via `syncFromStorage` on startup, so
`pendingTaskIds` is immediately large again and the loop re-enters the same
lock-holding scan.
Observed on Druid 33.0.0 (`httpRemote` task runner).
Thread-dump signature at the stall:
- One `hrtr-pending-tasks-runner-*` thread is `RUNNABLE`, holding
`statusLock`, deep in `ImmutableWorkerInfo.fromWorkerAnnouncements` →
`WorkerHolder.toImmutable` → `getWorkersEligibleToRunTasks` →
`findWorkerToRunTask` → `pendingTasksExecutionLoop`.
- The other pending-task-runner threads are idle in `statusLock.wait()`.
- `TaskQueue-Manager` is `BLOCKED` on the same monitor in
`HttpRemoteTaskRunner.run()`, while holding the `TaskQueue` giant lock.
- Many Jetty `qtp-*` handler threads are parked on the `TaskQueue` lock in
`OverlordResource.taskPost → TaskQueue.add`.
### Motivation
**Use case:** any Overlord using the `httpRemote` task runner that can
accumulate a large pending-task backlog while workers are saturated
**Why the change is beneficial:**
- No behavior change for correctness: within a single synchronized pass the
loop reserves at most one task (it `break`s right after
`workersWithUnacknowledgedTask.putIfAbsent`), so the eligible-worker set is
invariant across the inner loop — recomputing it per task produces identical
results.
**Proposed fix:**
**Compute the eligible-worker snapshot once per pass**, not once per pending
task. Build it just inside `synchronized (statusLock)` before iterating
`pendingTaskIds`, and pass it into an overload `findWorkerToRunTask(Task,
ImmutableMap<String, ImmutableWorkerInfo> eligibleWorkers)`. This drops the
dominant `fromWorkerAnnouncements` rebuild cost from O(pendingTasks × workers ×
tasksPerWorker) to O(workers × tasksPerWorker) per pass.
Sketch:
```java
synchronized (statusLock) {
final ImmutableMap<String, ImmutableWorkerInfo> eligibleWorkers =
ImmutableMap.copyOf(getWorkersEligibleToRunTasks()); // once per pass
// fast path: nothing can be placed this pass
boolean anyFreeCapacity =
eligibleWorkers.values().stream().anyMatch(w ->
w.getAvailableCapacity() > 0);
if (!anyFreeCapacity) {
statusLock.wait(TimeUnit.MINUTES.toMillis(1));
continue;
}
Iterator<String> iter = pendingTaskIds.iterator();
while (iter.hasNext()) {
...
immutableWorker = findWorkerToRunTask(ti.getTask(), eligibleWorkers);
// no per-task rebuild
...
}
...
}
```
### Affected Version
Reproduced on 33.0.0; code path unchanged on `master`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
