raghavyadav01 opened a new pull request, #20039:
URL: https://github.com/apache/pinot/pull/20039
## Problem
`maxSubtaskWaitTimeMs` does not measure how long a subtask has been queued.
For every subtask that has not started, `getTableTaskStatusSummary` reports the
age of the parent job:
```java
if (state == null && jobStartTime > 0) {
long waitingTimeMillis = (currentTime - jobStartTime);
```
So the gauge grows with job length. A job of 1,000 subtasks that minions
drain steadily for five hours reports a five-hour wait for its last subtasks,
the same value as a job whose queue has not moved in five hours. An alert on
this gauge (say, wait > 4h) fires on every large backfill or merge job, and
raising the threshold per cluster only moves the problem to the next larger job.
## Change
A new per-table, per-task-type gauge, `maxSubtaskQueueStallTimeMs`: for a
table that still has waiting subtasks, the time since any subtask of its job
last started or finished (the job start time when none has yet). It stays small
while a job keeps draining, however long the job runs, and grows only when the
queue stops moving, for example when no minion picks up the job's subtasks. It
is 0 for a table with nothing waiting, so an alert on it self-resolves like the
existing gauges.
- `PinotHelixTaskResourceManager.getTableTaskStatusSummary` tracks the
latest partition start or finish time of the job and sets
`TaskStatusSummary.queueStallTimeMs` on tables with waiting subtasks.
- `TaskMetricsEmitter` emits the per-table maximum across jobs as
`ControllerGauge.MAX_SUBTASK_QUEUE_STALL_TIME_MS` and removes it with the other
table gauges.
- The controller JMX exporter config exports it next to
`maxSubtaskWaitTimeMs`.
`maxSubtaskWaitTimeMs` is unchanged, so existing dashboards and alerts keep
working.
## Tests
-
`PinotHelixTaskResourceManagerTest.testGetTableTaskStatusSummaryQueueStallTime`
(new): a six-hour-old job whose last subtask finished a minute ago reports a 6h
wait but a ~1 min stall for the table with a waiting subtask, and 0 for a table
with none.
- `testGetTableTaskStatusSummaryWithTiming`: the stall is measured from the
last subtask start, not the job start.
- `TaskMetricsEmitterTest`: the gauge is emitted per table (1000 ms for the
table with a waiting subtask, 0 for the other) and the metric counts include it.
`TaskMetricsEmitterTest` (11) and `PinotHelixTaskResourceManagerTest` (58)
pass; checkstyle and license checks pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
