gortiz commented on PR #19364:
URL: https://github.com/apache/pinot/pull/19364#issuecomment-5425372045
Thanks both — I've pushed a rework that takes all of this. Rebased on master
so it now sits on top of #19365, which matters here: before that fix a mailbox
send's `executionTimeMs` was short by its entire final block, so a max over
workers would have been measuring the wrong thing.
**`maxClockTimeMs` (@Jackie-Jiang, @yashmayya)** — added, and named for
direct comparison with `clockTimeMs`: the same measure, taken on the worker
that took longest instead of averaged across all of them. No division involved,
since it is already a single worker's time. This is the stat that actually
answers the original problem; `clockTimeMs` greatly below it means the average
is understating the stage.
**`NON_ACTIVE_WORKERS` instead of `ACTIVE_WORKERS` (@Jackie-Jiang)** — done,
and it works out better than just being quieter. `StatMap` drops zero-valued
keys, so the field is now simply **absent** whenever every worker contributed,
and appears only when something is off. Nothing on the wire and nothing in the
output in the common case.
**Dropped `minNonzeroEmittedRows` (@Jackie-Jiang, @yashmayya)** — agreed,
gone. `maxEmittedRows` is kept: it answers "did one worker get most of the
data", which `emittedRows` cannot since it is the sum. If it still reads as
noise next to `maxClockTimeMs`, I am happy to drop that too.
**`CollectionUtils.isNotEmpty()`** — done, both places.
**Coverage** — added `ServerQueryRequestTest`, which covers the codecov gap.
One of its cases asserts that `getSegmentsToQuery()` is null on the
logical-table path, which is the trap `hasSegmentsToQuery()` exists to hide.
Rendered output now, on a balanced two-server `GROUP BY` — note no
`nonActiveWorkers` anywhere:
```
MAILBOX_SEND parallelism:2 emittedRows:5 maxEmittedRows:3
clockTimeMs:294 maxClockTimeMs:294
AGGREGATE
MAILBOX_RECEIVE parallelism:2 emittedRows:10
MAILBOX_SEND parallelism:2 emittedRows:10 maxEmittedRows:5
clockTimeMs:101 maxClockTimeMs:101
LEAF parallelism:2 emittedRows:10
```
and the same query with a filter matching nothing, where the leaf was given
segments but nothing came out of the stage:
```
MAILBOX_SEND parallelism:2 nonActiveWorkers:2 clockTimeMs:6
maxClockTimeMs:6
AGGREGATE
MAILBOX_RECEIVE parallelism:2 nonActiveWorkers:2
MAILBOX_SEND parallelism:2 nonActiveWorkers:2 clockTimeMs:2
maxClockTimeMs:2
LEAF parallelism:2 numSegmentsQueried:3
numSegmentsPrunedByServer:3
```
The `LEAF` reporting no idle worker while everything above it reports two is
the distinction that a single stage-level number could not make: the work was
handed out, and it is visible exactly where it stopped producing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
