FrankChen021 commented on code in PR #19627:
URL: https://github.com/apache/druid/pull/19627#discussion_r3474272577
##########
processing/src/main/java/org/apache/druid/query/groupby/GroupByStatsProvider.java:
##########
@@ -224,9 +266,24 @@ public void mergeBufferAcquisitionTime(long delay)
mergeBufferAcquisitionTimeNs.addAndGet(delay);
}
- public void maxMergeBufferUsedBytes(long bytes)
+ /**
+ * Accumulates the peak merge-buffer usage of one grouper (slice). Despite
the previous "max" naming, this method
+ * sums across the slices a query holds; see {@link #mergeBufferUsedBytes}.
+ */
+ public void addMergeBufferUsedBytes(long bytes)
+ {
+ mergeBufferUsedBytes.addAndGet(bytes);
+ }
+
+ /**
+ * Records one slice's peak used bytes and its spill threshold ({@code
sliceSize * maxLoadFactor}). Both are tracked
+ * as maxima, so after all slices close the pair describes the single
fullest slice — the one that drives spilling.
+ * Used to compute {@code mergeBuffer/maxSpillProximity}.
+ */
+ public void sliceUsage(long usedBytes, long spillThresholdBytes)
{
- maxMergeBufferUsedBytes.addAndGet(bytes);
+ maxSliceUsedBytes.accumulateAndGet(usedBytes, Math::max);
+ sliceSpillThresholdBytes.accumulateAndGet(spillThresholdBytes,
Math::max);
Review Comment:
[P2] Track spill proximity as a per-slice ratio
This stores the maximum used bytes and maximum threshold independently,
which breaks when one query reports groupers with different thresholds.
Nested/subtotal processing can pass the same PerQueryStats through a sliced
ConcurrentGrouper and later full-buffer SpillingGroupers; if a small slice
reaches its spill threshold, the larger full-buffer threshold can be retained
here and `getSpillProximity()` will divide by that larger value,
under-reporting the slice spill as roughly `1 / concurrencyHint`. Please track
the maximum `usedBytes / spillThresholdBytes` per `sliceUsage` call, or
otherwise keep the used/threshold pair together, so mixed grouper sizes still
report the true max proximity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
