abhishekrb19 commented on PR #18663: URL: https://github.com/apache/druid/pull/18663#issuecomment-3430306747
@kfaraz thanks for taking a look! >It turns out that the compact duty using the native engine just caps it using maxNumConcurrentSubTasks, regardless of the phase a current compact supervisor is in. @abhishekrb19 , I am not sure how the busyCompactionTaskSlots being emitted in this PR will avoid this problem, since the value we are emitting is effectively just maxSlots - availableSlots. Yeah, I was just noting my observation on how this metric is calculated. The docs for the `compactTask/availableSlot/count` metric tripped me up a bit: “This is the max number of task slots minus any currently running compaction tasks,” rather than mentioning that it’s an _estimated_ number of currently running compaction tasks. While at it, I was thinking that the busy slot estimate could just be emitted directly given the [condition](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java#L430-L438) - a consistent higher utilization of the busy slots metric (and a corresponding drop in available slots) would indicate the need to tune the compaction task slots, which is what we ended up doing for some clusters. >Although, it might be useful to emit the "actual" busy slot count which would be computed using the sub-task counts as mentioned above. Is that what you mean to do here? I wasn't necessarily thinking of emitting the "actual" busy slot count because it doesn't influence the auto-compaction algorithm currently; also I think this can be determined using `task/*/count`? Let me know if that makes sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
