kfaraz commented on PR #18663: URL: https://github.com/apache/druid/pull/18663#issuecomment-3417903838
> It turns out that the compact duty using the native engine just caps it using maxNumConcurrentSubTasks, regardless of the phase a current compact supervisor is in. @abhishekrb19 , I am not sure how the `busyCompactionTaskSlots` being emitted in this PR will avoid this problem, since the value we are emitting is effectively just `maxSlots - availableSlots`. Could you please clarify this? To determine the actual count of tasks, we could count the sub-tasks for each currently running `compact` tasks, but that can run into other issues. I feel it is better to launch fewer `compact` tasks (due to a smaller `availableSlot` count) than over-assign `compact` tasks causing them to potentially hog up slots reserved for ingestion. So I feel the current approach of capping at `maxNumConcurrentSubTasks` is reasonable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
