kfaraz commented on PR #17903: URL: https://github.com/apache/druid/pull/17903#issuecomment-2848510167
@jtuglu-netflix , I haven't taken a look at the patch yet. Could you please give me an idea of the following? - Do you feel that the current scale up logic adds too many tasks or too few? - Similarly, what do you feel about the current scale down logic? - In this patch, what is the formula to compute the "proportional" task count for a given value of lag? Does it also incorporate the current processing rate of each To add to @cryptoe 's point, in production clusters, we have seen that scaling to a number of tasks that does not uniformly divide the number of partitions in the streaming topic always causes grief. So, say for a Kafka supervisor reading from a topic with `100` partitions, the valid task count options should be: `1, 2, 4, 5, 10, 20, 25, 50, 100` So it would be nice if we could do take a combined approach in this PR i.e. _proportional_ scaling to a _valid_ task count. So, if proportional logic dictates that the required task count is 23, we go with 25. If we need 51, we just go with 50 and so on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
