Hi, Colin, Thanks for the comment. You brought up several points.
1. Should we set up a per user quota? To me, it does seem we need some sort of a quota. When the controller runs out of resources, ideally, we only want to penalize the bad behaving applications, instead of every application. To do that, we will need to know what each application is entitled to and the per user quota is intended to capture that. 2. How easy is it to configure a quota? The following is how an admin typically sets up a quota in our existing systems. Pick a generous default per user quota works for most applications. For the few resource intensive applications, customize a higher quota for them. Reserve enough resources in anticipation that a single (or a few) application will exceed the quota at a given time. 3. How should the quota be defined? In the discussion thread, we debated between a usage based model vs a rate based model. Dave and Anna argued for the rate based model mostly because it's simpler to implement. 4. If a quota is exceeded, how is that enforced? My understanding of the KIP is that, if a quota is exceeded, the broker immediately sends back a QUOTA_VIOLATED error and a throttle time back to the client, and the client will wait for the throttle time before issuing the next request. This seems to be the same as the BUSY error code you mentioned. I will let David chime in more on that. Thanks, Jun On Sun, Jun 7, 2020 at 2:30 PM Colin McCabe <cmcc...@apache.org> wrote: > Hi David, > > Thanks for the KIP. > > I thought about this for a while and I actually think this approach is not > quite right. The problem that I see here is that using an explicitly set > quota here requires careful tuning by the cluster operator. Even worse, > this tuning might be invalidated by changes in overall conditions or even > more efficient controller software. > > For example, if we empirically find that the controller can do 1000 topics > in a minute (or whatever), this tuning might actually be wrong if the next > version of the software can do 2000 topics in a minute because of > efficiency upgrades. Or, the broker that the controller is located on > might be experiencing heavy load from its non-controller operations, and so > it can only do 500 topics in a minute during this period. > > So the system administrator gets a very obscure tunable (it's not clear to > a non-Kafka-developer what "controller mutations" are or why they should > care). And even worse, they will have to significantly "sandbag" the value > that they set it to, so that even under the heaviest load and oldest > deployed version of the software, the controller can still function. Even > worse, this new quota adds a lot of complexity to the controller. > > What we really want is backpressure when the controller is overloaded. I > believe this is the alternative you discuss in "Rejected Alternatives" > under "Throttle the Execution instead of the Admission" Your reason for > rejecting it is that the client error handling does not work well in this > case. But actually, this is an artifact of our current implementation, > rather than a fundamental issue with backpressure. > > Consider the example of a CreateTopicsRequest. The controller could > return a special error code if the load was too high, and take the create > topics event off the controller queue. Let's call that error code BUSY. > Additionally, the controller could immediately refuse new events if the > queue had reached its maximum length, and simply return BUSY for that case > as well. > > Basically, the way we handle RPC timeouts in the controller right now is > not very good. As you know, we time out the RPC, so the client gets > TimeoutException, but then keep the event on the queue, so that it > eventually gets executed! There's no reason why we have to do that. We > could take the event off the queue if there is a timeout. This would > reduce load and mostly avoid the paradoxical situations you describe > (getting TopicExistsException for a CreateTopicsRequest retry, etc.) > > I say "mostly" because there are still cases where retries could go astray > (for example if we execute the topic creation but a network problem > prevents the response from being sent to the client). But this would still > be a very big improvement over what we have now. > > Sorry for commenting so late on this but I got distracted by some other > things. I hope we can figure this one out-- I feel like there is a chance > to significantly simplify this. > > best, > Colin > > > On Fri, May 29, 2020, at 07:57, David Jacot wrote: > > Hi folks, > > > > I'd like to start the vote for KIP-599 which proposes a new quota to > > throttle create topic, create partition, and delete topics operations to > > protect the Kafka controller: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+Topic+Operations > > > > Please, let me know what you think. > > > > Cheers, > > David > > >
