i dont think time/cpu% are easy to reason about. most user-facing quota systems i know (especially the commercial ones) focus on things users understand better - iops and bytes.
as for quotas and "overhead" requests like heartbeats - on the one hand subjecting them to the quota may cause clients to die out. on the other not subjecting them to the quota opens the broker up to DOS attacks. how about giving overhead requests their own quota, separate from "real" (user-initiated?) requests? slightly more complicated but i think solves the issue? how long are requests held in purgatory? wouldnt this, at some point, still cause resources to be taken? wouldnt it be better (for high enough delay values) to just return an error to the client (quota exceeded, try again in 3 seconds)? how would these work across an entire cluster? if these are enforced independently on every single broker you'd be hitting "monotonous" clients (who interact with fewer partitions) much harder than clients who operate across a lot of partitions. On Thu, Feb 23, 2017 at 8:02 AM, Ismael Juma <ism...@juma.me.uk> wrote: > Thanks for the KIP, Rajini. This is a welcome improvement and the KIP page > covers it well. A few comments: > > 1. Can you expand a bit on the motivation for throttling requests that fail > authorization for ClusterAction? Under what scenarios would this help? > > 2. I think we should rename `throttle_time_ms` in the new version of > produce/fetch response to make it clear that it refers to the byte rate > throttling. Also, it would be good to include the updated schema for the > responses (we typically try to do that whenever we update protocol APIs). > > 3. I think I am OK with using absolute units, but I am not sure about the > argument why it's better than a percentage. We are comparing request > threads to CPUs, but they're not the same as increasing the number of > request threads doesn't necessarily mean that the server can cope with more > requests. In the example where we double the number of threads, all the > existing users would still have the same capacity proportionally speaking > so it seems intuitive to me. One thing that would be helpful, I think, is > to describe a few scenarios where the setting needs to be adjusted and how > users would go about doing it. > > 4. I think it's worth mentioning that TLS increases the load on the network > thread significantly and for cases where there is mixed plaintext and TLS > traffic, the existing byte rate throttling may not do a great job. I think > it's OK to tackle this in a separate KIP, but worth mentioning the > limitation. > > 5. We mention DoS attacks in the document. It may be worth mentioning that > this mostly helps with clients that are not malicious. A malicious client > could generate a large number of connections to counteract the delays that > this KIP introduces. Kafka has connection limits per IP today, but not per > user, so a distributed DoS could bypass those. This is not easy to solve at > the Kafka level since the authentication step required to get the user may > be costly enough that the brokers will eventually be overwhelmed. > > 6. It's unfortunate that the existing byte rate quota configs use > underscores instead of dots (like every other config) as separators. It's > reasonable for `io_thread_units` to use the same convention as the byte > rate configs, but it's not great that we are adding to the inconsistency. I > don't have any great solutions apart from perhaps accepting the dot > notation for all these configs as well. > > Ismael > > On Fri, Feb 17, 2017 at 5:05 PM, Rajini Sivaram <rajinisiva...@gmail.com> > wrote: > > > Hi all, > > > > I have just created KIP-124 to introduce request rate quotas to Kafka: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+ > > Request+rate+quotas > > > > The proposal is for a simple percentage request handling time quota that > > can be allocated to *<client-id>*, *<user>* or *<user, client-id>*. There > > are a few other suggestions also under "Rejected alternatives". Feedback > > and suggestions are welcome. > > > > Thank you... > > > > Regards, > > > > Rajini > > >
