Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-08-06 Thread David Jacot
Hi all, I would like to inform you that we have slightly changed our thoughts about the implementation of the Token Bucket algorithm. Our initial idea was to change our existing Rate to behave like a Token Bucket. That works as we expected but we have realized that the value of the Rate is not

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-07-28 Thread David Jacot
Hi all, Just a quick update. We have made good progress regarding the implementation of this KIP. The major parts are already in trunk modulo the new "rate" implementation which is still under development. I would like to change the type of the `controller_mutations_rate` from a Long to a

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-15 Thread David Jacot
Hi all, The vote has passed with 5 binding votes (Gwen, Rajini, Mickael, Jun and Colin) and 2 non-binding votes (Tom, Anna). Thank you all for the fruitful discussion! I'd like to particularly thank Anna who has heavily contributed to the design of this KIP. Regards, David On Fri, Jun 12, 2020

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-12 Thread Colin McCabe
+1. Thanks, David! best, Colin On Thu, Jun 11, 2020, at 23:51, David Jacot wrote: > Colin, Jun, > > Do the proposed error code and the updated KIP look good to you guys? I’d > like to wrap up and close the vote. > > Thanks, > David > > Le mer. 10 juin 2020 à 14:50, David Jacot a écrit : >

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-12 Thread Jun Rao
Hi, David, Thanks for making those changes. They look fine to me. +1 Jun On Thu, Jun 11, 2020 at 11:51 PM David Jacot wrote: > Colin, Jun, > > Do the proposed error code and the updated KIP look good to you guys? I’d > like to wrap up and close the vote. > > Thanks, > David > > Le mer. 10

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-12 Thread David Jacot
Colin, Jun, Do the proposed error code and the updated KIP look good to you guys? I’d like to wrap up and close the vote. Thanks, David Le mer. 10 juin 2020 à 14:50, David Jacot a écrit : > Hi Colin and Jun, > > I have no problem if we have to rewrite part of it when the new controller >

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-10 Thread David Jacot
Hi Colin and Jun, I have no problem if we have to rewrite part of it when the new controller comes out. I will be more than happy to help out. Regarding KIP-590, I think that we can cope with a principal as a string when the time comes. The user entity name is defined with a string already.

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-09 Thread Jun Rao
Hi, Colin, Good point. Maybe sth like THROTTLING_QUOTA_VIOLATED will make this clear. Hi, David, We added a new quota name in the KIP. You chose not to bump up the version of DESCRIBE_CLIENT_QUOTAS and ALTER_CLIENT_QUOTAS, which seems ok since the quota name is represented as a string. However,

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-09 Thread Colin McCabe
On Tue, Jun 9, 2020, at 05:06, David Jacot wrote: > Hi Colin, > > Thank you for your feedback. > > Jun has summarized the situation pretty well. Thanks Jun! I would like to > complement it with the following points: > > 1. Indeed, when the quota is exceeded, the broker will reject the topic >

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-09 Thread Jun Rao
Hi, David, Sounds good then. Thanks, Jun On Tue, Jun 9, 2020 at 10:59 AM David Jacot wrote: > Hi Jun, > > Both are already in the KIP, see "New Broker Configurations" chapter. I > think > that we need them in order to be able to define different burst for the new > quota. > > Best, > David >

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-09 Thread David Jacot
Hi Jun, Both are already in the KIP, see "New Broker Configurations" chapter. I think that we need them in order to be able to define different burst for the new quota. Best, David On Tue, Jun 9, 2020 at 7:48 PM Jun Rao wrote: > Hi, David, > > Another thing. Should we add

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-09 Thread Jun Rao
Hi, David, Another thing. Should we add controller.quota.window.size.seconds and controller.quota.window.num or just reuse the existing quota.window.size.seconds and quota.window.num that are used for other types of quotas? Thanks, Jun On Tue, Jun 9, 2020 at 10:30 AM Jun Rao wrote: > Hi,

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-09 Thread Jun Rao
Hi, David, Thanks for the KIP. The name QUOTA_VIOLATED sounds reasonable to me. +1 on the KIP. Jun On Tue, Jun 9, 2020 at 5:07 AM David Jacot wrote: > Hi Colin, > > Thank you for your feedback. > > Jun has summarized the situation pretty well. Thanks Jun! I would like to > complement it with

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-09 Thread David Jacot
Hi Colin, Thank you for your feedback. Jun has summarized the situation pretty well. Thanks Jun! I would like to complement it with the following points: 1. Indeed, when the quota is exceeded, the broker will reject the topic creations, partition creations and topics deletions that are

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-08 Thread Colin McCabe
On Mon, Jun 8, 2020, at 14:41, Jun Rao wrote: > Hi, Colin, > > Thanks for the comment. You brought up several points. > > 1. Should we set up a per user quota? To me, it does seem we need some sort > of a quota. When the controller runs out of resources, ideally, we only > want to penalize the

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-08 Thread Jun Rao
Hi, Colin, Thanks for the comment. You brought up several points. 1. Should we set up a per user quota? To me, it does seem we need some sort of a quota. When the controller runs out of resources, ideally, we only want to penalize the bad behaving applications, instead of every application. To

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-07 Thread Colin McCabe
Hi David, Thanks for the KIP. I thought about this for a while and I actually think this approach is not quite right. The problem that I see here is that using an explicitly set quota here requires careful tuning by the cluster operator. Even worse, this tuning might be invalidated by

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-05 Thread Anna Povzner
+1 (not binding) Thanks for the KIP! -Anna On Thu, Jun 4, 2020 at 8:26 AM Mickael Maison wrote: > +1 (binding) > Thanks David for looking into this important issue > > On Thu, Jun 4, 2020 at 3:59 PM Tom Bentley wrote: > > > > +1 (non binding). > > > > Thanks! > > > > On Wed, Jun 3, 2020 at

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-04 Thread Mickael Maison
+1 (binding) Thanks David for looking into this important issue On Thu, Jun 4, 2020 at 3:59 PM Tom Bentley wrote: > > +1 (non binding). > > Thanks! > > On Wed, Jun 3, 2020 at 3:51 PM Rajini Sivaram > wrote: > > > +1 (binding) > > > > Thanks for the KIP, David! > > > > Regards, > > > > Rajini >

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-04 Thread Tom Bentley
+1 (non binding). Thanks! On Wed, Jun 3, 2020 at 3:51 PM Rajini Sivaram wrote: > +1 (binding) > > Thanks for the KIP, David! > > Regards, > > Rajini > > > On Sun, May 31, 2020 at 3:29 AM Gwen Shapira wrote: > > > +1 (binding) > > > > Looks great. Thank you for the in-depth design and

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-06-03 Thread Rajini Sivaram
+1 (binding) Thanks for the KIP, David! Regards, Rajini On Sun, May 31, 2020 at 3:29 AM Gwen Shapira wrote: > +1 (binding) > > Looks great. Thank you for the in-depth design and discussion. > > On Fri, May 29, 2020 at 7:58 AM David Jacot wrote: > > > Hi folks, > > > > I'd like to start the

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-05-30 Thread Gwen Shapira
+1 (binding) Looks great. Thank you for the in-depth design and discussion. On Fri, May 29, 2020 at 7:58 AM David Jacot wrote: > Hi folks, > > I'd like to start the vote for KIP-599 which proposes a new quota to > throttle create topic, create partition, and delete topics operations to >