This sounds a lot better to me--hopefully it isn't too much harder! I do think if it is possible to do this directly that will be better for users than having an intermediate step since we always have to work through migrating people who have setup quotas already from the old way to the new way.
-Jay On Fri, Jun 10, 2016 at 2:12 PM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > I do think client-id is a valid and useful grouping for quotas even in > secure clusters - but only if clientA of user1 is treated as a different > client-id from clientA of user2. Grouping of clients of a user enables > users to allocate their quota effectively to their clients (eg. guarantee > that critical event processing clients are not throttled by auditing > clients). When the KIP was down-sized to support only user-based quotas, I > was hoping that we could extend it at a later time to enable hierarchical > quotas. But I understand that it can be confusing to switch the semantics > of quotas based on modes set in the brokers. So let me try once again to > promote the original KIP-55. At the time, I did have a flag to revert to > the existing client-id behavior to maintain compatibility. But perhaps that > is not necessary. > > How does this sound? > > - Quotas may be configured for users. Sub-quotas may be configured for > client-ids of a user. Quotas may also be configured for client-ids of > users > with unlimited quota (Long.MaxValue). > - Users who don't have a quota override are allocated a configurable > default quota. > - Client-ids without a sub-quota override share the remainder of the > user quota if the user has a quota limit. Default quotas may be defined > for > clients of users with unlimited quota. > - For an insecure or single-user secure cluster, the existing client-id > based quota semantics can be achieved by configuring unlimited quota for > the user and sub-quota configuration for client-id that matches the > current > client-id quota configuration. > - For a cluster mixes both secure and insecure access, client-id quotas > can be set for unauthenticated clients (unlimited quota for ANONYMOUS, > quotas for client-ids) and user quotas can be set for authenticated > users. > - In a multi-user cluster, it is currently possible to define quotas for > client-ids that span multiple users. This will no longer be supported. > > > > > On Fri, Jun 10, 2016 at 6:43 PM, Gwen Shapira <g...@confluent.io> wrote: > > > I am not crazy about modes either. An earlier proposal supported both > > client-ids and users at the same time, and it made more sense to me. I > > believe it was dropped without proper discussion (Basically, Jun > > mentioned it is complex and Rajini agreed to drop it). We should > > probably rethink the complexity of supporting both vs the limitations > > of "modes". > > > > As you said, we will have secure clients authenticating with users and > > insecure clients authenticating with client-ids at the same time. > > > > On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps <j...@confluent.io> wrote: > > > Hey Rajini, > > > > > > 1. That makes sense to me. Is that reflected in the documentation > > anywhere > > > (I couldn't really find it)? Is there a way to discover that > definition? > > We > > > do way better when we right this stuff down so it has an official > > > definition users and developers can work off of... > > > 2. If client id is a thing that makes sense even when you have users, > why > > > would you not want to quota on it? > > > > > > I am not wild about these "modes" where you boot the cluster in mode X > > and > > > it behaves in one way and in mode Y and it behaves in another. It is > > > complex then for users who expect to be able to set quotas but then > have > > to > > > be able to get access to the filesystem of the kafka nodes to discover > > what > > > mode the cluster is in to know which kind of quota is applicable. > > > > > > I guess there are two ways to think about a feature: one is the > increment > > > from where we are, and the other is the resulting state after that > > > increment is taken. What I am asking is not "is this a low cost step > from > > > where we are?" which everyone can agree that it is, but rather "does > this > > > make sense as an end state--i.e. if you were starting fresh with > neither > > > users nor client ids nor quotas would you end up with this?". > > > > > > In terms of use cases, I think that we support mixing secure and > insecure > > > access on a single cluster so presumably in that case you would want to > > be > > > able to quota insecure users based on client id and secure users based > on > > > user, right? Likewise, as you said, client id is a valid grouping even > in > > > the presence of users, so it might be the case that several apps that > are > > > all part of the same system might access Kafka under a single user, but > > you > > > might have different quotas for these different apps. Basically if > client > > > id is a valid grouping even in the presence of users (willing to debate > > > this point, btw!) then you would want to quota on it. > > > > > > -Jay > > > > > > > > > > > > On Fri, Jun 10, 2016 at 4:49 AM, Rajini Sivaram < > > > rajinisiva...@googlemail.com> wrote: > > > > > >> Jay, > > >> > > >> Thank you for the feedback. > > >> > > >> 1. I think it is good to have a single concept of identity, but > multiple > > >> ways of grouping clients for different functions. Client-id is a > logical > > >> grouping of clients with a meaningful name that is used in client > > metrics > > >> and logs. User principal is an authenticated user or a grouping of > > >> unauthenticated users chosen by the broker and is used for ACLs. In my > > >> view, in a multi-user system, there is a hierarchy - user owns zero or > > more > > >> clients. (principal, client-id) defines a safe group, but the shorter > > >> unsafe client-id is sufficient in client metrics and logs. > > >> > > >> > > >> 2. KIP-55 was initially written to support hierarchical quotas (quotas > > for > > >> user1-clientA, user2-clientA etc), but Jun's view was that the > > complexity > > >> was not justified since there is no clear requirement for this. The > > >> cut-down version of the KIP is clearly a lot simpler. But I think your > > >> suggestion is to enable non-hierarchical user quotas and client-id > > quotas > > >> at the same time. Basically treat users and client-ids as distinct > > entities > > >> like topics and allow quotas to be applied to each of these entities. > I > > >> agree that we want to support quotas simultaneously on different > > entities > > >> like topics and users. I am less convinced of client-id and user being > > >> > > >> distinct entities that require separate quotas at the same time. And > > >> treating client-id and user as distinct entities makes it harder to > > >> implement hierarchical quotas in future. > > >> > > >> > > >> A single user system needs only client-id quotas, and multi-tenant > > system > > >> cannot use client-id quotas since we need to guarantee that one > tenant's > > >> quotas can never be used by another tenant. I suppose a multi-user > > cluster > > >> where users trust each other could apply separate quotas for both > > clients > > >> and users, but I am not sure if there is a usecase that can't be > > satisfied > > >> with just client-id based quotas for this case. Do you have a usecase > in > > >> mind where you want to apply separate quotas for clients and users > > >> simultaneously? > > >> > > >> > > >> > > >> > > >> On Thu, Jun 9, 2016 at 9:40 PM, Jay Kreps <j...@confluent.io> wrote: > > >> > > >> > Super sorry to come in late on this one. Rajini, I had two quick > > >> questions > > >> > I think we should be able to answer: > > >> > > > >> > 1. Do client ids make sense in a world which has users? If not > > should > > >> we > > >> > unify them the way Hadoop did (without auth the user is a kind of > > best > > >> > effort honor system identity). This came up in the discussion > > thread > > >> > but I > > >> > didn't really see a crisp answer. Basically, what is the > > definition of > > >> > "client id" and what is the definition of "user" and how do the > > >> concepts > > >> > relate? > > >> > 2. If both client ids and users are sensible distinct notions and > > we > > >> > want to maintain both, why don't we just support quotas on both? > If > > >> they > > >> > both make sense then you would have a reason to set quotas at > both > > >> > levels. > > >> > Why have this "mode" that you set that swaps between only being > > able > > >> to > > >> > use > > >> > one and the other? I should be able to set quotas at both levels. > > >> Going > > >> > forward the model we had discussed with quotas was potentially > > being > > >> > able > > >> > to set quotas for many things independently (say at the topic > > level), > > >> > and I > > >> > don't think it would make sense to extend this mode approach to > > those. > > >> > > > >> > -Jay > > >> > > > >> > On Wed, Jun 8, 2016 at 12:56 PM, Rajini Sivaram < > > >> > rajinisiva...@googlemail.com> wrote: > > >> > > > >> > > I would like to initiate the vote for KIP-55. > > >> > > > > >> > > The KIP details are here: KIP-55: Secure quotas for authenticated > > users > > >> > > < > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users > > >> > > > > > >> > > . > > >> > > > > >> > > The JIRA KAFKA-3492 < > > >> https://issues.apache.org/jira/browse/KAFKA-3492 > > >> > > >has > > >> > > a draft PR here: https://github.com/apache/kafka/pull/1256. > > >> > > > > >> > > Thank you... > > >> > > > > >> > > Regards, > > >> > > > > >> > > Rajini > > >> > > > > >> > > > >> > > >> > > >> > > >> -- > > >> Regards, > > >> > > >> Rajini > > >> > > > > > > -- > Regards, > > Rajini >
