I do think client-id is a valid and useful grouping for quotas even in secure clusters - but only if clientA of user1 is treated as a different client-id from clientA of user2. Grouping of clients of a user enables users to allocate their quota effectively to their clients (eg. guarantee that critical event processing clients are not throttled by auditing clients). When the KIP was down-sized to support only user-based quotas, I was hoping that we could extend it at a later time to enable hierarchical quotas. But I understand that it can be confusing to switch the semantics of quotas based on modes set in the brokers. So let me try once again to promote the original KIP-55. At the time, I did have a flag to revert to the existing client-id behavior to maintain compatibility. But perhaps that is not necessary.
How does this sound? - Quotas may be configured for users. Sub-quotas may be configured for client-ids of a user. Quotas may also be configured for client-ids of users with unlimited quota (Long.MaxValue). - Users who don't have a quota override are allocated a configurable default quota. - Client-ids without a sub-quota override share the remainder of the user quota if the user has a quota limit. Default quotas may be defined for clients of users with unlimited quota. - For an insecure or single-user secure cluster, the existing client-id based quota semantics can be achieved by configuring unlimited quota for the user and sub-quota configuration for client-id that matches the current client-id quota configuration. - For a cluster mixes both secure and insecure access, client-id quotas can be set for unauthenticated clients (unlimited quota for ANONYMOUS, quotas for client-ids) and user quotas can be set for authenticated users. - In a multi-user cluster, it is currently possible to define quotas for client-ids that span multiple users. This will no longer be supported. On Fri, Jun 10, 2016 at 6:43 PM, Gwen Shapira <g...@confluent.io> wrote: > I am not crazy about modes either. An earlier proposal supported both > client-ids and users at the same time, and it made more sense to me. I > believe it was dropped without proper discussion (Basically, Jun > mentioned it is complex and Rajini agreed to drop it). We should > probably rethink the complexity of supporting both vs the limitations > of "modes". > > As you said, we will have secure clients authenticating with users and > insecure clients authenticating with client-ids at the same time. > > On Fri, Jun 10, 2016 at 7:19 PM, Jay Kreps <j...@confluent.io> wrote: > > Hey Rajini, > > > > 1. That makes sense to me. Is that reflected in the documentation > anywhere > > (I couldn't really find it)? Is there a way to discover that definition? > We > > do way better when we right this stuff down so it has an official > > definition users and developers can work off of... > > 2. If client id is a thing that makes sense even when you have users, why > > would you not want to quota on it? > > > > I am not wild about these "modes" where you boot the cluster in mode X > and > > it behaves in one way and in mode Y and it behaves in another. It is > > complex then for users who expect to be able to set quotas but then have > to > > be able to get access to the filesystem of the kafka nodes to discover > what > > mode the cluster is in to know which kind of quota is applicable. > > > > I guess there are two ways to think about a feature: one is the increment > > from where we are, and the other is the resulting state after that > > increment is taken. What I am asking is not "is this a low cost step from > > where we are?" which everyone can agree that it is, but rather "does this > > make sense as an end state--i.e. if you were starting fresh with neither > > users nor client ids nor quotas would you end up with this?". > > > > In terms of use cases, I think that we support mixing secure and insecure > > access on a single cluster so presumably in that case you would want to > be > > able to quota insecure users based on client id and secure users based on > > user, right? Likewise, as you said, client id is a valid grouping even in > > the presence of users, so it might be the case that several apps that are > > all part of the same system might access Kafka under a single user, but > you > > might have different quotas for these different apps. Basically if client > > id is a valid grouping even in the presence of users (willing to debate > > this point, btw!) then you would want to quota on it. > > > > -Jay > > > > > > > > On Fri, Jun 10, 2016 at 4:49 AM, Rajini Sivaram < > > rajinisiva...@googlemail.com> wrote: > > > >> Jay, > >> > >> Thank you for the feedback. > >> > >> 1. I think it is good to have a single concept of identity, but multiple > >> ways of grouping clients for different functions. Client-id is a logical > >> grouping of clients with a meaningful name that is used in client > metrics > >> and logs. User principal is an authenticated user or a grouping of > >> unauthenticated users chosen by the broker and is used for ACLs. In my > >> view, in a multi-user system, there is a hierarchy - user owns zero or > more > >> clients. (principal, client-id) defines a safe group, but the shorter > >> unsafe client-id is sufficient in client metrics and logs. > >> > >> > >> 2. KIP-55 was initially written to support hierarchical quotas (quotas > for > >> user1-clientA, user2-clientA etc), but Jun's view was that the > complexity > >> was not justified since there is no clear requirement for this. The > >> cut-down version of the KIP is clearly a lot simpler. But I think your > >> suggestion is to enable non-hierarchical user quotas and client-id > quotas > >> at the same time. Basically treat users and client-ids as distinct > entities > >> like topics and allow quotas to be applied to each of these entities. I > >> agree that we want to support quotas simultaneously on different > entities > >> like topics and users. I am less convinced of client-id and user being > >> > >> distinct entities that require separate quotas at the same time. And > >> treating client-id and user as distinct entities makes it harder to > >> implement hierarchical quotas in future. > >> > >> > >> A single user system needs only client-id quotas, and multi-tenant > system > >> cannot use client-id quotas since we need to guarantee that one tenant's > >> quotas can never be used by another tenant. I suppose a multi-user > cluster > >> where users trust each other could apply separate quotas for both > clients > >> and users, but I am not sure if there is a usecase that can't be > satisfied > >> with just client-id based quotas for this case. Do you have a usecase in > >> mind where you want to apply separate quotas for clients and users > >> simultaneously? > >> > >> > >> > >> > >> On Thu, Jun 9, 2016 at 9:40 PM, Jay Kreps <j...@confluent.io> wrote: > >> > >> > Super sorry to come in late on this one. Rajini, I had two quick > >> questions > >> > I think we should be able to answer: > >> > > >> > 1. Do client ids make sense in a world which has users? If not > should > >> we > >> > unify them the way Hadoop did (without auth the user is a kind of > best > >> > effort honor system identity). This came up in the discussion > thread > >> > but I > >> > didn't really see a crisp answer. Basically, what is the > definition of > >> > "client id" and what is the definition of "user" and how do the > >> concepts > >> > relate? > >> > 2. If both client ids and users are sensible distinct notions and > we > >> > want to maintain both, why don't we just support quotas on both? If > >> they > >> > both make sense then you would have a reason to set quotas at both > >> > levels. > >> > Why have this "mode" that you set that swaps between only being > able > >> to > >> > use > >> > one and the other? I should be able to set quotas at both levels. > >> Going > >> > forward the model we had discussed with quotas was potentially > being > >> > able > >> > to set quotas for many things independently (say at the topic > level), > >> > and I > >> > don't think it would make sense to extend this mode approach to > those. > >> > > >> > -Jay > >> > > >> > On Wed, Jun 8, 2016 at 12:56 PM, Rajini Sivaram < > >> > rajinisiva...@googlemail.com> wrote: > >> > > >> > > I would like to initiate the vote for KIP-55. > >> > > > >> > > The KIP details are here: KIP-55: Secure quotas for authenticated > users > >> > > < > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users > >> > > > > >> > > . > >> > > > >> > > The JIRA KAFKA-3492 < > >> https://issues.apache.org/jira/browse/KAFKA-3492 > >> > > >has > >> > > a draft PR here: https://github.com/apache/kafka/pull/1256. > >> > > > >> > > Thank you... > >> > > > >> > > Regards, > >> > > > >> > > Rajini > >> > > > >> > > >> > >> > >> > >> -- > >> Regards, > >> > >> Rajini > >> > -- Regards, Rajini
