Makes sense. I'll update the KIP. Does anyone have any other comments? :)
Thanks Piyush Vijay On Thu, May 3, 2018 at 11:55 AM, Colin McCabe <cmcc...@apache.org> wrote: > Yeah, I guess that's a good point. It probably makes sense to support the > prefix scheme for consumer groups and transactional IDs as well as topics. > > I agree that the current situation where anything goes in consumer group > names and transactional ID names is not ideal. I wish we could rewind the > clock and impose restrictions on the names. However, it doesn't seem > practical at the moment. Adding new restrictions would break a lot of > existing users after an upgrade. It would be a really bad upgrade > experience. > > However, I think we can support this in a compatible way. From the > perspective of AdminClient, we just have to add a new field to > ResourceFilter. Currently, it has two fields, resourceType and name: > > > /** > > * A filter which matches Resource objects. > > * > > * The API for this class is still evolving and we may break > compatibility in minor releases, if necessary. > > */ > > @InterfaceStability.Evolving > > public class ResourceFilter { > > private final ResourceType resourceType; > > private final String name; > > We can add a third field, pattern. > > So the API will basically be, if I create a ResourceFilter(resourceType=GROUP, > name=foo*, pattern=null), it applies only to the consumer group named > "foo*". If I create a ResourceFilter(resourceType=GROUP, name=null, > pattern=foo*), it applies to any consumer group starting in "foo". name > and pattern cannot be both set at the same time. This preserves > compatibility at the AdminClient level. > > It's possible that we will want to add more types of pattern in the > future. So we should reserve "special characters" such as +, /, &, %, #, > $, etc. These characters should be treated as special unless they are > prefixed with a backslash to escape them. This will allow us to add > support for using these characters in the future without breaking > compatibility. > > At the protocol level, we need a new API version for CreateAclsRequest / > DeleteAclsRequest. The new API version will send all special characters > over the wire escaped rather than directly. (So there is no need for the > equivalent of both "name" and "pattern"-- we translate name into a validly > escaped pattern that matches only one thing, by adding escape characters as > appropriate.) The broker will validate the new API version and reject > malformed of unsupported patterns. > > At the ZK level, we can introduce a protocol version to the data in ZK-- > or store it under a different root. > > best, > Colin > > > On Wed, May 2, 2018, at 18:09, Piyush Vijay wrote: > > Thank you everyone for the interest and, prompt and valuable feedback. I > > really appreciate the quick turnaround. I’ve tried to organize the > comments > > into common headings. See my replies below: > > > > > > > > *Case of ‘*’ might already be present in consumer groups and > transactional > > ids* > > > > > > > > - We definitely need wildcard ACLs support for resources like consumer > > groups and transactional ids for the reason Andy mentioned. A big win > of > > this feature is that service providers don’t have to track and keep > > up-to-date all the consumer groups their customers are using. > > - I agree with Andy’s thoughts on the two possible ways. > > - My vote would be to do the breaking change because we should > restrict > > the format of consumer groups and transactional ids sooner than later. > > - Consumer groups and transactional ids are basic Kafka concepts. > > There is a lot of value in having a defined naming convention on > these > > concepts. > > - This will help us not run into more issues down the line. > > - I’m not sure if people actually use ‘*’ in their consumer group > > names anyway. > > - Escaping ‘*’ isn’t trivial because ‘\’ is an allowed character > too. > > > > > > *Why introduce two new APIs?* > > > > > > > > - It’s possible to make this change without introducing new APIs but > new > > APIs are required for inspection. > > - For example: If I want to fetch all ACLs that match ’topicA*’, it’s > > not possible without introducing new APIs AND maintaining backwards > > compatibility. > > > > > > *Matching multiple hosts* > > > > > > > > - Rajini is right that wildcard ACLs aren’t the correct fit for > > specifying range of hosts. > > - We will rely on KIP-252 for the proper functionality ( > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 252+-+Extend+ACLs+to+allow+filtering+based+on+ip+ranges+and+subnets > > ) > > > > > > *Implementation of matching algorithm and performance concerns* > > > > > > > > - Updated the KIP with an implementation. > > - Andy, you’re right. The length doesn’t play a part. The request will > > be authorized *iff* there is at least one matching ALLOW and no > matching > > DENY irrespective of the prefix length. Included this detail in the > KIP. > > - Since everything is stored in memory, the performance hit isn’t > really > > significantly worse than the current behavior. > > - Stephane’s performance improvement suggestion is a great idea but > > orthogonal. > > > > > > *Why extend this for principal?* > > > > > > > > - Thanks for the amazing points Rajini. > > - There is a use case where ALL principals from an org might want to > > access fix set of topics. > > - User group functionality is currently missing. > > - IIRC SASL doesn’t use custom principal builder. > > - However, prefixing is not the right choice in all cases. Agreed. > > - Thoughts? > > > > > > *Changes to AdminClient to support wildcard ACLs* > > > > > > > > - Thanks Colin for the implementation. It’s good to have you and > > others > > here for the expert opinions. > > - The current implementation uses two classes: AclBinding and > > AclBindingFilter. ( > > > > https://github.com/apache/kafka/blob/trunk/clients/src/ > main/java/org/apache/kafka/common/acl/AclBinding.java > > and > > > > https://github.com/apache/kafka/blob/trunk/clients/src/ > main/java/org/apache/kafka/common/acl/AclBindingFilter.java > > ) > > - AclBinding is definition of an Acl. It’s used to create ACLs. > > - AclBindingFilter is used to fetch or delete “matching’ ACLs. > > - In the context of wildcard suffixed ACLs, a stored ACL may have ‘*’ > > in > > it. *It really removes the distinction between these two classes.* > > - The current implementation uses ‘null’ to define wildcard ACL > > (‘*’). I > > think it’s not a good pattern and we should use ‘*’ for the wildcard > > ACL ( > > > > https://github.com/apache/kafka/blob/trunk/clients/src/ > main/java/org/apache/kafka/common/acl/AclBindingFilter.java#L39 > > and > > > > https://github.com/apache/kafka/blob/trunk/clients/src/ > main/java/org/apache/kafka/common/resource/ResourceFilter.java#L37 > > ). > > - However, the above two changes are breaking change but it should be > > acceptable because the API is marked with > > @InterfaceStability.Evolving. > > - If everyone agrees to the above two changes (merging the two > > classes > > and using non-null values for blanket access), the only other change > > is > > using the matching algorithm from the KIP to match ACLs. > > > > > > Other comments: > > > > > > > > - > It may be worth excluding delegation token ACLs from using > prefixed > > wildcards since it doesn't make much sense. > > > > I want to ask for clarification on what delegation token ACLs are before > > commenting. Wildcard suffixed ACLs are supported only for resource and > > principal names. > > > > > > > > - A quick read makes me believe that I’ve fixed the formatting issues > > reported by Ted. Let me know if something is still wrong and I would > be > > happy to fix it. > > - I’ve fixed the mismatch in signature reported by Ron. > > - Andy, I’ve updated the KIP with the security hole related to DENY > > wildcard ACLs ‘*’ on the downgrade path. > > - Wrt naming, wildcard suffixed ACLs sound reasonable to me until > > someone raise a major objection. > > > > > > > > Let me know your thoughts. Looking forward to the next iteration. > > > > > > Best, > > > > Piyush > > > > > > > > > > > > Piyush Vijay > > > > On Wed, May 2, 2018 at 1:33 PM, Andy Coates <a...@confluent.io> wrote: > > > > > > Perhaps there is a simpler way. It seems like the resource that > people > > > really want to use prefixed ACLs with is topic names. Because topic > names > > > can't contain "*", there is no ambiguity there, either. I think maybe > we > > > should restrict the prefix handling to topic resources. > > > > > > The KIP should cover consumer groups for sure, otherwise we're back to > the > > > situation users need to know, up front, the set of consumer groups an > > > KStreams topology is going to use. > > > > > > I'm assuming transactional producer Ids would be the same. > > > > > > The cleanest solution would be to restrict the characters in group and > > > transaction producer ids, but that's a breaking change that might > affect a > > > lot of people. > > > > > > Another solution might be to add a protocol version to the `Acl` type, > (not > > > the current `version` field used for optimistic concurrency control), > > > defaulting it to version 1 if it is not present, and releasing this > change > > > as version 2. This would at least allow us to leave the version 1 ACLs > > > as-is, (which makes for a cleaner storey if a cluster is downgraded). > There > > > is then potential to escape '*' when writing version 2 ACLs. (And > introduce > > > code to check for supported ACL versions going forward). Though... how > do > > > we escape? If the consumer group is free form, then any escape > sequence is > > > also valid. Aren't there metrics that use the group name? If there > are, > > > I'd of thought we'd need to restrict the char set anyway. > > > > > > On 2 May 2018 at 18:57, charly molter <charly.mol...@gmail.com> wrote: > > > > > > > The fact that consumer groups and transactionalProducerId don't have > a > > > > strict format is problematic (we have problems with people with empty > > > > spaces at the end of their consumer group for example). > > > > With 2.0 around the corner and the possibility to fix these errors > from > > > the > > > > past should we create a KIP to restrict these names like we do for > > > topics? > > > > > > > > Thanks! > > > > Charly > > > > > > > > On Wed, May 2, 2018 at 6:22 PM Colin McCabe <cmcc...@apache.org> > wrote: > > > > > > > > > Hi Piyush, > > > > > > > > > > Thanks for the KIP! It seems like it will be really useful. > > > > > > > > > > As Rajini commented, the names for some resources (such as consumer > > > > > groups) can include stars. So your consumer group might be named > > > "foo*". > > > > > We need a way of explicitly referring to that consumer group name, > > > rather > > > > > than to the foo prefix. A simple way would be to escape the star > with > > > a > > > > > backslash: "foo\*" During the software upgrade process, we also > need > > > to > > > > > translate all ACLs that refer to "foo*" into "foo\*". Otherwise, > the > > > > > upgrade could create a security hole by granting more access than > the > > > > > administrator intended. > > > > > > > > > > Perhaps there is a simpler way. It seems like the resource that > people > > > > > really want to use prefixed ACLs with is topic names. Because > topic > > > > names > > > > > can't contain "*", there is no ambiguity there, either. I think > maybe > > > we > > > > > should restrict the prefix handling to topic resources. > > > > > > > > > > Are the new "getMatchingAcls" methods needed? It seems like they > break > > > > > encapsulation. All that the calling code really needs to do is > call > > > > > Authorizer#authorize(session, operation, resource). The authorizer > > > knows > > > > > that if it has an ACL allowing access to topics starting with > "foo" and > > > > > someone calls authorize(foobar), it should allow access. It's not > > > > > necessary for the calling code to know exactly what the rules are > for > > > > > authorization. The Authorizer#getAcls, APIs are only needed when > the > > > > > AdminClient wants to list ACLs. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > > > > > > On Wed, May 2, 2018, at 03:31, Rajini Sivaram wrote: > > > > > > Hi Piyush, > > > > > > > > > > > > Thanks for the KIP for this widely requested feature. A few > > > > > > comments/questions: > > > > > > > > > > > > 1. Some resource names can contain '*'. For example, consumer > groups > > > or > > > > > > transactional ids. I am wondering whether we need to restrict > > > > characters > > > > > > for these entities or provide a way to distinguish wildcarded > > > resource > > > > > from > > > > > > a resource containing the wildcard character. > > > > > > > > > > > > 2. I am not sure we want to do wildcarded principals. It feels > like a > > > > > > workaround in the absence of user groups. In the longer term, I > think > > > > > > groups (without wildcards) would be a better option to configure > ACLs > > > > for > > > > > > groups of users, rather than building user principals which have > > > common > > > > > > prefixes. In the shorter term, since a configurable > PrincipalBuilder > > > is > > > > > > used to build the principal used for authorization, perhaps > using the > > > > > > prefix itself as the principal would work (unless different > quotas > > > are > > > > > > required for the full user name)? User principal strings take > > > different > > > > > > formats for different security protocols (eg. CN=xxx,O=org,C=UK > for > > > > SSL) > > > > > > and simple prefixing isn't probably the right grouping in many > cases. > > > > > > > > > > > > 3. I am assuming we don't want to do wildcarded hosts in this KIP > > > since > > > > > > wildcard suffix doesn't really work for hosts. > > > > > > > > > > > > 4. It may be worth excluding delegation token ACLs from using > > > prefixed > > > > > > wildcards since it doesn't make much sense. > > > > > > > > > > > > Regards, > > > > > > > > > > > > Rajini > > > > > > > > > > > > > > > > > > On Wed, May 2, 2018 at 2:05 AM, Stephane Maarek < > > > > > > steph...@simplemachines.com.au> wrote: > > > > > > > > > > > > > Hi, thanks for this badly needed feature > > > > > > > > > > > > > > 1) Why introduce two new APIs in authorizer instead of > replacing > > > the > > > > > > > implementation for simple ACL authorizer with adding the > wildcard > > > > > > > capability? > > > > > > > > > > > > > > 2) is there an impact to performance as now we're evaluating > more > > > > > rules ? A > > > > > > > while back I had evaluated the concept of cached Acl result so > > > > swallow > > > > > the > > > > > > > cost of computing an authorisation cost once and then doing in > > > memory > > > > > > > lookups. CF: https://issues.apache.org/jira/browse/KAFKA-5261 > > > > > > > > > > > > > > 3) is there any need to also extend this to consumer group > > > resources > > > > ? > > > > > > > > > > > > > > 4) create topics KIP as recently moved permissions out of > Cluster > > > > into > > > > > > > Topic. Will wildcard be supported for this action too? > > > > > > > > > > > > > > Thanks a lot for this ! > > > > > > > > > > > > > > On Wed., 2 May 2018, 1:37 am Ted Yu, <yuzhih...@gmail.com> > wrote: > > > > > > > > > > > > > > > w.r.t. naming, we can keep wildcard and drop 'prefixed' (or > > > > > 'suffixed') > > > > > > > > since the use of regex would always start with non-wildcard > > > > portion. > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > > > > > On Tue, May 1, 2018 at 12:13 PM, Andy Coates < > a...@confluent.io> > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Piyush, > > > > > > > > > > > > > > > > > > Can you also document in the Compatibility section what > would > > > > > happen > > > > > > > > should > > > > > > > > > the cluster be upgraded, wildcard-suffixed ACLs are added, > and > > > > > then the > > > > > > > > > cluster is rolled back to the previous version. On > downgrade > > > the > > > > > > > partial > > > > > > > > > wildcard ACLs will be treated as literals and hence never > match > > > > > > > anything. > > > > > > > > > This is fine for ALLOW ACLs, but some might consider this a > > > > > security > > > > > > > hole > > > > > > > > > if a DENY ACL is ignored, so this needs to be documented, > both > > > in > > > > > the > > > > > > > KIP > > > > > > > > > and the final docs. > > > > > > > > > > > > > > > > > > For some reason I find the term 'prefixed wildcard ACLs' > easier > > > > to > > > > > grok > > > > > > > > > than 'wildcard suffixed ACLs'. Probably because in the > former > > > the > > > > > > > > > 'wildcard' term comes after the positional adjective, which > > > > > matches the > > > > > > > > > position of the wildcard char in the resource name, i.e. > > > "some*". > > > > > It's > > > > > > > > > most likely a person thing, but I thought I'd mention it as > > > > naming > > > > > is > > > > > > > > > important when it comes to making this initiative for > users. > > > > > > > > > > > > > > > > > > On 1 May 2018 at 19:57, Andy Coates <a...@confluent.io> > wrote: > > > > > > > > > > > > > > > > > > > Hi Piyush, > > > > > > > > > > > > > > > > > > > > Thanks for raising this KIP - it's very much appreciated. > > > > > > > > > > > > > > > > > > > > I've not had chance to digest it yet, but... > > > > > > > > > > > > > > > > > > > > 1. you might want to add details of how the internals of > the > > > > > > > > > > `getMatchingAcls` is implemented. We'd want to make sure > the > > > > > > > complexity > > > > > > > > > of > > > > > > > > > > the operation isn't adversely affected. > > > > > > > > > > 2. You might want to be more explicit that the length of > a > > > > prefix > > > > > > > does > > > > > > > > > not > > > > > > > > > > play a part in the `authorize` call, e.g. given ACLs > {DENY, > > > > > some.*}, > > > > > > > > > {ALLOW, > > > > > > > > > > some.prefix.*}, the longer, i.e. more specific, allow ACL > > > does > > > > > _not_ > > > > > > > > > > override the more general deny ACL. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Andy > > > > > > > > > > > > > > > > > > > > On 1 May 2018 at 16:59, Ron Dagostino <rndg...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> Hi Piyush. I appreciated your talk at Kafka Summit and > > > > > appreciate > > > > > > > the > > > > > > > > > KIP > > > > > > > > > >> -- thanks. > > > > > > > > > >> > > > > > > > > > >> Could you explain these mismatching references? Near > the > > > top > > > > > of the > > > > > > > > KIP > > > > > > > > > >> you refer to these proposed new method signatures: > > > > > > > > > >> > > > > > > > > > >> def getMatchingAcls(resource: Resource): Set[Acl] > > > > > > > > > >> def getMatchingAcls(principal: KafkaPrincipal): > > > Map[Resource, > > > > > > > > Set[Acl]] > > > > > > > > > >> > > > > > > > > > >> But near the bottom of the KIP you refer to different > method > > > > > > > > > >> signatures that don't seem to match the above ones: > > > > > > > > > >> > > > > > > > > > >> getMatchingAcls(topicRegex) > > > > > > > > > >> getMatchingAcls(principalRegex) > > > > > > > > > >> > > > > > > > > > >> Ron > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> On Tue, May 1, 2018 at 11:48 AM, Ted Yu < > > > yuzhih...@gmail.com> > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >> > The KIP was well written. Minor comment on formatting: > > > > > > > > > >> > > > > > > > > > > >> > https://github.com/apache/kafka/blob/trunk/core/src/ > > > > > > > > > >> > main/scala/kafka/admin/AclCommand.scala > > > > > > > > > >> > to > > > > > > > > > >> > > > > > > > > > > >> > Leave space between the URL and 'to' > > > > > > > > > >> > > > > > > > > > > >> > Can you describe changes for the AdminClient ? > > > > > > > > > >> > > > > > > > > > > >> > Thanks > > > > > > > > > >> > > > > > > > > > > >> > On Tue, May 1, 2018 at 8:12 AM, Piyush Vijay < > > > > > > > > piyushvij...@gmail.com> > > > > > > > > > >> > wrote: > > > > > > > > > >> > > > > > > > > > > >> > > Hi all, > > > > > > > > > >> > > > > > > > > > > > >> > > I just opened a KIP to add support for wildcard > suffixed > > > > > ACLs. > > > > > > > > This > > > > > > > > > is > > > > > > > > > >> > one > > > > > > > > > >> > > of the feature I talked about in my Kafka summit > talk > > > and > > > > we > > > > > > > > > promised > > > > > > > > > >> to > > > > > > > > > >> > > upstream it :) > > > > > > > > > >> > > > > > > > > > > > >> > > The details are here - > > > > > > > > > >> > > https://cwiki.apache.org/ > confluence/display/KAFKA/KIP- > > > > > > > > > >> > > 290%3A+Support+for+wildcard+suffixed+ACLs > > > > > > > > > >> > > > > > > > > > > > >> > > There is an open question about the way to add the > > > support > > > > > in > > > > > > > the > > > > > > > > > >> > > AdminClient, which I can discuss here in more detail > > > once > > > > > > > everyone > > > > > > > > > has > > > > > > > > > >> > > taken a first look at the KIP. > > > > > > > > > >> > > > > > > > > > > > >> > > Looking forward to discuss the change. > > > > > > > > > >> > > > > > > > > > > > >> > > Best, > > > > > > > > > >> > > Piyush Vijay > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Charly Molter > > > > > > > >