Hi Parth The design looks good, a few minor comments below. Since I just started looking into the discussion and many previous discussions I may missed, I'm sorry if these comments had be discussed.
1. About SimpleAclAuthorizer (SimpleAuthorizer): a. As my understanding, I think there should only one type privilege(allow/deny) of a topic on a principle, or we make it deny > allow. For example, acl_1 " host1 -> group1-> user1 -> read->allow" and acl_2 " host1-> group1 -> user1 ->read->deny", if the two acls are for a same topic, it may be hard to understand, do you think it's necessary to add some details about this to wiki. b. And when we do authorize a user on a topic, we may should check user's user level acl first, then check user's group level acl, finally we check the host level and default level acl. do you think it's necessary we add some contents like these to wiki. For example, "host1 -> group1-> user1" > "host1 -> group1" > "host1" 2.About SimpleAclAuthorizer (Acl Json will be stored in zookeeper) a. It may be better to make acl json stored hierarchily. It may be easy to search and do authorize. For example, when we authorize a user, we only need user related acls. b. I found one acl may contains multi-principles, multi-operations and multi-hosts, I'm strongly agreed with we provide api like these, but the acls stored in zookeeper or memory we may better to separate to one-principle, one-operation and one host. So we could make sure there are not many acls with same meaning and make acl management easily. Regards Dapeng -----Original Message----- From: Jun Rao [mailto:j...@confluent.io] Sent: Monday, April 27, 2015 5:02 AM To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-11- Authorization design for kafka security A few more minor comments. 100. To make it clear, perhaps we should rename the resource "group" to consumer-group. We can probably make the same change in CLI as well so that it's not confused with user group. 101. Currently, create is only at the cluster level. Should it also be at topic level? For example, perhaps it's useful to allow only user X to create topic X. Thanks, Jun On Sun, Apr 26, 2015 at 12:36 AM, Gwen Shapira <gshap...@cloudera.com> wrote: > Thanks for clarifying, Parth. I think you are taking the right > approach here. > > On Fri, Apr 24, 2015 at 11:46 AM, Parth Brahmbhatt > <pbrahmbh...@hortonworks.com> wrote: > > Sorry Gwen, completely misunderstood the question :-). > > > > * Does everyone have the privilege to create a new Group and use it > > to consume from Topics he's already privileged on? > > Yes in current proposal. I did not see an API to create > > group > but if you > > have a READ permission on a TOPIC and WRITE permission on that Group > > you are free to join and consume. > > > > > > * Will the CLI tool be used to manage group membership too? > > Yes and I think that means I need to add ―group. Updating > > the > KIP. Thanks > > for pointing this out. > > > > * Groups are kind of ephemeral, right? If all consumers in the group > > disconnect the group is gone, AFAIK. Do we preserve the ACLs? Or do > > we treat the new group as completely new resource? Can we create > > ACLs before the group exists, in anticipation of it getting created? > > I have considered any auto delete and auto create as out of > scope for the > > first release. So Right now I was going with preserving the acls. Do > > you see any issues with this? Auto deleting would mean authorizer > > will now have to get into implementation details of kafka which I > > was trying to avoid. > > > > Thanks > > Parth > > > > On 4/24/15, 11:33 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote: > > > >>We are not talking about same Groups :) > >> > >>I meant, Groups of consumers (which KIP-11 lists as a separate > >>resource in the Privilege table) > >> > >>On Fri, Apr 24, 2015 at 11:31 AM, Parth Brahmbhatt > >><pbrahmbh...@hortonworks.com> wrote: > >>> I see Groups as something we can add incrementally in the current > model. > >>> The acls take principalType: name so groups can be represented as > group: > >>> groupName. We are not managing group memberships anywhere in kafka > >>> and > I > >>> don’t see the need to do so. > >>> > >>> So for a topic1 using the CLI an admin can add an acl to grant > >>> access > to > >>> group:kafka-test-users. > >>> > >>> The authorizer implementation can have a plugin to map > >>>authenticated user to groups ( This is how hadoop and storm > >>>works). The plugin could be mapping user to linux/ldap/active > >>>directory groups but that is again upto the implementation. > >>> > >>> What we are offering is an interface that is extensible so these > >>>features can be added incrementally. I can add support for this in > >>>the first release but don’t necessarily see why this would be > >>>absolute necessity. > >>> > >>> Thanks > >>> Parth > >>> > >>> On 4/24/15, 11:00 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote: > >>> > >>>>Thanks. > >>>> > >>>>One more thing I'm missing in the KIP is details on the Group > >>>>resource (I think we discussed this and it was just not fully updated): > >>>> > >>>>* Does everyone have the privilege to create a new Group and use > >>>>it to consume from Topics he's already privileged on? > >>>>* Will the CLI tool be used to manage group membership too? > >>>>* Groups are kind of ephemeral, right? If all consumers in the > >>>>group disconnect the group is gone, AFAIK. Do we preserve the > >>>>ACLs? Or do we treat the new group as completely new resource? Can > >>>>we create ACLs before the group exists, in anticipation of it getting > >>>>created? > >>>> > >>>>Its all small details, but it will be difficult to implement > >>>>KIP-11 without knowing the answers :) > >>>> > >>>>Gwen > >>>> > >>>> > >>>>On Fri, Apr 24, 2015 at 9:58 AM, Parth Brahmbhatt > >>>><pbrahmbh...@hortonworks.com> wrote: > >>>>> You are right, moved it to the default implementation section. > >>>>> > >>>>> Thanks > >>>>> Parth > >>>>> > >>>>> On 4/24/15, 9:52 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote: > >>>>> > >>>>>>Sample ACL JSON and Zookeeper is in public API, but I thought it > >>>>>>is part of DefaultAuthorizer (Since Sentry and Argus won't be > >>>>>>using Zookeeper). > >>>>>>Am I wrong? Or is it the KIP? > >>>>>> > >>>>>>On Fri, Apr 24, 2015 at 9:49 AM, Parth Brahmbhatt > >>>>>><pbrahmbh...@hortonworks.com> wrote: > >>>>>>> Thanks for clarifying Gwen, KIP updated. > >>>>>>> > >>>>>>> I tried to make the distinction by creating a section for all > public > >>>>>>>APIs > >>>>>>> > >>>>>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizat > >>>>>>>io > >>>>>>>n+ > >>>>>>>In > >>>>>>> > >>>>>>>terface#KIP-11-AuthorizationInterface-PublicInterfacesandclasse > >>>>>>>s > >>>>>>> > >>>>>>> Let me know if you think there is a better way to reflect this. > >>>>>>> > >>>>>>> Thanks > >>>>>>> Parth > >>>>>>> > >>>>>>> On 4/24/15, 9:37 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote: > >>>>>>> > >>>>>>>>+1 (non-binding) > >>>>>>>> > >>>>>>>>Two nitpicks for the wiki: > >>>>>>>>* Heartbeat is probably a READ and not CLUSTER operation. I'm > pretty > >>>>>>>>sure new consumers need it to be part of a consumer group. > >>>>>>>>* Can you clearly separate which parts are the API (common to > >>>>>>>>every > >>>>>>>>Authorizer) and which parts are DefaultAuthorizer implementation? > It > >>>>>>>>will make reviews and Authorizer implementations a bit easier > >>>>>>>>to know exactly which is which. > >>>>>>>> > >>>>>>>>Gwen > >>>>>>>> > >>>>>>>>On Fri, Apr 24, 2015 at 9:28 AM, Parth Brahmbhatt > >>>>>>>><pbrahmbh...@hortonworks.com> wrote: > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> I would like to open KIP-11 for voting. > >>>>>>>>> > >>>>>>>>> Thanks > >>>>>>>>> Parth > >>>>>>>>> > >>>>>>>>> On 4/22/15, 1:56 PM, "Parth Brahmbhatt" > >>>>>>>>><pbrahmbh...@hortonworks.com> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>>Hi Jeff, > >>>>>>>>>> > >>>>>>>>>>Thanks a lot for the review. I think you have a valid point > >>>>>>>>>>about acls being duplicated and the simplest solution would > >>>>>>>>>>be to modify > acls > >>>>>>>>>>class > >>>>>>>>>>so they hold a set of principals instead of single > >>>>>>>>>>principal. i.e > >>>>>>>>>> > >>>>>>>>>><user_a,user_b> has <READ,WRITE,DESCRIBE> Permissions on > >>>>>>>>>><Topic1> from <Host1, Host2, Host3>. > >>>>>>>>>> > >>>>>>>>>>I think the evaluation order only matters for the > >>>>>>>>>>permissionType which is Deny acls should be evaluated before > >>>>>>>>>>allow acls. To give you an example suppose we have following > >>>>>>>>>>acls > >>>>>>>>>> > >>>>>>>>>>acl1 -> user1 is allowed to READ from all hosts. > >>>>>>>>>>acl2 -> host1 is allowed to READ regardless of who is the user. > >>>>>>>>>>acl3 -> host2 is allowed to READ regardless of who is the user. > >>>>>>>>>> > >>>>>>>>>>acl4 -> user1 is denied to READ from host1. > >>>>>>>>>> > >>>>>>>>>>As stated in the KIP we first evaluate DENY so if user1 > >>>>>>>>>>tries to access from host1 he will be denied(acl4), even > >>>>>>>>>>though both user1 and > >>>>>>>>>>host1 > >>>>>>>>>>has > >>>>>>>>>>acl’s for allow with wildcards (acl1, acl2). > >>>>>>>>>>If user1 tried to READ from host2 , the action will be > >>>>>>>>>>allowed > and > >>>>>>>>>>it > >>>>>>>>>>does > >>>>>>>>>>not matter if we match acl3 or acl1 so I don’t think the > >>>>>>>>>>evaluation order matters here. > >>>>>>>>>> > >>>>>>>>>>“Will people actually use hosts with users?” I really don’t > >>>>>>>>>>know but given ACl’s are part of our Public APIs I thought > >>>>>>>>>>it is better to try and cover more use cases. If others > >>>>>>>>>>think this extra complexity is not > worth > >>>>>>>>>>the > >>>>>>>>>>value its adding please raise your concerns so we can > >>>>>>>>>>discuss if it should be removed from the acl structure. Note > >>>>>>>>>>that even in absence of hosts from ACL users will still be > >>>>>>>>>>able to whitelist/blacklist host as long as we start > >>>>>>>>>>supporting principalType = “host”, easy to add and can be > an > >>>>>>>>>>incremental improvement. They will however loose the ability > >>>>>>>>>>to restrict access to users just from a set of hosts. > >>>>>>>>>> > >>>>>>>>>>We agreed to offer a CLI to overcome the JSON acl config > >>>>>>>>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authori > >>>>>>>>>>za > >>>>>>>>>>ti > >>>>>>>>>>on > >>>>>>>>>>+I > >>>>>>>>>>n > >>>>>>>>>>terface#KIP-11-AuthorizationInterface-AclManagement(CLI). I > >>>>>>>>>>still like Jsons but that probably has something to do with > >>>>>>>>>>me being a developer :-). > >>>>>>>>>> > >>>>>>>>>>Thanks > >>>>>>>>>>Parth > >>>>>>>>>> > >>>>>>>>>>On 4/22/15, 11:38 AM, "Jeff Holoman" <jholo...@cloudera.com> > >>>>>>>>>>wrote: > >>>>>>>>>> > >>>>>>>>>>>Parth, > >>>>>>>>>>> > >>>>>>>>>>>This is a long thread, so trying to keep up here, sorry if > >>>>>>>>>>>this has been covered before. First, great job on the KIP > >>>>>>>>>>>proposal and work so far. > >>>>>>>>>>> > >>>>>>>>>>>Are we sure that we want to tie host level access to a > >>>>>>>>>>>given user? > >>>>>>>>>>>My > >>>>>>>>>>>understanding is that the ACL will be (omitting some > >>>>>>>>>>>fields) > >>>>>>>>>>> > >>>>>>>>>>>user_a, host1, host2, host3 user_b, host1, host2, host3 > >>>>>>>>>>> > >>>>>>>>>>>So there would potentially be a lot of redundancy in the > configs. > >>>>>>>>>>>Does > >>>>>>>>>>>it > >>>>>>>>>>>make sense to have hosts be at the same level as principal > >>>>>>>>>>>in > the > >>>>>>>>>>>hierarchy? This way you could just blanket the allowed / > >>>>>>>>>>>denied hosts and only have to worry about the users. So if > >>>>>>>>>>>you follow this, then > >>>>>>>>>>> > >>>>>>>>>>>we can wildcard the user so we can have a separate list of > >>>>>>>>>>>just host-based access. What's the order that the perms > >>>>>>>>>>>would be evaluated if a there was more than one match on a > >>>>>>>>>>>principal ? > >>>>>>>>>>> > >>>>>>>>>>>Is the thought that there wouldn't usually be much overlap > >>>>>>>>>>>on hosts? > >>>>>>>>>>>I > >>>>>>>>>>>guess I can imagine a scenario where I want to > >>>>>>>>>>>offline/online access to a particular hosts or set of hosts > >>>>>>>>>>>and if there was overlap, I'm doing a bunch of alter > >>>>>>>>>>>commands for just a single host. Maybe this is > too > >>>>>>>>>>>contrived > >>>>>>>>>>>an example? > >>>>>>>>>>> > >>>>>>>>>>>I agree that having this level of granularity gives > >>>>>>>>>>>flexibility but I wonder if people will actually use it and > >>>>>>>>>>>not just * the hosts for a given user and create separate > >>>>>>>>>>>"global" list as i mentioned above? > >>>>>>>>>>> > >>>>>>>>>>>The only other system I know of that ties users with hosts > >>>>>>>>>>>for access is MySql and I don't love that model. Companies > >>>>>>>>>>>usually standardize on group authorization anyway, are we > >>>>>>>>>>>complicating that issue with the inclusion of hosts > >>>>>>>>>>>attached to users? Additionally I worry about the debt of > >>>>>>>>>>>big JSON configs in the first place, most non-developers > >>>>>>>>>>>find them non-intuitive already, so anything to ease this I > >>>>>>>>>>>think would be beneficial. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>Thanks > >>>>>>>>>>> > >>>>>>>>>>>Jeff > >>>>>>>>>>> > >>>>>>>>>>>On Wed, Apr 22, 2015 at 2:22 PM, Parth Brahmbhatt < > >>>>>>>>>>>pbrahmbh...@hortonworks.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Sorry I missed your last questions. I am +0 on adding > >>>>>>>>>>>>―host option for ―list, we could add it for symmetry. > >>>>>>>>>>>>Again if this is only a CLI change it can be added later > >>>>>>>>>>>>if you mean adding this in authorizer interface then we > >>>>>>>>>>>>should make a decision now. > >>>>>>>>>>>> > >>>>>>>>>>>> Given a choice I would like to actually keep only one > >>>>>>>>>>>>option which is resource based get (remove even the get > >>>>>>>>>>>>based on principal). I see those (getAcl for principal or > >>>>>>>>>>>>host) as special filtering case which can easily be > >>>>>>>>>>>>achieved by a third party tool by doing "list all topics" > >>>>>>>>>>>>and > >>>>>>>>>>>>calling > >>>>>>>>>>>> getAcls for each topic and applying filtering logic on that. > I > >>>>>>>>>>>>really > >>>>>>>>>>>> don’t see the need to make those first class citizens of > >>>>>>>>>>>>the authorizer interface given these kind of queries will > >>>>>>>>>>>>be issued outside > of > >>>>>>>>>>>>broker > >>>>>>>>>>>>JVM > >>>>>>>>>>>> so they will not benefit from the caching and because the > >>>>>>>>>>>>storage will be indexed on resource both these options > >>>>>>>>>>>>even as a first class API will just scan all topic acls > >>>>>>>>>>>>and apply filtering logic. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks > >>>>>>>>>>>> Parth > >>>>>>>>>>>> > >>>>>>>>>>>> On 4/22/15, 11:08 AM, "Parth Brahmbhatt" > >>>>>>>>>>>><pbrahmbh...@hortonworks.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> >Please see all the available options here > >>>>>>>>>>>> > > >>>>>>>>>>>> > >>>>>>>>>>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Autho > >>>>>>>>>>>>ri > >>>>>>>>>>>>za > >>>>>>>>>>>>ti > >>>>>>>>>>>>on > >>>>>>>>>>>>+ > >>>>>>>>>>>>I > >>>>>>>>>>>> >nterface#KIP-11-AuthorizationInterface-AclManagement(CLI > >>>>>>>>>>>> >) . I > >>>>>>>>>>>>think > >>>>>>>>>>>>it > >>>>>>>>>>>> >covers both hosts and operations and allows to specify a > >>>>>>>>>>>> >list > >>>>>>>>>>>>for > >>>>>>>>>>>>both. > >>>>>>>>>>>> > > >>>>>>>>>>>> >Thanks > >>>>>>>>>>>> >Parth > >>>>>>>>>>>> > > >>>>>>>>>>>> >From: Tom Graves > >>>>>>>>>>>><tgraves...@yahoo.com<mailto:tgraves...@yahoo.com>> > >>>>>>>>>>>> >Reply-To: Tom Graves > >>>>>>>>>>>><tgraves...@yahoo.com<mailto:tgraves...@yahoo.com>> > >>>>>>>>>>>> >Date: Wednesday, April 22, 2015 at 11:02 AM > >>>>>>>>>>>> >To: Parth Brahmbhatt > >>>>>>>>>>>> > >>>>>>>>>>>>><pbrahmbh...@hortonworks.com<mailto: > pbrahmbh...@hortonworks.com > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>, > >>>>>>>>>>>> >"dev@kafka.apache.org<mailto:dev@kafka.apache.org>" > >>>>>>>>>>>> ><dev@kafka.apache.org<mailto:dev@kafka.apache.org>> > >>>>>>>>>>>> >Subject: Re: [DISCUSS] KIP-11- Authorization design for > >>>>>>>>>>>> >kafka > >>>>>>>>>>>>security > >>>>>>>>>>>> > > >>>>>>>>>>>> >Thanks for the explanations Parth. > >>>>>>>>>>>> > > >>>>>>>>>>>> >On the configs questions, the way I see it is its more > >>>>>>>>>>>> >likely > >>>>>>>>>>>>to > >>>>>>>>>>>> >accidentally give everyone access, especially since you > >>>>>>>>>>>> >have > >>>>>>>>>>>>to > >>>>>>>>>>>>run > >>>>>>>>>>>>a > >>>>>>>>>>>> >separate command to change the acls. If there was some > >>>>>>>>>>>> >config > >>>>>>>>>>>>for > >>>>>>>>>>>> >defaults, a cluster admin could change that to be nobody > >>>>>>>>>>>> >or > >>>>>>>>>>>>certain > >>>>>>>>>>>>set > >>>>>>>>>>>> >of users, then grant others permissions. This would > >>>>>>>>>>>> >also > >>>>>>>>>>>>remove > >>>>>>>>>>>>the > >>>>>>>>>>>>race > >>>>>>>>>>>> >between commands. This is something you can always add > >>>>>>>>>>>> >later > >>>>>>>>>>>>though > >>>>>>>>>>>>if > >>>>>>>>>>>> >people request it. > >>>>>>>>>>>> > > >>>>>>>>>>>> >So in kafka-acl.sh how do I actually tell it what the > >>>>>>>>>>>>operation > >>>>>>>>>>>>is? > >>>>>>>>>>>> >kafka-acl.sh --topic testtopic --add --grandprincipal > >>>>>>>>>>>>user:joe,user:kate > >>>>>>>>>>>> > > >>>>>>>>>>>> >where does READ, WRITE, etc go? Can specify as a list > >>>>>>>>>>>> >so I > >>>>>>>>>>>>don't > >>>>>>>>>>>>have > >>>>>>>>>>>>to > >>>>>>>>>>>> >run this a bunch of times for each. > >>>>>>>>>>>> > > >>>>>>>>>>>> >Do you want to have a --host option for --list so that > >>>>>>>>>>>> >admins > >>>>>>>>>>>>could > >>>>>>>>>>>>see > >>>>>>>>>>>> >what acls apply to specific host(s)? > >>>>>>>>>>>> > > >>>>>>>>>>>> >Tom > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> >On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt > >>>>>>>>>>>> > >>>>>>>>>>>>><pbrahmbh...@hortonworks.com<mailto: > pbrahmbh...@hortonworks.com > >>>>>>>>>>>>>>> > >>>>>>>>>>>>wrote: > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> >FYI, I have modified the KIP to include group as > >>>>>>>>>>>> >resource. In > >>>>>>>>>>>>order > >>>>>>>>>>>>to > >>>>>>>>>>>> >access “joinGroup” and “commitOFfset” APIs the user will > >>>>>>>>>>>> >need > >>>>>>>>>>>>a > >>>>>>>>>>>>read > >>>>>>>>>>>> >permission on topic and WRITE permission on group. > >>>>>>>>>>>> > > >>>>>>>>>>>> >I plan to open a VOTE thread by noon if there are no > >>>>>>>>>>>> >more > >>>>>>>>>>>>concerns. > >>>>>>>>>>>> > > >>>>>>>>>>>> >Thanks > >>>>>>>>>>>> >Parth > >>>>>>>>>>>> > > >>>>>>>>>>>> >On 4/22/15, 9:03 AM, "Tom Graves" > >>>>>>>>>>>> > >>>>>>>>>>>>><tgraves...@yahoo.com.INVALID<mailto: > tgraves...@yahoo.com.INVAL > >>>>>>>>>>>>>ID > >>>>>>>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > > >>>>>>>>>>>> >>Hey everyone, > >>>>>>>>>>>> >>Sorry to jump in on the conversation so late. I'm new > >>>>>>>>>>>> >>to > >>>>>>>>>>>>Kafka. > >>>>>>>>>>>>I'll > >>>>>>>>>>>> >>apologize in advance if you have already covered some > >>>>>>>>>>>> >>of my > >>>>>>>>>>>>questions. I > >>>>>>>>>>>> >>read through the wiki and had some comments and questions. > >>>>>>>>>>>> >>1) public enum Operation needs EDIT changed to ALTER > >>>>>>>>>>>> > > >>>>>>>>>>>> >> Done. > >>>>>>>>>>>> > > >>>>>>>>>>>> >>2) Does the Authorizer class need a setAcls? Rather > >>>>>>>>>>>> >>then > >>>>>>>>>>>>just > >>>>>>>>>>>>add > >>>>>>>>>>>>to > >>>>>>>>>>>>be > >>>>>>>>>>>> >>able to set to explicit list and overwrite what was there? > I > >>>>>>>>>>>>see > >>>>>>>>>>>>the > >>>>>>>>>>>> >>kafka-acl.sh lists a removeall so I guess you could do > >>>>>>>>>>>>removeall > >>>>>>>>>>>>and > >>>>>>>>>>>>then > >>>>>>>>>>>> >>add. I also don't see a removeall in the Authorizer > >>>>>>>>>>>> >>class, > >>>>>>>>>>>>is > >>>>>>>>>>>>it > >>>>>>>>>>>>going > >>>>>>>>>>>> >>to loop through them all to remove each one? > >>>>>>>>>>>> > > >>>>>>>>>>>> > There is an overloaded version of removeAcls in the > >>>>>>>>>>>>interface > >>>>>>>>>>>>that > >>>>>>>>>>>> >takes > >>>>>>>>>>>> >in resource as the only input and as described in the > >>>>>>>>>>>> >javadoc > >>>>>>>>>>>>all > >>>>>>>>>>>>the > >>>>>>>>>>>>acls > >>>>>>>>>>>> >attached to that resource will be deleted. To cover the > setAcl > >>>>>>>>>>>>use > >>>>>>>>>>>>case > >>>>>>>>>>>> >the caller can first call remove and then add. > >>>>>>>>>>>> > > >>>>>>>>>>>> >>3) Can someone tell me what the use case to do acls > >>>>>>>>>>>> >>based on > >>>>>>>>>>>>the > >>>>>>>>>>>>hosts? > >>>>>>>>>>>> >>I can see some possibilities just wondering if we can > >>>>>>>>>>>>concrete > >>>>>>>>>>>>ones > >>>>>>>>>>>>where > >>>>>>>>>>>> >>one user is allowed from one host but not another. > >>>>>>>>>>>> > > >>>>>>>>>>>> > I am not sure if I understand the question given the > >>>>>>>>>>>> > use > >>>>>>>>>>>>case > >>>>>>>>>>>>you > >>>>>>>>>>>> >described in your question is what we are trying to > >>>>>>>>>>>> >cover > with > >>>>>>>>>>>>use > >>>>>>>>>>>>of > >>>>>>>>>>>> >hosts in Acl. There are some additional use cases like > >>>>>>>>>>>> >“allow > >>>>>>>>>>>>access > >>>>>>>>>>>>to > >>>>>>>>>>>> >any user from host1,host2” but I think primarily it > >>>>>>>>>>>> >gives the > >>>>>>>>>>>>admins > >>>>>>>>>>>>the > >>>>>>>>>>>> >ability to define acls at a more granular level. > >>>>>>>>>>>> > > >>>>>>>>>>>> >>4) I'm a bit unclear how the "resource" works in the > >>>>>>>>>>>>Authorizer > >>>>>>>>>>>>class. > >>>>>>>>>>>> >>From what I see we have 2 resources - topics and cluster. > >>>>>>>>>>>>If I > >>>>>>>>>>>>want > >>>>>>>>>>>>to > >>>>>>>>>>>> >>add an acl to allow "joe" to CREATE for the cluster > >>>>>>>>>>>> >>then I > >>>>>>>>>>>>call > >>>>>>>>>>>>addAcls > >>>>>>>>>>>> >>with Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and > >>>>>>>>>>>>"cluster"? > >>>>>>>>>>>>What > >>>>>>>>>>>> >>if I want to call addAcls for DESCRIBE on a topic? Is > >>>>>>>>>>>> >>the > >>>>>>>>>>>>resource > >>>>>>>>>>>>then > >>>>>>>>>>>> >>"topic" or is it the topic name? > >>>>>>>>>>>> > > >>>>>>>>>>>> > We now have 3 resources(added group), please see the > >>>>>>>>>>>>updated > >>>>>>>>>>>>doc. > >>>>>>>>>>>>The > >>>>>>>>>>>> >CREATE acl that you described is correct. For any topic > >>>>>>>>>>>>operation > >>>>>>>>>>>>you > >>>>>>>>>>>> >should use topic name as the resource name and for group > >>>>>>>>>>>> >the > >>>>>>>>>>>>user > >>>>>>>>>>>>will > >>>>>>>>>>>> >provide groupId as resource name. > >>>>>>>>>>>> > > >>>>>>>>>>>> >>5) reassigning partitions is a CLUSTER_ACTION or superuser? > >>>>>>>>>>>>Its > >>>>>>>>>>>>not > >>>>>>>>>>>> >>totally clear to me the differences between these. > >>>>>>>>>>>> >>what > >>>>>>>>>>>>about > >>>>>>>>>>>>increasing > >>>>>>>>>>>> >># of partitions? > >>>>>>>>>>>> > > >>>>>>>>>>>> > I see this as an alter topic operation so it is at > >>>>>>>>>>>> > topic > >>>>>>>>>>>>level > >>>>>>>>>>>>and > >>>>>>>>>>>>the > >>>>>>>>>>>> >user must have alter permissions on topic. > >>>>>>>>>>>> > > >>>>>>>>>>>> >>6) groups are mentioned, are we supporting right away > >>>>>>>>>>>> >>or is > >>>>>>>>>>>>that > >>>>>>>>>>>>a > >>>>>>>>>>>>follow > >>>>>>>>>>>> >>on item? (is there going to be a kafka.supergroups) > >>>>>>>>>>>> > > >>>>>>>>>>>> > I think it can be a separate jira just for braking > >>>>>>>>>>>> > down > >>>>>>>>>>>>the > >>>>>>>>>>>>code > >>>>>>>>>>>> >review > >>>>>>>>>>>> >in smaller chunk. We will support it in first version > >>>>>>>>>>>> >but I > >>>>>>>>>>>>think > >>>>>>>>>>>>if > >>>>>>>>>>>>we > >>>>>>>>>>>> >can not do it for any reason that should not block a > >>>>>>>>>>>> >release > >>>>>>>>>>>>with > >>>>>>>>>>>>all > >>>>>>>>>>>>the > >>>>>>>>>>>> >other authZ work. We made deliberate design choices > >>>>>>>>>>>> >(like > >>>>>>>>>>>>introducing > >>>>>>>>>>>>a > >>>>>>>>>>>> >principalType in KafkaPrinciapl) to allow supporting > >>>>>>>>>>>> >groups > as > >>>>>>>>>>>>an > >>>>>>>>>>>> >incremental change. > >>>>>>>>>>>> > > >>>>>>>>>>>> >>7) Are there config options for setting acls when I > >>>>>>>>>>>> >>create > my > >>>>>>>>>>>>topic? > >>>>>>>>>>>>Or > >>>>>>>>>>>> >>do I have to create my topic and then run the > >>>>>>>>>>>> >>kafka-acl.sh > >>>>>>>>>>>>script > >>>>>>>>>>>>to > >>>>>>>>>>>>set > >>>>>>>>>>>> >>them? Although its very small, there would be possible > >>>>>>>>>>>> >>race > >>>>>>>>>>>>there > >>>>>>>>>>>>that > >>>>>>>>>>>> >>someone could start producing to topic before acls are set. > >>>>>>>>>>>> > > >>>>>>>>>>>> > We discussed this yesterday and we agreed to go with > >>>>>>>>>>>>kafka-acl.sh. > >>>>>>>>>>>>Yes > >>>>>>>>>>>> >there is a very very small window of vulnerability but I > think > >>>>>>>>>>>>that > >>>>>>>>>>>>really > >>>>>>>>>>>> >does not warrant to change the decision in this case. > >>>>>>>>>>>> > > >>>>>>>>>>>> >>8) are there configs for cluster level acl defaults? > >>>>>>>>>>>> >>Or > does > >>>>>>>>>>>>it > >>>>>>>>>>>>default > >>>>>>>>>>>> >>to superusers on bringing up new cluster and you have > >>>>>>>>>>>> >>to > >>>>>>>>>>>>modify > >>>>>>>>>>>>with > >>>>>>>>>>>>cli. > >>>>>>>>>>>> >>thanks,Tom > >>>>>>>>>>>> > > >>>>>>>>>>>> > No defaults, the default is superusers will have > >>>>>>>>>>>> > full > >>>>>>>>>>>>access. > >>>>>>>>>>>>I > >>>>>>>>>>>>don’t > >>>>>>>>>>>> >think making assumptions about ones security requirement > >>>>>>>>>>>>should > >>>>>>>>>>>>be > >>>>>>>>>>>>our > >>>>>>>>>>>> >burden. > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> On Tuesday, April 21, 2015 7:10 PM, Parth > >>>>>>>>>>>> >> Brahmbhatt > >>>>>>>>>>>> > >>>>>>>>>>>>>><pbrahmbh...@hortonworks.com<mailto: > pbrahmbh...@hortonworks.co > >>>>>>>>>>>>>>m> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>wrote: > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> I have added the notes to KIP-11 Open question sections. > >>>>>>>>>>>> >> > >>>>>>>>>>>> >>Thanks > >>>>>>>>>>>> >>Parth > >>>>>>>>>>>> >> > >>>>>>>>>>>> >>On 4/21/15, 4:49 PM, "Gwen Shapira" > >>>>>>>>>>>> >><gshap...@cloudera.com<mailto:gshap...@cloudera.com>> > wrote: > >>>>>>>>>>>> >> > >>>>>>>>>>>> >>>Adding my notes from today's call to the thread: > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>>** Deny or Allow all by default? We will add a > configuration > >>>>>>>>>>>>to > >>>>>>>>>>>> >>>control this. The configuration will default to > >>>>>>>>>>>> >>>“allow” for > >>>>>>>>>>>>backward > >>>>>>>>>>>> >>>compatibility. Security admins can set it to "deny" > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>>** Storing ACLs for default authorizers: We'll store > >>>>>>>>>>>> >>>them > in > >>>>>>>>>>>>ZK. > >>>>>>>>>>>>We'll > >>>>>>>>>>>> >>>support pointing the authorizer to any ZK. > >>>>>>>>>>>> >>>The use of ZK will be internal to the default authorizer. > >>>>>>>>>>>>Authorizer > >>>>>>>>>>>> >>>reads ACLs from cache every hour. We proposed having > >>>>>>>>>>>>mechanism > >>>>>>>>>>>> >>>(possibly via new ZK node) to tell broker to refresh > >>>>>>>>>>>> >>>the > >>>>>>>>>>>>cache > >>>>>>>>>>>> >>>immediately. > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>>** Support deny as permission type - we agreed to keep > this. > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>>** Mapping operations to API: We may need to add Group > >>>>>>>>>>>> >>>as a > >>>>>>>>>>>>resource, > >>>>>>>>>>>> >>>with JoinGroup and OffsetCommit require privilege on > >>>>>>>>>>>> >>>the > >>>>>>>>>>>>consumer > >>>>>>>>>>>> >>>group. > >>>>>>>>>>>> >>>This can be something we pass now and authorizers can > >>>>>>>>>>>>support > >>>>>>>>>>>>in > >>>>>>>>>>>> >>>future. - Jay will write specifics to the mailing list > >>>>>>>>>>>>discussion. > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps > >>>>>>>>>>>> >>><jay.kr...@gmail.com<mailto:jay.kr...@gmail.com>> wrote: > >>>>>>>>>>>> >>>> Following up on the KIP discussion. Two options for > >>>>>>>>>>>>authorizing > >>>>>>>>>>>> >>>>consumers > >>>>>>>>>>>> >>>> to read topic "t" as part of group "g": > >>>>>>>>>>>> >>>> 1. READ permission on resource /topic/t 2. READ > >>>>>>>>>>>> >>>>permission on resource /topic/t AND WRITE > >>>>>>>>>>>>permission > >>>>>>>>>>>>on > >>>>>>>>>>>> >>>>/group/g > >>>>>>>>>>>> >>>> > >>>>>>>>>>>> >>>> The advantage of (1) is that it is simpler. The > >>>>>>>>>>>>disadvantage > >>>>>>>>>>>>is > >>>>>>>>>>>>that > >>>>>>>>>>>> >>>>any > >>>>>>>>>>>> >>>> member of any group that reads from t can commit > >>>>>>>>>>>> >>>>offsets > >>>>>>>>>>>>as > >>>>>>>>>>>>any > >>>>>>>>>>>>other > >>>>>>>>>>>> >>>> member of a different group. This doesn't effect > >>>>>>>>>>>> >>>> data > >>>>>>>>>>>>security > >>>>>>>>>>>>(who > >>>>>>>>>>>> >>>>can > >>>>>>>>>>>> >>>> access what) but it is a bit of a management > >>>>>>>>>>>> >>>>issue--a > >>>>>>>>>>>>malicious > >>>>>>>>>>>>person > >>>>>>>>>>>> >>>>can > >>>>>>>>>>>> >>>> cause data loss or duplicates for another consumer > >>>>>>>>>>>> >>>>by > >>>>>>>>>>>>committing > >>>>>>>>>>>> >>>>offset. > >>>>>>>>>>>> >>>> > >>>>>>>>>>>> >>>> I think I favor (2) but it's worth it to think it > through. > >>>>>>>>>>>> >>>> > >>>>>>>>>>>> >>>> -Jay > >>>>>>>>>>>> >>>> > >>>>>>>>>>>> >>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt < > >>>>>>>>>>>> >>>> > >>>>>>>>>>>>pbrahmbh...@hortonworks.com<mailto:pbrahmbhatt@hortonworks > >>>>>>>>>>>>.com > >> > >>>>>>>>>>>> >>>>wrote: > >>>>>>>>>>>> >>>> > >>>>>>>>>>>> >>>>> Hey Jun, > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> Yes and we support wild cards for all acl entities > >>>>>>>>>>>>principal, > >>>>>>>>>>>>hosts > >>>>>>>>>>>> >>>>>and > >>>>>>>>>>>> >>>>> operation. > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> Thanks > >>>>>>>>>>>> >>>>> Parth > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> On 4/21/15, 9:06 AM, "Jun Rao" > >>>>>>>>>>>> >>>>><j...@confluent.io<mailto:j...@confluent.io>> wrote: > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> >Harsha, Parth, > >>>>>>>>>>>> >>>>> > > >>>>>>>>>>>> >>>>> >Thanks for the clarification. This makes sense. > >>>>>>>>>>>> >>>>> >Perhaps > >>>>>>>>>>>>we > >>>>>>>>>>>>can > >>>>>>>>>>>> >>>>>clarify the > >>>>>>>>>>>> >>>>> >meaning of those rules in the wiki. > >>>>>>>>>>>> >>>>> > > >>>>>>>>>>>> >>>>> >Related to this, it seems that we need to support > >>>>>>>>>>>>wildcard > >>>>>>>>>>>>in > >>>>>>>>>>>> >>>>>cli/request > >>>>>>>>>>>> >>>>> >protocol for topics? > >>>>>>>>>>>> >>>>> > > >>>>>>>>>>>> >>>>> >Jun > >>>>>>>>>>>> >>>>> > > >>>>>>>>>>>> >>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt > >>>>>>>>>>>> >>>>> >< > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>pbrahmbh...@hortonworks.com<mailto: > pbrahmbh...@hortonworks.com> > >>>>>>>>>>>>>> > >>>>>>>>>>>> >>>>>wrote: > >>>>>>>>>>>> >>>>> > > >>>>>>>>>>>> >>>>> >> The iptables on unix supports the DENY operator, > >>>>>>>>>>>> >>>>> >> not > >>>>>>>>>>>>that > >>>>>>>>>>>>it > >>>>>>>>>>>> >>>>>should > >>>>>>>>>>>> >>>>> >> matter. The deny operator can also be used to > >>>>>>>>>>>> >>>>> >> specify > >>>>>>>>>>>>³allow > >>>>>>>>>>>>user1 > >>>>>>>>>>>> >>>>>to > >>>>>>>>>>>> >>>>> >>READ > >>>>>>>>>>>> >>>>> >> from topic1 from all hosts but host1,host2². > >>>>>>>>>>>> >>>>> >>Again we > >>>>>>>>>>>>could > >>>>>>>>>>>>add a > >>>>>>>>>>>> >>>>>host > >>>>>>>>>>>> >>>>> >> group semantic and extra complexity around that, > >>>>>>>>>>>> >>>>> >> not > >>>>>>>>>>>>sure > >>>>>>>>>>>>if > >>>>>>>>>>>>its > >>>>>>>>>>>> >>>>>worth > >>>>>>>>>>>> >>>>> >>it. > >>>>>>>>>>>> >>>>> >> In addition with DENY operator you are now not > >>>>>>>>>>>> >>>>> >>forced > >>>>>>>>>>>>to > >>>>>>>>>>>>create a > >>>>>>>>>>>> >>>>> >>special > >>>>>>>>>>>> >>>>> >> group just to support the authorization use > >>>>>>>>>>>> >>>>> >>case. I > am > >>>>>>>>>>>>not > >>>>>>>>>>>> >>>>>convinced > >>>>>>>>>>>> >>>>> >>that > >>>>>>>>>>>> >>>>> >> the operator it self is really all that confusing. > >>>>>>>>>>>>There > >>>>>>>>>>>>are 3 > >>>>>>>>>>>> >>>>>practical > >>>>>>>>>>>> >>>>> >> use cases: > >>>>>>>>>>>> >>>>> >> - Resource with no acl what so ever -> allow > >>>>>>>>>>>> >>>>> >> access > to > >>>>>>>>>>>>everyone ( > >>>>>>>>>>>> >>>>>just > >>>>>>>>>>>> >>>>> >>for > >>>>>>>>>>>> >>>>> >> backward compatibility, I would much rather fail > close > >>>>>>>>>>>>and > >>>>>>>>>>>>force > >>>>>>>>>>>> >>>>>users > >>>>>>>>>>>> >>>>> >>to > >>>>>>>>>>>> >>>>> >> explicitly grant acls that allows access to all > >>>>>>>>>>>>users.) > >>>>>>>>>>>> >>>>> >> - Resource with some acl attached -> only users > >>>>>>>>>>>> >>>>> >> that > >>>>>>>>>>>>have > >>>>>>>>>>>>a > >>>>>>>>>>>> >>>>>matching > >>>>>>>>>>>> >>>>> >>allow > >>>>>>>>>>>> >>>>> >> acl are allowed (i.e. ³allow READ access to > >>>>>>>>>>>> >>>>> >>topic1 to > >>>>>>>>>>>>user1 > >>>>>>>>>>>>from > >>>>>>>>>>>> >>>>>all > >>>>>>>>>>>> >>>>> >> hosts², only user1 has READ access and no other > >>>>>>>>>>>> >>>>> >> user > >>>>>>>>>>>>has > >>>>>>>>>>>>access of > >>>>>>>>>>>> >>>>>any > >>>>>>>>>>>> >>>>> >> kind) > >>>>>>>>>>>> >>>>> >> - Resource with some allow and some deny acl > >>>>>>>>>>>> >>>>> >> attached > >>>>>>>>>>>>-> > >>>>>>>>>>>>users > >>>>>>>>>>>>are > >>>>>>>>>>>> >>>>> >>allowed > >>>>>>>>>>>> >>>>> >> to perform operation only when they satisfy > >>>>>>>>>>>> >>>>> >>allow acl > >>>>>>>>>>>>and > >>>>>>>>>>>>do > >>>>>>>>>>>>not > >>>>>>>>>>>> >>>>>have > >>>>>>>>>>>> >>>>> >> conflicting deny acl. Users that have no > >>>>>>>>>>>> >>>>> >> acl(allow or > >>>>>>>>>>>>deny) > >>>>>>>>>>>>will > >>>>>>>>>>>> >>>>>still > >>>>>>>>>>>> >>>>> >>not > >>>>>>>>>>>> >>>>> >> have any access. (i.e. ³allow READ access to > >>>>>>>>>>>> >>>>> >>topic1 > to > >>>>>>>>>>>>user1 > >>>>>>>>>>>>from > >>>>>>>>>>>> >>>>>all > >>>>>>>>>>>> >>>>> >> hosts except host1 and host², only user1 has > >>>>>>>>>>>> >>>>> >> access > >>>>>>>>>>>>but > >>>>>>>>>>>>not > >>>>>>>>>>>>from > >>>>>>>>>>>> >>>>>host1 > >>>>>>>>>>>> >>>>> >>an > >>>>>>>>>>>> >>>>> >> host2) > >>>>>>>>>>>> >>>>> >> > >>>>>>>>>>>> >>>>> >> I think we need to make a decision on deny > >>>>>>>>>>>> >>>>> >> primarily > >>>>>>>>>>>>because > >>>>>>>>>>>>with > >>>>>>>>>>>> >>>>> >> introduction of acl management API, Acl is now a > >>>>>>>>>>>>public > >>>>>>>>>>>>class > >>>>>>>>>>>>that > >>>>>>>>>>>> >>>>>will > >>>>>>>>>>>> >>>>> >>be > >>>>>>>>>>>> >>>>> >> used by Ranger/Santry and other authroization > >>>>>>>>>>>>providers. > >>>>>>>>>>>>In > >>>>>>>>>>>> >>>>>Current > >>>>>>>>>>>> >>>>> >>design > >>>>>>>>>>>> >>>>> >> the acl has a permissionType enum field with > >>>>>>>>>>>> >>>>> >>possible > >>>>>>>>>>>>values > >>>>>>>>>>>>of > >>>>>>>>>>>> >>>>>Allow > >>>>>>>>>>>> >>>>> >>and > >>>>>>>>>>>> >>>>> >> Deny. If we chose to remove deny we can assume > >>>>>>>>>>>> >>>>> >>all > >>>>>>>>>>>>acls > >>>>>>>>>>>>to > >>>>>>>>>>>>be > >>>>>>>>>>>>of > >>>>>>>>>>>> >>>>>allow > >>>>>>>>>>>> >>>>> >> type and remove the permissionType field completely. > >>>>>>>>>>>> >>>>> >> > >>>>>>>>>>>> >>>>> >> Thanks > >>>>>>>>>>>> >>>>> >> Parth > >>>>>>>>>>>> >>>>> >> > >>>>>>>>>>>> >>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira" > >>>>>>>>>>>> >>>>><gshap...@cloudera.com<mailto:gshap...@cloudera.com> > >>>>>>>>>>>> >>>>>> > >>>>>>>>>>>>wrote: > >>>>>>>>>>>> >>>>> >> > >>>>>>>>>>>> >>>>> >> >I think thats how its done in pretty much any > >>>>>>>>>>>> >>>>> >> >system > >>>>>>>>>>>>I > >>>>>>>>>>>>can > >>>>>>>>>>>>think > >>>>>>>>>>>> >>>>>of. > >>>>>>>>>>>> >>>>> >> > > >>>>>>>>>>>> >>>>> >> > >>>>>>>>>>>> >>>>> >> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>-- > >>>>>>>>>>>Jeff Holoman > >>>>>>>>>>>Systems Engineer > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>> > >>> > > >
