Sample ACL JSON and Zookeeper is in public API, but I thought it is part of DefaultAuthorizer (Since Sentry and Argus won't be using Zookeeper). Am I wrong? Or is it the KIP?
On Fri, Apr 24, 2015 at 9:49 AM, Parth Brahmbhatt <pbrahmbh...@hortonworks.com> wrote: > Thanks for clarifying Gwen, KIP updated. > > I tried to make the distinction by creating a section for all public APIs > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+In > terface#KIP-11-AuthorizationInterface-PublicInterfacesandclasses > > Let me know if you think there is a better way to reflect this. > > Thanks > Parth > > On 4/24/15, 9:37 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote: > >>+1 (non-binding) >> >>Two nitpicks for the wiki: >>* Heartbeat is probably a READ and not CLUSTER operation. I'm pretty >>sure new consumers need it to be part of a consumer group. >>* Can you clearly separate which parts are the API (common to every >>Authorizer) and which parts are DefaultAuthorizer implementation? It >>will make reviews and Authorizer implementations a bit easier to know >>exactly which is which. >> >>Gwen >> >>On Fri, Apr 24, 2015 at 9:28 AM, Parth Brahmbhatt >><pbrahmbh...@hortonworks.com> wrote: >>> Hi, >>> >>> I would like to open KIP-11 for voting. >>> >>> Thanks >>> Parth >>> >>> On 4/22/15, 1:56 PM, "Parth Brahmbhatt" <pbrahmbh...@hortonworks.com> >>> wrote: >>> >>>>Hi Jeff, >>>> >>>>Thanks a lot for the review. I think you have a valid point about acls >>>>being duplicated and the simplest solution would be to modify acls class >>>>so they hold a set of principals instead of single principal. i.e >>>> >>>><user_a,user_b> has <READ,WRITE,DESCRIBE> Permissions on <Topic1> from >>>><Host1, Host2, Host3>. >>>> >>>>I think the evaluation order only matters for the permissionType which >>>>is >>>>Deny acls should be evaluated before allow acls. To give you an example >>>>suppose we have following acls >>>> >>>>acl1 -> user1 is allowed to READ from all hosts. >>>>acl2 -> host1 is allowed to READ regardless of who is the user. >>>>acl3 -> host2 is allowed to READ regardless of who is the user. >>>> >>>>acl4 -> user1 is denied to READ from host1. >>>> >>>>As stated in the KIP we first evaluate DENY so if user1 tries to access >>>>from host1 he will be denied(acl4), even though both user1 and host1 has >>>>acl’s for allow with wildcards (acl1, acl2). >>>>If user1 tried to READ from host2 , the action will be allowed and it >>>>does >>>>not matter if we match acl3 or acl1 so I don’t think the evaluation >>>>order >>>>matters here. >>>> >>>>“Will people actually use hosts with users?” I really don’t know but >>>>given >>>>ACl’s are part of our Public APIs I thought it is better to try and >>>>cover >>>>more use cases. If others think this extra complexity is not worth the >>>>value its adding please raise your concerns so we can discuss if it >>>>should >>>>be removed from the acl structure. Note that even in absence of hosts >>>>from >>>>ACL users will still be able to whitelist/blacklist host as long as we >>>>start supporting principalType = “host”, easy to add and can be an >>>>incremental improvement. They will however loose the ability to restrict >>>>access to users just from a set of hosts. >>>> >>>>We agreed to offer a CLI to overcome the JSON acl config >>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization >>>>+I >>>>n >>>>terface#KIP-11-AuthorizationInterface-AclManagement(CLI). I still like >>>>Jsons but that probably has something to do with me being a developer >>>>:-). >>>> >>>>Thanks >>>>Parth >>>> >>>>On 4/22/15, 11:38 AM, "Jeff Holoman" <jholo...@cloudera.com> wrote: >>>> >>>>>Parth, >>>>> >>>>>This is a long thread, so trying to keep up here, sorry if this has >>>>>been >>>>>covered before. First, great job on the KIP proposal and work so far. >>>>> >>>>>Are we sure that we want to tie host level access to a given user? My >>>>>understanding is that the ACL will be (omitting some fields) >>>>> >>>>>user_a, host1, host2, host3 >>>>>user_b, host1, host2, host3 >>>>> >>>>>So there would potentially be a lot of redundancy in the configs. Does >>>>>it >>>>>make sense to have hosts be at the same level as principal in the >>>>>hierarchy? This way you could just blanket the allowed / denied hosts >>>>>and >>>>>only have to worry about the users. So if you follow this, then >>>>> >>>>>we can wildcard the user so we can have a separate list of just >>>>>host-based >>>>>access. What's the order that the perms would be evaluated if a there >>>>>was >>>>>more than one match on a principal ? >>>>> >>>>>Is the thought that there wouldn't usually be much overlap on hosts? I >>>>>guess I can imagine a scenario where I want to offline/online access >>>>>to a >>>>>particular hosts or set of hosts and if there was overlap, I'm doing a >>>>>bunch of alter commands for just a single host. Maybe this is too >>>>>contrived >>>>>an example? >>>>> >>>>>I agree that having this level of granularity gives flexibility but I >>>>>wonder if people will actually use it and not just * the hosts for a >>>>>given >>>>>user and create separate "global" list as i mentioned above? >>>>> >>>>>The only other system I know of that ties users with hosts for access >>>>>is >>>>>MySql and I don't love that model. Companies usually standardize on >>>>>group >>>>>authorization anyway, are we complicating that issue with the inclusion >>>>>of >>>>>hosts attached to users? Additionally I worry about the debt of big >>>>>JSON >>>>>configs in the first place, most non-developers find them non-intuitive >>>>>already, so anything to ease this I think would be beneficial. >>>>> >>>>> >>>>>Thanks >>>>> >>>>>Jeff >>>>> >>>>>On Wed, Apr 22, 2015 at 2:22 PM, Parth Brahmbhatt < >>>>>pbrahmbh...@hortonworks.com> wrote: >>>>> >>>>>> Sorry I missed your last questions. I am +0 on adding ―host option >>>>>>for >>>>>> ―list, we could add it for symmetry. Again if this is only a CLI >>>>>>change >>>>>>it >>>>>> can be added later if you mean adding this in authorizer interface >>>>>>then >>>>>>we >>>>>> should make a decision now. >>>>>> >>>>>> Given a choice I would like to actually keep only one option which is >>>>>> resource based get (remove even the get based on principal). I see >>>>>>those >>>>>> (getAcl for principal or host) as special filtering case which can >>>>>>easily >>>>>> be achieved by a third party tool by doing "list all topics" and >>>>>>calling >>>>>> getAcls for each topic and applying filtering logic on that. I >>>>>>really >>>>>> don’t see the need to make those first class citizens of the >>>>>>authorizer >>>>>> interface given these kind of queries will be issued outside of >>>>>>broker >>>>>>JVM >>>>>> so they will not benefit from the caching and because the storage >>>>>>will >>>>>>be >>>>>> indexed on resource both these options even as a first class API will >>>>>>just >>>>>> scan all topic acls and apply filtering logic. >>>>>> >>>>>> Thanks >>>>>> Parth >>>>>> >>>>>> On 4/22/15, 11:08 AM, "Parth Brahmbhatt" >>>>>><pbrahmbh...@hortonworks.com> >>>>>> wrote: >>>>>> >>>>>> >Please see all the available options here >>>>>> > >>>>>> >>>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizati >>>>>>on >>>>>>+ >>>>>>I >>>>>> >nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think >>>>>>it >>>>>> >covers both hosts and operations and allows to specify a list for >>>>>>both. >>>>>> > >>>>>> >Thanks >>>>>> >Parth >>>>>> > >>>>>> >From: Tom Graves <tgraves...@yahoo.com<mailto:tgraves...@yahoo.com>> >>>>>> >Reply-To: Tom Graves >>>>>><tgraves...@yahoo.com<mailto:tgraves...@yahoo.com>> >>>>>> >Date: Wednesday, April 22, 2015 at 11:02 AM >>>>>> >To: Parth Brahmbhatt >>>>>> ><pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com>>, >>>>>> >"dev@kafka.apache.org<mailto:dev@kafka.apache.org>" >>>>>> ><dev@kafka.apache.org<mailto:dev@kafka.apache.org>> >>>>>> >Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka >>>>>>security >>>>>> > >>>>>> >Thanks for the explanations Parth. >>>>>> > >>>>>> >On the configs questions, the way I see it is its more likely to >>>>>> >accidentally give everyone access, especially since you have to run >>>>>>a >>>>>> >separate command to change the acls. If there was some config for >>>>>> >defaults, a cluster admin could change that to be nobody or certain >>>>>>set >>>>>> >of users, then grant others permissions. This would also remove the >>>>>>race >>>>>> >between commands. This is something you can always add later though >>>>>>if >>>>>> >people request it. >>>>>> > >>>>>> >So in kafka-acl.sh how do I actually tell it what the operation is? >>>>>> >kafka-acl.sh --topic testtopic --add --grandprincipal >>>>>>user:joe,user:kate >>>>>> > >>>>>> >where does READ, WRITE, etc go? Can specify as a list so I don't >>>>>>have >>>>>>to >>>>>> >run this a bunch of times for each. >>>>>> > >>>>>> >Do you want to have a --host option for --list so that admins could >>>>>>see >>>>>> >what acls apply to specific host(s)? >>>>>> > >>>>>> >Tom >>>>>> > >>>>>> > >>>>>> > >>>>>> >On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt >>>>>> ><pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com>> >>>>>>wrote: >>>>>> > >>>>>> > >>>>>> > >>>>>> >FYI, I have modified the KIP to include group as resource. In order >>>>>>to >>>>>> >access “joinGroup” and “commitOFfset” APIs the user will need a read >>>>>> >permission on topic and WRITE permission on group. >>>>>> > >>>>>> >I plan to open a VOTE thread by noon if there are no more concerns. >>>>>> > >>>>>> >Thanks >>>>>> >Parth >>>>>> > >>>>>> >On 4/22/15, 9:03 AM, "Tom Graves" >>>>>> ><tgraves...@yahoo.com.INVALID<mailto:tgraves...@yahoo.com.INVALID>> >>>>>> wrote: >>>>>> > >>>>>> >>Hey everyone, >>>>>> >>Sorry to jump in on the conversation so late. I'm new to Kafka. >>>>>>I'll >>>>>> >>apologize in advance if you have already covered some of my >>>>>>questions. I >>>>>> >>read through the wiki and had some comments and questions. >>>>>> >>1) public enum Operation needs EDIT changed to ALTER >>>>>> > >>>>>> >> Done. >>>>>> > >>>>>> >>2) Does the Authorizer class need a setAcls? Rather then just add >>>>>>to >>>>>>be >>>>>> >>able to set to explicit list and overwrite what was there? I see >>>>>>the >>>>>> >>kafka-acl.sh lists a removeall so I guess you could do removeall >>>>>>and >>>>>>then >>>>>> >>add. I also don't see a removeall in the Authorizer class, is it >>>>>>going >>>>>> >>to loop through them all to remove each one? >>>>>> > >>>>>> > There is an overloaded version of removeAcls in the interface >>>>>>that >>>>>> >takes >>>>>> >in resource as the only input and as described in the javadoc all >>>>>>the >>>>>>acls >>>>>> >attached to that resource will be deleted. To cover the setAcl use >>>>>>case >>>>>> >the caller can first call remove and then add. >>>>>> > >>>>>> >>3) Can someone tell me what the use case to do acls based on the >>>>>>hosts? >>>>>> >>I can see some possibilities just wondering if we can concrete ones >>>>>>where >>>>>> >>one user is allowed from one host but not another. >>>>>> > >>>>>> > I am not sure if I understand the question given the use case >>>>>>you >>>>>> >described in your question is what we are trying to cover with use >>>>>>of >>>>>> >hosts in Acl. There are some additional use cases like “allow access >>>>>>to >>>>>> >any user from host1,host2” but I think primarily it gives the admins >>>>>>the >>>>>> >ability to define acls at a more granular level. >>>>>> > >>>>>> >>4) I'm a bit unclear how the "resource" works in the Authorizer >>>>>>class. >>>>>> >>From what I see we have 2 resources - topics and cluster. If I >>>>>>want >>>>>>to >>>>>> >>add an acl to allow "joe" to CREATE for the cluster then I call >>>>>>addAcls >>>>>> >>with Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"? >>>>>>What >>>>>> >>if I want to call addAcls for DESCRIBE on a topic? Is the resource >>>>>>then >>>>>> >>"topic" or is it the topic name? >>>>>> > >>>>>> > We now have 3 resources(added group), please see the updated >>>>>>doc. >>>>>>The >>>>>> >CREATE acl that you described is correct. For any topic operation >>>>>>you >>>>>> >should use topic name as the resource name and for group the user >>>>>>will >>>>>> >provide groupId as resource name. >>>>>> > >>>>>> >>5) reassigning partitions is a CLUSTER_ACTION or superuser? Its >>>>>>not >>>>>> >>totally clear to me the differences between these. what about >>>>>>increasing >>>>>> >># of partitions? >>>>>> > >>>>>> > I see this as an alter topic operation so it is at topic level >>>>>>and >>>>>>the >>>>>> >user must have alter permissions on topic. >>>>>> > >>>>>> >>6) groups are mentioned, are we supporting right away or is that a >>>>>>follow >>>>>> >>on item? (is there going to be a kafka.supergroups) >>>>>> > >>>>>> > I think it can be a separate jira just for braking down the code >>>>>> >review >>>>>> >in smaller chunk. We will support it in first version but I think if >>>>>>we >>>>>> >can not do it for any reason that should not block a release with >>>>>>all >>>>>>the >>>>>> >other authZ work. We made deliberate design choices (like >>>>>>introducing >>>>>>a >>>>>> >principalType in KafkaPrinciapl) to allow supporting groups as an >>>>>> >incremental change. >>>>>> > >>>>>> >>7) Are there config options for setting acls when I create my >>>>>>topic? >>>>>>Or >>>>>> >>do I have to create my topic and then run the kafka-acl.sh script >>>>>>to >>>>>>set >>>>>> >>them? Although its very small, there would be possible race there >>>>>>that >>>>>> >>someone could start producing to topic before acls are set. >>>>>> > >>>>>> > We discussed this yesterday and we agreed to go with >>>>>>kafka-acl.sh. >>>>>>Yes >>>>>> >there is a very very small window of vulnerability but I think that >>>>>>really >>>>>> >does not warrant to change the decision in this case. >>>>>> > >>>>>> >>8) are there configs for cluster level acl defaults? Or does it >>>>>>default >>>>>> >>to superusers on bringing up new cluster and you have to modify >>>>>>with >>>>>>cli. >>>>>> >>thanks,Tom >>>>>> > >>>>>> > No defaults, the default is superusers will have full access. I >>>>>>don’t >>>>>> >think making assumptions about ones security requirement should be >>>>>>our >>>>>> >burden. >>>>>> > >>>>>> > >>>>>> >> >>>>>> >> On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt >>>>>> >><pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com>> >>>>>>wrote: >>>>>> >> >>>>>> >> >>>>>> >> I have added the notes to KIP-11 Open question sections. >>>>>> >> >>>>>> >>Thanks >>>>>> >>Parth >>>>>> >> >>>>>> >>On 4/21/15, 4:49 PM, "Gwen Shapira" >>>>>> >><gshap...@cloudera.com<mailto:gshap...@cloudera.com>> wrote: >>>>>> >> >>>>>> >>>Adding my notes from today's call to the thread: >>>>>> >>> >>>>>> >>>** Deny or Allow all by default? We will add a configuration to >>>>>> >>>control this. The configuration will default to “allow” for >>>>>>backward >>>>>> >>>compatibility. Security admins can set it to "deny" >>>>>> >>> >>>>>> >>>** Storing ACLs for default authorizers: We'll store them in ZK. >>>>>>We'll >>>>>> >>>support pointing the authorizer to any ZK. >>>>>> >>>The use of ZK will be internal to the default authorizer. >>>>>>Authorizer >>>>>> >>>reads ACLs from cache every hour. We proposed having mechanism >>>>>> >>>(possibly via new ZK node) to tell broker to refresh the cache >>>>>> >>>immediately. >>>>>> >>> >>>>>> >>>** Support deny as permission type - we agreed to keep this. >>>>>> >>> >>>>>> >>>** Mapping operations to API: We may need to add Group as a >>>>>>resource, >>>>>> >>>with JoinGroup and OffsetCommit require privilege on the consumer >>>>>> >>>group. >>>>>> >>>This can be something we pass now and authorizers can support in >>>>>> >>>future. - Jay will write specifics to the mailing list discussion. >>>>>> >>> >>>>>> >>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps >>>>>> >>><jay.kr...@gmail.com<mailto:jay.kr...@gmail.com>> wrote: >>>>>> >>>> Following up on the KIP discussion. Two options for authorizing >>>>>> >>>>consumers >>>>>> >>>> to read topic "t" as part of group "g": >>>>>> >>>> 1. READ permission on resource /topic/t >>>>>> >>>> 2. READ permission on resource /topic/t AND WRITE permission on >>>>>> >>>>/group/g >>>>>> >>>> >>>>>> >>>> The advantage of (1) is that it is simpler. The disadvantage is >>>>>>that >>>>>> >>>>any >>>>>> >>>> member of any group that reads from t can commit offsets as any >>>>>>other >>>>>> >>>> member of a different group. This doesn't effect data security >>>>>>(who >>>>>> >>>>can >>>>>> >>>> access what) but it is a bit of a management issue--a malicious >>>>>>person >>>>>> >>>>can >>>>>> >>>> cause data loss or duplicates for another consumer by committing >>>>>> >>>>offset. >>>>>> >>>> >>>>>> >>>> I think I favor (2) but it's worth it to think it through. >>>>>> >>>> >>>>>> >>>> -Jay >>>>>> >>>> >>>>>> >>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt < >>>>>> >>>> pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com>> >>>>>> >>>>wrote: >>>>>> >>>> >>>>>> >>>>> Hey Jun, >>>>>> >>>>> >>>>>> >>>>> Yes and we support wild cards for all acl entities principal, >>>>>>hosts >>>>>> >>>>>and >>>>>> >>>>> operation. >>>>>> >>>>> >>>>>> >>>>> Thanks >>>>>> >>>>> Parth >>>>>> >>>>> >>>>>> >>>>> On 4/21/15, 9:06 AM, "Jun Rao" >>>>>> >>>>><j...@confluent.io<mailto:j...@confluent.io>> wrote: >>>>>> >>>>> >>>>>> >>>>> >Harsha, Parth, >>>>>> >>>>> > >>>>>> >>>>> >Thanks for the clarification. This makes sense. Perhaps we can >>>>>> >>>>>clarify the >>>>>> >>>>> >meaning of those rules in the wiki. >>>>>> >>>>> > >>>>>> >>>>> >Related to this, it seems that we need to support wildcard in >>>>>> >>>>>cli/request >>>>>> >>>>> >protocol for topics? >>>>>> >>>>> > >>>>>> >>>>> >Jun >>>>>> >>>>> > >>>>>> >>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt < >>>>>> >>>>> >>>>>>>pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com>> >>>>>> >>>>>wrote: >>>>>> >>>>> > >>>>>> >>>>> >> The iptables on unix supports the DENY operator, not that it >>>>>> >>>>>should >>>>>> >>>>> >> matter. The deny operator can also be used to specify ³allow >>>>>>user1 >>>>>> >>>>>to >>>>>> >>>>> >>READ >>>>>> >>>>> >> from topic1 from all hosts but host1,host2². Again we could >>>>>>add a >>>>>> >>>>>host >>>>>> >>>>> >> group semantic and extra complexity around that, not sure if >>>>>>its >>>>>> >>>>>worth >>>>>> >>>>> >>it. >>>>>> >>>>> >> In addition with DENY operator you are now not forced to >>>>>>create a >>>>>> >>>>> >>special >>>>>> >>>>> >> group just to support the authorization use case. I am not >>>>>> >>>>>convinced >>>>>> >>>>> >>that >>>>>> >>>>> >> the operator it self is really all that confusing. There >>>>>>are 3 >>>>>> >>>>>practical >>>>>> >>>>> >> use cases: >>>>>> >>>>> >> - Resource with no acl what so ever -> allow access to >>>>>>everyone ( >>>>>> >>>>>just >>>>>> >>>>> >>for >>>>>> >>>>> >> backward compatibility, I would much rather fail close and >>>>>>force >>>>>> >>>>>users >>>>>> >>>>> >>to >>>>>> >>>>> >> explicitly grant acls that allows access to all users.) >>>>>> >>>>> >> - Resource with some acl attached -> only users that have a >>>>>> >>>>>matching >>>>>> >>>>> >>allow >>>>>> >>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 >>>>>>from >>>>>> >>>>>all >>>>>> >>>>> >> hosts², only user1 has READ access and no other user has >>>>>>access of >>>>>> >>>>>any >>>>>> >>>>> >> kind) >>>>>> >>>>> >> - Resource with some allow and some deny acl attached -> >>>>>>users >>>>>>are >>>>>> >>>>> >>allowed >>>>>> >>>>> >> to perform operation only when they satisfy allow acl and do >>>>>>not >>>>>> >>>>>have >>>>>> >>>>> >> conflicting deny acl. Users that have no acl(allow or deny) >>>>>>will >>>>>> >>>>>still >>>>>> >>>>> >>not >>>>>> >>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 >>>>>>from >>>>>> >>>>>all >>>>>> >>>>> >> hosts except host1 and host², only user1 has access but not >>>>>>from >>>>>> >>>>>host1 >>>>>> >>>>> >>an >>>>>> >>>>> >> host2) >>>>>> >>>>> >> >>>>>> >>>>> >> I think we need to make a decision on deny primarily because >>>>>>with >>>>>> >>>>> >> introduction of acl management API, Acl is now a public >>>>>>class >>>>>>that >>>>>> >>>>>will >>>>>> >>>>> >>be >>>>>> >>>>> >> used by Ranger/Santry and other authroization providers. In >>>>>> >>>>>Current >>>>>> >>>>> >>design >>>>>> >>>>> >> the acl has a permissionType enum field with possible values >>>>>>of >>>>>> >>>>>Allow >>>>>> >>>>> >>and >>>>>> >>>>> >> Deny. If we chose to remove deny we can assume all acls to >>>>>>be >>>>>>of >>>>>> >>>>>allow >>>>>> >>>>> >> type and remove the permissionType field completely. >>>>>> >>>>> >> >>>>>> >>>>> >> Thanks >>>>>> >>>>> >> Parth >>>>>> >>>>> >> >>>>>> >>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira" >>>>>> >>>>><gshap...@cloudera.com<mailto:gshap...@cloudera.com>> wrote: >>>>>> >>>>> >> >>>>>> >>>>> >> >I think thats how its done in pretty much any system I can >>>>>>think >>>>>> >>>>>of. >>>>>> >>>>> >> > >>>>>> >>>>> >> >>>>>> >>>>> >> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> > >>>>>> > >>>>>> > >>>>>> >>>>>> >>>>> >>>>> >>>>>-- >>>>>Jeff Holoman >>>>>Systems Engineer >>>> >>> >
