I admit that I'm also not sure what we gain by having "deny" rules. A user is either in the "allow" list (one way or another) or it will be denied by default.
I think this is something we can skip for now and add later if needed? On Mon, Apr 20, 2015 at 5:24 PM, Jun Rao <j...@confluent.io> wrote: > According to the pseudo code, if you have a rule "deny user1", then it > essentially denies all users? > > Thanks, > > Jun > > On Mon, Apr 20, 2015 at 5:16 PM, Parth Brahmbhatt < > pbrahmbh...@hortonworks.com> wrote: > >> >> Here is a pseudo code that explains my current approach: >> >> acls = authorizer.getAcl(resource) >> if(acls == null || acls.isEmpty) { >> allow all requests for backward compatibility. (any topics that >> were >> created prior to security support will not have acls) This is debatable , >> generally we should block everyone which is what I would prefer but that >> means anyone moving to authorizer must go to all of his existing topics >> and add acl to allow all. If we are fine with imposing this requirement I >> can start returning deny when no acls are found. >> } else { >> //So the user has set some acls explicitly, this means they have >> knowingly enabled authorizer. Let’t first check if they have set an Acl to >> deny this user/host/operation combination. >> if some acl denies this request for this principal/host/operation >> combination , return deny >> >> //this principal/host/operation does not have any explicit deny >> acl, >> check if there is some explicit acl that allows the operation >> if at least one acl allows this request for this >> principal/host/operation >> combination , return allow >> >> // no acl was found for this principal/host/operation combination >> to >> allow this operation, so we will deny the request >> return deny >> } >> >> >> Thanks >> Parth >> >> >> On 4/20/15, 2:21 PM, "Jun Rao" <j...@confluent.io> wrote: >> >> >Hmm, I thought the semantics is that if you only have rule "deny user2", >> >it >> >means that everyone except user2 has access? >> > >> >Thanks, >> > >> >Jun >> > >> >On Mon, Apr 20, 2015 at 3:25 PM, Parth Brahmbhatt < >> >pbrahmbh...@hortonworks.com> wrote: >> > >> >> user3 does not have access and removing the deny rule does not grant him >> >> or user2 access. user2 even without the deny rule will not have access. >> >> >> >> Thanks >> >> Parth >> >> >> >> On 4/20/15, 12:03 PM, "Jun Rao" <j...@confluent.io> wrote: >> >> >> >> >Just a followup question. Suppose there are two rules. Rule1 allows >> >>user1 >> >> >and rule2 denies user2. Does user3 have access? If not, does removing >> >> >rule1 >> >> >enable user3 access? >> >> > >> >> >Thanks, >> >> > >> >> >Jun >> >> > >> >> >On Mon, Apr 20, 2015 at 1:34 PM, Parth Brahmbhatt < >> >> >pbrahmbh...@hortonworks.com> wrote: >> >> > >> >> >> >> >> >> Hi Joel, >> >> >> >> >> >> Thanks for the review and I plan to update the KIP today with all the >> >> >> updated info. My comments in line below. >> >> >> >> >> >> Thanks >> >> >> Parth >> >> >> >> >> >> >> >> >> On 4/20/15, 10:07 AM, "Joel Koshy" <jjkosh...@gmail.com<mailto: >> >> >> jjkosh...@gmail.com>> wrote: >> >> >> >> >> >> Hi Parth, >> >> >> >> >> >> Nice work on this KIP. I did another read through and had a few more >> >> >> comments (with edits after I went through the thread). Many of these >> >> >> comments were brought up by others as well, so it appears that the >> >>KIP >> >> >> would benefit from an update at this point to incorporate comments >> >> >> from the thread and last hangout. >> >> >> >> >> >> - The operation enum is mostly self-explanatory, but it would help >> >> >> (for the sake of clarity and completeness if nothing else) to >> >> >> document exactly what each of the enums are. E.g., I think this >> >>came >> >> >> up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't >> >> >> remember what was said about it. <Edit>: After going through the >> >> >> thread it seems the conclusion was to categorize operations. E.g., >> >> >> WRITE could apply to multiple requests. Again, this is unclear, so >> >> >> if it would be great if you could update the KIP to clarify what >> >>you >> >> >> intend. >> >> >> >> >> >> Will add to document. SEND_CONTROL_MESSAGE Probably a very bad name >> >>but >> >> >> these are intra borker API calls like controller notifying other >> >> >>brokers to >> >> >> update metadata or heartbeats. Any better naming suggestions? >> >> >> >> >> >> - When you update the KIP to categorize the requests it would also >> >> >> help to have a column for what the resource is for each. >> >> >> >> >> >> Will add to the KIP. >> >> >> >> >> >> - FWIW I prefer a 1-1 mapping between requests and operations. I >> >>think >> >> >> categorizing requests into these can be confusing because: >> >> >> - The resource being protected for different requests will be >> >> >> different. We are mostly thinking about topics (read/write) but >> >> >> there are requests for which topic is not the right resource. >> >> >> E.g., for topic creation, the resource as you suggested would be >> >> >> something global/common such as “cluster”. For >> >> >> OffsetCommit/FetchRequest, the resource may be the consumer >> >>group, >> >> >> or maybe a tuple of <consumer group, topic>. So this can be >> >> >> confusing - i.e., different resources and request types in the >> >> >> same category. It may be simpler and clearer to just have a 1-1 >> >> >> mapping between the operation enum and requests. >> >> >> >> >> >> I only see 2 resource categories right now cluster and topic. I >> >>don’t >> >> >> really care one way or another so we can probably make a quick >> >>decision >> >> >>in >> >> >> tomorrow’s meeting to either to 1-1 mapping or have categorization? >> >> >> >> >> >> - Some requests that are intuitively READ have WRITE side-effects. >> >> >> E.g., (currently) TopicMetadataRequest with auto-create, although >> >> >> that will eventually go away. ConsumerMetadataRequest still >> >> >> auto-creates the offsets topic. Likewise, ADMIN-type requests may >> >> >> be interpreted as having side-effects (depending on who you ask). >> >> >> >> >> >> Yes and what I am doing right now is checking authorization for all >> >> >> possible actions i.e. for auto-create it checks if the config has it >> >> >> enabled and if yes, check for read + create authorization. Its not >> >>very >> >> >> meaningful right now as there is no CREATE authorization but I think >> >> >>this >> >> >> is implementation detail, we need to ensure we call authorize with >> >>all >> >> >> possible operations from KafkaAPI. >> >> >> - <quote>When an ACL is missing - fail open</quote>. What does >> >>missing >> >> >> mean? i.e., no explicit ACL for a principal? I'm confused by this >> >> >> especially in relation to the precedence of DENY over ALLOW. So per >> >> >> the description: >> >> >> - If no ACLs exist for topic A then ALLOW all operations on it by >> >> >> anyone. >> >> >> - If I now add an ACL for a certain principal P to ALLOW (say) >> >>WRITE >> >> >> to the topic then either: >> >> >> - This has the effect of DENYing WRITE to all other principals >> >> >> - Or, this ACL serves no purpose >> >> >> - If the effect is to DENY WRITE to all other principals, what >> >>about >> >> >> READ. Do all principals (including P) have READ permissions to >> >> >> topic A? >> >> >> - In other words, it seems for a specific ACL to be meaningful then >> >> >> fail close is necessary for an absent ACL. >> >> >> - <edit>After through the thread: it appears that the DENY override >> >> >> only applies to the given principal. i.e., in the above case it >> >> >> appears that the other principals will in fact be granted access. >> >> >> Then this makes the ACL that was added pointless right? >> >> >> >> >> >> The rule I was going with is >> >> >> - If there is no ACL I.e. This might be a topic that was created in >> >>non >> >> >> secure mode or was created before we supported ACLs. We assume you do >> >> >>not >> >> >> want authorization and let all requests go through. >> >> >> - once you add any ACL, we assume you want authorization on the topic >> >> >>and >> >> >> all the general authorization rules now start to apply, I.e we fail >> >> >>close >> >> >> if we don’t find an ACL that allows access or if we find an ACL that >> >> >>denies >> >> >> access. It does not matter if you added a READACL or WRITEACL or >> >> >>ALLOWACL >> >> >> or DENY ACL. If you add any ACL, now every user gets checked against >> >> >>that >> >> >> and if it does not satisfy the ACL, request fails. I.e. If you add an >> >> >>ACL >> >> >> “Allow write to topic-1 form user1 from all hosts” , user-1 has write >> >> >> access from all hosts and no other user has any access(except for >> >> >> superusers who have all the access). >> >> >> - Deny ACLS are suppose to be used to restrict access authorized by >> >>some >> >> >> allow ACL, they are not suppose to be required. Implicitly anyone who >> >> >>does >> >> >> not have an allow acl, gets denied. The Deny ACLs are only added to >> >>give >> >> >> more control to administrators who wants more granular control with >> >> >>lesser >> >> >> config. The scenario described in mailing list was “Allow user X >> >>access >> >> >> from all hosts but Host1,Host2”. in absence of DENY operator you will >> >> >>have >> >> >> to exhaustively list all possible hosts in your ACL which is what we >> >>are >> >> >> trying to avoid. >> >> >> >> >> >> - On ZK ACLs: I think ZK will be closed to everyone except Kafka >> >> >> brokers. This is a dependency on KIP-4 though. i.e., eventually all >> >> >> clients should talk to brokers only via RPC. >> >> >> >> >> >> Yes. >> >> >> >> >> >> - Topic owner: list vs single entry - both have issues off the bat >> >> >> (although list is more intuitive at least to me), but perhaps you >> >> >> could write up some example workflows to clarify the current >> >> >> proposal. I was thinking that anyone in the owner list should be >> >> >> considered a super-user of the topic and can grant/revoke >> >> >> permissions. They should also be allowed to add other principals as >> >> >> owners. Even with this it is unclear who should be allowed to >> >>remove >> >> >> owners. >> >> >> >> >> >> As you pointed out in the last KIP meeting owners/creators have use >> >>out >> >> >> side of security context (plain simple auditing). I don’t think the >> >> >> authorizer work depends on this, it was my bad to even mention it in >> >> >>first >> >> >> place. I think we can have this discussion outside of >> >> >>authorizer/security >> >> >> context and once we have a way to get topic owners the default >> >> >>Authorizer >> >> >> can start using it. It makes sense to treat all owners as super users >> >> >>and I >> >> >> think it is safe to assume superusers can also modify ownership but I >> >> >>think >> >> >> this should not be treated as blocking work for authorization. >> >> >> >> >> >> - What is the effect of deleting a topic - should all associated ACLs >> >> >> be deleted as well? >> >> >> They should be and with acls being stored as part of TopicConfig this >> >> >>was >> >> >> taken care of automatically. With the new ACL management API the >> >>users >> >> >>will >> >> >> have to call remove ACLs explicitly to perform the cleanup. If >> >>everyone >> >> >> thinks this should be automated , with the new APIs we will need a >> >> >>hook(or >> >> >> poll) to be notified when a topic is deleted to perform cleanup. >> >> >> - TopicConfigCache to store topic-ACLs. As mentioned above, not all >> >> >> requests will be tied to topics. We may want to have an entirely >> >> >> separate ZK directory for ACLs. We have a similar issue with >> >>quotas. >> >> >> This ties in with dynamic config management. We can certainly >> >> >> leverage the dynamic config management part of topic configs but I >> >> >> think we need to have a story for non-topic resources. >> >> >> >> >> >> In the first proposal I was going with a topic-Acl and cluster-Acl >> >>where >> >> >> cluster-Acls were json acl local files on all brokers. With the new >> >>ACL >> >> >> management APIs we are planning to have /kafka-acl node under which >> >>all >> >> >> acls will be stored in /kakfa-acls/resource-name -> {acl json data}. >> >> >> Cluster acls will just have resource name kafka-cluster. >> >> >> >> >> >> Thanks, >> >> >> >> >> >> Joel >> >> >> >> >> >> On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote: >> >> >> Kafka currently stores logConfig overrides specified during topic >> >> >>creation >> >> >> in zookeeper, its just an instance of java.util.Properties converted >> >>to >> >> >> json. I am proposing in addition to that we store acls and owner as >> >>well >> >> >> as part of same Properties map. >> >> >> There is some infrastructure around reading this config, converting >> >>it >> >> >> back to Properties map and most importantly propagating any changes >> >> >> efficiently which we will be able to leverage. As this >> >>infrastructure is >> >> >> common to the cluster the reading (not interpreting) of config >> >>happens >> >> >> outside of any authorization code. >> >> >> If the TopicConfigCache just kept the json representation and left >> >>it to >> >> >> authorizer to parse it, the authorizer will have to either parse the >> >> >>json >> >> >> for each request(not acceptable) or it will have to keep one more >> >>layer >> >> >>of >> >> >> parsed ACL instance cache. Assuming authorizer will keep an >> >>additional >> >> >> caching layer we will now have to implement some way to invalidate >> >>the >> >> >> cache which means the TopicConfigCache will have to be an observable >> >> >>which >> >> >> the Authorizer observes and invalidates its cache entries when >> >> >> topicConfigCache gets updated. Seemed like unnecessary complexity >> >>with >> >> >>not >> >> >> lot to gain so I went with TopicConfigCache interpreting the json and >> >> >> caching a higher level modeled object. >> >> >> In summary, the interpretation is done for both optimization and >> >> >> simplicity. If you think it is important to allow custom ACL format >> >> >> support we can add one more pluggable config(acl.parser) and >> >> >> interface(AclParser) or it could just be another method in >> >>Authorizer. >> >> >> One thing to note the current ACL json is versioned so it is easy to >> >> >>make >> >> >> changes to it however it won’t be possible to support custom ACL >> >>formats >> >> >> with the current design. >> >> >> Thanks >> >> >> Parth >> >> >> On 4/15/15, 4:29 PM, "Michael Herstine" >> >><mherst...@linkedin.com.INVALID >> >> >> <mailto:mherst...@linkedin.com.INVALID>> >> >> >> wrote: >> >> >> >Hi Parth, >> >> >> > >> >> >> >I’m a little confused: why would Kafka need to interpret the JSON? >> >> >>IIRC >> >> >> >KIP-11 even says that the TopicConfigData will just store the JSON. >> >>I’m >> >> >> >not really making a design recommendation here, just trying to >> >> >>understand >> >> >> >what you’re proposing. >> >> >> > >> >> >> >On 4/15/15, 11:20 AM, "Parth Brahmbhatt" >> >><pbrahmbh...@hortonworks.com >> >> >> <mailto:pbrahmbh...@hortonworks.com>> >> >> >> >wrote: >> >> >> > >> >> >> >>Hi Michael, >> >> >> >> >> >> >> >>There is code in kafka codebase that reads and interprets the topic >> >> >> >>config JSON which has acls, owner and logconfig properties. There >> >>are >> >> >>3 >> >> >> >>use cases that we are supporting with current proposal: >> >> >> >> >> >> >> >> * You use out of box simpleAcl authorizer which is tied to the >> >>acl >> >> >> >>stored in topic config and the format is locked down. >> >> >> >> * You have a custom authorizer and a custom ACL store. >> >> >>Ranger/Argus >> >> >> >>falls under this as they have their own acl store and ui that users >> >> >>use >> >> >> >>to configure acls on the cluster and cluster resources like topic. >> >> >>It is >> >> >> >>upto the custom authorizer to leverage the kafka acl configs or >> >> >> >>completely ignore them as they have set a user expectation that >> >>only >> >> >>acls >> >> >> >>configured via their ui/system will be effective. >> >> >> >> * You have a custom authorizer but no custom Acl store. You are >> >> >> >>completely tied to Acl structure that we have provided in out of >> >>box >> >> >> >>implementation. >> >> >> >> >> >> >> >>Thanks >> >> >> >>Parth >> >> >> >> >> >> >> >>On 4/15/15, 10:31 AM, "Michael Herstine" >> >> >> >> >>>><mherst...@linkedin.com.INVALID<mailto:mherst...@linkedin.com.INVALID >> >> >> ><mailto:mherst...@linkedin.com.INVALID>> >> >> >> >>wrote: >> >> >> >> >> >> >> >>Hi Parth, >> >> >> >> >> >> >> >>One question that occurred to me at the end of today’s hangout: how >> >> >>tied >> >> >> >>are we to a particular ACL representation under your proposal? I >> >>know >> >> >> >>that >> >> >> >>TopicConfigCache will just contain JSON— if a particular site >> >>decides >> >> >> >>they >> >> >> >>want to represent their ACLs differently, and swap out the >> >>authorizer >> >> >> >>implementation, will that work? I guess what I’m asking is whether >> >> >> >>there’s any code in the Kafka codebase that will interpret that >> >>JSON, >> >> >>or >> >> >> >>does that logic live exclusively in the authorizer? >> >> >> >> >> >> >> >>On 4/14/15, 10:56 PM, "Don Bosco Durai" >> >> >> >> >>>><bo...@apache.org<mailto:bo...@apache.org><mailto:bo...@apache.org>> >> >> >> wrote: >> >> >> >> >> >> >> >>I also feel, having just IP would be more appropriate. Host lookup >> >> >>will >> >> >> >>unnecessary slow things down and would be insecure as you pointed >> >>out. >> >> >> >> >> >> >> >>With IP, it will be also able to setup policies (in future if >> >>needed) >> >> >> >>with >> >> >> >>ranges or netmasks and it would be more scalable. >> >> >> >> >> >> >> >>Bosco >> >> >> >> >> >> >> >> >> >> >> >>On 4/14/15, 1:40 PM, "Michael Herstine" >> >> >> >> >>>><mherst...@linkedin.com.INVALID<mailto:mherst...@linkedin.com.INVALID >> >> >> ><mailto:mherst...@linkedin.com.INVALID>> >> >> >> >>wrote: >> >> >> >> >> >> >> >>Hi Parth, >> >> >> >> >> >> >> >>Sorry to chime in so late, but I’ve got a minor question on the >> >>KIP. >> >> >> >> >> >> >> >>Several methods take a parameter named “host” of type String. Is >> >>that >> >> >> >>intended to be a hostname, or an IP address? If the former, I’m >> >> >>curious >> >> >> >>as >> >> >> >>to how that’s found (in my experience, when accepting an incoming >> >> >>socket >> >> >> >>connection, you only know the IP address, and there isn’t a way to >> >>map >> >> >> >>that to a hostname without a round trip to a DNS server, which is >> >> >> >>insecure >> >> >> >>anyway). >> >> >> >> >> >> >> >> >> >> >> >>On 3/25/15, 1:07 PM, "Parth Brahmbhatt" >> >> >> >> >> >>>><pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com >> >> ><mailto >> >> >>>>: >> >> >> pbrahmbh...@hortonworks.com>> >> >> >> >>wrote: >> >> >> >> >> >> >> >>Hi all, >> >> >> >> >> >> >> >>I have modified the KIP to reflect the recent change request from >> >>the >> >> >> >>reviewers. I have been working on the code and I have the server >> >>side >> >> >> >>code >> >> >> >>for authorization ready. I am now modifying the command line >> >> >>utilities. >> >> >> >>I >> >> >> >>would really appreciate if some of the committers can spend >> >>sometime >> >> >>to >> >> >> >>review the KIP so we can make progress on this. >> >> >> >> >> >> >> >>Thanks >> >> >> >>Parth >> >> >> >> >> >> >> >>On 3/18/15, 2:20 PM, "Michael Herstine" >> >> >> >> >>>><mherst...@linkedin.com.INVALID<mailto:mherst...@linkedin.com.INVALID >> >> >> ><mailto:mherst...@linkedin.com.INVALID>> >> >> >> >>wrote: >> >> >> >> >> >> >> >>Hi Parth, >> >> >> >> >> >> >> >>Thanks! A few questions: >> >> >> >> >> >> >> >>1. Do you want to permit rules in your ACLs that DENY access as >> >>well >> >> >>as >> >> >> >>ALLOW? This can be handy setting up rules that have exceptions. >> >>E.g. >> >> >> >>“Allow principal P to READ resource R from all hosts” with “Deny >> >> >> >>principal >> >> >> >>P READ access to resource R from host H1” in combination would >> >>allow P >> >> >> >>to >> >> >> >>READ R from all hosts *except* H1. >> >> >> >> >> >> >> >>2. When a topic is newly created, will there be an ACL created for >> >>it? >> >> >> >>If >> >> >> >>not, would that not deny subsequent access to it? >> >> >> >> >> >> >> >>(nit) Maybe use Principal instead of String to represent >> >>principals? >> >> >> >> >> >> >> >> >> >> >> >>On 3/9/15, 11:48 AM, "Don Bosco Durai" >> >> >> >> >>>><bo...@apache.org<mailto:bo...@apache.org><mailto:bo...@apache.org>> >> >> >> wrote: >> >> >> >> >> >> >> >>Parth >> >> >> >> >> >> >> >>Overall it is looking good. Couple of questionsŠ >> >> >> >> >> >> >> >>- Can you give an example how the policies will look like in the >> >> >> >>default >> >> >> >>implementation? >> >> >> >>- In the operations, can we support ³CONNECT² also? This can be >> >>used >> >> >> >>during Session connection >> >> >> >>- Regarding access control for ³Topic Creation², since we can¹t do >> >>it >> >> >> >>on >> >> >> >>the server side, can we de-scope it for? And plan it as a future >> >> >> >>feature >> >> >> >>request? >> >> >> >> >> >> >> >>Thanks >> >> >> >> >> >> >> >>Bosco >> >> >> >> >> >> >> >> >> >> >> >>On 3/6/15, 8:10 AM, "Harsha" >> >><ka...@harsha.io<mailto:ka...@harsha.io >> >> >> ><mailto:ka...@harsha.io>> >> >> >> >>wrote: >> >> >> >> >> >> >> >>Hi Parth, >> >> >> >> Thanks for putting this together. Overall it looks good >> >> >> >>to >> >> >> >> me. Although AdminUtils is a concern KIP-4 can >> >>probably >> >> >> >>fix >> >> >> >> that part. >> >> >> >>Thanks, >> >> >> >>Harsha >> >> >> >> >> >> >> >>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote: >> >> >> >>Forgot to add links to wiki and jira. >> >> >> >>Link to wiki: >> >> >> >> >>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza >> >> >> >>t >> >> >> >>i >> >> >> >>o >> >> >> >>n >> >> >> >>+ >> >> >> >>Interface >> >> >> >>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688 >> >> >> >>Thanks >> >> >> >>Parth >> >> >> >>From: Parth Brahmbhatt >> >> >> >> >> >>>><pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com >> >> ><mailto >> >> >>>>: >> >> >> pbrahmbh...@hortonworks.com><mailto:p >> >> >> >>b >> >> >> >>rahmbh...@hortonworks.com<mailto:rahmbh...@hortonworks.com>>> >> >> >> >>Date: Thursday, March 5, 2015 at 10:33 AM >> >> >> >>To: >> >> >> >>"dev@kafka.apache.org<mailto:dev@kafka.apache.org><mailto: >> >> >> dev@kafka.apache.org><mailto:dev@kafka.apach >> >> >> >>e >> >> >> >>.org>" >> >> >> >><dev@kafka.apache.org<mailto:dev@kafka.apache.org><mailto: >> >> >> dev@kafka.apache.org><mailto:dev@kafka.apach >> >> >> >>e >> >> >> >>.org>> >> >> >> >>Subject: [DISCUSS] KIP-11- Authorization design for kafka security >> >> >> >>Hi, >> >> >> >>KIP-11 is open for discussion , I have updated the wiki with the >> >> >> >>design >> >> >> >>and open questions. >> >> >> >>Thanks >> >> >> >>Parth >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>