Yes in case of kerberos we will use superACL and this will be equivalent to kafka broker’s principal name. But in SSL as two-way auth is not mandatory the only option if we want enforce authorizer in case of ssl is to force two-way auth. Again this can be an issue on client side , lets say if a producer doesn’t want to provide client auth and just needs wire encryption there won’t be any identity , in this case and we won’t be able to enforce an authorizer as the client will be anonymous.
-- Harsha On March 31, 2015 at 10:29:33 AM, Don Bosco Durai (bo...@apache.org) wrote: >Related interesting question: Since a broker is a consumer (of lead replicas), how do we handle the broker level of permissions? Do we hardcode a broker-principal name and automatically authorize brokers to do anything? Or is there a cleaner way? I feel, in Kerberos environment, “kafka” keytab would be the ideal solution. And “kafka” principal will need to be white listed. SSL certificate is another option, but it would be painful to set it up. IP whitelisting is another low impact, but less secure option. Bosco On 3/31/15, 10:20 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote: >Related interesting question: >Since a broker is a consumer (of lead replicas), how do we handle the >broker level of permissions? Do we hardcode a broker-principal name >and automatically authorize brokers to do anything? Or is there a >cleaner way? > > >On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai <bo...@apache.org> >wrote: >>>21. Operation: What about other types of requests not covered in the >>>list, >> such as committing and fetching offsets, list topics, fetching consumer >> metadata, heartbeat, join group, etc? >> >> Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add >> high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover >>related >> permissions? >> >> Bosco >> >> >> >> On 3/31/15, 9:21 AM, "Jun Rao" <j...@confluent.io> wrote: >> >>>Thanks for the writeup. A few more comments. >>> >>>20. I agree that it would be better to do this after KIP-4 (admin >>>commands) >>>is done. With KIP-4, all admin operations will be sent as requests to >>>the >>>brokers instead of accessing ZK directly. This will make authorization >>>easier. >>> >>>21. Operation: What about other types of requests not covered in the >>>list, >>>such as committing and fetching offsets, list topics, fetching consumer >>>metadata, heartbeat, join group, etc? >>> >>>22. TopicConfigCache: We will need such a cache in KIP-4 as well. It >>>would >>>be useful to make sure that the implementation can be reused. >>> >>>23. Authorizer: >>>23.1 Do cluster level operations go through authorize() too? If so, what >>>will be the resource? >>>23.2 I assume that the authorize() check will be called on every >>>request. >>>So, we will have to make sure that the check is cheap. >>> >>>24. The acl json string in the config: Should we version this so that we >>>can evolve it in the future (e.g., adding group support)? >>> >>>Jun >>> >>>On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt < >>>pbrahmbh...@hortonworks.com> wrote: >>> >>>> Hi Gwen, >>>> >>>> Thanks a lot for taking the time to review this. I have tried to >>>>address >>>> all your questions below. >>>> >>>> Thanks >>>> Parth >>>> On 3/28/15, 8:08 PM, "Gwen Shapira" <gshap...@cloudera.com<mailto: >>>> gshap...@cloudera.com>> wrote: >>>> >>>> Preparing for Tuesday meeting, I went over the KIP :) >>>> >>>> First, Parth did an amazing job, the KIP is fantastic - detailed and >>>> readable. Thank you! >>>> >>>> Second, I have a loooong list of questions :) No objections, just some >>>> things I'm unclear on and random minor comments. In general, I like >>>> the design, I just feel I'm missing parts of the picture. >>>> >>>> 1. "Yes, Create topic will have an optional acls, the output of >>>> describe will display owner and acls and alter topic will allow to >>>> modify the acls.” - will be nice to see what the CLI will look like. >>>> >>>> * I will modify the KIP but I was going to add “—acl >>>><acl-file.json>” >>>> to create-topic and alter-topic. >>>> >>>> 2. I like the addition of Topic owner. We made the mistake of >>>> forgetting about it when adding authorization to Sqoop2. We probably >>>> want to add “chown” command to the topic commands. >>>> >>>> * Again we can add “—owner <user-name>” to alter topic. >>>> >>>> 3. "Kafka server will read "authorizer.class” config value at startup >>>> time, create an instance of the specified class and call initialize >>>> method." >>>> We’ll need to validate that users specify only one of those. >>>> >>>> * The config type will be string so type validation should take >>>>care >>>> of it. >>>> >>>> 4. "One added assumption is that on non-secure connections the session >>>> will have principal set to an object whose name() method will return >>>> "Anonymous”." >>>> Can we keep DrWho? :) >>>> >>>> * Sure, its up to you actually as you are the owner of the jira >>>>that >>>> introduces session concept. >>>> >>>> 5. "For cluster actions that do not apply to a specific topic like >>>> CREATE we have 2 options. We can either add a broker config called >>>> broker.acls which will point to a json file. This file will be >>>> available on all broker hosts and authorizer will read the acls on >>>> initialization and keep refreshing it every X minutes. Any changes >>>> will require re-distribution of the acl json file. Alternatively we >>>> can add a zookeeper path /brokers/acls and store the acl json as data. >>>> Authorizer can refresh the acl from json every X minutes. In absence >>>> of broker acls the authorizer will fail open, in other words it will >>>> allow all users from all hosts to perform all cluster actions” >>>> I prefer a file to ZK - since thats where we store all use-defined >>>> configurations for now. Everyone knows how to secure a file system :) >>>> >>>> * I will let everyone vote, file system is fine by me. >>>> >>>> 6. "When an Acl is missing , this implementation will always fail open >>>> for backward compatibility. “ <- agree, but we need to document that >>>> this makes the default authorizer non-secure >>>> >>>> * Sure. >>>> >>>> 7. "If the value of authorizer.class.name is null, in secure mode the >>>> cluster will fail with ConfigException. In non secure mode in absence >>>> of config value forauthorizer.class.name the server will allow all >>>> requests to all topics that , even if the topic has configured acls” >>>> <- I don’t think Kafka has “secure mode” - it can support SSL and >>>> plaintext (un-authenticated) on two different ports simultaneously. I >>>> don’t object to adding such configuration, but we need to decide >>>> exactly what it means. >>>> >>>> * This is one area of confusion so I will add an open question. >>>> >>>> 8. "Currently all topic creation/modification/deletion actions are >>>> performed using KafkaAdminUtil which mostly interacts directly with >>>> zookeeper instead of forwarding requests to a broker host. Given all >>>> the code is executed on client side there is no easy way to perform >>>> authorization. “ - since we are adding the admin protocol requests in >>>> KIP-4, perhaps addressing those makes sense. >>>> >>>> * Yes, We will have to wait for KIP-4 to be delivered. >>>> >>>> 9. I didn’t see a specification of what is “resource”, I assume its an >>>> enum with things like Topic and… ? >>>> >>>> * Resource is not an enum, Right now its just name of the topic. >>>> Currently we will not be able to support cluster actions like CREATE >>>>in >>>> which case the resource can be some constant string like >>>>“Kafka-Cluster”. >>>> >>>> 10. I’m also unclear on where and how “PermissionType” will be used >>>> and what does “DENY takes precedence” mean here. >>>> >>>> * I originally did not have the “PermissionType” but someone >>>>suggested >>>> in DISCUSS thread that we should add support for ALLOW and DENY acls. >>>>The >>>> use case is to support acls like “Allow user U to Perform READ from >>>>all >>>> Hosts except from Host1 and Host2”. Deny take precedence only means if >>>>you >>>> have define 2 ACLs for the same user/resource/operation. DENY acl will >>>>take >>>> effect I.e. “Allow user X to read from *” and “Deny User X to Read >>>>from >>>> host1 and host2” will satisfy the use case described in the previous >>>> statement. >>>> >>>> 11. "What does acls on zookeeper node look like given all our admin >>>> APIs are currently performed directly from client?” <- “secure mode” >>>> Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both >>>> Kafka and everyone else will need to use them (this is limited to >>>> Kerberos authentication AFAIK.) >>>> >>>> * Yes, again I think this can be a separate issue for now and can >>>>only >>>> be worked on after KIP-4 is delivered. >>>> >>>> 12. "Do we want to support group acls as part of this authorizer? Do >>>> we want to support principal to local user mapping? If yes we need to >>>> add plugins for UserToGroupMapper and PrincipalToUserMapper.” <- >>>> Sentry uses Groups for authorizing, so we need to support that. I >>>> figured that as long as the API specifies Principal, it typically >>>> contains both user and group, so nothing else is needed. Did I miss >>>> anything? >>>> >>>> * Once we support group acls we will need someway to indicate if a >>>> principal is of type group or user(as part of acl) or we can have >>>>group >>>> acls and user acls stored separately with topic config. We will also >>>>need a >>>> way to map an authenticated user to list of groups the user belongs >>>>to. >>>> >>>> 13. It looks like the Authorizer stores the ACLs and not Kafka. So we >>>> need an API for Kafka to notify Authorizer when a topic is added and >>>> when ACLs are modified, right? I didn’t see that. >>>> >>>> * ACLs will be stored under /topic/config at time of topic >>>>creation >>>>in >>>> json format. Authorizer will get these acls using newly introduced >>>> TopicConfigCache which gets updated anytime topic config changes. >>>> >>>> 14. Are we going to have any API for Kafka to give out the ACLs on a >>>> topic? Or we leave this to the Authorizer? >>>> >>>> * I think it is better to leave this out side of Kafka. Mainly >>>>because >>>> most 3rd party authorizers will have their own ACL stores and >>>>configuration >>>> Uis like (Ranger-Argus not sure what are they calling it now). >>>> >>>> >>>> On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede >>>><n...@confluent.io<mailto: >>>> n...@confluent.io>> wrote: >>>> Parth, >>>> >>>> We can make some 15 mins or so to discuss this at the next KIP >>>>hangout. >>>> >>>> Thanks, >>>> Neha >>>> >>>> On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt < >>>> pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com>> >>>>wrote: >>>> >>>> Hi all, >>>> >>>> I have modified the KIP to reflect the recent change request from the >>>> reviewers. I have been working on the code and I have the server side >>>>code >>>> for authorization ready. I am now modifying the command line >>>>utilities. >>>>I >>>> would really appreciate if some of the committers can spend sometime >>>>to >>>> review the KIP so we can make progress on this. >>>> >>>> Thanks >>>> Parth >>>> >>>> On 3/18/15, 2:20 PM, "Michael Herstine" >>>><mherst...@linkedin.com.INVALID >>>> <mailto:mherst...@linkedin.com.INVALID>> >>>> wrote: >>>> >>>> >Hi Parth, >>>> > >>>> >Thanks! A few questions: >>>> > >>>> >1. Do you want to permit rules in your ACLs that DENY access as well >>>>as >>>> >ALLOW? This can be handy setting up rules that have exceptions. E.g. >>>> >“Allow principal P to READ resource R from all hosts” with “Deny >>>>principal >>>> >P READ access to resource R from host H1” in combination would allow >>>>P >>>>to >>>> >READ R from all hosts *except* H1. >>>> > >>>> >2. When a topic is newly created, will there be an ACL created for >>>>it? >>>>If >>>> >not, would that not deny subsequent access to it? >>>> > >>>> >(nit) Maybe use Principal instead of String to represent principals? >>>> > >>>> > >>>> >On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org<mailto: >>>> bo...@apache.org>> wrote: >>>> > >>>> >>Parth >>>> >> >>>> >>Overall it is looking good. Couple of questionsŠ >>>> >> >>>> >>- Can you give an example how the policies will look like in the >>>>default >>>> >>implementation? >>>> >>- In the operations, can we support ³CONNECT² also? This can be used >>>> >>during Session connection >>>> >>- Regarding access control for ³Topic Creation², since we can¹t do >>>>it >>>>on >>>> >>the server side, can we de-scope it for? And plan it as a future >>>>feature >>>> >>request? >>>> >> >>>> >>Thanks >>>> >> >>>> >>Bosco >>>> >> >>>> >> >>>> >> >>>> >>On 3/6/15, 8:10 AM, "Harsha" >>>><ka...@harsha.io<mailto:ka...@harsha.io>> >>>> wrote: >>>> >> >>>> >>>Hi Parth, >>>> >>> Thanks for putting this together. Overall it looks good >>>>to >>>> >>> me. Although AdminUtils is a concern KIP-4 can >>>>probably >>>>fix >>>> >>> that part. >>>> >>>Thanks, >>>> >>>Harsha >>>> >>> >>>> >>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote: >>>> >>>> Forgot to add links to wiki and jira. >>>> >>>> >>>> >>>> Link to wiki: >>>> >>>> >>>> >>>> >>>> >>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio >>>> >>>>n >>>> >>>>+ >>>> >>>>Interface >>>> >>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688 >>>> >>>> >>>> >>>> Thanks >>>> >>>> Parth >>>> >>>> >>>> >>>> From: Parth Brahmbhatt >>>> >>>> <pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com >>>> ><mailto:pbrahmbh...@hortonworks.com>> >>>> >>>> Date: Thursday, March 5, 2015 at 10:33 AM >>>> >>>> To: "dev@kafka.apache.org<mailto:dev@kafka.apache.org><mailto: >>>> dev@kafka.apache.org>" >>>> >>>> <dev@kafka.apache.org<mailto:dev@kafka.apache.org><mailto: >>>> dev@kafka.apache.org>> >>>> >>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka >>>>security >>>> >>>> >>>> >>>> Hi, >>>> >>>> >>>> >>>> KIP-11 is open for discussion , I have updated the wiki with the >>>> >>>>design >>>> >>>> and open questions. >>>> >>>> >>>> >>>> Thanks >>>> >>>> Parth >>>> >> >>>> >> >>>> > >>>> >>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Neha >>>> >>>> >> >>
