Thanks for reviewing Gwen. The wiki already has details on token expiration under token acquisition process <https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition>. Current proposal is that tokens will expire based on a server side configuration (default 24 hours) unless renewed. Renewal is only allowed until the max life time of token. Alternatively we could also make that an optional param and the server side default can serve as the upper bound.
To your second point it will be done exactly the same way we would support multiple keytabs. The calling client will have to put the tokens it wants to use in the subject instance and call produce/consume inside subject.doas. Each caller will have to keep track of its own subject. I will have to look at the code to see if we support this feature right now but my understanding is delegation token shouldn't need any special treatment as its just another type of Credential in the subject. I would also like to know what is your opinion about infinite renewal (my recommendation is to not support this), tokens renewing them self(my recommendation is to not support this) and most importantly your choice between the alternatives listed on this thread <http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism> ( I am leaning towards the alternative-2 minus controller distributing secret). Thanks again for reviewing. Thanks Parth On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <g...@confluent.io> wrote: > Harsha, > > I was thinking of the Rest Proxy. I didn't see your design yet, but in > our proxy, we have a set of producers, which will serve multiple users > going through the proxy. Since these users will have different > privileges, they'll need to authenticate separately, and can't share a > token. > > Am I missing anything? > > Gwen > > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote: > > Gwen, > > On your second point. Can you describe a usecase where > > mutliple clients ended up sharing a producer and even if they > > do why can't they not use single token that producer > > captures. Why would we need multiple clients with different > > tokens sharing a single instance of producer. Also in this > > case other clients have access all the tokens no? > > > > Thanks, > > Harsha > > > > > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote: > >> Sorry for the delay: > >> > >> Two questions that we didn't see in the wiki: > >> 1. Is there an expiration for delegation tokens? Renewal? How do we > >> revoke them? > >> 2. If we want to use delegation tokens for "do-as" (say, submit Storm > >> job as my user), we will need a producer for every job (we can't share > >> them between multiple jobs running on same node), since we only > >> authenticate when connecting. Is there a plan to change this for > >> delegation tokens, in order to allow multiple users with different > >> tokens to share a client? > >> > >> Gwen > >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt > >> <brahmbhatt.pa...@gmail.com> wrote: > >> > Bumping this up one more time, can other committers review? > >> > > >> > Thanks > >> > Parth > >> > > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote: > >> > > >> >> Parth, > >> >> Overall current design looks good to me. I am +1 on the > KIP. > >> >> > >> >> Gwen , Jun can you review this as well. > >> >> > >> >> -Harsha > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote: > >> >> > Thanks for review Jitendra. > >> >> > > >> >> > I don't like the idea of infinite lifetime but I see the Streaming > use > >> >> > case. Even for Streaming use case I was hoping there will be some > notion > >> >> > of > >> >> > master/driver that can get new delegation tokens at fixed interval > and > >> >> > distribute to workers. If that is not the case for we can discuss > >> >> > delegation tokens renewing them self and the security implications > of the > >> >> > same. > >> >> > > >> >> > I did not want clients to fetch tokens from zookeeper, overall I > think > >> >> > its > >> >> > better if clients don't rely on our metadata store and I think we > are > >> >> > moving in that direction with all the KIP-4 improvements. I chose > >> >> > zookeeper as in this case the client will still just talk to > broker , its > >> >> > the brokers that will use zookeeper which we already do for a lot > of > >> >> > other > >> >> > usecases + ease of development + and the ability so tokens will > survive > >> >> > even a rolling restart/cluster failure. if a majority agrees the > added > >> >> > complexity to have controller forwarding keys to all broker is > justified > >> >> > as > >> >> > it provides tighter security , I am fine with that option too. > >> >> > > >> >> > Given zookeeper does not support SSL we can not store master keys > in > >> >> > zookeeper as master keys will be exposed on wire. To support > rotation > >> >> > without affecting current clients is something I need to put more > thought > >> >> > in. My current proposal assumes the rotation will invalidate all > current > >> >> > tokens. > >> >> > > >> >> > I request committers to also review and post their comments so we > can > >> >> > make > >> >> > progress on this KIP. > >> >> > > >> >> > Thanks > >> >> > Parth > >> >> > > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <asi...@cloudera.com > > > >> >> > wrote: > >> >> > > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io> > wrote: > >> >> > > > >> >> > > > Unifying the two discussion threads on this KIP. > >> >> > > > > >> >> > > > Here is the response from Jitendra > >> >> > > > > >> >> > > > "The need for a large number of clients that are running all > over the > >> >> > > > cluster that authenticate with Kafka brokers, is very similar > to the > >> >> > > > Hadoop use case of large number of tasks running across the > cluster > >> >> that > >> >> > > > need authentication to Hdfs Namenode. Therefore, the > delegation token > >> >> > > > approach does seem like a good fit for this use case as we > have seen > >> >> it > >> >> > > > working at large scale in HDFS and YARN. > >> >> > > > > >> >> > > > The proposed design is very much inline with Hadoop > approach. A few > >> >> > > > comments: > >> >> > > > > >> >> > > > 1) Why do you guys want to allow infinite renewable lifetime > for a > >> >> > > > token? HDFS restricts a token to a max life time (default 7 > days). A > >> >> > > > token's vulnerability is believed to increase with time. > >> >> > > > > >> >> > > I agree that having infinite lifetime might not be the best idea. > >> >> > > > >> >> > > > > >> >> > > > 2) As I understand the tokens are stored in zookeeper as well, > and > >> >> can > >> >> > > > be updated there. This is clever as it can allow replacing the > tokens > >> >> > > > once they run out of max life time, and clients can download > new > >> >> tokens > >> >> > > > from zookeeper. It shouldn't be a big load on zookeeper as a > client > >> >> will > >> >> > > > need to get a new token once in several days. In this approach > you > >> >> don't > >> >> > > > need infinite lifetime on the token even for long running > clients. > >> >> > > > > >> >> > > > 3) The token password are generated using a master key. The > master > >> >> key > >> >> > > > should also be periodically changed. In Hadoop, the default > renewal > >> >> > > > period is 1 day.? > >> >> > > > > >> >> > > IIUC, this will require brokers maintaining a list of X most > recent > >> >> master > >> >> > > keys. This list will have to be persisted somewhere, as if a > broker > >> >> goes > >> >> > > down it will have to get that list again and storing master keys > on ZK > >> >> is > >> >> > > not the best idea. However, if a broker goes down then we have > much > >> >> bigger > >> >> > > issue to deal with and client can always re-authenticate is such > >> >> events. > >> >> > > > >> >> > > Did you happen to take a look at other alternatives this list has > >> >> > > suggested? > >> >> > > > >> >> > > > > >> >> > > > Thanks for a thorough proposal, great work!" > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote: > >> >> > > > > Makes sense to me. Thanks! > >> >> > > > > > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io> > wrote: > >> >> > > > > > It doesn't need any release vehicle but still the work can > move > >> >> > > > forward. > >> >> > > > > > If anyone is interested in the KIP please do the review and > >> >> provide > >> >> > > the > >> >> > > > > > comments. > >> >> > > > > > > >> >> > > > > > -Harsha > >> >> > > > > > > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote: > >> >> > > > > >> I agree that it would be good to have more time to review > and > >> >> > > discuss > >> >> > > > > >> KIP-48. > >> >> > > > > >> > >> >> > > > > >> Ismael > >> >> > > > > >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira < > >> >> g...@confluent.io> > >> >> > > > wrote: > >> >> > > > > >> > >> >> > > > > >> > Hi Team, > >> >> > > > > >> > > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit > of a > >> >> risk > >> >> > > for > >> >> > > > > >> > the next release - any chance we can delay delegation > tokens > >> >> to > >> >> > > > Kafka > >> >> > > > > >> > 0.10.1? > >> >> > > > > >> > With the community working on a release every 3 month, > this > >> >> is not > >> >> > > > a huge > >> >> > > > > >> > delay. > >> >> > > > > >> > > >> >> > > > > >> > Gwen > >> >> > > > > >> > > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh < > >> >> > > asi...@cloudera.com> > >> >> > > > wrote: > >> >> > > > > >> > > Parth, > >> >> > > > > >> > > > >> >> > > > > >> > > Thanks again for the awesome write up. Following our > >> >> discussion > >> >> > > > from the > >> >> > > > > >> > > JIRA, I think it will be easier to compare various > >> >> alternatives > >> >> > > > if they > >> >> > > > > >> > are > >> >> > > > > >> > > listed together. I am stating below a few > alternatives along > >> >> > > with > >> >> > > > a the > >> >> > > > > >> > > current proposal. > >> >> > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK. > >> >> > > > > >> > > > >> >> > > > > >> > > 1. Client authenticates with a broker. > >> >> > > > > >> > > 2. Once a client is authenticated, it will make a > broker > >> >> side > >> >> > > > call to > >> >> > > > > >> > > issue a delegation token. > >> >> > > > > >> > > 3. The broker generates a shared secret based on > >> >> > > HMAC-SHA256(a > >> >> > > > > >> > > Password/Secret shared between all brokers, > randomly > >> >> > > generated > >> >> > > > > >> > tokenId). > >> >> > > > > >> > > 4. Broker stores this token in its in memory cache. > >> >> Broker > >> >> > > > also stores > >> >> > > > > >> > > the DelegationToken without the hmac in the > zookeeper. > >> >> > > > > >> > > 5. All brokers will have a cache backed by > zookeeper so > >> >> they > >> >> > > > will all > >> >> > > > > >> > > get notified whenever a new token is generated and > they > >> >> will > >> >> > > > update > >> >> > > > > >> > their > >> >> > > > > >> > > local cache whenever token state changes. > >> >> > > > > >> > > 6. Broker returns the token to Client. > >> >> > > > > >> > > > >> >> > > > > >> > > Probable issues and fixes > >> >> > > > > >> > > > >> >> > > > > >> > > 1. Probable race condition, client tries to > authenticate > >> >> with > >> >> > > > a broker > >> >> > > > > >> > > that is yet to be updated with the newly generated > DT. > >> >> This > >> >> > > can > >> >> > > > > >> > probably be > >> >> > > > > >> > > dealt with making dtRequest block until all > brokers have > >> >> > > > updated > >> >> > > > > >> > their DT > >> >> > > > > >> > > cache. Zk barrier or similar mechanism can be used. > >> >> However, > >> >> > > > all such > >> >> > > > > >> > > mechanisms will increase complexity. > >> >> > > > > >> > > 2. Using a static secret key from config file. Will > >> >> require > >> >> > > yet > >> >> > > > > >> > another > >> >> > > > > >> > > config and uses a static secret key. It is advised > to > >> >> rotate > >> >> > > > secret > >> >> > > > > >> > keys > >> >> > > > > >> > > periodically. This can be avoided with controller > >> >> generating > >> >> > > > > >> > secretKey and > >> >> > > > > >> > > passing to brokers periodically. However, this will > >> >> require > >> >> > > > brokers to > >> >> > > > > >> > > maintain certain counts of secretKeys. > >> >> > > > > >> > > > >> >> > > > > >> > > (Alternative 1) Have controller generate delegation > token. > >> >> > > > > >> > > > >> >> > > > > >> > > 1. Client authenticates with a broker. > >> >> > > > > >> > > 2. Once a client is authenticated, it will make a > broker > >> >> side > >> >> > > > call to > >> >> > > > > >> > > issue a delegation token. > >> >> > > > > >> > > 3. Broker forwards the request to controller. > >> >> > > > > >> > > 4. Controller generates a DT and broadcasts to all > >> >> brokers. > >> >> > > > > >> > > 5. Broker stores this token in its memory cache. > >> >> > > > > >> > > 6. Controller responds to broker’s DT req. > >> >> > > > > >> > > 7. Broker returns the token to Client. > >> >> > > > > >> > > > >> >> > > > > >> > > Probable issues and fixes > >> >> > > > > >> > > > >> >> > > > > >> > > 1. We will have to add new APIs to support > controller > >> >> pushing > >> >> > > > tokens > >> >> > > > > >> > to > >> >> > > > > >> > > brokers on top of the minimal APIs that are > currently > >> >> > > proposed. > >> >> > > > > >> > > 2. We will also have to add APIs to support the > >> >> bootstrapping > >> >> > > > case, > >> >> > > > > >> > i.e, > >> >> > > > > >> > > when a new broker comes up it will have to get all > >> >> delegation > >> >> > > > tokens > >> >> > > > > >> > from > >> >> > > > > >> > > the controller. > >> >> > > > > >> > > 3. In catastrophic failures where all brokers go > down, > >> >> the > >> >> > > > tokens will > >> >> > > > > >> > > be lost even if servers are restarted as tokens > are not > >> >> > > > persisted > >> >> > > > > >> > anywhere. > >> >> > > > > >> > > If this happens, then there are more important > things to > >> >> > > worry > >> >> > > > about > >> >> > > > > >> > and > >> >> > > > > >> > > maybe it is better to re-authenticate. > >> >> > > > > >> > > > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to other brokers > at > >> >> all. > >> >> > > > > >> > > > >> >> > > > > >> > > 1. Client authenticates with a broker. > >> >> > > > > >> > > 2. Once a client is authenticated, it will make a > broker > >> >> side > >> >> > > > call to > >> >> > > > > >> > > issue a delegation token. > >> >> > > > > >> > > 3. The broker generates DT of form, [hmac + (owner, > >> >> renewer, > >> >> > > > > >> > > maxLifeTime, id, hmac, expirationTime)] and passes > back > >> >> this > >> >> > > > DT to > >> >> > > > > >> > client. > >> >> > > > > >> > > hmac is generated via {HMAC-SHA256(owner, renewer, > >> >> > > > maxLifeTime, id, > >> >> > > > > >> > hmac, > >> >> > > > > >> > > expirationTime) using SecretKey}. Note that all > brokers > >> >> have > >> >> > > > this > >> >> > > > > >> > SecretKey. > >> >> > > > > >> > > 4. Client then goes to any broker and to > authenticate > >> >> sends > >> >> > > > the DT. > >> >> > > > > >> > > Broker recalculates hmac using (owner, renewer, > >> >> maxLifeTime, > >> >> > > > id, hmac, > >> >> > > > > >> > > expirationTime) info from DT and its SecretKey. If > it > >> >> matches > >> >> > > > with > >> >> > > > > >> > hmac of > >> >> > > > > >> > > DT, client is authenticated. Yes, it will do other > >> >> obvious > >> >> > > > checks of > >> >> > > > > >> > > timestamp expiry and such. > >> >> > > > > >> > > > >> >> > > > > >> > > Note that secret key will be generated by controller > and > >> >> passed > >> >> > > to > >> >> > > > > >> > brokers > >> >> > > > > >> > > periodically. > >> >> > > > > >> > > Probable issues and fixes > >> >> > > > > >> > > > >> >> > > > > >> > > 1. How to delete a DT? Yes, that is a downside > here. > >> >> However, > >> >> > > > this can > >> >> > > > > >> > > be handled with brokers maintaining a blacklist of > DTs, > >> >> DTs > >> >> > > > from this > >> >> > > > > >> > list > >> >> > > > > >> > > can be removed after expiry. > >> >> > > > > >> > > 2. In catastrophic failures where all brokers go > down, > >> >> the > >> >> > > > tokens will > >> >> > > > > >> > > be lost even if servers are restarted as tokens > are not > >> >> > > > persisted > >> >> > > > > >> > anywhere. > >> >> > > > > >> > > If this happens, then there are more important > things to > >> >> > > worry > >> >> > > > about > >> >> > > > > >> > and > >> >> > > > > >> > > maybe it is better to re-authenticate. > >> >> > > > > >> > > > >> >> > > > > >> > > > >> >> > > > > >> > > > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt < > >> >> > > > > >> > > pbrahmbh...@hortonworks.com> wrote: > >> >> > > > > >> > > > >> >> > > > > >> > >> Hi, > >> >> > > > > >> > >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like > delegation > >> >> > > > tokens in > >> >> > > > > >> > >> kafka. You can review the design > >> >> > > > > >> > >> > >> >> > > > > >> > > >> >> > > > > >> >> > > > >> >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka > >> >> > > > > >> > . > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also > discussed an > >> >> > > > alternative to > >> >> > > > > >> > >> proposed design here< > >> >> > > > > >> > >> > >> >> > > > > >> > > >> >> > > > > >> >> > > > >> >> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800 > >> >> > > > > >> > >> >. > >> >> > > > > >> > >> > >> >> > > > > >> > >> Thanks > >> >> > > > > >> > >> Parth > >> >> > > > > >> > >> > >> >> > > > > >> > > > >> >> > > > > >> > > > >> >> > > > > >> > > > >> >> > > > > >> > > -- > >> >> > > > > >> > > > >> >> > > > > >> > > Regards, > >> >> > > > > >> > > Ashish > >> >> > > > > >> > > >> >> > > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > -- > >> >> > > > >> >> > > Regards, > >> >> > > Ashish > >> >> > > > >> >> >
