Parth, Thanks for the reply.
It makes sense to only allow the renewal by users that authenticated using *non* delegation token mechanism. Then, should we make the renewal a list? For example, in the case of rest proxy, it will be useful for every instance of rest proxy to be able to renew the tokens. It would be clearer if we can document the request protocol like https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0) . It would also be useful to document the client APIs. Thanks, Jun On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt < brahmbhatt.pa...@gmail.com> wrote: > Hi, > > I am suggesting that we will only allow the renewal by users that > authenticated using *non* delegation token mechanism. For example, If user > Alice authenticated using kerberos and requested delegation tokens, only > user Alice authenticated via non delegation token mechanism can renew. > Clients that have access to delegation tokens can not issue renewal > request for renewing their own token and this is primarily important to > reduce the time window for which a compromised token will be valid. > > To clarify, Yes any authenticated user can request delegation tokens but > even here I would recommend to avoid creating a chain where a client > authenticated via delegation token request for more delegation tokens. > Basically anyone can request delegation token, as long as they authenticate > via a non delegation token mechanism. > > Aren't classes listed here > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-PublicInterfaces > > > sufficient? > > Thanks > Parth > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <j...@confluent.io> wrote: > > > Parth, > > > > Thanks for the reply. A couple of comments inline below. > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt < > > brahmbhatt.pa...@gmail.com> wrote: > > > > > 1. Who / how are tokens renewed? By original requester only? or using > > > Kerberos > > > auth only? > > > My recommendation is to do this only using Kerberos auth and only threw > > the > > > renewer specified during the acquisition request. > > > > > > > > Hmm, not sure that I follow this. Are you saying that any client > > authenticated with the delegation token can renew, i.e. there is no > renewer > > needed? > > > > Also, just to be clear, any authenticated client (either through SASL or > > SSL) can request a delegation token for the authenticated user, right? > > > > > > > 2. Are tokens stored on each broker or in ZK? > > > My recommendation is still to store in ZK or not store them at all. The > > > whole controller based distribution is too much overhead with not much > to > > > achieve. > > > > > > 3. How are tokens invalidated / expired? > > > Either by expiration time out or through an explicit request to > > invalidate. > > > > > > 4. Which encryption algorithm is used? > > > SCRAM > > > > > > 5. What is the impersonation proposal (it wasn't in the KIP but was > > > discussed > > > in this thread)? > > > There is no imperonation proposal. I tried and explained how its a > > > different problem and why its not really necessary to discuss that as > > part > > > of this KIP. This KIP will not support any impersonation, it will just > > be > > > another way to authenticate. > > > > > > 6. Do we need new ACLs, if so - for what actions? > > > We do not need new ACLs. > > > > > > > > Could we document the format of the new request/response and their > > associated Resource and Operation for ACL? > > > > > > > 7. How would the delegation token be configured in the client? > > > Should be through config. I wasn't planning on supporting JAAS for > > tokens. > > > I don't believe hadoop does this either. > > > > > > Thanks > > > Parth > > > > > > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > Harsha, > > > > > > > > Another question. > > > > > > > > 9. How would the delegation token be configured in the client? The > > > standard > > > > way is to do this through JAAS. However, we will need to think > through > > if > > > > this is convenient in a shared environment. For example, when a new > > task > > > is > > > > added to a Storm worker node, do we need to dynamically add a new > > section > > > > in the JAAS file? It may be more convenient if we can pass in the > token > > > > through the config directly w/o going through JAAS. > > > > > > > > Are you or Parth still actively working on this KIP? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > Just to add on that list. > > > > > > > > > > 2. It would be good to document the format of the data stored in > ZK. > > > > > 7. Earlier, there was a discussion on whether the tokens should be > > > > > propagated through ZK like config/acl/quota, or through the > > controller. > > > > > Currently, the controller is only designed for propagating topic > > > > metadata, > > > > > but not other data. > > > > > 8. Should we use SCRAM to send the token instead of DIGEST-MD5 > since > > > it's > > > > > deprecated? > > > > > > > > > > Also, the images in the wiki seem broken. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <g...@confluent.io> > > > > wrote: > > > > > > > > > >> From what I can see, remaining questions are: > > > > >> > > > > >> 1. Who / how are tokens renewed? By original requester only? or > > using > > > > >> Kerberos auth only? > > > > >> 2. Are tokens stored on each broker or in ZK? > > > > >> 3. How are tokens invalidated / expired? > > > > >> 4. Which encryption algorithm is used? > > > > >> 5. What is the impersonation proposal (it wasn't in the KIP but > was > > > > >> discussed in this thread)? > > > > >> 6. Do we need new ACLs, if so - for what actions? > > > > >> > > > > >> Gwen > > > > >> > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io> wrote: > > > > >> > Jun & Ismael, > > > > >> > Unfortunately I couldn't attend the KIP > > > > meeting > > > > >> > when delegation tokens discussed. > > > Appreciate > > > > if > > > > >> > you can update the thread if you have > any > > > > >> > further questions. > > > > >> > Thanks, > > > > >> > Harsha > > > > >> > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote: > > > > >> >> It seems that the links to images in the KIP are broken. > > > > >> >> > > > > >> >> Liquan > > > > >> >> > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt < > > > > >> >> brahmbhatt.pa...@gmail.com> wrote: > > > > >> >> > > > > >> >> > 110. What does getDelegationTokenAs mean? > > > > >> >> > In the current proposal we only allow a user to get > delegation > > > > token > > > > >> for > > > > >> >> > the identity that it authenticated as using another > mechanism, > > > i.e. > > > > >> A user > > > > >> >> > that authenticate using a keytab for principal > > us...@example.com > > > > >> will get > > > > >> >> > delegation tokens for that user only. In future I think we > will > > > > have > > > > >> to > > > > >> >> > extend support such that we allow some set of users ( > > > > >> >> > kafka-rest-u...@example.com, storm-nim...@example.com) to > > > acquire > > > > >> >> > delegation tokens on behalf of other users whose identity > they > > > have > > > > >> >> > verified independently. Kafka brokers will have ACLs to > > control > > > > >> which > > > > >> >> > users are allowed to impersonate other users and get tokens > on > > > > >> behalf of > > > > >> >> > them. Overall Impersonation is a whole different problem in > my > > > > >> opinion and > > > > >> >> > I think we can tackle it in separate KIP. > > > > >> >> > > > > > >> >> > 111. What's the typical rate of getting and renewing > delegation > > > > >> tokens? > > > > >> >> > Typically this should be very very low, 1 request per minute > > is a > > > > >> >> > relatively high estimate. However it depends on the token > > > > >> expiration. I am > > > > >> >> > less worried about the extra load it puts on controller vs > the > > > > added > > > > >> >> > complexity and the value it offers. > > > > >> >> > > > > > >> >> > Thanks > > > > >> >> > Parth > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma < > > ism...@juma.me.uk> > > > > >> wrote: > > > > >> >> > > > > > >> >> > > Thanks Rajini. It would probably require a separate KIP as > it > > > > will > > > > >> >> > > introduce user visible changes. We could also update KIP-48 > > to > > > > >> have this > > > > >> >> > > information, but it seems cleaner to do it separately. We > can > > > > >> discuss > > > > >> >> > that > > > > >> >> > > in the KIP call today. > > > > >> >> > > > > > > >> >> > > Ismael > > > > >> >> > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram < > > > > >> >> > > rajinisiva...@googlemail.com> wrote: > > > > >> >> > > > > > > >> >> > > > Ismael, > > > > >> >> > > > > > > > >> >> > > > I have created a JIRA ( > > > > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751) > > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would that need > > another > > > > >> KIP? If > > > > >> >> > > > KIP-48 will use this mechanism, can this just be a JIRA > > that > > > > gets > > > > >> >> > > reviewed > > > > >> >> > > > when the PR is ready? > > > > >> >> > > > > > > > >> >> > > > Thank you, > > > > >> >> > > > > > > > >> >> > > > Rajini > > > > >> >> > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma < > > > > ism...@juma.me.uk> > > > > >> >> > wrote: > > > > >> >> > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good candidate. > > > > >> >> > > > > > > > > >> >> > > > > Gwen had independently mentioned this as a SASL > mechanism > > > > that > > > > >> might > > > > >> >> > be > > > > >> >> > > > > useful for Kafka and I have been meaning to check it in > > > more > > > > >> detail. > > > > >> >> > > Good > > > > >> >> > > > > to know that you are willing to contribute an > > > implementation. > > > > >> Maybe > > > > >> >> > we > > > > >> >> > > > > should file a separate JIRA for this? > > > > >> >> > > > > > > > > >> >> > > > > Ismael > > > > >> >> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram < > > > > >> >> > > > > rajinisiva...@googlemail.com> wrote: > > > > >> >> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response Authentication > > > Mechanism) > > > > >> is a > > > > >> >> > > better > > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't come with a > > > > built-in > > > > >> SCRAM > > > > >> >> > > > > > SaslServer or SaslClient, but I will be happy to add > > > > support > > > > >> in > > > > >> >> > Kafka > > > > >> >> > > > > since > > > > >> >> > > > > > it would be a useful mechanism to support anyway. > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677 describes the > > > protocol > > > > >> for > > > > >> >> > > > > > SCRAM-SHA-256. > > > > >> >> > > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao < > > > j...@confluent.io > > > > > > > > > >> wrote: > > > > >> >> > > > > > > > > > >> >> > > > > > > Parth, > > > > >> >> > > > > > > > > > > >> >> > > > > > > Thanks for the explanation. A couple of more > > questions. > > > > >> >> > > > > > > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs mean? > > > > >> >> > > > > > > > > > > >> >> > > > > > > 111. What's the typical rate of getting and > renewing > > > > >> delegation > > > > >> >> > > > tokens? > > > > >> >> > > > > > > That may have an impact on whether they should be > > > > directed > > > > >> to the > > > > >> >> > > > > > > controller. > > > > >> >> > > > > > > > > > > >> >> > > > > > > Jun > > > > >> >> > > > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt < > > > > >> >> > > > > > > brahmbhatt.pa...@gmail.com> wrote: > > > > >> >> > > > > > > > > > > >> >> > > > > > > > Hi Jun, > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > Thanks for reviewing. > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > * We could add a Cluster action to add acls on > who > > > can > > > > >> request > > > > >> >> > > > > > delegation > > > > >> >> > > > > > > > tokens. I don't see the use case for that yet but > > > down > > > > >> the line > > > > >> >> > > > when > > > > >> >> > > > > we > > > > >> >> > > > > > > > start supporting getDelegationTokenAs it will be > > > > >> necessary. > > > > >> >> > > > > > > > * Yes we recommend tokens to be only > > used/distributed > > > > >> over > > > > >> >> > secure > > > > >> >> > > > > > > channels. > > > > >> >> > > > > > > > * Depending on what design we end up choosing > > > > >> Invalidation will > > > > >> >> > > be > > > > >> >> > > > > > > > responsibility of every broker or controller. > > > > >> >> > > > > > > > * I am not sure if I documented somewhere that > > > > >> invalidation > > > > >> >> > will > > > > >> >> > > > > > directly > > > > >> >> > > > > > > > go through zookeeper but that is not the intent. > > > > >> Invalidation > > > > >> >> > > will > > > > >> >> > > > > > either > > > > >> >> > > > > > > > be request based or due to expiration. No direct > > > > >> zookeeper > > > > >> >> > > > > interaction > > > > >> >> > > > > > > from > > > > >> >> > > > > > > > any client. > > > > >> >> > > > > > > > * "Broker also stores the DelegationToken without > > the > > > > >> hmac in > > > > >> >> > the > > > > >> >> > > > > > > > zookeeper." : Sorry about the confusion. The sole > > > > >> purpose of > > > > >> >> > > > > zookeeper > > > > >> >> > > > > > in > > > > >> >> > > > > > > > this design is as distribution channel for tokens > > > > >> between all > > > > >> >> > > > brokers > > > > >> >> > > > > > > and a > > > > >> >> > > > > > > > layer that ensures only tokens that were > generated > > by > > > > >> making a > > > > >> >> > > > > request > > > > >> >> > > > > > > to a > > > > >> >> > > > > > > > broker will be accepted (more on this in second > > > > >> paragraph). The > > > > >> >> > > > token > > > > >> >> > > > > > > > consists of few elements (owner, renewer, uuid , > > > > >> expiration, > > > > >> >> > > hmac) > > > > >> >> > > > , > > > > >> >> > > > > > one > > > > >> >> > > > > > > of > > > > >> >> > > > > > > > which is the finally generated hmac but hmac it > > self > > > is > > > > >> >> > derivable > > > > >> >> > > > if > > > > >> >> > > > > > you > > > > >> >> > > > > > > > have all the other elements of the token + secret > > key > > > > to > > > > >> >> > generate > > > > >> >> > > > > hmac. > > > > >> >> > > > > > > > Given zookeeper does not provide SSL support we > do > > > not > > > > >> want the > > > > >> >> > > > > entire > > > > >> >> > > > > > > > token to be wire transferred to zookeeper as that > > > will > > > > >> be an > > > > >> >> > > > insecure > > > > >> >> > > > > > > wire > > > > >> >> > > > > > > > transfer. Instead we only store all the other > > > elements > > > > >> of a > > > > >> >> > > > > delegation > > > > >> >> > > > > > > > tokens. Brokers can read these elements and > because > > > > they > > > > >> also > > > > >> >> > > have > > > > >> >> > > > > > access > > > > >> >> > > > > > > > to secret key they will be able to generate hmac > on > > > > >> their end. > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > One of the alternative proposed is to avoid > > zookeeper > > > > >> >> > > altogether. A > > > > >> >> > > > > > > Client > > > > >> >> > > > > > > > will call broker with required information > (owner, > > > > >> renwer, > > > > >> >> > > > > expiration) > > > > >> >> > > > > > > and > > > > >> >> > > > > > > > get back (signed hmac, uuid). Broker won't store > > this > > > > in > > > > >> >> > > zookeeper. > > > > >> >> > > > > > From > > > > >> >> > > > > > > > this point a client can contact any broker with > all > > > the > > > > >> >> > > delegation > > > > >> >> > > > > > token > > > > >> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the > > > borker > > > > >> will > > > > >> >> > > > > regenerate > > > > >> >> > > > > > > the > > > > >> >> > > > > > > > hmac and as long as it matches with hmac > presented > > by > > > > >> client , > > > > >> >> > > > broker > > > > >> >> > > > > > > will > > > > >> >> > > > > > > > allow the request to authenticate. Only problem > > with > > > > >> this > > > > >> >> > > approach > > > > >> >> > > > > is > > > > >> >> > > > > > if > > > > >> >> > > > > > > > the secret key is compromised any client can now > > > > generate > > > > >> >> > random > > > > >> >> > > > > tokens > > > > >> >> > > > > > > and > > > > >> >> > > > > > > > they will still be able to authenticate as any > user > > > > they > > > > >> like. > > > > >> >> > > with > > > > >> >> > > > > > > > zookeeper we guarantee that only tokens acquired > > via > > > a > > > > >> broker > > > > >> >> > > > (using > > > > >> >> > > > > > some > > > > >> >> > > > > > > > auth scheme other than delegation token) will be > > > > >> accepted. We > > > > >> >> > > need > > > > >> >> > > > to > > > > >> >> > > > > > > > discuss which proposal makes more sense and we > can > > go > > > > >> over it > > > > >> >> > in > > > > >> >> > > > > > > tomorrow's > > > > >> >> > > > > > > > meeting. > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > Also, can you forward the invite to me? > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > Thanks > > > > >> >> > > > > > > > Parth > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao < > > > > >> j...@confluent.io> > > > > >> >> > > > wrote: > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few comments. > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > 100. This potentially can be useful for Kafka > > > Connect > > > > >> and > > > > >> >> > Kafka > > > > >> >> > > > > rest > > > > >> >> > > > > > > > proxy > > > > >> >> > > > > > > > > where a worker agent will need to run a task on > > > > behalf > > > > >> of a > > > > >> >> > > > client. > > > > >> >> > > > > > We > > > > >> >> > > > > > > > will > > > > >> >> > > > > > > > > likely need to change how those services use > > Kafka > > > > >> clients > > > > >> >> > > > > > > > > (producer/consumer). Instead of a shared client > > per > > > > >> worker, > > > > >> >> > we > > > > >> >> > > > will > > > > >> >> > > > > > > need > > > > >> >> > > > > > > > a > > > > >> >> > > > > > > > > client per user task since the authentication > > > happens > > > > >> at the > > > > >> >> > > > > > connection > > > > >> >> > > > > > > > > level. For Kafka Connect, the renewer will be > the > > > > >> workers. > > > > >> >> > So, > > > > >> >> > > we > > > > >> >> > > > > > > > probably > > > > >> >> > > > > > > > > need to allow multiple renewers. For Kafka rest > > > > proxy, > > > > >> the > > > > >> >> > > > renewer > > > > >> >> > > > > > can > > > > >> >> > > > > > > > > probably just be the creator of the token. > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who can request > > > delegation > > > > >> tokens? > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > 102. Do we recommend people to send delegation > > > tokens > > > > >> in an > > > > >> >> > > > > encrypted > > > > >> >> > > > > > > > > channel? > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > 103. Who is responsible for expiring tokens, > > every > > > > >> broker? > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > 104. For invalidating tokens, would it be > better > > to > > > > do > > > > >> it in > > > > >> >> > a > > > > >> >> > > > > > request > > > > >> >> > > > > > > > > instead of going to ZK directly? > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > 105. The terminology of client in the wiki > > > sometimes > > > > >> refers > > > > >> >> > to > > > > >> >> > > > the > > > > >> >> > > > > > end > > > > >> >> > > > > > > > > client and some other times refers to the > client > > > > using > > > > >> the > > > > >> >> > > > > delegation > > > > >> >> > > > > > > > > tokens. It would be useful to distinguish > between > > > the > > > > >> two. > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > 106. Could you explain the sentence "Broker > also > > > > >> stores the > > > > >> >> > > > > > > > DelegationToken > > > > >> >> > > > > > > > > without the hmac in the zookeeper." a bit > more? I > > > > >> thought the > > > > >> >> > > > > > > delegation > > > > >> >> > > > > > > > > token is the hmac. > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > Thanks, > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > Jun > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao < > > > > >> j...@confluent.io> > > > > >> >> > > > wrote: > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > Hi, Harsha, > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting invite. We can > > > discuss > > > > >> this in > > > > >> >> > > the > > > > >> >> > > > > > > meeting > > > > >> >> > > > > > > > > > tomorrow. > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > Thanks, > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > Jun > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha < > > > > >> ka...@harsha.io> > > > > >> >> > > > wrote: > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > >> Hi All, > > > > >> >> > > > > > > > > >> Can we have a KIP meeting around > > > this. > > > > >> The KIP > > > > >> >> > is > > > > >> >> > > > up > > > > >> >> > > > > > for > > > > >> >> > > > > > > > > >> sometime and if there are any > > > questions > > > > >> lets > > > > >> >> > > > quickly > > > > >> >> > > > > > hash > > > > >> >> > > > > > > > out > > > > >> >> > > > > > > > > >> details. > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> Thanks, > > > > >> >> > > > > > > > > >> Harsha > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth > > > > brahmbhatt > > > > >> wrote: > > > > >> >> > > > > > > > > >> > That is what the hadoop echo system uses > so > > no > > > > >> good > > > > >> >> > reason > > > > >> >> > > > > > really. > > > > >> >> > > > > > > > We > > > > >> >> > > > > > > > > >> > could > > > > >> >> > > > > > > > > >> > change it to whatever is the newest > > > recommended > > > > >> standard > > > > >> >> > > is. > > > > >> >> > > > > > > > > >> > > > > > >> >> > > > > > > > > >> > Thanks > > > > >> >> > > > > > > > > >> > Parth > > > > >> >> > > > > > > > > >> > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael > > Juma < > > > > >> >> > > > > ism...@juma.me.uk > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > wrote: > > > > >> >> > > > > > > > > >> > > > > > >> >> > > > > > > > > >> > > Hi Parth, > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only started > > reviewing > > > > >> this and > > > > >> >> > > may > > > > >> >> > > > > have > > > > >> >> > > > > > > > > >> additional > > > > >> >> > > > > > > > > >> > > questions later. The immediate question > > that > > > > >> came to > > > > >> >> > > mind > > > > >> >> > > > is > > > > >> >> > > > > > our > > > > >> >> > > > > > > > > >> choice of > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as > > > > >> OBSOLETE in > > > > >> >> > the > > > > >> >> > > > IANA > > > > >> >> > > > > > > > > Registry > > > > >> >> > > > > > > > > >> of > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the original RFC > > (2831) > > > > has > > > > >> been > > > > >> >> > > moved > > > > >> >> > > > > to > > > > >> >> > > > > > > > > Historic > > > > >> >> > > > > > > > > >> > > status: > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331 > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > >> > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > What is the reasoning behind that > choice? > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > Thanks, > > > > >> >> > > > > > > > > >> > > Ismael > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen > > > > Shapira < > > > > >> >> > > > > > > g...@confluent.io > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > >> wrote: > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > > Also comments inline :) > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > * I want to emphasize that even > though > > > > >> delegation > > > > >> >> > > > tokens > > > > >> >> > > > > > > are a > > > > >> >> > > > > > > > > >> Hadoop > > > > >> >> > > > > > > > > >> > > > > innovation, I feel very strongly > about > > > not > > > > >> adding > > > > >> >> > > > > > dependency > > > > >> >> > > > > > > > on > > > > >> >> > > > > > > > > >> Hadoop > > > > >> >> > > > > > > > > >> > > > > when implementing delegation tokens > > for > > > > >> Kafka. The > > > > >> >> > > KIP > > > > >> >> > > > > > > doesn't > > > > >> >> > > > > > > > > >> imply > > > > >> >> > > > > > > > > >> > > > > such dependency, but if you can > > > clarify... > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.* > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so no > one > > > will > > > > >> read > > > > >> >> > the > > > > >> >> > > > KIP > > > > >> >> > > > > > and > > > > >> >> > > > > > > > > panic > > > > >> >> > > > > > > > > >> > > > three weeks before the next release... > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get delegation token at any > > > time > > > > >> after > > > > >> >> > > > > > > > authenticating? > > > > >> >> > > > > > > > > >> only > > > > >> >> > > > > > > > > >> > > > > immediately after? > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you are authenticated > you > > > can > > > > >> get > > > > >> >> > > > delegation > > > > >> >> > > > > > > > tokens. > > > > >> >> > > > > > > > > >> We > > > > >> >> > > > > > > > > >> > > need > > > > >> >> > > > > > > > > >> > > > to > > > > >> >> > > > > > > > > >> > > > > discuss if a client authenticated > > using > > > > >> delegation > > > > >> >> > > > > token, > > > > >> >> > > > > > > can > > > > >> >> > > > > > > > > also > > > > >> >> > > > > > > > > >> > > > acquire > > > > >> >> > > > > > > > > >> > > > > delegation token again or not. Also > > > there > > > > >> is the > > > > >> >> > > > > question > > > > >> >> > > > > > of > > > > >> >> > > > > > > > do > > > > >> >> > > > > > > > > we > > > > >> >> > > > > > > > > >> > > allow > > > > >> >> > > > > > > > > >> > > > > anyone to acquire delegation token > or > > we > > > > >> want > > > > >> >> > > specific > > > > >> >> > > > > > ACLs > > > > >> >> > > > > > > (I > > > > >> >> > > > > > > > > >> think > > > > >> >> > > > > > > > > >> > > its > > > > >> >> > > > > > > > > >> > > > an > > > > >> >> > > > > > > > > >> > > > > overkill.)* > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an overkill. > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > I think we are debating two options: > > > Either > > > > >> require > > > > >> >> > > > > Kerberos > > > > >> >> > > > > > > > auth > > > > >> >> > > > > > > > > >> for > > > > >> >> > > > > > > > > >> > > > renewal or require non-owners to > renew. > > > > >> >> > > > > > > > > >> > > > I *think* the latter is simpler (it > > > > basically > > > > >> >> > require > > > > >> >> > > a > > > > >> >> > > > > "job > > > > >> >> > > > > > > > > master" > > > > >> >> > > > > > > > > >> > > > to take responsibility for the > renewal, > > it > > > > >> will have > > > > >> >> > > its > > > > >> >> > > > > own > > > > >> >> > > > > > > > > >> identity > > > > >> >> > > > > > > > > >> > > > anyway and I think this is the correct > > > > design > > > > >> >> > pattern > > > > >> >> > > > > > anyway. > > > > >> >> > > > > > > > For > > > > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate > > > > >> renewals?), > > > > >> >> > but > > > > >> >> > > > it > > > > >> >> > > > > is > > > > >> >> > > > > > > > hard > > > > >> >> > > > > > > > > to > > > > >> >> > > > > > > > > >> > > > debate simplicity without looking at > the > > > > code > > > > >> >> > changes > > > > >> >> > > > > > > required. > > > > >> >> > > > > > > > If > > > > >> >> > > > > > > > > >> you > > > > >> >> > > > > > > > > >> > > > have a draft of how the "require > > Kerberos" > > > > >> will look > > > > >> >> > > in > > > > >> >> > > > > > Kafka > > > > >> >> > > > > > > > > code, > > > > >> >> > > > > > > > > >> > > > I'll be happy to take a look. > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > * My understanding is that tokens > will > > > > >> propagate > > > > >> >> > via > > > > >> >> > > > ZK > > > > >> >> > > > > > but > > > > >> >> > > > > > > > > >> without > > > > >> >> > > > > > > > > >> > > > > additional changes to UpdateMetadata > > > > >> protocol, > > > > >> >> > > > correct? > > > > >> >> > > > > > > > Clients > > > > >> >> > > > > > > > > >> > > > > currently don't retry on SASL auth > > > failure > > > > >> (IIRC), > > > > >> >> > > but > > > > >> >> > > > > > since > > > > >> >> > > > > > > > the > > > > >> >> > > > > > > > > >> > > > > tokens propagate between brokers > > asynch, > > > > we > > > > >> will > > > > >> >> > > need > > > > >> >> > > > to > > > > >> >> > > > > > > > retry a > > > > >> >> > > > > > > > > >> bit > > > > >> >> > > > > > > > > >> > > > > to avoid clients failing auth due to > > > > timing > > > > >> >> > issues. > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > *I am considering 2 alternatives > right > > > > now. > > > > >> The > > > > >> >> > > > current > > > > >> >> > > > > > > > > documented > > > > >> >> > > > > > > > > >> > > > approach > > > > >> >> > > > > > > > > >> > > > > is zookeeper based and it does not > > > require > > > > >> any > > > > >> >> > > changes > > > > >> >> > > > > to > > > > >> >> > > > > > > > > >> > > UpdateMetadata > > > > >> >> > > > > > > > > >> > > > > protocol. An alternative approach > can > > > > remove > > > > >> >> > > zookeeper > > > > >> >> > > > > > > > > dependency > > > > >> >> > > > > > > > > >> as > > > > >> >> > > > > > > > > >> > > well > > > > >> >> > > > > > > > > >> > > > > but we can discuss that in KIP > > > discussion > > > > >> call.* > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want > to > > > > ping > > > > >> Jun to > > > > >> >> > > > > > arrange a > > > > >> >> > > > > > > > > call? > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of > > having > > > > >> just the > > > > >> >> > > > > > controller > > > > >> >> > > > > > > > > issue > > > > >> >> > > > > > > > > >> the > > > > >> >> > > > > > > > > >> > > > > delegation tokens, to avoid syncing > a > > > > shared > > > > >> >> > secret. > > > > >> >> > > > Not > > > > >> >> > > > > > > sure > > > > >> >> > > > > > > > if > > > > >> >> > > > > > > > > >> we > > > > >> >> > > > > > > > > >> > > > > want to continue the discussion here > > or > > > on > > > > >> the > > > > >> >> > > wiki. I > > > > >> >> > > > > > think > > > > >> >> > > > > > > > > that > > > > >> >> > > > > > > > > >> we > > > > >> >> > > > > > > > > >> > > > > can decouple the problem of "token > > > > >> distribution" > > > > >> >> > > from > > > > >> >> > > > > > > "shared > > > > >> >> > > > > > > > > >> secret > > > > >> >> > > > > > > > > >> > > > > distribution" and use the controller > > as > > > > the > > > > >> only > > > > >> >> > > token > > > > >> >> > > > > > > > generator > > > > >> >> > > > > > > > > >> to > > > > >> >> > > > > > > > > >> > > > > solve the second issue, while still > > > using > > > > >> ZK async > > > > >> >> > > to > > > > >> >> > > > > > > > distribute > > > > >> >> > > > > > > > > >> > > > > tokens. > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the previous Email > I > > am > > > > >> fine with > > > > >> >> > > > that > > > > >> >> > > > > > > > approach > > > > >> >> > > > > > > > > >> as > > > > >> >> > > > > > > > > >> > > long > > > > >> >> > > > > > > > > >> > > > as > > > > >> >> > > > > > > > > >> > > > > we agree that the extra complexity > of > > > > >> >> > > adding/updating > > > > >> >> > > > > APIS > > > > >> >> > > > > > > > adds > > > > >> >> > > > > > > > > >> enough > > > > >> >> > > > > > > > > >> > > > > value. The advantage with the > > controller > > > > >> approach > > > > >> >> > is > > > > >> >> > > > > > secret > > > > >> >> > > > > > > > > >> rotation > > > > >> >> > > > > > > > > >> > > can > > > > >> >> > > > > > > > > >> > > > be > > > > >> >> > > > > > > > > >> > > > > automated,frequent and would not > > require > > > > >> >> > > deployment. * > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > Can you detail the extra complexity > (or > > > > point > > > > >> me to > > > > >> >> > > the > > > > >> >> > > > > > email > > > > >> >> > > > > > > I > > > > >> >> > > > > > > > > >> > > > missed?) - which Apis are required? > > > > >> >> > > > > > > > > >> > > > As far as I can tell, clients can > > already > > > > >> find the > > > > >> >> > > > > > controller > > > > >> >> > > > > > > > from > > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned > about > > > > >> controller > > > > >> >> > > > load. > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > * While I like the idea of forcing > > > > kerberos > > > > >> auth > > > > >> >> > for > > > > >> >> > > > > > > renwal, I > > > > >> >> > > > > > > > > >> think > > > > >> >> > > > > > > > > >> > > > > it mixes the transport layer the the > > > > request > > > > >> >> > content > > > > >> >> > > > in > > > > >> >> > > > > a > > > > >> >> > > > > > > > pretty > > > > >> >> > > > > > > > > >> ugly > > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to > > > non-owner > > > > >> is > > > > >> >> > > better. > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > *I feel this is a necessary evil. > > While > > > > >> this will > > > > >> >> > > make > > > > >> >> > > > > the > > > > >> >> > > > > > > > kafka > > > > >> >> > > > > > > > > >> code > > > > >> >> > > > > > > > > >> > > > > pretty straight forward , forcing > > > renewer > > > > >> to > > > > >> >> > > > non-owner > > > > >> >> > > > > > > pushes > > > > >> >> > > > > > > > > >> the code > > > > >> >> > > > > > > > > >> > > > > ugliness to client and makes it even > > > > harder > > > > >> to > > > > >> >> > > > > > integrate. * > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > As mentioned before, I don't think the > > > > >> "renewal by > > > > >> >> > > > other" > > > > >> >> > > > > > > > approach > > > > >> >> > > > > > > > > >> is > > > > >> >> > > > > > > > > >> > > > that ugly for the clients we expect to > > use > > > > >> >> > delegation > > > > >> >> > > > > tokens > > > > >> >> > > > > > > > since > > > > >> >> > > > > > > > > >> > > > they will have an app-master of some > > sort > > > > who > > > > >> >> > > requested > > > > >> >> > > > > the > > > > >> >> > > > > > > > token > > > > >> >> > > > > > > > > to > > > > >> >> > > > > > > > > >> > > > begin with. > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > The response for my question on how > > > > multiple > > > > >> >> > > > identities > > > > >> >> > > > > > will > > > > >> >> > > > > > > > be > > > > >> >> > > > > > > > > >> > > > > handled wasn't super clear to me - > > > AFAIK, > > > > >> we don't > > > > >> >> > > > > > > > authenticate > > > > >> >> > > > > > > > > >> each > > > > >> >> > > > > > > > > >> > > > > request, we authenticate > connections. > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > *We authenticate connections, and > only > > > > when > > > > >> they > > > > >> >> > are > > > > >> >> > > > > being > > > > >> >> > > > > > > > > >> established. > > > > >> >> > > > > > > > > >> > > > Let > > > > >> >> > > > > > > > > >> > > > > me try to phrase this as a question, > > in > > > > >> absence of > > > > >> >> > > > > > > delegation > > > > >> >> > > > > > > > > >> tokens if > > > > >> >> > > > > > > > > >> > > > we > > > > >> >> > > > > > > > > >> > > > > had to support the use case using > user > > > > >> TGT's how > > > > >> >> > > would > > > > >> >> > > > > we > > > > >> >> > > > > > do > > > > >> >> > > > > > > > it? > > > > >> >> > > > > > > > > >> My > > > > >> >> > > > > > > > > >> > > point > > > > >> >> > > > > > > > > >> > > > > was it would be no different with > > > > delegation > > > > >> >> > tokens. > > > > >> >> > > > The > > > > >> >> > > > > > use > > > > >> >> > > > > > > > > case > > > > >> >> > > > > > > > > >> you > > > > >> >> > > > > > > > > >> > > are > > > > >> >> > > > > > > > > >> > > > > describing seems more like > > > impersonation.* > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of the things > > > that > > > > >> >> > delegation > > > > >> >> > > > > > tokens > > > > >> >> > > > > > > > > >> handled. > > > > >> >> > > > > > > > > >> > > > Maybe I got it wrong :) > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > Thanks for the detailed answers. > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > Gwen > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > Thanks > > > > >> >> > > > > > > > > >> > > > > Parth > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, > Gwen > > > > >> Shapira < > > > > >> >> > > > > > > > > g...@confluent.io > > > > >> >> > > > > > > > > >> > > > > > >> >> > > > > > > > > >> > > > wrote: > > > > >> >> > > > > > > > > >> > > > > > > > > >> >> > > > > > > > > >> > > > >> Hi Parth and Harsha, > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> Few more comments: > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * The API / RequestResponse section > > > > >> doesn't seem > > > > >> >> > to > > > > >> >> > > > > have > > > > >> >> > > > > > > good > > > > >> >> > > > > > > > > >> > > > >> description of the changes to the > > Kafka > > > > >> Protocol. > > > > >> >> > > > > Sounds > > > > >> >> > > > > > > like > > > > >> >> > > > > > > > > >> you are > > > > >> >> > > > > > > > > >> > > > >> proposing new > DelegationTokenRequest > > > and > > > > >> >> > > > > > RenewTokenRequest > > > > >> >> > > > > > > > (and > > > > >> >> > > > > > > > > >> > > > >> matching responses), without > > detailing > > > > the > > > > >> >> > contents > > > > >> >> > > > of > > > > >> >> > > > > > the > > > > >> >> > > > > > > > > >> requests > > > > >> >> > > > > > > > > >> > > > >> and responses? Or rather, you show > > the > > > > >> class > > > > >> >> > > > interface, > > > > >> >> > > > > > but > > > > >> >> > > > > > > > not > > > > >> >> > > > > > > > > >> the > > > > >> >> > > > > > > > > >> > > > >> underlying protocol. This could be > > seen > > > > as > > > > >> an > > > > >> >> > > > > > > implementation > > > > >> >> > > > > > > > > >> detail, > > > > >> >> > > > > > > > > >> > > > >> but since the binary protocol is > what > > > we > > > > >> provide > > > > >> >> > to > > > > >> >> > > > > > > non-Java > > > > >> >> > > > > > > > > >> clients, > > > > >> >> > > > > > > > > >> > > > >> we need to show the changes there. > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * getDelegationToken sounds like > > > > >> >> > > > > > > > delegationTokenRequestHandler? > > > > >> >> > > > > > > > > >> Is it > > > > >> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or > > > > Client? > > > > >> Its > > > > >> >> > > > > unclear... > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * I want to emphasize that even > > though > > > > >> delegation > > > > >> >> > > > > tokens > > > > >> >> > > > > > > are > > > > >> >> > > > > > > > a > > > > >> >> > > > > > > > > >> Hadoop > > > > >> >> > > > > > > > > >> > > > >> innovation, I feel very strongly > > about > > > > not > > > > >> adding > > > > >> >> > > > > > > dependency > > > > >> >> > > > > > > > on > > > > >> >> > > > > > > > > >> Hadoop > > > > >> >> > > > > > > > > >> > > > >> when implementing delegation tokens > > for > > > > >> Kafka. > > > > >> >> > The > > > > >> >> > > > KIP > > > > >> >> > > > > > > > doesn't > > > > >> >> > > > > > > > > >> imply > > > > >> >> > > > > > > > > >> > > > >> such dependency, but if you can > > > > clarify... > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * Can we get delegation token at > any > > > time > > > > >> after > > > > >> >> > > > > > > > authenticating? > > > > >> >> > > > > > > > > >> only > > > > >> >> > > > > > > > > >> > > > >> immediately after? > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * My understanding is that tokens > > will > > > > >> propagate > > > > >> >> > > via > > > > >> >> > > > ZK > > > > >> >> > > > > > but > > > > >> >> > > > > > > > > >> without > > > > >> >> > > > > > > > > >> > > > >> additional changes to > UpdateMetadata > > > > >> protocol, > > > > >> >> > > > correct? > > > > >> >> > > > > > > > Clients > > > > >> >> > > > > > > > > >> > > > >> currently don't retry on SASL auth > > > > failure > > > > >> >> > (IIRC), > > > > >> >> > > > but > > > > >> >> > > > > > > since > > > > >> >> > > > > > > > > the > > > > >> >> > > > > > > > > >> > > > >> tokens propagate between brokers > > > asynch, > > > > >> we will > > > > >> >> > > need > > > > >> >> > > > > to > > > > >> >> > > > > > > > retry > > > > >> >> > > > > > > > > a > > > > >> >> > > > > > > > > >> bit > > > > >> >> > > > > > > > > >> > > > >> to avoid clients failing auth due > to > > > > timing > > > > >> >> > issues. > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients not > > > > >> touching ZK > > > > >> >> > > > directly > > > > >> >> > > > > > :) > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of > > having > > > > >> just the > > > > >> >> > > > > > controller > > > > >> >> > > > > > > > > >> issue the > > > > >> >> > > > > > > > > >> > > > >> delegation tokens, to avoid > syncing a > > > > >> shared > > > > >> >> > > secret. > > > > >> >> > > > > Not > > > > >> >> > > > > > > sure > > > > >> >> > > > > > > > > if > > > > >> >> > > > > > > > > >> we > > > > >> >> > > > > > > > > >> > > > >> want to continue the discussion > here > > or > > > > on > > > > >> the > > > > >> >> > > wiki. > > > > >> >> > > > I > > > > >> >> > > > > > > think > > > > >> >> > > > > > > > > >> that we > > > > >> >> > > > > > > > > >> > > > >> can decouple the problem of "token > > > > >> distribution" > > > > >> >> > > from > > > > >> >> > > > > > > "shared > > > > >> >> > > > > > > > > >> secret > > > > >> >> > > > > > > > > >> > > > >> distribution" and use the > controller > > as > > > > >> the only > > > > >> >> > > > token > > > > >> >> > > > > > > > > generator > > > > >> >> > > > > > > > > >> to > > > > >> >> > > > > > > > > >> > > > >> solve the second issue, while still > > > using > > > > >> ZK > > > > >> >> > async > > > > >> >> > > to > > > > >> >> > > > > > > > > distribute > > > > >> >> > > > > > > > > >> > > > >> tokens. > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * I am also uncomfortable with > > infinite > > > > >> lifetime > > > > >> >> > of > > > > >> >> > > > > > tokens > > > > >> >> > > > > > > > (and > > > > >> >> > > > > > > > > >> hoped > > > > >> >> > > > > > > > > >> > > > >> to hear from others in the > > community) - > > > > but > > > > >> >> > having > > > > >> >> > > > the > > > > >> >> > > > > > > option > > > > >> >> > > > > > > > > and > > > > >> >> > > > > > > > > >> > > > >> documenting it as less secure, > allows > > > > >> users to > > > > >> >> > > > > configure > > > > >> >> > > > > > > > their > > > > >> >> > > > > > > > > >> system > > > > >> >> > > > > > > > > >> > > > >> as the wish. > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * While I like the idea of forcing > > > > >> kerberos auth > > > > >> >> > > for > > > > >> >> > > > > > > renwal, > > > > >> >> > > > > > > > I > > > > >> >> > > > > > > > > >> think > > > > >> >> > > > > > > > > >> > > > >> it mixes the transport layer the > the > > > > >> request > > > > >> >> > > content > > > > >> >> > > > > in a > > > > >> >> > > > > > > > > pretty > > > > >> >> > > > > > > > > >> ugly > > > > >> >> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to > > > > non-owner > > > > >> is > > > > >> >> > > better. > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> Things I'd still like to see: > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * More detailed explanation on what > > we > > > > >> plan to do > > > > >> >> > > for > > > > >> >> > > > > the > > > > >> >> > > > > > > > java > > > > >> >> > > > > > > > > >> clients > > > > >> >> > > > > > > > > >> > > > >> specifically - new configuration? > new > > > > APIs? > > > > >> >> > > > > > > > > >> > > > >> The response for my question on how > > > > >> multiple > > > > >> >> > > > identities > > > > >> >> > > > > > > will > > > > >> >> > > > > > > > be > > > > >> >> > > > > > > > > >> > > > >> handled wasn't super clear to me - > > > AFAIK, > > > > >> we > > > > >> >> > don't > > > > >> >> > > > > > > > authenticate > > > > >> >> > > > > > > > > >> each > > > > >> >> > > > > > > > > >> > > > >> request, we authenticate > connections. > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens > are > > > > only > > > > >> used > > > > >> >> > in > > > > >> >> > > > the > > > > >> >> > > > > > > Hadoop > > > > >> >> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there > are > > > > >> >> > alternatives > > > > >> >> > > in > > > > >> >> > > > > > other > > > > >> >> > > > > > > > > >> ecosystems > > > > >> >> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and > > > whether > > > > >> there > > > > >> >> > are > > > > >> >> > > > some > > > > >> >> > > > > > > > > >> advantages > > > > >> >> > > > > > > > > >> > > > >> there. > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> Gwen > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, > > > Harsha < > > > > >> >> > > > > ka...@harsha.io > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > wrote: > > > > >> >> > > > > > > > > >> > > > >> > Hi Gwen, > > > > >> >> > > > > > > > > >> > > > >> > Can you look at > Parth's > > > last > > > > >> reply. > > > > >> >> > > Does > > > > >> >> > > > > it > > > > >> >> > > > > > > > answer > > > > >> >> > > > > > > > > >> your > > > > >> >> > > > > > > > > >> > > > >> > concerns. > > > > >> >> > > > > > > > > >> > > > >> > > > > > >> >> > > > > > > > > >> > > > >> > Thanks, > > > > >> >> > > > > > > > > >> > > > >> > Harsha > > > > >> >> > > > > > > > > >> > > > >> > > > > > >> >> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, > > > parth > > > > >> >> > > brahmbhatt > > > > >> >> > > > > > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The > > wiki > > > > >> already > > > > >> >> > has > > > > >> >> > > > > > details > > > > >> >> > > > > > > on > > > > >> >> > > > > > > > > >> token > > > > >> >> > > > > > > > > >> > > > >> >> expiration > > > > >> >> > > > > > > > > >> > > > >> >> under token acquisition process > > > > >> >> > > > > > > > > >> > > > >> >> < > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition > > > > >> >> > > > > > > > > >> > > > >> >. > > > > >> >> > > > > > > > > >> > > > >> >> Current proposal is that tokens > > will > > > > >> expire > > > > >> >> > > based > > > > >> >> > > > > on a > > > > >> >> > > > > > > > > server > > > > >> >> > > > > > > > > >> side > > > > >> >> > > > > > > > > >> > > > >> >> configuration (default 24 hours) > > > > unless > > > > >> >> > renewed. > > > > >> >> > > > > > Renewal > > > > >> >> > > > > > > > is > > > > >> >> > > > > > > > > >> only > > > > >> >> > > > > > > > > >> > > > allowed > > > > >> >> > > > > > > > > >> > > > >> >> until the max life time of > token. > > > > >> >> > Alternatively > > > > >> >> > > we > > > > >> >> > > > > > could > > > > >> >> > > > > > > > > also > > > > >> >> > > > > > > > > >> make > > > > >> >> > > > > > > > > >> > > > that > > > > >> >> > > > > > > > > >> > > > >> >> an > > > > >> >> > > > > > > > > >> > > > >> >> optional param and the server > side > > > > >> default can > > > > >> >> > > > serve > > > > >> >> > > > > > as > > > > >> >> > > > > > > > the > > > > >> >> > > > > > > > > >> upper > > > > >> >> > > > > > > > > >> > > > bound. > > > > >> >> > > > > > > > > >> > > > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> To your second point it will be > > done > > > > >> exactly > > > > >> >> > the > > > > >> >> > > > > same > > > > >> >> > > > > > > way > > > > >> >> > > > > > > > we > > > > >> >> > > > > > > > > >> would > > > > >> >> > > > > > > > > >> > > > >> >> support > > > > >> >> > > > > > > > > >> > > > >> >> multiple keytabs. The calling > > client > > > > >> will have > > > > >> >> > > to > > > > >> >> > > > > put > > > > >> >> > > > > > > the > > > > >> >> > > > > > > > > >> tokens it > > > > >> >> > > > > > > > > >> > > > >> wants > > > > >> >> > > > > > > > > >> > > > >> >> to use in the subject instance > and > > > > call > > > > >> >> > > > > > produce/consume > > > > >> >> > > > > > > > > inside > > > > >> >> > > > > > > > > >> > > > >> >> subject.doas. Each caller will > > have > > > to > > > > >> keep > > > > >> >> > > track > > > > >> >> > > > of > > > > >> >> > > > > > its > > > > >> >> > > > > > > > own > > > > >> >> > > > > > > > > >> > > > subject. I > > > > >> >> > > > > > > > > >> > > > >> >> will have to look at the code to > > see > > > > if > > > > >> we > > > > >> >> > > support > > > > >> >> > > > > > this > > > > >> >> > > > > > > > > >> feature > > > > >> >> > > > > > > > > >> > > right > > > > >> >> > > > > > > > > >> > > > >> now > > > > >> >> > > > > > > > > >> > > > >> >> but my understanding is > delegation > > > > token > > > > >> >> > > shouldn't > > > > >> >> > > > > > need > > > > >> >> > > > > > > > any > > > > >> >> > > > > > > > > >> special > > > > >> >> > > > > > > > > >> > > > >> >> treatment as its just another > type > > > of > > > > >> >> > Credential > > > > >> >> > > > in > > > > >> >> > > > > > the > > > > >> >> > > > > > > > > >> subject. > > > > >> >> > > > > > > > > >> > > > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> I would also like to know what > is > > > your > > > > >> opinion > > > > >> >> > > > about > > > > >> >> > > > > > > > > infinite > > > > >> >> > > > > > > > > >> > > renewal > > > > >> >> > > > > > > > > >> > > > >> (my > > > > >> >> > > > > > > > > >> > > > >> >> recommendation is to not support > > > > this), > > > > >> tokens > > > > >> >> > > > > > renewing > > > > >> >> > > > > > > > them > > > > >> >> > > > > > > > > >> > > self(my > > > > >> >> > > > > > > > > >> > > > >> >> recommendation is to not support > > > this) > > > > >> and > > > > >> >> > most > > > > >> >> > > > > > > > importantly > > > > >> >> > > > > > > > > >> your > > > > >> >> > > > > > > > > >> > > > choice > > > > >> >> > > > > > > > > >> > > > >> >> between the alternatives listed > on > > > > this > > > > >> thread > > > > >> >> > > > > > > > > >> > > > >> >> < > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism > > > > >> >> > > > > > > > > >> > > > >> > > > > > >> >> > > > > > > > > >> > > > >> >> ( I am leaning towards the > > > > >> alternative-2 minus > > > > >> >> > > > > > > controller > > > > >> >> > > > > > > > > >> > > > distributing > > > > >> >> > > > > > > > > >> > > > >> >> secret). Thanks again for > > reviewing. > > > > >> >> > > > > > > > > >> > > > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> Thanks > > > > >> >> > > > > > > > > >> > > > >> >> Parth > > > > >> >> > > > > > > > > >> > > > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, > > Gwen > > > > >> Shapira < > > > > >> >> > > > > > > > > >> g...@confluent.io> > > > > >> >> > > > > > > > > >> > > > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> > Harsha, > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > I was thinking of the Rest > > Proxy. > > > I > > > > >> didn't > > > > >> >> > see > > > > >> >> > > > > your > > > > >> >> > > > > > > > design > > > > >> >> > > > > > > > > >> yet, > > > > >> >> > > > > > > > > >> > > > but in > > > > >> >> > > > > > > > > >> > > > >> >> > our proxy, we have a set of > > > > >> producers, which > > > > >> >> > > > will > > > > >> >> > > > > > > serve > > > > >> >> > > > > > > > > >> multiple > > > > >> >> > > > > > > > > >> > > > users > > > > >> >> > > > > > > > > >> > > > >> >> > going through the proxy. Since > > > these > > > > >> users > > > > >> >> > > will > > > > >> >> > > > > have > > > > >> >> > > > > > > > > >> different > > > > >> >> > > > > > > > > >> > > > >> >> > privileges, they'll need to > > > > >> authenticate > > > > >> >> > > > > separately, > > > > >> >> > > > > > > and > > > > >> >> > > > > > > > > >> can't > > > > >> >> > > > > > > > > >> > > > share a > > > > >> >> > > > > > > > > >> > > > >> >> > token. > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > Am I missing anything? > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > Gwen > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 > PM, > > > > >> Harsha < > > > > >> >> > > > > > > ka...@harsha.io > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > >> wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > > Gwen, > > > > >> >> > > > > > > > > >> > > > >> >> > > On your second > > point. > > > > >> Can you > > > > >> >> > > > > describe > > > > >> >> > > > > > a > > > > >> >> > > > > > > > > >> usecase > > > > >> >> > > > > > > > > >> > > where > > > > >> >> > > > > > > > > >> > > > >> >> > > mutliple clients > > > ended > > > > up > > > > >> >> > > sharing a > > > > >> >> > > > > > > > producer > > > > >> >> > > > > > > > > >> and > > > > >> >> > > > > > > > > >> > > even > > > > >> >> > > > > > > > > >> > > > if > > > > >> >> > > > > > > > > >> > > > >> they > > > > >> >> > > > > > > > > >> > > > >> >> > > do why can't they > > not > > > > use > > > > >> >> > single > > > > >> >> > > > > token > > > > >> >> > > > > > > that > > > > >> >> > > > > > > > > >> producer > > > > >> >> > > > > > > > > >> > > > >> >> > > captures. Why > would > > > we > > > > >> need > > > > >> >> > > > multiple > > > > >> >> > > > > > > > clients > > > > >> >> > > > > > > > > >> with > > > > >> >> > > > > > > > > >> > > > >> different > > > > >> >> > > > > > > > > >> > > > >> >> > > tokens sharing a > > > single > > > > >> >> > instance > > > > >> >> > > of > > > > >> >> > > > > > > > producer. > > > > >> >> > > > > > > > > >> Also > > > > >> >> > > > > > > > > >> > > in > > > > >> >> > > > > > > > > >> > > > >> this > > > > >> >> > > > > > > > > >> > > > >> >> > > case other > clients > > > have > > > > >> access > > > > >> >> > > all > > > > >> >> > > > > the > > > > >> >> > > > > > > > tokens > > > > >> >> > > > > > > > > >> no? > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > > Thanks, > > > > >> >> > > > > > > > > >> > > > >> >> > > Harsha > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at > 11:49 > > > AM, > > > > >> Gwen > > > > >> >> > > Shapira > > > > >> >> > > > > > > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> Sorry for the delay: > > > > >> >> > > > > > > > > >> > > > >> >> > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> Two questions that we > didn't > > > see > > > > >> in the > > > > >> >> > > wiki: > > > > >> >> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration > for > > > > >> delegation > > > > >> >> > > > > tokens? > > > > >> >> > > > > > > > > >> Renewal? How > > > > >> >> > > > > > > > > >> > > > do we > > > > >> >> > > > > > > > > >> > > > >> >> > >> revoke them? > > > > >> >> > > > > > > > > >> > > > >> >> > >> 2. If we want to use > > delegation > > > > >> tokens > > > > >> >> > for > > > > >> >> > > > > > "do-as" > > > > >> >> > > > > > > > > (say, > > > > >> >> > > > > > > > > >> > > submit > > > > >> >> > > > > > > > > >> > > > >> Storm > > > > >> >> > > > > > > > > >> > > > >> >> > >> job as my user), we will > > need a > > > > >> producer > > > > >> >> > > for > > > > >> >> > > > > > every > > > > >> >> > > > > > > > job > > > > >> >> > > > > > > > > >> (we > > > > >> >> > > > > > > > > >> > > can't > > > > >> >> > > > > > > > > >> > > > >> share > > > > >> >> > > > > > > > > >> > > > >> >> > >> them between multiple jobs > > > > running > > > > >> on > > > > >> >> > same > > > > >> >> > > > > node), > > > > >> >> > > > > > > > since > > > > >> >> > > > > > > > > >> we > > > > >> >> > > > > > > > > >> > > only > > > > >> >> > > > > > > > > >> > > > >> >> > >> authenticate when > connecting. > > > Is > > > > >> there a > > > > >> >> > > plan > > > > >> >> > > > > to > > > > >> >> > > > > > > > change > > > > >> >> > > > > > > > > >> this > > > > >> >> > > > > > > > > >> > > for > > > > >> >> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order > > to > > > > >> allow > > > > >> >> > > multiple > > > > >> >> > > > > > users > > > > >> >> > > > > > > > > with > > > > >> >> > > > > > > > > >> > > > different > > > > >> >> > > > > > > > > >> > > > >> >> > >> tokens to share a client? > > > > >> >> > > > > > > > > >> > > > >> >> > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> Gwen > > > > >> >> > > > > > > > > >> > > > >> >> > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 > > AM, > > > > >> parth > > > > >> >> > > > > brahmbhatt > > > > >> >> > > > > > > > > >> > > > >> >> > >> < > brahmbhatt.pa...@gmail.com> > > > > >> wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more > > > time, > > > > >> can > > > > >> >> > other > > > > >> >> > > > > > > committers > > > > >> >> > > > > > > > > >> review? > > > > >> >> > > > > > > > > >> > > > >> >> > >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> > Thanks > > > > >> >> > > > > > > > > >> > > > >> >> > >> > Parth > > > > >> >> > > > > > > > > >> > > > >> >> > >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at > > 9:07 > > > > AM, > > > > >> >> > Harsha < > > > > >> >> > > > > > > > > >> ka...@harsha.io> > > > > >> >> > > > > > > > > >> > > > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> Parth, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> Overall > current > > > > >> design looks > > > > >> >> > > > good > > > > >> >> > > > > to > > > > >> >> > > > > > > > me. I > > > > >> >> > > > > > > > > >> am +1 > > > > >> >> > > > > > > > > >> > > on > > > > >> >> > > > > > > > > >> > > > >> the > > > > >> >> > > > > > > > > >> > > > >> >> > KIP. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you > review > > > this > > > > >> as > > > > >> >> > well. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at > > > 09:57 > > > > >> AM, > > > > >> >> > parth > > > > >> >> > > > > > > > brahmbhatt > > > > >> >> > > > > > > > > >> wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review > > > Jitendra. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea > of > > > > >> infinite > > > > >> >> > > > lifetime > > > > >> >> > > > > > > but I > > > > >> >> > > > > > > > > >> see the > > > > >> >> > > > > > > > > >> > > > >> Streaming > > > > >> >> > > > > > > > > >> > > > >> >> > use > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > case. Even for > Streaming > > > use > > > > >> case I > > > > >> >> > > was > > > > >> >> > > > > > hoping > > > > >> >> > > > > > > > > >> there will > > > > >> >> > > > > > > > > >> > > > be > > > > >> >> > > > > > > > > >> > > > >> some > > > > >> >> > > > > > > > > >> > > > >> >> > notion > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > of > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can > > get > > > > new > > > > >> >> > > > delegation > > > > >> >> > > > > > > tokens > > > > >> >> > > > > > > > > at > > > > >> >> > > > > > > > > >> fixed > > > > >> >> > > > > > > > > >> > > > >> interval > > > > >> >> > > > > > > > > >> > > > >> >> > and > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers. > > If > > > > >> that is > > > > >> >> > not > > > > >> >> > > > the > > > > >> >> > > > > > case > > > > >> >> > > > > > > > for > > > > >> >> > > > > > > > > >> we can > > > > >> >> > > > > > > > > >> > > > >> discuss > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens > > renewing > > > > >> them self > > > > >> >> > > and > > > > >> >> > > > > the > > > > >> >> > > > > > > > > >> security > > > > >> >> > > > > > > > > >> > > > >> implications > > > > >> >> > > > > > > > > >> > > > >> >> > of the > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > same. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients > > to > > > > >> fetch > > > > >> >> > tokens > > > > >> >> > > > > from > > > > >> >> > > > > > > > > >> zookeeper, > > > > >> >> > > > > > > > > >> > > > >> overall I > > > > >> >> > > > > > > > > >> > > > >> >> > think > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > its > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > better if clients > don't > > > rely > > > > >> on our > > > > >> >> > > > > metadata > > > > >> >> > > > > > > > store > > > > >> >> > > > > > > > > >> and I > > > > >> >> > > > > > > > > >> > > > >> think we > > > > >> >> > > > > > > > > >> > > > >> >> > are > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > moving in that > direction > > > > with > > > > >> all > > > > >> >> > the > > > > >> >> > > > > KIP-4 > > > > >> >> > > > > > > > > >> improvements. > > > > >> >> > > > > > > > > >> > > > I > > > > >> >> > > > > > > > > >> > > > >> chose > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this > > case > > > > the > > > > >> client > > > > >> >> > > > will > > > > >> >> > > > > > > still > > > > >> >> > > > > > > > > >> just talk > > > > >> >> > > > > > > > > >> > > > to > > > > >> >> > > > > > > > > >> > > > >> >> > broker , its > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will > > use > > > > >> zookeeper > > > > >> >> > > > which > > > > >> >> > > > > we > > > > >> >> > > > > > > > > >> already do > > > > >> >> > > > > > > > > >> > > > for a > > > > >> >> > > > > > > > > >> > > > >> lot > > > > >> >> > > > > > > > > >> > > > >> >> > of > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > other > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of > > > > >> development + and > > > > >> >> > > the > > > > >> >> > > > > > > ability > > > > >> >> > > > > > > > > so > > > > >> >> > > > > > > > > >> > > tokens > > > > >> >> > > > > > > > > >> > > > >> will > > > > >> >> > > > > > > > > >> > > > >> >> > survive > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > even a rolling > > > > restart/cluster > > > > >> >> > > failure. > > > > >> >> > > > > if a > > > > >> >> > > > > > > > > >> majority > > > > >> >> > > > > > > > > >> > > > agrees > > > > >> >> > > > > > > > > >> > > > >> the > > > > >> >> > > > > > > > > >> > > > >> >> > added > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > complexity to have > > > > controller > > > > >> >> > > forwarding > > > > >> >> > > > > > keys > > > > >> >> > > > > > > to > > > > >> >> > > > > > > > > all > > > > >> >> > > > > > > > > >> > > > broker is > > > > >> >> > > > > > > > > >> > > > >> >> > justified > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > as > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter > > > security > > > > >> , I am > > > > >> >> > > fine > > > > >> >> > > > > > with > > > > >> >> > > > > > > > that > > > > >> >> > > > > > > > > >> option > > > > >> >> > > > > > > > > >> > > > too. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does > not > > > > >> support SSL > > > > >> >> > > we > > > > >> >> > > > > can > > > > >> >> > > > > > > not > > > > >> >> > > > > > > > > >> store > > > > >> >> > > > > > > > > >> > > > master > > > > >> >> > > > > > > > > >> > > > >> keys > > > > >> >> > > > > > > > > >> > > > >> >> > in > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master > keys > > > > will > > > > >> be > > > > >> >> > > exposed > > > > >> >> > > > > on > > > > >> >> > > > > > > > wire. > > > > >> >> > > > > > > > > To > > > > >> >> > > > > > > > > >> > > > support > > > > >> >> > > > > > > > > >> > > > >> >> > rotation > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > without affecting > > current > > > > >> clients is > > > > >> >> > > > > > > something I > > > > >> >> > > > > > > > > >> need to > > > > >> >> > > > > > > > > >> > > > put > > > > >> >> > > > > > > > > >> > > > >> more > > > > >> >> > > > > > > > > >> > > > >> >> > thought > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > in. My current > proposal > > > > >> assumes the > > > > >> >> > > > > rotation > > > > >> >> > > > > > > > will > > > > >> >> > > > > > > > > >> > > > invalidate > > > > >> >> > > > > > > > > >> > > > >> all > > > > >> >> > > > > > > > > >> > > > >> >> > current > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > tokens. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I request committers > to > > > also > > > > >> review > > > > >> >> > > and > > > > >> >> > > > > post > > > > >> >> > > > > > > > their > > > > >> >> > > > > > > > > >> > > comments > > > > >> >> > > > > > > > > >> > > > >> so we > > > > >> >> > > > > > > > > >> > > > >> >> > can > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > make > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Parth > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 > at > > > 8:39 > > > > >> AM, > > > > >> >> > > Ashish > > > > >> >> > > > > > Singh > > > > >> >> > > > > > > < > > > > >> >> > > > > > > > > >> > > > >> asi...@cloudera.com > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 > > at > > > > >> 11:26 AM, > > > > >> >> > > > > Harsha < > > > > >> >> > > > > > > > > >> > > > ka...@harsha.io> > > > > >> >> > > > > > > > > >> > > > >> >> > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two > > > > >> discussion > > > > >> >> > > threads > > > > >> >> > > > on > > > > >> >> > > > > > > this > > > > >> >> > > > > > > > > KIP. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the > response > > > > from > > > > >> >> > Jitendra > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a > > large > > > > >> number of > > > > >> >> > > > > clients > > > > >> >> > > > > > > that > > > > >> >> > > > > > > > > are > > > > >> >> > > > > > > > > >> > > > running > > > > >> >> > > > > > > > > >> > > > >> all > > > > >> >> > > > > > > > > >> > > > >> >> > over the > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that > > > > authenticate > > > > >> with > > > > >> >> > > Kafka > > > > >> >> > > > > > > > brokers, > > > > >> >> > > > > > > > > >> is very > > > > >> >> > > > > > > > > >> > > > >> similar > > > > >> >> > > > > > > > > >> > > > >> >> > to the > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of > > > large > > > > >> number > > > > >> >> > of > > > > >> >> > > > > tasks > > > > >> >> > > > > > > > > running > > > > >> >> > > > > > > > > >> > > across > > > > >> >> > > > > > > > > >> > > > >> the > > > > >> >> > > > > > > > > >> > > > >> >> > cluster > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> that > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need > authentication > > to > > > > >> Hdfs > > > > >> >> > > > Namenode. > > > > >> >> > > > > > > > > >> Therefore, the > > > > >> >> > > > > > > > > >> > > > >> >> > delegation token > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem > > > like > > > > a > > > > >> good > > > > >> >> > fit > > > > >> >> > > > for > > > > >> >> > > > > > > this > > > > >> >> > > > > > > > > use > > > > >> >> > > > > > > > > >> case > > > > >> >> > > > > > > > > >> > > > as we > > > > >> >> > > > > > > > > >> > > > >> >> > have seen > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> it > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > working at large > > scale > > > > in > > > > >> HDFS > > > > >> >> > and > > > > >> >> > > > > YARN. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > The proposed > > design > > > is > > > > >> very > > > > >> >> > much > > > > >> >> > > > > > inline > > > > >> >> > > > > > > > with > > > > >> >> > > > > > > > > >> Hadoop > > > > >> >> > > > > > > > > >> > > > >> >> > approach. A few > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > comments: > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys > > > want > > > > >> to allow > > > > >> >> > > > > > infinite > > > > >> >> > > > > > > > > >> renewable > > > > >> >> > > > > > > > > >> > > > >> lifetime > > > > >> >> > > > > > > > > >> > > > >> >> > for a > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS > > restricts > > > a > > > > >> token > > > > >> >> > to a > > > > >> >> > > > max > > > > >> >> > > > > > > life > > > > >> >> > > > > > > > > time > > > > >> >> > > > > > > > > >> > > > (default > > > > >> >> > > > > > > > > >> > > > >> 7 > > > > >> >> > > > > > > > > >> > > > >> >> > days). A > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token's > > vulnerability > > > is > > > > >> >> > believed > > > > >> >> > > to > > > > >> >> > > > > > > > increase > > > > >> >> > > > > > > > > >> with > > > > >> >> > > > > > > > > >> > > > time. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having > > > > infinite > > > > >> >> > > lifetime > > > > >> >> > > > > > might > > > > >> >> > > > > > > > not > > > > >> >> > > > > > > > > >> be the > > > > >> >> > > > > > > > > >> > > > best > > > > >> >> > > > > > > > > >> > > > >> idea. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand > > the > > > > >> tokens > > > > >> >> > are > > > > >> >> > > > > stored > > > > >> >> > > > > > > in > > > > >> >> > > > > > > > > >> zookeeper > > > > >> >> > > > > > > > > >> > > > as > > > > >> >> > > > > > > > > >> > > > >> well, > > > > >> >> > > > > > > > > >> > > > >> >> > and > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> can > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there. > > This > > > > is > > > > >> clever > > > > >> >> > > as > > > > >> >> > > > it > > > > >> >> > > > > > can > > > > >> >> > > > > > > > > allow > > > > >> >> > > > > > > > > >> > > > >> replacing the > > > > >> >> > > > > > > > > >> > > > >> >> > tokens > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out > of > > > max > > > > >> life > > > > >> >> > > time, > > > > >> >> > > > > and > > > > >> >> > > > > > > > > clients > > > > >> >> > > > > > > > > >> can > > > > >> >> > > > > > > > > >> > > > >> download > > > > >> >> > > > > > > > > >> > > > >> >> > new > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> tokens > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It > > > > >> shouldn't be > > > > >> >> > a > > > > >> >> > > > big > > > > >> >> > > > > > load > > > > >> >> > > > > > > > on > > > > >> >> > > > > > > > > >> > > zookeeper > > > > >> >> > > > > > > > > >> > > > >> as a > > > > >> >> > > > > > > > > >> > > > >> >> > client > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> will > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new > > > token > > > > >> once in > > > > >> >> > > > > several > > > > >> >> > > > > > > > days. > > > > >> >> > > > > > > > > >> In this > > > > >> >> > > > > > > > > >> > > > >> approach > > > > >> >> > > > > > > > > >> > > > >> >> > you > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> don't > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite > > lifetime > > > > on > > > > >> the > > > > >> >> > > token > > > > >> >> > > > > even > > > > >> >> > > > > > > for > > > > >> >> > > > > > > > > >> long > > > > >> >> > > > > > > > > >> > > > running > > > > >> >> > > > > > > > > >> > > > >> >> > clients. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token > > password > > > > are > > > > >> >> > > generated > > > > >> >> > > > > > using > > > > >> >> > > > > > > a > > > > >> >> > > > > > > > > >> master > > > > >> >> > > > > > > > > >> > > key. > > > > >> >> > > > > > > > > >> > > > >> The > > > > >> >> > > > > > > > > >> > > > >> >> > master > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> key > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > should also be > > > > >> periodically > > > > >> >> > > changed. > > > > >> >> > > > > In > > > > >> >> > > > > > > > > Hadoop, > > > > >> >> > > > > > > > > >> the > > > > >> >> > > > > > > > > >> > > > >> default > > > > >> >> > > > > > > > > >> > > > >> >> > renewal > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.? > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will > > require > > > > >> brokers > > > > >> >> > > > > > maintaining > > > > >> >> > > > > > > a > > > > >> >> > > > > > > > > >> list of X > > > > >> >> > > > > > > > > >> > > > most > > > > >> >> > > > > > > > > >> > > > >> >> > recent > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> master > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will > > > have > > > > >> to be > > > > >> >> > > > > persisted > > > > >> >> > > > > > > > > >> somewhere, as > > > > >> >> > > > > > > > > >> > > > if a > > > > >> >> > > > > > > > > >> > > > >> >> > broker > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> goes > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to > > get > > > > >> that list > > > > >> >> > > > again > > > > >> >> > > > > > and > > > > >> >> > > > > > > > > >> storing > > > > >> >> > > > > > > > > >> > > > master > > > > >> >> > > > > > > > > >> > > > >> keys > > > > >> >> > > > > > > > > >> > > > >> >> > on ZK > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> is > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea. > > > > However, > > > > >> if a > > > > >> >> > > > broker > > > > >> >> > > > > > goes > > > > >> >> > > > > > > > > down > > > > >> >> > > > > > > > > >> then > > > > >> >> > > > > > > > > >> > > we > > > > >> >> > > > > > > > > >> > > > >> have > > > > >> >> > > > > > > > > >> > > > >> >> > much > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> bigger > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with > and > > > > >> client can > > > > >> >> > > > always > > > > >> >> > > > > > > > > >> > > re-authenticate > > > > >> >> > > > > > > > > >> > > > is > > > > >> >> > > > > > > > > >> > > > >> such > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> events. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to > > take a > > > > >> look at > > > > >> >> > > other > > > > >> >> > > > > > > > > >> alternatives > > > > >> >> > > > > > > > > >> > > this > > > > >> >> > > > > > > > > >> > > > >> list has > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > suggested? > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a > > thorough > > > > >> proposal, > > > > >> >> > > > great > > > > >> >> > > > > > > work!" > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, > 2016, > > > at > > > > >> 10:28 > > > > >> >> > PM, > > > > >> >> > > > Gwen > > > > >> >> > > > > > > > Shapira > > > > >> >> > > > > > > > > >> wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to > me. > > > > >> Thanks! > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, > > 2016 > > > at > > > > >> 9:25 > > > > >> >> > PM, > > > > >> >> > > > > > Harsha < > > > > >> >> > > > > > > > > >> > > > ka...@harsha.io > > > > >> >> > > > > > > > > >> > > > >> > > > > > >> >> > > > > > > > > >> > > > >> >> > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't > need > > > any > > > > >> release > > > > >> >> > > > > vehicle > > > > >> >> > > > > > > but > > > > >> >> > > > > > > > > >> still the > > > > >> >> > > > > > > > > >> > > > >> work can > > > > >> >> > > > > > > > > >> > > > >> >> > move > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > forward. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is > > > > >> interested in > > > > >> >> > the > > > > >> >> > > > KIP > > > > >> >> > > > > > > > please > > > > >> >> > > > > > > > > >> do the > > > > >> >> > > > > > > > > >> > > > >> review and > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> provide > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > the > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, > > > 2016, > > > > >> at > > > > >> >> > 04:59 > > > > >> >> > > > PM, > > > > >> >> > > > > > > Ismael > > > > >> >> > > > > > > > > >> Juma > > > > >> >> > > > > > > > > >> > > > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that > it > > > > >> would be > > > > >> >> > good > > > > >> >> > > > to > > > > >> >> > > > > > have > > > > >> >> > > > > > > > > more > > > > >> >> > > > > > > > > >> time > > > > >> >> > > > > > > > > >> > > to > > > > >> >> > > > > > > > > >> > > > >> review > > > > >> >> > > > > > > > > >> > > > >> >> > and > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > discuss > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar > 8, > > > 2016 > > > > >> at > > > > >> >> > 12:55 > > > > >> >> > > > AM, > > > > >> >> > > > > > Gwen > > > > >> >> > > > > > > > > >> Shapira < > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> g...@confluent.io> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since > KIP-48 > > > > >> depends on > > > > >> >> > > > KIP-43, > > > > >> >> > > > > > > which > > > > >> >> > > > > > > > > is > > > > >> >> > > > > > > > > >> > > > already a > > > > >> >> > > > > > > > > >> > > > >> bit > > > > >> >> > > > > > > > > >> > > > >> >> > of a > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> risk > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > for > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next > > > release > > > > - > > > > >> any > > > > >> >> > > chance > > > > >> >> > > > > we > > > > >> >> > > > > > > can > > > > >> >> > > > > > > > > >> delay > > > > >> >> > > > > > > > > >> > > > >> delegation > > > > >> >> > > > > > > > > >> > > > >> >> > tokens > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1? > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the > > > > community > > > > >> >> > working > > > > >> >> > > > on a > > > > >> >> > > > > > > > release > > > > >> >> > > > > > > > > >> every > > > > >> >> > > > > > > > > >> > > 3 > > > > >> >> > > > > > > > > >> > > > >> month, > > > > >> >> > > > > > > > > >> > > > >> >> > this > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> is not > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a huge > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb > > 26, > > > > >> 2016 at > > > > >> >> > > 5:11 > > > > >> >> > > > > PM, > > > > >> >> > > > > > > > Ashish > > > > >> >> > > > > > > > > >> Singh > > > > >> >> > > > > > > > > >> > > < > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > asi...@cloudera.com > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks > > again > > > > for > > > > >> the > > > > >> >> > > > awesome > > > > >> >> > > > > > > write > > > > >> >> > > > > > > > > up. > > > > >> >> > > > > > > > > >> > > > Following > > > > >> >> > > > > > > > > >> > > > >> our > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> discussion > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from the > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I > > think > > > > it > > > > >> will > > > > >> >> > be > > > > >> >> > > > > easier > > > > >> >> > > > > > > to > > > > >> >> > > > > > > > > >> compare > > > > >> >> > > > > > > > > >> > > > >> various > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> alternatives > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > if they > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed > > > > together. > > > > >> I am > > > > >> >> > > > stating > > > > >> >> > > > > > > > below a > > > > >> >> > > > > > > > > >> few > > > > >> >> > > > > > > > > >> > > > >> >> > alternatives along > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > with > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a the > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current > > > > proposal. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current > > > > >> proposal) > > > > >> >> > Store > > > > >> >> > > > > > > Delegation > > > > >> >> > > > > > > > > >> Token, > > > > >> >> > > > > > > > > >> > > DT, > > > > >> >> > > > > > > > > >> > > > >> on ZK. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. > > Client > > > > >> >> > > authenticates > > > > >> >> > > > > > with a > > > > >> >> > > > > > > > > >> broker. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. > Once > > a > > > > >> client is > > > > >> >> > > > > > > > authenticated, > > > > >> >> > > > > > > > > >> it > > > > >> >> > > > > > > > > >> > > will > > > > >> >> > > > > > > > > >> > > > >> make a > > > > >> >> > > > > > > > > >> > > > >> >> > broker > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> side > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > issue > a > > > > >> delegation > > > > >> >> > > > token. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. The > > > > broker > > > > >> >> > > generates > > > > >> >> > > > a > > > > >> >> > > > > > > shared > > > > >> >> > > > > > > > > >> secret > > > > >> >> > > > > > > > > >> > > > based > > > > >> >> > > > > > > > > >> > > > >> on > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> Password/Secret > > > > >> >> > shared > > > > >> >> > > > > > between > > > > >> >> > > > > > > > all > > > > >> >> > > > > > > > > >> > > brokers, > > > > >> >> > > > > > > > > >> > > > >> >> > randomly > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > generated > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId). > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 4. > > Broker > > > > >> stores > > > > >> >> > this > > > > >> >> > > > > token > > > > >> >> > > > > > in > > > > >> >> > > > > > > > its > > > > >> >> > > > > > > > > >> in > > > > >> >> > > > > > > > > >> > > > memory > > > > >> >> > > > > > > > > >> > > > >> cache. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> Broker > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > also stores > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > the > > > > >> DelegationToken > > > > >> >> > > > > without > > > > >> >> > > > > > > the > > > > >> >> > > > > > > > > >> hmac in > > > > >> >> > > > > > > > > >> > > the > > > > >> >> > > > > > > > > >> > > > >> >> > zookeeper. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 5. All > > > > >> brokers will > > > > >> >> > > > have a > > > > >> >> > > > > > > cache > > > > >> >> > > > > > > > > >> backed > > > > >> >> > > > > > > > > >> > > by > > > > >> >> > > > > > > > > >> > > > >> >> > zookeeper so > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> they > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > will all > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > get > > > notified > > > > >> >> > whenever > > > > >> >> > > a > > > > >> >> > > > > new > > > > >> >> > > > > > > > token > > > > >> >> > > > > > > > > is > > > > >> >> > > > > > > > > >> > > > >> generated and > > > > >> >> > > > > > > > > >> > > > >> >> > they > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> will > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > update > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > local > > > cache > > > > >> whenever > > > > >> >> > > > token > > > > >> >> > > > > > > state > > > > >> >> > > > > > > > > >> changes. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 6. > > Broker > > > > >> returns > > > > >> >> > the > > > > >> >> > > > > token > > > > >> >> > > > > > to > > > > >> >> > > > > > > > > >> Client. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable > > > issues > > > > >> and > > > > >> >> > fixes > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. > > > Probable > > > > >> race > > > > >> >> > > > > condition, > > > > >> >> > > > > > > > client > > > > >> >> > > > > > > > > >> tries > > > > >> >> > > > > > > > > >> > > to > > > > >> >> > > > > > > > > >> > > > >> >> > authenticate > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> with > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a broker > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > that > is > > > yet > > > > >> to be > > > > >> >> > > > updated > > > > >> >> > > > > > with > > > > >> >> > > > > > > > the > > > > >> >> > > > > > > > > >> newly > > > > >> >> > > > > > > > > >> > > > >> generated > > > > >> >> > > > > > > > > >> > > > >> >> > DT. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> This > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > can > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > dealt > > with > > > > >> making > > > > >> >> > > > > dtRequest > > > > >> >> > > > > > > > block > > > > >> >> > > > > > > > > >> until > > > > >> >> > > > > > > > > >> > > all > > > > >> >> > > > > > > > > >> > > > >> >> > brokers have > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > updated > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > cache. > > Zk > > > > >> barrier or > > > > >> >> > > > > similar > > > > >> >> > > > > > > > > >> mechanism > > > > >> >> > > > > > > > > >> > > can > > > > >> >> > > > > > > > > >> > > > be > > > > >> >> > > > > > > > > >> > > > >> used. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> However, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > all such > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > mechanisms > > > > >> will > > > > >> >> > > increase > > > > >> >> > > > > > > > > complexity. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. > > Using a > > > > >> static > > > > >> >> > > secret > > > > >> >> > > > > key > > > > >> >> > > > > > > > from > > > > >> >> > > > > > > > > >> config > > > > >> >> > > > > > > > > >> > > > >> file. Will > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> require > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > yet > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > config > > and > > > > >> uses a > > > > >> >> > > static > > > > >> >> > > > > > > secret > > > > >> >> > > > > > > > > >> key. It > > > > >> >> > > > > > > > > >> > > is > > > > >> >> > > > > > > > > >> > > > >> advised > > > > >> >> > > > > > > > > >> > > > >> >> > to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> rotate > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > secret > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > periodically. > > > > >> This > > > > >> >> > can > > > > >> >> > > > be > > > > >> >> > > > > > > > avoided > > > > >> >> > > > > > > > > >> with > > > > >> >> > > > > > > > > >> > > > >> controller > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> generating > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey > and > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > passing > > to > > > > >> brokers > > > > >> >> > > > > > > periodically. > > > > >> >> > > > > > > > > >> However, > > > > >> >> > > > > > > > > >> > > > >> this will > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> require > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > maintain > > > > >> certain > > > > >> >> > > counts > > > > >> >> > > > of > > > > >> >> > > > > > > > > >> secretKeys. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > (Alternative > > > 1) > > > > >> Have > > > > >> >> > > > > controller > > > > >> >> > > > > > > > > >> generate > > > > >> >> > > > > > > > > >> > > > >> delegation > > > > >> >> > > > > > > > > >> > > > >> >> > token. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. > > Client > > > > >> >> > > authenticates > > > > >> >> > > > > > with a > > > > >> >> > > > > > > > > >> broker. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. > Once > > a > > > > >> client is > > > > >> >> > > > > > > > authenticated, > > > > >> >> > > > > > > > > >> it > > > > >> >> > > > > > > > > >> > > will > > > > >> >> > > > > > > > > >> > > > >> make a > > > > >> >> > > > > > > > > >> > > > >> >> > broker > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> side > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > issue > a > > > > >> delegation > > > > >> >> > > > token. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. > > Broker > > > > >> forwards > > > > >> >> > the > > > > >> >> > > > > > request > > > > >> >> > > > > > > > to > > > > >> >> > > > > > > > > >> > > > controller. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 4. > > > > Controller > > > > >> >> > > generates > > > > >> >> > > > a > > > > >> >> > > > > DT > > > > >> >> > > > > > > and > > > > >> >> > > > > > > > > >> > > broadcasts > > > > >> >> > > > > > > > > >> > > > >> to all > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> brokers. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 5. > > Broker > > > > >> stores > > > > >> >> > this > > > > >> >> > > > > token > > > > >> >> > > > > > in > > > > >> >> > > > > > > > its > > > > >> >> > > > > > > > > >> memory > > > > >> >> > > > > > > > > >> > > > >> cache. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 6. > > > > Controller > > > > >> >> > responds > > > > >> >> > > > to > > > > >> >> > > > > > > > broker’s > > > > >> >> > > > > > > > > >> DT > > > > >> >> > > > > > > > > >> > > req. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 7. > > Broker > > > > >> returns > > > > >> >> > the > > > > >> >> > > > > token > > > > >> >> > > > > > to > > > > >> >> > > > > > > > > >> Client. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable > > > issues > > > > >> and > > > > >> >> > fixes > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. We > > will > > > > >> have to > > > > >> >> > add > > > > >> >> > > > new > > > > >> >> > > > > > > APIs > > > > >> >> > > > > > > > to > > > > >> >> > > > > > > > > >> > > support > > > > >> >> > > > > > > > > >> > > > >> >> > controller > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> pushing > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > brokers > > on > > > > >> top of > > > > >> >> > the > > > > >> >> > > > > > minimal > > > > >> >> > > > > > > > APIs > > > > >> >> > > > > > > > > >> that > > > > >> >> > > > > > > > > >> > > are > > > > >> >> > > > > > > > > >> > > > >> >> > currently > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > proposed. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. We > > will > > > > >> also have > > > > >> >> > > to > > > > >> >> > > > > add > > > > >> >> > > > > > > APIs > > > > >> >> > > > > > > > > to > > > > >> >> > > > > > > > > >> > > support > > > > >> >> > > > > > > > > >> > > > >> the > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> bootstrapping > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > case, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > when a > > new > > > > >> broker > > > > >> >> > > comes > > > > >> >> > > > up > > > > >> >> > > > > > it > > > > >> >> > > > > > > > will > > > > >> >> > > > > > > > > >> have > > > > >> >> > > > > > > > > >> > > to > > > > >> >> > > > > > > > > >> > > > >> get all > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> delegation > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > the > > > > >> controller. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. In > > > > >> catastrophic > > > > >> >> > > > > failures > > > > >> >> > > > > > > > where > > > > >> >> > > > > > > > > >> all > > > > >> >> > > > > > > > > >> > > > brokers > > > > >> >> > > > > > > > > >> > > > >> go > > > > >> >> > > > > > > > > >> > > > >> >> > down, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> the > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > be > lost > > > even > > > > >> if > > > > >> >> > > servers > > > > >> >> > > > > are > > > > >> >> > > > > > > > > >> restarted as > > > > >> >> > > > > > > > > >> > > > >> tokens > > > > >> >> > > > > > > > > >> > > > >> >> > are not > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > If > this > > > > >> happens, > > > > >> >> > then > > > > >> >> > > > > there > > > > >> >> > > > > > > are > > > > >> >> > > > > > > > > more > > > > >> >> > > > > > > > > >> > > > important > > > > >> >> > > > > > > > > >> > > > >> >> > things to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > maybe > it > > > is > > > > >> better > > > > >> >> > to > > > > >> >> > > > > > > > > >> re-authenticate. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > (Alternative > > > 2) > > > > >> Do not > > > > >> >> > > > > > distribute > > > > >> >> > > > > > > > DT > > > > >> >> > > > > > > > > to > > > > >> >> > > > > > > > > >> > > other > > > > >> >> > > > > > > > > >> > > > >> brokers > > > > >> >> > > > > > > > > >> > > > >> >> > at > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> all. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. > > Client > > > > >> >> > > authenticates > > > > >> >> > > > > > with a > > > > >> >> > > > > > > > > >> broker. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. > Once > > a > > > > >> client is > > > > >> >> > > > > > > > authenticated, > > > > >> >> > > > > > > > > >> it > > > > >> >> > > > > > > > > >> > > will > > > > >> >> > > > > > > > > >> > > > >> make a > > > > >> >> > > > > > > > > >> > > > >> >> > broker > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> side > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > issue > a > > > > >> delegation > > > > >> >> > > > token. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. The > > > > broker > > > > >> >> > > generates > > > > >> >> > > > DT > > > > >> >> > > > > > of > > > > >> >> > > > > > > > > form, > > > > >> >> > > > > > > > > >> > > [hmac + > > > > >> >> > > > > > > > > >> > > > >> (owner, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> renewer, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > maxLifeTime, > > > > >> id, > > > > >> >> > hmac, > > > > >> >> > > > > > > > > >> expirationTime)] > > > > >> >> > > > > > > > > >> > > and > > > > >> >> > > > > > > > > >> > > > >> passes > > > > >> >> > > > > > > > > >> > > > >> >> > back > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> this > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > DT to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > hmac > is > > > > >> generated > > > > >> >> > via > > > > >> >> > > > > > > > > >> {HMAC-SHA256(owner, > > > > >> >> > > > > > > > > >> > > > >> renewer, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> expirationTime) > > > > >> >> > using > > > > >> >> > > > > > > > SecretKey}. > > > > >> >> > > > > > > > > >> Note > > > > >> >> > > > > > > > > >> > > that > > > > >> >> > > > > > > > > >> > > > >> all > > > > >> >> > > > > > > > > >> > > > >> >> > brokers > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> have > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 4. > > Client > > > > >> then goes > > > > >> >> > to > > > > >> >> > > > any > > > > >> >> > > > > > > > broker > > > > >> >> > > > > > > > > >> and to > > > > >> >> > > > > > > > > >> > > > >> >> > authenticate > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> sends > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > the DT. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Broker > > > > >> recalculates > > > > >> >> > > hmac > > > > >> >> > > > > > using > > > > >> >> > > > > > > > > >> (owner, > > > > >> >> > > > > > > > > >> > > > >> renewer, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> expirationTime) info > > > > >> >> > > > from > > > > >> >> > > > > DT > > > > >> >> > > > > > > and > > > > >> >> > > > > > > > > its > > > > >> >> > > > > > > > > >> > > > >> SecretKey. If > > > > >> >> > > > > > > > > >> > > > >> >> > it > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> matches > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > with > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > DT, > > client > > > > is > > > > >> >> > > > > authenticated. > > > > >> >> > > > > > > > Yes, > > > > >> >> > > > > > > > > >> it will > > > > >> >> > > > > > > > > >> > > > do > > > > >> >> > > > > > > > > >> > > > >> other > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> obvious > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > checks of > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > timestamp > > > > >> expiry and > > > > >> >> > > > such. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that > > > > secret > > > > >> key > > > > >> >> > will > > > > >> >> > > > be > > > > >> >> > > > > > > > > generated > > > > >> >> > > > > > > > > >> by > > > > >> >> > > > > > > > > >> > > > >> controller > > > > >> >> > > > > > > > > >> > > > >> >> > and > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> passed > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > periodically. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable > > > issues > > > > >> and > > > > >> >> > fixes > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. How > > to > > > > >> delete a > > > > >> >> > DT? > > > > >> >> > > > > Yes, > > > > >> >> > > > > > > that > > > > >> >> > > > > > > > > is > > > > >> >> > > > > > > > > >> a > > > > >> >> > > > > > > > > >> > > > downside > > > > >> >> > > > > > > > > >> > > > >> >> > here. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> However, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this can > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > be > > handled > > > > >> with > > > > >> >> > > brokers > > > > >> >> > > > > > > > > maintaining > > > > >> >> > > > > > > > > >> a > > > > >> >> > > > > > > > > >> > > > >> blacklist of > > > > >> >> > > > > > > > > >> > > > >> >> > DTs, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> DTs > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from this > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > can be > > > > >> removed after > > > > >> >> > > > > expiry. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. In > > > > >> catastrophic > > > > >> >> > > > > failures > > > > >> >> > > > > > > > where > > > > >> >> > > > > > > > > >> all > > > > >> >> > > > > > > > > >> > > > brokers > > > > >> >> > > > > > > > > >> > > > >> go > > > > >> >> > > > > > > > > >> > > > >> >> > down, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> the > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > be > lost > > > even > > > > >> if > > > > >> >> > > servers > > > > >> >> > > > > are > > > > >> >> > > > > > > > > >> restarted as > > > > >> >> > > > > > > > > >> > > > >> tokens > > > > >> >> > > > > > > > > >> > > > >> >> > are not > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > If > this > > > > >> happens, > > > > >> >> > then > > > > >> >> > > > > there > > > > >> >> > > > > > > are > > > > >> >> > > > > > > > > more > > > > >> >> > > > > > > > > >> > > > important > > > > >> >> > > > > > > > > >> > > > >> >> > things to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > maybe > it > > > is > > > > >> better > > > > >> >> > to > > > > >> >> > > > > > > > > >> re-authenticate. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, > Feb > > > 26, > > > > >> 2016 at > > > > >> >> > > > 1:58 > > > > >> >> > > > > > PM, > > > > >> >> > > > > > > > > Parth > > > > >> >> > > > > > > > > >> > > > >> Brahmbhatt < > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > pbrahmbh...@hortonworks.com> > > > > >> >> > > > > > > > wrote: > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have > > filed > > > > >> KIP-48 so > > > > >> >> > > we > > > > >> >> > > > > can > > > > >> >> > > > > > > > offer > > > > >> >> > > > > > > > > >> hadoop > > > > >> >> > > > > > > > > >> > > > like > > > > >> >> > > > > > > > > >> > > > >> >> > delegation > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. > You > > > can > > > > >> review > > > > >> >> > > the > > > > >> >> > > > > > design > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > . > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP > > > > >> depends on > > > > >> >> > > KIP-43 > > > > >> >> > > > > and > > > > >> >> > > > > > > we > > > > >> >> > > > > > > > > >> have also > > > > >> >> > > > > > > > > >> > > > >> >> > discussed an > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed > > > > design > > > > >> here< > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800 > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >. > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > -- > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > -- > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Regards, > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Ashish > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> > > > > > >> >> > > > > > > > > >> > > > >> > > > > >> >> > > > > > > > > >> > > > > > > > >> >> > > > > > > > > >> > > > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > -- > > > > >> >> > > > > > Regards, > > > > >> >> > > > > > > > > > >> >> > > > > > Rajini > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > -- > > > > >> >> > > > Regards, > > > > >> >> > > > > > > > >> >> > > > Rajini > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> -- > > > > >> >> Liquan Pei > > > > >> >> Software Engineer, Confluent Inc > > > > >> > > > > > > > > > > > > > > > > > > > >
