Harsha, Another question.
9. How would the delegation token be configured in the client? The standard way is to do this through JAAS. However, we will need to think through if this is convenient in a shared environment. For example, when a new task is added to a Storm worker node, do we need to dynamically add a new section in the JAAS file? It may be more convenient if we can pass in the token through the config directly w/o going through JAAS. Are you or Parth still actively working on this KIP? Thanks, Jun On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <j...@confluent.io> wrote: > Just to add on that list. > > 2. It would be good to document the format of the data stored in ZK. > 7. Earlier, there was a discussion on whether the tokens should be > propagated through ZK like config/acl/quota, or through the controller. > Currently, the controller is only designed for propagating topic metadata, > but not other data. > 8. Should we use SCRAM to send the token instead of DIGEST-MD5 since it's > deprecated? > > Also, the images in the wiki seem broken. > > Thanks, > > Jun > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <g...@confluent.io> wrote: > >> From what I can see, remaining questions are: >> >> 1. Who / how are tokens renewed? By original requester only? or using >> Kerberos auth only? >> 2. Are tokens stored on each broker or in ZK? >> 3. How are tokens invalidated / expired? >> 4. Which encryption algorithm is used? >> 5. What is the impersonation proposal (it wasn't in the KIP but was >> discussed in this thread)? >> 6. Do we need new ACLs, if so - for what actions? >> >> Gwen >> >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io> wrote: >> > Jun & Ismael, >> > Unfortunately I couldn't attend the KIP meeting >> > when delegation tokens discussed. Appreciate if >> > you can update the thread if you have any >> > further questions. >> > Thanks, >> > Harsha >> > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote: >> >> It seems that the links to images in the KIP are broken. >> >> >> >> Liquan >> >> >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt < >> >> brahmbhatt.pa...@gmail.com> wrote: >> >> >> >> > 110. What does getDelegationTokenAs mean? >> >> > In the current proposal we only allow a user to get delegation token >> for >> >> > the identity that it authenticated as using another mechanism, i.e. >> A user >> >> > that authenticate using a keytab for principal us...@example.com >> will get >> >> > delegation tokens for that user only. In future I think we will have >> to >> >> > extend support such that we allow some set of users ( >> >> > kafka-rest-u...@example.com, storm-nim...@example.com) to acquire >> >> > delegation tokens on behalf of other users whose identity they have >> >> > verified independently. Kafka brokers will have ACLs to control >> which >> >> > users are allowed to impersonate other users and get tokens on >> behalf of >> >> > them. Overall Impersonation is a whole different problem in my >> opinion and >> >> > I think we can tackle it in separate KIP. >> >> > >> >> > 111. What's the typical rate of getting and renewing delegation >> tokens? >> >> > Typically this should be very very low, 1 request per minute is a >> >> > relatively high estimate. However it depends on the token >> expiration. I am >> >> > less worried about the extra load it puts on controller vs the added >> >> > complexity and the value it offers. >> >> > >> >> > Thanks >> >> > Parth >> >> > >> >> > >> >> > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <ism...@juma.me.uk> >> wrote: >> >> > >> >> > > Thanks Rajini. It would probably require a separate KIP as it will >> >> > > introduce user visible changes. We could also update KIP-48 to >> have this >> >> > > information, but it seems cleaner to do it separately. We can >> discuss >> >> > that >> >> > > in the KIP call today. >> >> > > >> >> > > Ismael >> >> > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram < >> >> > > rajinisiva...@googlemail.com> wrote: >> >> > > >> >> > > > Ismael, >> >> > > > >> >> > > > I have created a JIRA ( >> >> > https://issues.apache.org/jira/browse/KAFKA-3751) >> >> > > > for adding SCRAM as a SASL mechanism. Would that need another >> KIP? If >> >> > > > KIP-48 will use this mechanism, can this just be a JIRA that gets >> >> > > reviewed >> >> > > > when the PR is ready? >> >> > > > >> >> > > > Thank you, >> >> > > > >> >> > > > Rajini >> >> > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <ism...@juma.me.uk> >> >> > wrote: >> >> > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good candidate. >> >> > > > > >> >> > > > > Gwen had independently mentioned this as a SASL mechanism that >> might >> >> > be >> >> > > > > useful for Kafka and I have been meaning to check it in more >> detail. >> >> > > Good >> >> > > > > to know that you are willing to contribute an implementation. >> Maybe >> >> > we >> >> > > > > should file a separate JIRA for this? >> >> > > > > >> >> > > > > Ismael >> >> > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram < >> >> > > > > rajinisiva...@googlemail.com> wrote: >> >> > > > > >> >> > > > > > SCRAM (Salted Challenge Response Authentication Mechanism) >> is a >> >> > > better >> >> > > > > > mechanism than Digest-MD5. Java doesn't come with a built-in >> SCRAM >> >> > > > > > SaslServer or SaslClient, but I will be happy to add support >> in >> >> > Kafka >> >> > > > > since >> >> > > > > > it would be a useful mechanism to support anyway. >> >> > > > > > https://tools.ietf.org/html/rfc7677 describes the protocol >> for >> >> > > > > > SCRAM-SHA-256. >> >> > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <j...@confluent.io> >> wrote: >> >> > > > > > >> >> > > > > > > Parth, >> >> > > > > > > >> >> > > > > > > Thanks for the explanation. A couple of more questions. >> >> > > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs mean? >> >> > > > > > > >> >> > > > > > > 111. What's the typical rate of getting and renewing >> delegation >> >> > > > tokens? >> >> > > > > > > That may have an impact on whether they should be directed >> to the >> >> > > > > > > controller. >> >> > > > > > > >> >> > > > > > > Jun >> >> > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt < >> >> > > > > > > brahmbhatt.pa...@gmail.com> wrote: >> >> > > > > > > >> >> > > > > > > > Hi Jun, >> >> > > > > > > > >> >> > > > > > > > Thanks for reviewing. >> >> > > > > > > > >> >> > > > > > > > * We could add a Cluster action to add acls on who can >> request >> >> > > > > > delegation >> >> > > > > > > > tokens. I don't see the use case for that yet but down >> the line >> >> > > > when >> >> > > > > we >> >> > > > > > > > start supporting getDelegationTokenAs it will be >> necessary. >> >> > > > > > > > * Yes we recommend tokens to be only used/distributed >> over >> >> > secure >> >> > > > > > > channels. >> >> > > > > > > > * Depending on what design we end up choosing >> Invalidation will >> >> > > be >> >> > > > > > > > responsibility of every broker or controller. >> >> > > > > > > > * I am not sure if I documented somewhere that >> invalidation >> >> > will >> >> > > > > > directly >> >> > > > > > > > go through zookeeper but that is not the intent. >> Invalidation >> >> > > will >> >> > > > > > either >> >> > > > > > > > be request based or due to expiration. No direct >> zookeeper >> >> > > > > interaction >> >> > > > > > > from >> >> > > > > > > > any client. >> >> > > > > > > > * "Broker also stores the DelegationToken without the >> hmac in >> >> > the >> >> > > > > > > > zookeeper." : Sorry about the confusion. The sole >> purpose of >> >> > > > > zookeeper >> >> > > > > > in >> >> > > > > > > > this design is as distribution channel for tokens >> between all >> >> > > > brokers >> >> > > > > > > and a >> >> > > > > > > > layer that ensures only tokens that were generated by >> making a >> >> > > > > request >> >> > > > > > > to a >> >> > > > > > > > broker will be accepted (more on this in second >> paragraph). The >> >> > > > token >> >> > > > > > > > consists of few elements (owner, renewer, uuid , >> expiration, >> >> > > hmac) >> >> > > > , >> >> > > > > > one >> >> > > > > > > of >> >> > > > > > > > which is the finally generated hmac but hmac it self is >> >> > derivable >> >> > > > if >> >> > > > > > you >> >> > > > > > > > have all the other elements of the token + secret key to >> >> > generate >> >> > > > > hmac. >> >> > > > > > > > Given zookeeper does not provide SSL support we do not >> want the >> >> > > > > entire >> >> > > > > > > > token to be wire transferred to zookeeper as that will >> be an >> >> > > > insecure >> >> > > > > > > wire >> >> > > > > > > > transfer. Instead we only store all the other elements >> of a >> >> > > > > delegation >> >> > > > > > > > tokens. Brokers can read these elements and because they >> also >> >> > > have >> >> > > > > > access >> >> > > > > > > > to secret key they will be able to generate hmac on >> their end. >> >> > > > > > > > >> >> > > > > > > > One of the alternative proposed is to avoid zookeeper >> >> > > altogether. A >> >> > > > > > > Client >> >> > > > > > > > will call broker with required information (owner, >> renwer, >> >> > > > > expiration) >> >> > > > > > > and >> >> > > > > > > > get back (signed hmac, uuid). Broker won't store this in >> >> > > zookeeper. >> >> > > > > > From >> >> > > > > > > > this point a client can contact any broker with all the >> >> > > delegation >> >> > > > > > token >> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the borker >> will >> >> > > > > regenerate >> >> > > > > > > the >> >> > > > > > > > hmac and as long as it matches with hmac presented by >> client , >> >> > > > broker >> >> > > > > > > will >> >> > > > > > > > allow the request to authenticate. Only problem with >> this >> >> > > approach >> >> > > > > is >> >> > > > > > if >> >> > > > > > > > the secret key is compromised any client can now generate >> >> > random >> >> > > > > tokens >> >> > > > > > > and >> >> > > > > > > > they will still be able to authenticate as any user they >> like. >> >> > > with >> >> > > > > > > > zookeeper we guarantee that only tokens acquired via a >> broker >> >> > > > (using >> >> > > > > > some >> >> > > > > > > > auth scheme other than delegation token) will be >> accepted. We >> >> > > need >> >> > > > to >> >> > > > > > > > discuss which proposal makes more sense and we can go >> over it >> >> > in >> >> > > > > > > tomorrow's >> >> > > > > > > > meeting. >> >> > > > > > > > >> >> > > > > > > > Also, can you forward the invite to me? >> >> > > > > > > > >> >> > > > > > > > Thanks >> >> > > > > > > > Parth >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao < >> j...@confluent.io> >> >> > > > wrote: >> >> > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few comments. >> >> > > > > > > > > >> >> > > > > > > > > 100. This potentially can be useful for Kafka Connect >> and >> >> > Kafka >> >> > > > > rest >> >> > > > > > > > proxy >> >> > > > > > > > > where a worker agent will need to run a task on behalf >> of a >> >> > > > client. >> >> > > > > > We >> >> > > > > > > > will >> >> > > > > > > > > likely need to change how those services use Kafka >> clients >> >> > > > > > > > > (producer/consumer). Instead of a shared client per >> worker, >> >> > we >> >> > > > will >> >> > > > > > > need >> >> > > > > > > > a >> >> > > > > > > > > client per user task since the authentication happens >> at the >> >> > > > > > connection >> >> > > > > > > > > level. For Kafka Connect, the renewer will be the >> workers. >> >> > So, >> >> > > we >> >> > > > > > > > probably >> >> > > > > > > > > need to allow multiple renewers. For Kafka rest proxy, >> the >> >> > > > renewer >> >> > > > > > can >> >> > > > > > > > > probably just be the creator of the token. >> >> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who can request delegation >> tokens? >> >> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend people to send delegation tokens >> in an >> >> > > > > encrypted >> >> > > > > > > > > channel? >> >> > > > > > > > > >> >> > > > > > > > > 103. Who is responsible for expiring tokens, every >> broker? >> >> > > > > > > > > >> >> > > > > > > > > 104. For invalidating tokens, would it be better to do >> it in >> >> > a >> >> > > > > > request >> >> > > > > > > > > instead of going to ZK directly? >> >> > > > > > > > > >> >> > > > > > > > > 105. The terminology of client in the wiki sometimes >> refers >> >> > to >> >> > > > the >> >> > > > > > end >> >> > > > > > > > > client and some other times refers to the client using >> the >> >> > > > > delegation >> >> > > > > > > > > tokens. It would be useful to distinguish between the >> two. >> >> > > > > > > > > >> >> > > > > > > > > 106. Could you explain the sentence "Broker also >> stores the >> >> > > > > > > > DelegationToken >> >> > > > > > > > > without the hmac in the zookeeper." a bit more? I >> thought the >> >> > > > > > > delegation >> >> > > > > > > > > token is the hmac. >> >> > > > > > > > > >> >> > > > > > > > > Thanks, >> >> > > > > > > > > >> >> > > > > > > > > Jun >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao < >> j...@confluent.io> >> >> > > > wrote: >> >> > > > > > > > > >> >> > > > > > > > > > Hi, Harsha, >> >> > > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting invite. We can discuss >> this in >> >> > > the >> >> > > > > > > meeting >> >> > > > > > > > > > tomorrow. >> >> > > > > > > > > > >> >> > > > > > > > > > Thanks, >> >> > > > > > > > > > >> >> > > > > > > > > > Jun >> >> > > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha < >> ka...@harsha.io> >> >> > > > wrote: >> >> > > > > > > > > > >> >> > > > > > > > > >> Hi All, >> >> > > > > > > > > >> Can we have a KIP meeting around this. >> The KIP >> >> > is >> >> > > > up >> >> > > > > > for >> >> > > > > > > > > >> sometime and if there are any questions >> lets >> >> > > > quickly >> >> > > > > > hash >> >> > > > > > > > out >> >> > > > > > > > > >> details. >> >> > > > > > > > > >> >> >> > > > > > > > > >> Thanks, >> >> > > > > > > > > >> Harsha >> >> > > > > > > > > >> >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt >> wrote: >> >> > > > > > > > > >> > That is what the hadoop echo system uses so no >> good >> >> > reason >> >> > > > > > really. >> >> > > > > > > > We >> >> > > > > > > > > >> > could >> >> > > > > > > > > >> > change it to whatever is the newest recommended >> standard >> >> > > is. >> >> > > > > > > > > >> > >> >> > > > > > > > > >> > Thanks >> >> > > > > > > > > >> > Parth >> >> > > > > > > > > >> > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma < >> >> > > > > ism...@juma.me.uk >> >> > > > > > > >> >> > > > > > > > > wrote: >> >> > > > > > > > > >> > >> >> > > > > > > > > >> > > Hi Parth, >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only started reviewing >> this and >> >> > > may >> >> > > > > have >> >> > > > > > > > > >> additional >> >> > > > > > > > > >> > > questions later. The immediate question that >> came to >> >> > > mind >> >> > > > is >> >> > > > > > our >> >> > > > > > > > > >> choice of >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as >> OBSOLETE in >> >> > the >> >> > > > IANA >> >> > > > > > > > > Registry >> >> > > > > > > > > >> of >> >> > > > > > > > > >> > > SASL mechanisms and the original RFC (2831) has >> been >> >> > > moved >> >> > > > > to >> >> > > > > > > > > Historic >> >> > > > > > > > > >> > > status: >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331 >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> >> > > > > > >> >> > > >> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> > > What is the reasoning behind that choice? >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> > > Thanks, >> >> > > > > > > > > >> > > Ismael >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira < >> >> > > > > > > g...@confluent.io >> >> > > > > > > > > >> >> > > > > > > > > >> wrote: >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> > > > Also comments inline :) >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > > * I want to emphasize that even though >> delegation >> >> > > > tokens >> >> > > > > > > are a >> >> > > > > > > > > >> Hadoop >> >> > > > > > > > > >> > > > > innovation, I feel very strongly about not >> adding >> >> > > > > > dependency >> >> > > > > > > > on >> >> > > > > > > > > >> Hadoop >> >> > > > > > > > > >> > > > > when implementing delegation tokens for >> Kafka. The >> >> > > KIP >> >> > > > > > > doesn't >> >> > > > > > > > > >> imply >> >> > > > > > > > > >> > > > > such dependency, but if you can clarify... >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.* >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so no one will >> read >> >> > the >> >> > > > KIP >> >> > > > > > and >> >> > > > > > > > > panic >> >> > > > > > > > > >> > > > three weeks before the next release... >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > > * Can we get delegation token at any time >> after >> >> > > > > > > > authenticating? >> >> > > > > > > > > >> only >> >> > > > > > > > > >> > > > > immediately after? >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > *As long as you are authenticated you can >> get >> >> > > > delegation >> >> > > > > > > > tokens. >> >> > > > > > > > > >> We >> >> > > > > > > > > >> > > need >> >> > > > > > > > > >> > > > to >> >> > > > > > > > > >> > > > > discuss if a client authenticated using >> delegation >> >> > > > > token, >> >> > > > > > > can >> >> > > > > > > > > also >> >> > > > > > > > > >> > > > acquire >> >> > > > > > > > > >> > > > > delegation token again or not. Also there >> is the >> >> > > > > question >> >> > > > > > of >> >> > > > > > > > do >> >> > > > > > > > > we >> >> > > > > > > > > >> > > allow >> >> > > > > > > > > >> > > > > anyone to acquire delegation token or we >> want >> >> > > specific >> >> > > > > > ACLs >> >> > > > > > > (I >> >> > > > > > > > > >> think >> >> > > > > > > > > >> > > its >> >> > > > > > > > > >> > > > an >> >> > > > > > > > > >> > > > > overkill.)* >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an overkill. >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > I think we are debating two options: Either >> require >> >> > > > > Kerberos >> >> > > > > > > > auth >> >> > > > > > > > > >> for >> >> > > > > > > > > >> > > > renewal or require non-owners to renew. >> >> > > > > > > > > >> > > > I *think* the latter is simpler (it basically >> >> > require >> >> > > a >> >> > > > > "job >> >> > > > > > > > > master" >> >> > > > > > > > > >> > > > to take responsibility for the renewal, it >> will have >> >> > > its >> >> > > > > own >> >> > > > > > > > > >> identity >> >> > > > > > > > > >> > > > anyway and I think this is the correct design >> >> > pattern >> >> > > > > > anyway. >> >> > > > > > > > For >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate >> renewals?), >> >> > but >> >> > > > it >> >> > > > > is >> >> > > > > > > > hard >> >> > > > > > > > > to >> >> > > > > > > > > >> > > > debate simplicity without looking at the code >> >> > changes >> >> > > > > > > required. >> >> > > > > > > > If >> >> > > > > > > > > >> you >> >> > > > > > > > > >> > > > have a draft of how the "require Kerberos" >> will look >> >> > > in >> >> > > > > > Kafka >> >> > > > > > > > > code, >> >> > > > > > > > > >> > > > I'll be happy to take a look. >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > > * My understanding is that tokens will >> propagate >> >> > via >> >> > > > ZK >> >> > > > > > but >> >> > > > > > > > > >> without >> >> > > > > > > > > >> > > > > additional changes to UpdateMetadata >> protocol, >> >> > > > correct? >> >> > > > > > > > Clients >> >> > > > > > > > > >> > > > > currently don't retry on SASL auth failure >> (IIRC), >> >> > > but >> >> > > > > > since >> >> > > > > > > > the >> >> > > > > > > > > >> > > > > tokens propagate between brokers asynch, we >> will >> >> > > need >> >> > > > to >> >> > > > > > > > retry a >> >> > > > > > > > > >> bit >> >> > > > > > > > > >> > > > > to avoid clients failing auth due to timing >> >> > issues. >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > *I am considering 2 alternatives right now. >> The >> >> > > > current >> >> > > > > > > > > documented >> >> > > > > > > > > >> > > > approach >> >> > > > > > > > > >> > > > > is zookeeper based and it does not require >> any >> >> > > changes >> >> > > > > to >> >> > > > > > > > > >> > > UpdateMetadata >> >> > > > > > > > > >> > > > > protocol. An alternative approach can remove >> >> > > zookeeper >> >> > > > > > > > > dependency >> >> > > > > > > > > >> as >> >> > > > > > > > > >> > > well >> >> > > > > > > > > >> > > > > but we can discuss that in KIP discussion >> call.* >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to ping >> Jun to >> >> > > > > > arrange a >> >> > > > > > > > > call? >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of having >> just the >> >> > > > > > controller >> >> > > > > > > > > issue >> >> > > > > > > > > >> the >> >> > > > > > > > > >> > > > > delegation tokens, to avoid syncing a shared >> >> > secret. >> >> > > > Not >> >> > > > > > > sure >> >> > > > > > > > if >> >> > > > > > > > > >> we >> >> > > > > > > > > >> > > > > want to continue the discussion here or on >> the >> >> > > wiki. I >> >> > > > > > think >> >> > > > > > > > > that >> >> > > > > > > > > >> we >> >> > > > > > > > > >> > > > > can decouple the problem of "token >> distribution" >> >> > > from >> >> > > > > > > "shared >> >> > > > > > > > > >> secret >> >> > > > > > > > > >> > > > > distribution" and use the controller as the >> only >> >> > > token >> >> > > > > > > > generator >> >> > > > > > > > > >> to >> >> > > > > > > > > >> > > > > solve the second issue, while still using >> ZK async >> >> > > to >> >> > > > > > > > distribute >> >> > > > > > > > > >> > > > > tokens. >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the previous Email I am >> fine with >> >> > > > that >> >> > > > > > > > approach >> >> > > > > > > > > >> as >> >> > > > > > > > > >> > > long >> >> > > > > > > > > >> > > > as >> >> > > > > > > > > >> > > > > we agree that the extra complexity of >> >> > > adding/updating >> >> > > > > APIS >> >> > > > > > > > adds >> >> > > > > > > > > >> enough >> >> > > > > > > > > >> > > > > value. The advantage with the controller >> approach >> >> > is >> >> > > > > > secret >> >> > > > > > > > > >> rotation >> >> > > > > > > > > >> > > can >> >> > > > > > > > > >> > > > be >> >> > > > > > > > > >> > > > > automated,frequent and would not require >> >> > > deployment. * >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > Can you detail the extra complexity (or point >> me to >> >> > > the >> >> > > > > > email >> >> > > > > > > I >> >> > > > > > > > > >> > > > missed?) - which Apis are required? >> >> > > > > > > > > >> > > > As far as I can tell, clients can already >> find the >> >> > > > > > controller >> >> > > > > > > > from >> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned about >> controller >> >> > > > load. >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > * While I like the idea of forcing kerberos >> auth >> >> > for >> >> > > > > > > renwal, I >> >> > > > > > > > > >> think >> >> > > > > > > > > >> > > > > it mixes the transport layer the the request >> >> > content >> >> > > > in >> >> > > > > a >> >> > > > > > > > pretty >> >> > > > > > > > > >> ugly >> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to non-owner >> is >> >> > > better. >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > *I feel this is a necessary evil. While >> this will >> >> > > make >> >> > > > > the >> >> > > > > > > > kafka >> >> > > > > > > > > >> code >> >> > > > > > > > > >> > > > > pretty straight forward , forcing renewer >> to >> >> > > > non-owner >> >> > > > > > > pushes >> >> > > > > > > > > >> the code >> >> > > > > > > > > >> > > > > ugliness to client and makes it even harder >> to >> >> > > > > > integrate. * >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > As mentioned before, I don't think the >> "renewal by >> >> > > > other" >> >> > > > > > > > approach >> >> > > > > > > > > >> is >> >> > > > > > > > > >> > > > that ugly for the clients we expect to use >> >> > delegation >> >> > > > > tokens >> >> > > > > > > > since >> >> > > > > > > > > >> > > > they will have an app-master of some sort who >> >> > > requested >> >> > > > > the >> >> > > > > > > > token >> >> > > > > > > > > to >> >> > > > > > > > > >> > > > begin with. >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > The response for my question on how multiple >> >> > > > identities >> >> > > > > > will >> >> > > > > > > > be >> >> > > > > > > > > >> > > > > handled wasn't super clear to me - AFAIK, >> we don't >> >> > > > > > > > authenticate >> >> > > > > > > > > >> each >> >> > > > > > > > > >> > > > > request, we authenticate connections. >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > *We authenticate connections, and only when >> they >> >> > are >> >> > > > > being >> >> > > > > > > > > >> established. >> >> > > > > > > > > >> > > > Let >> >> > > > > > > > > >> > > > > me try to phrase this as a question, in >> absence of >> >> > > > > > > delegation >> >> > > > > > > > > >> tokens if >> >> > > > > > > > > >> > > > we >> >> > > > > > > > > >> > > > > had to support the use case using user >> TGT's how >> >> > > would >> >> > > > > we >> >> > > > > > do >> >> > > > > > > > it? >> >> > > > > > > > > >> My >> >> > > > > > > > > >> > > point >> >> > > > > > > > > >> > > > > was it would be no different with delegation >> >> > tokens. >> >> > > > The >> >> > > > > > use >> >> > > > > > > > > case >> >> > > > > > > > > >> you >> >> > > > > > > > > >> > > are >> >> > > > > > > > > >> > > > > describing seems more like impersonation.* >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of the things that >> >> > delegation >> >> > > > > > tokens >> >> > > > > > > > > >> handled. >> >> > > > > > > > > >> > > > Maybe I got it wrong :) >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > Thanks for the detailed answers. >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > Gwen >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > > Thanks >> >> > > > > > > > > >> > > > > Parth >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen >> Shapira < >> >> > > > > > > > > g...@confluent.io >> >> > > > > > > > > >> > >> >> > > > > > > > > >> > > > wrote: >> >> > > > > > > > > >> > > > > >> >> > > > > > > > > >> > > > >> Hi Parth and Harsha, >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> Few more comments: >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * The API / RequestResponse section >> doesn't seem >> >> > to >> >> > > > > have >> >> > > > > > > good >> >> > > > > > > > > >> > > > >> description of the changes to the Kafka >> Protocol. >> >> > > > > Sounds >> >> > > > > > > like >> >> > > > > > > > > >> you are >> >> > > > > > > > > >> > > > >> proposing new DelegationTokenRequest and >> >> > > > > > RenewTokenRequest >> >> > > > > > > > (and >> >> > > > > > > > > >> > > > >> matching responses), without detailing the >> >> > contents >> >> > > > of >> >> > > > > > the >> >> > > > > > > > > >> requests >> >> > > > > > > > > >> > > > >> and responses? Or rather, you show the >> class >> >> > > > interface, >> >> > > > > > but >> >> > > > > > > > not >> >> > > > > > > > > >> the >> >> > > > > > > > > >> > > > >> underlying protocol. This could be seen as >> an >> >> > > > > > > implementation >> >> > > > > > > > > >> detail, >> >> > > > > > > > > >> > > > >> but since the binary protocol is what we >> provide >> >> > to >> >> > > > > > > non-Java >> >> > > > > > > > > >> clients, >> >> > > > > > > > > >> > > > >> we need to show the changes there. >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * getDelegationToken sounds like >> >> > > > > > > > delegationTokenRequestHandler? >> >> > > > > > > > > >> Is it >> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or Client? >> Its >> >> > > > > unclear... >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * I want to emphasize that even though >> delegation >> >> > > > > tokens >> >> > > > > > > are >> >> > > > > > > > a >> >> > > > > > > > > >> Hadoop >> >> > > > > > > > > >> > > > >> innovation, I feel very strongly about not >> adding >> >> > > > > > > dependency >> >> > > > > > > > on >> >> > > > > > > > > >> Hadoop >> >> > > > > > > > > >> > > > >> when implementing delegation tokens for >> Kafka. >> >> > The >> >> > > > KIP >> >> > > > > > > > doesn't >> >> > > > > > > > > >> imply >> >> > > > > > > > > >> > > > >> such dependency, but if you can clarify... >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * Can we get delegation token at any time >> after >> >> > > > > > > > authenticating? >> >> > > > > > > > > >> only >> >> > > > > > > > > >> > > > >> immediately after? >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * My understanding is that tokens will >> propagate >> >> > > via >> >> > > > ZK >> >> > > > > > but >> >> > > > > > > > > >> without >> >> > > > > > > > > >> > > > >> additional changes to UpdateMetadata >> protocol, >> >> > > > correct? >> >> > > > > > > > Clients >> >> > > > > > > > > >> > > > >> currently don't retry on SASL auth failure >> >> > (IIRC), >> >> > > > but >> >> > > > > > > since >> >> > > > > > > > > the >> >> > > > > > > > > >> > > > >> tokens propagate between brokers asynch, >> we will >> >> > > need >> >> > > > > to >> >> > > > > > > > retry >> >> > > > > > > > > a >> >> > > > > > > > > >> bit >> >> > > > > > > > > >> > > > >> to avoid clients failing auth due to timing >> >> > issues. >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients not >> touching ZK >> >> > > > directly >> >> > > > > > :) >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of having >> just the >> >> > > > > > controller >> >> > > > > > > > > >> issue the >> >> > > > > > > > > >> > > > >> delegation tokens, to avoid syncing a >> shared >> >> > > secret. >> >> > > > > Not >> >> > > > > > > sure >> >> > > > > > > > > if >> >> > > > > > > > > >> we >> >> > > > > > > > > >> > > > >> want to continue the discussion here or on >> the >> >> > > wiki. >> >> > > > I >> >> > > > > > > think >> >> > > > > > > > > >> that we >> >> > > > > > > > > >> > > > >> can decouple the problem of "token >> distribution" >> >> > > from >> >> > > > > > > "shared >> >> > > > > > > > > >> secret >> >> > > > > > > > > >> > > > >> distribution" and use the controller as >> the only >> >> > > > token >> >> > > > > > > > > generator >> >> > > > > > > > > >> to >> >> > > > > > > > > >> > > > >> solve the second issue, while still using >> ZK >> >> > async >> >> > > to >> >> > > > > > > > > distribute >> >> > > > > > > > > >> > > > >> tokens. >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * I am also uncomfortable with infinite >> lifetime >> >> > of >> >> > > > > > tokens >> >> > > > > > > > (and >> >> > > > > > > > > >> hoped >> >> > > > > > > > > >> > > > >> to hear from others in the community) - but >> >> > having >> >> > > > the >> >> > > > > > > option >> >> > > > > > > > > and >> >> > > > > > > > > >> > > > >> documenting it as less secure, allows >> users to >> >> > > > > configure >> >> > > > > > > > their >> >> > > > > > > > > >> system >> >> > > > > > > > > >> > > > >> as the wish. >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * While I like the idea of forcing >> kerberos auth >> >> > > for >> >> > > > > > > renwal, >> >> > > > > > > > I >> >> > > > > > > > > >> think >> >> > > > > > > > > >> > > > >> it mixes the transport layer the the >> request >> >> > > content >> >> > > > > in a >> >> > > > > > > > > pretty >> >> > > > > > > > > >> ugly >> >> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to non-owner >> is >> >> > > better. >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> Things I'd still like to see: >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * More detailed explanation on what we >> plan to do >> >> > > for >> >> > > > > the >> >> > > > > > > > java >> >> > > > > > > > > >> clients >> >> > > > > > > > > >> > > > >> specifically - new configuration? new APIs? >> >> > > > > > > > > >> > > > >> The response for my question on how >> multiple >> >> > > > identities >> >> > > > > > > will >> >> > > > > > > > be >> >> > > > > > > > > >> > > > >> handled wasn't super clear to me - AFAIK, >> we >> >> > don't >> >> > > > > > > > authenticate >> >> > > > > > > > > >> each >> >> > > > > > > > > >> > > > >> request, we authenticate connections. >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens are only >> used >> >> > in >> >> > > > the >> >> > > > > > > Hadoop >> >> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there are >> >> > alternatives >> >> > > in >> >> > > > > > other >> >> > > > > > > > > >> ecosystems >> >> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether >> there >> >> > are >> >> > > > some >> >> > > > > > > > > >> advantages >> >> > > > > > > > > >> > > > >> there. >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> Gwen >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha < >> >> > > > > ka...@harsha.io >> >> > > > > > > >> >> > > > > > > > > wrote: >> >> > > > > > > > > >> > > > >> > Hi Gwen, >> >> > > > > > > > > >> > > > >> > Can you look at Parth's last >> reply. >> >> > > Does >> >> > > > > it >> >> > > > > > > > answer >> >> > > > > > > > > >> your >> >> > > > > > > > > >> > > > >> > concerns. >> >> > > > > > > > > >> > > > >> > >> >> > > > > > > > > >> > > > >> > Thanks, >> >> > > > > > > > > >> > > > >> > Harsha >> >> > > > > > > > > >> > > > >> > >> >> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth >> >> > > brahmbhatt >> >> > > > > > wrote: >> >> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki >> already >> >> > has >> >> > > > > > details >> >> > > > > > > on >> >> > > > > > > > > >> token >> >> > > > > > > > > >> > > > >> >> expiration >> >> > > > > > > > > >> > > > >> >> under token acquisition process >> >> > > > > > > > > >> > > > >> >> < >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition >> >> > > > > > > > > >> > > > >> >. >> >> > > > > > > > > >> > > > >> >> Current proposal is that tokens will >> expire >> >> > > based >> >> > > > > on a >> >> > > > > > > > > server >> >> > > > > > > > > >> side >> >> > > > > > > > > >> > > > >> >> configuration (default 24 hours) unless >> >> > renewed. >> >> > > > > > Renewal >> >> > > > > > > > is >> >> > > > > > > > > >> only >> >> > > > > > > > > >> > > > allowed >> >> > > > > > > > > >> > > > >> >> until the max life time of token. >> >> > Alternatively >> >> > > we >> >> > > > > > could >> >> > > > > > > > > also >> >> > > > > > > > > >> make >> >> > > > > > > > > >> > > > that >> >> > > > > > > > > >> > > > >> >> an >> >> > > > > > > > > >> > > > >> >> optional param and the server side >> default can >> >> > > > serve >> >> > > > > > as >> >> > > > > > > > the >> >> > > > > > > > > >> upper >> >> > > > > > > > > >> > > > bound. >> >> > > > > > > > > >> > > > >> >> >> >> > > > > > > > > >> > > > >> >> To your second point it will be done >> exactly >> >> > the >> >> > > > > same >> >> > > > > > > way >> >> > > > > > > > we >> >> > > > > > > > > >> would >> >> > > > > > > > > >> > > > >> >> support >> >> > > > > > > > > >> > > > >> >> multiple keytabs. The calling client >> will have >> >> > > to >> >> > > > > put >> >> > > > > > > the >> >> > > > > > > > > >> tokens it >> >> > > > > > > > > >> > > > >> wants >> >> > > > > > > > > >> > > > >> >> to use in the subject instance and call >> >> > > > > > produce/consume >> >> > > > > > > > > inside >> >> > > > > > > > > >> > > > >> >> subject.doas. Each caller will have to >> keep >> >> > > track >> >> > > > of >> >> > > > > > its >> >> > > > > > > > own >> >> > > > > > > > > >> > > > subject. I >> >> > > > > > > > > >> > > > >> >> will have to look at the code to see if >> we >> >> > > support >> >> > > > > > this >> >> > > > > > > > > >> feature >> >> > > > > > > > > >> > > right >> >> > > > > > > > > >> > > > >> now >> >> > > > > > > > > >> > > > >> >> but my understanding is delegation token >> >> > > shouldn't >> >> > > > > > need >> >> > > > > > > > any >> >> > > > > > > > > >> special >> >> > > > > > > > > >> > > > >> >> treatment as its just another type of >> >> > Credential >> >> > > > in >> >> > > > > > the >> >> > > > > > > > > >> subject. >> >> > > > > > > > > >> > > > >> >> >> >> > > > > > > > > >> > > > >> >> I would also like to know what is your >> opinion >> >> > > > about >> >> > > > > > > > > infinite >> >> > > > > > > > > >> > > renewal >> >> > > > > > > > > >> > > > >> (my >> >> > > > > > > > > >> > > > >> >> recommendation is to not support this), >> tokens >> >> > > > > > renewing >> >> > > > > > > > them >> >> > > > > > > > > >> > > self(my >> >> > > > > > > > > >> > > > >> >> recommendation is to not support this) >> and >> >> > most >> >> > > > > > > > importantly >> >> > > > > > > > > >> your >> >> > > > > > > > > >> > > > choice >> >> > > > > > > > > >> > > > >> >> between the alternatives listed on this >> thread >> >> > > > > > > > > >> > > > >> >> < >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism >> >> > > > > > > > > >> > > > >> > >> >> > > > > > > > > >> > > > >> >> ( I am leaning towards the >> alternative-2 minus >> >> > > > > > > controller >> >> > > > > > > > > >> > > > distributing >> >> > > > > > > > > >> > > > >> >> secret). Thanks again for reviewing. >> >> > > > > > > > > >> > > > >> >> >> >> > > > > > > > > >> > > > >> >> Thanks >> >> > > > > > > > > >> > > > >> >> Parth >> >> > > > > > > > > >> > > > >> >> >> >> > > > > > > > > >> > > > >> >> >> >> > > > > > > > > >> > > > >> >> >> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen >> Shapira < >> >> > > > > > > > > >> g...@confluent.io> >> >> > > > > > > > > >> > > > wrote: >> >> > > > > > > > > >> > > > >> >> >> >> > > > > > > > > >> > > > >> >> > Harsha, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I >> didn't >> >> > see >> >> > > > > your >> >> > > > > > > > design >> >> > > > > > > > > >> yet, >> >> > > > > > > > > >> > > > but in >> >> > > > > > > > > >> > > > >> >> > our proxy, we have a set of >> producers, which >> >> > > > will >> >> > > > > > > serve >> >> > > > > > > > > >> multiple >> >> > > > > > > > > >> > > > users >> >> > > > > > > > > >> > > > >> >> > going through the proxy. Since these >> users >> >> > > will >> >> > > > > have >> >> > > > > > > > > >> different >> >> > > > > > > > > >> > > > >> >> > privileges, they'll need to >> authenticate >> >> > > > > separately, >> >> > > > > > > and >> >> > > > > > > > > >> can't >> >> > > > > > > > > >> > > > share a >> >> > > > > > > > > >> > > > >> >> > token. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> > Am I missing anything? >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> > Gwen >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, >> Harsha < >> >> > > > > > > ka...@harsha.io >> >> > > > > > > > > >> >> > > > > > > > > >> wrote: >> >> > > > > > > > > >> > > > >> >> > > Gwen, >> >> > > > > > > > > >> > > > >> >> > > On your second point. >> Can you >> >> > > > > describe >> >> > > > > > a >> >> > > > > > > > > >> usecase >> >> > > > > > > > > >> > > where >> >> > > > > > > > > >> > > > >> >> > > mutliple clients ended up >> >> > > sharing a >> >> > > > > > > > producer >> >> > > > > > > > > >> and >> >> > > > > > > > > >> > > even >> >> > > > > > > > > >> > > > if >> >> > > > > > > > > >> > > > >> they >> >> > > > > > > > > >> > > > >> >> > > do why can't they not use >> >> > single >> >> > > > > token >> >> > > > > > > that >> >> > > > > > > > > >> producer >> >> > > > > > > > > >> > > > >> >> > > captures. Why would we >> need >> >> > > > multiple >> >> > > > > > > > clients >> >> > > > > > > > > >> with >> >> > > > > > > > > >> > > > >> different >> >> > > > > > > > > >> > > > >> >> > > tokens sharing a single >> >> > instance >> >> > > of >> >> > > > > > > > producer. >> >> > > > > > > > > >> Also >> >> > > > > > > > > >> > > in >> >> > > > > > > > > >> > > > >> this >> >> > > > > > > > > >> > > > >> >> > > case other clients have >> access >> >> > > all >> >> > > > > the >> >> > > > > > > > tokens >> >> > > > > > > > > >> no? >> >> > > > > > > > > >> > > > >> >> > > >> >> > > > > > > > > >> > > > >> >> > > Thanks, >> >> > > > > > > > > >> > > > >> >> > > Harsha >> >> > > > > > > > > >> > > > >> >> > > >> >> > > > > > > > > >> > > > >> >> > > >> >> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, >> Gwen >> >> > > Shapira >> >> > > > > > > wrote: >> >> > > > > > > > > >> > > > >> >> > >> Sorry for the delay: >> >> > > > > > > > > >> > > > >> >> > >> >> >> > > > > > > > > >> > > > >> >> > >> Two questions that we didn't see >> in the >> >> > > wiki: >> >> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for >> delegation >> >> > > > > tokens? >> >> > > > > > > > > >> Renewal? How >> >> > > > > > > > > >> > > > do we >> >> > > > > > > > > >> > > > >> >> > >> revoke them? >> >> > > > > > > > > >> > > > >> >> > >> 2. If we want to use delegation >> tokens >> >> > for >> >> > > > > > "do-as" >> >> > > > > > > > > (say, >> >> > > > > > > > > >> > > submit >> >> > > > > > > > > >> > > > >> Storm >> >> > > > > > > > > >> > > > >> >> > >> job as my user), we will need a >> producer >> >> > > for >> >> > > > > > every >> >> > > > > > > > job >> >> > > > > > > > > >> (we >> >> > > > > > > > > >> > > can't >> >> > > > > > > > > >> > > > >> share >> >> > > > > > > > > >> > > > >> >> > >> them between multiple jobs running >> on >> >> > same >> >> > > > > node), >> >> > > > > > > > since >> >> > > > > > > > > >> we >> >> > > > > > > > > >> > > only >> >> > > > > > > > > >> > > > >> >> > >> authenticate when connecting. Is >> there a >> >> > > plan >> >> > > > > to >> >> > > > > > > > change >> >> > > > > > > > > >> this >> >> > > > > > > > > >> > > for >> >> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order to >> allow >> >> > > multiple >> >> > > > > > users >> >> > > > > > > > > with >> >> > > > > > > > > >> > > > different >> >> > > > > > > > > >> > > > >> >> > >> tokens to share a client? >> >> > > > > > > > > >> > > > >> >> > >> >> >> > > > > > > > > >> > > > >> >> > >> Gwen >> >> > > > > > > > > >> > > > >> >> > >> >> >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, >> parth >> >> > > > > brahmbhatt >> >> > > > > > > > > >> > > > >> >> > >> <brahmbhatt.pa...@gmail.com> >> wrote: >> >> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more time, >> can >> >> > other >> >> > > > > > > committers >> >> > > > > > > > > >> review? >> >> > > > > > > > > >> > > > >> >> > >> > >> >> > > > > > > > > >> > > > >> >> > >> > Thanks >> >> > > > > > > > > >> > > > >> >> > >> > Parth >> >> > > > > > > > > >> > > > >> >> > >> > >> >> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, >> >> > Harsha < >> >> > > > > > > > > >> ka...@harsha.io> >> >> > > > > > > > > >> > > > wrote: >> >> > > > > > > > > >> > > > >> >> > >> > >> >> > > > > > > > > >> > > > >> >> > >> >> Parth, >> >> > > > > > > > > >> > > > >> >> > >> >> Overall current >> design looks >> >> > > > good >> >> > > > > to >> >> > > > > > > > me. I >> >> > > > > > > > > >> am +1 >> >> > > > > > > > > >> > > on >> >> > > > > > > > > >> > > > >> the >> >> > > > > > > > > >> > > > >> >> > KIP. >> >> > > > > > > > > >> > > > >> >> > >> >> >> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this >> as >> >> > well. >> >> > > > > > > > > >> > > > >> >> > >> >> >> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha >> >> > > > > > > > > >> > > > >> >> > >> >> >> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 >> AM, >> >> > parth >> >> > > > > > > > brahmbhatt >> >> > > > > > > > > >> wrote: >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra. >> >> > > > > > > > > >> > > > >> >> > >> >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of >> infinite >> >> > > > lifetime >> >> > > > > > > but I >> >> > > > > > > > > >> see the >> >> > > > > > > > > >> > > > >> Streaming >> >> > > > > > > > > >> > > > >> >> > use >> >> > > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use >> case I >> >> > > was >> >> > > > > > hoping >> >> > > > > > > > > >> there will >> >> > > > > > > > > >> > > > be >> >> > > > > > > > > >> > > > >> some >> >> > > > > > > > > >> > > > >> >> > notion >> >> > > > > > > > > >> > > > >> >> > >> >> > of >> >> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can get new >> >> > > > delegation >> >> > > > > > > tokens >> >> > > > > > > > > at >> >> > > > > > > > > >> fixed >> >> > > > > > > > > >> > > > >> interval >> >> > > > > > > > > >> > > > >> >> > and >> >> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers. If >> that is >> >> > not >> >> > > > the >> >> > > > > > case >> >> > > > > > > > for >> >> > > > > > > > > >> we can >> >> > > > > > > > > >> > > > >> discuss >> >> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing >> them self >> >> > > and >> >> > > > > the >> >> > > > > > > > > >> security >> >> > > > > > > > > >> > > > >> implications >> >> > > > > > > > > >> > > > >> >> > of the >> >> > > > > > > > > >> > > > >> >> > >> >> > same. >> >> > > > > > > > > >> > > > >> >> > >> >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients to >> fetch >> >> > tokens >> >> > > > > from >> >> > > > > > > > > >> zookeeper, >> >> > > > > > > > > >> > > > >> overall I >> >> > > > > > > > > >> > > > >> >> > think >> >> > > > > > > > > >> > > > >> >> > >> >> > its >> >> > > > > > > > > >> > > > >> >> > >> >> > better if clients don't rely >> on our >> >> > > > > metadata >> >> > > > > > > > store >> >> > > > > > > > > >> and I >> >> > > > > > > > > >> > > > >> think we >> >> > > > > > > > > >> > > > >> >> > are >> >> > > > > > > > > >> > > > >> >> > >> >> > moving in that direction with >> all >> >> > the >> >> > > > > KIP-4 >> >> > > > > > > > > >> improvements. >> >> > > > > > > > > >> > > > I >> >> > > > > > > > > >> > > > >> chose >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case the >> client >> >> > > > will >> >> > > > > > > still >> >> > > > > > > > > >> just talk >> >> > > > > > > > > >> > > > to >> >> > > > > > > > > >> > > > >> >> > broker , its >> >> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will use >> zookeeper >> >> > > > which >> >> > > > > we >> >> > > > > > > > > >> already do >> >> > > > > > > > > >> > > > for a >> >> > > > > > > > > >> > > > >> lot >> >> > > > > > > > > >> > > > >> >> > of >> >> > > > > > > > > >> > > > >> >> > >> >> > other >> >> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of >> development + and >> >> > > the >> >> > > > > > > ability >> >> > > > > > > > > so >> >> > > > > > > > > >> > > tokens >> >> > > > > > > > > >> > > > >> will >> >> > > > > > > > > >> > > > >> >> > survive >> >> > > > > > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster >> >> > > failure. >> >> > > > > if a >> >> > > > > > > > > >> majority >> >> > > > > > > > > >> > > > agrees >> >> > > > > > > > > >> > > > >> the >> >> > > > > > > > > >> > > > >> >> > added >> >> > > > > > > > > >> > > > >> >> > >> >> > complexity to have controller >> >> > > forwarding >> >> > > > > > keys >> >> > > > > > > to >> >> > > > > > > > > all >> >> > > > > > > > > >> > > > broker is >> >> > > > > > > > > >> > > > >> >> > justified >> >> > > > > > > > > >> > > > >> >> > >> >> > as >> >> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter security >> , I am >> >> > > fine >> >> > > > > > with >> >> > > > > > > > that >> >> > > > > > > > > >> option >> >> > > > > > > > > >> > > > too. >> >> > > > > > > > > >> > > > >> >> > >> >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not >> support SSL >> >> > > we >> >> > > > > can >> >> > > > > > > not >> >> > > > > > > > > >> store >> >> > > > > > > > > >> > > > master >> >> > > > > > > > > >> > > > >> keys >> >> > > > > > > > > >> > > > >> >> > in >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys will >> be >> >> > > exposed >> >> > > > > on >> >> > > > > > > > wire. >> >> > > > > > > > > To >> >> > > > > > > > > >> > > > support >> >> > > > > > > > > >> > > > >> >> > rotation >> >> > > > > > > > > >> > > > >> >> > >> >> > without affecting current >> clients is >> >> > > > > > > something I >> >> > > > > > > > > >> need to >> >> > > > > > > > > >> > > > put >> >> > > > > > > > > >> > > > >> more >> >> > > > > > > > > >> > > > >> >> > thought >> >> > > > > > > > > >> > > > >> >> > >> >> > in. My current proposal >> assumes the >> >> > > > > rotation >> >> > > > > > > > will >> >> > > > > > > > > >> > > > invalidate >> >> > > > > > > > > >> > > > >> all >> >> > > > > > > > > >> > > > >> >> > current >> >> > > > > > > > > >> > > > >> >> > >> >> > tokens. >> >> > > > > > > > > >> > > > >> >> > >> >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > I request committers to also >> review >> >> > > and >> >> > > > > post >> >> > > > > > > > their >> >> > > > > > > > > >> > > comments >> >> > > > > > > > > >> > > > >> so we >> >> > > > > > > > > >> > > > >> >> > can >> >> > > > > > > > > >> > > > >> >> > >> >> > make >> >> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP. >> >> > > > > > > > > >> > > > >> >> > >> >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks >> >> > > > > > > > > >> > > > >> >> > >> >> > Parth >> >> > > > > > > > > >> > > > >> >> > >> >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 >> AM, >> >> > > Ashish >> >> > > > > > Singh >> >> > > > > > > < >> >> > > > > > > > > >> > > > >> asi...@cloudera.com >> >> > > > > > > > > >> > > > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > wrote: >> >> > > > > > > > > >> > > > >> >> > >> >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at >> 11:26 AM, >> >> > > > > Harsha < >> >> > > > > > > > > >> > > > ka...@harsha.io> >> >> > > > > > > > > >> > > > >> >> > wrote: >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two >> discussion >> >> > > threads >> >> > > > on >> >> > > > > > > this >> >> > > > > > > > > KIP. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the response from >> >> > Jitendra >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a large >> number of >> >> > > > > clients >> >> > > > > > > that >> >> > > > > > > > > are >> >> > > > > > > > > >> > > > running >> >> > > > > > > > > >> > > > >> all >> >> > > > > > > > > >> > > > >> >> > over the >> >> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that authenticate >> with >> >> > > Kafka >> >> > > > > > > > brokers, >> >> > > > > > > > > >> is very >> >> > > > > > > > > >> > > > >> similar >> >> > > > > > > > > >> > > > >> >> > to the >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large >> number >> >> > of >> >> > > > > tasks >> >> > > > > > > > > running >> >> > > > > > > > > >> > > across >> >> > > > > > > > > >> > > > >> the >> >> > > > > > > > > >> > > > >> >> > cluster >> >> > > > > > > > > >> > > > >> >> > >> >> that >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need authentication to >> Hdfs >> >> > > > Namenode. >> >> > > > > > > > > >> Therefore, the >> >> > > > > > > > > >> > > > >> >> > delegation token >> >> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem like a >> good >> >> > fit >> >> > > > for >> >> > > > > > > this >> >> > > > > > > > > use >> >> > > > > > > > > >> case >> >> > > > > > > > > >> > > > as we >> >> > > > > > > > > >> > > > >> >> > have seen >> >> > > > > > > > > >> > > > >> >> > >> >> it >> >> > > > > > > > > >> > > > >> >> > >> >> > > > working at large scale in >> HDFS >> >> > and >> >> > > > > YARN. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > The proposed design is >> very >> >> > much >> >> > > > > > inline >> >> > > > > > > > with >> >> > > > > > > > > >> Hadoop >> >> > > > > > > > > >> > > > >> >> > approach. A few >> >> > > > > > > > > >> > > > >> >> > >> >> > > > comments: >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want >> to allow >> >> > > > > > infinite >> >> > > > > > > > > >> renewable >> >> > > > > > > > > >> > > > >> lifetime >> >> > > > > > > > > >> > > > >> >> > for a >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a >> token >> >> > to a >> >> > > > max >> >> > > > > > > life >> >> > > > > > > > > time >> >> > > > > > > > > >> > > > (default >> >> > > > > > > > > >> > > > >> 7 >> >> > > > > > > > > >> > > > >> >> > days). A >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is >> >> > believed >> >> > > to >> >> > > > > > > > increase >> >> > > > > > > > > >> with >> >> > > > > > > > > >> > > > time. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having infinite >> >> > > lifetime >> >> > > > > > might >> >> > > > > > > > not >> >> > > > > > > > > >> be the >> >> > > > > > > > > >> > > > best >> >> > > > > > > > > >> > > > >> idea. >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the >> tokens >> >> > are >> >> > > > > stored >> >> > > > > > > in >> >> > > > > > > > > >> zookeeper >> >> > > > > > > > > >> > > > as >> >> > > > > > > > > >> > > > >> well, >> >> > > > > > > > > >> > > > >> >> > and >> >> > > > > > > > > >> > > > >> >> > >> >> can >> >> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there. This is >> clever >> >> > > as >> >> > > > it >> >> > > > > > can >> >> > > > > > > > > allow >> >> > > > > > > > > >> > > > >> replacing the >> >> > > > > > > > > >> > > > >> >> > tokens >> >> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out of max >> life >> >> > > time, >> >> > > > > and >> >> > > > > > > > > clients >> >> > > > > > > > > >> can >> >> > > > > > > > > >> > > > >> download >> >> > > > > > > > > >> > > > >> >> > new >> >> > > > > > > > > >> > > > >> >> > >> >> tokens >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It >> shouldn't be >> >> > a >> >> > > > big >> >> > > > > > load >> >> > > > > > > > on >> >> > > > > > > > > >> > > zookeeper >> >> > > > > > > > > >> > > > >> as a >> >> > > > > > > > > >> > > > >> >> > client >> >> > > > > > > > > >> > > > >> >> > >> >> will >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new token >> once in >> >> > > > > several >> >> > > > > > > > days. >> >> > > > > > > > > >> In this >> >> > > > > > > > > >> > > > >> approach >> >> > > > > > > > > >> > > > >> >> > you >> >> > > > > > > > > >> > > > >> >> > >> >> don't >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on >> the >> >> > > token >> >> > > > > even >> >> > > > > > > for >> >> > > > > > > > > >> long >> >> > > > > > > > > >> > > > running >> >> > > > > > > > > >> > > > >> >> > clients. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token password are >> >> > > generated >> >> > > > > > using >> >> > > > > > > a >> >> > > > > > > > > >> master >> >> > > > > > > > > >> > > key. >> >> > > > > > > > > >> > > > >> The >> >> > > > > > > > > >> > > > >> >> > master >> >> > > > > > > > > >> > > > >> >> > >> >> key >> >> > > > > > > > > >> > > > >> >> > >> >> > > > should also be >> periodically >> >> > > changed. >> >> > > > > In >> >> > > > > > > > > Hadoop, >> >> > > > > > > > > >> the >> >> > > > > > > > > >> > > > >> default >> >> > > > > > > > > >> > > > >> >> > renewal >> >> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.? >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require >> brokers >> >> > > > > > maintaining >> >> > > > > > > a >> >> > > > > > > > > >> list of X >> >> > > > > > > > > >> > > > most >> >> > > > > > > > > >> > > > >> >> > recent >> >> > > > > > > > > >> > > > >> >> > >> >> master >> >> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will have >> to be >> >> > > > > persisted >> >> > > > > > > > > >> somewhere, as >> >> > > > > > > > > >> > > > if a >> >> > > > > > > > > >> > > > >> >> > broker >> >> > > > > > > > > >> > > > >> >> > >> >> goes >> >> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to get >> that list >> >> > > > again >> >> > > > > > and >> >> > > > > > > > > >> storing >> >> > > > > > > > > >> > > > master >> >> > > > > > > > > >> > > > >> keys >> >> > > > > > > > > >> > > > >> >> > on ZK >> >> > > > > > > > > >> > > > >> >> > >> >> is >> >> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea. However, >> if a >> >> > > > broker >> >> > > > > > goes >> >> > > > > > > > > down >> >> > > > > > > > > >> then >> >> > > > > > > > > >> > > we >> >> > > > > > > > > >> > > > >> have >> >> > > > > > > > > >> > > > >> >> > much >> >> > > > > > > > > >> > > > >> >> > >> >> bigger >> >> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and >> client can >> >> > > > always >> >> > > > > > > > > >> > > re-authenticate >> >> > > > > > > > > >> > > > is >> >> > > > > > > > > >> > > > >> such >> >> > > > > > > > > >> > > > >> >> > >> >> events. >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a >> look at >> >> > > other >> >> > > > > > > > > >> alternatives >> >> > > > > > > > > >> > > this >> >> > > > > > > > > >> > > > >> list has >> >> > > > > > > > > >> > > > >> >> > >> >> > > suggested? >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough >> proposal, >> >> > > > great >> >> > > > > > > work!" >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at >> 10:28 >> >> > PM, >> >> > > > Gwen >> >> > > > > > > > Shapira >> >> > > > > > > > > >> wrote: >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me. >> Thanks! >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at >> 9:25 >> >> > PM, >> >> > > > > > Harsha < >> >> > > > > > > > > >> > > > ka...@harsha.io >> >> > > > > > > > > >> > > > >> > >> >> > > > > > > > > >> > > > >> >> > wrote: >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any >> release >> >> > > > > vehicle >> >> > > > > > > but >> >> > > > > > > > > >> still the >> >> > > > > > > > > >> > > > >> work can >> >> > > > > > > > > >> > > > >> >> > move >> >> > > > > > > > > >> > > > >> >> > >> >> > > > forward. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is >> interested in >> >> > the >> >> > > > KIP >> >> > > > > > > > please >> >> > > > > > > > > >> do the >> >> > > > > > > > > >> > > > >> review and >> >> > > > > > > > > >> > > > >> >> > >> >> provide >> >> > > > > > > > > >> > > > >> >> > >> >> > > the >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, >> at >> >> > 04:59 >> >> > > > PM, >> >> > > > > > > Ismael >> >> > > > > > > > > >> Juma >> >> > > > > > > > > >> > > > wrote: >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it >> would be >> >> > good >> >> > > > to >> >> > > > > > have >> >> > > > > > > > > more >> >> > > > > > > > > >> time >> >> > > > > > > > > >> > > to >> >> > > > > > > > > >> > > > >> review >> >> > > > > > > > > >> > > > >> >> > and >> >> > > > > > > > > >> > > > >> >> > >> >> > > discuss >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 >> at >> >> > 12:55 >> >> > > > AM, >> >> > > > > > Gwen >> >> > > > > > > > > >> Shapira < >> >> > > > > > > > > >> > > > >> >> > >> >> g...@confluent.io> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote: >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 >> depends on >> >> > > > KIP-43, >> >> > > > > > > which >> >> > > > > > > > > is >> >> > > > > > > > > >> > > > already a >> >> > > > > > > > > >> > > > >> bit >> >> > > > > > > > > >> > > > >> >> > of a >> >> > > > > > > > > >> > > > >> >> > >> >> risk >> >> > > > > > > > > >> > > > >> >> > >> >> > > for >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next release - >> any >> >> > > chance >> >> > > > > we >> >> > > > > > > can >> >> > > > > > > > > >> delay >> >> > > > > > > > > >> > > > >> delegation >> >> > > > > > > > > >> > > > >> >> > tokens >> >> > > > > > > > > >> > > > >> >> > >> >> to >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1? >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the community >> >> > working >> >> > > > on a >> >> > > > > > > > release >> >> > > > > > > > > >> every >> >> > > > > > > > > >> > > 3 >> >> > > > > > > > > >> > > > >> month, >> >> > > > > > > > > >> > > > >> >> > this >> >> > > > > > > > > >> > > > >> >> > >> >> is not >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a huge >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, >> 2016 at >> >> > > 5:11 >> >> > > > > PM, >> >> > > > > > > > Ashish >> >> > > > > > > > > >> Singh >> >> > > > > > > > > >> > > < >> >> > > > > > > > > >> > > > >> >> > >> >> > > asi...@cloudera.com> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote: >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for >> the >> >> > > > awesome >> >> > > > > > > write >> >> > > > > > > > > up. >> >> > > > > > > > > >> > > > Following >> >> > > > > > > > > >> > > > >> our >> >> > > > > > > > > >> > > > >> >> > >> >> discussion >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from the >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it >> will >> >> > be >> >> > > > > easier >> >> > > > > > > to >> >> > > > > > > > > >> compare >> >> > > > > > > > > >> > > > >> various >> >> > > > > > > > > >> > > > >> >> > >> >> alternatives >> >> > > > > > > > > >> > > > >> >> > >> >> > > > if they >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed together. >> I am >> >> > > > stating >> >> > > > > > > > below a >> >> > > > > > > > > >> few >> >> > > > > > > > > >> > > > >> >> > alternatives along >> >> > > > > > > > > >> > > > >> >> > >> >> > > with >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a the >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current >> proposal) >> >> > Store >> >> > > > > > > Delegation >> >> > > > > > > > > >> Token, >> >> > > > > > > > > >> > > DT, >> >> > > > > > > > > >> > > > >> on ZK. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. Client >> >> > > authenticates >> >> > > > > > with a >> >> > > > > > > > > >> broker. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. Once a >> client is >> >> > > > > > > > authenticated, >> >> > > > > > > > > >> it >> >> > > > > > > > > >> > > will >> >> > > > > > > > > >> > > > >> make a >> >> > > > > > > > > >> > > > >> >> > broker >> >> > > > > > > > > >> > > > >> >> > >> >> side >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > issue a >> delegation >> >> > > > token. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. The broker >> >> > > generates >> >> > > > a >> >> > > > > > > shared >> >> > > > > > > > > >> secret >> >> > > > > > > > > >> > > > based >> >> > > > > > > > > >> > > > >> on >> >> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> Password/Secret >> >> > shared >> >> > > > > > between >> >> > > > > > > > all >> >> > > > > > > > > >> > > brokers, >> >> > > > > > > > > >> > > > >> >> > randomly >> >> > > > > > > > > >> > > > >> >> > >> >> > > generated >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId). >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 4. Broker >> stores >> >> > this >> >> > > > > token >> >> > > > > > in >> >> > > > > > > > its >> >> > > > > > > > > >> in >> >> > > > > > > > > >> > > > memory >> >> > > > > > > > > >> > > > >> cache. >> >> > > > > > > > > >> > > > >> >> > >> >> Broker >> >> > > > > > > > > >> > > > >> >> > >> >> > > > also stores >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > the >> DelegationToken >> >> > > > > without >> >> > > > > > > the >> >> > > > > > > > > >> hmac in >> >> > > > > > > > > >> > > the >> >> > > > > > > > > >> > > > >> >> > zookeeper. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 5. All >> brokers will >> >> > > > have a >> >> > > > > > > cache >> >> > > > > > > > > >> backed >> >> > > > > > > > > >> > > by >> >> > > > > > > > > >> > > > >> >> > zookeeper so >> >> > > > > > > > > >> > > > >> >> > >> >> they >> >> > > > > > > > > >> > > > >> >> > >> >> > > > will all >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > get notified >> >> > whenever >> >> > > a >> >> > > > > new >> >> > > > > > > > token >> >> > > > > > > > > is >> >> > > > > > > > > >> > > > >> generated and >> >> > > > > > > > > >> > > > >> >> > they >> >> > > > > > > > > >> > > > >> >> > >> >> will >> >> > > > > > > > > >> > > > >> >> > >> >> > > > update >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > local cache >> whenever >> >> > > > token >> >> > > > > > > state >> >> > > > > > > > > >> changes. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 6. Broker >> returns >> >> > the >> >> > > > > token >> >> > > > > > to >> >> > > > > > > > > >> Client. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues >> and >> >> > fixes >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. Probable >> race >> >> > > > > condition, >> >> > > > > > > > client >> >> > > > > > > > > >> tries >> >> > > > > > > > > >> > > to >> >> > > > > > > > > >> > > > >> >> > authenticate >> >> > > > > > > > > >> > > > >> >> > >> >> with >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a broker >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > that is yet >> to be >> >> > > > updated >> >> > > > > > with >> >> > > > > > > > the >> >> > > > > > > > > >> newly >> >> > > > > > > > > >> > > > >> generated >> >> > > > > > > > > >> > > > >> >> > DT. >> >> > > > > > > > > >> > > > >> >> > >> >> This >> >> > > > > > > > > >> > > > >> >> > >> >> > > can >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > dealt with >> making >> >> > > > > dtRequest >> >> > > > > > > > block >> >> > > > > > > > > >> until >> >> > > > > > > > > >> > > all >> >> > > > > > > > > >> > > > >> >> > brokers have >> >> > > > > > > > > >> > > > >> >> > >> >> > > > updated >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > cache. Zk >> barrier or >> >> > > > > similar >> >> > > > > > > > > >> mechanism >> >> > > > > > > > > >> > > can >> >> > > > > > > > > >> > > > be >> >> > > > > > > > > >> > > > >> used. >> >> > > > > > > > > >> > > > >> >> > >> >> However, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > all such >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > mechanisms >> will >> >> > > increase >> >> > > > > > > > > complexity. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. Using a >> static >> >> > > secret >> >> > > > > key >> >> > > > > > > > from >> >> > > > > > > > > >> config >> >> > > > > > > > > >> > > > >> file. Will >> >> > > > > > > > > >> > > > >> >> > >> >> require >> >> > > > > > > > > >> > > > >> >> > >> >> > > yet >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > config and >> uses a >> >> > > static >> >> > > > > > > secret >> >> > > > > > > > > >> key. It >> >> > > > > > > > > >> > > is >> >> > > > > > > > > >> > > > >> advised >> >> > > > > > > > > >> > > > >> >> > to >> >> > > > > > > > > >> > > > >> >> > >> >> rotate >> >> > > > > > > > > >> > > > >> >> > >> >> > > > secret >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically. >> This >> >> > can >> >> > > > be >> >> > > > > > > > avoided >> >> > > > > > > > > >> with >> >> > > > > > > > > >> > > > >> controller >> >> > > > > > > > > >> > > > >> >> > >> >> generating >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > passing to >> brokers >> >> > > > > > > periodically. >> >> > > > > > > > > >> However, >> >> > > > > > > > > >> > > > >> this will >> >> > > > > > > > > >> > > > >> >> > >> >> require >> >> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > maintain >> certain >> >> > > counts >> >> > > > of >> >> > > > > > > > > >> secretKeys. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) >> Have >> >> > > > > controller >> >> > > > > > > > > >> generate >> >> > > > > > > > > >> > > > >> delegation >> >> > > > > > > > > >> > > > >> >> > token. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. Client >> >> > > authenticates >> >> > > > > > with a >> >> > > > > > > > > >> broker. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. Once a >> client is >> >> > > > > > > > authenticated, >> >> > > > > > > > > >> it >> >> > > > > > > > > >> > > will >> >> > > > > > > > > >> > > > >> make a >> >> > > > > > > > > >> > > > >> >> > broker >> >> > > > > > > > > >> > > > >> >> > >> >> side >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > issue a >> delegation >> >> > > > token. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. Broker >> forwards >> >> > the >> >> > > > > > request >> >> > > > > > > > to >> >> > > > > > > > > >> > > > controller. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 4. Controller >> >> > > generates >> >> > > > a >> >> > > > > DT >> >> > > > > > > and >> >> > > > > > > > > >> > > broadcasts >> >> > > > > > > > > >> > > > >> to all >> >> > > > > > > > > >> > > > >> >> > >> >> brokers. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 5. Broker >> stores >> >> > this >> >> > > > > token >> >> > > > > > in >> >> > > > > > > > its >> >> > > > > > > > > >> memory >> >> > > > > > > > > >> > > > >> cache. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 6. Controller >> >> > responds >> >> > > > to >> >> > > > > > > > broker’s >> >> > > > > > > > > >> DT >> >> > > > > > > > > >> > > req. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 7. Broker >> returns >> >> > the >> >> > > > > token >> >> > > > > > to >> >> > > > > > > > > >> Client. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues >> and >> >> > fixes >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. We will >> have to >> >> > add >> >> > > > new >> >> > > > > > > APIs >> >> > > > > > > > to >> >> > > > > > > > > >> > > support >> >> > > > > > > > > >> > > > >> >> > controller >> >> > > > > > > > > >> > > > >> >> > >> >> pushing >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > brokers on >> top of >> >> > the >> >> > > > > > minimal >> >> > > > > > > > APIs >> >> > > > > > > > > >> that >> >> > > > > > > > > >> > > are >> >> > > > > > > > > >> > > > >> >> > currently >> >> > > > > > > > > >> > > > >> >> > >> >> > > proposed. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. We will >> also have >> >> > > to >> >> > > > > add >> >> > > > > > > APIs >> >> > > > > > > > > to >> >> > > > > > > > > >> > > support >> >> > > > > > > > > >> > > > >> the >> >> > > > > > > > > >> > > > >> >> > >> >> bootstrapping >> >> > > > > > > > > >> > > > >> >> > >> >> > > > case, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > when a new >> broker >> >> > > comes >> >> > > > up >> >> > > > > > it >> >> > > > > > > > will >> >> > > > > > > > > >> have >> >> > > > > > > > > >> > > to >> >> > > > > > > > > >> > > > >> get all >> >> > > > > > > > > >> > > > >> >> > >> >> delegation >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > the >> controller. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. In >> catastrophic >> >> > > > > failures >> >> > > > > > > > where >> >> > > > > > > > > >> all >> >> > > > > > > > > >> > > > brokers >> >> > > > > > > > > >> > > > >> go >> >> > > > > > > > > >> > > > >> >> > down, >> >> > > > > > > > > >> > > > >> >> > >> >> the >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > be lost even >> if >> >> > > servers >> >> > > > > are >> >> > > > > > > > > >> restarted as >> >> > > > > > > > > >> > > > >> tokens >> >> > > > > > > > > >> > > > >> >> > are not >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > If this >> happens, >> >> > then >> >> > > > > there >> >> > > > > > > are >> >> > > > > > > > > more >> >> > > > > > > > > >> > > > important >> >> > > > > > > > > >> > > > >> >> > things to >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > maybe it is >> better >> >> > to >> >> > > > > > > > > >> re-authenticate. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) >> Do not >> >> > > > > > distribute >> >> > > > > > > > DT >> >> > > > > > > > > to >> >> > > > > > > > > >> > > other >> >> > > > > > > > > >> > > > >> brokers >> >> > > > > > > > > >> > > > >> >> > at >> >> > > > > > > > > >> > > > >> >> > >> >> all. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. Client >> >> > > authenticates >> >> > > > > > with a >> >> > > > > > > > > >> broker. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. Once a >> client is >> >> > > > > > > > authenticated, >> >> > > > > > > > > >> it >> >> > > > > > > > > >> > > will >> >> > > > > > > > > >> > > > >> make a >> >> > > > > > > > > >> > > > >> >> > broker >> >> > > > > > > > > >> > > > >> >> > >> >> side >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > issue a >> delegation >> >> > > > token. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. The broker >> >> > > generates >> >> > > > DT >> >> > > > > > of >> >> > > > > > > > > form, >> >> > > > > > > > > >> > > [hmac + >> >> > > > > > > > > >> > > > >> (owner, >> >> > > > > > > > > >> > > > >> >> > >> >> renewer, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > maxLifeTime, >> id, >> >> > hmac, >> >> > > > > > > > > >> expirationTime)] >> >> > > > > > > > > >> > > and >> >> > > > > > > > > >> > > > >> passes >> >> > > > > > > > > >> > > > >> >> > back >> >> > > > > > > > > >> > > > >> >> > >> >> this >> >> > > > > > > > > >> > > > >> >> > >> >> > > > DT to >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > hmac is >> generated >> >> > via >> >> > > > > > > > > >> {HMAC-SHA256(owner, >> >> > > > > > > > > >> > > > >> renewer, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> expirationTime) >> >> > using >> >> > > > > > > > SecretKey}. >> >> > > > > > > > > >> Note >> >> > > > > > > > > >> > > that >> >> > > > > > > > > >> > > > >> all >> >> > > > > > > > > >> > > > >> >> > brokers >> >> > > > > > > > > >> > > > >> >> > >> >> have >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 4. Client >> then goes >> >> > to >> >> > > > any >> >> > > > > > > > broker >> >> > > > > > > > > >> and to >> >> > > > > > > > > >> > > > >> >> > authenticate >> >> > > > > > > > > >> > > > >> >> > >> >> sends >> >> > > > > > > > > >> > > > >> >> > >> >> > > > the DT. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Broker >> recalculates >> >> > > hmac >> >> > > > > > using >> >> > > > > > > > > >> (owner, >> >> > > > > > > > > >> > > > >> renewer, >> >> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> expirationTime) info >> >> > > > from >> >> > > > > DT >> >> > > > > > > and >> >> > > > > > > > > its >> >> > > > > > > > > >> > > > >> SecretKey. If >> >> > > > > > > > > >> > > > >> >> > it >> >> > > > > > > > > >> > > > >> >> > >> >> matches >> >> > > > > > > > > >> > > > >> >> > >> >> > > > with >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > DT, client is >> >> > > > > authenticated. >> >> > > > > > > > Yes, >> >> > > > > > > > > >> it will >> >> > > > > > > > > >> > > > do >> >> > > > > > > > > >> > > > >> other >> >> > > > > > > > > >> > > > >> >> > >> >> obvious >> >> > > > > > > > > >> > > > >> >> > >> >> > > > checks of >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > timestamp >> expiry and >> >> > > > such. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret >> key >> >> > will >> >> > > > be >> >> > > > > > > > > generated >> >> > > > > > > > > >> by >> >> > > > > > > > > >> > > > >> controller >> >> > > > > > > > > >> > > > >> >> > and >> >> > > > > > > > > >> > > > >> >> > >> >> passed >> >> > > > > > > > > >> > > > >> >> > >> >> > > to >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues >> and >> >> > fixes >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. How to >> delete a >> >> > DT? >> >> > > > > Yes, >> >> > > > > > > that >> >> > > > > > > > > is >> >> > > > > > > > > >> a >> >> > > > > > > > > >> > > > downside >> >> > > > > > > > > >> > > > >> >> > here. >> >> > > > > > > > > >> > > > >> >> > >> >> However, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this can >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > be handled >> with >> >> > > brokers >> >> > > > > > > > > maintaining >> >> > > > > > > > > >> a >> >> > > > > > > > > >> > > > >> blacklist of >> >> > > > > > > > > >> > > > >> >> > DTs, >> >> > > > > > > > > >> > > > >> >> > >> >> DTs >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from this >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > can be >> removed after >> >> > > > > expiry. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. In >> catastrophic >> >> > > > > failures >> >> > > > > > > > where >> >> > > > > > > > > >> all >> >> > > > > > > > > >> > > > brokers >> >> > > > > > > > > >> > > > >> go >> >> > > > > > > > > >> > > > >> >> > down, >> >> > > > > > > > > >> > > > >> >> > >> >> the >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > be lost even >> if >> >> > > servers >> >> > > > > are >> >> > > > > > > > > >> restarted as >> >> > > > > > > > > >> > > > >> tokens >> >> > > > > > > > > >> > > > >> >> > are not >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > If this >> happens, >> >> > then >> >> > > > > there >> >> > > > > > > are >> >> > > > > > > > > more >> >> > > > > > > > > >> > > > important >> >> > > > > > > > > >> > > > >> >> > things to >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > maybe it is >> better >> >> > to >> >> > > > > > > > > >> re-authenticate. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, >> 2016 at >> >> > > > 1:58 >> >> > > > > > PM, >> >> > > > > > > > > Parth >> >> > > > > > > > > >> > > > >> Brahmbhatt < >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > pbrahmbh...@hortonworks.com> >> >> > > > > > > > wrote: >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed >> KIP-48 so >> >> > > we >> >> > > > > can >> >> > > > > > > > offer >> >> > > > > > > > > >> hadoop >> >> > > > > > > > > >> > > > like >> >> > > > > > > > > >> > > > >> >> > delegation >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can >> review >> >> > > the >> >> > > > > > design >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > . >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP >> depends on >> >> > > KIP-43 >> >> > > > > and >> >> > > > > > > we >> >> > > > > > > > > >> have also >> >> > > > > > > > > >> > > > >> >> > discussed an >> >> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design >> here< >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800 >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >. >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > -- >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards, >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > -- >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Regards, >> >> > > > > > > > > >> > > > >> >> > >> >> > > Ashish >> >> > > > > > > > > >> > > > >> >> > >> >> > > >> >> > > > > > > > > >> > > > >> >> > >> >> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > >> >> > > > > > > > > >> >> >> > > > > > > > > >> >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > -- >> >> > > > > > Regards, >> >> > > > > > >> >> > > > > > Rajini >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > -- >> >> > > > Regards, >> >> > > > >> >> > > > Rajini >> >> > > > >> >> > > >> >> > >> >> >> >> >> >> >> >> -- >> >> Liquan Pei >> >> Software Engineer, Confluent Inc >> > >
