Also comments inline :) > * I want to emphasize that even though delegation tokens are a Hadoop > innovation, I feel very strongly about not adding dependency on Hadoop > when implementing delegation tokens for Kafka. The KIP doesn't imply > such dependency, but if you can clarify... > > > *No hadoop dependency.*
Yay! Just add this to the KIP so no one will read the KIP and panic three weeks before the next release... > * Can we get delegation token at any time after authenticating? only > immediately after? > > > *As long as you are authenticated you can get delegation tokens. We need to > discuss if a client authenticated using delegation token, can also acquire > delegation token again or not. Also there is the question of do we allow > anyone to acquire delegation token or we want specific ACLs (I think its an > overkill.)* I agree that ACLs is an overkill. I think we are debating two options: Either require Kerberos auth for renewal or require non-owners to renew. I *think* the latter is simpler (it basically require a "job master" to take responsibility for the renewal, it will have its own identity anyway and I think this is the correct design pattern anyway. For storm, I'd expect Nimbus to coordinate renewals?), but it is hard to debate simplicity without looking at the code changes required. If you have a draft of how the "require Kerberos" will look in Kafka code, I'll be happy to take a look. > * My understanding is that tokens will propagate via ZK but without > additional changes to UpdateMetadata protocol, correct? Clients > currently don't retry on SASL auth failure (IIRC), but since the > tokens propagate between brokers asynch, we will need to retry a bit > to avoid clients failing auth due to timing issues. > > *I am considering 2 alternatives right now. The current documented approach > is zookeeper based and it does not require any changes to UpdateMetadata > protocol. An alternative approach can remove zookeeper dependency as well > but we can discuss that in KIP discussion call.* Oooh! Sounds interesting. Do you want to ping Jun to arrange a call? > * I liked Ashish's suggestion of having just the controller issue the > delegation tokens, to avoid syncing a shared secret. Not sure if we > want to continue the discussion here or on the wiki. I think that we > can decouple the problem of "token distribution" from "shared secret > distribution" and use the controller as the only token generator to > solve the second issue, while still using ZK async to distribute > tokens. > > > *As mentioned in the previous Email I am fine with that approach as long as > we agree that the extra complexity of adding/updating APIS adds enough > value. The advantage with the controller approach is secret rotation can be > automated,frequent and would not require deployment. * Can you detail the extra complexity (or point me to the email I missed?) - which Apis are required? As far as I can tell, clients can already find the controller from metadata. I'm a bit more concerned about controller load. > > * While I like the idea of forcing kerberos auth for renwal, I think > it mixes the transport layer the the request content in a pretty ugly > way. Perhaps limiting renewer to non-owner is better. > > *I feel this is a necessary evil. While this will make the kafka code > pretty straight forward , forcing renewer to non-owner pushes the code > ugliness to client and makes it even harder to integrate. * As mentioned before, I don't think the "renewal by other" approach is that ugly for the clients we expect to use delegation tokens since they will have an app-master of some sort who requested the token to begin with. > > The response for my question on how multiple identities will be > handled wasn't super clear to me - AFAIK, we don't authenticate each > request, we authenticate connections. > > *We authenticate connections, and only when they are being established. Let > me try to phrase this as a question, in absence of delegation tokens if we > had to support the use case using user TGT's how would we do it? My point > was it would be no different with delegation tokens. The use case you are > describing seems more like impersonation.* Yeah, I thought that one of the things that delegation tokens handled. Maybe I got it wrong :) Thanks for the detailed answers. Gwen > Thanks > Parth > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <g...@confluent.io> wrote: > >> Hi Parth and Harsha, >> >> Few more comments: >> >> * The API / RequestResponse section doesn't seem to have good >> description of the changes to the Kafka Protocol. Sounds like you are >> proposing new DelegationTokenRequest and RenewTokenRequest (and >> matching responses), without detailing the contents of the requests >> and responses? Or rather, you show the class interface, but not the >> underlying protocol. This could be seen as an implementation detail, >> but since the binary protocol is what we provide to non-Java clients, >> we need to show the changes there. >> >> * getDelegationToken sounds like delegationTokenRequestHandler? Is it >> planned to be part of KafkaApi? or Client? Its unclear... >> >> * I want to emphasize that even though delegation tokens are a Hadoop >> innovation, I feel very strongly about not adding dependency on Hadoop >> when implementing delegation tokens for Kafka. The KIP doesn't imply >> such dependency, but if you can clarify... >> >> * Can we get delegation token at any time after authenticating? only >> immediately after? >> >> * My understanding is that tokens will propagate via ZK but without >> additional changes to UpdateMetadata protocol, correct? Clients >> currently don't retry on SASL auth failure (IIRC), but since the >> tokens propagate between brokers asynch, we will need to retry a bit >> to avoid clients failing auth due to timing issues. >> >> * Strongly agreeing on clients not touching ZK directly :) >> >> * I liked Ashish's suggestion of having just the controller issue the >> delegation tokens, to avoid syncing a shared secret. Not sure if we >> want to continue the discussion here or on the wiki. I think that we >> can decouple the problem of "token distribution" from "shared secret >> distribution" and use the controller as the only token generator to >> solve the second issue, while still using ZK async to distribute >> tokens. >> >> * I am also uncomfortable with infinite lifetime of tokens (and hoped >> to hear from others in the community) - but having the option and >> documenting it as less secure, allows users to configure their system >> as the wish. >> >> * While I like the idea of forcing kerberos auth for renwal, I think >> it mixes the transport layer the the request content in a pretty ugly >> way. Perhaps limiting renewer to non-owner is better. >> >> Things I'd still like to see: >> >> * More detailed explanation on what we plan to do for the java clients >> specifically - new configuration? new APIs? >> The response for my question on how multiple identities will be >> handled wasn't super clear to me - AFAIK, we don't authenticate each >> request, we authenticate connections. >> >> * Alternatives: Delegation tokens are only used in the Hadoop >> ecosystem. I'm wondering if there are alternatives in other ecosystems >> (Mesos? Tachyon? Cassandra?) and whether there are some advantages >> there. >> >> Gwen >> >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> wrote: >> > Hi Gwen, >> > Can you look at Parth's last reply. Does it answer your >> > concerns. >> > >> > Thanks, >> > Harsha >> > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote: >> >> Thanks for reviewing Gwen. The wiki already has details on token >> >> expiration >> >> under token acquisition process >> >> < >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition >> >. >> >> Current proposal is that tokens will expire based on a server side >> >> configuration (default 24 hours) unless renewed. Renewal is only allowed >> >> until the max life time of token. Alternatively we could also make that >> >> an >> >> optional param and the server side default can serve as the upper bound. >> >> >> >> To your second point it will be done exactly the same way we would >> >> support >> >> multiple keytabs. The calling client will have to put the tokens it >> wants >> >> to use in the subject instance and call produce/consume inside >> >> subject.doas. Each caller will have to keep track of its own subject. I >> >> will have to look at the code to see if we support this feature right >> now >> >> but my understanding is delegation token shouldn't need any special >> >> treatment as its just another type of Credential in the subject. >> >> >> >> I would also like to know what is your opinion about infinite renewal >> (my >> >> recommendation is to not support this), tokens renewing them self(my >> >> recommendation is to not support this) and most importantly your choice >> >> between the alternatives listed on this thread >> >> < >> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism >> > >> >> ( I am leaning towards the alternative-2 minus controller distributing >> >> secret). Thanks again for reviewing. >> >> >> >> Thanks >> >> Parth >> >> >> >> >> >> >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <g...@confluent.io> wrote: >> >> >> >> > Harsha, >> >> > >> >> > I was thinking of the Rest Proxy. I didn't see your design yet, but in >> >> > our proxy, we have a set of producers, which will serve multiple users >> >> > going through the proxy. Since these users will have different >> >> > privileges, they'll need to authenticate separately, and can't share a >> >> > token. >> >> > >> >> > Am I missing anything? >> >> > >> >> > Gwen >> >> > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote: >> >> > > Gwen, >> >> > > On your second point. Can you describe a usecase where >> >> > > mutliple clients ended up sharing a producer and even if >> they >> >> > > do why can't they not use single token that producer >> >> > > captures. Why would we need multiple clients with >> different >> >> > > tokens sharing a single instance of producer. Also in >> this >> >> > > case other clients have access all the tokens no? >> >> > > >> >> > > Thanks, >> >> > > Harsha >> >> > > >> >> > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote: >> >> > >> Sorry for the delay: >> >> > >> >> >> > >> Two questions that we didn't see in the wiki: >> >> > >> 1. Is there an expiration for delegation tokens? Renewal? How do we >> >> > >> revoke them? >> >> > >> 2. If we want to use delegation tokens for "do-as" (say, submit >> Storm >> >> > >> job as my user), we will need a producer for every job (we can't >> share >> >> > >> them between multiple jobs running on same node), since we only >> >> > >> authenticate when connecting. Is there a plan to change this for >> >> > >> delegation tokens, in order to allow multiple users with different >> >> > >> tokens to share a client? >> >> > >> >> >> > >> Gwen >> >> > >> >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt >> >> > >> <brahmbhatt.pa...@gmail.com> wrote: >> >> > >> > Bumping this up one more time, can other committers review? >> >> > >> > >> >> > >> > Thanks >> >> > >> > Parth >> >> > >> > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote: >> >> > >> > >> >> > >> >> Parth, >> >> > >> >> Overall current design looks good to me. I am +1 on >> the >> >> > KIP. >> >> > >> >> >> >> > >> >> Gwen , Jun can you review this as well. >> >> > >> >> >> >> > >> >> -Harsha >> >> > >> >> >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote: >> >> > >> >> > Thanks for review Jitendra. >> >> > >> >> > >> >> > >> >> > I don't like the idea of infinite lifetime but I see the >> Streaming >> >> > use >> >> > >> >> > case. Even for Streaming use case I was hoping there will be >> some >> >> > notion >> >> > >> >> > of >> >> > >> >> > master/driver that can get new delegation tokens at fixed >> interval >> >> > and >> >> > >> >> > distribute to workers. If that is not the case for we can >> discuss >> >> > >> >> > delegation tokens renewing them self and the security >> implications >> >> > of the >> >> > >> >> > same. >> >> > >> >> > >> >> > >> >> > I did not want clients to fetch tokens from zookeeper, >> overall I >> >> > think >> >> > >> >> > its >> >> > >> >> > better if clients don't rely on our metadata store and I >> think we >> >> > are >> >> > >> >> > moving in that direction with all the KIP-4 improvements. I >> chose >> >> > >> >> > zookeeper as in this case the client will still just talk to >> >> > broker , its >> >> > >> >> > the brokers that will use zookeeper which we already do for a >> lot >> >> > of >> >> > >> >> > other >> >> > >> >> > usecases + ease of development + and the ability so tokens >> will >> >> > survive >> >> > >> >> > even a rolling restart/cluster failure. if a majority agrees >> the >> >> > added >> >> > >> >> > complexity to have controller forwarding keys to all broker is >> >> > justified >> >> > >> >> > as >> >> > >> >> > it provides tighter security , I am fine with that option too. >> >> > >> >> > >> >> > >> >> > Given zookeeper does not support SSL we can not store master >> keys >> >> > in >> >> > >> >> > zookeeper as master keys will be exposed on wire. To support >> >> > rotation >> >> > >> >> > without affecting current clients is something I need to put >> more >> >> > thought >> >> > >> >> > in. My current proposal assumes the rotation will invalidate >> all >> >> > current >> >> > >> >> > tokens. >> >> > >> >> > >> >> > >> >> > I request committers to also review and post their comments >> so we >> >> > can >> >> > >> >> > make >> >> > >> >> > progress on this KIP. >> >> > >> >> > >> >> > >> >> > Thanks >> >> > >> >> > Parth >> >> > >> >> > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh < >> asi...@cloudera.com >> >> > > >> >> > >> >> > wrote: >> >> > >> >> > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io> >> >> > wrote: >> >> > >> >> > > >> >> > >> >> > > > Unifying the two discussion threads on this KIP. >> >> > >> >> > > > >> >> > >> >> > > > Here is the response from Jitendra >> >> > >> >> > > > >> >> > >> >> > > > "The need for a large number of clients that are running >> all >> >> > over the >> >> > >> >> > > > cluster that authenticate with Kafka brokers, is very >> similar >> >> > to the >> >> > >> >> > > > Hadoop use case of large number of tasks running across >> the >> >> > cluster >> >> > >> >> that >> >> > >> >> > > > need authentication to Hdfs Namenode. Therefore, the >> >> > delegation token >> >> > >> >> > > > approach does seem like a good fit for this use case as we >> >> > have seen >> >> > >> >> it >> >> > >> >> > > > working at large scale in HDFS and YARN. >> >> > >> >> > > > >> >> > >> >> > > > The proposed design is very much inline with Hadoop >> >> > approach. A few >> >> > >> >> > > > comments: >> >> > >> >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow infinite renewable >> lifetime >> >> > for a >> >> > >> >> > > > token? HDFS restricts a token to a max life time (default >> 7 >> >> > days). A >> >> > >> >> > > > token's vulnerability is believed to increase with time. >> >> > >> >> > > > >> >> > >> >> > > I agree that having infinite lifetime might not be the best >> idea. >> >> > >> >> > > >> >> > >> >> > > > >> >> > >> >> > > > 2) As I understand the tokens are stored in zookeeper as >> well, >> >> > and >> >> > >> >> can >> >> > >> >> > > > be updated there. This is clever as it can allow >> replacing the >> >> > tokens >> >> > >> >> > > > once they run out of max life time, and clients can >> download >> >> > new >> >> > >> >> tokens >> >> > >> >> > > > from zookeeper. It shouldn't be a big load on zookeeper >> as a >> >> > client >> >> > >> >> will >> >> > >> >> > > > need to get a new token once in several days. In this >> approach >> >> > you >> >> > >> >> don't >> >> > >> >> > > > need infinite lifetime on the token even for long running >> >> > clients. >> >> > >> >> > > > >> >> > >> >> > > > 3) The token password are generated using a master key. >> The >> >> > master >> >> > >> >> key >> >> > >> >> > > > should also be periodically changed. In Hadoop, the >> default >> >> > renewal >> >> > >> >> > > > period is 1 day.? >> >> > >> >> > > > >> >> > >> >> > > IIUC, this will require brokers maintaining a list of X most >> >> > recent >> >> > >> >> master >> >> > >> >> > > keys. This list will have to be persisted somewhere, as if a >> >> > broker >> >> > >> >> goes >> >> > >> >> > > down it will have to get that list again and storing master >> keys >> >> > on ZK >> >> > >> >> is >> >> > >> >> > > not the best idea. However, if a broker goes down then we >> have >> >> > much >> >> > >> >> bigger >> >> > >> >> > > issue to deal with and client can always re-authenticate is >> such >> >> > >> >> events. >> >> > >> >> > > >> >> > >> >> > > Did you happen to take a look at other alternatives this >> list has >> >> > >> >> > > suggested? >> >> > >> >> > > >> >> > >> >> > > > >> >> > >> >> > > > Thanks for a thorough proposal, great work!" >> >> > >> >> > > > >> >> > >> >> > > > >> >> > >> >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote: >> >> > >> >> > > > > Makes sense to me. Thanks! >> >> > >> >> > > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io >> > >> >> > wrote: >> >> > >> >> > > > > > It doesn't need any release vehicle but still the >> work can >> >> > move >> >> > >> >> > > > forward. >> >> > >> >> > > > > > If anyone is interested in the KIP please do the >> review and >> >> > >> >> provide >> >> > >> >> > > the >> >> > >> >> > > > > > comments. >> >> > >> >> > > > > > >> >> > >> >> > > > > > -Harsha >> >> > >> >> > > > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote: >> >> > >> >> > > > > >> I agree that it would be good to have more time to >> review >> >> > and >> >> > >> >> > > discuss >> >> > >> >> > > > > >> KIP-48. >> >> > >> >> > > > > >> >> >> > >> >> > > > > >> Ismael >> >> > >> >> > > > > >> >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira < >> >> > >> >> g...@confluent.io> >> >> > >> >> > > > wrote: >> >> > >> >> > > > > >> >> >> > >> >> > > > > >> > Hi Team, >> >> > >> >> > > > > >> > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a >> bit >> >> > of a >> >> > >> >> risk >> >> > >> >> > > for >> >> > >> >> > > > > >> > the next release - any chance we can delay >> delegation >> >> > tokens >> >> > >> >> to >> >> > >> >> > > > Kafka >> >> > >> >> > > > > >> > 0.10.1? >> >> > >> >> > > > > >> > With the community working on a release every 3 >> month, >> >> > this >> >> > >> >> is not >> >> > >> >> > > > a huge >> >> > >> >> > > > > >> > delay. >> >> > >> >> > > > > >> > >> >> > >> >> > > > > >> > Gwen >> >> > >> >> > > > > >> > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh < >> >> > >> >> > > asi...@cloudera.com> >> >> > >> >> > > > wrote: >> >> > >> >> > > > > >> > > Parth, >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > Thanks again for the awesome write up. Following >> our >> >> > >> >> discussion >> >> > >> >> > > > from the >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to compare >> various >> >> > >> >> alternatives >> >> > >> >> > > > if they >> >> > >> >> > > > > >> > are >> >> > >> >> > > > > >> > > listed together. I am stating below a few >> >> > alternatives along >> >> > >> >> > > with >> >> > >> >> > > > a the >> >> > >> >> > > > > >> > > current proposal. >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation Token, DT, >> on ZK. >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > 1. Client authenticates with a broker. >> >> > >> >> > > > > >> > > 2. Once a client is authenticated, it will >> make a >> >> > broker >> >> > >> >> side >> >> > >> >> > > > call to >> >> > >> >> > > > > >> > > issue a delegation token. >> >> > >> >> > > > > >> > > 3. The broker generates a shared secret based >> on >> >> > >> >> > > HMAC-SHA256(a >> >> > >> >> > > > > >> > > Password/Secret shared between all brokers, >> >> > randomly >> >> > >> >> > > generated >> >> > >> >> > > > > >> > tokenId). >> >> > >> >> > > > > >> > > 4. Broker stores this token in its in memory >> cache. >> >> > >> >> Broker >> >> > >> >> > > > also stores >> >> > >> >> > > > > >> > > the DelegationToken without the hmac in the >> >> > zookeeper. >> >> > >> >> > > > > >> > > 5. All brokers will have a cache backed by >> >> > zookeeper so >> >> > >> >> they >> >> > >> >> > > > will all >> >> > >> >> > > > > >> > > get notified whenever a new token is >> generated and >> >> > they >> >> > >> >> will >> >> > >> >> > > > update >> >> > >> >> > > > > >> > their >> >> > >> >> > > > > >> > > local cache whenever token state changes. >> >> > >> >> > > > > >> > > 6. Broker returns the token to Client. >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > Probable issues and fixes >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > 1. Probable race condition, client tries to >> >> > authenticate >> >> > >> >> with >> >> > >> >> > > > a broker >> >> > >> >> > > > > >> > > that is yet to be updated with the newly >> generated >> >> > DT. >> >> > >> >> This >> >> > >> >> > > can >> >> > >> >> > > > > >> > probably be >> >> > >> >> > > > > >> > > dealt with making dtRequest block until all >> >> > brokers have >> >> > >> >> > > > updated >> >> > >> >> > > > > >> > their DT >> >> > >> >> > > > > >> > > cache. Zk barrier or similar mechanism can be >> used. >> >> > >> >> However, >> >> > >> >> > > > all such >> >> > >> >> > > > > >> > > mechanisms will increase complexity. >> >> > >> >> > > > > >> > > 2. Using a static secret key from config >> file. Will >> >> > >> >> require >> >> > >> >> > > yet >> >> > >> >> > > > > >> > another >> >> > >> >> > > > > >> > > config and uses a static secret key. It is >> advised >> >> > to >> >> > >> >> rotate >> >> > >> >> > > > secret >> >> > >> >> > > > > >> > keys >> >> > >> >> > > > > >> > > periodically. This can be avoided with >> controller >> >> > >> >> generating >> >> > >> >> > > > > >> > secretKey and >> >> > >> >> > > > > >> > > passing to brokers periodically. However, >> this will >> >> > >> >> require >> >> > >> >> > > > brokers to >> >> > >> >> > > > > >> > > maintain certain counts of secretKeys. >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller generate >> delegation >> >> > token. >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > 1. Client authenticates with a broker. >> >> > >> >> > > > > >> > > 2. Once a client is authenticated, it will >> make a >> >> > broker >> >> > >> >> side >> >> > >> >> > > > call to >> >> > >> >> > > > > >> > > issue a delegation token. >> >> > >> >> > > > > >> > > 3. Broker forwards the request to controller. >> >> > >> >> > > > > >> > > 4. Controller generates a DT and broadcasts >> to all >> >> > >> >> brokers. >> >> > >> >> > > > > >> > > 5. Broker stores this token in its memory >> cache. >> >> > >> >> > > > > >> > > 6. Controller responds to broker’s DT req. >> >> > >> >> > > > > >> > > 7. Broker returns the token to Client. >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > Probable issues and fixes >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > 1. We will have to add new APIs to support >> >> > controller >> >> > >> >> pushing >> >> > >> >> > > > tokens >> >> > >> >> > > > > >> > to >> >> > >> >> > > > > >> > > brokers on top of the minimal APIs that are >> >> > currently >> >> > >> >> > > proposed. >> >> > >> >> > > > > >> > > 2. We will also have to add APIs to support >> the >> >> > >> >> bootstrapping >> >> > >> >> > > > case, >> >> > >> >> > > > > >> > i.e, >> >> > >> >> > > > > >> > > when a new broker comes up it will have to >> get all >> >> > >> >> delegation >> >> > >> >> > > > tokens >> >> > >> >> > > > > >> > from >> >> > >> >> > > > > >> > > the controller. >> >> > >> >> > > > > >> > > 3. In catastrophic failures where all brokers >> go >> >> > down, >> >> > >> >> the >> >> > >> >> > > > tokens will >> >> > >> >> > > > > >> > > be lost even if servers are restarted as >> tokens >> >> > are not >> >> > >> >> > > > persisted >> >> > >> >> > > > > >> > anywhere. >> >> > >> >> > > > > >> > > If this happens, then there are more important >> >> > things to >> >> > >> >> > > worry >> >> > >> >> > > > about >> >> > >> >> > > > > >> > and >> >> > >> >> > > > > >> > > maybe it is better to re-authenticate. >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to other >> brokers >> >> > at >> >> > >> >> all. >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > 1. Client authenticates with a broker. >> >> > >> >> > > > > >> > > 2. Once a client is authenticated, it will >> make a >> >> > broker >> >> > >> >> side >> >> > >> >> > > > call to >> >> > >> >> > > > > >> > > issue a delegation token. >> >> > >> >> > > > > >> > > 3. The broker generates DT of form, [hmac + >> (owner, >> >> > >> >> renewer, >> >> > >> >> > > > > >> > > maxLifeTime, id, hmac, expirationTime)] and >> passes >> >> > back >> >> > >> >> this >> >> > >> >> > > > DT to >> >> > >> >> > > > > >> > client. >> >> > >> >> > > > > >> > > hmac is generated via {HMAC-SHA256(owner, >> renewer, >> >> > >> >> > > > maxLifeTime, id, >> >> > >> >> > > > > >> > hmac, >> >> > >> >> > > > > >> > > expirationTime) using SecretKey}. Note that >> all >> >> > brokers >> >> > >> >> have >> >> > >> >> > > > this >> >> > >> >> > > > > >> > SecretKey. >> >> > >> >> > > > > >> > > 4. Client then goes to any broker and to >> >> > authenticate >> >> > >> >> sends >> >> > >> >> > > > the DT. >> >> > >> >> > > > > >> > > Broker recalculates hmac using (owner, >> renewer, >> >> > >> >> maxLifeTime, >> >> > >> >> > > > id, hmac, >> >> > >> >> > > > > >> > > expirationTime) info from DT and its >> SecretKey. If >> >> > it >> >> > >> >> matches >> >> > >> >> > > > with >> >> > >> >> > > > > >> > hmac of >> >> > >> >> > > > > >> > > DT, client is authenticated. Yes, it will do >> other >> >> > >> >> obvious >> >> > >> >> > > > checks of >> >> > >> >> > > > > >> > > timestamp expiry and such. >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > Note that secret key will be generated by >> controller >> >> > and >> >> > >> >> passed >> >> > >> >> > > to >> >> > >> >> > > > > >> > brokers >> >> > >> >> > > > > >> > > periodically. >> >> > >> >> > > > > >> > > Probable issues and fixes >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > 1. How to delete a DT? Yes, that is a downside >> >> > here. >> >> > >> >> However, >> >> > >> >> > > > this can >> >> > >> >> > > > > >> > > be handled with brokers maintaining a >> blacklist of >> >> > DTs, >> >> > >> >> DTs >> >> > >> >> > > > from this >> >> > >> >> > > > > >> > list >> >> > >> >> > > > > >> > > can be removed after expiry. >> >> > >> >> > > > > >> > > 2. In catastrophic failures where all brokers >> go >> >> > down, >> >> > >> >> the >> >> > >> >> > > > tokens will >> >> > >> >> > > > > >> > > be lost even if servers are restarted as >> tokens >> >> > are not >> >> > >> >> > > > persisted >> >> > >> >> > > > > >> > anywhere. >> >> > >> >> > > > > >> > > If this happens, then there are more important >> >> > things to >> >> > >> >> > > worry >> >> > >> >> > > > about >> >> > >> >> > > > > >> > and >> >> > >> >> > > > > >> > > maybe it is better to re-authenticate. >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth >> Brahmbhatt < >> >> > >> >> > > > > >> > > pbrahmbh...@hortonworks.com> wrote: >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > >> Hi, >> >> > >> >> > > > > >> > >> >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like >> >> > delegation >> >> > >> >> > > > tokens in >> >> > >> >> > > > > >> > >> kafka. You can review the design >> >> > >> >> > > > > >> > >> >> >> > >> >> > > > > >> > >> >> > >> >> > > > >> >> > >> >> > > >> >> > >> >> >> >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka >> >> > >> >> > > > > >> > . >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also >> >> > discussed an >> >> > >> >> > > > alternative to >> >> > >> >> > > > > >> > >> proposed design here< >> >> > >> >> > > > > >> > >> >> >> > >> >> > > > > >> > >> >> > >> >> > > > >> >> > >> >> > > >> >> > >> >> >> >> > >> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800 >> >> > >> >> > > > > >> > >> >. >> >> > >> >> > > > > >> > >> >> >> > >> >> > > > > >> > >> Thanks >> >> > >> >> > > > > >> > >> Parth >> >> > >> >> > > > > >> > >> >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > -- >> >> > >> >> > > > > >> > > >> >> > >> >> > > > > >> > > Regards, >> >> > >> >> > > > > >> > > Ashish >> >> > >> >> > > > > >> > >> >> > >> >> > > > >> >> > >> >> > > >> >> > >> >> > > >> >> > >> >> > > >> >> > >> >> > > -- >> >> > >> >> > > >> >> > >> >> > > Regards, >> >> > >> >> > > Ashish >> >> > >> >> > > >> >> > >> >> >> >> > >>
