Parth,
Overall current design looks good to me. I am +1 on the KIP.
Gwen , Jun can you review this as well.
-Harsha
On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> Thanks for review Jitendra.
>
> I don't like the idea of infinite lifetime but I see the Streaming use
> case. Even for Streaming use case I was hoping there will be some notion
> of
> master/driver that can get new delegation tokens at fixed interval and
> distribute to workers. If that is not the case for we can discuss
> delegation tokens renewing them self and the security implications of the
> same.
>
> I did not want clients to fetch tokens from zookeeper, overall I think
> its
> better if clients don't rely on our metadata store and I think we are
> moving in that direction with all the KIP-4 improvements. I chose
> zookeeper as in this case the client will still just talk to broker , its
> the brokers that will use zookeeper which we already do for a lot of
> other
> usecases + ease of development + and the ability so tokens will survive
> even a rolling restart/cluster failure. if a majority agrees the added
> complexity to have controller forwarding keys to all broker is justified
> as
> it provides tighter security , I am fine with that option too.
>
> Given zookeeper does not support SSL we can not store master keys in
> zookeeper as master keys will be exposed on wire. To support rotation
> without affecting current clients is something I need to put more thought
> in. My current proposal assumes the rotation will invalidate all current
> tokens.
>
> I request committers to also review and post their comments so we can
> make
> progress on this KIP.
>
> Thanks
> Parth
>
> On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <asi...@cloudera.com>
> wrote:
>
> > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io> wrote:
> >
> > > Unifying the two discussion threads on this KIP.
> > >
> > > Here is the response from Jitendra
> > >
> > > "The need for a large number of clients that are running all over the
> > > cluster that authenticate with Kafka brokers, is very similar to the
> > > Hadoop use case of large number of tasks running across the cluster that
> > > need authentication to Hdfs Namenode. Therefore, the delegation token
> > > approach does seem like a good fit for this use case as we have seen it
> > > working at large scale in HDFS and YARN.
> > >
> > > The proposed design is very much inline with Hadoop approach. A few
> > > comments:
> > >
> > > 1) Why do you guys want to allow infinite renewable lifetime for a
> > > token? HDFS restricts a token to a max life time (default 7 days). A
> > > token's vulnerability is believed to increase with time.
> > >
> > I agree that having infinite lifetime might not be the best idea.
> >
> > >
> > > 2) As I understand the tokens are stored in zookeeper as well, and can
> > > be updated there. This is clever as it can allow replacing the tokens
> > > once they run out of max life time, and clients can download new tokens
> > > from zookeeper. It shouldn't be a big load on zookeeper as a client will
> > > need to get a new token once in several days. In this approach you don't
> > > need infinite lifetime on the token even for long running clients.
> > >
> > > 3) The token password are generated using a master key. The master key
> > > should also be periodically changed. In Hadoop, the default renewal
> > > period is 1 day.?
> > >
> > IIUC, this will require brokers maintaining a list of X most recent master
> > keys. This list will have to be persisted somewhere, as if a broker goes
> > down it will have to get that list again and storing master keys on ZK is
> > not the best idea. However, if a broker goes down then we have much bigger
> > issue to deal with and client can always re-authenticate is such events.
> >
> > Did you happen to take a look at other alternatives this list has
> > suggested?
> >
> > >
> > > Thanks for a thorough proposal, great work!"
> > >
> > >
> > >
> > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> > > > Makes sense to me. Thanks!
> > > >
> > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io> wrote:
> > > > > It doesn't need any release vehicle but still the work can move
> > > forward.
> > > > > If anyone is interested in the KIP please do the review and provide
> > the
> > > > > comments.
> > > > >
> > > > > -Harsha
> > > > >
> > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
> > > > >> I agree that it would be good to have more time to review and
> > discuss
> > > > >> KIP-48.
> > > > >>
> > > > >> Ismael
> > > > >>
> > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <g...@confluent.io>
> > > wrote:
> > > > >>
> > > > >> > Hi Team,
> > > > >> >
> > > > >> > Since KIP-48 depends on KIP-43, which is already a bit of a risk
> > for
> > > > >> > the next release - any chance we can delay delegation tokens to
> > > Kafka
> > > > >> > 0.10.1?
> > > > >> > With the community working on a release every 3 month, this is not
> > > a huge
> > > > >> > delay.
> > > > >> >
> > > > >> > Gwen
> > > > >> >
> > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
> > asi...@cloudera.com>
> > > wrote:
> > > > >> > > Parth,
> > > > >> > >
> > > > >> > > Thanks again for the awesome write up. Following our discussion
> > > from the
> > > > >> > > JIRA, I think it will be easier to compare various alternatives
> > > if they
> > > > >> > are
> > > > >> > > listed together. I am stating below a few alternatives along
> > with
> > > a the
> > > > >> > > current proposal.
> > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
> > > > >> > >
> > > > >> > > 1. Client authenticates with a broker.
> > > > >> > > 2. Once a client is authenticated, it will make a broker side
> > > call to
> > > > >> > > issue a delegation token.
> > > > >> > > 3. The broker generates a shared secret based on
> > HMAC-SHA256(a
> > > > >> > > Password/Secret shared between all brokers, randomly
> > generated
> > > > >> > tokenId).
> > > > >> > > 4. Broker stores this token in its in memory cache. Broker
> > > also stores
> > > > >> > > the DelegationToken without the hmac in the zookeeper.
> > > > >> > > 5. All brokers will have a cache backed by zookeeper so they
> > > will all
> > > > >> > > get notified whenever a new token is generated and they will
> > > update
> > > > >> > their
> > > > >> > > local cache whenever token state changes.
> > > > >> > > 6. Broker returns the token to Client.
> > > > >> > >
> > > > >> > > Probable issues and fixes
> > > > >> > >
> > > > >> > > 1. Probable race condition, client tries to authenticate with
> > > a broker
> > > > >> > > that is yet to be updated with the newly generated DT. This
> > can
> > > > >> > probably be
> > > > >> > > dealt with making dtRequest block until all brokers have
> > > updated
> > > > >> > their DT
> > > > >> > > cache. Zk barrier or similar mechanism can be used. However,
> > > all such
> > > > >> > > mechanisms will increase complexity.
> > > > >> > > 2. Using a static secret key from config file. Will require
> > yet
> > > > >> > another
> > > > >> > > config and uses a static secret key. It is advised to rotate
> > > secret
> > > > >> > keys
> > > > >> > > periodically. This can be avoided with controller generating
> > > > >> > secretKey and
> > > > >> > > passing to brokers periodically. However, this will require
> > > brokers to
> > > > >> > > maintain certain counts of secretKeys.
> > > > >> > >
> > > > >> > > (Alternative 1) Have controller generate delegation token.
> > > > >> > >
> > > > >> > > 1. Client authenticates with a broker.
> > > > >> > > 2. Once a client is authenticated, it will make a broker side
> > > call to
> > > > >> > > issue a delegation token.
> > > > >> > > 3. Broker forwards the request to controller.
> > > > >> > > 4. Controller generates a DT and broadcasts to all brokers.
> > > > >> > > 5. Broker stores this token in its memory cache.
> > > > >> > > 6. Controller responds to broker’s DT req.
> > > > >> > > 7. Broker returns the token to Client.
> > > > >> > >
> > > > >> > > Probable issues and fixes
> > > > >> > >
> > > > >> > > 1. We will have to add new APIs to support controller pushing
> > > tokens
> > > > >> > to
> > > > >> > > brokers on top of the minimal APIs that are currently
> > proposed.
> > > > >> > > 2. We will also have to add APIs to support the bootstrapping
> > > case,
> > > > >> > i.e,
> > > > >> > > when a new broker comes up it will have to get all delegation
> > > tokens
> > > > >> > from
> > > > >> > > the controller.
> > > > >> > > 3. In catastrophic failures where all brokers go down, the
> > > tokens will
> > > > >> > > be lost even if servers are restarted as tokens are not
> > > persisted
> > > > >> > anywhere.
> > > > >> > > If this happens, then there are more important things to
> > worry
> > > about
> > > > >> > and
> > > > >> > > maybe it is better to re-authenticate.
> > > > >> > >
> > > > >> > > (Alternative 2) Do not distribute DT to other brokers at all.
> > > > >> > >
> > > > >> > > 1. Client authenticates with a broker.
> > > > >> > > 2. Once a client is authenticated, it will make a broker side
> > > call to
> > > > >> > > issue a delegation token.
> > > > >> > > 3. The broker generates DT of form, [hmac + (owner, renewer,
> > > > >> > > maxLifeTime, id, hmac, expirationTime)] and passes back this
> > > DT to
> > > > >> > client.
> > > > >> > > hmac is generated via {HMAC-SHA256(owner, renewer,
> > > maxLifeTime, id,
> > > > >> > hmac,
> > > > >> > > expirationTime) using SecretKey}. Note that all brokers have
> > > this
> > > > >> > SecretKey.
> > > > >> > > 4. Client then goes to any broker and to authenticate sends
> > > the DT.
> > > > >> > > Broker recalculates hmac using (owner, renewer, maxLifeTime,
> > > id, hmac,
> > > > >> > > expirationTime) info from DT and its SecretKey. If it matches
> > > with
> > > > >> > hmac of
> > > > >> > > DT, client is authenticated. Yes, it will do other obvious
> > > checks of
> > > > >> > > timestamp expiry and such.
> > > > >> > >
> > > > >> > > Note that secret key will be generated by controller and passed
> > to
> > > > >> > brokers
> > > > >> > > periodically.
> > > > >> > > Probable issues and fixes
> > > > >> > >
> > > > >> > > 1. How to delete a DT? Yes, that is a downside here. However,
> > > this can
> > > > >> > > be handled with brokers maintaining a blacklist of DTs, DTs
> > > from this
> > > > >> > list
> > > > >> > > can be removed after expiry.
> > > > >> > > 2. In catastrophic failures where all brokers go down, the
> > > tokens will
> > > > >> > > be lost even if servers are restarted as tokens are not
> > > persisted
> > > > >> > anywhere.
> > > > >> > > If this happens, then there are more important things to
> > worry
> > > about
> > > > >> > and
> > > > >> > > maybe it is better to re-authenticate.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> > > > >> > > pbrahmbh...@hortonworks.com> wrote:
> > > > >> > >
> > > > >> > >> Hi,
> > > > >> > >>
> > > > >> > >> I have filed KIP-48 so we can offer hadoop like delegation
> > > tokens in
> > > > >> > >> kafka. You can review the design
> > > > >> > >>
> > > > >> >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > >> > .
> > > > >> > >> This KIP depends on KIP-43 and we have also discussed an
> > > alternative to
> > > > >> > >> proposed design here<
> > > > >> > >>
> > > > >> >
> > >
> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > >> > >> >.
> > > > >> > >>
> > > > >> > >> Thanks
> > > > >> > >> Parth
> > > > >> > >>
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > >
> > > > >> > > Regards,
> > > > >> > > Ashish
> > > > >> >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
