Hi Gwen,
Can you look at Parth's last reply. Does it answer your
concerns.
Thanks,
Harsha
On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> Thanks for reviewing Gwen. The wiki already has details on token
> expiration
> under token acquisition process
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition>.
> Current proposal is that tokens will expire based on a server side
> configuration (default 24 hours) unless renewed. Renewal is only allowed
> until the max life time of token. Alternatively we could also make that
> an
> optional param and the server side default can serve as the upper bound.
>
> To your second point it will be done exactly the same way we would
> support
> multiple keytabs. The calling client will have to put the tokens it wants
> to use in the subject instance and call produce/consume inside
> subject.doas. Each caller will have to keep track of its own subject. I
> will have to look at the code to see if we support this feature right now
> but my understanding is delegation token shouldn't need any special
> treatment as its just another type of Credential in the subject.
>
> I would also like to know what is your opinion about infinite renewal (my
> recommendation is to not support this), tokens renewing them self(my
> recommendation is to not support this) and most importantly your choice
> between the alternatives listed on this thread
> <http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism>
> ( I am leaning towards the alternative-2 minus controller distributing
> secret). Thanks again for reviewing.
>
> Thanks
> Parth
>
>
>
> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <g...@confluent.io> wrote:
>
> > Harsha,
> >
> > I was thinking of the Rest Proxy. I didn't see your design yet, but in
> > our proxy, we have a set of producers, which will serve multiple users
> > going through the proxy. Since these users will have different
> > privileges, they'll need to authenticate separately, and can't share a
> > token.
> >
> > Am I missing anything?
> >
> > Gwen
> >
> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
> > > Gwen,
> > > On your second point. Can you describe a usecase where
> > > mutliple clients ended up sharing a producer and even if they
> > > do why can't they not use single token that producer
> > > captures. Why would we need multiple clients with different
> > > tokens sharing a single instance of producer. Also in this
> > > case other clients have access all the tokens no?
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> > >> Sorry for the delay:
> > >>
> > >> Two questions that we didn't see in the wiki:
> > >> 1. Is there an expiration for delegation tokens? Renewal? How do we
> > >> revoke them?
> > >> 2. If we want to use delegation tokens for "do-as" (say, submit Storm
> > >> job as my user), we will need a producer for every job (we can't share
> > >> them between multiple jobs running on same node), since we only
> > >> authenticate when connecting. Is there a plan to change this for
> > >> delegation tokens, in order to allow multiple users with different
> > >> tokens to share a client?
> > >>
> > >> Gwen
> > >>
> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> > >> <brahmbhatt.pa...@gmail.com> wrote:
> > >> > Bumping this up one more time, can other committers review?
> > >> >
> > >> > Thanks
> > >> > Parth
> > >> >
> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
> > >> >
> > >> >> Parth,
> > >> >> Overall current design looks good to me. I am +1 on the
> > KIP.
> > >> >>
> > >> >> Gwen , Jun can you review this as well.
> > >> >>
> > >> >> -Harsha
> > >> >>
> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> > >> >> > Thanks for review Jitendra.
> > >> >> >
> > >> >> > I don't like the idea of infinite lifetime but I see the Streaming
> > use
> > >> >> > case. Even for Streaming use case I was hoping there will be some
> > notion
> > >> >> > of
> > >> >> > master/driver that can get new delegation tokens at fixed interval
> > and
> > >> >> > distribute to workers. If that is not the case for we can discuss
> > >> >> > delegation tokens renewing them self and the security implications
> > of the
> > >> >> > same.
> > >> >> >
> > >> >> > I did not want clients to fetch tokens from zookeeper, overall I
> > think
> > >> >> > its
> > >> >> > better if clients don't rely on our metadata store and I think we
> > are
> > >> >> > moving in that direction with all the KIP-4 improvements. I chose
> > >> >> > zookeeper as in this case the client will still just talk to
> > broker , its
> > >> >> > the brokers that will use zookeeper which we already do for a lot
> > of
> > >> >> > other
> > >> >> > usecases + ease of development + and the ability so tokens will
> > survive
> > >> >> > even a rolling restart/cluster failure. if a majority agrees the
> > added
> > >> >> > complexity to have controller forwarding keys to all broker is
> > justified
> > >> >> > as
> > >> >> > it provides tighter security , I am fine with that option too.
> > >> >> >
> > >> >> > Given zookeeper does not support SSL we can not store master keys
> > in
> > >> >> > zookeeper as master keys will be exposed on wire. To support
> > rotation
> > >> >> > without affecting current clients is something I need to put more
> > thought
> > >> >> > in. My current proposal assumes the rotation will invalidate all
> > current
> > >> >> > tokens.
> > >> >> >
> > >> >> > I request committers to also review and post their comments so we
> > can
> > >> >> > make
> > >> >> > progress on this KIP.
> > >> >> >
> > >> >> > Thanks
> > >> >> > Parth
> > >> >> >
> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <asi...@cloudera.com
> > >
> > >> >> > wrote:
> > >> >> >
> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io>
> > wrote:
> > >> >> > >
> > >> >> > > > Unifying the two discussion threads on this KIP.
> > >> >> > > >
> > >> >> > > > Here is the response from Jitendra
> > >> >> > > >
> > >> >> > > > "The need for a large number of clients that are running all
> > over the
> > >> >> > > > cluster that authenticate with Kafka brokers, is very similar
> > to the
> > >> >> > > > Hadoop use case of large number of tasks running across the
> > cluster
> > >> >> that
> > >> >> > > > need authentication to Hdfs Namenode. Therefore, the
> > delegation token
> > >> >> > > > approach does seem like a good fit for this use case as we
> > have seen
> > >> >> it
> > >> >> > > > working at large scale in HDFS and YARN.
> > >> >> > > >
> > >> >> > > > The proposed design is very much inline with Hadoop
> > approach. A few
> > >> >> > > > comments:
> > >> >> > > >
> > >> >> > > > 1) Why do you guys want to allow infinite renewable lifetime
> > for a
> > >> >> > > > token? HDFS restricts a token to a max life time (default 7
> > days). A
> > >> >> > > > token's vulnerability is believed to increase with time.
> > >> >> > > >
> > >> >> > > I agree that having infinite lifetime might not be the best idea.
> > >> >> > >
> > >> >> > > >
> > >> >> > > > 2) As I understand the tokens are stored in zookeeper as well,
> > and
> > >> >> can
> > >> >> > > > be updated there. This is clever as it can allow replacing the
> > tokens
> > >> >> > > > once they run out of max life time, and clients can download
> > new
> > >> >> tokens
> > >> >> > > > from zookeeper. It shouldn't be a big load on zookeeper as a
> > client
> > >> >> will
> > >> >> > > > need to get a new token once in several days. In this approach
> > you
> > >> >> don't
> > >> >> > > > need infinite lifetime on the token even for long running
> > clients.
> > >> >> > > >
> > >> >> > > > 3) The token password are generated using a master key. The
> > master
> > >> >> key
> > >> >> > > > should also be periodically changed. In Hadoop, the default
> > renewal
> > >> >> > > > period is 1 day.?
> > >> >> > > >
> > >> >> > > IIUC, this will require brokers maintaining a list of X most
> > recent
> > >> >> master
> > >> >> > > keys. This list will have to be persisted somewhere, as if a
> > broker
> > >> >> goes
> > >> >> > > down it will have to get that list again and storing master keys
> > on ZK
> > >> >> is
> > >> >> > > not the best idea. However, if a broker goes down then we have
> > much
> > >> >> bigger
> > >> >> > > issue to deal with and client can always re-authenticate is such
> > >> >> events.
> > >> >> > >
> > >> >> > > Did you happen to take a look at other alternatives this list has
> > >> >> > > suggested?
> > >> >> > >
> > >> >> > > >
> > >> >> > > > Thanks for a thorough proposal, great work!"
> > >> >> > > >
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> > >> >> > > > > Makes sense to me. Thanks!
> > >> >> > > > >
> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io>
> > wrote:
> > >> >> > > > > > It doesn't need any release vehicle but still the work can
> > move
> > >> >> > > > forward.
> > >> >> > > > > > If anyone is interested in the KIP please do the review and
> > >> >> provide
> > >> >> > > the
> > >> >> > > > > > comments.
> > >> >> > > > > >
> > >> >> > > > > > -Harsha
> > >> >> > > > > >
> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
> > >> >> > > > > >> I agree that it would be good to have more time to review
> > and
> > >> >> > > discuss
> > >> >> > > > > >> KIP-48.
> > >> >> > > > > >>
> > >> >> > > > > >> Ismael
> > >> >> > > > > >>
> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> > >> >> g...@confluent.io>
> > >> >> > > > wrote:
> > >> >> > > > > >>
> > >> >> > > > > >> > Hi Team,
> > >> >> > > > > >> >
> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit
> > of a
> > >> >> risk
> > >> >> > > for
> > >> >> > > > > >> > the next release - any chance we can delay delegation
> > tokens
> > >> >> to
> > >> >> > > > Kafka
> > >> >> > > > > >> > 0.10.1?
> > >> >> > > > > >> > With the community working on a release every 3 month,
> > this
> > >> >> is not
> > >> >> > > > a huge
> > >> >> > > > > >> > delay.
> > >> >> > > > > >> >
> > >> >> > > > > >> > Gwen
> > >> >> > > > > >> >
> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
> > >> >> > > asi...@cloudera.com>
> > >> >> > > > wrote:
> > >> >> > > > > >> > > Parth,
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > Thanks again for the awesome write up. Following our
> > >> >> discussion
> > >> >> > > > from the
> > >> >> > > > > >> > > JIRA, I think it will be easier to compare various
> > >> >> alternatives
> > >> >> > > > if they
> > >> >> > > > > >> > are
> > >> >> > > > > >> > > listed together. I am stating below a few
> > alternatives along
> > >> >> > > with
> > >> >> > > > a the
> > >> >> > > > > >> > > current proposal.
> > >> >> > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > 1. Client authenticates with a broker.
> > >> >> > > > > >> > > 2. Once a client is authenticated, it will make a
> > broker
> > >> >> side
> > >> >> > > > call to
> > >> >> > > > > >> > > issue a delegation token.
> > >> >> > > > > >> > > 3. The broker generates a shared secret based on
> > >> >> > > HMAC-SHA256(a
> > >> >> > > > > >> > > Password/Secret shared between all brokers,
> > randomly
> > >> >> > > generated
> > >> >> > > > > >> > tokenId).
> > >> >> > > > > >> > > 4. Broker stores this token in its in memory cache.
> > >> >> Broker
> > >> >> > > > also stores
> > >> >> > > > > >> > > the DelegationToken without the hmac in the
> > zookeeper.
> > >> >> > > > > >> > > 5. All brokers will have a cache backed by
> > zookeeper so
> > >> >> they
> > >> >> > > > will all
> > >> >> > > > > >> > > get notified whenever a new token is generated and
> > they
> > >> >> will
> > >> >> > > > update
> > >> >> > > > > >> > their
> > >> >> > > > > >> > > local cache whenever token state changes.
> > >> >> > > > > >> > > 6. Broker returns the token to Client.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > Probable issues and fixes
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > 1. Probable race condition, client tries to
> > authenticate
> > >> >> with
> > >> >> > > > a broker
> > >> >> > > > > >> > > that is yet to be updated with the newly generated
> > DT.
> > >> >> This
> > >> >> > > can
> > >> >> > > > > >> > probably be
> > >> >> > > > > >> > > dealt with making dtRequest block until all
> > brokers have
> > >> >> > > > updated
> > >> >> > > > > >> > their DT
> > >> >> > > > > >> > > cache. Zk barrier or similar mechanism can be used.
> > >> >> However,
> > >> >> > > > all such
> > >> >> > > > > >> > > mechanisms will increase complexity.
> > >> >> > > > > >> > > 2. Using a static secret key from config file. Will
> > >> >> require
> > >> >> > > yet
> > >> >> > > > > >> > another
> > >> >> > > > > >> > > config and uses a static secret key. It is advised
> > to
> > >> >> rotate
> > >> >> > > > secret
> > >> >> > > > > >> > keys
> > >> >> > > > > >> > > periodically. This can be avoided with controller
> > >> >> generating
> > >> >> > > > > >> > secretKey and
> > >> >> > > > > >> > > passing to brokers periodically. However, this will
> > >> >> require
> > >> >> > > > brokers to
> > >> >> > > > > >> > > maintain certain counts of secretKeys.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > (Alternative 1) Have controller generate delegation
> > token.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > 1. Client authenticates with a broker.
> > >> >> > > > > >> > > 2. Once a client is authenticated, it will make a
> > broker
> > >> >> side
> > >> >> > > > call to
> > >> >> > > > > >> > > issue a delegation token.
> > >> >> > > > > >> > > 3. Broker forwards the request to controller.
> > >> >> > > > > >> > > 4. Controller generates a DT and broadcasts to all
> > >> >> brokers.
> > >> >> > > > > >> > > 5. Broker stores this token in its memory cache.
> > >> >> > > > > >> > > 6. Controller responds to broker’s DT req.
> > >> >> > > > > >> > > 7. Broker returns the token to Client.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > Probable issues and fixes
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > 1. We will have to add new APIs to support
> > controller
> > >> >> pushing
> > >> >> > > > tokens
> > >> >> > > > > >> > to
> > >> >> > > > > >> > > brokers on top of the minimal APIs that are
> > currently
> > >> >> > > proposed.
> > >> >> > > > > >> > > 2. We will also have to add APIs to support the
> > >> >> bootstrapping
> > >> >> > > > case,
> > >> >> > > > > >> > i.e,
> > >> >> > > > > >> > > when a new broker comes up it will have to get all
> > >> >> delegation
> > >> >> > > > tokens
> > >> >> > > > > >> > from
> > >> >> > > > > >> > > the controller.
> > >> >> > > > > >> > > 3. In catastrophic failures where all brokers go
> > down,
> > >> >> the
> > >> >> > > > tokens will
> > >> >> > > > > >> > > be lost even if servers are restarted as tokens
> > are not
> > >> >> > > > persisted
> > >> >> > > > > >> > anywhere.
> > >> >> > > > > >> > > If this happens, then there are more important
> > things to
> > >> >> > > worry
> > >> >> > > > about
> > >> >> > > > > >> > and
> > >> >> > > > > >> > > maybe it is better to re-authenticate.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to other brokers
> > at
> > >> >> all.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > 1. Client authenticates with a broker.
> > >> >> > > > > >> > > 2. Once a client is authenticated, it will make a
> > broker
> > >> >> side
> > >> >> > > > call to
> > >> >> > > > > >> > > issue a delegation token.
> > >> >> > > > > >> > > 3. The broker generates DT of form, [hmac + (owner,
> > >> >> renewer,
> > >> >> > > > > >> > > maxLifeTime, id, hmac, expirationTime)] and passes
> > back
> > >> >> this
> > >> >> > > > DT to
> > >> >> > > > > >> > client.
> > >> >> > > > > >> > > hmac is generated via {HMAC-SHA256(owner, renewer,
> > >> >> > > > maxLifeTime, id,
> > >> >> > > > > >> > hmac,
> > >> >> > > > > >> > > expirationTime) using SecretKey}. Note that all
> > brokers
> > >> >> have
> > >> >> > > > this
> > >> >> > > > > >> > SecretKey.
> > >> >> > > > > >> > > 4. Client then goes to any broker and to
> > authenticate
> > >> >> sends
> > >> >> > > > the DT.
> > >> >> > > > > >> > > Broker recalculates hmac using (owner, renewer,
> > >> >> maxLifeTime,
> > >> >> > > > id, hmac,
> > >> >> > > > > >> > > expirationTime) info from DT and its SecretKey. If
> > it
> > >> >> matches
> > >> >> > > > with
> > >> >> > > > > >> > hmac of
> > >> >> > > > > >> > > DT, client is authenticated. Yes, it will do other
> > >> >> obvious
> > >> >> > > > checks of
> > >> >> > > > > >> > > timestamp expiry and such.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > Note that secret key will be generated by controller
> > and
> > >> >> passed
> > >> >> > > to
> > >> >> > > > > >> > brokers
> > >> >> > > > > >> > > periodically.
> > >> >> > > > > >> > > Probable issues and fixes
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > 1. How to delete a DT? Yes, that is a downside
> > here.
> > >> >> However,
> > >> >> > > > this can
> > >> >> > > > > >> > > be handled with brokers maintaining a blacklist of
> > DTs,
> > >> >> DTs
> > >> >> > > > from this
> > >> >> > > > > >> > list
> > >> >> > > > > >> > > can be removed after expiry.
> > >> >> > > > > >> > > 2. In catastrophic failures where all brokers go
> > down,
> > >> >> the
> > >> >> > > > tokens will
> > >> >> > > > > >> > > be lost even if servers are restarted as tokens
> > are not
> > >> >> > > > persisted
> > >> >> > > > > >> > anywhere.
> > >> >> > > > > >> > > If this happens, then there are more important
> > things to
> > >> >> > > worry
> > >> >> > > > about
> > >> >> > > > > >> > and
> > >> >> > > > > >> > > maybe it is better to re-authenticate.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> > >> >> > > > > >> > > pbrahmbh...@hortonworks.com> wrote:
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >> Hi,
> > >> >> > > > > >> > >>
> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like
> > delegation
> > >> >> > > > tokens in
> > >> >> > > > > >> > >> kafka. You can review the design
> > >> >> > > > > >> > >>
> > >> >> > > > > >> >
> > >> >> > > >
> > >> >> > >
> > >> >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > >> >> > > > > >> > .
> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also
> > discussed an
> > >> >> > > > alternative to
> > >> >> > > > > >> > >> proposed design here<
> > >> >> > > > > >> > >>
> > >> >> > > > > >> >
> > >> >> > > >
> > >> >> > >
> > >> >>
> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > >> >> > > > > >> > >> >.
> > >> >> > > > > >> > >>
> > >> >> > > > > >> > >> Thanks
> > >> >> > > > > >> > >> Parth
> > >> >> > > > > >> > >>
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > --
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > Regards,
> > >> >> > > > > >> > > Ashish
> > >> >> > > > > >> >
> > >> >> > > >
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> > > --
> > >> >> > >
> > >> >> > > Regards,
> > >> >> > > Ashish
> > >> >> > >
> > >> >>
> >
