Harsha,
I was thinking of the Rest Proxy. I didn't see your design yet, but in
our proxy, we have a set of producers, which will serve multiple users
going through the proxy. Since these users will have different
privileges, they'll need to authenticate separately, and can't share a
token.
Am I missing anything?
Gwen
On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
> Gwen,
> On your second point. Can you describe a usecase where
> mutliple clients ended up sharing a producer and even if they
> do why can't they not use single token that producer
> captures. Why would we need multiple clients with different
> tokens sharing a single instance of producer. Also in this
> case other clients have access all the tokens no?
>
> Thanks,
> Harsha
>
>
> On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
>> Sorry for the delay:
>>
>> Two questions that we didn't see in the wiki:
>> 1. Is there an expiration for delegation tokens? Renewal? How do we
>> revoke them?
>> 2. If we want to use delegation tokens for "do-as" (say, submit Storm
>> job as my user), we will need a producer for every job (we can't share
>> them between multiple jobs running on same node), since we only
>> authenticate when connecting. Is there a plan to change this for
>> delegation tokens, in order to allow multiple users with different
>> tokens to share a client?
>>
>> Gwen
>>
>> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
>> <brahmbhatt.pa...@gmail.com> wrote:
>> > Bumping this up one more time, can other committers review?
>> >
>> > Thanks
>> > Parth
>> >
>> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
>> >
>> >> Parth,
>> >> Overall current design looks good to me. I am +1 on the KIP.
>> >>
>> >> Gwen , Jun can you review this as well.
>> >>
>> >> -Harsha
>> >>
>> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
>> >> > Thanks for review Jitendra.
>> >> >
>> >> > I don't like the idea of infinite lifetime but I see the Streaming use
>> >> > case. Even for Streaming use case I was hoping there will be some notion
>> >> > of
>> >> > master/driver that can get new delegation tokens at fixed interval and
>> >> > distribute to workers. If that is not the case for we can discuss
>> >> > delegation tokens renewing them self and the security implications of
>> >> > the
>> >> > same.
>> >> >
>> >> > I did not want clients to fetch tokens from zookeeper, overall I think
>> >> > its
>> >> > better if clients don't rely on our metadata store and I think we are
>> >> > moving in that direction with all the KIP-4 improvements. I chose
>> >> > zookeeper as in this case the client will still just talk to broker ,
>> >> > its
>> >> > the brokers that will use zookeeper which we already do for a lot of
>> >> > other
>> >> > usecases + ease of development + and the ability so tokens will survive
>> >> > even a rolling restart/cluster failure. if a majority agrees the added
>> >> > complexity to have controller forwarding keys to all broker is justified
>> >> > as
>> >> > it provides tighter security , I am fine with that option too.
>> >> >
>> >> > Given zookeeper does not support SSL we can not store master keys in
>> >> > zookeeper as master keys will be exposed on wire. To support rotation
>> >> > without affecting current clients is something I need to put more
>> >> > thought
>> >> > in. My current proposal assumes the rotation will invalidate all current
>> >> > tokens.
>> >> >
>> >> > I request committers to also review and post their comments so we can
>> >> > make
>> >> > progress on this KIP.
>> >> >
>> >> > Thanks
>> >> > Parth
>> >> >
>> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <asi...@cloudera.com>
>> >> > wrote:
>> >> >
>> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io> wrote:
>> >> > >
>> >> > > > Unifying the two discussion threads on this KIP.
>> >> > > >
>> >> > > > Here is the response from Jitendra
>> >> > > >
>> >> > > > "The need for a large number of clients that are running all over
>> >> > > > the
>> >> > > > cluster that authenticate with Kafka brokers, is very similar to the
>> >> > > > Hadoop use case of large number of tasks running across the cluster
>> >> that
>> >> > > > need authentication to Hdfs Namenode. Therefore, the delegation
>> >> > > > token
>> >> > > > approach does seem like a good fit for this use case as we have seen
>> >> it
>> >> > > > working at large scale in HDFS and YARN.
>> >> > > >
>> >> > > > The proposed design is very much inline with Hadoop approach. A
>> >> > > > few
>> >> > > > comments:
>> >> > > >
>> >> > > > 1) Why do you guys want to allow infinite renewable lifetime for a
>> >> > > > token? HDFS restricts a token to a max life time (default 7 days).
>> >> > > > A
>> >> > > > token's vulnerability is believed to increase with time.
>> >> > > >
>> >> > > I agree that having infinite lifetime might not be the best idea.
>> >> > >
>> >> > > >
>> >> > > > 2) As I understand the tokens are stored in zookeeper as well, and
>> >> can
>> >> > > > be updated there. This is clever as it can allow replacing the
>> >> > > > tokens
>> >> > > > once they run out of max life time, and clients can download new
>> >> tokens
>> >> > > > from zookeeper. It shouldn't be a big load on zookeeper as a client
>> >> will
>> >> > > > need to get a new token once in several days. In this approach you
>> >> don't
>> >> > > > need infinite lifetime on the token even for long running clients.
>> >> > > >
>> >> > > > 3) The token password are generated using a master key. The master
>> >> key
>> >> > > > should also be periodically changed. In Hadoop, the default renewal
>> >> > > > period is 1 day.?
>> >> > > >
>> >> > > IIUC, this will require brokers maintaining a list of X most recent
>> >> master
>> >> > > keys. This list will have to be persisted somewhere, as if a broker
>> >> goes
>> >> > > down it will have to get that list again and storing master keys on ZK
>> >> is
>> >> > > not the best idea. However, if a broker goes down then we have much
>> >> bigger
>> >> > > issue to deal with and client can always re-authenticate is such
>> >> events.
>> >> > >
>> >> > > Did you happen to take a look at other alternatives this list has
>> >> > > suggested?
>> >> > >
>> >> > > >
>> >> > > > Thanks for a thorough proposal, great work!"
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
>> >> > > > > Makes sense to me. Thanks!
>> >> > > > >
>> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io> wrote:
>> >> > > > > > It doesn't need any release vehicle but still the work can move
>> >> > > > forward.
>> >> > > > > > If anyone is interested in the KIP please do the review and
>> >> provide
>> >> > > the
>> >> > > > > > comments.
>> >> > > > > >
>> >> > > > > > -Harsha
>> >> > > > > >
>> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
>> >> > > > > >> I agree that it would be good to have more time to review and
>> >> > > discuss
>> >> > > > > >> KIP-48.
>> >> > > > > >>
>> >> > > > > >> Ismael
>> >> > > > > >>
>> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
>> >> g...@confluent.io>
>> >> > > > wrote:
>> >> > > > > >>
>> >> > > > > >> > Hi Team,
>> >> > > > > >> >
>> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit of a
>> >> risk
>> >> > > for
>> >> > > > > >> > the next release - any chance we can delay delegation tokens
>> >> to
>> >> > > > Kafka
>> >> > > > > >> > 0.10.1?
>> >> > > > > >> > With the community working on a release every 3 month, this
>> >> is not
>> >> > > > a huge
>> >> > > > > >> > delay.
>> >> > > > > >> >
>> >> > > > > >> > Gwen
>> >> > > > > >> >
>> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
>> >> > > asi...@cloudera.com>
>> >> > > > wrote:
>> >> > > > > >> > > Parth,
>> >> > > > > >> > >
>> >> > > > > >> > > Thanks again for the awesome write up. Following our
>> >> discussion
>> >> > > > from the
>> >> > > > > >> > > JIRA, I think it will be easier to compare various
>> >> alternatives
>> >> > > > if they
>> >> > > > > >> > are
>> >> > > > > >> > > listed together. I am stating below a few alternatives
>> >> > > > > >> > > along
>> >> > > with
>> >> > > > a the
>> >> > > > > >> > > current proposal.
>> >> > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
>> >> > > > > >> > >
>> >> > > > > >> > > 1. Client authenticates with a broker.
>> >> > > > > >> > > 2. Once a client is authenticated, it will make a broker
>> >> side
>> >> > > > call to
>> >> > > > > >> > > issue a delegation token.
>> >> > > > > >> > > 3. The broker generates a shared secret based on
>> >> > > HMAC-SHA256(a
>> >> > > > > >> > > Password/Secret shared between all brokers, randomly
>> >> > > generated
>> >> > > > > >> > tokenId).
>> >> > > > > >> > > 4. Broker stores this token in its in memory cache.
>> >> Broker
>> >> > > > also stores
>> >> > > > > >> > > the DelegationToken without the hmac in the zookeeper.
>> >> > > > > >> > > 5. All brokers will have a cache backed by zookeeper so
>> >> they
>> >> > > > will all
>> >> > > > > >> > > get notified whenever a new token is generated and they
>> >> will
>> >> > > > update
>> >> > > > > >> > their
>> >> > > > > >> > > local cache whenever token state changes.
>> >> > > > > >> > > 6. Broker returns the token to Client.
>> >> > > > > >> > >
>> >> > > > > >> > > Probable issues and fixes
>> >> > > > > >> > >
>> >> > > > > >> > > 1. Probable race condition, client tries to authenticate
>> >> with
>> >> > > > a broker
>> >> > > > > >> > > that is yet to be updated with the newly generated DT.
>> >> This
>> >> > > can
>> >> > > > > >> > probably be
>> >> > > > > >> > > dealt with making dtRequest block until all brokers have
>> >> > > > updated
>> >> > > > > >> > their DT
>> >> > > > > >> > > cache. Zk barrier or similar mechanism can be used.
>> >> However,
>> >> > > > all such
>> >> > > > > >> > > mechanisms will increase complexity.
>> >> > > > > >> > > 2. Using a static secret key from config file. Will
>> >> require
>> >> > > yet
>> >> > > > > >> > another
>> >> > > > > >> > > config and uses a static secret key. It is advised to
>> >> rotate
>> >> > > > secret
>> >> > > > > >> > keys
>> >> > > > > >> > > periodically. This can be avoided with controller
>> >> generating
>> >> > > > > >> > secretKey and
>> >> > > > > >> > > passing to brokers periodically. However, this will
>> >> require
>> >> > > > brokers to
>> >> > > > > >> > > maintain certain counts of secretKeys.
>> >> > > > > >> > >
>> >> > > > > >> > > (Alternative 1) Have controller generate delegation token.
>> >> > > > > >> > >
>> >> > > > > >> > > 1. Client authenticates with a broker.
>> >> > > > > >> > > 2. Once a client is authenticated, it will make a broker
>> >> side
>> >> > > > call to
>> >> > > > > >> > > issue a delegation token.
>> >> > > > > >> > > 3. Broker forwards the request to controller.
>> >> > > > > >> > > 4. Controller generates a DT and broadcasts to all
>> >> brokers.
>> >> > > > > >> > > 5. Broker stores this token in its memory cache.
>> >> > > > > >> > > 6. Controller responds to broker’s DT req.
>> >> > > > > >> > > 7. Broker returns the token to Client.
>> >> > > > > >> > >
>> >> > > > > >> > > Probable issues and fixes
>> >> > > > > >> > >
>> >> > > > > >> > > 1. We will have to add new APIs to support controller
>> >> pushing
>> >> > > > tokens
>> >> > > > > >> > to
>> >> > > > > >> > > brokers on top of the minimal APIs that are currently
>> >> > > proposed.
>> >> > > > > >> > > 2. We will also have to add APIs to support the
>> >> bootstrapping
>> >> > > > case,
>> >> > > > > >> > i.e,
>> >> > > > > >> > > when a new broker comes up it will have to get all
>> >> delegation
>> >> > > > tokens
>> >> > > > > >> > from
>> >> > > > > >> > > the controller.
>> >> > > > > >> > > 3. In catastrophic failures where all brokers go down,
>> >> the
>> >> > > > tokens will
>> >> > > > > >> > > be lost even if servers are restarted as tokens are not
>> >> > > > persisted
>> >> > > > > >> > anywhere.
>> >> > > > > >> > > If this happens, then there are more important things to
>> >> > > worry
>> >> > > > about
>> >> > > > > >> > and
>> >> > > > > >> > > maybe it is better to re-authenticate.
>> >> > > > > >> > >
>> >> > > > > >> > > (Alternative 2) Do not distribute DT to other brokers at
>> >> all.
>> >> > > > > >> > >
>> >> > > > > >> > > 1. Client authenticates with a broker.
>> >> > > > > >> > > 2. Once a client is authenticated, it will make a broker
>> >> side
>> >> > > > call to
>> >> > > > > >> > > issue a delegation token.
>> >> > > > > >> > > 3. The broker generates DT of form, [hmac + (owner,
>> >> renewer,
>> >> > > > > >> > > maxLifeTime, id, hmac, expirationTime)] and passes back
>> >> this
>> >> > > > DT to
>> >> > > > > >> > client.
>> >> > > > > >> > > hmac is generated via {HMAC-SHA256(owner, renewer,
>> >> > > > maxLifeTime, id,
>> >> > > > > >> > hmac,
>> >> > > > > >> > > expirationTime) using SecretKey}. Note that all brokers
>> >> have
>> >> > > > this
>> >> > > > > >> > SecretKey.
>> >> > > > > >> > > 4. Client then goes to any broker and to authenticate
>> >> sends
>> >> > > > the DT.
>> >> > > > > >> > > Broker recalculates hmac using (owner, renewer,
>> >> maxLifeTime,
>> >> > > > id, hmac,
>> >> > > > > >> > > expirationTime) info from DT and its SecretKey. If it
>> >> matches
>> >> > > > with
>> >> > > > > >> > hmac of
>> >> > > > > >> > > DT, client is authenticated. Yes, it will do other
>> >> obvious
>> >> > > > checks of
>> >> > > > > >> > > timestamp expiry and such.
>> >> > > > > >> > >
>> >> > > > > >> > > Note that secret key will be generated by controller and
>> >> passed
>> >> > > to
>> >> > > > > >> > brokers
>> >> > > > > >> > > periodically.
>> >> > > > > >> > > Probable issues and fixes
>> >> > > > > >> > >
>> >> > > > > >> > > 1. How to delete a DT? Yes, that is a downside here.
>> >> However,
>> >> > > > this can
>> >> > > > > >> > > be handled with brokers maintaining a blacklist of DTs,
>> >> DTs
>> >> > > > from this
>> >> > > > > >> > list
>> >> > > > > >> > > can be removed after expiry.
>> >> > > > > >> > > 2. In catastrophic failures where all brokers go down,
>> >> the
>> >> > > > tokens will
>> >> > > > > >> > > be lost even if servers are restarted as tokens are not
>> >> > > > persisted
>> >> > > > > >> > anywhere.
>> >> > > > > >> > > If this happens, then there are more important things to
>> >> > > worry
>> >> > > > about
>> >> > > > > >> > and
>> >> > > > > >> > > maybe it is better to re-authenticate.
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
>> >> > > > > >> > > pbrahmbh...@hortonworks.com> wrote:
>> >> > > > > >> > >
>> >> > > > > >> > >> Hi,
>> >> > > > > >> > >>
>> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like delegation
>> >> > > > tokens in
>> >> > > > > >> > >> kafka. You can review the design
>> >> > > > > >> > >>
>> >> > > > > >> >
>> >> > > >
>> >> > >
>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
>> >> > > > > >> > .
>> >> > > > > >> > >> This KIP depends on KIP-43 and we have also discussed an
>> >> > > > alternative to
>> >> > > > > >> > >> proposed design here<
>> >> > > > > >> > >>
>> >> > > > > >> >
>> >> > > >
>> >> > >
>> >> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
>> >> > > > > >> > >> >.
>> >> > > > > >> > >>
>> >> > > > > >> > >> Thanks
>> >> > > > > >> > >> Parth
>> >> > > > > >> > >>
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > > --
>> >> > > > > >> > >
>> >> > > > > >> > > Regards,
>> >> > > > > >> > > Ashish
>> >> > > > > >> >
>> >> > > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > >
>> >> > > Regards,
>> >> > > Ashish
>> >> > >
>> >>
