Hi Jun, Thanks for reviewing.
* We could add a Cluster action to add acls on who can request delegation tokens. I don't see the use case for that yet but down the line when we start supporting getDelegationTokenAs it will be necessary. * Yes we recommend tokens to be only used/distributed over secure channels. * Depending on what design we end up choosing Invalidation will be responsibility of every broker or controller. * I am not sure if I documented somewhere that invalidation will directly go through zookeeper but that is not the intent. Invalidation will either be request based or due to expiration. No direct zookeeper interaction from any client. * "Broker also stores the DelegationToken without the hmac in the zookeeper." : Sorry about the confusion. The sole purpose of zookeeper in this design is as distribution channel for tokens between all brokers and a layer that ensures only tokens that were generated by making a request to a broker will be accepted (more on this in second paragraph). The token consists of few elements (owner, renewer, uuid , expiration, hmac) , one of which is the finally generated hmac but hmac it self is derivable if you have all the other elements of the token + secret key to generate hmac. Given zookeeper does not provide SSL support we do not want the entire token to be wire transferred to zookeeper as that will be an insecure wire transfer. Instead we only store all the other elements of a delegation tokens. Brokers can read these elements and because they also have access to secret key they will be able to generate hmac on their end. One of the alternative proposed is to avoid zookeeper altogether. A Client will call broker with required information (owner, renwer, expiration) and get back (signed hmac, uuid). Broker won't store this in zookeeper. From this point a client can contact any broker with all the delegation token info (owner, rewner, expiration, hmac, uuid) the borker will regenerate the hmac and as long as it matches with hmac presented by client , broker will allow the request to authenticate. Only problem with this approach is if the secret key is compromised any client can now generate random tokens and they will still be able to authenticate as any user they like. with zookeeper we guarantee that only tokens acquired via a broker (using some auth scheme other than delegation token) will be accepted. We need to discuss which proposal makes more sense and we can go over it in tomorrow's meeting. Also, can you forward the invite to me? Thanks Parth On Mon, May 23, 2016 at 10:35 AM, Jun Rao <j...@confluent.io> wrote: > Thanks for the KIP. A few comments. > > 100. This potentially can be useful for Kafka Connect and Kafka rest proxy > where a worker agent will need to run a task on behalf of a client. We will > likely need to change how those services use Kafka clients > (producer/consumer). Instead of a shared client per worker, we will need a > client per user task since the authentication happens at the connection > level. For Kafka Connect, the renewer will be the workers. So, we probably > need to allow multiple renewers. For Kafka rest proxy, the renewer can > probably just be the creator of the token. > > 101. Do we need new acl on who can request delegation tokens? > > 102. Do we recommend people to send delegation tokens in an encrypted > channel? > > 103. Who is responsible for expiring tokens, every broker? > > 104. For invalidating tokens, would it be better to do it in a request > instead of going to ZK directly? > > 105. The terminology of client in the wiki sometimes refers to the end > client and some other times refers to the client using the delegation > tokens. It would be useful to distinguish between the two. > > 106. Could you explain the sentence "Broker also stores the DelegationToken > without the hmac in the zookeeper." a bit more? I thought the delegation > token is the hmac. > > Thanks, > > Jun > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <j...@confluent.io> wrote: > > > Hi, Harsha, > > > > Just sent out a KIP meeting invite. We can discuss this in the meeting > > tomorrow. > > > > Thanks, > > > > Jun > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io> wrote: > > > >> Hi All, > >> Can we have a KIP meeting around this. The KIP is up for > >> sometime and if there are any questions lets quickly hash out > >> details. > >> > >> Thanks, > >> Harsha > >> > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote: > >> > That is what the hadoop echo system uses so no good reason really. We > >> > could > >> > change it to whatever is the newest recommended standard is. > >> > > >> > Thanks > >> > Parth > >> > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <ism...@juma.me.uk> > wrote: > >> > > >> > > Hi Parth, > >> > > > >> > > Thanks for the KIP. I only started reviewing this and may have > >> additional > >> > > questions later. The immediate question that came to mind is our > >> choice of > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA > Registry > >> of > >> > > SASL mechanisms and the original RFC (2831) has been moved to > Historic > >> > > status: > >> > > > >> > > https://tools.ietf.org/html/rfc6331 > >> > > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml > >> > > > >> > > What is the reasoning behind that choice? > >> > > > >> > > Thanks, > >> > > Ismael > >> > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <g...@confluent.io> > >> wrote: > >> > > > >> > > > Also comments inline :) > >> > > > > >> > > > > * I want to emphasize that even though delegation tokens are a > >> Hadoop > >> > > > > innovation, I feel very strongly about not adding dependency on > >> Hadoop > >> > > > > when implementing delegation tokens for Kafka. The KIP doesn't > >> imply > >> > > > > such dependency, but if you can clarify... > >> > > > > > >> > > > > > >> > > > > *No hadoop dependency.* > >> > > > > >> > > > Yay! Just add this to the KIP so no one will read the KIP and > panic > >> > > > three weeks before the next release... > >> > > > > >> > > > > * Can we get delegation token at any time after authenticating? > >> only > >> > > > > immediately after? > >> > > > > > >> > > > > > >> > > > > *As long as you are authenticated you can get delegation tokens. > >> We > >> > > need > >> > > > to > >> > > > > discuss if a client authenticated using delegation token, can > also > >> > > > acquire > >> > > > > delegation token again or not. Also there is the question of do > we > >> > > allow > >> > > > > anyone to acquire delegation token or we want specific ACLs (I > >> think > >> > > its > >> > > > an > >> > > > > overkill.)* > >> > > > > >> > > > I agree that ACLs is an overkill. > >> > > > > >> > > > I think we are debating two options: Either require Kerberos auth > >> for > >> > > > renewal or require non-owners to renew. > >> > > > I *think* the latter is simpler (it basically require a "job > master" > >> > > > to take responsibility for the renewal, it will have its own > >> identity > >> > > > anyway and I think this is the correct design pattern anyway. For > >> > > > storm, I'd expect Nimbus to coordinate renewals?), but it is hard > to > >> > > > debate simplicity without looking at the code changes required. If > >> you > >> > > > have a draft of how the "require Kerberos" will look in Kafka > code, > >> > > > I'll be happy to take a look. > >> > > > > >> > > > > * My understanding is that tokens will propagate via ZK but > >> without > >> > > > > additional changes to UpdateMetadata protocol, correct? Clients > >> > > > > currently don't retry on SASL auth failure (IIRC), but since the > >> > > > > tokens propagate between brokers asynch, we will need to retry a > >> bit > >> > > > > to avoid clients failing auth due to timing issues. > >> > > > > > >> > > > > *I am considering 2 alternatives right now. The current > documented > >> > > > approach > >> > > > > is zookeeper based and it does not require any changes to > >> > > UpdateMetadata > >> > > > > protocol. An alternative approach can remove zookeeper > dependency > >> as > >> > > well > >> > > > > but we can discuss that in KIP discussion call.* > >> > > > > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to arrange a > call? > >> > > > > >> > > > > * I liked Ashish's suggestion of having just the controller > issue > >> the > >> > > > > delegation tokens, to avoid syncing a shared secret. Not sure if > >> we > >> > > > > want to continue the discussion here or on the wiki. I think > that > >> we > >> > > > > can decouple the problem of "token distribution" from "shared > >> secret > >> > > > > distribution" and use the controller as the only token generator > >> to > >> > > > > solve the second issue, while still using ZK async to distribute > >> > > > > tokens. > >> > > > > > >> > > > > > >> > > > > *As mentioned in the previous Email I am fine with that approach > >> as > >> > > long > >> > > > as > >> > > > > we agree that the extra complexity of adding/updating APIS adds > >> enough > >> > > > > value. The advantage with the controller approach is secret > >> rotation > >> > > can > >> > > > be > >> > > > > automated,frequent and would not require deployment. * > >> > > > > >> > > > Can you detail the extra complexity (or point me to the email I > >> > > > missed?) - which Apis are required? > >> > > > As far as I can tell, clients can already find the controller from > >> > > > metadata. I'm a bit more concerned about controller load. > >> > > > > >> > > > > > >> > > > > * While I like the idea of forcing kerberos auth for renwal, I > >> think > >> > > > > it mixes the transport layer the the request content in a pretty > >> ugly > >> > > > > way. Perhaps limiting renewer to non-owner is better. > >> > > > > > >> > > > > *I feel this is a necessary evil. While this will make the kafka > >> code > >> > > > > pretty straight forward , forcing renewer to non-owner pushes > >> the code > >> > > > > ugliness to client and makes it even harder to integrate. * > >> > > > > >> > > > As mentioned before, I don't think the "renewal by other" approach > >> is > >> > > > that ugly for the clients we expect to use delegation tokens since > >> > > > they will have an app-master of some sort who requested the token > to > >> > > > begin with. > >> > > > > >> > > > > > >> > > > > The response for my question on how multiple identities will be > >> > > > > handled wasn't super clear to me - AFAIK, we don't authenticate > >> each > >> > > > > request, we authenticate connections. > >> > > > > > >> > > > > *We authenticate connections, and only when they are being > >> established. > >> > > > Let > >> > > > > me try to phrase this as a question, in absence of delegation > >> tokens if > >> > > > we > >> > > > > had to support the use case using user TGT's how would we do it? > >> My > >> > > point > >> > > > > was it would be no different with delegation tokens. The use > case > >> you > >> > > are > >> > > > > describing seems more like impersonation.* > >> > > > > >> > > > Yeah, I thought that one of the things that delegation tokens > >> handled. > >> > > > Maybe I got it wrong :) > >> > > > > >> > > > Thanks for the detailed answers. > >> > > > > >> > > > Gwen > >> > > > > >> > > > > >> > > > > Thanks > >> > > > > Parth > >> > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira < > g...@confluent.io > >> > > >> > > > wrote: > >> > > > > > >> > > > >> Hi Parth and Harsha, > >> > > > >> > >> > > > >> Few more comments: > >> > > > >> > >> > > > >> * The API / RequestResponse section doesn't seem to have good > >> > > > >> description of the changes to the Kafka Protocol. Sounds like > >> you are > >> > > > >> proposing new DelegationTokenRequest and RenewTokenRequest (and > >> > > > >> matching responses), without detailing the contents of the > >> requests > >> > > > >> and responses? Or rather, you show the class interface, but not > >> the > >> > > > >> underlying protocol. This could be seen as an implementation > >> detail, > >> > > > >> but since the binary protocol is what we provide to non-Java > >> clients, > >> > > > >> we need to show the changes there. > >> > > > >> > >> > > > >> * getDelegationToken sounds like delegationTokenRequestHandler? > >> Is it > >> > > > >> planned to be part of KafkaApi? or Client? Its unclear... > >> > > > >> > >> > > > >> * I want to emphasize that even though delegation tokens are a > >> Hadoop > >> > > > >> innovation, I feel very strongly about not adding dependency on > >> Hadoop > >> > > > >> when implementing delegation tokens for Kafka. The KIP doesn't > >> imply > >> > > > >> such dependency, but if you can clarify... > >> > > > >> > >> > > > >> * Can we get delegation token at any time after authenticating? > >> only > >> > > > >> immediately after? > >> > > > >> > >> > > > >> * My understanding is that tokens will propagate via ZK but > >> without > >> > > > >> additional changes to UpdateMetadata protocol, correct? Clients > >> > > > >> currently don't retry on SASL auth failure (IIRC), but since > the > >> > > > >> tokens propagate between brokers asynch, we will need to retry > a > >> bit > >> > > > >> to avoid clients failing auth due to timing issues. > >> > > > >> > >> > > > >> * Strongly agreeing on clients not touching ZK directly :) > >> > > > >> > >> > > > >> * I liked Ashish's suggestion of having just the controller > >> issue the > >> > > > >> delegation tokens, to avoid syncing a shared secret. Not sure > if > >> we > >> > > > >> want to continue the discussion here or on the wiki. I think > >> that we > >> > > > >> can decouple the problem of "token distribution" from "shared > >> secret > >> > > > >> distribution" and use the controller as the only token > generator > >> to > >> > > > >> solve the second issue, while still using ZK async to > distribute > >> > > > >> tokens. > >> > > > >> > >> > > > >> * I am also uncomfortable with infinite lifetime of tokens (and > >> hoped > >> > > > >> to hear from others in the community) - but having the option > and > >> > > > >> documenting it as less secure, allows users to configure their > >> system > >> > > > >> as the wish. > >> > > > >> > >> > > > >> * While I like the idea of forcing kerberos auth for renwal, I > >> think > >> > > > >> it mixes the transport layer the the request content in a > pretty > >> ugly > >> > > > >> way. Perhaps limiting renewer to non-owner is better. > >> > > > >> > >> > > > >> Things I'd still like to see: > >> > > > >> > >> > > > >> * More detailed explanation on what we plan to do for the java > >> clients > >> > > > >> specifically - new configuration? new APIs? > >> > > > >> The response for my question on how multiple identities will be > >> > > > >> handled wasn't super clear to me - AFAIK, we don't authenticate > >> each > >> > > > >> request, we authenticate connections. > >> > > > >> > >> > > > >> * Alternatives: Delegation tokens are only used in the Hadoop > >> > > > >> ecosystem. I'm wondering if there are alternatives in other > >> ecosystems > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are some > >> advantages > >> > > > >> there. > >> > > > >> > >> > > > >> Gwen > >> > > > >> > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> > wrote: > >> > > > >> > Hi Gwen, > >> > > > >> > Can you look at Parth's last reply. Does it answer > >> your > >> > > > >> > concerns. > >> > > > >> > > >> > > > >> > Thanks, > >> > > > >> > Harsha > >> > > > >> > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote: > >> > > > >> >> Thanks for reviewing Gwen. The wiki already has details on > >> token > >> > > > >> >> expiration > >> > > > >> >> under token acquisition process > >> > > > >> >> < > >> > > > >> > >> > > > > >> > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition > >> > > > >> >. > >> > > > >> >> Current proposal is that tokens will expire based on a > server > >> side > >> > > > >> >> configuration (default 24 hours) unless renewed. Renewal is > >> only > >> > > > allowed > >> > > > >> >> until the max life time of token. Alternatively we could > also > >> make > >> > > > that > >> > > > >> >> an > >> > > > >> >> optional param and the server side default can serve as the > >> upper > >> > > > bound. > >> > > > >> >> > >> > > > >> >> To your second point it will be done exactly the same way we > >> would > >> > > > >> >> support > >> > > > >> >> multiple keytabs. The calling client will have to put the > >> tokens it > >> > > > >> wants > >> > > > >> >> to use in the subject instance and call produce/consume > inside > >> > > > >> >> subject.doas. Each caller will have to keep track of its own > >> > > > subject. I > >> > > > >> >> will have to look at the code to see if we support this > >> feature > >> > > right > >> > > > >> now > >> > > > >> >> but my understanding is delegation token shouldn't need any > >> special > >> > > > >> >> treatment as its just another type of Credential in the > >> subject. > >> > > > >> >> > >> > > > >> >> I would also like to know what is your opinion about > infinite > >> > > renewal > >> > > > >> (my > >> > > > >> >> recommendation is to not support this), tokens renewing them > >> > > self(my > >> > > > >> >> recommendation is to not support this) and most importantly > >> your > >> > > > choice > >> > > > >> >> between the alternatives listed on this thread > >> > > > >> >> < > >> > > > >> > >> > > > > >> > > > >> > http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism > >> > > > >> > > >> > > > >> >> ( I am leaning towards the alternative-2 minus controller > >> > > > distributing > >> > > > >> >> secret). Thanks again for reviewing. > >> > > > >> >> > >> > > > >> >> Thanks > >> > > > >> >> Parth > >> > > > >> >> > >> > > > >> >> > >> > > > >> >> > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira < > >> g...@confluent.io> > >> > > > wrote: > >> > > > >> >> > >> > > > >> >> > Harsha, > >> > > > >> >> > > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't see your design > >> yet, > >> > > > but in > >> > > > >> >> > our proxy, we have a set of producers, which will serve > >> multiple > >> > > > users > >> > > > >> >> > going through the proxy. Since these users will have > >> different > >> > > > >> >> > privileges, they'll need to authenticate separately, and > >> can't > >> > > > share a > >> > > > >> >> > token. > >> > > > >> >> > > >> > > > >> >> > Am I missing anything? > >> > > > >> >> > > >> > > > >> >> > Gwen > >> > > > >> >> > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> > >> wrote: > >> > > > >> >> > > Gwen, > >> > > > >> >> > > On your second point. Can you describe a > >> usecase > >> > > where > >> > > > >> >> > > mutliple clients ended up sharing a producer > >> and > >> > > even > >> > > > if > >> > > > >> they > >> > > > >> >> > > do why can't they not use single token that > >> producer > >> > > > >> >> > > captures. Why would we need multiple clients > >> with > >> > > > >> different > >> > > > >> >> > > tokens sharing a single instance of producer. > >> Also > >> > > in > >> > > > >> this > >> > > > >> >> > > case other clients have access all the tokens > >> no? > >> > > > >> >> > > > >> > > > >> >> > > Thanks, > >> > > > >> >> > > Harsha > >> > > > >> >> > > > >> > > > >> >> > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote: > >> > > > >> >> > >> Sorry for the delay: > >> > > > >> >> > >> > >> > > > >> >> > >> Two questions that we didn't see in the wiki: > >> > > > >> >> > >> 1. Is there an expiration for delegation tokens? > >> Renewal? How > >> > > > do we > >> > > > >> >> > >> revoke them? > >> > > > >> >> > >> 2. If we want to use delegation tokens for "do-as" > (say, > >> > > submit > >> > > > >> Storm > >> > > > >> >> > >> job as my user), we will need a producer for every job > >> (we > >> > > can't > >> > > > >> share > >> > > > >> >> > >> them between multiple jobs running on same node), since > >> we > >> > > only > >> > > > >> >> > >> authenticate when connecting. Is there a plan to change > >> this > >> > > for > >> > > > >> >> > >> delegation tokens, in order to allow multiple users > with > >> > > > different > >> > > > >> >> > >> tokens to share a client? > >> > > > >> >> > >> > >> > > > >> >> > >> Gwen > >> > > > >> >> > >> > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt > >> > > > >> >> > >> <brahmbhatt.pa...@gmail.com> wrote: > >> > > > >> >> > >> > Bumping this up one more time, can other committers > >> review? > >> > > > >> >> > >> > > >> > > > >> >> > >> > Thanks > >> > > > >> >> > >> > Parth > >> > > > >> >> > >> > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha < > >> ka...@harsha.io> > >> > > > wrote: > >> > > > >> >> > >> > > >> > > > >> >> > >> >> Parth, > >> > > > >> >> > >> >> Overall current design looks good to me. I > >> am +1 > >> > > on > >> > > > >> the > >> > > > >> >> > KIP. > >> > > > >> >> > >> >> > >> > > > >> >> > >> >> Gwen , Jun can you review this as well. > >> > > > >> >> > >> >> > >> > > > >> >> > >> >> -Harsha > >> > > > >> >> > >> >> > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt > >> wrote: > >> > > > >> >> > >> >> > Thanks for review Jitendra. > >> > > > >> >> > >> >> > > >> > > > >> >> > >> >> > I don't like the idea of infinite lifetime but I > >> see the > >> > > > >> Streaming > >> > > > >> >> > use > >> > > > >> >> > >> >> > case. Even for Streaming use case I was hoping > >> there will > >> > > > be > >> > > > >> some > >> > > > >> >> > notion > >> > > > >> >> > >> >> > of > >> > > > >> >> > >> >> > master/driver that can get new delegation tokens > at > >> fixed > >> > > > >> interval > >> > > > >> >> > and > >> > > > >> >> > >> >> > distribute to workers. If that is not the case for > >> we can > >> > > > >> discuss > >> > > > >> >> > >> >> > delegation tokens renewing them self and the > >> security > >> > > > >> implications > >> > > > >> >> > of the > >> > > > >> >> > >> >> > same. > >> > > > >> >> > >> >> > > >> > > > >> >> > >> >> > I did not want clients to fetch tokens from > >> zookeeper, > >> > > > >> overall I > >> > > > >> >> > think > >> > > > >> >> > >> >> > its > >> > > > >> >> > >> >> > better if clients don't rely on our metadata store > >> and I > >> > > > >> think we > >> > > > >> >> > are > >> > > > >> >> > >> >> > moving in that direction with all the KIP-4 > >> improvements. > >> > > > I > >> > > > >> chose > >> > > > >> >> > >> >> > zookeeper as in this case the client will still > >> just talk > >> > > > to > >> > > > >> >> > broker , its > >> > > > >> >> > >> >> > the brokers that will use zookeeper which we > >> already do > >> > > > for a > >> > > > >> lot > >> > > > >> >> > of > >> > > > >> >> > >> >> > other > >> > > > >> >> > >> >> > usecases + ease of development + and the ability > so > >> > > tokens > >> > > > >> will > >> > > > >> >> > survive > >> > > > >> >> > >> >> > even a rolling restart/cluster failure. if a > >> majority > >> > > > agrees > >> > > > >> the > >> > > > >> >> > added > >> > > > >> >> > >> >> > complexity to have controller forwarding keys to > all > >> > > > broker is > >> > > > >> >> > justified > >> > > > >> >> > >> >> > as > >> > > > >> >> > >> >> > it provides tighter security , I am fine with that > >> option > >> > > > too. > >> > > > >> >> > >> >> > > >> > > > >> >> > >> >> > Given zookeeper does not support SSL we can not > >> store > >> > > > master > >> > > > >> keys > >> > > > >> >> > in > >> > > > >> >> > >> >> > zookeeper as master keys will be exposed on wire. > To > >> > > > support > >> > > > >> >> > rotation > >> > > > >> >> > >> >> > without affecting current clients is something I > >> need to > >> > > > put > >> > > > >> more > >> > > > >> >> > thought > >> > > > >> >> > >> >> > in. My current proposal assumes the rotation will > >> > > > invalidate > >> > > > >> all > >> > > > >> >> > current > >> > > > >> >> > >> >> > tokens. > >> > > > >> >> > >> >> > > >> > > > >> >> > >> >> > I request committers to also review and post their > >> > > comments > >> > > > >> so we > >> > > > >> >> > can > >> > > > >> >> > >> >> > make > >> > > > >> >> > >> >> > progress on this KIP. > >> > > > >> >> > >> >> > > >> > > > >> >> > >> >> > Thanks > >> > > > >> >> > >> >> > Parth > >> > > > >> >> > >> >> > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh < > >> > > > >> asi...@cloudera.com > >> > > > >> >> > > > >> > > > >> >> > >> >> > wrote: > >> > > > >> >> > >> >> > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha < > >> > > > ka...@harsha.io> > >> > > > >> >> > wrote: > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > > > Unifying the two discussion threads on this > KIP. > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > Here is the response from Jitendra > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > "The need for a large number of clients that > are > >> > > > running > >> > > > >> all > >> > > > >> >> > over the > >> > > > >> >> > >> >> > > > cluster that authenticate with Kafka brokers, > >> is very > >> > > > >> similar > >> > > > >> >> > to the > >> > > > >> >> > >> >> > > > Hadoop use case of large number of tasks > running > >> > > across > >> > > > >> the > >> > > > >> >> > cluster > >> > > > >> >> > >> >> that > >> > > > >> >> > >> >> > > > need authentication to Hdfs Namenode. > >> Therefore, the > >> > > > >> >> > delegation token > >> > > > >> >> > >> >> > > > approach does seem like a good fit for this > use > >> case > >> > > > as we > >> > > > >> >> > have seen > >> > > > >> >> > >> >> it > >> > > > >> >> > >> >> > > > working at large scale in HDFS and YARN. > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > The proposed design is very much inline with > >> Hadoop > >> > > > >> >> > approach. A few > >> > > > >> >> > >> >> > > > comments: > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow infinite > >> renewable > >> > > > >> lifetime > >> > > > >> >> > for a > >> > > > >> >> > >> >> > > > token? HDFS restricts a token to a max life > time > >> > > > (default > >> > > > >> 7 > >> > > > >> >> > days). A > >> > > > >> >> > >> >> > > > token's vulnerability is believed to increase > >> with > >> > > > time. > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > I agree that having infinite lifetime might not > >> be the > >> > > > best > >> > > > >> idea. > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > 2) As I understand the tokens are stored in > >> zookeeper > >> > > > as > >> > > > >> well, > >> > > > >> >> > and > >> > > > >> >> > >> >> can > >> > > > >> >> > >> >> > > > be updated there. This is clever as it can > allow > >> > > > >> replacing the > >> > > > >> >> > tokens > >> > > > >> >> > >> >> > > > once they run out of max life time, and > clients > >> can > >> > > > >> download > >> > > > >> >> > new > >> > > > >> >> > >> >> tokens > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big load on > >> > > zookeeper > >> > > > >> as a > >> > > > >> >> > client > >> > > > >> >> > >> >> will > >> > > > >> >> > >> >> > > > need to get a new token once in several days. > >> In this > >> > > > >> approach > >> > > > >> >> > you > >> > > > >> >> > >> >> don't > >> > > > >> >> > >> >> > > > need infinite lifetime on the token even for > >> long > >> > > > running > >> > > > >> >> > clients. > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > 3) The token password are generated using a > >> master > >> > > key. > >> > > > >> The > >> > > > >> >> > master > >> > > > >> >> > >> >> key > >> > > > >> >> > >> >> > > > should also be periodically changed. In > Hadoop, > >> the > >> > > > >> default > >> > > > >> >> > renewal > >> > > > >> >> > >> >> > > > period is 1 day.? > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > IIUC, this will require brokers maintaining a > >> list of X > >> > > > most > >> > > > >> >> > recent > >> > > > >> >> > >> >> master > >> > > > >> >> > >> >> > > keys. This list will have to be persisted > >> somewhere, as > >> > > > if a > >> > > > >> >> > broker > >> > > > >> >> > >> >> goes > >> > > > >> >> > >> >> > > down it will have to get that list again and > >> storing > >> > > > master > >> > > > >> keys > >> > > > >> >> > on ZK > >> > > > >> >> > >> >> is > >> > > > >> >> > >> >> > > not the best idea. However, if a broker goes > down > >> then > >> > > we > >> > > > >> have > >> > > > >> >> > much > >> > > > >> >> > >> >> bigger > >> > > > >> >> > >> >> > > issue to deal with and client can always > >> > > re-authenticate > >> > > > is > >> > > > >> such > >> > > > >> >> > >> >> events. > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > > Did you happen to take a look at other > >> alternatives > >> > > this > >> > > > >> list has > >> > > > >> >> > >> >> > > suggested? > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal, great work!" > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira > >> wrote: > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks! > >> > > > >> >> > >> >> > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha < > >> > > > ka...@harsha.io > >> > > > >> > > >> > > > >> >> > wrote: > >> > > > >> >> > >> >> > > > > > It doesn't need any release vehicle but > >> still the > >> > > > >> work can > >> > > > >> >> > move > >> > > > >> >> > >> >> > > > forward. > >> > > > >> >> > >> >> > > > > > If anyone is interested in the KIP please > >> do the > >> > > > >> review and > >> > > > >> >> > >> >> provide > >> > > > >> >> > >> >> > > the > >> > > > >> >> > >> >> > > > > > comments. > >> > > > >> >> > >> >> > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha > >> > > > >> >> > >> >> > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael > >> Juma > >> > > > wrote: > >> > > > >> >> > >> >> > > > > >> I agree that it would be good to have > more > >> time > >> > > to > >> > > > >> review > >> > > > >> >> > and > >> > > > >> >> > >> >> > > discuss > >> > > > >> >> > >> >> > > > > >> KIP-48. > >> > > > >> >> > >> >> > > > > >> > >> > > > >> >> > >> >> > > > > >> Ismael > >> > > > >> >> > >> >> > > > > >> > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen > >> Shapira < > >> > > > >> >> > >> >> g...@confluent.io> > >> > > > >> >> > >> >> > > > wrote: > >> > > > >> >> > >> >> > > > > >> > >> > > > >> >> > >> >> > > > > >> > Hi Team, > >> > > > >> >> > >> >> > > > > >> > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which > is > >> > > > already a > >> > > > >> bit > >> > > > >> >> > of a > >> > > > >> >> > >> >> risk > >> > > > >> >> > >> >> > > for > >> > > > >> >> > >> >> > > > > >> > the next release - any chance we can > >> delay > >> > > > >> delegation > >> > > > >> >> > tokens > >> > > > >> >> > >> >> to > >> > > > >> >> > >> >> > > > Kafka > >> > > > >> >> > >> >> > > > > >> > 0.10.1? > >> > > > >> >> > >> >> > > > > >> > With the community working on a release > >> every > >> > > 3 > >> > > > >> month, > >> > > > >> >> > this > >> > > > >> >> > >> >> is not > >> > > > >> >> > >> >> > > > a huge > >> > > > >> >> > >> >> > > > > >> > delay. > >> > > > >> >> > >> >> > > > > >> > > >> > > > >> >> > >> >> > > > > >> > Gwen > >> > > > >> >> > >> >> > > > > >> > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish > >> Singh > >> > > < > >> > > > >> >> > >> >> > > asi...@cloudera.com> > >> > > > >> >> > >> >> > > > wrote: > >> > > > >> >> > >> >> > > > > >> > > Parth, > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the awesome write > up. > >> > > > Following > >> > > > >> our > >> > > > >> >> > >> >> discussion > >> > > > >> >> > >> >> > > > from the > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to > >> compare > >> > > > >> various > >> > > > >> >> > >> >> alternatives > >> > > > >> >> > >> >> > > > if they > >> > > > >> >> > >> >> > > > > >> > are > >> > > > >> >> > >> >> > > > > >> > > listed together. I am stating below a > >> few > >> > > > >> >> > alternatives along > >> > > > >> >> > >> >> > > with > >> > > > >> >> > >> >> > > > a the > >> > > > >> >> > >> >> > > > > >> > > current proposal. > >> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation > >> Token, > >> > > DT, > >> > > > >> on ZK. > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > 1. Client authenticates with a > >> broker. > >> > > > >> >> > >> >> > > > > >> > > 2. Once a client is authenticated, > >> it > >> > > will > >> > > > >> make a > >> > > > >> >> > broker > >> > > > >> >> > >> >> side > >> > > > >> >> > >> >> > > > call to > >> > > > >> >> > >> >> > > > > >> > > issue a delegation token. > >> > > > >> >> > >> >> > > > > >> > > 3. The broker generates a shared > >> secret > >> > > > based > >> > > > >> on > >> > > > >> >> > >> >> > > HMAC-SHA256(a > >> > > > >> >> > >> >> > > > > >> > > Password/Secret shared between all > >> > > brokers, > >> > > > >> >> > randomly > >> > > > >> >> > >> >> > > generated > >> > > > >> >> > >> >> > > > > >> > tokenId). > >> > > > >> >> > >> >> > > > > >> > > 4. Broker stores this token in its > >> in > >> > > > memory > >> > > > >> cache. > >> > > > >> >> > >> >> Broker > >> > > > >> >> > >> >> > > > also stores > >> > > > >> >> > >> >> > > > > >> > > the DelegationToken without the > >> hmac in > >> > > the > >> > > > >> >> > zookeeper. > >> > > > >> >> > >> >> > > > > >> > > 5. All brokers will have a cache > >> backed > >> > > by > >> > > > >> >> > zookeeper so > >> > > > >> >> > >> >> they > >> > > > >> >> > >> >> > > > will all > >> > > > >> >> > >> >> > > > > >> > > get notified whenever a new token > is > >> > > > >> generated and > >> > > > >> >> > they > >> > > > >> >> > >> >> will > >> > > > >> >> > >> >> > > > update > >> > > > >> >> > >> >> > > > > >> > their > >> > > > >> >> > >> >> > > > > >> > > local cache whenever token state > >> changes. > >> > > > >> >> > >> >> > > > > >> > > 6. Broker returns the token to > >> Client. > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > 1. Probable race condition, client > >> tries > >> > > to > >> > > > >> >> > authenticate > >> > > > >> >> > >> >> with > >> > > > >> >> > >> >> > > > a broker > >> > > > >> >> > >> >> > > > > >> > > that is yet to be updated with the > >> newly > >> > > > >> generated > >> > > > >> >> > DT. > >> > > > >> >> > >> >> This > >> > > > >> >> > >> >> > > can > >> > > > >> >> > >> >> > > > > >> > probably be > >> > > > >> >> > >> >> > > > > >> > > dealt with making dtRequest block > >> until > >> > > all > >> > > > >> >> > brokers have > >> > > > >> >> > >> >> > > > updated > >> > > > >> >> > >> >> > > > > >> > their DT > >> > > > >> >> > >> >> > > > > >> > > cache. Zk barrier or similar > >> mechanism > >> > > can > >> > > > be > >> > > > >> used. > >> > > > >> >> > >> >> However, > >> > > > >> >> > >> >> > > > all such > >> > > > >> >> > >> >> > > > > >> > > mechanisms will increase > complexity. > >> > > > >> >> > >> >> > > > > >> > > 2. Using a static secret key from > >> config > >> > > > >> file. Will > >> > > > >> >> > >> >> require > >> > > > >> >> > >> >> > > yet > >> > > > >> >> > >> >> > > > > >> > another > >> > > > >> >> > >> >> > > > > >> > > config and uses a static secret > >> key. It > >> > > is > >> > > > >> advised > >> > > > >> >> > to > >> > > > >> >> > >> >> rotate > >> > > > >> >> > >> >> > > > secret > >> > > > >> >> > >> >> > > > > >> > keys > >> > > > >> >> > >> >> > > > > >> > > periodically. This can be avoided > >> with > >> > > > >> controller > >> > > > >> >> > >> >> generating > >> > > > >> >> > >> >> > > > > >> > secretKey and > >> > > > >> >> > >> >> > > > > >> > > passing to brokers periodically. > >> However, > >> > > > >> this will > >> > > > >> >> > >> >> require > >> > > > >> >> > >> >> > > > brokers to > >> > > > >> >> > >> >> > > > > >> > > maintain certain counts of > >> secretKeys. > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller > >> generate > >> > > > >> delegation > >> > > > >> >> > token. > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > 1. Client authenticates with a > >> broker. > >> > > > >> >> > >> >> > > > > >> > > 2. Once a client is authenticated, > >> it > >> > > will > >> > > > >> make a > >> > > > >> >> > broker > >> > > > >> >> > >> >> side > >> > > > >> >> > >> >> > > > call to > >> > > > >> >> > >> >> > > > > >> > > issue a delegation token. > >> > > > >> >> > >> >> > > > > >> > > 3. Broker forwards the request to > >> > > > controller. > >> > > > >> >> > >> >> > > > > >> > > 4. Controller generates a DT and > >> > > broadcasts > >> > > > >> to all > >> > > > >> >> > >> >> brokers. > >> > > > >> >> > >> >> > > > > >> > > 5. Broker stores this token in its > >> memory > >> > > > >> cache. > >> > > > >> >> > >> >> > > > > >> > > 6. Controller responds to broker’s > >> DT > >> > > req. > >> > > > >> >> > >> >> > > > > >> > > 7. Broker returns the token to > >> Client. > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > 1. We will have to add new APIs to > >> > > support > >> > > > >> >> > controller > >> > > > >> >> > >> >> pushing > >> > > > >> >> > >> >> > > > tokens > >> > > > >> >> > >> >> > > > > >> > to > >> > > > >> >> > >> >> > > > > >> > > brokers on top of the minimal APIs > >> that > >> > > are > >> > > > >> >> > currently > >> > > > >> >> > >> >> > > proposed. > >> > > > >> >> > >> >> > > > > >> > > 2. We will also have to add APIs > to > >> > > support > >> > > > >> the > >> > > > >> >> > >> >> bootstrapping > >> > > > >> >> > >> >> > > > case, > >> > > > >> >> > >> >> > > > > >> > i.e, > >> > > > >> >> > >> >> > > > > >> > > when a new broker comes up it will > >> have > >> > > to > >> > > > >> get all > >> > > > >> >> > >> >> delegation > >> > > > >> >> > >> >> > > > tokens > >> > > > >> >> > >> >> > > > > >> > from > >> > > > >> >> > >> >> > > > > >> > > the controller. > >> > > > >> >> > >> >> > > > > >> > > 3. In catastrophic failures where > >> all > >> > > > brokers > >> > > > >> go > >> > > > >> >> > down, > >> > > > >> >> > >> >> the > >> > > > >> >> > >> >> > > > tokens will > >> > > > >> >> > >> >> > > > > >> > > be lost even if servers are > >> restarted as > >> > > > >> tokens > >> > > > >> >> > are not > >> > > > >> >> > >> >> > > > persisted > >> > > > >> >> > >> >> > > > > >> > anywhere. > >> > > > >> >> > >> >> > > > > >> > > If this happens, then there are > more > >> > > > important > >> > > > >> >> > things to > >> > > > >> >> > >> >> > > worry > >> > > > >> >> > >> >> > > > about > >> > > > >> >> > >> >> > > > > >> > and > >> > > > >> >> > >> >> > > > > >> > > maybe it is better to > >> re-authenticate. > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT > to > >> > > other > >> > > > >> brokers > >> > > > >> >> > at > >> > > > >> >> > >> >> all. > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > 1. Client authenticates with a > >> broker. > >> > > > >> >> > >> >> > > > > >> > > 2. Once a client is authenticated, > >> it > >> > > will > >> > > > >> make a > >> > > > >> >> > broker > >> > > > >> >> > >> >> side > >> > > > >> >> > >> >> > > > call to > >> > > > >> >> > >> >> > > > > >> > > issue a delegation token. > >> > > > >> >> > >> >> > > > > >> > > 3. The broker generates DT of > form, > >> > > [hmac + > >> > > > >> (owner, > >> > > > >> >> > >> >> renewer, > >> > > > >> >> > >> >> > > > > >> > > maxLifeTime, id, hmac, > >> expirationTime)] > >> > > and > >> > > > >> passes > >> > > > >> >> > back > >> > > > >> >> > >> >> this > >> > > > >> >> > >> >> > > > DT to > >> > > > >> >> > >> >> > > > > >> > client. > >> > > > >> >> > >> >> > > > > >> > > hmac is generated via > >> {HMAC-SHA256(owner, > >> > > > >> renewer, > >> > > > >> >> > >> >> > > > maxLifeTime, id, > >> > > > >> >> > >> >> > > > > >> > hmac, > >> > > > >> >> > >> >> > > > > >> > > expirationTime) using SecretKey}. > >> Note > >> > > that > >> > > > >> all > >> > > > >> >> > brokers > >> > > > >> >> > >> >> have > >> > > > >> >> > >> >> > > > this > >> > > > >> >> > >> >> > > > > >> > SecretKey. > >> > > > >> >> > >> >> > > > > >> > > 4. Client then goes to any broker > >> and to > >> > > > >> >> > authenticate > >> > > > >> >> > >> >> sends > >> > > > >> >> > >> >> > > > the DT. > >> > > > >> >> > >> >> > > > > >> > > Broker recalculates hmac using > >> (owner, > >> > > > >> renewer, > >> > > > >> >> > >> >> maxLifeTime, > >> > > > >> >> > >> >> > > > id, hmac, > >> > > > >> >> > >> >> > > > > >> > > expirationTime) info from DT and > its > >> > > > >> SecretKey. If > >> > > > >> >> > it > >> > > > >> >> > >> >> matches > >> > > > >> >> > >> >> > > > with > >> > > > >> >> > >> >> > > > > >> > hmac of > >> > > > >> >> > >> >> > > > > >> > > DT, client is authenticated. Yes, > >> it will > >> > > > do > >> > > > >> other > >> > > > >> >> > >> >> obvious > >> > > > >> >> > >> >> > > > checks of > >> > > > >> >> > >> >> > > > > >> > > timestamp expiry and such. > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret key will be > generated > >> by > >> > > > >> controller > >> > > > >> >> > and > >> > > > >> >> > >> >> passed > >> > > > >> >> > >> >> > > to > >> > > > >> >> > >> >> > > > > >> > brokers > >> > > > >> >> > >> >> > > > > >> > > periodically. > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > 1. How to delete a DT? Yes, that > is > >> a > >> > > > downside > >> > > > >> >> > here. > >> > > > >> >> > >> >> However, > >> > > > >> >> > >> >> > > > this can > >> > > > >> >> > >> >> > > > > >> > > be handled with brokers > maintaining > >> a > >> > > > >> blacklist of > >> > > > >> >> > DTs, > >> > > > >> >> > >> >> DTs > >> > > > >> >> > >> >> > > > from this > >> > > > >> >> > >> >> > > > > >> > list > >> > > > >> >> > >> >> > > > > >> > > can be removed after expiry. > >> > > > >> >> > >> >> > > > > >> > > 2. In catastrophic failures where > >> all > >> > > > brokers > >> > > > >> go > >> > > > >> >> > down, > >> > > > >> >> > >> >> the > >> > > > >> >> > >> >> > > > tokens will > >> > > > >> >> > >> >> > > > > >> > > be lost even if servers are > >> restarted as > >> > > > >> tokens > >> > > > >> >> > are not > >> > > > >> >> > >> >> > > > persisted > >> > > > >> >> > >> >> > > > > >> > anywhere. > >> > > > >> >> > >> >> > > > > >> > > If this happens, then there are > more > >> > > > important > >> > > > >> >> > things to > >> > > > >> >> > >> >> > > worry > >> > > > >> >> > >> >> > > > about > >> > > > >> >> > >> >> > > > > >> > and > >> > > > >> >> > >> >> > > > > >> > > maybe it is better to > >> re-authenticate. > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, > Parth > >> > > > >> Brahmbhatt < > >> > > > >> >> > >> >> > > > > >> > > pbrahmbh...@hortonworks.com> wrote: > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > >> Hi, > >> > > > >> >> > >> >> > > > > >> > >> > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer > >> hadoop > >> > > > like > >> > > > >> >> > delegation > >> > > > >> >> > >> >> > > > tokens in > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review the design > >> > > > >> >> > >> >> > > > > >> > >> > >> > > > >> >> > >> >> > > > > >> > > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > >> > > > >> >> > > >> > > > >> > >> > > > > >> > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka > >> > > > >> >> > >> >> > > > > >> > . > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we > >> have also > >> > > > >> >> > discussed an > >> > > > >> >> > >> >> > > > alternative to > >> > > > >> >> > >> >> > > > > >> > >> proposed design here< > >> > > > >> >> > >> >> > > > > >> > >> > >> > > > >> >> > >> >> > > > > >> > > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > >> > > > >> >> > > >> > > > >> > >> > > > > >> > > > >> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800 > >> > > > >> >> > >> >> > > > > >> > >> >. > >> > > > >> >> > >> >> > > > > >> > >> > >> > > > >> >> > >> >> > > > > >> > >> Thanks > >> > > > >> >> > >> >> > > > > >> > >> Parth > >> > > > >> >> > >> >> > > > > >> > >> > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > -- > >> > > > >> >> > >> >> > > > > >> > > > >> > > > >> >> > >> >> > > > > >> > > Regards, > >> > > > >> >> > >> >> > > > > >> > > Ashish > >> > > > >> >> > >> >> > > > > >> > > >> > > > >> >> > >> >> > > > > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > > -- > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > > Regards, > >> > > > >> >> > >> >> > > Ashish > >> > > > >> >> > >> >> > > > >> > > > >> >> > >> >> > >> > > > >> >> > > >> > > > >> > >> > > > > >> > > > >> > >> > > >
