I am not saying that the developer at Example Inc. would develop a Login
implementation that combines Smartcard and Kerberos because Retailer uses
both. I am saying that Example Inc develops the LoginModule (similar to JVM
security providers developing Kerberos modules). But there is no standard
interface for Login to allow ticket refresh. So, it is very unlikely that
Example Inc would develop a Login implementation that works with an Apache
Kafka defined interface ( Kafka developers wrote this code for Kerberos).
For a custom integration, the user (i.e. Retailer) would be expected to
develop this code if required.
You could imagine that Smartcard is a commonly used mechanism and a 3rd
party develops code for integrating Smartcard with Kafka and makes the
integration code (Login and CallbackHandler implementation) widely
available, If Retailer wants to use clients or a broker with just Smartcard
enabled in their broker, they configure Kafka to use the 3rd party code,
with no additional code development. But to combine Smartcard and Kerberos,
Retailer needs to write a few lines of code to incorporate both Smartcard
and Kerberos. I believe this is an edge case.
Smartcard is likely to use standard NameCallback and PasswordCallback
already implemented in Kafka and Kerberos support exists in Kafka. So it is
very likely that Retailer doesn't need to override Login or CallbackHandler
in this case. And it would just be a matter of configuring the mechanisms.
On Wed, Mar 9, 2016 at 12:48 AM, Gwen Shapira <g...@confluent.io> wrote:
> "Since smart card logins are not built into Kafka (or the JDK), you need a
> developer to build the login module. So the developer implements
> example.SmartcardLoginModule. In addition, the developer may also implement
> callback handlers for the SASL client or server and a login class to keep
> this login refreshed. The callback handlers and login implementation
> support all the mechanisms that the organisation supports - in this case
> Kerberos and smart card."
>
> In this case, the developer works for Example Inc (which develops
> SmartCard authentication modules), while I work for Retailer and need
> to use his module.
> You assume that developer from Example Inc knows about all the
> mechanisms that I have enabled in my Kafka cluster and he is capable
> of developing a Login class and maybe a callback handler that supports
> all of them. I think this is an unreasonable expectation.
>
> I think that allowing plug in of Login method and single Callback
> handler is not a good extensible solution for multiple mechanisms. We
> should probably supply a Kafka-based LoginContext and CallbackHandler
> that will be able to delegate to implementations of LoginModules and
> Callback Handlers based on mechanism out of a list supplied by a user.
> I agree it is more complex, but otherwise we are asking of too much
> coordination from whoever implements CallbackHandlers and Login
> methods.
>
> Gwen
>
> On Tue, Mar 8, 2016 at 3:35 PM, Rajini Sivaram
> <rajinisiva...@googlemail.com> wrote:
> > As an admin, you would configure login modules using standard JAAS
> > configuration. So, if you are supporting Kerberos and smart card login
> > modules within a single broker, in your jaas.conf you would specify:
> >
> > KafkaServer {
> >
> > com.ibm.security.auth.module.Krb5LoginModule required
> > credsType=both
> > useKeytab="file:/kafka/key.tab"
> > principal="kafka/localh...@example.com";
> >
> > example.SmartcardLoginModule required
> >
> > cardNumber=123
> >
> > };
> >
> >
> > Since smart card logins are not built into Kafka (or the JDK), you need a
> > developer to build the login module. So the developer implements
> > example.SmartcardLoginModule. In addition, the developer may also
> implement
> > callback handlers for the SASL client or server and a login class to
> keep
> > this login refreshed. The callback handlers and login implementation
> > support all the mechanisms that the organisation supports - in this case
> > Kerberos and smart card.
> >
> > As an admin, if you are using mechanisms built into Kafka, you simply
> > specify a JAAS config with the combination of mechanisms you want and you
> > are done. If you are adding new mechanisms, it clearly can't be done
> > without some additional code to implement that mechanism. The callback
> and
> > login interfaces are configurable to ensure that (as a developer), you
> can
> > build custom mechanisms without modifying Kafka. SASL was designed
> > precisely to do this. But at the moment, Kafka restricts what type of
> > authentication you can perform with SASL.
> >
> > Back to the questions:
> > 1. *How does the proposed Login class interact with LoginContext and *
> > *LoginModule.*
> > Login class performs *LoginContext.login(). *This performs login for all
> > the login modules of that context (this is standard Java function). Login
> > is cached with a reference counter to keep it alive.
> >
> > 2. *Do we really need a pluggable Login class, when users
> **(administrators,
> > not developers) are more likely to want to specify LoginModules?*
> > Administrators specify *LoginModules. *Developers implement new
> mechanisms.
> > The pluggable *Login* class enables the developer to implement new
> > mechanisms without changing Kafka. If Kafka didn't have a default
> Kerberos
> > implementation and a user wanted to use Kerberos, pluggability of
> *Login* class
> > enables the user to include Kerberos without changing Kafka. Not that we
> > want another Kerberos, but Tao had a requirement to enable a custom
> > mechanism with functionality similar to Kerberos (see KIP-44). Pluggable
> > *Login* supports this use case without replacing the entire
> authentication
> > mechanism in Kafka.
> >
> > *3. How will users get to specify multiple LoginModules?*
> > Standard Java JAAS configuration - see example above.
> >
> >
> > *4. It sounds like different LoginModules may need
> > different CallbackHandlers. Since we only allow a single
> > pluggableCallbackHandler class in the KIP-43 configuration, how will this
> > be handled?*
> > One LoginContext, one subject, one callback handler, multiple login
> > modules. This is the Java model for login. And it works well because
> > callback handlers tend to be fairly standard. Even if the smart card SASL
> > callbacks needed to do something wildly different and you wanted a broker
> > that combined this with Kerberos, you would implement a callback that
> > looked like the code below.
> >
> > private CallbackHandler handler;
> >
> > public void configure(Map<String, ?> configs, Mode mode, Subject
> > subject, String mechanism) {public void configure(Map<String, ?> configs,
> > Mode mode, Subject subject, String mechanism) {
> >
> > switch (mechanism) {
> >
> > case "GSSAPI" : handler = new GssapiCallbackHandler();
> break
> > ;
> >
> > case "SMARTCARD" : handler = new
> SmartcardCallbackHandler();
> > break;
> >
> > }
> >
> > }
> >
> >
> > public void handle(Callback[] callbacks) throws IOException,
> > UnsupportedCallbackException {
> >
> > handler.handle(callbacks);
> >
> > }
> >
> > But I don't see this as just an admin configuration. This is some code
> > developed alongside a new mechanism. Yes, you can add a new configuration
> > type in Kafka that maps mechanism to callback handler class to avoid the
> > switch statement. But that doesn't remove the need for new code
> altogether
> > since you still need to implement SmartcardCallbackHandler. Multiple
> > conflicting callback handlers within one broker to support complete
> > different behaviour is an edge case. Since the design supports
> > full flexibility in terms of functionality and you have to write new code
> > for new mechanisms anyway, this feels like the simplest solution.
> >
> >
> >
> > On Tue, Mar 8, 2016 at 8:23 PM, Gwen Shapira <g...@confluent.io> wrote:
> >
> >> I guess what is unclear to me is:
> >>
> >> 1. How does the proposed Login class interact with LoginContext and
> >> LoginModule.
> >> 2. Do we really need a pluggable Login class, when users
> >> (administrators, not developers) are more likely to want to specify
> >> LoginModules?
> >> 3. How will users get to specify multiple LoginModules?
> >> 4. It sounds like different LoginModules may need different
> >> CallbackHandlers. Since we only allow a single pluggable
> >> CallbackHandler class in the KIP-43 configuration, how will this be
> >> handled?
> >>
> >> In other words, suppose I want my Kafka installation to support both
> >> Kerberos and SmartCard mechanisms, can you tell me how to configure
> >> Kafka? (as admin, not developer)?
> >>
> >> Gwen
> >>
> >>
> >>
> >> On Tue, Mar 8, 2016 at 11:46 AM, Rajini Sivaram
> >> <rajinisiva...@googlemail.com> wrote:
> >> > Gwen,
> >> >
> >> > I am not sure I understand your concern. Java login contexts support
> >> > multiple login modules precisely to support this type of real-world
> >> > scenario.
> >> >
> >> > From Javadoc of *javax.security.auth.login.LoginContext*:
> >> >
> >> > *In addition to supporting pluggable authentication, this class also
> >> > supports the notion of stacked authentication. Applications may be
> >> > configured to use more than one LoginModule. For example, one could
> >> > configure both a Kerberos LoginModule and a smart card LoginModule
> under
> >> an
> >> > application. *
> >> >
> >> > * A typical caller instantiates a LoginContext with a name and a
> >> > CallbackHandler. LoginContext uses the name as the index into a
> >> > Configuration to determine which LoginModules should be used, and
> which
> >> > ones must succeed in order for the overall authentication to succeed.
> The
> >> > CallbackHandler is passed to the underlying LoginModules so they may
> >> > communicate and interact with users (prompting for a username and
> >> password
> >> > via a graphical user interface, for example).*
> >> >
> >> > If it is good enough for Java, surely it must be good enough for
> Kafka.
> >> We
> >> > don't expect users to directly use existing classes when they want to
> >> > support complex new mechanisms. Login is not a standard interface,
> and we
> >> > want to make callback handlers configurable. So we do require users to
> >> wrap
> >> > their existing classes in our interface. And the interfaces make it
> >> > possible to plugin any number of mechanisms. Am I missing something?
> >> >
> >> >
> >> >
> >> > On Tue, Mar 8, 2016 at 4:49 PM, Gwen Shapira <g...@confluent.io>
> wrote:
> >> >
> >> >> Yes, I understand that.
> >> >>
> >> >> However, because the current configuration only allows a single
> >> >> callback and a single login module, I need to supply a class that
> >> >> supports callbacks or logins for every single mechanism that I need
> to
> >> >> support. I question whether this is applicable in real-world scenario
> >> >> where a user may need to support mechanisms that were written by
> >> >> different vendors and where a single class that implements all the
> >> >> necessary modules or callsbacks is very unlikely to exist.
> >> >>
> >> >> I agree that supporting a list of classes and dynamically using the
> >> >> correct callbacks and modules based on configuration is complex, but
> I
> >> >> don't think this feature is useful enough without it.
> >> >>
> >> >> Gwen
> >> >>
> >> >> On Tue, Mar 8, 2016 at 2:40 AM, Rajini Sivaram
> >> >> <rajinisiva...@googlemail.com> wrote:
> >> >> > Gwen,
> >> >> >
> >> >> > Sorry if this is not clear in the KIP. The goal of the KIP is to
> >> enable
> >> >> new
> >> >> > mechanisms to be added without any changes to Apache Kafka. The
> >> >> > configuration proposed by the KIP is based on the configurability
> of
> >> the
> >> >> > Java SASL API and what it would take to add a new mechanism
> similar to
> >> >> > Kerberos/PLAIN/Digest-MD5 and the requirements that Tao raised for
> >> custom
> >> >> > authentication.
> >> >> >
> >> >> > The PR includes a test for Digest-MD5 which is not included in
> Kafka,
> >> but
> >> >> > is tested using the configuration options that allow new mechanisms
> >> to be
> >> >> > added.
> >> >> >
> >> >> > If you want to add a more complex mechanism which requires
> integration
> >> >> with
> >> >> > some external authentication libraries for login or callback
> handlers,
> >> >> you
> >> >> > can do that by configuring your own callback handler and login
> >> >> > implementation. The login class allows you to add additional
> threads
> >> (for
> >> >> > instance for token refresh), while the callback handler extends the
> >> >> > standard Java CallbackHandler interface, enabling you to override
> any
> >> of
> >> >> > the default callbacks in Kafka to provide your own behaviour. If
> you
> >> want
> >> >> > to add your own login modules, SASL server or SASL client
> >> implementation
> >> >> > for a mechanism that is not supported in your JDK, you can do that
> by
> >> >> > adding new Java security modules, using the built-in pluggability
> of
> >> Java
> >> >> > security.
> >> >> >
> >> >> > And finally, if you want to enable multiple mechanisms in your
> Kafka
> >> >> > brokers, you can specify a callback handler class that either
> provides
> >> >> the
> >> >> > handlers for all the mechanisms (as the PR does for Kerberos and
> >> PLAIN)
> >> >> or
> >> >> > a delegator that calls different callback handlers based on the
> >> >> mechanism.
> >> >> > And for login, you provide a single or a compound class that
> provides
> >> any
> >> >> > additional login logic for all the login modules that you have
> >> specified
> >> >> > for KafkaServer.
> >> >> >
> >> >> > Hope that helps.
> >> >> >
> >> >> >
> >> >> > On Tue, Mar 8, 2016 at 1:17 AM, Gwen Shapira <g...@confluent.io>
> >> wrote:
> >> >> >
> >> >> >> Can you explain the process for a adding a new mechanism based on
> >> >> current
> >> >> >> KIP?
> >> >> >>
> >> >> >> My thought is that if it requires modifying Apache Kafka code, it
> is
> >> >> >> not pluggable enough.
> >> >> >>
> >> >> >> On Mon, Mar 7, 2016 at 4:04 PM, Rajini Sivaram
> >> >> >> <rajinisiva...@googlemail.com> wrote:
> >> >> >> > Gwen,
> >> >> >> >
> >> >> >> > In cases where you want completely different callbacks for
> >> different
> >> >> >> > mechanisms, I was thinking that the choice would be between a
> map
> >> of
> >> >> >> > classes (mechanism -> callbackHandler class) or a delegator
> class
> >> that
> >> >> >> > chooses the appropriate callback handler class based on
> mechanism.
> >> I
> >> >> >> chose
> >> >> >> > the latter since it makes it easier to configure in Kafka.
> Since we
> >> >> >> create
> >> >> >> > a callback handler for each channel and configure it with the
> >> >> >> > client-selected mechanism, it is straightforward to have one
> >> wrapper
> >> >> >> class
> >> >> >> > that delegates to the right mechanism-specific class to handle
> >> >> callbacks.
> >> >> >> > In many cases, a single class may be sufficient (the PR uses a
> >> single
> >> >> >> > callback class for Kerberos and PLAIN). I do see your point
> about
> >> the
> >> >> >> > flexibility that multiple classes would provide, but since you
> need
> >> >> to be
> >> >> >> > able to associate the callback with a mechanism for this to be
> >> >> useful, I
> >> >> >> am
> >> >> >> > not sure if just a list would add value.
> >> >> >> >
> >> >> >> > Login class is slightly different since the proposal is to use a
> >> >> single
> >> >> >> > login context with multiple login modules to handle multiple
> >> >> mechanisms.
> >> >> >> In
> >> >> >> > this case, you want to perform login for all the mechanisms that
> >> are
> >> >> >> > enabled. And you want to call loginContext.login() only once.
> >> Again,
> >> >> you
> >> >> >> > can delegate to multiple classes if you wish to add some complex
> >> >> >> > mechanism-specific logic, but a single login class makes the
> >> mapping
> >> >> to a
> >> >> >> > single login context and the login cache more obvious (the PR
> has a
> >> >> test
> >> >> >> > that includes Kerberos and PLAIN).
> >> >> >> >
> >> >> >> > Thoughts?
> >> >> >> >
> >> >> >> > On Mon, Mar 7, 2016 at 9:57 PM, Gwen Shapira <g...@confluent.io
> >
> >> >> wrote:
> >> >> >> >
> >> >> >> >> Hi,
> >> >> >> >>
> >> >> >> >> Before I can vote on this KIP, I have two additional questions
> /
> >> >> >> >> comments on the new configuration:
> >> >> >> >>
> >> >> >> >> 1. sasl.callback.handler.class - it looks like we want a single
> >> class
> >> >> >> >> that implements all mechanisms. I think this will make it
> >> difficult
> >> >> to
> >> >> >> >> extend since the only way I can add a mechanism will be by
> >> >> >> >> implementing every single existing mechanism (otherwise
> customers
> >> >> will
> >> >> >> >> need to choose between new and existing when selecting which
> >> class to
> >> >> >> >> use). If Microsoft releases a proprietary "AD Mechanism" and
> >> Oracle
> >> >> >> >> releases "OID mechanism", there will be no class that
> implements
> >> >> both.
> >> >> >> >> Can we make it a list of classes instead? I realize this
> >> complicates
> >> >> >> >> the code a bit (some kind of factory will be required to choose
> >> the
> >> >> >> >> right class to use), but important IMO.
> >> >> >> >> 2. similar for sasl.login.class - if I have a class for
> Kerberos
> >> >> (with
> >> >> >> >> refresh thread) and a class for "plain", we need to be able to
> >> load
> >> >> >> >> both.
> >> >> >> >>
> >> >> >> >> Gwen
> >> >> >> >>
> >> >> >> >> On Wed, Mar 2, 2016 at 12:30 AM, Rajini Sivaram
> >> >> >> >> <rajinisiva...@googlemail.com> wrote:
> >> >> >> >> > Jun,
> >> >> >> >> >
> >> >> >> >> > Thanks, I have added a note to the KIP. I will add a comment
> in
> >> the
> >> >> >> >> > implementation and also add a unit test to ensure that
> conflicts
> >> >> are
> >> >> >> >> > avoided when version number is modified.
> >> >> >> >> >
> >> >> >> >> > On Tue, Mar 1, 2016 at 5:43 PM, Jun Rao <j...@confluent.io>
> >> wrote:
> >> >> >> >> >
> >> >> >> >> >> Rajini,
> >> >> >> >> >>
> >> >> >> >> >> Thanks for the explanation. For 1, this implies that we have
> >> to be
> >> >> >> >> careful
> >> >> >> >> >> with changing the 2-byte version in the future to avoid
> >> conflict.
> >> >> >> Could
> >> >> >> >> you
> >> >> >> >> >> document this in the KIP and also in the implementation?
> >> >> >> >> >>
> >> >> >> >> >> Jun
> >> >> >> >> >>
> >> >> >> >> >> On Tue, Mar 1, 2016 at 2:47 AM, Rajini Sivaram <
> >> >> >> >> >> rajinisiva...@googlemail.com
> >> >> >> >> >> > wrote:
> >> >> >> >> >>
> >> >> >> >> >> > Jun,
> >> >> >> >> >> >
> >> >> >> >> >> > Thank you for the review.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > 1. With GSSAPI, the first context establishment packet
> >> starts
> >> >> >> with
> >> >> >> >> the
> >> >> >> >> >> > byte 0x60 (tag for APPLICATION-0) followed by a
> >> >> variable-length
> >> >> >> >> >> encoded
> >> >> >> >> >> > size, followed by various tags and contents. And the
> >> packet
> >> >> also
> >> >> >> >> >> > contains a
> >> >> >> >> >> > checksum. This is completely different from the
> mechanism
> >> >> packet
> >> >> >> >> from
> >> >> >> >> >> > Kafka
> >> >> >> >> >> > clients which start with a two-byte version set to zero
> >> >> >> currently,
> >> >> >> >> >> > followed
> >> >> >> >> >> > by just a String mechanism.
> >> >> >> >> >> > 2. Agreed, I have removed the version from the server
> >> >> response
> >> >> >> in
> >> >> >> >> the
> >> >> >> >> >> > KIP. Thanks.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > On Tue, Mar 1, 2016 at 2:33 AM, Jun Rao <j...@confluent.io
> >
> >> >> wrote:
> >> >> >> >> >> >
> >> >> >> >> >> > > Rajini,
> >> >> >> >> >> > >
> >> >> >> >> >> > > Thanks for the updates. Just a couple of minor comments.
> >> >> >> >> >> > >
> >> >> >> >> >> > > 1. With the default GSSAPI, what's the first packet that
> >> the
> >> >> >> client
> >> >> >> >> >> sends
> >> >> >> >> >> > > to the server? Is that completely different from the
> packet
> >> >> >> format
> >> >> >> >> that
> >> >> >> >> >> > we
> >> >> >> >> >> > > will use for non-GSSAPI mechanisms?
> >> >> >> >> >> > >
> >> >> >> >> >> > > 2. In the server response, it doesn't seem that we need
> to
> >> >> >> include
> >> >> >> >> the
> >> >> >> >> >> > > version since the client knows the version of the
> request
> >> >> that it
> >> >> >> >> >> sends.
> >> >> >> >> >> > >
> >> >> >> >> >> > > Jun
> >> >> >> >> >> > >
> >> >> >> >> >> > > On Mon, Feb 29, 2016 at 10:14 AM, Rajini Sivaram <
> >> >> >> >> >> > > rajinisiva...@googlemail.com> wrote:
> >> >> >> >> >> > >
> >> >> >> >> >> > > > Harsha,
> >> >> >> >> >> > > >
> >> >> >> >> >> > > > Thank you for the review. I will wait another day to
> see
> >> if
> >> >> >> there
> >> >> >> >> is
> >> >> >> >> >> > more
> >> >> >> >> >> > > > feedback and then start a voting thread.
> >> >> >> >> >> > > >
> >> >> >> >> >> > > > Rajini
> >> >> >> >> >> > > >
> >> >> >> >> >> > > > On Mon, Feb 29, 2016 at 2:51 PM, Harsha <
> ka...@harsha.io
> >> >
> >> >> >> wrote:
> >> >> >> >> >> > > >
> >> >> >> >> >> > > > > Rajini,
> >> >> >> >> >> > > > > Thanks for the changes to the KIP. It
> >> looks
> >> >> >> good
> >> >> >> >> to
> >> >> >> >> >> > me. I
> >> >> >> >> >> > > > > think we can move to voting.
> >> >> >> >> >> > > > > Thanks,
> >> >> >> >> >> > > > > Harsha
> >> >> >> >> >> > > > >
> >> >> >> >> >> > > > > On Mon, Feb 29, 2016, at 12:43 AM, Rajini Sivaram
> >> wrote:
> >> >> >> >> >> > > > > > I have added some more detail to the KIP based on
> the
> >> >> >> >> discussion
> >> >> >> >> >> in
> >> >> >> >> >> > > the
> >> >> >> >> >> > > > > > last KIP meeting to simplify support for multiple
> >> >> >> mechanisms.
> >> >> >> >> >> Have
> >> >> >> >> >> > > also
> >> >> >> >> >> > > > > > changed the property names to reflect this.
> >> >> >> >> >> > > > > >
> >> >> >> >> >> > > > > > Also updated the PR in
> >> >> >> >> >> > > > https://issues.apache.org/jira/browse/KAFKA-3149
> >> >> >> >> >> > > > > > to
> >> >> >> >> >> > > > > > reflect the KIP.
> >> >> >> >> >> > > > > >
> >> >> >> >> >> > > > > > Any feedback is appreciated.
> >> >> >> >> >> > > > > >
> >> >> >> >> >> > > > > >
> >> >> >> >> >> > > > > > On Tue, Feb 23, 2016 at 9:36 PM, Rajini Sivaram <
> >> >> >> >> >> > > > > > rajinisiva...@googlemail.com> wrote:
> >> >> >> >> >> > > > > >
> >> >> >> >> >> > > > > > > I have updated the KIP based on the discussion
> in
> >> the
> >> >> KIP
> >> >> >> >> >> meeting
> >> >> >> >> >> > > > > today.
> >> >> >> >> >> > > > > > >
> >> >> >> >> >> > > > > > > Comments and feedback are welcome.
> >> >> >> >> >> > > > > > >
> >> >> >> >> >> > > > > > > On Wed, Feb 3, 2016 at 7:20 PM, Rajini Sivaram <
> >> >> >> >> >> > > > > > > rajinisiva...@googlemail.com> wrote:
> >> >> >> >> >> > > > > > >
> >> >> >> >> >> > > > > > >> Hi Harsha,
> >> >> >> >> >> > > > > > >>
> >> >> >> >> >> > > > > > >> Thank you for the review. Can you clarify - I
> >> think
> >> >> you
> >> >> >> are
> >> >> >> >> >> > saying
> >> >> >> >> >> > > > > that
> >> >> >> >> >> > > > > > >> the client should send its mechanism over the
> >> wire to
> >> >> >> the
> >> >> >> >> >> > server.
> >> >> >> >> >> > > Is
> >> >> >> >> >> > > > > that
> >> >> >> >> >> > > > > > >> correct? The exchange is slightly different in
> the
> >> >> KIP
> >> >> >> >> (the PR
> >> >> >> >> >> > > > > matches the
> >> >> >> >> >> > > > > > >> KIP) from the one you described to enable
> >> >> >> interoperability
> >> >> >> >> >> with
> >> >> >> >> >> > > > > 0.9.0.0.
> >> >> >> >> >> > > > > > >>
> >> >> >> >> >> > > > > > >>
> >> >> >> >> >> > > > > > >> On Wed, Feb 3, 2016 at 1:56 PM, Harsha <
> >> >> m...@harsha.io>
> >> >> >> >> >> wrote:
> >> >> >> >> >> > > > > > >>
> >> >> >> >> >> > > > > > >>> Rajini,
> >> >> >> >> >> > > > > > >>> I looked at the PR you have. I
> think
> >> its
> >> >> >> better
> >> >> >> >> >> with
> >> >> >> >> >> > > > your
> >> >> >> >> >> > > > > > >>> earlier approach rather than
> extending
> >> >> the
> >> >> >> >> >> protocol.
> >> >> >> >> >> > > > > > >>> What I was thinking initially is, Broker has a
> >> >> config
> >> >> >> >> option
> >> >> >> >> >> of
> >> >> >> >> >> > > say
> >> >> >> >> >> > > > > > >>> sasl.mechanism = GSSAPI, PLAIN
> >> >> >> >> >> > > > > > >>> and the client can have similar config of
> >> >> >> >> >> sasl.mechanism=PLAIN.
> >> >> >> >> >> > > > > Client
> >> >> >> >> >> > > > > > >>> can send its sasl mechanism before the
> handshake
> >> >> starts
> >> >> >> >> and
> >> >> >> >> >> if
> >> >> >> >> >> > > the
> >> >> >> >> >> > > > > > >>> broker accepts that particular mechanism than
> it
> >> >> can go
> >> >> >> >> ahead
> >> >> >> >> >> > > with
> >> >> >> >> >> > > > > > >>> handshake otherwise return a error saying that
> >> the
> >> >> >> >> mechanism
> >> >> >> >> >> > not
> >> >> >> >> >> > > > > > >>> allowed.
> >> >> >> >> >> > > > > > >>>
> >> >> >> >> >> > > > > > >>> Thanks,
> >> >> >> >> >> > > > > > >>> Harsha
> >> >> >> >> >> > > > > > >>>
> >> >> >> >> >> > > > > > >>> On Wed, Feb 3, 2016, at 04:58 AM, Rajini
> Sivaram
> >> >> wrote:
> >> >> >> >> >> > > > > > >>> > A slightly different approach for supporting
> >> >> >> different
> >> >> >> >> SASL
> >> >> >> >> >> > > > > mechanisms
> >> >> >> >> >> > > > > > >>> > within a broker is to allow the same
> "*security
> >> >> >> >> protocol*"
> >> >> >> >> >> to
> >> >> >> >> >> > > be
> >> >> >> >> >> > > > > used
> >> >> >> >> >> > > > > > >>> on
> >> >> >> >> >> > > > > > >>> > different ports with different configuration
> >> >> >> options. An
> >> >> >> >> >> > > > advantage
> >> >> >> >> >> > > > > of
> >> >> >> >> >> > > > > > >>> > this
> >> >> >> >> >> > > > > > >>> > approach is that it extends the
> >> configurability of
> >> >> >> not
> >> >> >> >> just
> >> >> >> >> >> > > SASL,
> >> >> >> >> >> > > > > but
> >> >> >> >> >> > > > > > >>> any
> >> >> >> >> >> > > > > > >>> > protocol. For instance, it would enable the
> >> use of
> >> >> >> SSL
> >> >> >> >> with
> >> >> >> >> >> > > > mutual
> >> >> >> >> >> > > > > > >>> client
> >> >> >> >> >> > > > > > >>> > authentication on one port or different
> >> >> certificate
> >> >> >> >> chains
> >> >> >> >> >> on
> >> >> >> >> >> > > > > another.
> >> >> >> >> >> > > > > > >>> > And
> >> >> >> >> >> > > > > > >>> > it avoids the need for SASL mechanism
> >> negotiation.
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > Kafka would have the same "*security
> protocols"
> >> >> >> >> *defined as
> >> >> >> >> >> > > > today,
> >> >> >> >> >> > > > > but
> >> >> >> >> >> > > > > > >>> > with
> >> >> >> >> >> > > > > > >>> > (a single) configurable SASL mechanism. To
> have
> >> >> >> >> different
> >> >> >> >> >> > > > > > >>> configurations
> >> >> >> >> >> > > > > > >>> > of
> >> >> >> >> >> > > > > > >>> > a protocol within a broker, users can define
> >> new
> >> >> >> >> protocol
> >> >> >> >> >> > names
> >> >> >> >> >> > > > > which
> >> >> >> >> >> > > > > > >>> are
> >> >> >> >> >> > > > > > >>> > configured versions of existing protocols,
> >> perhaps
> >> >> >> using
> >> >> >> >> >> just
> >> >> >> >> >> > > > > > >>> > configuration
> >> >> >> >> >> > > > > > >>> > entries and no additional code.
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > For example:
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > A single mechanism broker would be
> configured
> >> as:
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > listeners=SASL_SSL://:9092
> >> >> >> >> >> > > > > > >>> > sasl.mechanism=GSSAPI
> >> >> >> >> >> > > > > > >>> > sasl.kerberos.class.name=kafka
> >> >> >> >> >> > > > > > >>> > ...
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > And a multi-mechanism broker would be
> >> configured
> >> >> as:
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> >
> >> >> listeners=gssapi://:9092,plain://:9093,custom://:9094
> >> >> >> >> >> > > > > > >>> > gssapi.security.protocol=SASL_SSL
> >> >> >> >> >> > > > > > >>> > gssapi.sasl.mechanism=GSSAPI
> >> >> >> >> >> > > > > > >>> > gssapi.sasl.kerberos.class.name=kafka
> >> >> >> >> >> > > > > > >>> > ...
> >> >> >> >> >> > > > > > >>> > plain.security.protocol=SASL_SSL
> >> >> >> >> >> > > > > > >>> > plain.sasl.mechanism=PLAIN
> >> >> >> >> >> > > > > > >>> > ..
> >> >> >> >> >> > > > > > >>> > custom.security.protocol=SASL_PLAINTEXT
> >> >> >> >> >> > > > > > >>> > custom.sasl.mechanism=CUSTOM
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > >
> >> >> custom.sasl.callback.handler.class=example.CustomCallbackHandler
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > This is still a big change because it
> affects
> >> the
> >> >> >> >> currently
> >> >> >> >> >> > > fixed
> >> >> >> >> >> > > > > > >>> > enumeration of security protocol
> definitions,
> >> but
> >> >> one
> >> >> >> >> that
> >> >> >> >> >> is
> >> >> >> >> >> > > > > perhaps
> >> >> >> >> >> > > > > > >>> > more
> >> >> >> >> >> > > > > > >>> > flexible than defining every new SASL
> mechanism
> >> >> as a
> >> >> >> new
> >> >> >> >> >> > > security
> >> >> >> >> >> > > > > > >>> > protocol.
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > Thoughts?
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > On Tue, Feb 2, 2016 at 12:20 PM, Rajini
> >> Sivaram <
> >> >> >> >> >> > > > > > >>> > rajinisiva...@googlemail.com> wrote:
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > > As Ismael has said, we do not have a
> >> >> requirement to
> >> >> >> >> >> support
> >> >> >> >> >> > > > > multiple
> >> >> >> >> >> > > > > > >>> > > protocols in a broker. But I agree with
> Jun's
> >> >> >> >> observation
> >> >> >> >> >> > > that
> >> >> >> >> >> > > > > some
> >> >> >> >> >> > > > > > >>> > > companies might want to support a
> different
> >> >> >> >> >> authentication
> >> >> >> >> >> > > > > mechanism
> >> >> >> >> >> > > > > > >>> for
> >> >> >> >> >> > > > > > >>> > > internal users or partners. For instance,
> we
> >> do
> >> >> use
> >> >> >> >> two
> >> >> >> >> >> > > > different
> >> >> >> >> >> > > > > > >>> > > authentication mechanisms, it just so
> happens
> >> >> that
> >> >> >> we
> >> >> >> >> are
> >> >> >> >> >> > > able
> >> >> >> >> >> > > > > to use
> >> >> >> >> >> > > > > > >>> > > certificate-based authentication for
> internal
> >> >> >> users,
> >> >> >> >> and
> >> >> >> >> >> > > hence
> >> >> >> >> >> > > > > don't
> >> >> >> >> >> > > > > > >>> > > require multiple SASL mechanisms in a
> broker.
> >> >> >> >> >> > > > > > >>> > >
> >> >> >> >> >> > > > > > >>> > > As Tao has pointed out, mechanism
> negotiation
> >> >> is a
> >> >> >> >> common
> >> >> >> >> >> > > usage
> >> >> >> >> >> > > > > > >>> pattern.
> >> >> >> >> >> > > > > > >>> > > Many existing protocols that support SASL
> do
> >> >> >> already
> >> >> >> >> use
> >> >> >> >> >> > this
> >> >> >> >> >> > > > > > >>> pattern. AMQP
> >> >> >> >> >> > > > > > >>> > > (
> >> >> >> >> >> > > > > > >>> > >
> >> >> >> >> >> > > > > > >>>
> >> >> >> >> >> > > > >
> >> >> >> >> >> > > >
> >> >> >> >> >> > >
> >> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-security-v1.0-os.html#type-sasl-mechanisms
> >> >> >> >> >> > > > > > >>> ),
> >> >> >> >> >> > > > > > >>> > > which, as a messaging protocol maybe
> closer
> >> to
> >> >> >> Kafka
> >> >> >> >> in
> >> >> >> >> >> use
> >> >> >> >> >> > > > cases
> >> >> >> >> >> > > > > > >>> than
> >> >> >> >> >> > > > > > >>> > > Zookeeper, is an example. Other examples
> >> where
> >> >> the
> >> >> >> >> client
> >> >> >> >> >> > > > > negotiates
> >> >> >> >> >> > > > > > >>> or
> >> >> >> >> >> > > > > > >>> > > sends SASL mechanism to server include
> ACAP
> >> >> that is
> >> >> >> >> used
> >> >> >> >> >> as
> >> >> >> >> >> > > an
> >> >> >> >> >> > > > > > >>> example in
> >> >> >> >> >> > > > > > >>> > > the SASL RFCs, POP3, LDAP, SMTP etc. This
> is
> >> >> not to
> >> >> >> >> say
> >> >> >> >> >> > that
> >> >> >> >> >> > > > > Kafka
> >> >> >> >> >> > > > > > >>> > > shouldn't use a different type of
> mechanism
> >> >> >> selection
> >> >> >> >> >> that
> >> >> >> >> >> > > fits
> >> >> >> >> >> > > > > > >>> better with
> >> >> >> >> >> > > > > > >>> > > the existing Kafka design. Just that
> >> negotiation
> >> >> >> is a
> >> >> >> >> >> > common
> >> >> >> >> >> > > > > pattern
> >> >> >> >> >> > > > > > >>> and
> >> >> >> >> >> > > > > > >>> > > since we typically turn on
> javax.net.debug to
> >> >> debug
> >> >> >> >> TLS
> >> >> >> >> >> > > > > negotiation
> >> >> >> >> >> > > > > > >>> issues,
> >> >> >> >> >> > > > > > >>> > > having to use Kafka logging to debug SASL
> >> >> >> negotiation
> >> >> >> >> >> > issues
> >> >> >> >> >> > > is
> >> >> >> >> >> > > > > not
> >> >> >> >> >> > > > > > >>> that
> >> >> >> >> >> > > > > > >>> > > dissimilar.
> >> >> >> >> >> > > > > > >>> > >
> >> >> >> >> >> > > > > > >>> > >
> >> >> >> >> >> > > > > > >>> > > On Tue, Feb 2, 2016 at 6:12 AM, tao xiao <
> >> >> >> >> >> > > xiaotao...@gmail.com
> >> >> >> >> >> > > > >
> >> >> >> >> >> > > > > > >>> wrote:
> >> >> >> >> >> > > > > > >>> > >
> >> >> >> >> >> > > > > > >>> > >> I am the author of KIP-44. I hope my use
> >> case
> >> >> will
> >> >> >> >> add
> >> >> >> >> >> > some
> >> >> >> >> >> > > > > values
> >> >> >> >> >> > > > > > >>> to this
> >> >> >> >> >> > > > > > >>> > >> discussion. The reason I raised KIP44 is
> >> that I
> >> >> >> want
> >> >> >> >> to
> >> >> >> >> >> be
> >> >> >> >> >> > > > able
> >> >> >> >> >> > > > > to
> >> >> >> >> >> > > > > > >>> > >> implement a custom security protocol that
> >> can
> >> >> >> fulfill
> >> >> >> >> >> the
> >> >> >> >> >> > > need
> >> >> >> >> >> > > > > of my
> >> >> >> >> >> > > > > > >>> > >> company. As pointed out by Ismael KIP-43
> now
> >> >> >> >> supports a
> >> >> >> >> >> > > > > pluggable
> >> >> >> >> >> > > > > > >>> way to
> >> >> >> >> >> > > > > > >>> > >> inject custom security provider to SASL I
> >> think
> >> >> >> it is
> >> >> >> >> >> > enough
> >> >> >> >> >> > > > to
> >> >> >> >> >> > > > > > >>> cover the
> >> >> >> >> >> > > > > > >>> > >> use case I have and address the concerns
> >> >> raised in
> >> >> >> >> >> KIP-44.
> >> >> >> >> >> > > > > > >>> > >>
> >> >> >> >> >> > > > > > >>> > >> For multiple security protocols support
> >> >> >> >> simultaneously
> >> >> >> >> >> it
> >> >> >> >> >> > is
> >> >> >> >> >> > > > not
> >> >> >> >> >> > > > > > >>> needed in
> >> >> >> >> >> > > > > > >>> > >> my use case and I don't foresee it is
> >> needed in
> >> >> >> the
> >> >> >> >> >> future
> >> >> >> >> >> > > but
> >> >> >> >> >> > > > > as i
> >> >> >> >> >> > > > > > >>> said
> >> >> >> >> >> > > > > > >>> > >> this is my use case only there may be
> other
> >> use
> >> >> >> cases
> >> >> >> >> >> that
> >> >> >> >> >> > > > need
> >> >> >> >> >> > > > > it.
> >> >> >> >> >> > > > > > >>> But if
> >> >> >> >> >> > > > > > >>> > >> we want to support it in the future I
> >> prefer to
> >> >> >> get
> >> >> >> >> it
> >> >> >> >> >> > right
> >> >> >> >> >> > > > at
> >> >> >> >> >> > > > > the
> >> >> >> >> >> > > > > > >>> first
> >> >> >> >> >> > > > > > >>> > >> place given the fact that security
> protocol
> >> is
> >> >> an
> >> >> >> >> ENUM
> >> >> >> >> >> and
> >> >> >> >> >> > > if
> >> >> >> >> >> > > > we
> >> >> >> >> >> > > > > > >>> stick to
> >> >> >> >> >> > > > > > >>> > >> that implementation it is very hard to
> >> extend
> >> >> in
> >> >> >> the
> >> >> >> >> >> > future
> >> >> >> >> >> > > > > when we
> >> >> >> >> >> > > > > > >>> decide
> >> >> >> >> >> > > > > > >>> > >> multiple security protocols is needed.
> >> >> >> >> >> > > > > > >>> > >>
> >> >> >> >> >> > > > > > >>> > >> Protocol negotiation is a very common
> usage
> >> >> >> pattern
> >> >> >> >> in
> >> >> >> >> >> > > > security
> >> >> >> >> >> > > > > > >>> domain. As
> >> >> >> >> >> > > > > > >>> > >> suggested in Java SASL doc
> >> >> >> >> >> > > > > > >>> > >>
> >> >> >> >> >> > > > > > >>> > >>
> >> >> >> >> >> > > > > > >>>
> >> >> >> >> >> > > > >
> >> >> >> >> >> > > >
> >> >> >> >> >> > >
> >> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/sasl/sasl-refguide.html
> >> >> >> >> >> > > > > > >>> > >> client
> >> >> >> >> >> > > > > > >>> > >> first sends out a packet to server and
> >> server
> >> >> >> >> responds
> >> >> >> >> >> > with
> >> >> >> >> >> > > a
> >> >> >> >> >> > > > > list
> >> >> >> >> >> > > > > > >>> of
> >> >> >> >> >> > > > > > >>> > >> mechanisms it supports. This is very
> >> similar to
> >> >> >> >> SSL/TLS
> >> >> >> >> >> > > > > negotiation.
> >> >> >> >> >> > > > > > >>> > >>
> >> >> >> >> >> > > > > > >>> > >> On Tue, 2 Feb 2016 at 06:39 Ismael Juma <
> >> >> >> >> >> > ism...@juma.me.uk>
> >> >> >> >> >> > > > > wrote:
> >> >> >> >> >> > > > > > >>> > >>
> >> >> >> >> >> > > > > > >>> > >> > On Mon, Feb 1, 2016 at 7:04 PM, Gwen
> >> Shapira
> >> >> <
> >> >> >> >> >> > > > > g...@confluent.io>
> >> >> >> >> >> > > > > > >>> wrote:
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >> > > Looking at "existing solutions", it
> >> looks
> >> >> like
> >> >> >> >> >> > Zookeeper
> >> >> >> >> >> > > > > allows
> >> >> >> >> >> > > > > > >>> > >> plugging
> >> >> >> >> >> > > > > > >>> > >> > in
> >> >> >> >> >> > > > > > >>> > >> > > any SASL mechanism, but the server
> will
> >> >> only
> >> >> >> >> support
> >> >> >> >> >> > one
> >> >> >> >> >> > > > > > >>> mechanism at
> >> >> >> >> >> > > > > > >>> > >> a
> >> >> >> >> >> > > > > > >>> > >> > > time.
> >> >> >> >> >> > > > > > >>> > >> > >
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >> > This was the original proposal from
> >> Rajini as
> >> >> >> that
> >> >> >> >> is
> >> >> >> >> >> > > enough
> >> >> >> >> >> > > > > for
> >> >> >> >> >> > > > > > >>> their
> >> >> >> >> >> > > > > > >>> > >> > needs.
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >> > > If this is good enough for our
> use-case
> >> >> (do we
> >> >> >> >> >> > actually
> >> >> >> >> >> > > > > need to
> >> >> >> >> >> > > > > > >>> > >> support
> >> >> >> >> >> > > > > > >>> > >> > > multiple mechanisms at once?), it
> will
> >> >> >> simplify
> >> >> >> >> >> life a
> >> >> >> >> >> > > lot
> >> >> >> >> >> > > > > for
> >> >> >> >> >> > > > > > >>> us (
> >> >> >> >> >> > > > > > >>> > >> > >
> >> >> >> >> >> > > > > > >>> > >>
> >> >> >> >> >> > > > > > >>>
> >> >> >> >> >> > > > >
> >> >> >> >> >> > >
> >> >> >> >> >>
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zookeeper+and+SASL
> >> >> >> >> >> > > > > > >>> > >> > )
> >> >> >> >> >> > > > > > >>> > >> > >
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >> > The current thinking is that it would
> be
> >> >> useful
> >> >> >> to
> >> >> >> >> >> > support
> >> >> >> >> >> > > > > > >>> multiple SASL
> >> >> >> >> >> > > > > > >>> > >> > mechanisms simultaneously. In the KIP
> >> >> meeting,
> >> >> >> Jun
> >> >> >> >> >> > > mentioned
> >> >> >> >> >> > > > > that
> >> >> >> >> >> > > > > > >>> > >> companies
> >> >> >> >> >> > > > > > >>> > >> > sometimes support additional
> >> authentication
> >> >> >> >> mechanisms
> >> >> >> >> >> > for
> >> >> >> >> >> > > > > > >>> partners, for
> >> >> >> >> >> > > > > > >>> > >> > example. It does make things more
> >> complex, as
> >> >> >> you
> >> >> >> >> say,
> >> >> >> >> >> > so
> >> >> >> >> >> > > we
> >> >> >> >> >> > > > > need
> >> >> >> >> >> > > > > > >>> to be
> >> >> >> >> >> > > > > > >>> > >> > sure the complexity is worth it.
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >> > Two more points:
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >> > 1. It has been suggested that custom
> >> security
> >> >> >> >> protocol
> >> >> >> >> >> > > > > support is
> >> >> >> >> >> > > > > > >>> > >> needed by
> >> >> >> >> >> > > > > > >>> > >> > some (KIP-44). Rajini enhanced KIP-43
> so
> >> >> that a
> >> >> >> >> SASL
> >> >> >> >> >> > > > mechanism
> >> >> >> >> >> > > > > > >>> with a
> >> >> >> >> >> > > > > > >>> > >> > custom provider can be used for this
> >> purpose
> >> >> >> >> instead.
> >> >> >> >> >> > > Given
> >> >> >> >> >> > > > > this,
> >> >> >> >> >> > > > > > >>> it
> >> >> >> >> >> > > > > > >>> > >> seems
> >> >> >> >> >> > > > > > >>> > >> > a bit inconsistent and restrictive not
> to
> >> >> allow
> >> >> >> >> >> multiple
> >> >> >> >> >> > > > SASL
> >> >> >> >> >> > > > > > >>> mechanisms
> >> >> >> >> >> > > > > > >>> > >> > simultaneously (we do allow SSL and
> SASL
> >> >> >> >> >> authentication
> >> >> >> >> >> > > > > > >>> simultaneously,
> >> >> >> >> >> > > > > > >>> > >> > after all).
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >> > 2. The other option would be to
> support a
> >> >> single
> >> >> >> >> SASL
> >> >> >> >> >> > > > > mechanism
> >> >> >> >> >> > > > > > >>> > >> > simultaneously to start with and then
> >> extend
> >> >> >> this
> >> >> >> >> to
> >> >> >> >> >> > > > multiple
> >> >> >> >> >> > > > > > >>> mechanisms
> >> >> >> >> >> > > > > > >>> > >> > simultaneously later (if and when
> >> needed). It
> >> >> >> seems
> >> >> >> >> >> like
> >> >> >> >> >> > > it
> >> >> >> >> >> > > > > would
> >> >> >> >> >> > > > > > >>> be
> >> >> >> >> >> > > > > > >>> > >> harder
> >> >> >> >> >> > > > > > >>> > >> > to support the latter in the future if
> we
> >> go
> >> >> >> down
> >> >> >> >> this
> >> >> >> >> >> > > > route,
> >> >> >> >> >> > > > > but
> >> >> >> >> >> > > > > > >>> maybe
> >> >> >> >> >> > > > > > >>> > >> > there are ways around this.
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >> > Thoughts?
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >> > Ismael
> >> >> >> >> >> > > > > > >>> > >> >
> >> >> >> >> >> > > > > > >>> > >>
> >> >> >> >> >> > > > > > >>> > >
> >> >> >> >> >> > > > > > >>> > >
> >> >> >> >> >> > > > > > >>> > >
> >> >> >> >> >> > > > > > >>> > > --
> >> >> >> >> >> > > > > > >>> > > Regards,
> >> >> >> >> >> > > > > > >>> > >
> >> >> >> >> >> > > > > > >>> > > Rajini
> >> >> >> >> >> > > > > > >>> > >
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > --
> >> >> >> >> >> > > > > > >>> > Regards,
> >> >> >> >> >> > > > > > >>> >
> >> >> >> >> >> > > > > > >>> > Rajini
> >> >> >> >> >> > > > > > >>>
> >> >> >> >> >> > > > > > >>
> >> >> >> >> >> > > > > > >>
> >> >> >> >> >> > > > > > >>
> >> >> >> >> >> > > > > > >> --
> >> >> >> >> >> > > > > > >> Regards,
> >> >> >> >> >> > > > > > >>
> >> >> >> >> >> > > > > > >> Rajini
> >> >> >> >> >> > > > > > >>
> >> >> >> >> >> > > > > > >
> >> >> >> >> >> > > > > > >
> >> >> >> >> >> > > > > > >
> >> >> >> >> >> > > > > > > --
> >> >> >> >> >> > > > > > > Regards,
> >> >> >> >> >> > > > > > >
> >> >> >> >> >> > > > > > > Rajini
> >> >> >> >> >> > > > > > >
> >> >> >> >> >> > > > > >
> >> >> >> >> >> > > > > >
> >> >> >> >> >> > > > > >
> >> >> >> >> >> > > > > > --
> >> >> >> >> >> > > > > > Regards,
> >> >> >> >> >> > > > > >
> >> >> >> >> >> > > > > > Rajini
> >> >> >> >> >> > > > >
> >> >> >> >> >> > > >
> >> >> >> >> >> > > >
> >> >> >> >> >> > > >
> >> >> >> >> >> > > > --
> >> >> >> >> >> > > > Regards,
> >> >> >> >> >> > > >
> >> >> >> >> >> > > > Rajini
> >> >> >> >> >> > > >
> >> >> >> >> >> > >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > --
> >> >> >> >> >> > Regards,
> >> >> >> >> >> >
> >> >> >> >> >> > Rajini
> >> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > --
> >> >> >> >> > Regards,
> >> >> >> >> >
> >> >> >> >> > Rajini
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> > Regards,
> >> >> >> >
> >> >> >> > Rajini
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Regards,
> >> >> >
> >> >> > Rajini
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> >
> >> > Rajini
> >>
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
>
>
--
Regards,
Rajini
