Magnus,
*"Why would it be harder to achieve? And is it harder for the Kafka developeror the user?Wouldnt it be possible to accept both non-SSL-auth and SSL-auth clients onthe same port?"* Maybe I am missing something, so perhaps it would be easier if I explained our scenario and you could tell me how it would work with the new design? a) Inter-broker communication - uses SSL (without SASL) - is on an internal network - advertised host is internal IP address - broker uses TLS client auth b) External clients - use SASL_SSL - are on the public network (and traffic goes through an additional TLS proxy for security) - advertised host is a registered host name also included in the certificate and validated by the client during TLS handshake - cannot use TLS client auth since that would require a complex PKI infrastructure for certificate management - should not be allowed to connect to Kafka without SASL authentication (broker should prevent connection without SASL) Questions: 1. How do I configure a) and b) to be on different network interfaces? 2. How do I advertise different hosts for a) and b)? 3. How do I configure broker to disable TLS client auth for b) while enabling it for a)? 4. How does the broker prevent external clients from connecting without SASL while allowing internal clients to connect without SASL? Thanks, Rajini On Wed, Apr 6, 2016 at 8:33 PM, Magnus Edenhill <mag...@edenhill.se> wrote: > 2016-04-06 19:16 GMT+02:00 Rajini Sivaram <rajinisiva...@googlemail.com>: > > > Magnus, > > > > I have not looked at your proposal in detail yet, > > > > Please do :) > > > > but I have a few comments: > > > > 1. We need to support SSL and SASL_SSL within the same broker (for > > instance, to support certificate-based authentication for replication > > and > > SASL for external connections). At the moment, there is a one-to-one > > mapping between port and security protocol. If SASL is a Kafka-level > > option > > on the same port rather than a security protocol, this would be harder > > to > > achieve. With separate protocols, we are also able to turn off SSL > > client > > auth for SASL. > > > > Why would it be harder to achieve? And is it harder for the Kafka developer > or the user? > Wouldnt it be possible to accept both non-SSL-auth and SSL-auth clients on > the same port? > > > > > > 2. SASL is a challenge-response protocol. Server generates a > challenge, > > client responds. Fitting this into Kafka API is possible, but feels > > unnatural. > > > > I dont see any problems with, the handshake is half-duplex and driven by > the SASL library, all the client needs > to do is "long poll" the server for data. > > Something like this in pseudo code (where sasl. is the SASL library > context): > until sasl.auth_done() { > sendbuf = sasl.wants_to_send() > if (!sendbuf) // Nothing to send, send empty payload to poll > server for data > response = SaslHandshakeRequest(payload=none) > else > response = SaslHandshakeRequest(payload=sendbuf) > > // handle response (possibly empty) from server and continue > handshake. > sasl.parse(response) > } > > > 3. Personally, I like the current Network|Security|Application layering > > of the code in Kafka. Handshake code is quite complicated and hard to > > debug > > as it is, moving that to the application layer could make it worse. > With > > handshake in the application layer, you would need to set Principal > > based > > on SSL channel for non-SASL and sometime later on for SASL. It may be > > just > > me, but it doesn't feel quite right. > > > > I actually think reusing the existing Kafka protocol implementation for > this makes it > more robust and easier to debug and understand. > > How do you plan on handling feature discovery for new requests or request > versions in this special protocol? > I.e., what KIP-35 provides for the Kafka protocol. > > > Regards, > Magnus > > > > > > On Wed, Apr 6, 2016 at 12:47 PM, Magnus Edenhill <mag...@edenhill.se> > > wrote: > > > > > Engineers, > > > > > > sorry to stir up some dust on this KIP but following yesterday's > > discussion > > > on the KIP call I'm a bit concerned about the way the SASL handshake > > > protocol is diverging even more from the main Kafka protocol. > > > The SASL handshake is already its own special protocol and the latest > > > addition of SaslMechanisms request in KIP-43 > > > adds yet another protocol on top of that, making assumptions on what > SASL > > > libraries are communicating in the opaque data channel. > > > Unlike TLS, SASL has no standard handshake protocol so conceiving a > > special > > > non-Kafka protocol just for SASL in Kafka is unwarranted given that the > > > Kafka protocol natively supports the same semantics. > > > There's also a non-neglectable maintenance aspect of maintaining a > > separate > > > protocol, in regards to documentation, versioning, testing, protocol > > stack > > > implementations (broker and clients * N), etc. > > > > > > > > > > > > So let me suggest a different approach that pulls the SASL handshake > into > > > the Kafka protocol properly and makes the > > > authentication an in-band connection setup step. > > > > > > > > > Create two new proper Kafka requests: > > > > > > SaslMechanismsRequest V0 { > > > // empty (see discussion below) > > > } > > > > > > SaslMechanismResponse V0 { > > > int16 errroCode; // e.g. AuthenticationNotRequired (when SASL > is > > > not configured) - or maybe just return an empty list in that case > > > Array[String] mechanisms; // server's supported mechanisms. > Client > > > may choose which one to use. > > > } > > > > > > > > > SaslHandshakeRequest V0 { > > > string selected_mechanism; // client's selected mechanism, MUST > not > > > change during a handshake (ignore in sub-sequent requests?) > > > bytes data; // opaque SASL library handshake data. > > > } > > > > > > SaslHandshakeResponse V0 { > > > int16 errorCode; // e.g. UnsupportedMechanism, and arbitrary SASL > > > library exceptions, etc. > > > bytes data; // opaque SASL library handshake data. > > > } > > > > > > (Note: I'll leave the exact format of the above Request/Responses to > the > > > people in the know, this is a proof of concept layout.) > > > > > > The protocol semantics of the current SASL handshake, which is driven > by > > > the underlying SASL libraries, remains in place, > > > such as sending zero length requests to "poll" the remote side, etc. > > > > > > > > > > > > Regarding empty SaslMechanismRequest: > > > A previous proposal suggests sending the client's configured mechanism > in > > > the SaslMechanismRequest, which is fine but if we do that there is no > > > point in returning the broker's list of mechanisms since the client has > > > already chosen - only the error code is needed. > > > For clients wanting to support multiple mechanisms it can iterate over > > its > > > configured mechanisms and send a SaslMechanismRequest for each one, > but > > > that just adds complexity at no gain, and at the end of the day the > > > desired_mechanism check in the broker is purely cosmetic. > > > > > > If we instead skip the desired mechanism and let the broker always > > return > > > its list of supported mechanisms the client can choose the one it > needs, > > > this allows a client to optionally support multiple mechanisms and also > > > puts the decision logic in the client rather than the server. > > > > > > If the argument is that a client will only support one mechanism > (which > > I > > > think is a bold claim to make) then the SaslMechanismRequest isn't > needed > > > at all, the client can jump directly SaslHandshakeRequest { > > > selected_mechanism="mymechanism" } and look for UnsupportedMechanism in > > the > > > response's errorCode. > > > > > > > > > > > > > > > > > > Typical client MO using the above requests: > > > > > > # Use ApiVersion info to select the proper SaslMechanismRequest version > > to > > > use (for future use..) > > > Client > Broker: ApiVersionRequest // (optional) query for > > > supported API versions, possibly limited to > > > {SaslHandshakeReq,SaslMechanismsReq..} > > > Broker < Client: ApiVersionResponse { .. } > > > > > > > > > # Request broker's supported mechanisms list > > > Client > Broker: SaslMechanismRequest { } > > > Broker < Client: SaslMechanismResponse { mechanisms = { "GSSAPI", > > "KRB", > > > .. } > > > > > > # If broker responded with an empty mechanisms list, alternative an > > > AuthenticationNotRequired error code, the client skips to its > operational > > > phase. > > > # If the client can't find a supported mechanism in the returned > > mechanisms > > > list it raises an error to the application. > > > > > > # Otherwise the client selects a proper mechanism and the auth > handshake > > > begins: > > > Client > Broker: SaslHandshakeRequest { mechanism="KRB", data = > > > KRB-specific data.. } > > > Broker > Client: SaslHandshakeResponse { data = KRB-specific data } > > > repeat as necessary > > > > > > # Client is now authenticated and may start its usual operation of > > > MetadataRequest, Produce, Fetch, etc... > > > > > > > > > > > > Pros: > > > + Using the standard Kafka protocol provides proper versioning, > > > documentation, feature detection, etc, out of the box. > > > + Less complex protcol implementations in both client and brokers. > > > Standard protocol stack can be used. > > > + No need to define a specific SASL port, SASL is now an in-band > > operation > > > on any type of connection (PLAINTEXT, SSL) > > > + Requires proper per-API auth enforcement in the broker, which is a > > good > > > thing. (i.e., if SASL is configured an unauthenticated client should > not > > be > > > permitted to call any API but ApiVersion, SaslHandshake, > SaslMechanisms, > > > ..). > > > + Future proof, the SASL/Auth handshake protocol can evolve naturally > > > using KIP-35 for feature detection. > > > + Looking forward this also enables re-authentication in the current > > > session, if there's ever a use case for that. > > > > > > Cons: > > > - Requires proper per-API auth enforcement in the broker, but this is > a > > > good thing. An initial naiive approach to this is to only allow Sasl* > and > > > ApiVersion requests prior to authentication (if auth is required). > > > > > > > > > > > > Compatibility: > > > - Leave the current legacy SASL handshake in place for SASL_PLAINTEXT > > and > > > SASL_SSL connections to not break existing 0.9 clients (that is the > Java > > > client and librdkafka, no other clients have implemented SASL yet > > AFAIK). > > > Deprecate this method in documentation. > > > - SSL and PLAINTEXT connections will use in-band authentication if > SASL > > is > > > configured. > > > > > > > > > > > > /Magnus > > > > > > 2016-03-16 0:41 GMT+01:00 Rajini Sivaram <rajinisiva...@googlemail.com > >: > > > > > > > Both the KIP and the PR have been updated to a cut-down version as > > > > discussed in the KIP meeting today. > > > > > > > > Any feedback is appreciated. > > > > > > > > On Tue, Mar 15, 2016 at 7:39 PM, Rajini Sivaram < > > > > rajinisiva...@googlemail.com> wrote: > > > > > > > > > Following on from the discussions in the KIP meeting today, the > > > > suggestion > > > > > is to implement a cut-down version of KIP-43 for 0.10.0.0 with a > > > > follow-on > > > > > KIP after the release to address support for custom mechanisms. > > > > > > > > > > Changes to be removed from KIP-43: > > > > > > > > > > 1. Remove the configuration for CallbackHandler. The callback > > > handler > > > > > implementation in Kafka will support Kerberos, PLAIN and > > Digest-MD5. > > > > It > > > > > will not support custom or more complex mechanisms which require > > > > additional > > > > > callbacks. > > > > > 2. Remove the configuration for Login. The Login implementation > in > > > > > Kafka will support Kerberos and any other mechanism (PLAIN, > > > > Digest-MD5 etc) > > > > > that doesn't require functionality like token refresh. > > > > > > > > > > Changes included in KIP-43: > > > > > > > > > > 1. Configurable mechanism > > > > > 2. Support for multiple mechanisms in the broker > > > > > 3. Implementation of SASL/PLAIN > > > > > > > > > > If there are no objections to this, I can update the KIP and the PR > > by > > > > > tomorrow. And move the support for custom mechanisms into another > KIP > > > and > > > > > PR for review after the release of 0.10.0.0. > > > > > > > > > > > > > > > On Mon, Mar 14, 2016 at 7:48 AM, Rajini Sivaram < > > > > > rajinisiva...@googlemail.com> wrote: > > > > > > > > > >> Harsha, > > > > >> > > > > >> You are right, we don't expect to override callback handler or > login > > > for > > > > >> Digest-MD5. > > > > >> > > > > >> Pluggable CallbackHandler and Login modules enable custom SASL > > > > mechanisms > > > > >> to be implemented without modifying Kafka. For instance, it would > > > enable > > > > >> KIP-44 ( > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-44+-+Allow+Kafka+to+have+a+customized+security+protocol > > > > ) > > > > >> to be implemented without making the whole security protocol > > > pluggable. > > > > Tao > > > > >> Xiao has already confirmed earlier in this discussion thread that > > the > > > > >> proposed callback handler and login interfaces are suitable for > > their > > > > >> custom authentication. > > > > >> > > > > >> > > > > >> > > > > >> On Sun, Mar 13, 2016 at 6:59 PM, Harsha <ka...@harsha.io> wrote: > > > > >> > > > > >>> Agree with Gwen here. I feel like these additional pluggable > Login > > > > >>> Modules are making this KIP complex. Since the main goal of the > KIP > > > is > > > > >>> to enable additional mechanism , can we limit the scope to that > and > > > If > > > > >>> we feel necessary for pluggable Login and callback handler > classes > > we > > > > >>> can address in another JIRA. > > > > >>> > > > > >>> Adding digest-md5 ,password callbacks can be done to existing > > > > >>> callbackhandler without expose it as pluggable class. It would > be > > > > >>> useful to have broker support multiple mechanisms. I haven't > seen > > > > >>> anyone using more than this in hadoop . It might be different for > > > Kafka > > > > >>> but I personally haven't seen anyone asking for this yet. > > > > >>> > > > > >>> Thanks, > > > > >>> Harsha > > > > >>> > > > > >>> > > > > >>> On Thu, Mar 10, 2016, at 01:44 AM, Rajini Sivaram wrote: > > > > >>> > Gwen, > > > > >>> > > > > > >>> > Just to be clear, the alternative would be: > > > > >>> > > > > > >>> > *jaas.conf:* > > > > >>> > > > > > >>> > GssapiKafkaServer { > > > > >>> > > > > > >>> > com.ibm.security.auth.module.Krb5LoginModule required > > > > >>> > credsType=both > > > > >>> > useKeytab="file:/kafka/key.tab" > > > > >>> > principal="kafka/localh...@example.com <http://example.com/>"; > > > > >>> > > > > > >>> > }; > > > > >>> > > > > > >>> > SmartcardKafkaServer { > > > > >>> > > > > > >>> > example.SmartcardLoginModule required > > > > >>> > > > > > >>> > cardNumber=123; > > > > >>> > > > > > >>> > }; > > > > >>> > > > > > >>> > > > > > >>> > *KafkaConfig* > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > - login.context.map={"GSSAPI="GssapiKafkaServer", > > > > >>> > "SMARTCARD"=SmartcardKafkaServer} > > > > >>> > - login.class.map={"GSSAPI=GssapiLogin.class, > > > > >>> > "SMARTCARD"=SmartcardLogin.class} > > > > >>> > - > > > > >>> callback.handler.class.map={"GSSAPI"=GssapiCallbackHandler.class, > > > > >>> > "SMARTCARD"=SmartcardCallbackHandler.class} > > > > >>> > > > > > >>> > *Client Config * > > > > >>> > Same as the server, but with only one entry allowed in each map > > and > > > > >>> > jaas.conf > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > This is a different model from the Java standard for supporting > > > > >>> multiple > > > > >>> > logins. As a developer, I am inclined to stick with approaches > > that > > > > are > > > > >>> > widely in use like JSSE. But this alternative can be made to > work > > > if > > > > >>> the > > > > >>> > Kafka community feels it is more appropriate for Kafka. If you > > know > > > > of > > > > >>> > other systems which use this approach, that would be helpful. > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > On Thu, Mar 10, 2016 at 2:07 AM, Gwen Shapira < > g...@confluent.io > > > > > > > >>> wrote: > > > > >>> > > > > > >>> > > What I'm hearing is that: > > > > >>> > > > > > > >>> > > 1. In order to support authentication mechanisms that were > not > > > > >>> written > > > > >>> > > specifically with Kafka in mind, someone will need to write > the > > > > >>> > > integration between the mechanism and Kafka. This may include > > > Login > > > > >>> > > and CallbackHandler classes. This can be the mechanism > vendor, > > > the > > > > >>> > > user or a 3rd party vendor. > > > > >>> > > 2. If someone wrote the code to support a mechanism in Kafka, > > > and a > > > > >>> > > user will want to use more than one mechanism, they will > still > > > need > > > > >>> to > > > > >>> > > write a wrapper. > > > > >>> > > 3. In reality, #2 will not be necessary ("edge-case") because > > > Kafka > > > > >>> > > will actually already provide the callback needed (and > > presumably > > > > >>> also > > > > >>> > > the code to load the LoginModule provided by Example.com)? > > > > >>> > > > > > > >>> > > Tradeoff #1 sounds reasonable. > > > > >>> > > #2 and #3 do not sound reasonable considering one of the > goals > > of > > > > the > > > > >>> > > patch is to support multiple mechanisms. I don't think we > > should > > > > >>> force > > > > >>> > > our users to write code just to avoid writing it ourselves. > > > > >>> > > Configuring security is complex enough as is. > > > > >>> > > Furthermore, if we believe that "Smartcard is likely to use > > > > standard > > > > >>> > > NameCallback and PasswordCallback already implemented in > > Kafka" - > > > > why > > > > >>> > > do we even provide configuration for Login and > CallbackHandler > > > > >>> > > classes? Either we support multiple mechanisms written by > > > different > > > > >>> > > vendors, or we don't. > > > > >>> > > > > > > >>> > > Gwen > > > > >>> > > > > > > >>> > > > > > > >>> > > On Wed, Mar 9, 2016 at 12:32 AM, Rajini Sivaram > > > > >>> > > <rajinisiva...@googlemail.com> wrote: > > > > >>> > > > I am not saying that the developer at Example Inc. would > > > develop > > > > a > > > > >>> Login > > > > >>> > > > implementation that combines Smartcard and Kerberos because > > > > >>> Retailer uses > > > > >>> > > > both. I am saying that Example Inc develops the LoginModule > > > > >>> (similar to > > > > >>> > > JVM > > > > >>> > > > security providers developing Kerberos modules). But there > is > > > no > > > > >>> standard > > > > >>> > > > interface for Login to allow ticket refresh. So, it is very > > > > >>> unlikely that > > > > >>> > > > Example Inc would develop a Login implementation that works > > > with > > > > an > > > > >>> > > Apache > > > > >>> > > > Kafka defined interface ( Kafka developers wrote this code > > for > > > > >>> Kerberos). > > > > >>> > > > For a custom integration, the user (i.e. Retailer) would be > > > > >>> expected to > > > > >>> > > > develop this code if required. > > > > >>> > > > > > > > >>> > > > You could imagine that Smartcard is a commonly used > mechanism > > > and > > > > >>> a 3rd > > > > >>> > > > party develops code for integrating Smartcard with Kafka > and > > > > makes > > > > >>> the > > > > >>> > > > integration code (Login and CallbackHandler implementation) > > > > widely > > > > >>> > > > available, If Retailer wants to use clients or a broker > with > > > just > > > > >>> > > Smartcard > > > > >>> > > > enabled in their broker, they configure Kafka to use the > 3rd > > > > party > > > > >>> code, > > > > >>> > > > with no additional code development. But to combine > Smartcard > > > and > > > > >>> > > Kerberos, > > > > >>> > > > Retailer needs to write a few lines of code to incorporate > > both > > > > >>> Smartcard > > > > >>> > > > and Kerberos. I believe this is an edge case. > > > > >>> > > > > > > > >>> > > > Smartcard is likely to use standard NameCallback and > > > > >>> PasswordCallback > > > > >>> > > > already implemented in Kafka and Kerberos support exists in > > > > Kafka. > > > > >>> So it > > > > >>> > > is > > > > >>> > > > very likely that Retailer doesn't need to override Login or > > > > >>> > > CallbackHandler > > > > >>> > > > in this case. And it would just be a matter of configuring > > the > > > > >>> > > mechanisms. > > > > >>> > > > > > > > >>> > > > On Wed, Mar 9, 2016 at 12:48 AM, Gwen Shapira < > > > g...@confluent.io > > > > > > > > > >>> wrote: > > > > >>> > > > > > > > >>> > > >> "Since smart card logins are not built into Kafka (or the > > > JDK), > > > > >>> you > > > > >>> > > need a > > > > >>> > > >> developer to build the login module. So the developer > > > implements > > > > >>> > > >> example.SmartcardLoginModule. In addition, the developer > may > > > > also > > > > >>> > > implement > > > > >>> > > >> callback handlers for the SASL client or server and a > login > > > > >>> class to > > > > >>> > > keep > > > > >>> > > >> this login refreshed. The callback handlers and login > > > > >>> implementation > > > > >>> > > >> support all the mechanisms that the organisation supports > - > > in > > > > >>> this case > > > > >>> > > >> Kerberos and smart card." > > > > >>> > > >> > > > > >>> > > >> In this case, the developer works for Example Inc (which > > > > develops > > > > >>> > > >> SmartCard authentication modules), while I work for > Retailer > > > and > > > > >>> need > > > > >>> > > >> to use his module. > > > > >>> > > >> You assume that developer from Example Inc knows about all > > the > > > > >>> > > >> mechanisms that I have enabled in my Kafka cluster and he > is > > > > >>> capable > > > > >>> > > >> of developing a Login class and maybe a callback handler > > that > > > > >>> supports > > > > >>> > > >> all of them. I think this is an unreasonable expectation. > > > > >>> > > >> > > > > >>> > > >> I think that allowing plug in of Login method and single > > > > Callback > > > > >>> > > >> handler is not a good extensible solution for multiple > > > > >>> mechanisms. We > > > > >>> > > >> should probably supply a Kafka-based LoginContext and > > > > >>> CallbackHandler > > > > >>> > > >> that will be able to delegate to implementations of > > > LoginModules > > > > >>> and > > > > >>> > > >> Callback Handlers based on mechanism out of a list > supplied > > > by a > > > > >>> user. > > > > >>> > > >> I agree it is more complex, but otherwise we are asking of > > too > > > > >>> much > > > > >>> > > >> coordination from whoever implements CallbackHandlers and > > > Login > > > > >>> > > >> methods. > > > > >>> > > >> > > > > >>> > > >> Gwen > > > > >>> > > >> > > > > >>> > > >> On Tue, Mar 8, 2016 at 3:35 PM, Rajini Sivaram > > > > >>> > > >> <rajinisiva...@googlemail.com> wrote: > > > > >>> > > >> > As an admin, you would configure login modules using > > > standard > > > > >>> JAAS > > > > >>> > > >> > configuration. So, if you are supporting Kerberos and > > smart > > > > >>> card login > > > > >>> > > >> > modules within a single broker, in your jaas.conf you > > would > > > > >>> specify: > > > > >>> > > >> > > > > > >>> > > >> > KafkaServer { > > > > >>> > > >> > > > > > >>> > > >> > com.ibm.security.auth.module.Krb5LoginModule required > > > > >>> > > >> > credsType=both > > > > >>> > > >> > useKeytab="file:/kafka/key.tab" > > > > >>> > > >> > principal="kafka/localh...@example.com"; > > > > >>> > > >> > > > > > >>> > > >> > example.SmartcardLoginModule required > > > > >>> > > >> > > > > > >>> > > >> > cardNumber=123 > > > > >>> > > >> > > > > > >>> > > >> > }; > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> > > >> > Since smart card logins are not built into Kafka (or the > > > JDK), > > > > >>> you > > > > >>> > > need a > > > > >>> > > >> > developer to build the login module. So the developer > > > > implements > > > > >>> > > >> > example.SmartcardLoginModule. In addition, the developer > > may > > > > >>> also > > > > >>> > > >> implement > > > > >>> > > >> > callback handlers for the SASL client or server and a > > login > > > > >>> class to > > > > >>> > > >> keep > > > > >>> > > >> > this login refreshed. The callback handlers and login > > > > >>> implementation > > > > >>> > > >> > support all the mechanisms that the organisation > supports > > - > > > in > > > > >>> this > > > > >>> > > case > > > > >>> > > >> > Kerberos and smart card. > > > > >>> > > >> > > > > > >>> > > >> > As an admin, if you are using mechanisms built into > Kafka, > > > you > > > > >>> simply > > > > >>> > > >> > specify a JAAS config with the combination of mechanisms > > you > > > > >>> want and > > > > >>> > > you > > > > >>> > > >> > are done. If you are adding new mechanisms, it clearly > > can't > > > > be > > > > >>> done > > > > >>> > > >> > without some additional code to implement that > mechanism. > > > The > > > > >>> callback > > > > >>> > > >> and > > > > >>> > > >> > login interfaces are configurable to ensure that (as a > > > > >>> developer), you > > > > >>> > > >> can > > > > >>> > > >> > build custom mechanisms without modifying Kafka. SASL > was > > > > >>> designed > > > > >>> > > >> > precisely to do this. But at the moment, Kafka restricts > > > what > > > > >>> type of > > > > >>> > > >> > authentication you can perform with SASL. > > > > >>> > > >> > > > > > >>> > > >> > Back to the questions: > > > > >>> > > >> > 1. *How does the proposed Login class interact with > > > > >>> LoginContext and * > > > > >>> > > >> > *LoginModule.* > > > > >>> > > >> > Login class performs *LoginContext.login(). *This > performs > > > > >>> login for > > > > >>> > > all > > > > >>> > > >> > the login modules of that context (this is standard Java > > > > >>> function). > > > > >>> > > Login > > > > >>> > > >> > is cached with a reference counter to keep it alive. > > > > >>> > > >> > > > > > >>> > > >> > 2. *Do we really need a pluggable Login class, when > users > > > > >>> > > >> **(administrators, > > > > >>> > > >> > not developers) are more likely to want to specify > > > > >>> LoginModules?* > > > > >>> > > >> > Administrators specify *LoginModules. *Developers > > implement > > > > new > > > > >>> > > >> mechanisms. > > > > >>> > > >> > The pluggable *Login* class enables the developer to > > > implement > > > > >>> new > > > > >>> > > >> > mechanisms without changing Kafka. If Kafka didn't have > a > > > > >>> default > > > > >>> > > >> Kerberos > > > > >>> > > >> > implementation and a user wanted to use Kerberos, > > > pluggability > > > > >>> of > > > > >>> > > >> *Login* class > > > > >>> > > >> > enables the user to include Kerberos without changing > > Kafka. > > > > >>> Not that > > > > >>> > > we > > > > >>> > > >> > want another Kerberos, but Tao had a requirement to > > enable a > > > > >>> custom > > > > >>> > > >> > mechanism with functionality similar to Kerberos (see > > > KIP-44). > > > > >>> > > Pluggable > > > > >>> > > >> > *Login* supports this use case without replacing the > > entire > > > > >>> > > >> authentication > > > > >>> > > >> > mechanism in Kafka. > > > > >>> > > >> > > > > > >>> > > >> > *3. How will users get to specify multiple > LoginModules?* > > > > >>> > > >> > Standard Java JAAS configuration - see example above. > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> > > >> > *4. It sounds like different LoginModules may need > > > > >>> > > >> > different CallbackHandlers. Since we only allow a single > > > > >>> > > >> > pluggableCallbackHandler class in the KIP-43 > > configuration, > > > > how > > > > >>> will > > > > >>> > > this > > > > >>> > > >> > be handled?* > > > > >>> > > >> > One LoginContext, one subject, one callback handler, > > > multiple > > > > >>> login > > > > >>> > > >> > modules. This is the Java model for login. And it works > > well > > > > >>> because > > > > >>> > > >> > callback handlers tend to be fairly standard. Even if > the > > > > smart > > > > >>> card > > > > >>> > > SASL > > > > >>> > > >> > callbacks needed to do something wildly different and > you > > > > >>> wanted a > > > > >>> > > broker > > > > >>> > > >> > that combined this with Kerberos, you would implement a > > > > >>> callback that > > > > >>> > > >> > looked like the code below. > > > > >>> > > >> > > > > > >>> > > >> > private CallbackHandler handler; > > > > >>> > > >> > > > > > >>> > > >> > public void configure(Map<String, ?> configs, > Mode > > > > mode, > > > > >>> > > Subject > > > > >>> > > >> > subject, String mechanism) {public void > > > configure(Map<String, > > > > ?> > > > > >>> > > configs, > > > > >>> > > >> > Mode mode, Subject subject, String mechanism) { > > > > >>> > > >> > > > > > >>> > > >> > switch (mechanism) { > > > > >>> > > >> > > > > > >>> > > >> > case "GSSAPI" : handler = new > > > > >>> GssapiCallbackHandler(); > > > > >>> > > >> break > > > > >>> > > >> > ; > > > > >>> > > >> > > > > > >>> > > >> > case "SMARTCARD" : handler = new > > > > >>> > > >> SmartcardCallbackHandler(); > > > > >>> > > >> > break; > > > > >>> > > >> > > > > > >>> > > >> > } > > > > >>> > > >> > > > > > >>> > > >> > } > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> > > >> > public void handle(Callback[] callbacks) throws > > > > >>> IOException, > > > > >>> > > >> > UnsupportedCallbackException { > > > > >>> > > >> > > > > > >>> > > >> > handler.handle(callbacks); > > > > >>> > > >> > > > > > >>> > > >> > } > > > > >>> > > >> > > > > > >>> > > >> > But I don't see this as just an admin configuration. > This > > is > > > > >>> some code > > > > >>> > > >> > developed alongside a new mechanism. Yes, you can add a > > new > > > > >>> > > configuration > > > > >>> > > >> > type in Kafka that maps mechanism to callback handler > > class > > > to > > > > >>> avoid > > > > >>> > > the > > > > >>> > > >> > switch statement. But that doesn't remove the need for > new > > > > code > > > > >>> > > >> altogether > > > > >>> > > >> > since you still need to implement > > SmartcardCallbackHandler. > > > > >>> Multiple > > > > >>> > > >> > conflicting callback handlers within one broker to > support > > > > >>> complete > > > > >>> > > >> > different behaviour is an edge case. Since the design > > > supports > > > > >>> > > >> > full flexibility in terms of functionality and you have > to > > > > >>> write new > > > > >>> > > code > > > > >>> > > >> > for new mechanisms anyway, this feels like the simplest > > > > >>> solution. > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> > > >> > On Tue, Mar 8, 2016 at 8:23 PM, Gwen Shapira < > > > > g...@confluent.io > > > > >>> > > > > > >>> > > wrote: > > > > >>> > > >> > > > > > >>> > > >> >> I guess what is unclear to me is: > > > > >>> > > >> >> > > > > >>> > > >> >> 1. How does the proposed Login class interact with > > > > >>> LoginContext and > > > > >>> > > >> >> LoginModule. > > > > >>> > > >> >> 2. Do we really need a pluggable Login class, when > users > > > > >>> > > >> >> (administrators, not developers) are more likely to > want > > to > > > > >>> specify > > > > >>> > > >> >> LoginModules? > > > > >>> > > >> >> 3. How will users get to specify multiple LoginModules? > > > > >>> > > >> >> 4. It sounds like different LoginModules may need > > different > > > > >>> > > >> >> CallbackHandlers. Since we only allow a single > pluggable > > > > >>> > > >> >> CallbackHandler class in the KIP-43 configuration, how > > will > > > > >>> this be > > > > >>> > > >> >> handled? > > > > >>> > > >> >> > > > > >>> > > >> >> In other words, suppose I want my Kafka installation to > > > > >>> support both > > > > >>> > > >> >> Kerberos and SmartCard mechanisms, can you tell me how > to > > > > >>> configure > > > > >>> > > >> >> Kafka? (as admin, not developer)? > > > > >>> > > >> >> > > > > >>> > > >> >> Gwen > > > > >>> > > >> >> > > > > >>> > > >> >> > > > > >>> > > >> >> > > > > >>> > > >> >> On Tue, Mar 8, 2016 at 11:46 AM, Rajini Sivaram > > > > >>> > > >> >> <rajinisiva...@googlemail.com> wrote: > > > > >>> > > >> >> > Gwen, > > > > >>> > > >> >> > > > > > >>> > > >> >> > I am not sure I understand your concern. Java login > > > > contexts > > > > >>> > > support > > > > >>> > > >> >> > multiple login modules precisely to support this type > > of > > > > >>> real-world > > > > >>> > > >> >> > scenario. > > > > >>> > > >> >> > > > > > >>> > > >> >> > From Javadoc of > > *javax.security.auth.login.LoginContext*: > > > > >>> > > >> >> > > > > > >>> > > >> >> > *In addition to supporting pluggable authentication, > > this > > > > >>> class > > > > >>> > > also > > > > >>> > > >> >> > supports the notion of stacked authentication. > > > Applications > > > > >>> may be > > > > >>> > > >> >> > configured to use more than one LoginModule. For > > example, > > > > >>> one could > > > > >>> > > >> >> > configure both a Kerberos LoginModule and a smart > card > > > > >>> LoginModule > > > > >>> > > >> under > > > > >>> > > >> >> an > > > > >>> > > >> >> > application. * > > > > >>> > > >> >> > > > > > >>> > > >> >> > * A typical caller instantiates a LoginContext with a > > > name > > > > >>> and a > > > > >>> > > >> >> > CallbackHandler. LoginContext uses the name as the > > index > > > > >>> into a > > > > >>> > > >> >> > Configuration to determine which LoginModules should > be > > > > >>> used, and > > > > >>> > > >> which > > > > >>> > > >> >> > ones must succeed in order for the overall > > authentication > > > > to > > > > >>> > > succeed. > > > > >>> > > >> The > > > > >>> > > >> >> > CallbackHandler is passed to the underlying > > LoginModules > > > so > > > > >>> they > > > > >>> > > may > > > > >>> > > >> >> > communicate and interact with users (prompting for a > > > > >>> username and > > > > >>> > > >> >> password > > > > >>> > > >> >> > via a graphical user interface, for example).* > > > > >>> > > >> >> > > > > > >>> > > >> >> > If it is good enough for Java, surely it must be good > > > > enough > > > > >>> for > > > > >>> > > >> Kafka. > > > > >>> > > >> >> We > > > > >>> > > >> >> > don't expect users to directly use existing classes > > when > > > > >>> they want > > > > >>> > > to > > > > >>> > > >> >> > support complex new mechanisms. Login is not a > standard > > > > >>> interface, > > > > >>> > > >> and we > > > > >>> > > >> >> > want to make callback handlers configurable. So we do > > > > require > > > > >>> > > users to > > > > >>> > > >> >> wrap > > > > >>> > > >> >> > their existing classes in our interface. And the > > > interfaces > > > > >>> make it > > > > >>> > > >> >> > possible to plugin any number of mechanisms. Am I > > missing > > > > >>> > > something? > > > > >>> > > >> >> > > > > > >>> > > >> >> > > > > > >>> > > >> >> > > > > > >>> > > >> >> > On Tue, Mar 8, 2016 at 4:49 PM, Gwen Shapira < > > > > >>> g...@confluent.io> > > > > >>> > > >> wrote: > > > > >>> > > >> >> > > > > > >>> > > >> >> >> Yes, I understand that. > > > > >>> > > >> >> >> > > > > >>> > > >> >> >> However, because the current configuration only > > allows a > > > > >>> single > > > > >>> > > >> >> >> callback and a single login module, I need to > supply a > > > > >>> class that > > > > >>> > > >> >> >> supports callbacks or logins for every single > > mechanism > > > > >>> that I > > > > >>> > > need > > > > >>> > > >> to > > > > >>> > > >> >> >> support. I question whether this is applicable in > > > > real-world > > > > >>> > > scenario > > > > >>> > > >> >> >> where a user may need to support mechanisms that > were > > > > >>> written by > > > > >>> > > >> >> >> different vendors and where a single class that > > > implements > > > > >>> all the > > > > >>> > > >> >> >> necessary modules or callsbacks is very unlikely to > > > exist. > > > > >>> > > >> >> >> > > > > >>> > > >> >> >> I agree that supporting a list of classes and > > > dynamically > > > > >>> using > > > > >>> > > the > > > > >>> > > >> >> >> correct callbacks and modules based on configuration > > is > > > > >>> complex, > > > > >>> > > but > > > > >>> > > >> I > > > > >>> > > >> >> >> don't think this feature is useful enough without > it. > > > > >>> > > >> >> >> > > > > >>> > > >> >> >> Gwen > > > > >>> > > >> >> >> > > > > >>> > > >> >> >> On Tue, Mar 8, 2016 at 2:40 AM, Rajini Sivaram > > > > >>> > > >> >> >> <rajinisiva...@googlemail.com> wrote: > > > > >>> > > >> >> >> > Gwen, > > > > >>> > > >> >> >> > > > > > >>> > > >> >> >> > Sorry if this is not clear in the KIP. The goal of > > the > > > > >>> KIP is to > > > > >>> > > >> >> enable > > > > >>> > > >> >> >> new > > > > >>> > > >> >> >> > mechanisms to be added without any changes to > Apache > > > > >>> Kafka. The > > > > >>> > > >> >> >> > configuration proposed by the KIP is based on the > > > > >>> > > configurability > > > > >>> > > >> of > > > > >>> > > >> >> the > > > > >>> > > >> >> >> > Java SASL API and what it would take to add a new > > > > >>> mechanism > > > > >>> > > >> similar to > > > > >>> > > >> >> >> > Kerberos/PLAIN/Digest-MD5 and the requirements > that > > > Tao > > > > >>> raised > > > > >>> > > for > > > > >>> > > >> >> custom > > > > >>> > > >> >> >> > authentication. > > > > >>> > > >> >> >> > > > > > >>> > > >> >> >> > The PR includes a test for Digest-MD5 which is not > > > > >>> included in > > > > >>> > > >> Kafka, > > > > >>> > > >> >> but > > > > >>> > > >> >> >> > is tested using the configuration options that > allow > > > new > > > > >>> > > mechanisms > > > > >>> > > >> >> to be > > > > >>> > > >> >> >> > added. > > > > >>> > > >> >> >> > > > > > >>> > > >> >> >> > If you want to add a more complex mechanism which > > > > requires > > > > >>> > > >> integration > > > > >>> > > >> >> >> with > > > > >>> > > >> >> >> > some external authentication libraries for login > or > > > > >>> callback > > > > >>> > > >> handlers, > > > > >>> > > >> >> >> you > > > > >>> > > >> >> >> > can do that by configuring your own callback > handler > > > and > > > > >>> login > > > > >>> > > >> >> >> > implementation. The login class allows you to add > > > > >>> additional > > > > >>> > > >> threads > > > > >>> > > >> >> (for > > > > >>> > > >> >> >> > instance for token refresh), while the callback > > > handler > > > > >>> extends > > > > >>> > > the > > > > >>> > > >> >> >> > standard Java CallbackHandler interface, enabling > > you > > > to > > > > >>> > > override > > > > >>> > > >> any > > > > >>> > > >> >> of > > > > >>> > > >> >> >> > the default callbacks in Kafka to provide your own > > > > >>> behaviour. If > > > > >>> > > >> you > > > > >>> > > >> >> want > > > > >>> > > >> >> >> > to add your own login modules, SASL server or SASL > > > > client > > > > >>> > > >> >> implementation > > > > >>> > > >> >> >> > for a mechanism that is not supported in your JDK, > > you > > > > >>> can do > > > > >>> > > that > > > > >>> > > >> by > > > > >>> > > >> >> >> > adding new Java security modules, using the > built-in > > > > >>> > > pluggability > > > > >>> > > >> of > > > > >>> > > >> >> Java > > > > >>> > > >> >> >> > security. > > > > >>> > > >> >> >> > > > > > >>> > > >> >> >> > And finally, if you want to enable multiple > > mechanisms > > > > in > > > > >>> your > > > > >>> > > >> Kafka > > > > >>> > > >> >> >> > brokers, you can specify a callback handler class > > that > > > > >>> either > > > > >>> > > >> provides > > > > >>> > > >> >> >> the > > > > >>> > > >> >> >> > handlers for all the mechanisms (as the PR does > for > > > > >>> Kerberos and > > > > >>> > > >> >> PLAIN) > > > > >>> > > >> >> >> or > > > > >>> > > >> >> >> > a delegator that calls different callback handlers > > > based > > > > >>> on the > > > > >>> > > >> >> >> mechanism. > > > > >>> > > >> >> >> > And for login, you provide a single or a compound > > > class > > > > >>> that > > > > >>> > > >> provides > > > > >>> > > >> >> any > > > > >>> > > >> >> >> > additional login logic for all the login modules > > that > > > > you > > > > >>> have > > > > >>> > > >> >> specified > > > > >>> > > >> >> >> > for KafkaServer. > > > > >>> > > >> >> >> > > > > > >>> > > >> >> >> > Hope that helps. > > > > >>> > > >> >> >> > > > > > >>> > > >> >> >> > > > > > >>> > > >> >> >> > On Tue, Mar 8, 2016 at 1:17 AM, Gwen Shapira < > > > > >>> g...@confluent.io > > > > >>> > > > > > > > >>> > > >> >> wrote: > > > > >>> > > >> >> >> > > > > > >>> > > >> >> >> >> Can you explain the process for a adding a new > > > > mechanism > > > > >>> based > > > > >>> > > on > > > > >>> > > >> >> >> current > > > > >>> > > >> >> >> >> KIP? > > > > >>> > > >> >> >> >> > > > > >>> > > >> >> >> >> My thought is that if it requires modifying > Apache > > > > Kafka > > > > >>> code, > > > > >>> > > it > > > > >>> > > >> is > > > > >>> > > >> >> >> >> not pluggable enough. > > > > >>> > > >> >> >> >> > > > > >>> > > >> >> >> >> On Mon, Mar 7, 2016 at 4:04 PM, Rajini Sivaram > > > > >>> > > >> >> >> >> <rajinisiva...@googlemail.com> wrote: > > > > >>> > > >> >> >> >> > Gwen, > > > > >>> > > >> >> >> >> > > > > > >>> > > >> >> >> >> > In cases where you want completely different > > > > callbacks > > > > >>> for > > > > >>> > > >> >> different > > > > >>> > > >> >> >> >> > mechanisms, I was thinking that the choice > would > > be > > > > >>> between a > > > > >>> > > >> map > > > > >>> > > >> >> of > > > > >>> > > >> >> >> >> > classes (mechanism -> callbackHandler class) > or a > > > > >>> delegator > > > > >>> > > >> class > > > > >>> > > >> >> that > > > > >>> > > >> >> >> >> > chooses the appropriate callback handler class > > > based > > > > on > > > > >>> > > >> mechanism. > > > > >>> > > >> >> I > > > > >>> > > >> >> >> >> chose > > > > >>> > > >> >> >> >> > the latter since it makes it easier to > configure > > in > > > > >>> Kafka. > > > > >>> > > >> Since we > > > > >>> > > >> >> >> >> create > > > > >>> > > >> >> >> >> > a callback handler for each channel and > configure > > > it > > > > >>> with the > > > > >>> > > >> >> >> >> > client-selected mechanism, it is > straightforward > > to > > > > >>> have one > > > > >>> > > >> >> wrapper > > > > >>> > > >> >> >> >> class > > > > >>> > > >> >> >> >> > that delegates to the right mechanism-specific > > > class > > > > to > > > > >>> > > handle > > > > >>> > > >> >> >> callbacks. > > > > >>> > > >> >> >> >> > In many cases, a single class may be sufficient > > > (the > > > > >>> PR uses > > > > >>> > > a > > > > >>> > > >> >> single > > > > >>> > > >> >> >> >> > callback class for Kerberos and PLAIN). I do > see > > > your > > > > >>> point > > > > >>> > > >> about > > > > >>> > > >> >> the > > > > >>> > > >> >> >> >> > flexibility that multiple classes would > provide, > > > but > > > > >>> since > > > > >>> > > you > > > > >>> > > >> need > > > > >>> > > >> >> >> to be > > > > >>> > > >> >> >> >> > able to associate the callback with a mechanism > > for > > > > >>> this to > > > > >>> > > be > > > > >>> > > >> >> >> useful, I > > > > >>> > > >> >> >> >> am > > > > >>> > > >> >> >> >> > not sure if just a list would add value. > > > > >>> > > >> >> >> >> > > > > > >>> > > >> >> >> >> > Login class is slightly different since the > > > proposal > > > > >>> is to > > > > >>> > > use a > > > > >>> > > >> >> >> single > > > > >>> > > >> >> >> >> > login context with multiple login modules to > > handle > > > > >>> multiple > > > > >>> > > >> >> >> mechanisms. > > > > >>> > > >> >> >> >> In > > > > >>> > > >> >> >> >> > this case, you want to perform login for all > the > > > > >>> mechanisms > > > > >>> > > that > > > > >>> > > >> >> are > > > > >>> > > >> >> >> >> > enabled. And you want to call > > loginContext.login() > > > > >>> only once. > > > > >>> > > >> >> Again, > > > > >>> > > >> >> >> you > > > > >>> > > >> >> >> >> > can delegate to multiple classes if you wish to > > add > > > > >>> some > > > > >>> > > complex > > > > >>> > > >> >> >> >> > mechanism-specific logic, but a single login > > class > > > > >>> makes the > > > > >>> > > >> >> mapping > > > > >>> > > >> >> >> to a > > > > >>> > > >> >> >> >> > single login context and the login cache more > > > obvious > > > > >>> (the PR > > > > >>> > > >> has a > > > > >>> > > >> >> >> test > > > > >>> > > >> >> >> >> > that includes Kerberos and PLAIN). > > > > >>> > > >> >> >> >> > > > > > >>> > > >> >> >> >> > Thoughts? > > > > >>> > > >> >> >> >> > > > > > >>> > > >> >> >> >> > On Mon, Mar 7, 2016 at 9:57 PM, Gwen Shapira < > > > > >>> > > g...@confluent.io > > > > >>> > > >> > > > > > >>> > > >> >> >> wrote: > > > > >>> > > >> >> >> >> > > > > > >>> > > >> >> >> >> >> Hi, > > > > >>> > > >> >> >> >> >> > > > > >>> > > >> >> >> >> >> Before I can vote on this KIP, I have two > > > additional > > > > >>> > > questions > > > > >>> > > >> / > > > > >>> > > >> >> >> >> >> comments on the new configuration: > > > > >>> > > >> >> >> >> >> > > > > >>> > > >> >> >> >> >> 1. sasl.callback.handler.class - it looks like > > we > > > > >>> want a > > > > >>> > > single > > > > >>> > > >> >> class > > > > >>> > > >> >> >> >> >> that implements all mechanisms. I think this > > will > > > > >>> make it > > > > >>> > > >> >> difficult > > > > >>> > > >> >> >> to > > > > >>> > > >> >> >> >> >> extend since the only way I can add a > mechanism > > > will > > > > >>> be by > > > > >>> > > >> >> >> >> >> implementing every single existing mechanism > > > > >>> (otherwise > > > > >>> > > >> customers > > > > >>> > > >> >> >> will > > > > >>> > > >> >> >> >> >> need to choose between new and existing when > > > > >>> selecting which > > > > >>> > > >> >> class to > > > > >>> > > >> >> >> >> >> use). If Microsoft releases a proprietary "AD > > > > >>> Mechanism" and > > > > >>> > > >> >> Oracle > > > > >>> > > >> >> >> >> >> releases "OID mechanism", there will be no > class > > > > that > > > > >>> > > >> implements > > > > >>> > > >> >> >> both. > > > > >>> > > >> >> >> >> >> Can we make it a list of classes instead? I > > > realize > > > > >>> this > > > > >>> > > >> >> complicates > > > > >>> > > >> >> >> >> >> the code a bit (some kind of factory will be > > > > required > > > > >>> to > > > > >>> > > choose > > > > >>> > > >> >> the > > > > >>> > > >> >> >> >> >> right class to use), but important IMO. > > > > >>> > > >> >> >> >> >> 2. similar for sasl.login.class - if I have a > > > class > > > > >>> for > > > > >>> > > >> Kerberos > > > > >>> > > >> >> >> (with > > > > >>> > > >> >> >> >> >> refresh thread) and a class for "plain", we > need > > > to > > > > >>> be able > > > > >>> > > to > > > > >>> > > >> >> load > > > > >>> > > >> >> >> >> >> both. > > > > >>> > > >> >> >> >> >> > > > > >>> > > >> >> >> >> >> Gwen > > > > >>> > > >> >> >> >> >> > > > > >>> > > >> >> >> >> >> On Wed, Mar 2, 2016 at 12:30 AM, Rajini > Sivaram > > > > >>> > > >> >> >> >> >> <rajinisiva...@googlemail.com> wrote: > > > > >>> > > >> >> >> >> >> > Jun, > > > > >>> > > >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> > Thanks, I have added a note to the KIP. I > will > > > > add a > > > > >>> > > comment > > > > >>> > > >> in > > > > >>> > > >> >> the > > > > >>> > > >> >> >> >> >> > implementation and also add a unit test to > > > ensure > > > > >>> that > > > > >>> > > >> conflicts > > > > >>> > > >> >> >> are > > > > >>> > > >> >> >> >> >> > avoided when version number is modified. > > > > >>> > > >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> > On Tue, Mar 1, 2016 at 5:43 PM, Jun Rao < > > > > >>> j...@confluent.io > > > > >>> > > > > > > > >>> > > >> >> wrote: > > > > >>> > > >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> >> Rajini, > > > > >>> > > >> >> >> >> >> >> > > > > >>> > > >> >> >> >> >> >> Thanks for the explanation. For 1, this > > implies > > > > >>> that we > > > > >>> > > have > > > > >>> > > >> >> to be > > > > >>> > > >> >> >> >> >> careful > > > > >>> > > >> >> >> >> >> >> with changing the 2-byte version in the > > future > > > to > > > > >>> avoid > > > > >>> > > >> >> conflict. > > > > >>> > > >> >> >> >> Could > > > > >>> > > >> >> >> >> >> you > > > > >>> > > >> >> >> >> >> >> document this in the KIP and also in the > > > > >>> implementation? > > > > >>> > > >> >> >> >> >> >> > > > > >>> > > >> >> >> >> >> >> Jun > > > > >>> > > >> >> >> >> >> >> > > > > >>> > > >> >> >> >> >> >> On Tue, Mar 1, 2016 at 2:47 AM, Rajini > > Sivaram > > > < > > > > >>> > > >> >> >> >> >> >> rajinisiva...@googlemail.com > > > > >>> > > >> >> >> >> >> >> > wrote: > > > > >>> > > >> >> >> >> >> >> > > > > >>> > > >> >> >> >> >> >> > Jun, > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> >> > Thank you for the review. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> >> > 1. With GSSAPI, the first context > > > > >>> establishment > > > > >>> > > packet > > > > >>> > > >> >> starts > > > > >>> > > >> >> >> >> with > > > > >>> > > >> >> >> >> >> the > > > > >>> > > >> >> >> >> >> >> > byte 0x60 (tag for APPLICATION-0) > > followed > > > > by > > > > >>> a > > > > >>> > > >> >> >> variable-length > > > > >>> > > >> >> >> >> >> >> encoded > > > > >>> > > >> >> >> >> >> >> > size, followed by various tags and > > > contents. > > > > >>> And the > > > > >>> > > >> >> packet > > > > >>> > > >> >> >> also > > > > >>> > > >> >> >> >> >> >> > contains a > > > > >>> > > >> >> >> >> >> >> > checksum. This is completely different > > > from > > > > >>> the > > > > >>> > > >> mechanism > > > > >>> > > >> >> >> packet > > > > >>> > > >> >> >> >> >> from > > > > >>> > > >> >> >> >> >> >> > Kafka > > > > >>> > > >> >> >> >> >> >> > clients which start with a two-byte > > > version > > > > >>> set to > > > > >>> > > zero > > > > >>> > > >> >> >> >> currently, > > > > >>> > > >> >> >> >> >> >> > followed > > > > >>> > > >> >> >> >> >> >> > by just a String mechanism. > > > > >>> > > >> >> >> >> >> >> > 2. Agreed, I have removed the version > > from > > > > the > > > > >>> > > server > > > > >>> > > >> >> >> response > > > > >>> > > >> >> >> >> in > > > > >>> > > >> >> >> >> >> the > > > > >>> > > >> >> >> >> >> >> > KIP. Thanks. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> >> > On Tue, Mar 1, 2016 at 2:33 AM, Jun Rao < > > > > >>> > > j...@confluent.io > > > > >>> > > >> > > > > > >>> > > >> >> >> wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> >> > > Rajini, > > > > >>> > > >> >> >> >> >> >> > > > > > > >>> > > >> >> >> >> >> >> > > Thanks for the updates. Just a couple > of > > > > minor > > > > >>> > > comments. > > > > >>> > > >> >> >> >> >> >> > > > > > > >>> > > >> >> >> >> >> >> > > 1. With the default GSSAPI, what's the > > > first > > > > >>> packet > > > > >>> > > that > > > > >>> > > >> >> the > > > > >>> > > >> >> >> >> client > > > > >>> > > >> >> >> >> >> >> sends > > > > >>> > > >> >> >> >> >> >> > > to the server? Is that completely > > different > > > > >>> from the > > > > >>> > > >> packet > > > > >>> > > >> >> >> >> format > > > > >>> > > >> >> >> >> >> that > > > > >>> > > >> >> >> >> >> >> > we > > > > >>> > > >> >> >> >> >> >> > > will use for non-GSSAPI mechanisms? > > > > >>> > > >> >> >> >> >> >> > > > > > > >>> > > >> >> >> >> >> >> > > 2. In the server response, it doesn't > > seem > > > > >>> that we > > > > >>> > > need > > > > >>> > > >> to > > > > >>> > > >> >> >> >> include > > > > >>> > > >> >> >> >> >> the > > > > >>> > > >> >> >> >> >> >> > > version since the client knows the > > version > > > of > > > > >>> the > > > > >>> > > >> request > > > > >>> > > >> >> >> that it > > > > >>> > > >> >> >> >> >> >> sends. > > > > >>> > > >> >> >> >> >> >> > > > > > > >>> > > >> >> >> >> >> >> > > Jun > > > > >>> > > >> >> >> >> >> >> > > > > > > >>> > > >> >> >> >> >> >> > > On Mon, Feb 29, 2016 at 10:14 AM, > Rajini > > > > >>> Sivaram < > > > > >>> > > >> >> >> >> >> >> > > rajinisiva...@googlemail.com> wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > > >>> > > >> >> >> >> >> >> > > > Harsha, > > > > >>> > > >> >> >> >> >> >> > > > > > > > >>> > > >> >> >> >> >> >> > > > Thank you for the review. I will wait > > > > >>> another day > > > > >>> > > to > > > > >>> > > >> see > > > > >>> > > >> >> if > > > > >>> > > >> >> >> >> there > > > > >>> > > >> >> >> >> >> is > > > > >>> > > >> >> >> >> >> >> > more > > > > >>> > > >> >> >> >> >> >> > > > feedback and then start a voting > > thread. > > > > >>> > > >> >> >> >> >> >> > > > > > > > >>> > > >> >> >> >> >> >> > > > Rajini > > > > >>> > > >> >> >> >> >> >> > > > > > > > >>> > > >> >> >> >> >> >> > > > On Mon, Feb 29, 2016 at 2:51 PM, > > Harsha < > > > > >>> > > >> ka...@harsha.io > > > > >>> > > >> >> > > > > > >>> > > >> >> >> >> wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > > > >>> > > >> >> >> >> >> >> > > > > Rajini, > > > > >>> > > >> >> >> >> >> >> > > > > Thanks for the > changes > > to > > > > >>> the KIP. > > > > >>> > > It > > > > >>> > > >> >> looks > > > > >>> > > >> >> >> >> good > > > > >>> > > >> >> >> >> >> to > > > > >>> > > >> >> >> >> >> >> > me. I > > > > >>> > > >> >> >> >> >> >> > > > > think we can move to > > > > voting. > > > > >>> > > >> >> >> >> >> >> > > > > Thanks, > > > > >>> > > >> >> >> >> >> >> > > > > Harsha > > > > >>> > > >> >> >> >> >> >> > > > > > > > > >>> > > >> >> >> >> >> >> > > > > On Mon, Feb 29, 2016, at 12:43 AM, > > > Rajini > > > > >>> Sivaram > > > > >>> > > >> >> wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > I have added some more detail to > > the > > > > KIP > > > > >>> based > > > > >>> > > on > > > > >>> > > >> the > > > > >>> > > >> >> >> >> >> discussion > > > > >>> > > >> >> >> >> >> >> in > > > > >>> > > >> >> >> >> >> >> > > the > > > > >>> > > >> >> >> >> >> >> > > > > > last KIP meeting to simplify > > support > > > > for > > > > >>> > > multiple > > > > >>> > > >> >> >> >> mechanisms. > > > > >>> > > >> >> >> >> >> >> Have > > > > >>> > > >> >> >> >> >> >> > > also > > > > >>> > > >> >> >> >> >> >> > > > > > changed the property names to > > reflect > > > > >>> this. > > > > >>> > > >> >> >> >> >> >> > > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > Also updated the PR in > > > > >>> > > >> >> >> >> >> >> > > > > > > > >>> https://issues.apache.org/jira/browse/KAFKA-3149 > > > > >>> > > >> >> >> >> >> >> > > > > > to > > > > >>> > > >> >> >> >> >> >> > > > > > reflect the KIP. > > > > >>> > > >> >> >> >> >> >> > > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > Any feedback is appreciated. > > > > >>> > > >> >> >> >> >> >> > > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > On Tue, Feb 23, 2016 at 9:36 PM, > > > Rajini > > > > >>> > > Sivaram < > > > > >>> > > >> >> >> >> >> >> > > > > > rajinisiva...@googlemail.com> > > wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > > I have updated the KIP based on > > the > > > > >>> > > discussion > > > > >>> > > >> in > > > > >>> > > >> >> the > > > > >>> > > >> >> >> KIP > > > > >>> > > >> >> >> >> >> >> meeting > > > > >>> > > >> >> >> >> >> >> > > > > today. > > > > >>> > > >> >> >> >> >> >> > > > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > > Comments and feedback are > > welcome. > > > > >>> > > >> >> >> >> >> >> > > > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > > On Wed, Feb 3, 2016 at 7:20 PM, > > > > Rajini > > > > >>> > > Sivaram < > > > > >>> > > >> >> >> >> >> >> > > > > > > rajinisiva...@googlemail.com> > > > wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > >> Hi Harsha, > > > > >>> > > >> >> >> >> >> >> > > > > > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >> Thank you for the review. Can > > you > > > > >>> clarify - > > > > >>> > > I > > > > >>> > > >> >> think > > > > >>> > > >> >> >> you > > > > >>> > > >> >> >> >> are > > > > >>> > > >> >> >> >> >> >> > saying > > > > >>> > > >> >> >> >> >> >> > > > > that > > > > >>> > > >> >> >> >> >> >> > > > > > >> the client should send its > > > mechanism > > > > >>> over > > > > >>> > > the > > > > >>> > > >> >> wire to > > > > >>> > > >> >> >> >> the > > > > >>> > > >> >> >> >> >> >> > server. > > > > >>> > > >> >> >> >> >> >> > > Is > > > > >>> > > >> >> >> >> >> >> > > > > that > > > > >>> > > >> >> >> >> >> >> > > > > > >> correct? The exchange is > > slightly > > > > >>> different > > > > >>> > > in > > > > >>> > > >> the > > > > >>> > > >> >> >> KIP > > > > >>> > > >> >> >> >> >> (the PR > > > > >>> > > >> >> >> >> >> >> > > > > matches the > > > > >>> > > >> >> >> >> >> >> > > > > > >> KIP) from the one you > described > > to > > > > >>> enable > > > > >>> > > >> >> >> >> interoperability > > > > >>> > > >> >> >> >> >> >> with > > > > >>> > > >> >> >> >> >> >> > > > > 0.9.0.0. > > > > >>> > > >> >> >> >> >> >> > > > > > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >> On Wed, Feb 3, 2016 at 1:56 > PM, > > > > >>> Harsha < > > > > >>> > > >> >> >> m...@harsha.io> > > > > >>> > > >> >> >> >> >> >> wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> Rajini, > > > > >>> > > >> >> >> >> >> >> > > > > > >>> I looked at the PR > > you > > > > >>> have. I > > > > >>> > > >> think > > > > >>> > > >> >> its > > > > >>> > > >> >> >> >> better > > > > >>> > > >> >> >> >> >> >> with > > > > >>> > > >> >> >> >> >> >> > > > your > > > > >>> > > >> >> >> >> >> >> > > > > > >>> earlier approach > > > rather > > > > >>> than > > > > >>> > > >> extending > > > > >>> > > >> >> >> the > > > > >>> > > >> >> >> >> >> >> protocol. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> What I was thinking initially > > is, > > > > >>> Broker > > > > >>> > > has a > > > > >>> > > >> >> >> config > > > > >>> > > >> >> >> >> >> option > > > > >>> > > >> >> >> >> >> >> of > > > > >>> > > >> >> >> >> >> >> > > say > > > > >>> > > >> >> >> >> >> >> > > > > > >>> sasl.mechanism = GSSAPI, > PLAIN > > > > >>> > > >> >> >> >> >> >> > > > > > >>> and the client can have > similar > > > > >>> config of > > > > >>> > > >> >> >> >> >> >> sasl.mechanism=PLAIN. > > > > >>> > > >> >> >> >> >> >> > > > > Client > > > > >>> > > >> >> >> >> >> >> > > > > > >>> can send its sasl mechanism > > > before > > > > >>> the > > > > >>> > > >> handshake > > > > >>> > > >> >> >> starts > > > > >>> > > >> >> >> >> >> and > > > > >>> > > >> >> >> >> >> >> if > > > > >>> > > >> >> >> >> >> >> > > the > > > > >>> > > >> >> >> >> >> >> > > > > > >>> broker accepts that > particular > > > > >>> mechanism > > > > >>> > > than > > > > >>> > > >> it > > > > >>> > > >> >> >> can go > > > > >>> > > >> >> >> >> >> ahead > > > > >>> > > >> >> >> >> >> >> > > with > > > > >>> > > >> >> >> >> >> >> > > > > > >>> handshake otherwise return a > > > error > > > > >>> saying > > > > >>> > > that > > > > >>> > > >> >> the > > > > >>> > > >> >> >> >> >> mechanism > > > > >>> > > >> >> >> >> >> >> > not > > > > >>> > > >> >> >> >> >> >> > > > > > >>> allowed. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> Thanks, > > > > >>> > > >> >> >> >> >> >> > > > > > >>> Harsha > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> On Wed, Feb 3, 2016, at 04:58 > > AM, > > > > >>> Rajini > > > > >>> > > >> Sivaram > > > > >>> > > >> >> >> wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > A slightly different > approach > > > for > > > > >>> > > supporting > > > > >>> > > >> >> >> >> different > > > > >>> > > >> >> >> >> >> SASL > > > > >>> > > >> >> >> >> >> >> > > > > mechanisms > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > within a broker is to allow > > the > > > > >>> same > > > > >>> > > >> "*security > > > > >>> > > >> >> >> >> >> protocol*" > > > > >>> > > >> >> >> >> >> >> to > > > > >>> > > >> >> >> >> >> >> > > be > > > > >>> > > >> >> >> >> >> >> > > > > used > > > > >>> > > >> >> >> >> >> >> > > > > > >>> on > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > different ports with > > different > > > > >>> > > configuration > > > > >>> > > >> >> >> >> options. An > > > > >>> > > >> >> >> >> >> >> > > > advantage > > > > >>> > > >> >> >> >> >> >> > > > > of > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > this > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > approach is that it extends > > the > > > > >>> > > >> >> configurability of > > > > >>> > > >> >> >> >> not > > > > >>> > > >> >> >> >> >> just > > > > >>> > > >> >> >> >> >> >> > > SASL, > > > > >>> > > >> >> >> >> >> >> > > > > but > > > > >>> > > >> >> >> >> >> >> > > > > > >>> any > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > protocol. For instance, it > > > would > > > > >>> enable > > > > >>> > > the > > > > >>> > > >> >> use of > > > > >>> > > >> >> >> >> SSL > > > > >>> > > >> >> >> >> >> with > > > > >>> > > >> >> >> >> >> >> > > > mutual > > > > >>> > > >> >> >> >> >> >> > > > > > >>> client > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > authentication on one port > or > > > > >>> different > > > > >>> > > >> >> >> certificate > > > > >>> > > >> >> >> >> >> chains > > > > >>> > > >> >> >> >> >> >> on > > > > >>> > > >> >> >> >> >> >> > > > > another. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > And > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > it avoids the need for SASL > > > > >>> mechanism > > > > >>> > > >> >> negotiation. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > Kafka would have the same > > > > >>> "*security > > > > >>> > > >> protocols" > > > > >>> > > >> >> >> >> >> *defined as > > > > >>> > > >> >> >> >> >> >> > > > today, > > > > >>> > > >> >> >> >> >> >> > > > > but > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > with > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > (a single) configurable > SASL > > > > >>> mechanism. > > > > >>> > > To > > > > >>> > > >> have > > > > >>> > > >> >> >> >> >> different > > > > >>> > > >> >> >> >> >> >> > > > > > >>> configurations > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > of > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > a protocol within a broker, > > > users > > > > >>> can > > > > >>> > > define > > > > >>> > > >> >> new > > > > >>> > > >> >> >> >> >> protocol > > > > >>> > > >> >> >> >> >> >> > names > > > > >>> > > >> >> >> >> >> >> > > > > which > > > > >>> > > >> >> >> >> >> >> > > > > > >>> are > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > configured versions of > > existing > > > > >>> > > protocols, > > > > >>> > > >> >> perhaps > > > > >>> > > >> >> >> >> using > > > > >>> > > >> >> >> >> >> >> just > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > configuration > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > entries and no additional > > code. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > For example: > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > A single mechanism broker > > would > > > > be > > > > >>> > > >> configured > > > > >>> > > >> >> as: > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > listeners=SASL_SSL://:9092 > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > sasl.mechanism=GSSAPI > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > sasl.kerberos.class.name > > =kafka > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > ... > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > And a multi-mechanism > broker > > > > would > > > > >>> be > > > > >>> > > >> >> configured > > > > >>> > > >> >> >> as: > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> > listeners=gssapi://:9092,plain://:9093,custom://:9094 > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > gssapi.security.protocol=SASL_SSL > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > gssapi.sasl.mechanism=GSSAPI > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > gssapi.sasl.kerberos.class.name > > > > >>> =kafka > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > ... > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > plain.security.protocol=SASL_SSL > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > plain.sasl.mechanism=PLAIN > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > .. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> custom.security.protocol=SASL_PLAINTEXT > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > custom.sasl.mechanism=CUSTOM > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > > >>> > > >> >> >> > > > > >>> custom.sasl.callback.handler.class=example.CustomCallbackHandler > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > This is still a big change > > > > because > > > > >>> it > > > > >>> > > >> affects > > > > >>> > > >> >> the > > > > >>> > > >> >> >> >> >> currently > > > > >>> > > >> >> >> >> >> >> > > fixed > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > enumeration of security > > > protocol > > > > >>> > > >> definitions, > > > > >>> > > >> >> but > > > > >>> > > >> >> >> one > > > > >>> > > >> >> >> >> >> that > > > > >>> > > >> >> >> >> >> >> is > > > > >>> > > >> >> >> >> >> >> > > > > perhaps > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > more > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > flexible than defining > every > > > new > > > > >>> SASL > > > > >>> > > >> mechanism > > > > >>> > > >> >> >> as a > > > > >>> > > >> >> >> >> new > > > > >>> > > >> >> >> >> >> >> > > security > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > protocol. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > Thoughts? > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > On Tue, Feb 2, 2016 at > 12:20 > > > PM, > > > > >>> Rajini > > > > >>> > > >> >> Sivaram < > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > rajinisiva...@googlemail.com > > > > > > > >>> wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > As Ismael has said, we do > > not > > > > >>> have a > > > > >>> > > >> >> >> requirement to > > > > >>> > > >> >> >> >> >> >> support > > > > >>> > > >> >> >> >> >> >> > > > > multiple > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > protocols in a broker. > But > > I > > > > >>> agree with > > > > >>> > > >> Jun's > > > > >>> > > >> >> >> >> >> observation > > > > >>> > > >> >> >> >> >> >> > > that > > > > >>> > > >> >> >> >> >> >> > > > > some > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > companies might want to > > > > support a > > > > >>> > > >> different > > > > >>> > > >> >> >> >> >> >> authentication > > > > >>> > > >> >> >> >> >> >> > > > > mechanism > > > > >>> > > >> >> >> >> >> >> > > > > > >>> for > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > internal users or > partners. > > > For > > > > >>> > > instance, > > > > >>> > > >> we > > > > >>> > > >> >> do > > > > >>> > > >> >> >> use > > > > >>> > > >> >> >> >> >> two > > > > >>> > > >> >> >> >> >> >> > > > different > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > authentication > mechanisms, > > it > > > > >>> just so > > > > >>> > > >> happens > > > > >>> > > >> >> >> that > > > > >>> > > >> >> >> >> we > > > > >>> > > >> >> >> >> >> are > > > > >>> > > >> >> >> >> >> >> > > able > > > > >>> > > >> >> >> >> >> >> > > > > to use > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > certificate-based > > > > authentication > > > > >>> for > > > > >>> > > >> internal > > > > >>> > > >> >> >> >> users, > > > > >>> > > >> >> >> >> >> and > > > > >>> > > >> >> >> >> >> >> > > hence > > > > >>> > > >> >> >> >> >> >> > > > > don't > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > require multiple SASL > > > > mechanisms > > > > >>> in a > > > > >>> > > >> broker. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > As Tao has pointed out, > > > > mechanism > > > > >>> > > >> negotiation > > > > >>> > > >> >> >> is a > > > > >>> > > >> >> >> >> >> common > > > > >>> > > >> >> >> >> >> >> > > usage > > > > >>> > > >> >> >> >> >> >> > > > > > >>> pattern. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > Many existing protocols > > that > > > > >>> support > > > > >>> > > SASL > > > > >>> > > >> do > > > > >>> > > >> >> >> >> already > > > > >>> > > >> >> >> >> >> use > > > > >>> > > >> >> >> >> >> >> > this > > > > >>> > > >> >> >> >> >> >> > > > > > >>> pattern. AMQP > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > ( > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > >>> > > >> >> >> >> >> >> > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > > > >>> > > >> >> >> >> >> >> > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> >> > > > > >>> > > >> >> >> >> >> > > > > >>> > > >> >> >> >> > > > > >>> > > >> >> >> > > > > >>> > > >> >> > > > > >>> > > >> > > > > >>> > > > > > > >>> > > > > > > > > > > http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-security-v1.0-os.html#type-sasl-mechanisms > > > > >>> > > >> >> >> >> >> >> > > > > > >>> ), > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > which, as a messaging > > > protocol > > > > >>> maybe > > > > >>> > > >> closer > > > > >>> > > >> >> to > > > > >>> > > >> >> >> >> Kafka > > > > >>> > > >> >> >> >> >> in > > > > >>> > > >> >> >> >> >> >> use > > > > >>> > > >> >> >> >> >> >> > > > cases > > > > >>> > > >> >> >> >> >> >> > > > > > >>> than > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > Zookeeper, is an example. > > > Other > > > > >>> > > examples > > > > >>> > > >> >> where > > > > >>> > > >> >> >> the > > > > >>> > > >> >> >> >> >> client > > > > >>> > > >> >> >> >> >> >> > > > > negotiates > > > > >>> > > >> >> >> >> >> >> > > > > > >>> or > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > sends SASL mechanism to > > > server > > > > >>> include > > > > >>> > > >> ACAP > > > > >>> > > >> >> >> that is > > > > >>> > > >> >> >> >> >> used > > > > >>> > > >> >> >> >> >> >> as > > > > >>> > > >> >> >> >> >> >> > > an > > > > >>> > > >> >> >> >> >> >> > > > > > >>> example in > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > the SASL RFCs, POP3, > LDAP, > > > SMTP > > > > >>> etc. > > > > >>> > > This > > > > >>> > > >> is > > > > >>> > > >> >> >> not to > > > > >>> > > >> >> >> >> >> say > > > > >>> > > >> >> >> >> >> >> > that > > > > >>> > > >> >> >> >> >> >> > > > > Kafka > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > shouldn't use a different > > > type > > > > of > > > > >>> > > >> mechanism > > > > >>> > > >> >> >> >> selection > > > > >>> > > >> >> >> >> >> >> that > > > > >>> > > >> >> >> >> >> >> > > fits > > > > >>> > > >> >> >> >> >> >> > > > > > >>> better with > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > the existing Kafka > design. > > > Just > > > > >>> that > > > > >>> > > >> >> negotiation > > > > >>> > > >> >> >> >> is a > > > > >>> > > >> >> >> >> >> >> > common > > > > >>> > > >> >> >> >> >> >> > > > > pattern > > > > >>> > > >> >> >> >> >> >> > > > > > >>> and > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > since we typically turn > on > > > > >>> > > >> javax.net.debug to > > > > >>> > > >> >> >> debug > > > > >>> > > >> >> >> >> >> TLS > > > > >>> > > >> >> >> >> >> >> > > > > negotiation > > > > >>> > > >> >> >> >> >> >> > > > > > >>> issues, > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > having to use Kafka > logging > > > to > > > > >>> debug > > > > >>> > > SASL > > > > >>> > > >> >> >> >> negotiation > > > > >>> > > >> >> >> >> >> >> > issues > > > > >>> > > >> >> >> >> >> >> > > is > > > > >>> > > >> >> >> >> >> >> > > > > not > > > > >>> > > >> >> >> >> >> >> > > > > > >>> that > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > dissimilar. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > On Tue, Feb 2, 2016 at > 6:12 > > > AM, > > > > >>> tao > > > > >>> > > xiao < > > > > >>> > > >> >> >> >> >> >> > > xiaotao...@gmail.com > > > > >>> > > >> >> >> >> >> >> > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> I am the author of > > KIP-44. I > > > > >>> hope my > > > > >>> > > use > > > > >>> > > >> >> case > > > > >>> > > >> >> >> will > > > > >>> > > >> >> >> >> >> add > > > > >>> > > >> >> >> >> >> >> > some > > > > >>> > > >> >> >> >> >> >> > > > > values > > > > >>> > > >> >> >> >> >> >> > > > > > >>> to this > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> discussion. The reason I > > > > raised > > > > >>> KIP44 > > > > >>> > > is > > > > >>> > > >> >> that I > > > > >>> > > >> >> >> >> want > > > > >>> > > >> >> >> >> >> to > > > > >>> > > >> >> >> >> >> >> be > > > > >>> > > >> >> >> >> >> >> > > > able > > > > >>> > > >> >> >> >> >> >> > > > > to > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> implement a custom > > security > > > > >>> protocol > > > > >>> > > that > > > > >>> > > >> >> can > > > > >>> > > >> >> >> >> fulfill > > > > >>> > > >> >> >> >> >> >> the > > > > >>> > > >> >> >> >> >> >> > > need > > > > >>> > > >> >> >> >> >> >> > > > > of my > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> company. As pointed out > by > > > > >>> Ismael > > > > >>> > > KIP-43 > > > > >>> > > >> now > > > > >>> > > >> >> >> >> >> supports a > > > > >>> > > >> >> >> >> >> >> > > > > pluggable > > > > >>> > > >> >> >> >> >> >> > > > > > >>> way to > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> inject custom security > > > > provider > > > > >>> to > > > > >>> > > SASL I > > > > >>> > > >> >> think > > > > >>> > > >> >> >> >> it is > > > > >>> > > >> >> >> >> >> >> > enough > > > > >>> > > >> >> >> >> >> >> > > > to > > > > >>> > > >> >> >> >> >> >> > > > > > >>> cover the > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> use case I have and > > address > > > > the > > > > >>> > > concerns > > > > >>> > > >> >> >> raised in > > > > >>> > > >> >> >> >> >> >> KIP-44. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> For multiple security > > > > protocols > > > > >>> > > support > > > > >>> > > >> >> >> >> >> simultaneously > > > > >>> > > >> >> >> >> >> >> it > > > > >>> > > >> >> >> >> >> >> > is > > > > >>> > > >> >> >> >> >> >> > > > not > > > > >>> > > >> >> >> >> >> >> > > > > > >>> needed in > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> my use case and I don't > > > > foresee > > > > >>> it is > > > > >>> > > >> >> needed in > > > > >>> > > >> >> >> >> the > > > > >>> > > >> >> >> >> >> >> future > > > > >>> > > >> >> >> >> >> >> > > but > > > > >>> > > >> >> >> >> >> >> > > > > as i > > > > >>> > > >> >> >> >> >> >> > > > > > >>> said > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> this is my use case only > > > there > > > > >>> may be > > > > >>> > > >> other > > > > >>> > > >> >> use > > > > >>> > > >> >> >> >> cases > > > > >>> > > >> >> >> >> >> >> that > > > > >>> > > >> >> >> >> >> >> > > > need > > > > >>> > > >> >> >> >> >> >> > > > > it. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> But if > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> we want to support it in > > the > > > > >>> future I > > > > >>> > > >> >> prefer to > > > > >>> > > >> >> >> >> get > > > > >>> > > >> >> >> >> >> it > > > > >>> > > >> >> >> >> >> >> > right > > > > >>> > > >> >> >> >> >> >> > > > at > > > > >>> > > >> >> >> >> >> >> > > > > the > > > > >>> > > >> >> >> >> >> >> > > > > > >>> first > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> place given the fact > that > > > > >>> security > > > > >>> > > >> protocol > > > > >>> > > >> >> is > > > > >>> > > >> >> >> an > > > > >>> > > >> >> >> >> >> ENUM > > > > >>> > > >> >> >> >> >> >> and > > > > >>> > > >> >> >> >> >> >> > > if > > > > >>> > > >> >> >> >> >> >> > > > we > > > > >>> > > >> >> >> >> >> >> > > > > > >>> stick to > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> that implementation it > is > > > very > > > > >>> hard to > > > > >>> > > >> >> extend > > > > >>> > > >> >> >> in > > > > >>> > > >> >> >> >> the > > > > >>> > > >> >> >> >> >> >> > future > > > > >>> > > >> >> >> >> >> >> > > > > when we > > > > >>> > > >> >> >> >> >> >> > > > > > >>> decide > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> multiple security > > protocols > > > is > > > > >>> needed. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> Protocol negotiation is > a > > > very > > > > >>> common > > > > >>> > > >> usage > > > > >>> > > >> >> >> >> pattern > > > > >>> > > >> >> >> >> >> in > > > > >>> > > >> >> >> >> >> >> > > > security > > > > >>> > > >> >> >> >> >> >> > > > > > >>> domain. As > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> suggested in Java SASL > doc > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > >>> > > >> >> >> >> >> >> > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > > > >>> > > >> >> >> >> >> >> > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > >> >> >> >> >> >> > > > > >>> > > >> >> >> >> >> > > > > >>> > > >> >> >> >> > > > > >>> > > >> >> >> > > > > >>> > > >> >> > > > > >>> > > >> > > > > >>> > > > > > > >>> > > > > > > > > > > http://docs.oracle.com/javase/7/docs/technotes/guides/security/sasl/sasl-refguide.html > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> client > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> first sends out a packet > > to > > > > >>> server and > > > > >>> > > >> >> server > > > > >>> > > >> >> >> >> >> responds > > > > >>> > > >> >> >> >> >> >> > with > > > > >>> > > >> >> >> >> >> >> > > a > > > > >>> > > >> >> >> >> >> >> > > > > list > > > > >>> > > >> >> >> >> >> >> > > > > > >>> of > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> mechanisms it supports. > > This > > > > is > > > > >>> very > > > > >>> > > >> >> similar to > > > > >>> > > >> >> >> >> >> SSL/TLS > > > > >>> > > >> >> >> >> >> >> > > > > negotiation. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> On Tue, 2 Feb 2016 at > > 06:39 > > > > >>> Ismael > > > > >>> > > Juma < > > > > >>> > > >> >> >> >> >> >> > ism...@juma.me.uk> > > > > >>> > > >> >> >> >> >> >> > > > > wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > On Mon, Feb 1, 2016 at > > > 7:04 > > > > >>> PM, Gwen > > > > >>> > > >> >> Shapira > > > > >>> > > >> >> >> < > > > > >>> > > >> >> >> >> >> >> > > > > g...@confluent.io> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> wrote: > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > Looking at "existing > > > > >>> solutions", > > > > >>> > > it > > > > >>> > > >> >> looks > > > > >>> > > >> >> >> like > > > > >>> > > >> >> >> >> >> >> > Zookeeper > > > > >>> > > >> >> >> >> >> >> > > > > allows > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> plugging > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > in > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > any SASL mechanism, > > but > > > > the > > > > >>> server > > > > >>> > > >> will > > > > >>> > > >> >> >> only > > > > >>> > > >> >> >> >> >> support > > > > >>> > > >> >> >> >> >> >> > one > > > > >>> > > >> >> >> >> >> >> > > > > > >>> mechanism at > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> a > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > time. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > This was the original > > > > >>> proposal from > > > > >>> > > >> >> Rajini as > > > > >>> > > >> >> >> >> that > > > > >>> > > >> >> >> >> >> is > > > > >>> > > >> >> >> >> >> >> > > enough > > > > >>> > > >> >> >> >> >> >> > > > > for > > > > >>> > > >> >> >> >> >> >> > > > > > >>> their > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > needs. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > If this is good > enough > > > for > > > > >>> our > > > > >>> > > >> use-case > > > > >>> > > >> >> >> (do we > > > > >>> > > >> >> >> >> >> >> > actually > > > > >>> > > >> >> >> >> >> >> > > > > need to > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> support > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > multiple mechanisms > at > > > > >>> once?), it > > > > >>> > > >> will > > > > >>> > > >> >> >> >> simplify > > > > >>> > > >> >> >> >> >> >> life a > > > > >>> > > >> >> >> >> >> >> > > lot > > > > >>> > > >> >> >> >> >> >> > > > > for > > > > >>> > > >> >> >> >> >> >> > > > > > >>> us ( > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > > > > >>> > > >> >> >> >> >> >> > > > > > > > > >>> > > >> >> >> >> >> >> > > > > > > >>> > > >> >> >> >> >> >> > > > > >>> > > >> >> >> >> >> > > > > >>> > > >> >> >> >> > > > > >>> > > >> >> >> > > > > >>> > > >> >> > > > > >>> > > >> > > > > >>> > > > > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zookeeper+and+SASL > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > ) > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > The current thinking > is > > > that > > > > >>> it > > > > >>> > > would > > > > >>> > > >> be > > > > >>> > > >> >> >> useful > > > > >>> > > >> >> >> >> to > > > > >>> > > >> >> >> >> >> >> > support > > > > >>> > > >> >> >> >> >> >> > > > > > >>> multiple SASL > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > mechanisms > > simultaneously. > > > > In > > > > >>> the > > > > >>> > > KIP > > > > >>> > > >> >> >> meeting, > > > > >>> > > >> >> >> >> Jun > > > > >>> > > >> >> >> >> >> >> > > mentioned > > > > >>> > > >> >> >> >> >> >> > > > > that > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> companies > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > sometimes support > > > additional > > > > >>> > > >> >> authentication > > > > >>> > > >> >> >> >> >> mechanisms > > > > >>> > > >> >> >> >> >> >> > for > > > > >>> > > >> >> >> >> >> >> > > > > > >>> partners, for > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > example. It does make > > > things > > > > >>> more > > > > >>> > > >> >> complex, as > > > > >>> > > >> >> >> >> you > > > > >>> > > >> >> >> >> >> say, > > > > >>> > > >> >> >> >> >> >> > so > > > > >>> > > >> >> >> >> >> >> > > we > > > > >>> > > >> >> >> >> >> >> > > > > need > > > > >>> > > >> >> >> >> >> >> > > > > > >>> to be > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > sure the complexity is > > > worth > > > > >>> it. > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > Two more points: > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > 1. It has been > suggested > > > > that > > > > >>> custom > > > > >>> > > >> >> security > > > > >>> > > >> >> >> >> >> protocol > > > > >>> > > >> >> >> >> >> >> > > > > support is > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> needed by > > > > >>> > > >> >> >> >> >> >> > > > > > >>> > >> > some (KIP-44). Rajini > > > > enhanced > > > > >>> > > KIP-43 > > > > >>> > > >> so > > > > >>> > > >> >> >> that a > > > > >>> > > >> >> >> >> >> SASL > > > > >>> > > >> >> >> >> >> >> > > > mechanism > > > > >>> > > >> >> >> >> > > > > > >>> > > > > >> ... > > > > >> > > > > >> [Message clipped] > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Rajini > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Rajini > > > > > > > > > > > > > > > -- > > Regards, > > > > Rajini > > > -- Regards, Rajini