Re: KAFKA-6654 custom SSLContext
Hi Clement, I think many of the previous discussions around custom SslFactory was to do with config options (e.g. avoid cleartext password, configure alias to choose certificate, OCSP support etc.) For these, alternative config options make the most sense because these are generic config options that are useful in many environments, so we should support these without having to write Java code. But perhaps in your case where you have an existing implementation you want to reuse (or share an implementation across multiple applications), customization of SslFactory would be the simplest approach. I would suggest writing a KIP that describes the motivation and rejected alternatives to move this forward. Regards, Rajini On Tue, Oct 16, 2018 at 4:07 AM Colin McCabe wrote: > Hi Clement, > > Thanks for the clarification. Perhaps a pluggable interface makes sense > here. Maybe someone more familiar with the SSL code can comment. > > best, > Colin > > > On Mon, Oct 15, 2018, at 19:53, Pellerin, Clement wrote: > > OK, I can see why passing an instance is not language neutral. > > All the libraries I can think of accept the SSLSocketFactory, but they > > most likely don't support C or Python. > > > > My exact use case is to reuse the SSLContext configured in my > > application outside Kafka. > > I'm afraid no amount of extra configuration properties can achieve that. > > It appears the creator of KAFKA-6654 agrees with me. > > > > I could solve my problem if I could convince SslChannelBuilder to create > > my own SslFactory implementation. > > The Kafka config already contains properties that hold class names. > > Like I suggested before, we could have a property for the class name > > that implements an SslFactory interface. > > I would also need to pass custom config parameters to my SslFactory > > implementation without causing warnings. > > By default, the SslFactory implementation would be the one built into > > Kafka which uses all the Kafka ssl properties. > > > > Is that acceptable to resolve KAFKA-6654? > > Can you think of a better solution? > > > > > > -----Original Message- > > From: Colin McCabe [mailto:cmcc...@apache.org] > > Sent: Monday, October 15, 2018 7:58 PM > > To: dev@kafka.apache.org > > Subject: Re: KAFKA-6654 custom SSLContext > > > > In general Kafka makes an effort to be langauge-neutral so that Kafka > > clients can be implemented on platforms other than Java. For example, > > we have things like librdkafka which allow people to access Kafka from C > > or Python. Unless I'm misunderstanding something, giving direct access > > to the SSLContext and SSLSocketFactory seems like it would make that > > kind of compatibility harder, if it were even still possible. I'm > > curious if there's a way to do this by adding configuration entries for > > what you need? > > > > best, > > Colin > > > > > > On Mon, Oct 15, 2018, at 13:20, Pellerin, Clement wrote: > > > I am new to this mailing list. I am not sure what I should do next. > > > Should I create a KIP to discuss this? > > > > > > -Original Message- > > > From: Pellerin, Clement > > > Sent: Wednesday, October 10, 2018 4:38 PM > > > To: dev@kafka.apache.org > > > Subject: KAFKA-6654 custom SSLContext > > > > > > KAFKA-6654 correctly states that there will never be enough > > > configuration parameters to fully configure the SSLContext/ > > > SSLSocketFactory created by Kafka. > > > For example, in our case, we need an alias to choose the key in the > > > keystore, and we need an implementation of OCSP. > > > KAFKA-6654 suggests to make the creation of the SSLContext a pluggable > > > implementation. > > > Maybe by declaring an interface and passing the name of an > > > implementation class in a new parameter. > > > > > > Many libraries solve this problem by accepting the SSLContextFactory > > > instance from the application. > > > How about passing the instance as the value of a runtime configuration > > > parameter? > > > If that parameter is set, all other ssl.* parameters would be ignored. > > > Obviously, this parameter could only be set programmatically. > > > > > > I would like to hear the proposed solution by the Kafka maintainers. > > > > > > I can help implementing a patch if there is an agreement on the > desired > > > solution. >
Re: KAFKA-6654 custom SSLContext
Hi Clement, Thanks for the clarification. Perhaps a pluggable interface makes sense here. Maybe someone more familiar with the SSL code can comment. best, Colin On Mon, Oct 15, 2018, at 19:53, Pellerin, Clement wrote: > OK, I can see why passing an instance is not language neutral. > All the libraries I can think of accept the SSLSocketFactory, but they > most likely don't support C or Python. > > My exact use case is to reuse the SSLContext configured in my > application outside Kafka. > I'm afraid no amount of extra configuration properties can achieve that. > It appears the creator of KAFKA-6654 agrees with me. > > I could solve my problem if I could convince SslChannelBuilder to create > my own SslFactory implementation. > The Kafka config already contains properties that hold class names. > Like I suggested before, we could have a property for the class name > that implements an SslFactory interface. > I would also need to pass custom config parameters to my SslFactory > implementation without causing warnings. > By default, the SslFactory implementation would be the one built into > Kafka which uses all the Kafka ssl properties. > > Is that acceptable to resolve KAFKA-6654? > Can you think of a better solution? > > > -Original Message- > From: Colin McCabe [mailto:cmcc...@apache.org] > Sent: Monday, October 15, 2018 7:58 PM > To: dev@kafka.apache.org > Subject: Re: KAFKA-6654 custom SSLContext > > In general Kafka makes an effort to be langauge-neutral so that Kafka > clients can be implemented on platforms other than Java. For example, > we have things like librdkafka which allow people to access Kafka from C > or Python. Unless I'm misunderstanding something, giving direct access > to the SSLContext and SSLSocketFactory seems like it would make that > kind of compatibility harder, if it were even still possible. I'm > curious if there's a way to do this by adding configuration entries for > what you need? > > best, > Colin > > > On Mon, Oct 15, 2018, at 13:20, Pellerin, Clement wrote: > > I am new to this mailing list. I am not sure what I should do next. > > Should I create a KIP to discuss this? > > > > -Original Message- > > From: Pellerin, Clement > > Sent: Wednesday, October 10, 2018 4:38 PM > > To: dev@kafka.apache.org > > Subject: KAFKA-6654 custom SSLContext > > > > KAFKA-6654 correctly states that there will never be enough > > configuration parameters to fully configure the SSLContext/ > > SSLSocketFactory created by Kafka. > > For example, in our case, we need an alias to choose the key in the > > keystore, and we need an implementation of OCSP. > > KAFKA-6654 suggests to make the creation of the SSLContext a pluggable > > implementation. > > Maybe by declaring an interface and passing the name of an > > implementation class in a new parameter. > > > > Many libraries solve this problem by accepting the SSLContextFactory > > instance from the application. > > How about passing the instance as the value of a runtime configuration > > parameter? > > If that parameter is set, all other ssl.* parameters would be ignored. > > Obviously, this parameter could only be set programmatically. > > > > I would like to hear the proposed solution by the Kafka maintainers. > > > > I can help implementing a patch if there is an agreement on the desired > > solution.
RE: KAFKA-6654 custom SSLContext
OK, I can see why passing an instance is not language neutral. All the libraries I can think of accept the SSLSocketFactory, but they most likely don't support C or Python. My exact use case is to reuse the SSLContext configured in my application outside Kafka. I'm afraid no amount of extra configuration properties can achieve that. It appears the creator of KAFKA-6654 agrees with me. I could solve my problem if I could convince SslChannelBuilder to create my own SslFactory implementation. The Kafka config already contains properties that hold class names. Like I suggested before, we could have a property for the class name that implements an SslFactory interface. I would also need to pass custom config parameters to my SslFactory implementation without causing warnings. By default, the SslFactory implementation would be the one built into Kafka which uses all the Kafka ssl properties. Is that acceptable to resolve KAFKA-6654? Can you think of a better solution? -Original Message- From: Colin McCabe [mailto:cmcc...@apache.org] Sent: Monday, October 15, 2018 7:58 PM To: dev@kafka.apache.org Subject: Re: KAFKA-6654 custom SSLContext In general Kafka makes an effort to be langauge-neutral so that Kafka clients can be implemented on platforms other than Java. For example, we have things like librdkafka which allow people to access Kafka from C or Python. Unless I'm misunderstanding something, giving direct access to the SSLContext and SSLSocketFactory seems like it would make that kind of compatibility harder, if it were even still possible. I'm curious if there's a way to do this by adding configuration entries for what you need? best, Colin On Mon, Oct 15, 2018, at 13:20, Pellerin, Clement wrote: > I am new to this mailing list. I am not sure what I should do next. > Should I create a KIP to discuss this? > > -Original Message- > From: Pellerin, Clement > Sent: Wednesday, October 10, 2018 4:38 PM > To: dev@kafka.apache.org > Subject: KAFKA-6654 custom SSLContext > > KAFKA-6654 correctly states that there will never be enough > configuration parameters to fully configure the SSLContext/ > SSLSocketFactory created by Kafka. > For example, in our case, we need an alias to choose the key in the > keystore, and we need an implementation of OCSP. > KAFKA-6654 suggests to make the creation of the SSLContext a pluggable > implementation. > Maybe by declaring an interface and passing the name of an > implementation class in a new parameter. > > Many libraries solve this problem by accepting the SSLContextFactory > instance from the application. > How about passing the instance as the value of a runtime configuration > parameter? > If that parameter is set, all other ssl.* parameters would be ignored. > Obviously, this parameter could only be set programmatically. > > I would like to hear the proposed solution by the Kafka maintainers. > > I can help implementing a patch if there is an agreement on the desired > solution.
Re: KAFKA-6654 custom SSLContext
In general Kafka makes an effort to be langauge-neutral so that Kafka clients can be implemented on platforms other than Java. For example, we have things like librdkafka which allow people to access Kafka from C or Python. Unless I'm misunderstanding something, giving direct access to the SSLContext and SSLSocketFactory seems like it would make that kind of compatibility harder, if it were even still possible. I'm curious if there's a way to do this by adding configuration entries for what you need? best, Colin On Mon, Oct 15, 2018, at 13:20, Pellerin, Clement wrote: > I am new to this mailing list. I am not sure what I should do next. > Should I create a KIP to discuss this? > > -Original Message- > From: Pellerin, Clement > Sent: Wednesday, October 10, 2018 4:38 PM > To: dev@kafka.apache.org > Subject: KAFKA-6654 custom SSLContext > > KAFKA-6654 correctly states that there will never be enough > configuration parameters to fully configure the SSLContext/ > SSLSocketFactory created by Kafka. > For example, in our case, we need an alias to choose the key in the > keystore, and we need an implementation of OCSP. > KAFKA-6654 suggests to make the creation of the SSLContext a pluggable > implementation. > Maybe by declaring an interface and passing the name of an > implementation class in a new parameter. > > Many libraries solve this problem by accepting the SSLContextFactory > instance from the application. > How about passing the instance as the value of a runtime configuration > parameter? > If that parameter is set, all other ssl.* parameters would be ignored. > Obviously, this parameter could only be set programmatically. > > I would like to hear the proposed solution by the Kafka maintainers. > > I can help implementing a patch if there is an agreement on the desired > solution.
RE: KAFKA-6654 custom SSLContext
I am new to this mailing list. I am not sure what I should do next. Should I create a KIP to discuss this? -Original Message- From: Pellerin, Clement Sent: Wednesday, October 10, 2018 4:38 PM To: dev@kafka.apache.org Subject: KAFKA-6654 custom SSLContext KAFKA-6654 correctly states that there will never be enough configuration parameters to fully configure the SSLContext/SSLSocketFactory created by Kafka. For example, in our case, we need an alias to choose the key in the keystore, and we need an implementation of OCSP. KAFKA-6654 suggests to make the creation of the SSLContext a pluggable implementation. Maybe by declaring an interface and passing the name of an implementation class in a new parameter. Many libraries solve this problem by accepting the SSLContextFactory instance from the application. How about passing the instance as the value of a runtime configuration parameter? If that parameter is set, all other ssl.* parameters would be ignored. Obviously, this parameter could only be set programmatically. I would like to hear the proposed solution by the Kafka maintainers. I can help implementing a patch if there is an agreement on the desired solution.
