Re: [VOTE] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

[Maulin Vasavada]

+1 (non-binding).

Makes sense, Rajini. This would be a great addition.

On Tue, Nov 15, 2022 at 10:55 AM Rajini Sivaram 
wrote:

> Hi all,
>
> I would like to start vote on KIP-881 to support rack-aware partition
> assignment for Kafka consumers:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers
> .
>
> Thank you!
>
> Regards,
>
> Rajini
>

Re: [DISCUSS] KIP-651 - Support PEM format for SSL certificates and

[Maulin Vasavada]

just a clarification on the part "... we are not left with any choice but
to use the file based config .." - That is "if" we want material rotations
to be picked up automatically.

On Tue, Mar 22, 2022 at 1:57 PM Maulin Vasavada 
wrote:

> Hi Dejan and Rajini
>
> I tend to agree with both the arguments when I read them in isolation :)
> However, what I feel is - we have to also take an operator's view of this
> setup. If we have encrypted keys/certs already it may be easier to fit with
> file based material but if we don't have material encrypted already, it
> would mean we have to additionally get into the encrypted material's
> management to use the file based configuration OR adapt to in-line PEM
> config. If the inline config is not able to take advantage of
> auto-refreshes based on the file's modified timestamp, we are not left with
> any choice but to use the file based config and it adds
> operations/maintenance overhead.
>
> I can imagine a system where the unencrypted PEM keys are rotated fairly
> frequently, like once every few hours or once a week and keeping them
> protected with file permissions. In this case, it would be best if the
> Kafka system is operator friendly.
>
> So all in all I am thinking if it is a norm to have PEM certificates
> unencrypted outside of the Kafka world, we should try to stick to that
> without adding new process overhead requirements for the Kafka users.
>
> Thanks
> Maulin
>
>
> On Mon, Mar 21, 2022 at 10:03 AM Rajini Sivaram 
> wrote:
>
>> For the background on the current implementation: We use Java's keystore
>> loading for JKS/PKCS12 keystore files and these files require passwords.
>> We
>> retained the same requirement for PEM files as well for consistency, even
>> though it is not a language restriction anymore. When PEM keys are
>> provided
>> as actual config values rather than files, we do allow non-encrypted keys
>> with the expectation that config externalization will be used to protect
>> these, similar to SASL configs (e.g. private keys may be in vault or
>> encrypted using a secure config provider). We could take the view that
>> file
>> protection should be up to the user as well and permit non-encrypted keys,
>> making it simpler to adopt PEM. But since this is a deviation from the
>> other SSL file configs, it will be good to know what others think.
>>
>> Regards,
>>
>> Rajini
>>
>>
>> On Mon, Mar 21, 2022 at 2:16 PM Dejan Maric  wrote:
>>
>> > Regarding this KIP,  we've created an MR that removes the need for
>> > specifying *keyPassword *when PEM certificates and private key are
>> provided
>> > as files: https://github.com/apache/kafka/pull/11916
>> >
>> > We think that Kafka should not enforce the use of passwords on private
>> > keys. It would be sufficient to *recommend* encrypted private keys.
>> >
>> > The reason we suggested the PR is because this issue is currently
>> blocking
>> > us from using PEM-formatted certificates issued by cert-manager
>> > <https://github.com/cert-manager/cert-manager> with Kafka - since
>> > cert-manager does not support encrypted private keys.
>> >
>> > We asked one of the main cert-manager maintainers about this, and got
>> the
>> > following comments:
>> >
>> > >There is a benefit to encrypting this stuff if the cert is being
>> persisted
>> > to disk and if the decryption key isn't written to the same disk, which
>> is
>> > probably a pretty common setup in a lot of older non-cloud systems. The
>> > issue in a lot of more modern or cloud environments - especially in k8s
>> -
>> > is that certs are ideally never going to be written to disk and wherever
>> > they are stored, they'd likely be stored next to their decryption keys.
>> >
>> > >
>> > >So I guess everyone's right - in some environments encrypting the key
>> is
>> > probably worth it, and in some it's a false sense of security and wasted
>> > CPU cycles. That to me means that being able to choose to not encrypt
>> the
>> > key is desirable.
>> > >(This is assuming that the encryption is actually secure, which might
>> not
>> > be true for all methods of encrypting private keys)
>> >
>> >
>> > We'd like to hear other opinions on this.
>> >
>> > Dejan
>> >
>> >
>> > On 2020/08/03 11:28:42 Rajini Sivaram wrote:
>> > > Hi all,
>> > >
>> > > I have submitted KIP-651 to support PEM format for SSL key and trust
>> > stores:
>> > >
>> > >-
>> > >
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
>> >
>> > >
>> > > This enables better integration with other Kafka features like dynamic
>> > > config update and password protection.
>> > >
>> > > Feedback and suggestions are welcome.
>> > >
>> > > Thank you...
>> > >
>> > > Regards,
>> > >
>> > > Rajini
>> > >
>> >
>>
>

Re: [DISCUSS] KIP-651 - Support PEM format for SSL certificates and

[Maulin Vasavada]

Hi Dejan and Rajini

I tend to agree with both the arguments when I read them in isolation :)
However, what I feel is - we have to also take an operator's view of this
setup. If we have encrypted keys/certs already it may be easier to fit with
file based material but if we don't have material encrypted already, it
would mean we have to additionally get into the encrypted material's
management to use the file based configuration OR adapt to in-line PEM
config. If the inline config is not able to take advantage of
auto-refreshes based on the file's modified timestamp, we are not left with
any choice but to use the file based config and it adds
operations/maintenance overhead.

I can imagine a system where the unencrypted PEM keys are rotated fairly
frequently, like once every few hours or once a week and keeping them
protected with file permissions. In this case, it would be best if the
Kafka system is operator friendly.

So all in all I am thinking if it is a norm to have PEM certificates
unencrypted outside of the Kafka world, we should try to stick to that
without adding new process overhead requirements for the Kafka users.

Thanks
Maulin


On Mon, Mar 21, 2022 at 10:03 AM Rajini Sivaram 
wrote:

> For the background on the current implementation: We use Java's keystore
> loading for JKS/PKCS12 keystore files and these files require passwords. We
> retained the same requirement for PEM files as well for consistency, even
> though it is not a language restriction anymore. When PEM keys are provided
> as actual config values rather than files, we do allow non-encrypted keys
> with the expectation that config externalization will be used to protect
> these, similar to SASL configs (e.g. private keys may be in vault or
> encrypted using a secure config provider). We could take the view that file
> protection should be up to the user as well and permit non-encrypted keys,
> making it simpler to adopt PEM. But since this is a deviation from the
> other SSL file configs, it will be good to know what others think.
>
> Regards,
>
> Rajini
>
>
> On Mon, Mar 21, 2022 at 2:16 PM Dejan Maric  wrote:
>
> > Regarding this KIP,  we've created an MR that removes the need for
> > specifying *keyPassword *when PEM certificates and private key are
> provided
> > as files: https://github.com/apache/kafka/pull/11916
> >
> > We think that Kafka should not enforce the use of passwords on private
> > keys. It would be sufficient to *recommend* encrypted private keys.
> >
> > The reason we suggested the PR is because this issue is currently
> blocking
> > us from using PEM-formatted certificates issued by cert-manager
> >  with Kafka - since
> > cert-manager does not support encrypted private keys.
> >
> > We asked one of the main cert-manager maintainers about this, and got the
> > following comments:
> >
> > >There is a benefit to encrypting this stuff if the cert is being
> persisted
> > to disk and if the decryption key isn't written to the same disk, which
> is
> > probably a pretty common setup in a lot of older non-cloud systems. The
> > issue in a lot of more modern or cloud environments - especially in k8s -
> > is that certs are ideally never going to be written to disk and wherever
> > they are stored, they'd likely be stored next to their decryption keys.
> >
> > >
> > >So I guess everyone's right - in some environments encrypting the key is
> > probably worth it, and in some it's a false sense of security and wasted
> > CPU cycles. That to me means that being able to choose to not encrypt the
> > key is desirable.
> > >(This is assuming that the encryption is actually secure, which might
> not
> > be true for all methods of encrypting private keys)
> >
> >
> > We'd like to hear other opinions on this.
> >
> > Dejan
> >
> >
> > On 2020/08/03 11:28:42 Rajini Sivaram wrote:
> > > Hi all,
> > >
> > > I have submitted KIP-651 to support PEM format for SSL key and trust
> > stores:
> > >
> > >-
> > >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
> >
> > >
> > > This enables better integration with other Kafka features like dynamic
> > > config update and password protection.
> > >
> > > Feedback and suggestions are welcome.
> > >
> > > Thank you...
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> >
>

Re: [VOTE] KIP-651 - Support PEM format for SSL certificates and private key

[Maulin Vasavada]

+1 (non-binding)

On Thu, Aug 6, 2020 at 11:36 AM Ron Dagostino  wrote:

> +1 (non-binding)
>
> Ron
>
> > On Aug 6, 2020, at 1:42 PM, Rajini Sivaram 
> wrote:
> >
> > Hi all,
> >
> > I would like to start vote on KIP-651 to support SSL key stores and trust
> > stores in PEM format:
> >
> >   -
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
> >
> >
> > Thank you...
> >
> > Regards,
> >
> > Rajini
>

Re: [DISCUSS] KIP-651 - Support PEM format for SSL certificates and private key

[Maulin Vasavada]

Looks great Rajini. Indeed PEM is a standard needed to be supported in a
first class way on Kafka.

On Wed, Aug 5, 2020 at 9:55 PM Manikumar  wrote:

> Thanks for the KIP. LGTM.
>
> On Wed, Aug 5, 2020 at 3:43 AM Ron Dagostino  wrote:
>
> > Yes, Rajinit, it looks good -- an excellently-written KIP.  Thanks!
> >
> > Ron
> >
> > On Mon, Aug 3, 2020 at 9:17 AM Andrew Otto  wrote:
> >
> > > Yes please!  I had to build custom tooling to automate generation and
> > > distribution of java truststore and keystore files at Wikimedia for
> Kafka
> > > brokers and clients.  PEM is much more standard outside of the Java
> world
> > > and it would be simpler to not have to maintain custom tooling for
> > > keystores.
> > >
> > > - Andrew Otto
> > >
> > > On Mon, Aug 3, 2020 at 7:29 AM Rajini Sivaram  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have submitted KIP-651 to support PEM format for SSL key and trust
> > > > stores:
> > > >
> > > >-
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
> > > >
> > > > This enables better integration with other Kafka features like
> dynamic
> > > > config update and password protection.
> > > >
> > > > Feedback and suggestions are welcome.
> > > >
> > > > Thank you...
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
>

Re: Preliminary blog post about the Apache 2.6.0 release

[Maulin Vasavada]

Thanks Randall. That clarifies things. Makes sense!

On Wed, Aug 5, 2020 at 11:38 AM Randall Hauch  wrote:

> Hi, Maulin.
>
> We have traditionally not mentioned *every* KIP in the project's blog posts
> -- unless a release just happens to involve a small number of KIPs. This
> means that we do have to choose a subset of the KIPs -- AK 2.6.0 had 30
> KIPs that were implemented, and a blog summarizing each would be too long
> and much less widely received. I tried to choose KIPs that clearly impacted
> the broadest set of users, and finding a subset wasn't that easy.
>
> I chose not to include KIP-519 in the blog simply because it requires
> installing and integration into the broker components that are not included
> in the official distribution. I hope that helps explain my thought process.
>
> Best regards,
>
> Randall
>
> On Wed, Aug 5, 2020 at 12:52 PM Maulin Vasavada  >
> wrote:
>
> > Hi Randall
> >
> > One question: Do we mention all KIPs/NewFeatures in the blog that are
> > listed in the release notes document -
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html ?
> >
> > I see that [KAFKA-8890 <https://issues.apache.org/jira/browse/KAFKA-8890
> >]
> > - KIP-519: Make SSL context/engine configuration extensible is missing
> from
> > the google doc that you shared.
> >
> > Thanks
> > Maulin
> >
> >
> > On Wed, Aug 5, 2020 at 9:49 AM Randall Hauch  wrote:
> >
> > > Thanks, Jason. We haven't done that for a few releases, but I think
> it's
> > a
> > > great idea. I've updated the blog post draft and the Google doc to
> > mention
> > > the 127 contributors by name that will also be mentioned in the email
> > > release announcement.
> > >
> > > I also linked to the project's downloads page in the opening sentence,
> > and
> > > tweaked the wording slightly in the first paragraph.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Wed, Aug 5, 2020 at 11:21 AM Jason Gustafson 
> > > wrote:
> > >
> > > > Hey Randall,
> > > >
> > > > Thanks for putting this together. I think it would be great if the
> blog
> > > > included the names of the release contributors. It's an easy way to
> > give
> > > > some recognition. I know we have done that in the past.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch 
> > wrote:
> > > >
> > > > > I've prepared a preliminary blog post about the upcoming Apache
> Kafka
> > > > 2.6.0
> > > > > release.
> > > > > Please take a look and let me know via this thread if you want to
> > > > > add/modify details.
> > > > > Thanks to all who contributed to this blog post.
> > > > >
> > > > > Unfortunately, the preview is not currently publicly visible at
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
> > > > > (I've
> > > > > logged https://issues.apache.org/jira/browse/INFRA-20646), so in
> the
> > > > > interest of time I created a Google doc with read privilege for
> > > everyone
> > > > by
> > > > > pasting the preview content directly into:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
> > > > >
> > > > > I will continue to update this doc with any changes to the draft
> blog
> > > > post.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Randall
> > > > >
> > > >
> > >
> >
>

Re: Preliminary blog post about the Apache 2.6.0 release

[Maulin Vasavada]

Hi Randall

One question: Do we mention all KIPs/NewFeatures in the blog that are
listed in the release notes document -
https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html ?

I see that [KAFKA-8890 ]
- KIP-519: Make SSL context/engine configuration extensible is missing from
the google doc that you shared.

Thanks
Maulin


On Wed, Aug 5, 2020 at 9:49 AM Randall Hauch  wrote:

> Thanks, Jason. We haven't done that for a few releases, but I think it's a
> great idea. I've updated the blog post draft and the Google doc to mention
> the 127 contributors by name that will also be mentioned in the email
> release announcement.
>
> I also linked to the project's downloads page in the opening sentence, and
> tweaked the wording slightly in the first paragraph.
>
> Best regards,
>
> Randall
>
> On Wed, Aug 5, 2020 at 11:21 AM Jason Gustafson 
> wrote:
>
> > Hey Randall,
> >
> > Thanks for putting this together. I think it would be great if the blog
> > included the names of the release contributors. It's an easy way to give
> > some recognition. I know we have done that in the past.
> >
> > Thanks,
> > Jason
> >
> > On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch  wrote:
> >
> > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > 2.6.0
> > > release.
> > > Please take a look and let me know via this thread if you want to
> > > add/modify details.
> > > Thanks to all who contributed to this blog post.
> > >
> > > Unfortunately, the preview is not currently publicly visible at
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
> > > (I've
> > > logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
> > > interest of time I created a Google doc with read privilege for
> everyone
> > by
> > > pasting the preview content directly into:
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
> > >
> > > I will continue to update this doc with any changes to the draft blog
> > post.
> > >
> > >
> > > Thanks,
> > > Randall
> > >
> >
>

Re: [VOTE] KIP-597: MirrorMaker2 internal topics Formatters

[Maulin Vasavada]

+1 (non-binding)

On Wed, Jun 10, 2020 at 9:47 AM Mickael Maison 
wrote:

> Bumping this thread. Let me know if you have any questions or feedback.
>
> So far, we have 2 binding and 5 non-binding votes.
>
> Thanks
>
> On Tue, May 19, 2020 at 10:56 AM Manikumar 
> wrote:
> >
> > +1 (binding)
> >
> > Thanks for the KIP.
> >
> > On Tue, May 19, 2020 at 2:57 PM Mickael Maison  >
> > wrote:
> >
> > > Thanks Konstantine for the feedback (and vote)!
> > >
> > > 1) I've added example commands using the new formatters
> > >
> > > 2) I updated the Compatiblity section to be more explicit about the
> > > need for recompilation
> > >
> > > 3) Good point, updated
> > >
> > > On Tue, May 19, 2020 at 3:18 AM Konstantine Karantasis
> > >  wrote:
> > > >
> > > > Thanks Michael.
> > > > I think it's useful to enable specialized message formatters by
> adding
> > > this
> > > > interface to the public API.
> > > >
> > > > You have my vote: +1 (binding)
> > > >
> > > > Just a few optional comments below:
> > > >
> > > > 1. Would you mind adding the equivalent command line example in the
> > > places
> > > > where you have an example output?
> > > >
> > > > Something equivalent to
> > > > ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
> --topic
> > > > __consumer_offsets --from-beginning --formatter
> > > >
> > >
> "kafka.coordinator.group.GroupMetadataManager\$GroupMetadataMessageFormatter"
> > > >
> > > > but with the equivalent formatter classes and expected topic names.
> > > >
> > > > 2. I have to note that breaking old formatters by requiring
> recompilation
> > > > could be avoided if we didn't change kafka.common.MessageFormatter to
> > > > extend the new org.apache.kafka.common.MessageFormatter. We could
> > > maintain
> > > > both, while the old one would remain deprecated and we could attempt
> to
> > > > instantiate one or the other type when reading the config and use
> either
> > > of
> > > > the two different types in the few places in ConsoleConsumer that a
> > > > formatter is used. But I admit that for this use case, it's not worth
> > > > maintaining both types. The interface wasn't public before anyways.
> > > >
> > > > Given that, my small request would be to rephrase in the
> compatibility
> > > > section to say something like:
> > > > 'Existing MessageFormatters implementations will require no changes
> > > beyond
> > > > recompilation.' or similar. Because, to be precise, existing
> formatters
> > > > _won't_ work if they are given as a parameter to a 2.6 console
> consumer,
> > > > without recompilation as you mention.
> > > >
> > > > 3. Finally, a minor comment on skipping the use of the `public`
> specifier
> > > > in the interface because it's redundant.
> > > >
> > > > Best regards,
> > > > Konstantine
> > > >
> > > > On Mon, May 18, 2020 at 3:26 PM Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Mon, May 18, 2020 at 8:49 AM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Bumping this thread as KIP freeze is approaching.
> > > > > >
> > > > > > It's a pretty small change and I have a PR ready:
> > > > > > https://github.com/apache/kafka/pull/8604
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Mon, May 4, 2020 at 5:26 PM Ryanne Dolan <
> ryannedo...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > +1, non-binding
> > > > > > >
> > > > > > > On Mon, May 4, 2020, 9:24 AM Christopher Egerton <
> > > chr...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > On Mon, May 4, 2020 at

Re: [DISCUSS] Apache Kafka 2.6.0 release

[Maulin Vasavada]

Thanks Randall Hauch!

On Sat, May 30, 2020 at 12:52 PM Randall Hauch  wrote:

> Hi, Kowshik,
>
> Thanks for the update on KIP-584. This is listed on the "Postponed" section
> of the AK 2.6.0 release plan (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> ).
>
> Best regards,
>
> Randall
>
> On Fri, May 29, 2020 at 4:51 PM Kowshik Prakasam 
> wrote:
>
> > Hi Randall,
> >
> > We have to remove KIP-584 from the release plan, as this item will not be
> > completed for 2.6 release (although KIP is accepted). We plan to include
> it
> > in a next release.
> >
> >
> > Cheers,
> > Kowshik
> >
> >
> > On Fri, May 29, 2020 at 11:43 AM Maulin Vasavada <
> > maulin.vasav...@gmail.com>
> > wrote:
> >
> > > Hi Randall Hauch
> > >
> > > Can we add KIP-519 to 2.6? It was merged to Trunk already in April -
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952
> > > .
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Fri, May 29, 2020 at 11:01 AM Randall Hauch 
> wrote:
> > >
> > > > Here's an update on the AK 2.6.0 release.
> > > >
> > > > Code freeze was Wednesday, and the release plan [1] has been updated
> to
> > > > reflect all of the KIPs that made the release. We've also cut the
> `2.6`
> > > > branch that we'll use for the release; see separate email announcing
> > the
> > > > new branch.
> > > >
> > > > The next important date for the 2.6.0 release is CODE FREEZE on JUNE
> > 10,
> > > > and until that date all bug fixes are still welcome on the release
> > > branch.
> > > > But after that, only blocker bugs can be merged to the release
> branch.
> > > >
> > > > If you have any questions or concerns, please contact me or (better
> > yet)
> > > > reply to this thread.
> > > >
> > > > Thanks, and best regards!
> > > >
> > > > Randall
> > > >
> > > > [1] AK 2.6.0 Release Plan:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > > >
> > > >
> > > > On Wed, May 27, 2020 at 5:53 PM Matthias J. Sax 
> > > wrote:
> > > >
> > > > > Thanks Randall!
> > > > >
> > > > > I added missing KIP-594.
> > > > >
> > > > >
> > > > > For the postponed KIP section: I removed KIP-441 and KIP-444 as
> both
> > > are
> > > > > completed.
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 5/27/20 2:31 PM, Randall Hauch wrote:
> > > > > > Hey everyone, just a quick update on the 2.6.0 release.
> > > > > >
> > > > > > Based on the release plan (
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > > > > ),
> > > > > > today (May 27) is feature freeze. Any major feature work that is
> > not
> > > > > > already complete will need to push out to the next release
> (either
> > > 2.7
> > > > or
> > > > > > 3.0). There are a few PRs for KIPs that are nearing completion,
> and
> > > > we're
> > > > > > having some Jenkins build issues. I will send another email later
> > > today
> > > > > or
> > > > > > early tomorrow with an update, and I plan to cut the release
> branch
> > > > > shortly
> > > > > > thereafter.
> > > > > >
> > > > > > I have also updated the list of planned KIPs on the release plan
> > > page (
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > > > > ),
> > > > > > and I've moved to the "Postponed" table any KIP that looks like
> it
> > is
> > > > not
> > > > > > going to be complete today. If any KIP is in the wrong table,
> > please
> > > > let
> > > > > me
> > > > > > know.
> > > > &

Re: [DISCUSS] Apache Kafka 2.6.0 release

[Maulin Vasavada]

Hi Randall Hauch

Can we add KIP-519 to 2.6? It was merged to Trunk already in April -
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952.

Thanks
Maulin

On Fri, May 29, 2020 at 11:01 AM Randall Hauch  wrote:

> Here's an update on the AK 2.6.0 release.
>
> Code freeze was Wednesday, and the release plan [1] has been updated to
> reflect all of the KIPs that made the release. We've also cut the `2.6`
> branch that we'll use for the release; see separate email announcing the
> new branch.
>
> The next important date for the 2.6.0 release is CODE FREEZE on JUNE 10,
> and until that date all bug fixes are still welcome on the release branch.
> But after that, only blocker bugs can be merged to the release branch.
>
> If you have any questions or concerns, please contact me or (better yet)
> reply to this thread.
>
> Thanks, and best regards!
>
> Randall
>
> [1] AK 2.6.0 Release Plan:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
>
>
> On Wed, May 27, 2020 at 5:53 PM Matthias J. Sax  wrote:
>
> > Thanks Randall!
> >
> > I added missing KIP-594.
> >
> >
> > For the postponed KIP section: I removed KIP-441 and KIP-444 as both are
> > completed.
> >
> >
> > -Matthias
> >
> > On 5/27/20 2:31 PM, Randall Hauch wrote:
> > > Hey everyone, just a quick update on the 2.6.0 release.
> > >
> > > Based on the release plan (
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > ),
> > > today (May 27) is feature freeze. Any major feature work that is not
> > > already complete will need to push out to the next release (either 2.7
> or
> > > 3.0). There are a few PRs for KIPs that are nearing completion, and
> we're
> > > having some Jenkins build issues. I will send another email later today
> > or
> > > early tomorrow with an update, and I plan to cut the release branch
> > shortly
> > > thereafter.
> > >
> > > I have also updated the list of planned KIPs on the release plan page (
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > ),
> > > and I've moved to the "Postponed" table any KIP that looks like it is
> not
> > > going to be complete today. If any KIP is in the wrong table, please
> let
> > me
> > > know.
> > >
> > > If you have any questions or concerns, please feel free to reply to
> this
> > > thread.
> > >
> > > Thanks, and best regards!
> > >
> > > Randall
> > >
> > > On Wed, May 20, 2020 at 2:16 PM Sophie Blee-Goldman <
> sop...@confluent.io
> > >
> > > wrote:
> > >
> > >> Hey Randall,
> > >>
> > >> Can you also add KIP-613 which was accepted yesterday?
> > >>
> > >> Thanks!
> > >> Sophie
> > >>
> > >> On Wed, May 20, 2020 at 6:47 AM Randall Hauch 
> wrote:
> > >>
> > >>> Hi, Tom. I saw last night that the KIP had enough votes before
> today’s
> > >>> deadline and I will add it to the roadmap today. Thanks for driving
> > this!
> > >>>
> > >>> On Wed, May 20, 2020 at 6:18 AM Tom Bentley 
> > wrote:
> > >>>
> >  Hi Randall,
> > 
> >  Can we add KIP-585? (I'm not quite sure of the protocol here, but
> > >> thought
> >  it better to ask than to just add it myself).
> > 
> >  Thanks,
> > 
> >  Tom
> > 
> >  On Tue, May 5, 2020 at 6:54 PM Randall Hauch 
> > >> wrote:
> > 
> > > Greetings!
> > >
> > > I'd like to volunteer to be release manager for the next time-based
> >  feature
> > > release which will be 2.6.0. I've published a release plan at
> > >
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > > ,
> > > and have included all of the KIPs that are currently approved or
> > >>> actively
> > > in discussion (though I'm happy to adjust as necessary).
> > >
> > > To stay on our time-based cadence, the KIP freeze is on May 20
> with a
> > > target release date of June 24.
> > >
> > > Let me know if there are any objections.
> > >
> > > Thanks,
> > > Randall Hauch
> > >
> > 
> > >>>
> > >>
> > >
> >
> >
>

Re: [VOTE] KIP-545 support automated consumer offset sync across clusters in MM 2.0

[Maulin Vasavada]

Thank you for the KIP. I sincerely hope we get enough votes on this KIP. I
was thinking of similar changes while working on DR capabilities and
offsets are Achilles Heels and this KIP addresses it.

On Mon, May 18, 2020 at 6:10 PM Maulin Vasavada 
wrote:

> +1 (non-binding)
>
> On Mon, May 18, 2020 at 9:41 AM Ryanne Dolan 
> wrote:
>
>> Bump. Looks like we've got 6 non-binding votes and 1 binding.
>>
>> On Thu, Feb 20, 2020 at 11:25 AM Ning Zhang 
>> wrote:
>>
>> > Hello committers,
>> >
>> > I am the author of the KIP-545 and if we still miss votes from the
>> > committers, please review the KIP and vote for it, so that the
>> > corresponding PR will be reviewed soon.
>> >
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
>> >
>> > Thank you
>> >
>> > On 2020/02/06 17:05:41, Edoardo Comar  wrote:
>> > > +1 (non-binding)
>> > > thanks for the KIP !
>> > >
>> > > On Tue, 14 Jan 2020 at 13:57, Navinder Brar > > .invalid>
>> > > wrote:
>> > >
>> > > > +1 (non-binding)
>> > > > Navinder
>> > > > On Tuesday, 14 January, 2020, 07:24:02 pm IST, Ryanne Dolan <
>> > > > ryannedo...@gmail.com> wrote:
>> > > >
>> > > >  Bump. We've got 4 non-binding and one binding vote.
>> > > >
>> > > > Ryanne
>> > > >
>> > > > On Fri, Dec 13, 2019, 1:44 AM Tom Bentley 
>> wrote:
>> > > >
>> > > > > +1 (non-binding)
>> > > > >
>> > > > > On Thu, Dec 12, 2019 at 6:33 PM Andrew Schofield <
>> > > > > andrew_schofi...@live.com>
>> > > > > wrote:
>> > > > >
>> > > > > > +1 (non-binding)
>> > > > > >
>> > > > > > On 12/12/2019, 14:20, "Mickael Maison" <
>> mickael.mai...@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > >+1 (binding)
>> > > > > >Thanks for the KIP!
>> > > > > >
>> > > > > >On Thu, Dec 5, 2019 at 12:56 AM Ryanne Dolan <
>> > ryannedo...@gmail.com
>> > > > >
>> > > > > > wrote:
>> > > > > >>
>> > > > > >> Bump. We've got 2 non-binding votes so far.
>> > > > > >>
>> > > > > >> On Wed, Nov 13, 2019 at 3:32 PM Ning Zhang <
>> > > > ning2008w...@gmail.com
>> > > > > >
>> > > > > > wrote:
>> > > > > >>
>> > > > > >> > My current plan is to implement this in
>> > "MirrorCheckpointTask"
>> > > > > >> >
>> > > > > >> > On 2019/11/02 03:30:11, Xu Jianhai > >
>> > > > wrote:
>> > > > > >> > > I think this kip will implement a task in sinkTask ?
>> > right?
>> > > > > >> > >
>> > > > > >> > > On Sat, Nov 2, 2019 at 1:06 AM Ryanne Dolan <
>> > > > > > ryannedo...@gmail.com>
>> > > > > >> > wrote:
>> > > > > >> > >
>> > > > > >> > > > Hey y'all, Ning Zhang and I would like to start the
>> > vote for
>> > > > > > the
>> > > > > >> > following
>> > > > > >> > > > small KIP:
>> > > > > >> > > >
>> > > > > >> > > >
>> > > > > >> > > >
>> > > > > >> >
>> > > > > >
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
>> > > > > >> > > >
>> > > > > >> > > > This is an elegant way to automatically write
>> consumer
>> > group
>> > > > > > offsets to
>> > > > > >> > > > downstream clusters without breaking existing use
>> cases.
>> > > > > > Currently, we
>> > > > > >> > rely
>> > > > > >> > > > on external tooling based on RemoteClusterUtils and
>> > > > > >> > kafka-consumer-groups
>> > > > > >> > > > command to write offsets. This KIP bakes this
>> > functionality
>> > > > > > into MM2
>> > > > > >> > > > itself, reducing the effort required to
>> > failover/failback
>> > > > > > workloads
>> > > > > >> > between
>> > > > > >> > > > clusters.
>> > > > > >> > > >
>> > > > > >> > > > Thanks for the votes!
>> > > > > >> > > >
>> > > > > >> > > > Ryanne
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>>
>

Re: [VOTE] KIP-545 support automated consumer offset sync across clusters in MM 2.0

[Maulin Vasavada]

+1 (non-binding)

On Mon, May 18, 2020 at 9:41 AM Ryanne Dolan  wrote:

> Bump. Looks like we've got 6 non-binding votes and 1 binding.
>
> On Thu, Feb 20, 2020 at 11:25 AM Ning Zhang 
> wrote:
>
> > Hello committers,
> >
> > I am the author of the KIP-545 and if we still miss votes from the
> > committers, please review the KIP and vote for it, so that the
> > corresponding PR will be reviewed soon.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> >
> > Thank you
> >
> > On 2020/02/06 17:05:41, Edoardo Comar  wrote:
> > > +1 (non-binding)
> > > thanks for the KIP !
> > >
> > > On Tue, 14 Jan 2020 at 13:57, Navinder Brar  > .invalid>
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > > Navinder
> > > > On Tuesday, 14 January, 2020, 07:24:02 pm IST, Ryanne Dolan <
> > > > ryannedo...@gmail.com> wrote:
> > > >
> > > >  Bump. We've got 4 non-binding and one binding vote.
> > > >
> > > > Ryanne
> > > >
> > > > On Fri, Dec 13, 2019, 1:44 AM Tom Bentley 
> wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Thu, Dec 12, 2019 at 6:33 PM Andrew Schofield <
> > > > > andrew_schofi...@live.com>
> > > > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > On 12/12/2019, 14:20, "Mickael Maison" <
> mickael.mai...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >+1 (binding)
> > > > > >Thanks for the KIP!
> > > > > >
> > > > > >On Thu, Dec 5, 2019 at 12:56 AM Ryanne Dolan <
> > ryannedo...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >>
> > > > > >> Bump. We've got 2 non-binding votes so far.
> > > > > >>
> > > > > >> On Wed, Nov 13, 2019 at 3:32 PM Ning Zhang <
> > > > ning2008w...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >>
> > > > > >> > My current plan is to implement this in
> > "MirrorCheckpointTask"
> > > > > >> >
> > > > > >> > On 2019/11/02 03:30:11, Xu Jianhai 
> > > > wrote:
> > > > > >> > > I think this kip will implement a task in sinkTask ?
> > right?
> > > > > >> > >
> > > > > >> > > On Sat, Nov 2, 2019 at 1:06 AM Ryanne Dolan <
> > > > > > ryannedo...@gmail.com>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > > > Hey y'all, Ning Zhang and I would like to start the
> > vote for
> > > > > > the
> > > > > >> > following
> > > > > >> > > > small KIP:
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> > > > > >> > > >
> > > > > >> > > > This is an elegant way to automatically write consumer
> > group
> > > > > > offsets to
> > > > > >> > > > downstream clusters without breaking existing use
> cases.
> > > > > > Currently, we
> > > > > >> > rely
> > > > > >> > > > on external tooling based on RemoteClusterUtils and
> > > > > >> > kafka-consumer-groups
> > > > > >> > > > command to write offsets. This KIP bakes this
> > functionality
> > > > > > into MM2
> > > > > >> > > > itself, reducing the effort required to
> > failover/failback
> > > > > > workloads
> > > > > >> > between
> > > > > >> > > > clusters.
> > > > > >> > > >
> > > > > >> > > > Thanks for the votes!
> > > > > >> > > >
> > > > > >> > > > Ryanne
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: [VOTE] KIP-597: MirrorMaker2 internal topics Formatters

[Maulin Vasavada]

+1 (non-binding)

On Mon, May 18, 2020 at 8:49 AM Mickael Maison 
wrote:

> Bumping this thread as KIP freeze is approaching.
>
> It's a pretty small change and I have a PR ready:
> https://github.com/apache/kafka/pull/8604
>
> Thanks
>
> On Mon, May 4, 2020 at 5:26 PM Ryanne Dolan  wrote:
> >
> > +1, non-binding
> >
> > On Mon, May 4, 2020, 9:24 AM Christopher Egerton 
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Mon, May 4, 2020 at 5:02 AM Edoardo Comar 
> wrote:
> > >
> > > > +1 (non-binding)
> > > > Thanks Mickael
> > > >
> > > > --
> > > >
> > > > Edoardo Comar
> > > >
> > > > Event Streams for IBM Cloud
> > > > IBM UK Ltd, Hursley Park, SO21 2JN
> > > >
> > > >
> > > >
> > > >
> > > > From:   Mickael Maison 
> > > > To: dev 
> > > > Date:   04/05/2020 11:45
> > > > Subject:[EXTERNAL] [VOTE] KIP-597: MirrorMaker2 internal
> topics
> > > > Formatters
> > > >
> > > >
> > > >
> > > > Hi all,
> > > >
> > > > I'd like to start a vote on KIP-597:
> > > >
> > > >
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D597-253A-2BMirrorMaker2-2Binternal-2Btopics-2BFormatters=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=r-_T9EFUWNEUGi1GuX7klXNZIH2sJmxGTtySV3lAjoQ=iyBSxabuEi1h7ksmzoXgJT8jJoMR0xKYsJy_MpvtCRQ=
> > > >
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > >
> > > > Unless stated otherwise above:
> > > > IBM United Kingdom Limited - Registered in England and Wales with
> number
> > > > 741598.
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
> PO6
> > > 3AU
> > > >
> > > >
> > >
>

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

[Maulin Vasavada]

Hi Sonke

Thanks for bringing this for discussion. There are lot of considerations
even if we assume we have end-to-end encryption done. Example depending
upon company's setup there could be restrictions on how/which encryption
keys are shared. Environment could have multiple security and network
boundaries beyond which keys are not allowed to be shared. That will mean
that consumers may not be able to decrypt the messages at all if the data
is moved from one zone to another. If we have mirroring done, are
mirror-makers supposed to decrypt and encrypt again OR they would be pretty
much bytes-in bytes-out paradigm that it is today? Also having a polyglot
Kafka client base will force you to support encryption/decryption libraries
that work for all the languages and that may not work depending upon the
scope of the team owning Kafka Infrastructure.

Combining disk encryption with TLS+ACLs could be enough instead of having
end-to-end message level encryption. What is your opinion on that?

We have experimented with end-to-end encryption with custom
serializers/deserializers and I felt that was good enough because
other challenges I mentioned before may not be ease to address with a
generic solution.

Thanks
Maulin



Thanks
Maulin




On Sat, May 9, 2020 at 2:05 PM Ryanne Dolan  wrote:

> Adam, I agree, seems reasonable to limit the broker's responsibility to
> encrypting only data at rest. I guess whole segment files could be
> encrypted with the same key, and rotating keys would just involve
> re-encrypting entire segments. Maybe a key rotation would involve closing
> all affected segments and kicking off a background task to re-encrypt them.
> Certainly that would not impede ingestion of new records, and seems
> consumers could use the old segments until they are replaced with the newly
> encrypted ones.
>
> Seems that could still get us per-topic keys (vs encrypting the entire
> volume), which would be my main requirement.
>
> Not really "end-to-end", but combined with TLS or something, seems
> reasonable.
>
> Ryanne
>
> On Sat, May 9, 2020, 11:00 AM Adam Bellemare 
> wrote:
>
> > Hi All
> >
> > I typed up a number of replies which I have below, but I have one major
> > overriding question: Is there a reason we aren't implementing
> > encryption-at-rest almost exactly the same way that most relational
> > databases do? ie:
> > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption
> >
> > I ask this because it seems like we're going to end up with something
> > similar to what they did in terms of requirements, plus...
> >
> > "For the *past 16 months*, there has been discussion about whether and
> how
> > to implement Transparent Data Encryption (tde) in Postgres. Many other
> > relational databases support tde, and *some security standards require*
> it.
> > However, it is also debatable how much security value tde provides.
> > The tde *400-email
> > thread* became difficult for people to follow..."
> > What still isn't clear to me is the scope that we're trying to cover
> here.
> > Encryption at rest suggests that we need to have the data encrypted on
> the
> > brokers, and *only* on the brokers, since they're the durable units of
> > storage. Any encryption over the wire should be covered by TLS.  I think
> > that our goals for this should be (from
> >
> https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Threat_models
> > )
> >
> > > TDE protects data from theft when file system access controls are
> > > compromised:
> > >
> > >- Malicious user steals storage devices and reads database files
> > >directly.
> > >- Malicious backup operator takes backup.
> > >- Protecting data at rest (persistent data)
> > >
> > > This does not protect from users who can read system memory, e.g.,
> shared
> > > buffers, which root users can do.
> > >
> >
> > I am not a security expert nor am I an expert on relational databases.
> > However, I can't identify any reason why the approach outlined by
> > PostgresDB, which is very similar to MySQL/InnoDB and IBM (from my
> > understanding) wouldn't work for data-at-rest encryption. In addition,
> we'd
> > get the added benefit of being consistent with other solutions, which is
> an
> > easier sell when discussing security with management (Kafka? Oh yeah,
> their
> > encryption solution is just like the one we already have in place for our
> > Postgres solutions), and may let us avoid reinventing a good part of the
> > wheel.
> >
> >
> > --
> >
> > @Ryanne
> > One more complicating factor, regarding joins - the foreign key joiner
> > requires access to the value to extract the foreign key - if it's
> > encrypted, the FKJ would need to decrypt it to apply the value extractor.
> >
> > @Soenk re (1)
> > > When people hear that this is not part of Apache Kafka itself, but that
> > > would need to develop something themselves that more often than not is
> > the
> > > end of that discussion. Using something that is not "stock" is 

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

HI all

The summary of the vote: *Total 6 +ve* with *+4 Binding*, +2 non-Binding.
I'll move the KIP to Accepted state.

Thanks
Maulin

On Tue, Mar 31, 2020 at 10:32 PM Maulin Vasavada 
wrote:

> Thanks Rajini and Jun. I'll update the shouldBeRebuilt() docs on what
> happens to existing SSL Connections.
>
> Thanks everybody who participated in the discussion thread and voting
> thread and spending the valuable time reviewing the KIP/PR. Could not have
> done it without your support. Really appreciate it.
>
> Now we can close on the voting phase since we had it open for > 72 hrs but
> since this is my first KIP, can you please guide what happens now? Can I
> move the KIP to accepted state myself?
>
> Thanks
> Maulin
>
> On Tue, Mar 31, 2020 at 10:27 AM Jun Rao  wrote:
>
>> Hi, Rajini, Maulin,
>>
>> 1. Ok. Then we can keep the package name as it is.
>>
>> 2. Thanks for updating the javadoc for shouldBeRebuilt(). Could you also
>> clarify after the SslEngine is rebuilt, what happens to existing SSL
>> connections?
>>
>> Thanks,
>>
>> Jun
>>
>> On Tue, Mar 31, 2020 at 2:07 AM Rajini Sivaram 
>> wrote:
>>
>> > Hi Jun, Maulin,
>> >
>> >  org.apache.kafka.common.security.ssl contains internal classes like
>> > SslFactory.  org.apache.kafka.common.security.auth is a public package
>> > which contains all our current authentication-related classes. If we
>> want
>> > to move the new interface into an SSL-specific package, we should
>> perhaps
>> > create a new public package rather than use an existing internal one?
>> >
>> > On Tue, Mar 31, 2020 at 7:56 AM Manikumar 
>> > wrote:
>> >
>> > > +1 (binding).
>> > > Thanks for the KIP.
>> > >
>> > > Thanks,
>> > > Manikumar
>> > >
>> > > On Tue, Mar 31, 2020 at 11:24 AM Maulin Vasavada <
>> > > maulin.vasav...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > So far we got 3 Binding votes. I am planning to keep the voting
>> phase
>> > > open
>> > > > until Tuesday 10 PM Pacific Time which will be more than 72 hours
>> from
>> > > the
>> > > > first binding vote on Thursday 12:36 PM Pacific Time.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Mon, Mar 30, 2020 at 10:32 PM Maulin Vasavada <
>> > > > maulin.vasav...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I updated the Javadoc in the KIP details and the actual
>> > > SslEngineFactory
>> > > > > interface for shouldBeRebuilt(). For the first comment, probably
>> I'll
>> > > try
>> > > > > to address it tomorrow.
>> > > > >
>> > > > > Thanks
>> > > > > Maulin
>> > > > >
>> > > > > On Mon, Mar 30, 2020 at 7:44 PM Maulin Vasavada <
>> > > > maulin.vasav...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > >> Thanks Jun Rao for your vote and comments.
>> > > > >>
>> > > > >> For 1) Earlier it was the security.ssl package but after a
>> review I
>> > > > >> changed it to .auth since there are some public interfaces in
>> that
>> > > > package.
>> > > > >> I am open to move it under .ssl package.
>> > > > >>
>> > > > >> For 2) Sure. Will document in Javadocs for the method.
>> > > > >>
>> > > > >> Thanks
>> > > > >> Maulin
>> > > > >>
>> > > > >> On Mon, Mar 30, 2020 at 5:46 PM Jun Rao 
>> wrote:
>> > > > >>
>> > > > >>> Hi, Maulin,
>> > > > >>>
>> > > > >>> Thanks for the KIP. +1 from me. Just a couple of minor comments
>> > > below.
>> > > > >>>
>> > > > >>> 1. Should the package name of the new
>> > > > >>> interface SslEngineFactory be
>> org.apache.kafka.common.security.ssl
>> > > > >>> instead
>> > > > >>> of org.apache.kafka.common.security.auth?
>&

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Thanks Rajini and Jun. I'll update the shouldBeRebuilt() docs on what
happens to existing SSL Connections.

Thanks everybody who participated in the discussion thread and voting
thread and spending the valuable time reviewing the KIP/PR. Could not have
done it without your support. Really appreciate it.

Now we can close on the voting phase since we had it open for > 72 hrs but
since this is my first KIP, can you please guide what happens now? Can I
move the KIP to accepted state myself?

Thanks
Maulin

On Tue, Mar 31, 2020 at 10:27 AM Jun Rao  wrote:

> Hi, Rajini, Maulin,
>
> 1. Ok. Then we can keep the package name as it is.
>
> 2. Thanks for updating the javadoc for shouldBeRebuilt(). Could you also
> clarify after the SslEngine is rebuilt, what happens to existing SSL
> connections?
>
> Thanks,
>
> Jun
>
> On Tue, Mar 31, 2020 at 2:07 AM Rajini Sivaram 
> wrote:
>
> > Hi Jun, Maulin,
> >
> >  org.apache.kafka.common.security.ssl contains internal classes like
> > SslFactory.  org.apache.kafka.common.security.auth is a public package
> > which contains all our current authentication-related classes. If we want
> > to move the new interface into an SSL-specific package, we should perhaps
> > create a new public package rather than use an existing internal one?
> >
> > On Tue, Mar 31, 2020 at 7:56 AM Manikumar 
> > wrote:
> >
> > > +1 (binding).
> > > Thanks for the KIP.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > > On Tue, Mar 31, 2020 at 11:24 AM Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > So far we got 3 Binding votes. I am planning to keep the voting phase
> > > open
> > > > until Tuesday 10 PM Pacific Time which will be more than 72 hours
> from
> > > the
> > > > first binding vote on Thursday 12:36 PM Pacific Time.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Mon, Mar 30, 2020 at 10:32 PM Maulin Vasavada <
> > > > maulin.vasav...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I updated the Javadoc in the KIP details and the actual
> > > SslEngineFactory
> > > > > interface for shouldBeRebuilt(). For the first comment, probably
> I'll
> > > try
> > > > > to address it tomorrow.
> > > > >
> > > > > Thanks
> > > > > Maulin
> > > > >
> > > > > On Mon, Mar 30, 2020 at 7:44 PM Maulin Vasavada <
> > > > maulin.vasav...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Thanks Jun Rao for your vote and comments.
> > > > >>
> > > > >> For 1) Earlier it was the security.ssl package but after a review
> I
> > > > >> changed it to .auth since there are some public interfaces in that
> > > > package.
> > > > >> I am open to move it under .ssl package.
> > > > >>
> > > > >> For 2) Sure. Will document in Javadocs for the method.
> > > > >>
> > > > >> Thanks
> > > > >> Maulin
> > > > >>
> > > > >> On Mon, Mar 30, 2020 at 5:46 PM Jun Rao  wrote:
> > > > >>
> > > > >>> Hi, Maulin,
> > > > >>>
> > > > >>> Thanks for the KIP. +1 from me. Just a couple of minor comments
> > > below.
> > > > >>>
> > > > >>> 1. Should the package name of the new
> > > > >>> interface SslEngineFactory be
> org.apache.kafka.common.security.ssl
> > > > >>> instead
> > > > >>> of org.apache.kafka.common.security.auth?
> > > > >>> 2. Could you document when shouldBeRebuilt() will be called?
> > > > >>>
> > > > >>> Jun
> > > > >>>
> > > > >>> On Mon, Mar 30, 2020 at 5:07 PM Maulin Vasavada <
> > > > >>> maulin.vasav...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > ^^^  bump ^^^ The vote is open for 2-3 days and gotten 1
> Binding
> > > vote
> > > > >>> so
> > > > >>> > far, can you please vote so that we can try to move forward
> with
> > > > >>

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all,

So far we got 3 Binding votes. I am planning to keep the voting phase open
until Tuesday 10 PM Pacific Time which will be more than 72 hours from the
first binding vote on Thursday 12:36 PM Pacific Time.

Thanks
Maulin

On Mon, Mar 30, 2020 at 10:32 PM Maulin Vasavada 
wrote:

> Hi all,
>
> I updated the Javadoc in the KIP details and the actual SslEngineFactory
> interface for shouldBeRebuilt(). For the first comment, probably I'll try
> to address it tomorrow.
>
> Thanks
> Maulin
>
> On Mon, Mar 30, 2020 at 7:44 PM Maulin Vasavada 
> wrote:
>
>> Thanks Jun Rao for your vote and comments.
>>
>> For 1) Earlier it was the security.ssl package but after a review I
>> changed it to .auth since there are some public interfaces in that package.
>> I am open to move it under .ssl package.
>>
>> For 2) Sure. Will document in Javadocs for the method.
>>
>> Thanks
>> Maulin
>>
>> On Mon, Mar 30, 2020 at 5:46 PM Jun Rao  wrote:
>>
>>> Hi, Maulin,
>>>
>>> Thanks for the KIP. +1 from me. Just a couple of minor comments below.
>>>
>>> 1. Should the package name of the new
>>> interface SslEngineFactory be org.apache.kafka.common.security.ssl
>>> instead
>>> of org.apache.kafka.common.security.auth?
>>> 2. Could you document when shouldBeRebuilt() will be called?
>>>
>>> Jun
>>>
>>> On Mon, Mar 30, 2020 at 5:07 PM Maulin Vasavada <
>>> maulin.vasav...@gmail.com>
>>> wrote:
>>>
>>> > ^^^  bump ^^^ The vote is open for 2-3 days and gotten 1 Binding vote
>>> so
>>> > far, can you please vote so that we can try to move forward with
>>> changes?
>>> >
>>> > On Thu, Mar 26, 2020 at 4:11 PM Zhou, Thomas >> >
>>> > wrote:
>>> >
>>> > > +1 (non-binding)
>>> > >
>>> > > Regards,
>>> > > Thomas
>>> > >
>>> > > On 3/26/20, 12:36 PM, "Rajini Sivaram" 
>>> wrote:
>>> > >
>>> > > +1 (binding)
>>> > > Thanks for the KIP, Maulin!
>>> > >
>>> > > Regards,
>>> > >
>>> > > Rajini
>>> > >
>>> > > On Thu, Mar 26, 2020 at 4:14 PM Maulin Vasavada <
>>> > > maulin.vasav...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > FYI - we have updated the KIP documentation also with
>>> appropriate
>>> > > code
>>> > > > samples for interfaces and few important changes.
>>> > > >
>>> > > > Thanks
>>> > > > Maulin
>>> > > >
>>> > > > On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada <
>>> > > > maulin.vasav...@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > bump
>>> > > > >
>>> > > > > On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada <
>>> > > > > maulin.vasav...@gmail.com> wrote:
>>> > > > >
>>> > > > >> Hi all
>>> > > > >>
>>> > > > >> After much await on the approach conclusion we have a PR
>>> > > > >>
>>> > >
>>> >
>>> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F8338data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=1ydk0OMaucb8QhTyyQ8Ua3ereGzcS4usRlavU1RixkE%3Dreserved=0
>>> > > .
>>> > > > >>
>>> > > > >> Can you please provide your vote so that we can more this
>>> > forward?
>>> > > > >>
>>> > > > >> Thanks
>>> > > > >> Maulin
>>> > > > >>
>>> > > > >> On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada <
>>> > > > >> maulin.vasav...@gmail.com> wrote:
>>> > > > >>
>>> > > > >>> Hi all
>>> > > > >>>
>>> > > > >>> After a good discussion on the KIP at
>>> > > > >>>
>>> > >
>>> >
>>> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fdev%40kafka.apache.org%2Fmsg101011.htmldata=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=qsvbqkoxL6NSPDV6rm9B9xqZG5xvYaZkj0cYrTM6bPw%3Dreserved=0
>>> > > I
>>> > > > >>> think we are ready to start voting.
>>> > > > >>>
>>> > > > >>> KIP:
>>> > > > >>>
>>> > > >
>>> > >
>>> >
>>> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D128650952data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=rcqWc2inIbrWlMj2jssHPKcMlHuDuLvicmYHHDYWrF8%3Dreserved=0
>>> > > > >>>
>>> > > > >>> The KIP proposes - Making SSLEngine creation pluggable to
>>> > support
>>> > > > >>> customization of various security related aspects.
>>> > > > >>>
>>> > > > >>> Thanks
>>> > > > >>> Maulin
>>> > > > >>>
>>> > > > >>
>>> > > >
>>> > >
>>> > >
>>> > >
>>> >
>>>
>>

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all,

I updated the Javadoc in the KIP details and the actual SslEngineFactory
interface for shouldBeRebuilt(). For the first comment, probably I'll try
to address it tomorrow.

Thanks
Maulin

On Mon, Mar 30, 2020 at 7:44 PM Maulin Vasavada 
wrote:

> Thanks Jun Rao for your vote and comments.
>
> For 1) Earlier it was the security.ssl package but after a review I
> changed it to .auth since there are some public interfaces in that package.
> I am open to move it under .ssl package.
>
> For 2) Sure. Will document in Javadocs for the method.
>
> Thanks
> Maulin
>
> On Mon, Mar 30, 2020 at 5:46 PM Jun Rao  wrote:
>
>> Hi, Maulin,
>>
>> Thanks for the KIP. +1 from me. Just a couple of minor comments below.
>>
>> 1. Should the package name of the new
>> interface SslEngineFactory be org.apache.kafka.common.security.ssl instead
>> of org.apache.kafka.common.security.auth?
>> 2. Could you document when shouldBeRebuilt() will be called?
>>
>> Jun
>>
>> On Mon, Mar 30, 2020 at 5:07 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> wrote:
>>
>> > ^^^  bump ^^^ The vote is open for 2-3 days and gotten 1 Binding vote so
>> > far, can you please vote so that we can try to move forward with
>> changes?
>> >
>> > On Thu, Mar 26, 2020 at 4:11 PM Zhou, Thomas > >
>> > wrote:
>> >
>> > > +1 (non-binding)
>> > >
>> > > Regards,
>> > > Thomas
>> > >
>> > > On 3/26/20, 12:36 PM, "Rajini Sivaram" 
>> wrote:
>> > >
>> > > +1 (binding)
>> > > Thanks for the KIP, Maulin!
>> > >
>> > > Regards,
>> > >
>> > > Rajini
>> > >
>> > > On Thu, Mar 26, 2020 at 4:14 PM Maulin Vasavada <
>> > > maulin.vasav...@gmail.com>
>> > > wrote:
>> > >
>> > > > FYI - we have updated the KIP documentation also with
>> appropriate
>> > > code
>> > > > samples for interfaces and few important changes.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada <
>> > > > maulin.vasav...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > bump
>> > > > >
>> > > > > On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada <
>> > > > > maulin.vasav...@gmail.com> wrote:
>> > > > >
>> > > > >> Hi all
>> > > > >>
>> > > > >> After much await on the approach conclusion we have a PR
>> > > > >>
>> > >
>> >
>> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F8338data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=1ydk0OMaucb8QhTyyQ8Ua3ereGzcS4usRlavU1RixkE%3Dreserved=0
>> > > .
>> > > > >>
>> > > > >> Can you please provide your vote so that we can more this
>> > forward?
>> > > > >>
>> > > > >> Thanks
>> > > > >> Maulin
>> > > > >>
>> > > > >> On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada <
>> > > > >> maulin.vasav...@gmail.com> wrote:
>> > > > >>
>> > > > >>> Hi all
>> > > > >>>
>> > > > >>> After a good discussion on the KIP at
>> > > > >>>
>> > >
>> >
>> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fdev%40kafka.apache.org%2Fmsg101011.htmldata=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=qsvbqkoxL6NSPDV6rm9B9xqZG5xvYaZkj0cYrTM6bPw%3Dreserved=0
>> > > I
>> > > > >>> think we are ready to start voting.
>> > > > >>>
>> > > > >>> KIP:
>> > > > >>>
>> > > >
>> > >
>> >
>> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D128650952data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=rcqWc2inIbrWlMj2jssHPKcMlHuDuLvicmYHHDYWrF8%3Dreserved=0
>> > > > >>>
>> > > > >>> The KIP proposes - Making SSLEngine creation pluggable to
>> > support
>> > > > >>> customization of various security related aspects.
>> > > > >>>
>> > > > >>> Thanks
>> > > > >>> Maulin
>> > > > >>>
>> > > > >>
>> > > >
>> > >
>> > >
>> > >
>> >
>>
>

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Thanks Jun Rao for your vote and comments.

For 1) Earlier it was the security.ssl package but after a review I changed
it to .auth since there are some public interfaces in that package. I am
open to move it under .ssl package.

For 2) Sure. Will document in Javadocs for the method.

Thanks
Maulin

On Mon, Mar 30, 2020 at 5:46 PM Jun Rao  wrote:

> Hi, Maulin,
>
> Thanks for the KIP. +1 from me. Just a couple of minor comments below.
>
> 1. Should the package name of the new
> interface SslEngineFactory be org.apache.kafka.common.security.ssl instead
> of org.apache.kafka.common.security.auth?
> 2. Could you document when shouldBeRebuilt() will be called?
>
> Jun
>
> On Mon, Mar 30, 2020 at 5:07 PM Maulin Vasavada  >
> wrote:
>
> > ^^^  bump ^^^ The vote is open for 2-3 days and gotten 1 Binding vote so
> > far, can you please vote so that we can try to move forward with changes?
> >
> > On Thu, Mar 26, 2020 at 4:11 PM Zhou, Thomas 
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > Regards,
> > > Thomas
> > >
> > > On 3/26/20, 12:36 PM, "Rajini Sivaram" 
> wrote:
> > >
> > > +1 (binding)
> > > Thanks for the KIP, Maulin!
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Thu, Mar 26, 2020 at 4:14 PM Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > wrote:
> > >
> > > > FYI - we have updated the KIP documentation also with appropriate
> > > code
> > > > samples for interfaces and few important changes.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada <
> > > > maulin.vasav...@gmail.com>
> > > > wrote:
> > > >
> > > > > bump
> > > > >
> > > > > On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada <
> > > > > maulin.vasav...@gmail.com> wrote:
> > > > >
> > > > >> Hi all
> > > > >>
> > > > >> After much await on the approach conclusion we have a PR
> > > > >>
> > >
> >
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F8338data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=1ydk0OMaucb8QhTyyQ8Ua3ereGzcS4usRlavU1RixkE%3Dreserved=0
> > > .
> > > > >>
> > > > >> Can you please provide your vote so that we can more this
> > forward?
> > > > >>
> > > > >> Thanks
> > > > >> Maulin
> > > > >>
> > > > >> On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada <
> > > > >> maulin.vasav...@gmail.com> wrote:
> > > > >>
> > > > >>> Hi all
> > > > >>>
> > > > >>> After a good discussion on the KIP at
> > > > >>>
> > >
> >
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fdev%40kafka.apache.org%2Fmsg101011.htmldata=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=qsvbqkoxL6NSPDV6rm9B9xqZG5xvYaZkj0cYrTM6bPw%3Dreserved=0
> > > I
> > > > >>> think we are ready to start voting.
> > > > >>>
> > > > >>> KIP:
> > > > >>>
> > > >
> > >
> >
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D128650952data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=rcqWc2inIbrWlMj2jssHPKcMlHuDuLvicmYHHDYWrF8%3Dreserved=0
> > > > >>>
> > > > >>> The KIP proposes - Making SSLEngine creation pluggable to
> > support
> > > > >>> customization of various security related aspects.
> > > > >>>
> > > > >>> Thanks
> > > > >>> Maulin
> > > > >>>
> > > > >>
> > > >
> > >
> > >
> > >
> >
>

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

^^^  bump ^^^ The vote is open for 2-3 days and gotten 1 Binding vote so
far, can you please vote so that we can try to move forward with changes?

On Thu, Mar 26, 2020 at 4:11 PM Zhou, Thomas 
wrote:

> +1 (non-binding)
>
> Regards,
> Thomas
>
> On 3/26/20, 12:36 PM, "Rajini Sivaram"  wrote:
>
> +1 (binding)
> Thanks for the KIP, Maulin!
>
> Regards,
>
>     Rajini
>
> On Thu, Mar 26, 2020 at 4:14 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> wrote:
>
> > FYI - we have updated the KIP documentation also with appropriate
> code
> > samples for interfaces and few important changes.
> >
> > Thanks
> > Maulin
> >
> > On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada <
> > maulin.vasav...@gmail.com>
> > wrote:
> >
> > > bump
> > >
> > > On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada <
> > > maulin.vasav...@gmail.com> wrote:
> > >
> > >> Hi all
> > >>
> > >> After much await on the approach conclusion we have a PR
> > >>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F8338data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=1ydk0OMaucb8QhTyyQ8Ua3ereGzcS4usRlavU1RixkE%3Dreserved=0
> .
> > >>
> > >> Can you please provide your vote so that we can more this forward?
> > >>
> > >> Thanks
> > >> Maulin
> > >>
> > >> On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada <
> > >> maulin.vasav...@gmail.com> wrote:
> > >>
> > >>> Hi all
> > >>>
> > >>> After a good discussion on the KIP at
> > >>>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fdev%40kafka.apache.org%2Fmsg101011.htmldata=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=qsvbqkoxL6NSPDV6rm9B9xqZG5xvYaZkj0cYrTM6bPw%3Dreserved=0
> I
> > >>> think we are ready to start voting.
> > >>>
> > >>> KIP:
> > >>>
> >
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D128650952data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=rcqWc2inIbrWlMj2jssHPKcMlHuDuLvicmYHHDYWrF8%3Dreserved=0
> > >>>
> > >>> The KIP proposes - Making SSLEngine creation pluggable to support
> > >>> customization of various security related aspects.
> > >>>
> > >>> Thanks
> > >>> Maulin
> > >>>
> > >>
> >
>
>
>

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

FYI - we have updated the KIP documentation also with appropriate code
samples for interfaces and few important changes.

Thanks
Maulin

On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada 
wrote:

> bump
>
> On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada <
> maulin.vasav...@gmail.com> wrote:
>
>> Hi all
>>
>> After much await on the approach conclusion we have a PR
>> https://github.com/apache/kafka/pull/8338.
>>
>> Can you please provide your vote so that we can more this forward?
>>
>> Thanks
>> Maulin
>>
>> On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com> wrote:
>>
>>> Hi all
>>>
>>> After a good discussion on the KIP at
>>> https://www.mail-archive.com/dev@kafka.apache.org/msg101011.html I
>>> think we are ready to start voting.
>>>
>>> KIP:
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952
>>>
>>> The KIP proposes - Making SSLEngine creation pluggable to support
>>> customization of various security related aspects.
>>>
>>> Thanks
>>> Maulin
>>>
>>

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

bump

On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada 
wrote:

> Hi all
>
> After much await on the approach conclusion we have a PR
> https://github.com/apache/kafka/pull/8338.
>
> Can you please provide your vote so that we can more this forward?
>
> Thanks
> Maulin
>
> On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada <
> maulin.vasav...@gmail.com> wrote:
>
>> Hi all
>>
>> After a good discussion on the KIP at
>> https://www.mail-archive.com/dev@kafka.apache.org/msg101011.html I think
>> we are ready to start voting.
>>
>> KIP:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952
>>
>> The KIP proposes - Making SSLEngine creation pluggable to support
>> customization of various security related aspects.
>>
>> Thanks
>> Maulin
>>
>

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all

After much await on the approach conclusion we have a PR
https://github.com/apache/kafka/pull/8338.

Can you please provide your vote so that we can more this forward?

Thanks
Maulin

On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada 
wrote:

> Hi all
>
> After a good discussion on the KIP at
> https://www.mail-archive.com/dev@kafka.apache.org/msg101011.html I think
> we are ready to start voting.
>
> KIP:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952
>
> The KIP proposes - Making SSLEngine creation pluggable to support
> customization of various security related aspects.
>
> Thanks
> Maulin
>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all,

I resolved the challenge with having Class Type for the
ssl.engine.factory.class configuration. Raised pull request to the trunk -
https://github.com/apache/kafka/pull/8338. Updated on the JIRA -
https://issues.apache.org/jira/browse/KAFKA-8890

Please review & Vote!

Thanks
Maulin

On Mon, Mar 23, 2020 at 6:57 PM Maulin Vasavada 
wrote:

> Hi Rajini
>
> When I configure the Default value for ssl.engine.factory.class as Type
> Class, it is resulting into lot of test cases failures since in many places
> - tests and actual classes/scala code it is converting the configuration
> maps value to value.toString(). I verified that was the issues by fixing
> some of it but still evaluating how many places we need to fix if we really
> want to support Class Type key for the configuration. Password and List
> Types are also of non-String types but it seems their defaults are Null and
> they are optional vs in my case the ssl.engine.factory.class is *not*
> optional - it needs a value by default.
>
> Will keep you posted. I am thinking of reverting the config type to String
> and then load it as String and do Class loading in SslFactory.
>
> Thanks
> Maulin
>
>
> On Mon, Mar 23, 2020 at 1:38 AM Maulin Vasavada 
> wrote:
>
>> still working on the pull request. hopefully will be done soon.
>>
>> On Wed, Mar 11, 2020 at 11:48 AM Maulin Vasavada <
>> maulin.vasav...@gmail.com> wrote:
>>
>>> Thanks Rajini. Sounds good. I'll make changes and update this thread.
>>>
>>> On Wed, Mar 11, 2020 at 6:41 AM Rajini Sivaram 
>>> wrote:
>>>
>>>> Hi Maulin,
>>>>
>>>> I have reviewed the PR and left some comments, can you turn it into a PR
>>>> for https://github.com/apache/kafka? It looks good overall.
>>>>
>>>> We pass all configs to other plugins, so we can do the same here. That
>>>> would be safer than assuming that all custom SSL-related configs start
>>>> with
>>>> `ssl.`. You can use principal builder as an example for what we do
>>>> today.
>>>>
>>>> Regards,
>>>>
>>>> Rajini
>>>>
>>>> On Thu, Mar 5, 2020 at 12:03 AM Maulin Vasavada <
>>>> maulin.vasav...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi Rajini
>>>> >
>>>> > I made changes suggested by you on
>>>> > https://github.com/maulin-vasavada/kafka/pull/4. Please check.
>>>> >
>>>> > In particular I had challenge in 'SslFactory#configure()' method the
>>>> first
>>>> > time to know which configs I have to add without having actual
>>>> > SslEngineFactory impl. I think it is best to just copy all configs
>>>> with
>>>> > "ssl." prefix. Can you please review
>>>> >
>>>> >
>>>> https://github.com/maulin-vasavada/kafka/pull/4/files#diff-1e3432211fdbb7b2e2b44b5d8838a40bR90
>>>> >  particularly?
>>>> >
>>>> > Clement, I missed to see your point about Mode in previous post but
>>>> even
>>>> > after I realized what you are suggesting, my response would be the
>>>> same as
>>>> > before :)
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> >
>>>> > On Wed, Feb 5, 2020 at 8:40 PM Maulin Vasavada <
>>>> maulin.vasav...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Hi Rajini
>>>> > >
>>>> > > Will accommodate your comments.
>>>> > >
>>>> > > Celement, while SSLContext factories are common, in this particular
>>>> case,
>>>> > > we need SSLEngine object because Kafka is using SSLEngine (as
>>>> mentioned
>>>> > > much before in this email thread, the SSLContext acts as factory for
>>>> > > getting SSLEngine, SSLSocketFactory or SSLServerSocketFactory and
>>>> Kafka
>>>> > > chooses SSLEngine to be used for SSL Connections). The 'Mode'
>>>> challenge
>>>> > > doesn't go away easily since Kafka is using the "same" classes for
>>>> Client
>>>> > > side and Server side. Other stacks where you don't see this
>>>> challenge is
>>>> > > because either libraries are client centric or server centric and
>>>> not
>>>> > both
>>>> > > at the same t

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi Rajini

When I configure the Default value for ssl.engine.factory.class as Type
Class, it is resulting into lot of test cases failures since in many places
- tests and actual classes/scala code it is converting the configuration
maps value to value.toString(). I verified that was the issues by fixing
some of it but still evaluating how many places we need to fix if we really
want to support Class Type key for the configuration. Password and List
Types are also of non-String types but it seems their defaults are Null and
they are optional vs in my case the ssl.engine.factory.class is *not*
optional - it needs a value by default.

Will keep you posted. I am thinking of reverting the config type to String
and then load it as String and do Class loading in SslFactory.

Thanks
Maulin


On Mon, Mar 23, 2020 at 1:38 AM Maulin Vasavada 
wrote:

> still working on the pull request. hopefully will be done soon.
>
> On Wed, Mar 11, 2020 at 11:48 AM Maulin Vasavada <
> maulin.vasav...@gmail.com> wrote:
>
>> Thanks Rajini. Sounds good. I'll make changes and update this thread.
>>
>> On Wed, Mar 11, 2020 at 6:41 AM Rajini Sivaram 
>> wrote:
>>
>>> Hi Maulin,
>>>
>>> I have reviewed the PR and left some comments, can you turn it into a PR
>>> for https://github.com/apache/kafka? It looks good overall.
>>>
>>> We pass all configs to other plugins, so we can do the same here. That
>>> would be safer than assuming that all custom SSL-related configs start
>>> with
>>> `ssl.`. You can use principal builder as an example for what we do today.
>>>
>>> Regards,
>>>
>>> Rajini
>>>
>>> On Thu, Mar 5, 2020 at 12:03 AM Maulin Vasavada <
>>> maulin.vasav...@gmail.com>
>>> wrote:
>>>
>>> > Hi Rajini
>>> >
>>> > I made changes suggested by you on
>>> > https://github.com/maulin-vasavada/kafka/pull/4. Please check.
>>> >
>>> > In particular I had challenge in 'SslFactory#configure()' method the
>>> first
>>> > time to know which configs I have to add without having actual
>>> > SslEngineFactory impl. I think it is best to just copy all configs with
>>> > "ssl." prefix. Can you please review
>>> >
>>> >
>>> https://github.com/maulin-vasavada/kafka/pull/4/files#diff-1e3432211fdbb7b2e2b44b5d8838a40bR90
>>> >  particularly?
>>> >
>>> > Clement, I missed to see your point about Mode in previous post but
>>> even
>>> > after I realized what you are suggesting, my response would be the
>>> same as
>>> > before :)
>>> >
>>> > Thanks
>>> > Maulin
>>> >
>>> >
>>> > On Wed, Feb 5, 2020 at 8:40 PM Maulin Vasavada <
>>> maulin.vasav...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Rajini
>>> > >
>>> > > Will accommodate your comments.
>>> > >
>>> > > Celement, while SSLContext factories are common, in this particular
>>> case,
>>> > > we need SSLEngine object because Kafka is using SSLEngine (as
>>> mentioned
>>> > > much before in this email thread, the SSLContext acts as factory for
>>> > > getting SSLEngine, SSLSocketFactory or SSLServerSocketFactory and
>>> Kafka
>>> > > chooses SSLEngine to be used for SSL Connections). The 'Mode'
>>> challenge
>>> > > doesn't go away easily since Kafka is using the "same" classes for
>>> Client
>>> > > side and Server side. Other stacks where you don't see this
>>> challenge is
>>> > > because either libraries are client centric or server centric and not
>>> > both
>>> > > at the same time. I would suggest you do a deeper dive into the
>>> sample
>>> > Pull
>>> > > request, build the code to get better idea about it. I don't find it
>>> > > strange to have 'Mode' argument in this context of Kafka. Kafka is
>>> not
>>> > > doing anything out of norm here since ultimately it is using JSSE
>>> spec
>>> > for
>>> > > SSL Connection.
>>> > >
>>> > > I will try to respond to code comments in couple of weeks since I am
>>> out
>>> > > for few weeks. Will keep you guys posted.
>>> > >
>>> > > Thanks
>>> > > Maulin
>>> > >
>>> > >
>>> > 

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

still working on the pull request. hopefully will be done soon.

On Wed, Mar 11, 2020 at 11:48 AM Maulin Vasavada 
wrote:

> Thanks Rajini. Sounds good. I'll make changes and update this thread.
>
> On Wed, Mar 11, 2020 at 6:41 AM Rajini Sivaram 
> wrote:
>
>> Hi Maulin,
>>
>> I have reviewed the PR and left some comments, can you turn it into a PR
>> for https://github.com/apache/kafka? It looks good overall.
>>
>> We pass all configs to other plugins, so we can do the same here. That
>> would be safer than assuming that all custom SSL-related configs start
>> with
>> `ssl.`. You can use principal builder as an example for what we do today.
>>
>> Regards,
>>
>> Rajini
>>
>> On Thu, Mar 5, 2020 at 12:03 AM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> wrote:
>>
>> > Hi Rajini
>> >
>> > I made changes suggested by you on
>> > https://github.com/maulin-vasavada/kafka/pull/4. Please check.
>> >
>> > In particular I had challenge in 'SslFactory#configure()' method the
>> first
>> > time to know which configs I have to add without having actual
>> > SslEngineFactory impl. I think it is best to just copy all configs with
>> > "ssl." prefix. Can you please review
>> >
>> >
>> https://github.com/maulin-vasavada/kafka/pull/4/files#diff-1e3432211fdbb7b2e2b44b5d8838a40bR90
>> >  particularly?
>> >
>> > Clement, I missed to see your point about Mode in previous post but even
>> > after I realized what you are suggesting, my response would be the same
>> as
>> > before :)
>> >
>> > Thanks
>> > Maulin
>> >
>> >
>> > On Wed, Feb 5, 2020 at 8:40 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> > wrote:
>> >
>> > > Hi Rajini
>> > >
>> > > Will accommodate your comments.
>> > >
>> > > Celement, while SSLContext factories are common, in this particular
>> case,
>> > > we need SSLEngine object because Kafka is using SSLEngine (as
>> mentioned
>> > > much before in this email thread, the SSLContext acts as factory for
>> > > getting SSLEngine, SSLSocketFactory or SSLServerSocketFactory and
>> Kafka
>> > > chooses SSLEngine to be used for SSL Connections). The 'Mode'
>> challenge
>> > > doesn't go away easily since Kafka is using the "same" classes for
>> Client
>> > > side and Server side. Other stacks where you don't see this challenge
>> is
>> > > because either libraries are client centric or server centric and not
>> > both
>> > > at the same time. I would suggest you do a deeper dive into the sample
>> > Pull
>> > > request, build the code to get better idea about it. I don't find it
>> > > strange to have 'Mode' argument in this context of Kafka. Kafka is not
>> > > doing anything out of norm here since ultimately it is using JSSE spec
>> > for
>> > > SSL Connection.
>> > >
>> > > I will try to respond to code comments in couple of weeks since I am
>> out
>> > > for few weeks. Will keep you guys posted.
>> > >
>> > > Thanks
>> > > Maulin
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Wed, Feb 5, 2020 at 9:50 PM Pellerin, Clement <
>> > clement_pelle...@ibi.com>
>> > > wrote:
>> > >
>> > >> Many of these points came up before.
>> > >>
>> > >> I had great hope when Maulin suggested the custom factory could
>> > >> return an SSLContext instead of SSLEngine.  SSLContext factories are
>> > >> common,
>> > >> whereas I have never seen an SSLEngine factory being used before.
>> > >> He must have hit the same problem I had with the Mode.
>> > >>
>> > >> If the Mode can be removed, can we find a way to return an SSLContext
>> > now?
>> > >> What is so special about Kafka that it needs to hardcode the Mode
>> when
>> > >> everyone
>> > >> else works with the SSLContext and ignores the other mode they don't
>> > use.
>> > >>
>> > >> -Original Message-
>> > >> From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
>> > >> Sent: Wednesday, February 5, 2020 10:03 AM

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Thanks Rajini. Sounds good. I'll make changes and update this thread.

On Wed, Mar 11, 2020 at 6:41 AM Rajini Sivaram 
wrote:

> Hi Maulin,
>
> I have reviewed the PR and left some comments, can you turn it into a PR
> for https://github.com/apache/kafka? It looks good overall.
>
> We pass all configs to other plugins, so we can do the same here. That
> would be safer than assuming that all custom SSL-related configs start with
> `ssl.`. You can use principal builder as an example for what we do today.
>
> Regards,
>
> Rajini
>
> On Thu, Mar 5, 2020 at 12:03 AM Maulin Vasavada  >
> wrote:
>
> > Hi Rajini
> >
> > I made changes suggested by you on
> > https://github.com/maulin-vasavada/kafka/pull/4. Please check.
> >
> > In particular I had challenge in 'SslFactory#configure()' method the
> first
> > time to know which configs I have to add without having actual
> > SslEngineFactory impl. I think it is best to just copy all configs with
> > "ssl." prefix. Can you please review
> >
> >
> https://github.com/maulin-vasavada/kafka/pull/4/files#diff-1e3432211fdbb7b2e2b44b5d8838a40bR90
> >  particularly?
> >
> > Clement, I missed to see your point about Mode in previous post but even
> > after I realized what you are suggesting, my response would be the same
> as
> > before :)
> >
> > Thanks
> > Maulin
> >
> >
> > On Wed, Feb 5, 2020 at 8:40 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > wrote:
> >
> > > Hi Rajini
> > >
> > > Will accommodate your comments.
> > >
> > > Celement, while SSLContext factories are common, in this particular
> case,
> > > we need SSLEngine object because Kafka is using SSLEngine (as mentioned
> > > much before in this email thread, the SSLContext acts as factory for
> > > getting SSLEngine, SSLSocketFactory or SSLServerSocketFactory and Kafka
> > > chooses SSLEngine to be used for SSL Connections). The 'Mode' challenge
> > > doesn't go away easily since Kafka is using the "same" classes for
> Client
> > > side and Server side. Other stacks where you don't see this challenge
> is
> > > because either libraries are client centric or server centric and not
> > both
> > > at the same time. I would suggest you do a deeper dive into the sample
> > Pull
> > > request, build the code to get better idea about it. I don't find it
> > > strange to have 'Mode' argument in this context of Kafka. Kafka is not
> > > doing anything out of norm here since ultimately it is using JSSE spec
> > for
> > > SSL Connection.
> > >
> > > I will try to respond to code comments in couple of weeks since I am
> out
> > > for few weeks. Will keep you guys posted.
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Feb 5, 2020 at 9:50 PM Pellerin, Clement <
> > clement_pelle...@ibi.com>
> > > wrote:
> > >
> > >> Many of these points came up before.
> > >>
> > >> I had great hope when Maulin suggested the custom factory could
> > >> return an SSLContext instead of SSLEngine.  SSLContext factories are
> > >> common,
> > >> whereas I have never seen an SSLEngine factory being used before.
> > >> He must have hit the same problem I had with the Mode.
> > >>
> > >> If the Mode can be removed, can we find a way to return an SSLContext
> > now?
> > >> What is so special about Kafka that it needs to hardcode the Mode when
> > >> everyone
> > >> else works with the SSLContext and ignores the other mode they don't
> > use.
> > >>
> > >> -Original Message-
> > >> From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
> > >> Sent: Wednesday, February 5, 2020 10:03 AM
> > >> To: dev
> > >> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> > >> extensible
> > >>
> > >> One more point:
> > >> 5) We should also add a method to SslEngineFactory that returns
> > >> `Set
> > >> reconfigurableConfigs()`
> > >>
> > >> On Wed, Feb 5, 2020 at 1:50 PM Rajini Sivaram <
> rajinisiva...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Maulin,
> > >> >
> > >> > Thanks for the updates. 

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi Rajini

I made changes suggested by you on
https://github.com/maulin-vasavada/kafka/pull/4. Please check.

In particular I had challenge in 'SslFactory#configure()' method the first
time to know which configs I have to add without having actual
SslEngineFactory impl. I think it is best to just copy all configs with
"ssl." prefix. Can you please review
https://github.com/maulin-vasavada/kafka/pull/4/files#diff-1e3432211fdbb7b2e2b44b5d8838a40bR90
 particularly?

Clement, I missed to see your point about Mode in previous post but even
after I realized what you are suggesting, my response would be the same as
before :)

Thanks
Maulin


On Wed, Feb 5, 2020 at 8:40 PM Maulin Vasavada 
wrote:

> Hi Rajini
>
> Will accommodate your comments.
>
> Celement, while SSLContext factories are common, in this particular case,
> we need SSLEngine object because Kafka is using SSLEngine (as mentioned
> much before in this email thread, the SSLContext acts as factory for
> getting SSLEngine, SSLSocketFactory or SSLServerSocketFactory and Kafka
> chooses SSLEngine to be used for SSL Connections). The 'Mode' challenge
> doesn't go away easily since Kafka is using the "same" classes for Client
> side and Server side. Other stacks where you don't see this challenge is
> because either libraries are client centric or server centric and not both
> at the same time. I would suggest you do a deeper dive into the sample Pull
> request, build the code to get better idea about it. I don't find it
> strange to have 'Mode' argument in this context of Kafka. Kafka is not
> doing anything out of norm here since ultimately it is using JSSE spec for
> SSL Connection.
>
> I will try to respond to code comments in couple of weeks since I am out
> for few weeks. Will keep you guys posted.
>
> Thanks
> Maulin
>
>
>
>
>
>
>
>
> On Wed, Feb 5, 2020 at 9:50 PM Pellerin, Clement 
> wrote:
>
>> Many of these points came up before.
>>
>> I had great hope when Maulin suggested the custom factory could
>> return an SSLContext instead of SSLEngine.  SSLContext factories are
>> common,
>> whereas I have never seen an SSLEngine factory being used before.
>> He must have hit the same problem I had with the Mode.
>>
>> If the Mode can be removed, can we find a way to return an SSLContext now?
>> What is so special about Kafka that it needs to hardcode the Mode when
>> everyone
>> else works with the SSLContext and ignores the other mode they don't use.
>>
>> -Original Message-
>> From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
>> Sent: Wednesday, February 5, 2020 10:03 AM
>> To: dev
>> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
>> extensible
>>
>> One more point:
>> 5) We should also add a method to SslEngineFactory that returns
>> `Set
>> reconfigurableConfigs()`
>>
>> On Wed, Feb 5, 2020 at 1:50 PM Rajini Sivaram 
>> wrote:
>>
>> > Hi Maulin,
>> >
>> > Thanks for the updates. A few comments below:
>> >
>> > 1) SslEngineFactory is currently in the internal package
>> > org.apache.kafka.common.security.ssl. We should move it to the public
>> > package org.apache.kafka.common.security.auth.
>> > 2) SslEngineFactory should extend Configurable and Closeable. We should
>> > expect the implementation class to have a default constructor and
>> invoke configure()
>> > to be consistent with otSslher pluggable classes.
>> > 3) SslEngineFactory.createSslEngine uses the internal enum `Mode`. It
>> > would be better to add two methods instead:
>> >
>> >
>> >- SSLEngine createClientSslEngine(String peerHost, int peerPort,
>> String endpointIdentification);
>> >- SSLEngine createServerSslEngine(String peerHost, int peerPort);
>> >
>> > 4) Don't think we need a method on SslEngineFactory to return configs.
>> It seems like an odd thing to do in a pubic Configurable API and is
>> inconsistent with other APIs. We can track configs in the internal
>> SslFactory class instead.
>>
>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi Rajini

Will accommodate your comments.

Celement, while SSLContext factories are common, in this particular case,
we need SSLEngine object because Kafka is using SSLEngine (as mentioned
much before in this email thread, the SSLContext acts as factory for
getting SSLEngine, SSLSocketFactory or SSLServerSocketFactory and Kafka
chooses SSLEngine to be used for SSL Connections). The 'Mode' challenge
doesn't go away easily since Kafka is using the "same" classes for Client
side and Server side. Other stacks where you don't see this challenge is
because either libraries are client centric or server centric and not both
at the same time. I would suggest you do a deeper dive into the sample Pull
request, build the code to get better idea about it. I don't find it
strange to have 'Mode' argument in this context of Kafka. Kafka is not
doing anything out of norm here since ultimately it is using JSSE spec for
SSL Connection.

I will try to respond to code comments in couple of weeks since I am out
for few weeks. Will keep you guys posted.

Thanks
Maulin








On Wed, Feb 5, 2020 at 9:50 PM Pellerin, Clement 
wrote:

> Many of these points came up before.
>
> I had great hope when Maulin suggested the custom factory could
> return an SSLContext instead of SSLEngine.  SSLContext factories are
> common,
> whereas I have never seen an SSLEngine factory being used before.
> He must have hit the same problem I had with the Mode.
>
> If the Mode can be removed, can we find a way to return an SSLContext now?
> What is so special about Kafka that it needs to hardcode the Mode when
> everyone
> else works with the SSLContext and ignores the other mode they don't use.
>
> -Original Message-
> From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
> Sent: Wednesday, February 5, 2020 10:03 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> One more point:
> 5) We should also add a method to SslEngineFactory that returns
> `Set
> reconfigurableConfigs()`
>
> On Wed, Feb 5, 2020 at 1:50 PM Rajini Sivaram 
> wrote:
>
> > Hi Maulin,
> >
> > Thanks for the updates. A few comments below:
> >
> > 1) SslEngineFactory is currently in the internal package
> > org.apache.kafka.common.security.ssl. We should move it to the public
> > package org.apache.kafka.common.security.auth.
> > 2) SslEngineFactory should extend Configurable and Closeable. We should
> > expect the implementation class to have a default constructor and invoke
> configure()
> > to be consistent with otSslher pluggable classes.
> > 3) SslEngineFactory.createSslEngine uses the internal enum `Mode`. It
> > would be better to add two methods instead:
> >
> >
> >- SSLEngine createClientSslEngine(String peerHost, int peerPort,
> String endpointIdentification);
> >- SSLEngine createServerSslEngine(String peerHost, int peerPort);
> >
> > 4) Don't think we need a method on SslEngineFactory to return configs.
> It seems like an odd thing to do in a pubic Configurable API and is
> inconsistent with other APIs. We can track configs in the internal
> SslFactory class instead.
>

[VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all

After a good discussion on the KIP at
https://www.mail-archive.com/dev@kafka.apache.org/msg101011.html I think we
are ready to start voting.

KIP:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952

The KIP proposes - Making SSLEngine creation pluggable to support
customization of various security related aspects.

Thanks
Maulin

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all

I will start the voting thread on this now.

Thanks
Maulin

On Thu, Jan 23, 2020 at 12:51 AM Maulin Vasavada 
wrote:

> Hi all,
>
> I have updated the KIP document with the current state of conclusions.
> Please review it and see if we are ready to move to Voting!
>
> Thanks
> Maulin
>
> On Wed, Jan 22, 2020 at 12:42 AM Maulin Vasavada <
> maulin.vasav...@gmail.com> wrote:
>
>> Hi all,
>>
>> Finally I squeezed time and I've a suggested code changes shown at
>> https://github.com/maulin-vasavada/kafka/pull/4/files for discussing
>> this further. I'll update the KIP document soon. Meanwhile, can you please
>> take a look and continue the discussion?
>>
>> One challenge is at:
>> https://github.com/maulin-vasavada/kafka/pull/4/files#diff-1e3432211fdbb7b2e2b44b5d8838a40bR89
>>
>> Thanks
>> Maulin
>>
>>
>> On Tue, Oct 22, 2019 at 11:13 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com> wrote:
>>
>>> bump! Clement/Rajini? Any responses based on the latest posts?
>>>
>>> On Wed, Oct 16, 2019 at 10:58 PM Maulin Vasavada <
>>> maulin.vasav...@gmail.com> wrote:
>>>
>>>> bump!
>>>>
>>>> On Sun, Oct 13, 2019 at 11:16 PM Maulin Vasavada <
>>>> maulin.vasav...@gmail.com> wrote:
>>>>
>>>>> Hi Clement
>>>>>
>>>>> 1) existing validation code will remain in SslFactory
>>>>> 2) the createEngine() method in SslEngineBuilder will move to
>>>>> SslFactory and the client/server mode setting will go there (I documented
>>>>> this in the latest KIP update)
>>>>>
>>>>> In the current KIP I am proposing (as per the latest updates) to make
>>>>> SSLContext loading/configuration/creation pluggable. I am not suggesting 
>>>>> we
>>>>> do/repeat anything that is already addressed by the existing Providers for
>>>>> SSLContext implementation. The createEngine() method (which will move to
>>>>> SslFactory) will call SslContextFactory.create() to get references to the
>>>>> SSLContext and then call SSLContext#createEngine(peer, host) and set
>>>>> client/server mode as it does today. I'll try to put that in a sequence
>>>>> diagram and update the KIP to make it clearer.
>>>>>
>>>>> So to your question about SslFactory returning SSLContext - I am
>>>>> saying register SslContextFactory interface to provide the SSLContext
>>>>> object instead and keep SslFactory more-or-less as it is today with some
>>>>> additional responsibility of createEngine() method.
>>>>>
>>>>> Thanks
>>>>> Maulin
>>>>>
>>>>> Thanks
>>>>> Maulin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 11, 2019 at 6:17 AM Pellerin, Clement <
>>>>> clement_pelle...@ibi.com> wrote:
>>>>>
>>>>>> Can you clarify a few points for me?
>>>>>>
>>>>>> The two stumbling blocks we have are:
>>>>>> 1) reuse of the validation code in the existing SslFactory
>>>>>> 2) the client/server mode on the SSLEngine
>>>>>>
>>>>>> How do you deal with those issues in your new proposal?
>>>>>>
>>>>>> My use case is to register a custom SslFactory that returns an
>>>>>> SSLContext previously created elsewhere in the application. Can your new
>>>>>> proposal handle this use case?
>>>>>>
>>>>>> -Original Message-
>>>>>> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
>>>>>> Sent: Friday, October 11, 2019 2:13 AM
>>>>>> To: dev@kafka.apache.org
>>>>>> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
>>>>>> extensible
>>>>>>
>>>>>> Check this out-
>>>>>>
>>>>>> https://github.com/apache/httpcomponents-core/blob/master/httpcore5/src/main/java/org/apache/hc/core5/ssl/SSLContextBuilder.java#L349
>>>>>>
>>>>>> This is exactly what I mean by using existing provider's SSLContext
>>>>>> implementation and customizing it with our data points. The similar
>>>>>> thing
>>>>>> Kafka's SslEngineBuilder is doing right now.
>&

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all,

I have updated the KIP document with the current state of conclusions.
Please review it and see if we are ready to move to Voting!

Thanks
Maulin

On Wed, Jan 22, 2020 at 12:42 AM Maulin Vasavada 
wrote:

> Hi all,
>
> Finally I squeezed time and I've a suggested code changes shown at
> https://github.com/maulin-vasavada/kafka/pull/4/files for discussing this
> further. I'll update the KIP document soon. Meanwhile, can you please take
> a look and continue the discussion?
>
> One challenge is at:
> https://github.com/maulin-vasavada/kafka/pull/4/files#diff-1e3432211fdbb7b2e2b44b5d8838a40bR89
>
> Thanks
> Maulin
>
>
> On Tue, Oct 22, 2019 at 11:13 PM Maulin Vasavada <
> maulin.vasav...@gmail.com> wrote:
>
>> bump! Clement/Rajini? Any responses based on the latest posts?
>>
>> On Wed, Oct 16, 2019 at 10:58 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com> wrote:
>>
>>> bump!
>>>
>>> On Sun, Oct 13, 2019 at 11:16 PM Maulin Vasavada <
>>> maulin.vasav...@gmail.com> wrote:
>>>
>>>> Hi Clement
>>>>
>>>> 1) existing validation code will remain in SslFactory
>>>> 2) the createEngine() method in SslEngineBuilder will move to
>>>> SslFactory and the client/server mode setting will go there (I documented
>>>> this in the latest KIP update)
>>>>
>>>> In the current KIP I am proposing (as per the latest updates) to make
>>>> SSLContext loading/configuration/creation pluggable. I am not suggesting we
>>>> do/repeat anything that is already addressed by the existing Providers for
>>>> SSLContext implementation. The createEngine() method (which will move to
>>>> SslFactory) will call SslContextFactory.create() to get references to the
>>>> SSLContext and then call SSLContext#createEngine(peer, host) and set
>>>> client/server mode as it does today. I'll try to put that in a sequence
>>>> diagram and update the KIP to make it clearer.
>>>>
>>>> So to your question about SslFactory returning SSLContext - I am saying
>>>> register SslContextFactory interface to provide the SSLContext object
>>>> instead and keep SslFactory more-or-less as it is today with some
>>>> additional responsibility of createEngine() method.
>>>>
>>>> Thanks
>>>> Maulin
>>>>
>>>> Thanks
>>>> Maulin
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Oct 11, 2019 at 6:17 AM Pellerin, Clement <
>>>> clement_pelle...@ibi.com> wrote:
>>>>
>>>>> Can you clarify a few points for me?
>>>>>
>>>>> The two stumbling blocks we have are:
>>>>> 1) reuse of the validation code in the existing SslFactory
>>>>> 2) the client/server mode on the SSLEngine
>>>>>
>>>>> How do you deal with those issues in your new proposal?
>>>>>
>>>>> My use case is to register a custom SslFactory that returns an
>>>>> SSLContext previously created elsewhere in the application. Can your new
>>>>> proposal handle this use case?
>>>>>
>>>>> -Original Message-
>>>>> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
>>>>> Sent: Friday, October 11, 2019 2:13 AM
>>>>> To: dev@kafka.apache.org
>>>>> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
>>>>> extensible
>>>>>
>>>>> Check this out-
>>>>>
>>>>> https://github.com/apache/httpcomponents-core/blob/master/httpcore5/src/main/java/org/apache/hc/core5/ssl/SSLContextBuilder.java#L349
>>>>>
>>>>> This is exactly what I mean by using existing provider's SSLContext
>>>>> implementation and customizing it with our data points. The similar
>>>>> thing
>>>>> Kafka's SslEngineBuilder is doing right now.
>>>>>
>>>>> On Thu, Oct 10, 2019 at 11:06 PM Maulin Vasavada <
>>>>> maulin.vasav...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > You meant JSSE not JCE right? We are not talking about cryptographic
>>>>> > providers we are talking about ssl providers hence JSSE.
>>>>> >
>>>>> > I do understand how JSSE Providers work and also the impact of
>>>>> multiple
>>>>> > JSSE providers with same algorithms in same JVM a

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all,

Finally I squeezed time and I've a suggested code changes shown at
https://github.com/maulin-vasavada/kafka/pull/4/files for discussing this
further. I'll update the KIP document soon. Meanwhile, can you please take
a look and continue the discussion?

One challenge is at:
https://github.com/maulin-vasavada/kafka/pull/4/files#diff-1e3432211fdbb7b2e2b44b5d8838a40bR89

Thanks
Maulin


On Tue, Oct 22, 2019 at 11:13 PM Maulin Vasavada 
wrote:

> bump! Clement/Rajini? Any responses based on the latest posts?
>
> On Wed, Oct 16, 2019 at 10:58 PM Maulin Vasavada <
> maulin.vasav...@gmail.com> wrote:
>
>> bump!
>>
>> On Sun, Oct 13, 2019 at 11:16 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com> wrote:
>>
>>> Hi Clement
>>>
>>> 1) existing validation code will remain in SslFactory
>>> 2) the createEngine() method in SslEngineBuilder will move to SslFactory
>>> and the client/server mode setting will go there (I documented this in the
>>> latest KIP update)
>>>
>>> In the current KIP I am proposing (as per the latest updates) to make
>>> SSLContext loading/configuration/creation pluggable. I am not suggesting we
>>> do/repeat anything that is already addressed by the existing Providers for
>>> SSLContext implementation. The createEngine() method (which will move to
>>> SslFactory) will call SslContextFactory.create() to get references to the
>>> SSLContext and then call SSLContext#createEngine(peer, host) and set
>>> client/server mode as it does today. I'll try to put that in a sequence
>>> diagram and update the KIP to make it clearer.
>>>
>>> So to your question about SslFactory returning SSLContext - I am saying
>>> register SslContextFactory interface to provide the SSLContext object
>>> instead and keep SslFactory more-or-less as it is today with some
>>> additional responsibility of createEngine() method.
>>>
>>> Thanks
>>> Maulin
>>>
>>> Thanks
>>> Maulin
>>>
>>>
>>>
>>>
>>> On Fri, Oct 11, 2019 at 6:17 AM Pellerin, Clement <
>>> clement_pelle...@ibi.com> wrote:
>>>
>>>> Can you clarify a few points for me?
>>>>
>>>> The two stumbling blocks we have are:
>>>> 1) reuse of the validation code in the existing SslFactory
>>>> 2) the client/server mode on the SSLEngine
>>>>
>>>> How do you deal with those issues in your new proposal?
>>>>
>>>> My use case is to register a custom SslFactory that returns an
>>>> SSLContext previously created elsewhere in the application. Can your new
>>>> proposal handle this use case?
>>>>
>>>> -Original Message-
>>>> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
>>>> Sent: Friday, October 11, 2019 2:13 AM
>>>> To: dev@kafka.apache.org
>>>> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
>>>> extensible
>>>>
>>>> Check this out-
>>>>
>>>> https://github.com/apache/httpcomponents-core/blob/master/httpcore5/src/main/java/org/apache/hc/core5/ssl/SSLContextBuilder.java#L349
>>>>
>>>> This is exactly what I mean by using existing provider's SSLContext
>>>> implementation and customizing it with our data points. The similar
>>>> thing
>>>> Kafka's SslEngineBuilder is doing right now.
>>>>
>>>> On Thu, Oct 10, 2019 at 11:06 PM Maulin Vasavada <
>>>> maulin.vasav...@gmail.com>
>>>> wrote:
>>>>
>>>> > You meant JSSE not JCE right? We are not talking about cryptographic
>>>> > providers we are talking about ssl providers hence JSSE.
>>>> >
>>>> > I do understand how JSSE Providers work and also the impact of
>>>> multiple
>>>> > JSSE providers with same algorithms in same JVM along with sequencing
>>>> > challenges for the same.
>>>> >
>>>> > Like you said- we need to allow customizing the configuration for
>>>> > SSLContext, so how many ways we have?
>>>> >
>>>> > Option-1: Write a custom JSSE Provider with our SSLContext
>>>> >
>>>> > Option-2: Use whichever SSLContext impl that you get from existing
>>>> JSSE
>>>> > Provider for SSLContext AND customize data for key material, trust
>>>> material
>>>> > AND secure random.
>>>> &

Re: [ANNOUNCE] New committer: Mickael Maison

[Maulin Vasavada]

Congratulations Mickael!

On Thu, Nov 7, 2019 at 8:27 PM Manikumar  wrote:

> Congrats Mickeal!
>
> On Fri, Nov 8, 2019 at 9:05 AM Dong Lin  wrote:
>
> > Congratulations Mickael!
> >
> > On Thu, Nov 7, 2019 at 1:38 PM Jun Rao  wrote:
> >
> > > Hi, Everyone,
> > >
> > > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > > Mickael
> > > Maison.
> > >
> > > Mickael has been contributing to Kafka since 2016. He proposed and
> > > implemented multiple KIPs. He has also been propomating Kafka through
> > blogs
> > > and public talks.
> > >
> > > Congratulations, Mickael!
> > >
> > > Thanks,
> > >
> > > Jun (on behalf of the Apache Kafka PMC)
> > >
> >
>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

bump! Clement/Rajini? Any responses based on the latest posts?

On Wed, Oct 16, 2019 at 10:58 PM Maulin Vasavada 
wrote:

> bump!
>
> On Sun, Oct 13, 2019 at 11:16 PM Maulin Vasavada <
> maulin.vasav...@gmail.com> wrote:
>
>> Hi Clement
>>
>> 1) existing validation code will remain in SslFactory
>> 2) the createEngine() method in SslEngineBuilder will move to SslFactory
>> and the client/server mode setting will go there (I documented this in the
>> latest KIP update)
>>
>> In the current KIP I am proposing (as per the latest updates) to make
>> SSLContext loading/configuration/creation pluggable. I am not suggesting we
>> do/repeat anything that is already addressed by the existing Providers for
>> SSLContext implementation. The createEngine() method (which will move to
>> SslFactory) will call SslContextFactory.create() to get references to the
>> SSLContext and then call SSLContext#createEngine(peer, host) and set
>> client/server mode as it does today. I'll try to put that in a sequence
>> diagram and update the KIP to make it clearer.
>>
>> So to your question about SslFactory returning SSLContext - I am saying
>> register SslContextFactory interface to provide the SSLContext object
>> instead and keep SslFactory more-or-less as it is today with some
>> additional responsibility of createEngine() method.
>>
>> Thanks
>> Maulin
>>
>> Thanks
>> Maulin
>>
>>
>>
>>
>> On Fri, Oct 11, 2019 at 6:17 AM Pellerin, Clement <
>> clement_pelle...@ibi.com> wrote:
>>
>>> Can you clarify a few points for me?
>>>
>>> The two stumbling blocks we have are:
>>> 1) reuse of the validation code in the existing SslFactory
>>> 2) the client/server mode on the SSLEngine
>>>
>>> How do you deal with those issues in your new proposal?
>>>
>>> My use case is to register a custom SslFactory that returns an
>>> SSLContext previously created elsewhere in the application. Can your new
>>> proposal handle this use case?
>>>
>>> -Original Message-
>>> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
>>> Sent: Friday, October 11, 2019 2:13 AM
>>> To: dev@kafka.apache.org
>>> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
>>> extensible
>>>
>>> Check this out-
>>>
>>> https://github.com/apache/httpcomponents-core/blob/master/httpcore5/src/main/java/org/apache/hc/core5/ssl/SSLContextBuilder.java#L349
>>>
>>> This is exactly what I mean by using existing provider's SSLContext
>>> implementation and customizing it with our data points. The similar thing
>>> Kafka's SslEngineBuilder is doing right now.
>>>
>>> On Thu, Oct 10, 2019 at 11:06 PM Maulin Vasavada <
>>> maulin.vasav...@gmail.com>
>>> wrote:
>>>
>>> > You meant JSSE not JCE right? We are not talking about cryptographic
>>> > providers we are talking about ssl providers hence JSSE.
>>> >
>>> > I do understand how JSSE Providers work and also the impact of multiple
>>> > JSSE providers with same algorithms in same JVM along with sequencing
>>> > challenges for the same.
>>> >
>>> > Like you said- we need to allow customizing the configuration for
>>> > SSLContext, so how many ways we have?
>>> >
>>> > Option-1: Write a custom JSSE Provider with our SSLContext
>>> >
>>> > Option-2: Use whichever SSLContext impl that you get from existing JSSE
>>> > Provider for SSLContext AND customize data for key material, trust
>>> material
>>> > AND secure random.
>>> >
>>> > Which one you prefer for this context?
>>> >
>>> > I feel we are making it complicated for no reason. It is very simple -
>>> > When we need to have SSL we need data points like - 1) Keys, 2) Trust
>>> certs
>>> > and 3) Secure Random which is feed to SSLContext and we are done. So
>>> we can
>>> > keep existing Kafka implementation as is by just making those data
>>> points
>>> > pluggable. Now SecureRandom is already pluggable via
>>> > 'ssl.secure.random.implementation' so that leaves us with keys and
>>> trusted
>>> > certs. For that purpose I raised KIP-486 BUT everybody feels we still
>>> need
>>> > higher level of pluggability hence this KIP.
>>> >
>>> > I"ve been taking advice from the dom

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

bump!

On Sun, Oct 13, 2019 at 11:16 PM Maulin Vasavada 
wrote:

> Hi Clement
>
> 1) existing validation code will remain in SslFactory
> 2) the createEngine() method in SslEngineBuilder will move to SslFactory
> and the client/server mode setting will go there (I documented this in the
> latest KIP update)
>
> In the current KIP I am proposing (as per the latest updates) to make
> SSLContext loading/configuration/creation pluggable. I am not suggesting we
> do/repeat anything that is already addressed by the existing Providers for
> SSLContext implementation. The createEngine() method (which will move to
> SslFactory) will call SslContextFactory.create() to get references to the
> SSLContext and then call SSLContext#createEngine(peer, host) and set
> client/server mode as it does today. I'll try to put that in a sequence
> diagram and update the KIP to make it clearer.
>
> So to your question about SslFactory returning SSLContext - I am saying
> register SslContextFactory interface to provide the SSLContext object
> instead and keep SslFactory more-or-less as it is today with some
> additional responsibility of createEngine() method.
>
> Thanks
> Maulin
>
> Thanks
> Maulin
>
>
>
>
> On Fri, Oct 11, 2019 at 6:17 AM Pellerin, Clement <
> clement_pelle...@ibi.com> wrote:
>
>> Can you clarify a few points for me?
>>
>> The two stumbling blocks we have are:
>> 1) reuse of the validation code in the existing SslFactory
>> 2) the client/server mode on the SSLEngine
>>
>> How do you deal with those issues in your new proposal?
>>
>> My use case is to register a custom SslFactory that returns an SSLContext
>> previously created elsewhere in the application. Can your new proposal
>> handle this use case?
>>
>> -Original Message-
>> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
>> Sent: Friday, October 11, 2019 2:13 AM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
>> extensible
>>
>> Check this out-
>>
>> https://github.com/apache/httpcomponents-core/blob/master/httpcore5/src/main/java/org/apache/hc/core5/ssl/SSLContextBuilder.java#L349
>>
>> This is exactly what I mean by using existing provider's SSLContext
>> implementation and customizing it with our data points. The similar thing
>> Kafka's SslEngineBuilder is doing right now.
>>
>> On Thu, Oct 10, 2019 at 11:06 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> wrote:
>>
>> > You meant JSSE not JCE right? We are not talking about cryptographic
>> > providers we are talking about ssl providers hence JSSE.
>> >
>> > I do understand how JSSE Providers work and also the impact of multiple
>> > JSSE providers with same algorithms in same JVM along with sequencing
>> > challenges for the same.
>> >
>> > Like you said- we need to allow customizing the configuration for
>> > SSLContext, so how many ways we have?
>> >
>> > Option-1: Write a custom JSSE Provider with our SSLContext
>> >
>> > Option-2: Use whichever SSLContext impl that you get from existing JSSE
>> > Provider for SSLContext AND customize data for key material, trust
>> material
>> > AND secure random.
>> >
>> > Which one you prefer for this context?
>> >
>> > I feel we are making it complicated for no reason. It is very simple -
>> > When we need to have SSL we need data points like - 1) Keys, 2) Trust
>> certs
>> > and 3) Secure Random which is feed to SSLContext and we are done. So we
>> can
>> > keep existing Kafka implementation as is by just making those data
>> points
>> > pluggable. Now SecureRandom is already pluggable via
>> > 'ssl.secure.random.implementation' so that leaves us with keys and
>> trusted
>> > certs. For that purpose I raised KIP-486 BUT everybody feels we still
>> need
>> > higher level of pluggability hence this KIP.
>> >
>> > I"ve been taking advice from the domain experts and Application security
>> > teams and to them it is very straight-forward - Make SSLContext
>> > configuration/building pluggable and that's it!
>> >
>> > Thanks
>> > Maulin
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Oct 7, 2019 at 5:47 AM Pellerin, Clement <
>> clement_pelle...@ibi.com>
>> > wrote:
>> >
>> >> If I understand correctly, you are pro

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi Clement

1) existing validation code will remain in SslFactory
2) the createEngine() method in SslEngineBuilder will move to SslFactory
and the client/server mode setting will go there (I documented this in the
latest KIP update)

In the current KIP I am proposing (as per the latest updates) to make
SSLContext loading/configuration/creation pluggable. I am not suggesting we
do/repeat anything that is already addressed by the existing Providers for
SSLContext implementation. The createEngine() method (which will move to
SslFactory) will call SslContextFactory.create() to get references to the
SSLContext and then call SSLContext#createEngine(peer, host) and set
client/server mode as it does today. I'll try to put that in a sequence
diagram and update the KIP to make it clearer.

So to your question about SslFactory returning SSLContext - I am saying
register SslContextFactory interface to provide the SSLContext object
instead and keep SslFactory more-or-less as it is today with some
additional responsibility of createEngine() method.

Thanks
Maulin

Thanks
Maulin




On Fri, Oct 11, 2019 at 6:17 AM Pellerin, Clement 
wrote:

> Can you clarify a few points for me?
>
> The two stumbling blocks we have are:
> 1) reuse of the validation code in the existing SslFactory
> 2) the client/server mode on the SSLEngine
>
> How do you deal with those issues in your new proposal?
>
> My use case is to register a custom SslFactory that returns an SSLContext
> previously created elsewhere in the application. Can your new proposal
> handle this use case?
>
> -----Original Message-
> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> Sent: Friday, October 11, 2019 2:13 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> Check this out-
>
> https://github.com/apache/httpcomponents-core/blob/master/httpcore5/src/main/java/org/apache/hc/core5/ssl/SSLContextBuilder.java#L349
>
> This is exactly what I mean by using existing provider's SSLContext
> implementation and customizing it with our data points. The similar thing
> Kafka's SslEngineBuilder is doing right now.
>
> On Thu, Oct 10, 2019 at 11:06 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> wrote:
>
> > You meant JSSE not JCE right? We are not talking about cryptographic
> > providers we are talking about ssl providers hence JSSE.
> >
> > I do understand how JSSE Providers work and also the impact of multiple
> > JSSE providers with same algorithms in same JVM along with sequencing
> > challenges for the same.
> >
> > Like you said- we need to allow customizing the configuration for
> > SSLContext, so how many ways we have?
> >
> > Option-1: Write a custom JSSE Provider with our SSLContext
> >
> > Option-2: Use whichever SSLContext impl that you get from existing JSSE
> > Provider for SSLContext AND customize data for key material, trust
> material
> > AND secure random.
> >
> > Which one you prefer for this context?
> >
> > I feel we are making it complicated for no reason. It is very simple -
> > When we need to have SSL we need data points like - 1) Keys, 2) Trust
> certs
> > and 3) Secure Random which is feed to SSLContext and we are done. So we
> can
> > keep existing Kafka implementation as is by just making those data points
> > pluggable. Now SecureRandom is already pluggable via
> > 'ssl.secure.random.implementation' so that leaves us with keys and
> trusted
> > certs. For that purpose I raised KIP-486 BUT everybody feels we still
> need
> > higher level of pluggability hence this KIP.
> >
> > I"ve been taking advice from the domain experts and Application security
> > teams and to them it is very straight-forward - Make SSLContext
> > configuration/building pluggable and that's it!
> >
> > Thanks
> > Maulin
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Oct 7, 2019 at 5:47 AM Pellerin, Clement <
> clement_pelle...@ibi.com>
> > wrote:
> >
> >> If I understand correctly, you are proposing to abandon the idea of a
> >> pluggable extension point for SSL in Kafka because we could rely on the
> JCE
> >> provider mechanism.
> >>
> >> I will reiterate that nobody does it that way. That in itself should be
> >> enough but let's discuss some of the reasons why.
> >>
> >> Changing the order of the JCE providers in the java.security file
> affects
> >> all java applications so you probably don't want to do it there.
> Changing
> >> the order of the JCE providers in the JVM instance affects 

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Check this out-
https://github.com/apache/httpcomponents-core/blob/master/httpcore5/src/main/java/org/apache/hc/core5/ssl/SSLContextBuilder.java#L349

This is exactly what I mean by using existing provider's SSLContext
implementation and customizing it with our data points. The similar thing
Kafka's SslEngineBuilder is doing right now.

On Thu, Oct 10, 2019 at 11:06 PM Maulin Vasavada 
wrote:

> You meant JSSE not JCE right? We are not talking about cryptographic
> providers we are talking about ssl providers hence JSSE.
>
> I do understand how JSSE Providers work and also the impact of multiple
> JSSE providers with same algorithms in same JVM along with sequencing
> challenges for the same.
>
> Like you said- we need to allow customizing the configuration for
> SSLContext, so how many ways we have?
>
> Option-1: Write a custom JSSE Provider with our SSLContext
>
> Option-2: Use whichever SSLContext impl that you get from existing JSSE
> Provider for SSLContext AND customize data for key material, trust material
> AND secure random.
>
> Which one you prefer for this context?
>
> I feel we are making it complicated for no reason. It is very simple -
> When we need to have SSL we need data points like - 1) Keys, 2) Trust certs
> and 3) Secure Random which is feed to SSLContext and we are done. So we can
> keep existing Kafka implementation as is by just making those data points
> pluggable. Now SecureRandom is already pluggable via
> 'ssl.secure.random.implementation' so that leaves us with keys and trusted
> certs. For that purpose I raised KIP-486 BUT everybody feels we still need
> higher level of pluggability hence this KIP.
>
> I"ve been taking advice from the domain experts and Application security
> teams and to them it is very straight-forward - Make SSLContext
> configuration/building pluggable and that's it!
>
> Thanks
> Maulin
>
>
>
>
>
>
>
>
>
>
> On Mon, Oct 7, 2019 at 5:47 AM Pellerin, Clement 
> wrote:
>
>> If I understand correctly, you are proposing to abandon the idea of a
>> pluggable extension point for SSL in Kafka because we could rely on the JCE
>> provider mechanism.
>>
>> I will reiterate that nobody does it that way. That in itself should be
>> enough but let's discuss some of the reasons why.
>>
>> Changing the order of the JCE providers in the java.security file affects
>> all java applications so you probably don't want to do it there. Changing
>> the order of the JCE providers in the JVM instance affects all code it
>> runs. Your library is not alone in the JVM process and other code will want
>> regular SSLContext instances. That leaves you with the only option of
>> specifying the provider explicitly when you create the SSLContext instance
>> in Kafka. That would work, as long as your users don't mess things up with
>> the very common configuration approaches above.
>>
>> A JCE SSLContext provider is intended to be a mechanism to replace the
>> SSLContext implementation. Our purpose is to customize the configuration,
>> not to replace it. This becomes hard to do when your only chance is at
>> creation time. Kafka then does its thing and you have no way to modify that
>> behavior in Kafka. You no longer support many legitimate use cases.
>>
>> The final blow is the need to sign JCE providers using a certificate
>> signed by Oracle's JCE Code Signing Certification Authority.
>>
>> https://www.oracle.com/technetwork/java/javase/tech/getcodesigningcertificate-361306.html
>> JCE will refuse to load your provider if it is not signed. Getting the
>> certificate is a pain and it takes time. You also have to worry about the
>> certificate expiration date. There are JVMs that don't require signed JCE
>> providers, but you cannot limit Kafka to just those JVMs.
>>
>> -Original Message-
>> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
>> Sent: Friday, October 4, 2019 5:31 PM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
>> extensible
>>
>> In other words, Kafka doesn't necessarily need to derive another
>> interface/mechanism to make SSLEngine pluggable. That interface/mechanism
>> exists in Java with Security Provider's SSLContext Algorithms.
>> Ref-1:
>>
>> https://docs.oracle.com/javase/9/docs/specs/security/standard-names.html#sslcontext-algorithms
>> Ref-2
>> <https://docs.oracle.com/javase/9/docs/specs/security/standard-names.html#sslcontext-algorithmsRef-2>
>> :
>>
>> https://github.com/bcgit/bc-java/blob/master/tls/src/main/java/org/bouncycastle/jsse/provider/BouncyCast

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

You meant JSSE not JCE right? We are not talking about cryptographic
providers we are talking about ssl providers hence JSSE.

I do understand how JSSE Providers work and also the impact of multiple
JSSE providers with same algorithms in same JVM along with sequencing
challenges for the same.

Like you said- we need to allow customizing the configuration for
SSLContext, so how many ways we have?

Option-1: Write a custom JSSE Provider with our SSLContext

Option-2: Use whichever SSLContext impl that you get from existing JSSE
Provider for SSLContext AND customize data for key material, trust material
AND secure random.

Which one you prefer for this context?

I feel we are making it complicated for no reason. It is very simple - When
we need to have SSL we need data points like - 1) Keys, 2) Trust certs and
3) Secure Random which is feed to SSLContext and we are done. So we can
keep existing Kafka implementation as is by just making those data points
pluggable. Now SecureRandom is already pluggable via
'ssl.secure.random.implementation' so that leaves us with keys and trusted
certs. For that purpose I raised KIP-486 BUT everybody feels we still need
higher level of pluggability hence this KIP.

I"ve been taking advice from the domain experts and Application security
teams and to them it is very straight-forward - Make SSLContext
configuration/building pluggable and that's it!

Thanks
Maulin










On Mon, Oct 7, 2019 at 5:47 AM Pellerin, Clement 
wrote:

> If I understand correctly, you are proposing to abandon the idea of a
> pluggable extension point for SSL in Kafka because we could rely on the JCE
> provider mechanism.
>
> I will reiterate that nobody does it that way. That in itself should be
> enough but let's discuss some of the reasons why.
>
> Changing the order of the JCE providers in the java.security file affects
> all java applications so you probably don't want to do it there. Changing
> the order of the JCE providers in the JVM instance affects all code it
> runs. Your library is not alone in the JVM process and other code will want
> regular SSLContext instances. That leaves you with the only option of
> specifying the provider explicitly when you create the SSLContext instance
> in Kafka. That would work, as long as your users don't mess things up with
> the very common configuration approaches above.
>
> A JCE SSLContext provider is intended to be a mechanism to replace the
> SSLContext implementation. Our purpose is to customize the configuration,
> not to replace it. This becomes hard to do when your only chance is at
> creation time. Kafka then does its thing and you have no way to modify that
> behavior in Kafka. You no longer support many legitimate use cases.
>
> The final blow is the need to sign JCE providers using a certificate
> signed by Oracle's JCE Code Signing Certification Authority.
>
> https://www.oracle.com/technetwork/java/javase/tech/getcodesigningcertificate-361306.html
> JCE will refuse to load your provider if it is not signed. Getting the
> certificate is a pain and it takes time. You also have to worry about the
> certificate expiration date. There are JVMs that don't require signed JCE
> providers, but you cannot limit Kafka to just those JVMs.
>
> -Original Message-
> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> Sent: Friday, October 4, 2019 5:31 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> In other words, Kafka doesn't necessarily need to derive another
> interface/mechanism to make SSLEngine pluggable. That interface/mechanism
> exists in Java with Security Provider's SSLContext Algorithms.
> Ref-1:
>
> https://docs.oracle.com/javase/9/docs/specs/security/standard-names.html#sslcontext-algorithms
> Ref-2
> <https://docs.oracle.com/javase/9/docs/specs/security/standard-names.html#sslcontext-algorithmsRef-2>
> :
>
> https://github.com/bcgit/bc-java/blob/master/tls/src/main/java/org/bouncycastle/jsse/provider/BouncyCastleJsseProvider.java#L193
>
> About the " whole world chooses to make the javax.net.ssl.SSLSocketFactory
> pluggable" I found the official documentation reinforcing my point I made
> above,
> "The javax.net.ssl.SSLSocket class represents a network socket that
> encapsulates SSL/TLS support on top of a normal stream socket (
> java.net.Socket). Some applications might want to use alternate data
> transport abstractions (e.g., New-I/O); the javax.net.ssl.SSLEngine class
> is available to produce and consume SSL/TLS packets."
> Reference:
>
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/overview/jsoverview.html
>
> I feel that we have to think about building SSLContext in a pluggable way
> since that is the class that takes "key/trust" material and secure-random
> config and help creates SSLEngine, SocketFactories via the TLS algorithm's
> provider specified by Security Provider configuration.
>
> Thanks
> Maulin
>
>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

In other words, Kafka doesn't necessarily need to derive another
interface/mechanism to make SSLEngine pluggable. That interface/mechanism
exists in Java with Security Provider's SSLContext Algorithms.
Ref-1:
https://docs.oracle.com/javase/9/docs/specs/security/standard-names.html#sslcontext-algorithms
Ref-2:
https://github.com/bcgit/bc-java/blob/master/tls/src/main/java/org/bouncycastle/jsse/provider/BouncyCastleJsseProvider.java#L193

About the " whole world chooses to make the javax.net.ssl.SSLSocketFactory
pluggable" I found the official documentation reinforcing my point I made
above,
"The javax.net.ssl.SSLSocket class represents a network socket that
encapsulates SSL/TLS support on top of a normal stream socket (
java.net.Socket). Some applications might want to use alternate data
transport abstractions (e.g., New-I/O); the javax.net.ssl.SSLEngine class
is available to produce and consume SSL/TLS packets."
Reference:
https://docs.oracle.com/javase/7/docs/technotes/guides/security/overview/jsoverview.html

I feel that we have to think about building SSLContext in a pluggable way
since that is the class that takes "key/trust" material and secure-random
config and help creates SSLEngine, SocketFactories via the TLS algorithm's
provider specified by Security Provider configuration.

Thanks
Maulin



On Fri, Oct 4, 2019 at 10:28 AM Maulin Vasavada 
wrote:

> Hi Clement
>
> The explanation about using Java Security Providers if you want to use
> customize SSLEngine is - similar to BouncyCastleJsseProvider we can have a
> provider that registers SSLContext+(TLS/SSL) services and have
> implementation for SSLContextSpi  which provides various methods to
> get/create different kind of objects offered by the specific TLS
> implementation - example: SSLEngine, ServerSocketFactory, SocketFactory.
> SSLContext works as kind of a "bridge" pattern that once we get SSLContext
> for a specific protocol (TLSv1.1,TLSv1.2 etc) then ssl engine, socket
> factory everything is for that protocol implementation.
>
> About the SSLContext vs SSLEngine, I debated the exact point you raise in
> my mind and I came to the conclusion that - The implementation of Kafka or
> HTTPs protocol (like ApacheHttpClient) ultimately relies on Java IO/NIO
> packages and uses some kind of Channel (example:
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/network/TransportLayer.java).
> Now they have a choice to either use SSLEngine or SSLSocket. It is entirely
> upto the protocol's implementation to decide that. Hence we can't say
> everybody must use SSLSocketFacotry vs SSLEngine. I will dig into Apache
> Http Client and find how they are using SocketFactory.
>
> Thanks
> Maulin
>
>
>
>
>
>
>
> On Fri, Oct 4, 2019 at 8:37 AM Pellerin, Clement 
> wrote:
>
>> I am happy to see I am not the only one with reservations on the
>> direction we were heading.
>>
>> >> I feel if you want to plug any JSSE compatible  SSLEngine
>> >> (example BouncyCastle, wildfly-OpenSSL etc) then we have to follow
>> path of
>> >> using Java security provider
>>
>> I like the conclusion but I did not understand your explanation.
>>
>> >> What should be pluggable - SSLContext or SSLEngine?
>> I can only remark that the whole world chooses to make the
>> javax.net.ssl.SSLSocketFactory pluggable. You can find projects making the
>> SSLContext pluggable but I have never seen a project that makes the
>> SSLEngine pluggable.
>>
>> The stumbling blocks in our case are the client/server mode customization
>> and the reusability of the reconfiguration checks. We can always compromise
>> on the reusability of the reconfiguration checks. On the other hand, we
>> really have to learn more about the client/server mode customization
>> because that's unavoidable at the moment.
>>
>> -Original Message-
>> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
>> Sent: Friday, October 4, 2019 4:13 AM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
>> extensible
>>
>> Hi all
>>
>> I've been having more thoughts on SSLEngine vs SSLContext pluggability
>> (reasons for hiatus from my side until now). Based on my further research
>> and understanding, various TLS implementations
>> https://en.wikipedia.org/wiki/Comparison_of_TLS_implementations , makes
>> it
>> clear that there are implementations that may not be originally complying
>> to JSSE BUT they eventually might provide JSSE Provider, example -
>> BouncyCastleJsseProvider -
>>
>> https://github.com/bcgit/bc-java/blob/mast

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi Clement

The explanation about using Java Security Providers if you want to use
customize SSLEngine is - similar to BouncyCastleJsseProvider we can have a
provider that registers SSLContext+(TLS/SSL) services and have
implementation for SSLContextSpi  which provides various methods to
get/create different kind of objects offered by the specific TLS
implementation - example: SSLEngine, ServerSocketFactory, SocketFactory.
SSLContext works as kind of a "bridge" pattern that once we get SSLContext
for a specific protocol (TLSv1.1,TLSv1.2 etc) then ssl engine, socket
factory everything is for that protocol implementation.

About the SSLContext vs SSLEngine, I debated the exact point you raise in
my mind and I came to the conclusion that - The implementation of Kafka or
HTTPs protocol (like ApacheHttpClient) ultimately relies on Java IO/NIO
packages and uses some kind of Channel (example:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/network/TransportLayer.java).
Now they have a choice to either use SSLEngine or SSLSocket. It is entirely
upto the protocol's implementation to decide that. Hence we can't say
everybody must use SSLSocketFacotry vs SSLEngine. I will dig into Apache
Http Client and find how they are using SocketFactory.

Thanks
Maulin







On Fri, Oct 4, 2019 at 8:37 AM Pellerin, Clement 
wrote:

> I am happy to see I am not the only one with reservations on the direction
> we were heading.
>
> >> I feel if you want to plug any JSSE compatible  SSLEngine
> >> (example BouncyCastle, wildfly-OpenSSL etc) then we have to follow path
> of
> >> using Java security provider
>
> I like the conclusion but I did not understand your explanation.
>
> >> What should be pluggable - SSLContext or SSLEngine?
> I can only remark that the whole world chooses to make the
> javax.net.ssl.SSLSocketFactory pluggable. You can find projects making the
> SSLContext pluggable but I have never seen a project that makes the
> SSLEngine pluggable.
>
> The stumbling blocks in our case are the client/server mode customization
> and the reusability of the reconfiguration checks. We can always compromise
> on the reusability of the reconfiguration checks. On the other hand, we
> really have to learn more about the client/server mode customization
> because that's unavoidable at the moment.
>
> -Original Message-
> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> Sent: Friday, October 4, 2019 4:13 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> Hi all
>
> I've been having more thoughts on SSLEngine vs SSLContext pluggability
> (reasons for hiatus from my side until now). Based on my further research
> and understanding, various TLS implementations
> https://en.wikipedia.org/wiki/Comparison_of_TLS_implementations , makes it
> clear that there are implementations that may not be originally complying
> to JSSE BUT they eventually might provide JSSE Provider, example -
> BouncyCastleJsseProvider -
>
> https://github.com/bcgit/bc-java/blob/master/tls/src/main/java/org/bouncycastle/jsse/provider/BouncyCastleJsseProvider.java
> .
> In those cases, I feel if you want to plug any JSSE compatible  SSLEngine
> (example BouncyCastle, wildfly-OpenSSL etc) then we have to follow path of
> using Java security provider (like using BouncyCastleJsseProvider class).
> We can't easily just use the proposed SslEngineFactory interface.
>
> Also, the existing logic of createEngine() method actually just does
> sslContext.createEngine() and then customize the object further based on
> the Client/Server mode. It doesn't really do any customization of SSLEngine
> wrap/unwrap methods which are the heart of it. That urges me to think more
> that actually we need SSLContext to be pluggable.
>
> Either way, point of discussions about reconfigurability and questions
> Clement asked remains similar BUT I think we might have to first really
> resolve "What should be pluggable - SSLContext or SSLEngine?" question.
>
> Let me know your thoughts!
>
> Thanks
> Maulin
>
>
>
>
>
>
>
>
> On Wed, Sep 25, 2019 at 10:56 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> wrote:
>
> > Ack. I should be able to get back to this on Friday.
> >
> > On Mon, Sep 23, 2019 at 10:35 AM Pellerin, Clement <
> > clement_pelle...@ibi.com> wrote:
> >
> >> When I worked on KIP-383 I was told the way to pass extra arguments to
> an
> >> instance is to add extra arguments to configure. I would now suggest we
> do
> >> like the KeySerializer. If you look in KafkaProducer, it creates a
> >> KeySerializer usi

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all

I've been having more thoughts on SSLEngine vs SSLContext pluggability
(reasons for hiatus from my side until now). Based on my further research
and understanding, various TLS implementations
https://en.wikipedia.org/wiki/Comparison_of_TLS_implementations , makes it
clear that there are implementations that may not be originally complying
to JSSE BUT they eventually might provide JSSE Provider, example -
BouncyCastleJsseProvider -
https://github.com/bcgit/bc-java/blob/master/tls/src/main/java/org/bouncycastle/jsse/provider/BouncyCastleJsseProvider.java.
In those cases, I feel if you want to plug any JSSE compatible  SSLEngine
(example BouncyCastle, wildfly-OpenSSL etc) then we have to follow path of
using Java security provider (like using BouncyCastleJsseProvider class).
We can't easily just use the proposed SslEngineFactory interface.

Also, the existing logic of createEngine() method actually just does
sslContext.createEngine() and then customize the object further based on
the Client/Server mode. It doesn't really do any customization of SSLEngine
wrap/unwrap methods which are the heart of it. That urges me to think more
that actually we need SSLContext to be pluggable.

Either way, point of discussions about reconfigurability and questions
Clement asked remains similar BUT I think we might have to first really
resolve "What should be pluggable - SSLContext or SSLEngine?" question.

Let me know your thoughts!

Thanks
Maulin








On Wed, Sep 25, 2019 at 10:56 PM Maulin Vasavada 
wrote:

> Ack. I should be able to get back to this on Friday.
>
> On Mon, Sep 23, 2019 at 10:35 AM Pellerin, Clement <
> clement_pelle...@ibi.com> wrote:
>
>> When I worked on KIP-383 I was told the way to pass extra arguments to an
>> instance is to add extra arguments to configure. I would now suggest we do
>> like the KeySerializer. If you look in KafkaProducer, it creates a
>> KeySerializer using AbstractConfig. getConfiguredInstance(). Since
>> KeySerializer does not extend KeySerializer, AbstractConfig does not call
>> configure(Map). KafkaProducer then calls configure() with two arguments.
>> This removes the need for the init() method which would be called too late
>> after configure(). By the way, the KeySerializer is not the only interface
>> that does this.
>>
>> In summary, SslEngineFactory does not extend Configurable and it has a
>> configure() method with more than 1 argument.
>>
>> The next item is to spec how config.originals() is passed to
>> SslChannelBuilder: the KIP needs to explain we will push the choice of
>> configs within the switch in ChannelBuilders.create()
>>
>> Finally, we need to spec which configs are passed to shouldRebuiltFor().
>> I assume these are now originals() instead of values(). We also need to
>> specify if all configs are passed or just the reconfigurable ones.
>>
>>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Ack. I should be able to get back to this on Friday.

On Mon, Sep 23, 2019 at 10:35 AM Pellerin, Clement 
wrote:

> When I worked on KIP-383 I was told the way to pass extra arguments to an
> instance is to add extra arguments to configure. I would now suggest we do
> like the KeySerializer. If you look in KafkaProducer, it creates a
> KeySerializer using AbstractConfig. getConfiguredInstance(). Since
> KeySerializer does not extend KeySerializer, AbstractConfig does not call
> configure(Map). KafkaProducer then calls configure() with two arguments.
> This removes the need for the init() method which would be called too late
> after configure(). By the way, the KeySerializer is not the only interface
> that does this.
>
> In summary, SslEngineFactory does not extend Configurable and it has a
> configure() method with more than 1 argument.
>
> The next item is to spec how config.originals() is passed to
> SslChannelBuilder: the KIP needs to explain we will push the choice of
> configs within the switch in ChannelBuilders.create()
>
> Finally, we need to spec which configs are passed to shouldRebuiltFor(). I
> assume these are now originals() instead of values(). We also need to
> specify if all configs are passed or just the reconfigurable ones.
>
>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Yes the checks should be mandatory and as I mentioned before we should keep
them outside of the pluggable implementation - the way currently it is in
SslFactory.

I think we can wait for comments from  @Colin McCabe 
 and @Rajini Sivaram  as soon as they can get some
time out of 2.4 release. May be later this week or early next week?

Thanks
Maulin

On Sat, Sep 21, 2019 at 5:24 AM Pellerin, Clement 
wrote:

> I may have found the technical reason.
>
> It would be logical if the customizable reconfigurable extension point
> extends the Reconfigurable interface. That's what the first iteration of
> KIP-383 proposed. It lost its vote because the voters wanted to preserve
> the reconfiguration checks in SslFactory. The next thing to try is to push
> those checks in the creator which is the channel builder. A Reconfigurable
> instance is mutable by definition. It can either be in its current
> configuration, or the reconfigured configuration but not in between. The
> checks require to partially build the new configuration before accepting it
> and therefore this approach does not work.
>
> We could move the checks in a helper validator class, then we would rely
> on the Reconfigurable instance to call that code. This would reuse the code
> but the checks would not be mandatory.
>
> The conclusion is the checks are mandatory or the interface extends the
> Reconfigurable interface but not both.
>
> Is that reasoning what you were saying?
> Do we make the checks mandatory or not?
> Do we support all the use cases we want?
>
> -Original Message-
> From: Pellerin, Clement
> Sent: Friday, September 20, 2019 5:24 PM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> The KIP now says: We believe that making SSLEngine creation pluggable is
> worth to allow SSL experts to write their own implementation having the SSL
> domain knowledge and keep them free of knowing much about Kafka's
> reconfigurability.
>
> The other custom reconfigurable extension points don't seem to have a
> problem with that. You may have a point though, so I need to look at the
> reconfiguration code you mention.
>
> Is your latest prototype available so I can study it?
>
> -Original Message-
> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> Sent: Friday, September 20, 2019 4:40 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> Thanks Clement for your thoughts. According to my current experience
> rewriting the code twice I would say I did what you suggest in the last
> point - " We must make an attempt, if only to explain why it fails in the
> Rejected Alternatives section of the KIP." In the rejected alternatives I
> already mention why not merge SslFactory and SslEngineFactory and make
> SslFactory reconfigurable.
>
> @Colin can you please express your thoughts on this discussion so far?
> Since you refactored the SslFactory's code to have SslEngineBuilder I feel
> you would have more insights into future changes.
>
> Thanks
> Maulin
>
>
> On Fri, Sep 20, 2019 at 7:29 AM Pellerin, Clement <
> clement_pelle...@ibi.com>
> wrote:
>
> > There will be chaperon code in the base class of the channel builders.
> > The arguments you gave me are emotional not technical.
> > The SSL extension point is reconfigurable hence it should extend
> > Reconfigurable.
> > We will encounter issues when we try to prototype it that way.
> > We will solve the issues or backtrack to another solution.
> > We must make an attempt, if only to explain why it fails in the Rejected
> > Alternatives section of the KIP.
> >
> > -Original Message-
> > From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> > Sent: Friday, September 20, 2019 2:40 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> > extensible
> >
> > Overall my thinking is - When somebody wants to customize creation of
> > SSLEngine, most likely they are more expert in dealing with SSL domain
> > related stuff than "Kafka's reconfigurability" aspect. As a custom
> > implementation it makes more sense to me to say - Hey I'll control how I
> > initialize my SSL context/engine and btw if Kafka can give me a way to
> just
> > get re-created whenever some set of config keys changes, it is great!
> This
> > is similar to my thinking on KIP-486 which is- as a Kafka operator I am
> > just trying to be compliant to my company's security policies to load
> > keys/certs in certain way. For that, I should not be

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Thanks Clement for your thoughts. According to my current experience
rewriting the code twice I would say I did what you suggest in the last
point - " We must make an attempt, if only to explain why it fails in the
Rejected Alternatives section of the KIP." In the rejected alternatives I
already mention why not merge SslFactory and SslEngineFactory and make
SslFactory reconfigurable.

@Colin can you please express your thoughts on this discussion so far?
Since you refactored the SslFactory's code to have SslEngineBuilder I feel
you would have more insights into future changes.

Thanks
Maulin


On Fri, Sep 20, 2019 at 7:29 AM Pellerin, Clement 
wrote:

> There will be chaperon code in the base class of the channel builders.
> The arguments you gave me are emotional not technical.
> The SSL extension point is reconfigurable hence it should extend
> Reconfigurable.
> We will encounter issues when we try to prototype it that way.
> We will solve the issues or backtrack to another solution.
> We must make an attempt, if only to explain why it fails in the Rejected
> Alternatives section of the KIP.
>
> -----Original Message-
> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> Sent: Friday, September 20, 2019 2:40 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> Overall my thinking is - When somebody wants to customize creation of
> SSLEngine, most likely they are more expert in dealing with SSL domain
> related stuff than "Kafka's reconfigurability" aspect. As a custom
> implementation it makes more sense to me to say - Hey I'll control how I
> initialize my SSL context/engine and btw if Kafka can give me a way to just
> get re-created whenever some set of config keys changes, it is great! This
> is similar to my thinking on KIP-486 which is- as a Kafka operator I am
> just trying to be compliant to my company's security policies to load
> keys/certs in certain way. For that, I should not be penalized by Kafka to
> know all about Java security Providers and how to really create SSLContext
> object etc given Java already provides a way to feed in KeyStore object
> regardless of how I load it.
>
> On Thu, Sep 19, 2019 at 10:57 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> wrote:
>
> > Hi Clement
> >
> > There will be good amount of state in the SslEngineFactory's default
> > implementation. Hence I feel we might anyway have a chaperon class to
> > provide reconfigurable functionality and will have one more class to host
> > the state/behavior of actual SSLContext/SSLEngine creation. While doing
> the
> > internal rewrite (so far two times) both of the times I reached to the
> same
> > conclusion.I feel if we leave the reconfigurations to the implementation
> -
> > it will repeat the same pattern of having two classes to manage it -
> since
> > most likely they will also have similar state information. Instead keep
> > that reconfigurations in SslFactory as is today and just allow "plugin of
> > creation of SSLEngine".
> >
> > One note I would like to make is: You are comparing this to
> MetricReporter
> > but we have to keep in mind that SSL configuration is inherently more
> > complex than a MetricReporters functionality. There are no JSSE
> equivalent
> > documents needed to be written for MetricReporter for example. So what
> > works best for simpler solutions may not work equally well for more
> complex
> > scenarios.
> >
> >
> > Thanks
> > Maulin
> >
> >
> > On Thu, Sep 19, 2019 at 8:36 PM Pellerin, Clement <
> > clement_pelle...@ibi.com> wrote:
> >
> >> We will get there eventually but I need to address another point first.
> >>
> >> My goal is to do exactly what the "other extension points with
> >> reconfigurable custom configs" are doing unless there is a good reason
> not
> >> to. They provide a ready-made solution that will let us reuse code,
> avoid
> >> pitfalls and show consistency.
> >>
> >> So far the roadblocks are
> >> - the need to enforce mandatory compatibility checks for the keystores
> >> and SSL handshake
> >> - SslFactory is used in two channel builders.
> >>
> >> Both of these roadblocks can be overcome by moving the checks to a new
> >> common base class of SslChannelBuilder and SaslChannelBuilder. This is
> easy
> >> since both classes extend Object directly. The new base class is not a
> >> public API so any implementation will do. The chaperon class SslFactory
> >> disappears and the int

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Overall my thinking is - When somebody wants to customize creation of
SSLEngine, most likely they are more expert in dealing with SSL domain
related stuff than "Kafka's reconfigurability" aspect. As a custom
implementation it makes more sense to me to say - Hey I'll control how I
initialize my SSL context/engine and btw if Kafka can give me a way to just
get re-created whenever some set of config keys changes, it is great! This
is similar to my thinking on KIP-486 which is- as a Kafka operator I am
just trying to be compliant to my company's security policies to load
keys/certs in certain way. For that, I should not be penalized by Kafka to
know all about Java security Providers and how to really create SSLContext
object etc given Java already provides a way to feed in KeyStore object
regardless of how I load it.

On Thu, Sep 19, 2019 at 10:57 PM Maulin Vasavada 
wrote:

> Hi Clement
>
> There will be good amount of state in the SslEngineFactory's default
> implementation. Hence I feel we might anyway have a chaperon class to
> provide reconfigurable functionality and will have one more class to host
> the state/behavior of actual SSLContext/SSLEngine creation. While doing the
> internal rewrite (so far two times) both of the times I reached to the same
> conclusion.I feel if we leave the reconfigurations to the implementation -
> it will repeat the same pattern of having two classes to manage it - since
> most likely they will also have similar state information. Instead keep
> that reconfigurations in SslFactory as is today and just allow "plugin of
> creation of SSLEngine".
>
> One note I would like to make is: You are comparing this to MetricReporter
> but we have to keep in mind that SSL configuration is inherently more
> complex than a MetricReporters functionality. There are no JSSE equivalent
> documents needed to be written for MetricReporter for example. So what
> works best for simpler solutions may not work equally well for more complex
> scenarios.
>
>
> Thanks
> Maulin
>
>
> On Thu, Sep 19, 2019 at 8:36 PM Pellerin, Clement <
> clement_pelle...@ibi.com> wrote:
>
>> We will get there eventually but I need to address another point first.
>>
>> My goal is to do exactly what the "other extension points with
>> reconfigurable custom configs" are doing unless there is a good reason not
>> to. They provide a ready-made solution that will let us reuse code, avoid
>> pitfalls and show consistency.
>>
>> So far the roadblocks are
>> - the need to enforce mandatory compatibility checks for the keystores
>> and SSL handshake
>> - SslFactory is used in two channel builders.
>>
>> Both of these roadblocks can be overcome by moving the checks to a new
>> common base class of SslChannelBuilder and SaslChannelBuilder. This is easy
>> since both classes extend Object directly. The new base class is not a
>> public API so any implementation will do. The chaperon class SslFactory
>> disappears and the interface extends Reconfigurable.
>>
>> Does this proposal address all the reasons you had not to do exactly what
>> other extension points are doing?
>>
>> -Original Message-
>> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
>> Sent: Thursday, September 19, 2019 10:21 PM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
>> extensible
>>
>> Hi Clement
>>
>> So assuming there are two classes - SslFactory and SslEngineFactory like I
>> suggested in my detailed post before this, we can use
>> config.getConfiguredInstance() in SslFactory for SslEngineFactory class
>> configuration and then followed by init() method. I don't see a challenge
>> there. Can you please provide your input on my detailed post along with
>> this recent point I am making?
>>
>> Thanks
>> Maulin
>>
>> On Thu, Sep 19, 2019 at 5:04 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> wrote:
>>
>> > Hi Clement,
>> >
>> > Thanks for pointing to AbstractConfig. Now I understand what you were
>> > saying. I'll respond by tonight with more thoughts.
>> >
>> > Thanks
>> > Maulin
>> >
>> > On Thu, Sep 19, 2019 at 5:46 AM Pellerin, Clement <
>> > clement_pelle...@ibi.com> wrote:
>> >
>> >> I appreciate the effort you put into this.
>> >>
>> >> Lets do this in steps. You had a question on getConfiguredInstance().
>> >>
>> >> The method getConfiguredInstance(key, Class) implemented in
>> >> AbstractConfig is how the Metric

Re: [VOTE] KIP-517: Add consumer metrics to observe user poll behavior

[Maulin Vasavada]

+1 (non-binding). Thanks for the KIP.

On Thu, Sep 19, 2019 at 10:38 PM Manikumar 
wrote:

> +1 (binding), Thanks for the KIP.
>
> On Fri, Sep 20, 2019 at 12:41 AM Harsha Chintalapani 
> wrote:
>
> > +1 (binding). Thanks for the KIP.
> >
> > -Harsha
> >
> > On Wed, Sep 18, 2019 at 9:07 AM Mickael Maison  >
> > wrote:
> >
> > > +1 (non binding)
> > > Thanks for the KIP, I can see this being really useful!
> > >
> > > On Wed, Sep 18, 2019 at 4:40 PM Kevin Lu 
> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > Since we have a bit of support, I'd like to start the vote on
> KIP-517:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-517%3A+Add+consumer+metrics+to+observe+user+poll+behavior
> > > >
> > > > Thanks!
> > > >
> > > > Regards,
> > > > Kevin
> > >
> >
>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi Clement

There will be good amount of state in the SslEngineFactory's default
implementation. Hence I feel we might anyway have a chaperon class to
provide reconfigurable functionality and will have one more class to host
the state/behavior of actual SSLContext/SSLEngine creation. While doing the
internal rewrite (so far two times) both of the times I reached to the same
conclusion.I feel if we leave the reconfigurations to the implementation -
it will repeat the same pattern of having two classes to manage it - since
most likely they will also have similar state information. Instead keep
that reconfigurations in SslFactory as is today and just allow "plugin of
creation of SSLEngine".

One note I would like to make is: You are comparing this to MetricReporter
but we have to keep in mind that SSL configuration is inherently more
complex than a MetricReporters functionality. There are no JSSE equivalent
documents needed to be written for MetricReporter for example. So what
works best for simpler solutions may not work equally well for more complex
scenarios.


Thanks
Maulin


On Thu, Sep 19, 2019 at 8:36 PM Pellerin, Clement 
wrote:

> We will get there eventually but I need to address another point first.
>
> My goal is to do exactly what the "other extension points with
> reconfigurable custom configs" are doing unless there is a good reason not
> to. They provide a ready-made solution that will let us reuse code, avoid
> pitfalls and show consistency.
>
> So far the roadblocks are
> - the need to enforce mandatory compatibility checks for the keystores and
> SSL handshake
> - SslFactory is used in two channel builders.
>
> Both of these roadblocks can be overcome by moving the checks to a new
> common base class of SslChannelBuilder and SaslChannelBuilder. This is easy
> since both classes extend Object directly. The new base class is not a
> public API so any implementation will do. The chaperon class SslFactory
> disappears and the interface extends Reconfigurable.
>
> Does this proposal address all the reasons you had not to do exactly what
> other extension points are doing?
>
> -Original Message-
> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> Sent: Thursday, September 19, 2019 10:21 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> Hi Clement
>
> So assuming there are two classes - SslFactory and SslEngineFactory like I
> suggested in my detailed post before this, we can use
> config.getConfiguredInstance() in SslFactory for SslEngineFactory class
> configuration and then followed by init() method. I don't see a challenge
> there. Can you please provide your input on my detailed post along with
> this recent point I am making?
>
> Thanks
> Maulin
>
> On Thu, Sep 19, 2019 at 5:04 PM Maulin Vasavada  >
> wrote:
>
> > Hi Clement,
> >
> > Thanks for pointing to AbstractConfig. Now I understand what you were
> > saying. I'll respond by tonight with more thoughts.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Sep 19, 2019 at 5:46 AM Pellerin, Clement <
> > clement_pelle...@ibi.com> wrote:
> >
> >> I appreciate the effort you put into this.
> >>
> >> Lets do this in steps. You had a question on getConfiguredInstance().
> >>
> >> The method getConfiguredInstance(key, Class) implemented in
> >> AbstractConfig is how the MetricsReporter and other extension points are
> >> intantiated. Creating the extension point this way calls the default
> >> constructor which is good. Since the (Re)Configurable interface dictates
> >> the signature of the configure() method, that forces the addition of a
> new
> >> init(...) method to pass the other constructor arguments.
> >>
> >> Do we agree on that before we move on to other issues?
> >>
> >> -Original Message-
> >> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> >> Sent: Wednesday, September 18, 2019 4:37 PM
> >> To: dev@kafka.apache.org
> >> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> >> extensible
> >>
> >> Hi Clement
> >>
> >> Here are my thoughts based on my latest re-write attempt and learnings,
> >>
> >> 1. I think that it will be a great value to keep both classes separate -
> >> SslFactory and SslEngineFactory and having method
> reconfigurableConfigs()
> >> in the SslEngineFactory. Here is the reasoning,
> >>
> >> a.  It is kind of a Decorator pattern to me - even without named like
> one
> >> SslFactory is acting as a decor

Re: [ DISCUSS ] KIP-512:Adding headers to RecordMetaData

[Maulin Vasavada]

Hi all,

Can this KIP get to voting? I don't see any major concerns so far. It is
already adding ProducerRecord in Producer Interceptor which was one of the
main ask.

Thanks
Maulin

On Mon, Sep 16, 2019 at 4:09 PM Renuka M  wrote:

> Hi All,
>
> The motivation behind this KIP is have info about record in
> ProducerInterceptor.onAcknowledgement(..), so that we can link record to
> its metadata which is missing now. Initially we proposed to add headers to
> RecordMetadata, so that we can link record to its metadata.
> After thinking little bit more about it, we thought adding a record itself
> to ProducerInterceptor.onAcknowledgement(..) makes sense since we can have
> all data to link record to its metadata in Interceptors.
>
> *public interface ProducerInterceptor extends Configurable *{
> 
>
>
>
>
>
> *@Deprecated   public void onAcknowledgement(RecordMetadata metadata,
> Exception exception);   public default void
> onAcknowledgement(RecordMetadata metadata, Exception exception,
> ProducerRecord record){ onAcknowledgement(metadata, exception);
>   }*
> }
>
> we will not be adding new method to Callback which has same method
> signature, since in callback the record information is  already available
> to link.
>
>
> Please let us know your thoughts on this? If its makes sense, we will drop
> original plan of adding headers to metadata, instead we can add an
> overloaded onAcknowledgement(...) method to ProducerInterceptor with record
> itself.
> By providing above default implementation, we will have backward
> compatibility, at the same time we can have deprecation plan for existing
> method.
>
>
> Thanks
> Renuka M
>
>
> On Thu, Sep 12, 2019 at 12:22 PM Renuka M  wrote:
>
> > Bumping this in the hope I can get more feedback and/or votes.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3AAdding+headers+to+RecordMetaData
> >
> > Thanks,
> > Renuka M
> >
> > On Mon, Sep 9, 2019 at 2:41 PM Renuka M  wrote:
> >
> >>
> >> Hi All,
> >>
> >> Could you please take a look and let us know what you think on KIP-512
> --
> >> adding record headers to RecordMetaData.
> >> Since headers also provides metadata about the record, adding these
> >> to RecordMetaData, will allow to link record to its acknowledgement in
> >> Interceptors.
> >>
> >> The message tracing solutions like inca -
> >>
> https://medium.com/@NetflixTechBlog/inca-message-tracing-and-loss-detection-for-streaming-data-netflix-de4836fc38c9
> >> using producer callback to link records to its acknowledgement.
> >> If we have headers info as part of RecordMetaData, we can capture the
> >> record tracking in ProducerInterceptors itself.
> >>
> >> This is what we are trying to solve.
> >>
> >> Thanks
> >> Renuka M
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Aug 29, 2019 at 10:49 AM Renuka M 
> wrote:
> >>
> >>> Hi Colin,
> >>>
> >>> yes we agree but RecordMetadata in Interceptors and callbacks  will not
> >>> have headers which gives context on for which record this MetaData
> belongs
> >>> to. To fill that Gap, we are proposing these changes.
> >>>
> >>>
> >>> Thanks
> >>> Renuka M
> >>>
> >>> On Thu, Aug 29, 2019 at 10:20 AM Colin McCabe 
> >>> wrote:
> >>>
>  As Gwen commented earlier, the client already has the record that it
>  sent, including all the headers.
> 
>  >
>  > Future future = producer.send(myRecord, null);
>  > future.get();
>  > System.out.println("I sent myRecord with headers " +
>  myRecord.headers());
>  >
> 
>  best,
>  Colin
> 
> 
>  On Tue, Aug 27, 2019, at 17:06, Renuka M wrote:
>  > Hi  Gwen/Team
>  >
>  > Can you please review the KIP. Hope we have clarified the question
>  you have
>  > regarding proposal.
>  >
>  > Thanks
>  > Renuka M
>  >
>  > On Mon, Aug 26, 2019 at 3:35 PM Renuka M 
>  wrote:
>  >
>  > > Hi Eric,
>  > >
>  > > We thought about that but we didn't find the strong  enough reason
>  for
>  > > having record itself in Acknowledgement.
>  > > Headers are supposed to carry metadata and that is the reason
>  headers are
>  > > added to producer/consumer records.
>  > > Also we feel having headers information in record metadata is good
>  enough
>  > > to bridge the gap and link the record to its metadata.
>  > > Its simple change since we are not adding any new method
> signatures.
>  > > Adding new method signatures requires adoption and deprecation of
>  old ones
>  > > to reduce duplication.
>  > > If we get enough votes on adding new method signature, we are open
>  to add
>  > > it.
>  > >
>  > > Thanks
>  > > Renuka M
>  > >
>  > > On Mon, Aug 26, 2019 at 10:54 AM Eric Azama 
>  wrote:
>  > >
>  > >> Have you considered adding a new onAcknowledgement method to the
>  > >> ProducerInterceptor with the signature
>  onAcknowledgement(RecordMetadata
>  > >> metadata, 

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi Clement

So assuming there are two classes - SslFactory and SslEngineFactory like I
suggested in my detailed post before this, we can use
config.getConfiguredInstance() in SslFactory for SslEngineFactory class
configuration and then followed by init() method. I don't see a challenge
there. Can you please provide your input on my detailed post along with
this recent point I am making?

Thanks
Maulin

On Thu, Sep 19, 2019 at 5:04 PM Maulin Vasavada 
wrote:

> Hi Clement,
>
> Thanks for pointing to AbstractConfig. Now I understand what you were
> saying. I'll respond by tonight with more thoughts.
>
> Thanks
> Maulin
>
> On Thu, Sep 19, 2019 at 5:46 AM Pellerin, Clement <
> clement_pelle...@ibi.com> wrote:
>
>> I appreciate the effort you put into this.
>>
>> Lets do this in steps. You had a question on getConfiguredInstance().
>>
>> The method getConfiguredInstance(key, Class) implemented in
>> AbstractConfig is how the MetricsReporter and other extension points are
>> intantiated. Creating the extension point this way calls the default
>> constructor which is good. Since the (Re)Configurable interface dictates
>> the signature of the configure() method, that forces the addition of a new
>> init(...) method to pass the other constructor arguments.
>>
>> Do we agree on that before we move on to other issues?
>>
>> -Original Message-
>> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
>> Sent: Wednesday, September 18, 2019 4:37 PM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
>> extensible
>>
>> Hi Clement
>>
>> Here are my thoughts based on my latest re-write attempt and learnings,
>>
>> 1. I think that it will be a great value to keep both classes separate -
>> SslFactory and SslEngineFactory and having method reconfigurableConfigs()
>> in the SslEngineFactory. Here is the reasoning,
>>
>> a.  It is kind of a Decorator pattern to me - even without named like one
>> SslFactory is acting as a decorator/surrogate to the SslEngineFactory and
>> helping it get created and re-created as needed based on the
>> terms/conditions specified by SslEngineFactory (via
>> reconfigurableConfigs()
>> method)
>>
>> b. SslEngineFactory will be pluggable class. By keeping the SslFactory
>> reconfigurable with delegation of reconfigurableConfigs() to
>> SslEngineFactory it allows the implementation of SslEngineFactory to be
>> worry free of - How Kafka manages reconfigurations. The contract is -
>> Kafka's SslFactory will ask the implementation to provide which
>> configurations it is ready to be reconfigured for. Rest of the logic for
>> triggering and reconfiguring and validation is in SslFactory.
>>
>> c. The current validation in SslFactory about inter-broker-ssl handshake
>> AND verifying that certificate chain doesn't change via dynamic config
>> changes is rightly owned by SslFactory. We should not give flexibility to
>> SslEngineFactory to decide if they want that validation or not.
>>
>> d. If SslEngineFactory fails to be re-created with new dynamic config
>> changes the constructor will throw some exception and the SslFactory will
>> fail the validateReconfiguration() call resulting in no-change. Hence the
>> validation if the new config is right is still controlled by the
>> SslEngineFactory without necessarily having explicit validate method
>> (assuming if you had a point about - we should keep validation of changed
>> configs in the pluggable class)
>>
>>
>> 2. About the keystore validation in SslFactory - as I mentioned in above
>> points,
>>
>> a. I feel it is Kafka's policy that it wants to mandate that validation
>> regardless of the SslEngineFactory's implementation. I feel that
>> regardless
>> of customized implementation it is doing a 'logical' enforcement. I don't
>> see many cases where you will end up changing certificate chain (I can't
>> say the same about SANs entries though. see my below points). Hence that
>> validation is reasonable to be generally enforced for dynamic config
>> changes. If you change something violating that validation, you can avoid
>> making such changes via dynamic configuration and do a rolling restarts of
>> the boxes.
>>
>> b. If the implementation doesn't use keystore then automatically no
>> validation will happen. Hence I don't see any issue with
>> SslEngineFactory's
>> implementations not having requirement to use keystores.
>>
>> c. There could be an argument however about - what it validates current

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi Clement,

Thanks for pointing to AbstractConfig. Now I understand what you were
saying. I'll respond by tonight with more thoughts.

Thanks
Maulin

On Thu, Sep 19, 2019 at 5:46 AM Pellerin, Clement 
wrote:

> I appreciate the effort you put into this.
>
> Lets do this in steps. You had a question on getConfiguredInstance().
>
> The method getConfiguredInstance(key, Class) implemented in AbstractConfig
> is how the MetricsReporter and other extension points are intantiated.
> Creating the extension point this way calls the default constructor which
> is good. Since the (Re)Configurable interface dictates the signature of the
> configure() method, that forces the addition of a new init(...) method to
> pass the other constructor arguments.
>
> Do we agree on that before we move on to other issues?
>
> -Original Message-
> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> Sent: Wednesday, September 18, 2019 4:37 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> Hi Clement
>
> Here are my thoughts based on my latest re-write attempt and learnings,
>
> 1. I think that it will be a great value to keep both classes separate -
> SslFactory and SslEngineFactory and having method reconfigurableConfigs()
> in the SslEngineFactory. Here is the reasoning,
>
> a.  It is kind of a Decorator pattern to me - even without named like one
> SslFactory is acting as a decorator/surrogate to the SslEngineFactory and
> helping it get created and re-created as needed based on the
> terms/conditions specified by SslEngineFactory (via reconfigurableConfigs()
> method)
>
> b. SslEngineFactory will be pluggable class. By keeping the SslFactory
> reconfigurable with delegation of reconfigurableConfigs() to
> SslEngineFactory it allows the implementation of SslEngineFactory to be
> worry free of - How Kafka manages reconfigurations. The contract is -
> Kafka's SslFactory will ask the implementation to provide which
> configurations it is ready to be reconfigured for. Rest of the logic for
> triggering and reconfiguring and validation is in SslFactory.
>
> c. The current validation in SslFactory about inter-broker-ssl handshake
> AND verifying that certificate chain doesn't change via dynamic config
> changes is rightly owned by SslFactory. We should not give flexibility to
> SslEngineFactory to decide if they want that validation or not.
>
> d. If SslEngineFactory fails to be re-created with new dynamic config
> changes the constructor will throw some exception and the SslFactory will
> fail the validateReconfiguration() call resulting in no-change. Hence the
> validation if the new config is right is still controlled by the
> SslEngineFactory without necessarily having explicit validate method
> (assuming if you had a point about - we should keep validation of changed
> configs in the pluggable class)
>
>
> 2. About the keystore validation in SslFactory - as I mentioned in above
> points,
>
> a. I feel it is Kafka's policy that it wants to mandate that validation
> regardless of the SslEngineFactory's implementation. I feel that regardless
> of customized implementation it is doing a 'logical' enforcement. I don't
> see many cases where you will end up changing certificate chain (I can't
> say the same about SANs entries though. see my below points). Hence that
> validation is reasonable to be generally enforced for dynamic config
> changes. If you change something violating that validation, you can avoid
> making such changes via dynamic configuration and do a rolling restarts of
> the boxes.
>
> b. If the implementation doesn't use keystore then automatically no
> validation will happen. Hence I don't see any issue with SslEngineFactory's
> implementations not having requirement to use keystores.
>
> c. There could be an argument however about - what it validates currently
> and is there a scope of change. Example: It validates SANs entries and that
> to me is a challenge because I have had scenarios where I kept adding more
> VIPs in my certs SANs entries without really changing any certificate
> chain. The existing validation will fail that setup unnecessarily. Given
> that - there could be change in SslFactory but that doesn't still make that
> validation eligible to go to SslEngineFactory implementations.
>
>
> 3. I am still in two minds about your point on - not using existing SSL
> Reconfigurable configs to be used by SslFactory on top of
> SslEngineFactory's reconfigurable configs. The reason for that is-
>
> a. I agree with you on that we should not worry about existing SSL
> reconfigurable configs in new changed code for SslFactory. Why depend on
> something you really do

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

ctory to be involved
   at all."

I think I am missing something. If we just have SslEngineFactory
reconfigure itself it will generate new SSLContext and in-turn new
SSLEgnine but how will it communicate that to the ChannelBuilders? Don't
they have to refresh the reference to the SslEngineFactory via SslFactory's
reconfigure() method in order to pick up that change?


Thanks
Maulin











On Tue, Sep 17, 2019 at 4:49 AM Pellerin, Clement 
wrote:

> Good point about the two callers of SslFactory. We can move the SslEngine
> validation to a separate class and call it in both places. That SslEngine
> validation class would not be part of the public API and therefore we don't
> need to fuss about its API.
>
> -Original Message-
> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> Sent: Tuesday, September 17, 2019 2:28 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> Hi Clement/Rajini
>
> When I read your responses - I swing between both of your suggestions :) I
> see both of your points. Let me ponder little bit more and give me take in
> a day or so.
>
> I tend to agree with Clement in a sense that we need to define clear
> responsibilities of classes. Right now I feel it's not clear. Also, I tend
> to agree to both of you about keystore/truststore validation - the conflict
> I've to propose a clean agreeable solution to.
>
> One clarification to Clement is - there are two classes using SslFactory
> today - SslChannelBuilder and SaslChannelBuilder so we have to keep that in
> mind. However, once we have clear responsibilities of classes, that should
> automatically clear what goes where.
>
> Thanks
> Maulin
>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

iented design that requires the
> > reconfigurableConfigs() method but declares the interface to be
> > non-reconfigurable with just the Configurable interface.
> >
> > My use case removes all built-in SSL configs (except the interface class
> > name of course). SslFactory should not hardcode any SSL configs in the
> > reconfigurableConfigs. It should delegate to the interface instance for
> all
> > reconfigurableConfigs. In particular, it cannot assume there are
> keystores
> > and truststores to validate. These checks should be moved to
> > DefaultSslEngineFactory. We can then consider moving the SslEngine
> > validation from SslFactory to SslChannelBuilder. What would be left in
> > SslFactory that forces us to keep it instead of making it the
> > Reconfigurable extension point itself?
> >
> > I believe we don't need the sslContext() method since it is only used by
> a
> > junit.
> > If we support my use case, there is a good chance we don't need the
> > keystore() and truststore() method.
> >
> > I am still not comfortable with the fact that reconfigurableConfigs() are
> > not known until the SslEngineFactory implementation is created and that
> > happens only after configure() is called. Notice this goes away if
> > SslFactory is the extension point, which would explain why this might not
> > have been an issue with the other extension points exposing
> reconfigurable
> > custom configs.
> >
> > We must document if the configs sent to the extension point
> implementation
> > have been validated or not. I am pretty sure they are not since
> > config.getConfiguredInstance() passes the originals() and there is no
> > configurableConfigs() method (there is only the subset in
> > reconfigurableConfigs()). Non-validated configs might be of the wrong
> type,
> > be out of range, or missing since the default value is not applied. This
> is
> > a burden to the extension point developer and Kafka should provide
> > utilities for this.
> >
> > Can you confirm where the special case for the reconfiguration of
> > keystore/truststore is implemented? I am still trying to determine if it
> is
> > possible to trigger a reconfiguration when none of the known configs have
> > changed.
> >
> >
> > -Original Message-
> > From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
> > Sent: Monday, September 16, 2019 5:29 AM
> > To: dev
> > Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> > extensible
> >
> > Hi Maulin,
> >
> > Thanks for writing that up. I think we are getting close.
> >
> > As an example interface (which can be adjusted as necessary, but just
> > including something here to refer to), I think we would have an interface
> >  along these lines:
> >
> > public interface SslEngineFactory extends Configurable, Closeable {
> >
> > Set reconfigurableConfigs();
> > boolean shouldBeRebuilt(Map nextConfigs);
> >
> > SSLEngine createSslEngine(Mode mode, String peerHost, int
> > peerPort, String endpointIdentification);
> > SSLContext sslContext();
> > KeyStore keystore();
> > KeyStore truststore();
> > }
> >
> > A) *Creation:*
> >
> >- We typically tend to include configs in constructor for
> non-pluggable
> >classes, but use Configurable for pluggable classes. So Option 2 works
> >better
> >
> > B) *Returning reconfigurable configs: *
> >
> >- Custom SslEngineFactory implementations will only return what they
> >care about in their implementation of reconfigurableConfigs().
> >- SslChannelBuilder will delegate to SslFactory as you mentioned.
> >- SslFactory.reconfigurableConfigs() will return
> >SslConfigs.RECONFIGURABLE_CONFIGS plus SslEngineFactory.
> > reconfigurableConfigs(). So one day if we make endpoint validation
> >reconfigurable, it would all just work. We can easily find a different
> > way
> >of continuing to reconfigure SslFactory without config changes if we
> >    needed to since it is not a pluggable class.
> >
> > C) *Triggering reconfiguration:*
> >
> >- We continue to use AdminClient for reconfiguration in brokers with
> >validation. That goes through SslEngineFactory.shouldBeRebuilt() and
> >creates a new SslEngineFactory instance if needed.
> >
> > D) *Handling push notification kind of needs*
> >
> >- For brokers, we want SslFactory to manage reconfiguration since
> >configs may change. Also

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi Clement/Rajini

Based on what I get from my own understanding and your latest inputs, I'll
write down what my proposal is here and let us see if we can discuss on
that,

A) Creation

1. SslEngineFactory interface will not extend Configurable. Instead it will
require the implementation to have constructor with config map
2. Currently, the caller is co-ordinating the call sequences between
configure() and reconfigurableConfigs() in existing code base for
SslChannelBuilder, SslFactory, ChannelBuilders and SocketServer. Hence I am
not sure if we should get stuck arguing having a constructor vs having
configure() to initialize the object. Personally I don't have a preference.

B) Returning reconfigurable configs

1. SslEngineFactory interface will have reconfigurableConfigs() method
which will return Set with values for custom (or with existing)
config keys that it cares about.
Here I agree that all existing SSL configurations may not at all be
applicable to the pluggable engine. I don't see a point in keep referring
to existing SSL Configs. Of course, the default implementation of the
SslEngineFactory will return current SSL Configs since it relies on them to
initialize keys/certs.

2. SslFacotry's reconfigurableConfigs() will delegate the call to the
SslEngineFactory's reconfigurableConfigs(). The similar delegation will
need to be done by SslChannelBuilder/SaslChannelBuilder to SslFactory for
the same method (currently that are returning SslConfigs.
RECONFIGURABLE_CONFIGS)

C) Triggering reconfiguration

1. We can use AdminClient to trigger AlterConfig for any configs needed by
the SslEngineFactory's implementation. For the default implementation it
will follow the SSL Configs it uses today. For the custom implementation
everything will funnel via 'custom config' as it is dealt with in below
method,

https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/DynamicBrokerConfig.scala#L507

Btw, I don't see any check like if the reconfiguration needs to be
triggered at the below line for Listeners,
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/DynamicBrokerConfig.scala#L515

We do seem to have a check on Line#518 for other reconfigurables.

D) Handling push notification kind of needs

As noted by both of you, the reconfiguration will come via AdminClient
probably from the SslEngineFactory's implementation itself when it knows
the keys/certs changed and I need to ask Kafka to re-create myself.
However, that approach ONLY works for Brokers. With keys/cert rotations we
should address Client side challenge also. As you would have noticed in
KIP-486 motivation the deployment challenge on Broker and Client side both
makes it challenging to manage key/cert rotations etc. Hence I feel we
should consider option of method in the SslEngineFactory to take a
reconfigurable and call it when it knows it needs to be reconfigured.

Overall, I feel we should avoid any coupling with existing handling of SSL
keystore and truststore via the SSL Configs. When we are allowing
customization for the whole SSLEngine then we don't want it to rely on
existing configs to assume/necessarily-reuse any mechanism for loading
keystore and Truststore.

For MetricReporters I feel it is using custom configs like this except that
MetricReporter interface defines init() and extends Configurable. Please
clarify if I am missing something.

Thanks
Maulin















On Fri, Sep 13, 2019 at 12:15 PM Maulin Vasavada 
wrote:

> Hi Clement/Rajini
>
> I've gone through the code to understand how reconfigruation work
> currently. I sent both of you a note separately also. Let us reconvene next
> week and proceed.
>
> Thanks
> Maulin
>
> On Thu, Sep 12, 2019 at 12:25 PM Pellerin, Clement <
> clement_pelle...@ibi.com> wrote:
>
>> You are proposing a different design pattern for the SSL reconfigurable
>> extension point than the one already implemented for MetricsReporter. You
>> need a good reason to justify this decision. It is as if you consider
>> SslFactory the extension point. This is indeed something I proposed
>> multiple times and was always shut down.
>>
>> Going back to your latest proposal, notice you cannot call
>> reconfigurableConfigs() until configure() is called because you need to
>> instantiate SslEngineFactory() first. There is no way to enforce this in
>> the API.
>>
>> You did not react to my observation that ConfigDef does a better job
>> validating/casting configs based on the ConfigKey declarations compared to
>> ad hoc code you force people to write in their extension point
>> implementations.
>>
>> You suggest not to augment ConfigDef with custom configs, so that takes
>> care of the recursive dependency.
>> I just noticed reconfigurableConfigs() returns Set and that does
>> not force the creation of a ConfigKey for 

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

re is a config change. SslFactory will invoke
> MySslEngineFactory.shouldBeRebuilt(). This gives MySslEngineFactory the
> opportunity to provide a new SslEngine whenever there is an AdminClient
> request to alter configs, regardless of whether any config changed or not.
>
> Hope that helps,
>
> Rajini
>
> On Thu, Sep 12, 2019 at 5:26 PM Pellerin, Clement <
> clement_pelle...@ibi.com>
> wrote:
>
> > I'm confused. Can you launch a reconfiguration without a config change or
> > not?
> >
> > If I understand the test case correctly, the design pattern to implement
> a
> > reconfigurable extension point in Kafka is to implement the
> Reconfigurable
> > interface. This means SslEngineFactory would be created once and mutate
> > with reconfigure. There is no ConfigKey created for the custom config and
> > therefore there is no validation by ConfigDef.
> >
> > Optionally, to expose the built-in validation, it might be worthwhile to
> > consider making ConfigKey a public API and move an individual config
> parse
> > from ConfigDef to ConfigKey. It would be more object oriented anyway.
> >
> > One of the use case of custom configs in SslEngineFactory is to remove
> the
> > need to specify the keystore and truststore locations. The special
> handling
> > to detect changes in keystore/truststore should be pushed to
> > DefaultSslEngineFactory and all calls to reconfigure should reach the
> > SslEngineFactory instance. Am I missing something?
> >
> > -Original Message-
> > From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
> > Sent: Thursday, September 12, 2019 12:01 PM
> > To: dev
> > Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> > extensible
> >
> > Correction on my previous email. Custom implementations can use
> AlterConfig
> > request without a config change by including `ssl.keystore.location` or
> > `ssl.truststore.location` in their `reconfigurabkeConfigs()`. This will
> > trigger the same codepath as we use for keystore updates when the file
> has
> > changed.
> >
> > On Thu, Sep 12, 2019 at 4:43 PM Rajini Sivaram 
> > wrote:
> >
> > > Hi Clement,
> > >
> > > Kafka does special handling for keystore/truststore file changes when
> an
> > > AlterConfig request is processed, but that is not easy to extend to
> > custom
> > > configs. I was thinking we could just add a custom config to trigger
> > > reconfiguration. For example, a config like
> `my.custom.keystore.version`
> > that
> > > is incremented when the custom implementation detects a change in
> > keystore.
> > > And the custom implementation would invoke admin client to update
> > `my.custom.keystore.version`.
> > > Kafka would do the rest to reconfigure SslFactory. And the custom
> > > implementation can then create the new builder.
> > >
> > > For an example of custom config reconfiguration, this test may be
> useful:
> > >
> >
> https://github.com/apache/kafka/blob/d3559f628b2ccb23a9faf531796675376ac06abb/core/src/test/scala/integration/kafka/server/DynamicBrokerReconfigurationTest.scala#L792
> > >
> > >
> > >
> > > On Thu, Sep 12, 2019 at 3:24 PM Pellerin, Clement <
> > > clement_pelle...@ibi.com> wrote:
> > >
> > >> For the push notification, Rajini prefers if the trigger to
> reconfigure
> > >> comes from an admin client. He says the admin client can reconfigure
> > even
> > >> though none of the config values have changed. This allows your custom
> > impl
> > >> to discover other conditions that have changed, for example the
> > contents of
> > >> the keystore.
> > >>
> > >> @Rajini, can you point us to an existing example of a Kafka extension
> > >> point that implements reconfigurable custom configs. This way we could
> > >> study it and do the same thing. Consistency is good. It would be nice
> if
> > >> there was a KIP that describes that design pattern.
> > >>
> > >> -Original Message-
> > >> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> > >> Sent: Thursday, September 12, 2019 2:24 AM
> > >> To: dev@kafka.apache.org
> > >> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> > >> extensible
> > >>
> > >> Thanks Clement and Rajini. Let me digest what both of you said.
> Clearly
> > I
> > >> need to understand more about th

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

accept this weaker validation.
> >
> > It appears we are trying to invent the notion of a meta config. Anyway, I
> > think we have shown asking an instance of SslEngineFactory to contribute
> to
> > ConfigDef is way too late.
> >
> > For your push notification, would it be simpler to just let your
> > SslEngineFactory notice the change and make it effective the next time it
> > is called. SslFactory would not cache the SslEngine and always ask
> > SslEngineFactory for it. You don't even need an inner thread if
> > SslEngineFactory checks for a change when it is called. SslEngineFactory
> > would no longer be immutable, so maybe it is worth reconsidering how
> > reconfigure works for it.
> >
> > -Original Message-
> > From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> > Sent: Wednesday, September 11, 2019 3:29 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> > extensible
> >
> > Hi all,
> >
> > Since the "custom config" seems the main topic of interest let us talk
> > about it.
> >
> > 1. I want to confirm that I interpret the definition of 'custom config of
> > SslEngineFactory' the same way Clement is suggesting - "a config that
> does
> > not exist in Kafka but is specified by a custom implementation of
> > SslEngineFactory".  If there is a disagreement to that we have to bring
> it
> > up here sooner.
> >
> > 2. I've been thinking about it and I question why we are trying to make a
> > custom config a first class citizen in standard config?
> > The reasoning for that question is-
> > Kafka wants to delegate creation of SSLEngine to a class which is "not"
> > part of Kafka's distribution. Since the interface for SSLEngine creator
> > will be defined by the public method of createSSLEngine(), why would
> Kafka
> > care what does the implementation do other than fulfilling the contract
> of
> > creation of SSLEngine. The implementation can use any special configs
> i.e.
> > configs coming from input Map OR configs defined in a new file only known
> > to itself. Making the configs part of the interface contract in any way
> is
> > not necessary. This way we keep it simple and straightforward.
> >
> > 3. Now, 2nd point raises a question - if we follow that suggestion - how
> > can we ever re-create the SSLEngineFactory object and allow new object to
> > be created when something changes in the implementation. That is a valid
> > question. If you noticed in the KIP section titled "Other challenge" - we
> > do have scenario where the SslEngineFactory implementation ONLY knows
> that
> > something changed - example: keystore got updated by a local daemon
> process
> > only known to the specific implementation. This means we have a need of
> > "push notification" from the SslEngineFactory's implementation to the
> > SslFactory actually. I feel if we build the "push notification" via
> adding
> > a method in the interface as "public void
> > registerReconfigurableListener(Reconfigurable r)" and make SslFactory
> > register itself with the SslEngineFactory's impl class, we can trigger
> the
> > reconfiguration of SslEngineFactory implementation based on its own terms
> > and conditions without getting into making custom configs complicated.
> >
> > I am just thinking out loud here and expressing my opinion not to avoid
> > addressing custom configs BUT what I genuinely believe might be a better
> > approach.
> >
> > Thanks
> > Maulin
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Sep 10, 2019 at 9:06 PM Pellerin, Clement <
> > clement_pelle...@ibi.com>
> > wrote:
> >
> > > Regarding what I labeled the simplest solution below, SslConfigs could
> > > instantiate the custom interface only if the yet to be validated
> configs
> > > were passed in to the call to get the list of known SSL configs.
> > >
> > > -Original Message-
> > > From: Pellerin, Clement
> > > Sent: Tuesday, September 10, 2019 11:36 AM
> > > To: dev@kafka.apache.org
> > > Subject: [EXTERNAL]RE: [DISCUSS] KIP-519: Make SSL context/engine
> > > configuration extensible
> > >
> > > Another solution could be a new standard ssl config that holds a list
> of
> > > extra custom configs to accept.
>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all,

Since the "custom config" seems the main topic of interest let us talk
about it.

1. I want to confirm that I interpret the definition of 'custom config of
SslEngineFactory' the same way Clement is suggesting - "a config that does
not exist in Kafka but is specified by a custom implementation of
SslEngineFactory".  If there is a disagreement to that we have to bring it
up here sooner.

2. I've been thinking about it and I question why we are trying to make a
custom config a first class citizen in standard config?
The reasoning for that question is-
Kafka wants to delegate creation of SSLEngine to a class which is "not"
part of Kafka's distribution. Since the interface for SSLEngine creator
will be defined by the public method of createSSLEngine(), why would Kafka
care what does the implementation do other than fulfilling the contract of
creation of SSLEngine. The implementation can use any special configs i.e.
configs coming from input Map OR configs defined in a new file only known
to itself. Making the configs part of the interface contract in any way is
not necessary. This way we keep it simple and straightforward.

3. Now, 2nd point raises a question - if we follow that suggestion - how
can we ever re-create the SSLEngineFactory object and allow new object to
be created when something changes in the implementation. That is a valid
question. If you noticed in the KIP section titled "Other challenge" - we
do have scenario where the SslEngineFactory implementation ONLY knows that
something changed - example: keystore got updated by a local daemon process
only known to the specific implementation. This means we have a need of
"push notification" from the SslEngineFactory's implementation to the
SslFactory actually. I feel if we build the "push notification" via adding
a method in the interface as "public void
registerReconfigurableListener(Reconfigurable r)" and make SslFactory
register itself with the SslEngineFactory's impl class, we can trigger the
reconfiguration of SslEngineFactory implementation based on its own terms
and conditions without getting into making custom configs complicated.

I am just thinking out loud here and expressing my opinion not to avoid
addressing custom configs BUT what I genuinely believe might be a better
approach.

Thanks
Maulin



















On Tue, Sep 10, 2019 at 9:06 PM Pellerin, Clement 
wrote:

> Regarding what I labeled the simplest solution below, SslConfigs could
> instantiate the custom interface only if the yet to be validated configs
> were passed in to the call to get the list of known SSL configs.
>
> -Original Message-
> From: Pellerin, Clement
> Sent: Tuesday, September 10, 2019 11:36 AM
> To: dev@kafka.apache.org
> Subject: [EXTERNAL]RE: [DISCUSS] KIP-519: Make SSL context/engine
> configuration extensible
>
> Another solution could be a new standard ssl config that holds a list of
> extra custom configs to accept.
> Using a custom SslEngineFactory with custom configs would require setting
> two configs, one for the class name and another for the list of custom
> configs.
> In SslConfigs, we see that declaring a single config takes 5 values, so
> I'm not sure how it would work exactly.
>
> We could also declare a new interface to return the list of custom ssl
> configs and the extra standard ssl config I'm proposing holds the name of
> the implementation class instead. The reason a different interface is
> needed is because it would be instantiated by SslConfigs, not SslFactory.
> This seems the simplest solution.
>
> Anyway, the point of this exercise is to prove an acceptable solution for
> custom configs is not affecting the public API in KIP-519.
>
>
> -Original Message-
> From: Pellerin, Clement
> Sent: Tuesday, September 10, 2019 9:35 AM
> To: dev@kafka.apache.org
> Subject: [EXTERNAL]RE: [DISCUSS] KIP-519: Make SSL context/engine
> configuration extensible
>
> Custom config is a term I invented to mean a config that does not exist in
> Kafka but is specified by a custom implementation of SslEngineFactory.
> The problem with custom configs (as I remember it) is that the list of
> configs is static in SslConfigs and config names are checked before
> SslFactory is created.
> ==> You must solve this problem because that is what killed KIP-383 and
> therefore is the sole reason why KIP-519 exists.
> ==> Your KIP does not have to implement the solution since it can be done
> in a future KIP, but your KIP must be compatible with the solution found.
> ==> A method to return the list of configs would help. This cannot be a
> static method in the interface since it cannot be overridden.
> ==> You could require a static method in the implementation class by
> convention, just like the constructor you require.
>
> This email did not originate from inside Information Builders.
>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi Clement/Rajini/Colin

Please review our latest updates on the KIP and let me know your thoughts.

Clement, please let me know if my understanding about the "custom configs"
is correct based on what I wrote in the KIP.

Thanks
Maulin

On Mon, Sep 9, 2019 at 3:28 PM Maulin Vasavada 
wrote:

> Hi all
>
> Based on longer discussion on another KIP-486 we are opening this KIP-519 (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952
> ).
>
> Please help us review this and provide your suggestions.
>
> Thanks
> Maulin
>

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Thanks Clement. I"ll start accommodating your suggestions and addressing
comments on the KIP-519.  However, can we please start discussing on the
KIP-519 discussion thread?

On Mon, Sep 9, 2019 at 9:23 PM Pellerin, Clement 
wrote:

> Please specify in the KIP if the new config is reconfigurable or not.
>
> -Original Message-
> From: Pellerin, Clement
> Sent: Monday, September 9, 2019 8:28 PM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
>
> This is a good start.
>
> Please document the constants for the config you are creating.
> You will notice you need to make the name of your default implementation
> public.
>
> How do the configs get into your factory instance? Is it through the
> constructor or you will add method arguments?
> Please define your constructor in the KIP even if it is the default
> constructor.
> You will also need to document the constructor in the class comment of
> your interface.
>
> In your implementation, I suggest to default the config value and always
> call the constructor dynamically even for the default implementation.
>
> I don't understand why this is a factory for both SSLContext and SslEngine.
> The name of the factory depends on this choice.
>
> I am not a fan of the method name shouldBeRebuilt()
> This was OK in a private implementation before but this will become a
> public API.
> What would be a better name that fits with the naming conventions in
> SslFactory?
>
> Please mention how your interface could handle custom configs.
> I would assume you plan to merge the SslFactory reconfigurableConfigs with
> those of the SslEngineFactory.
> How can you be sure SslFactory always creates its SslEngineFactory before
> any calls to SslFactory.reconfigurableConfigs()?
>
> Your Rejected Alternatives need to be beefed up.
> In particular, why is SslFactory not the pluggable interface directly?
> Hint: because we want to reuse all the complex validation code and make it
> mandatory.
> This is also the place to argue against KIP-383. Hint: because it does not
> handle reconfiguration in the presence of custom configs.
>
> When I wrote KIP-383, I felt I needed a prototype before I could solidify
> the proposal.
> That's part of the reason why there was never a third iteration.
>
> -Original Message-
> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> Sent: Monday, September 9, 2019 6:41 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
>
> Hi all
>
> I created a KIP-519 for pluggability of SSLContext/Engine object. Looking
> forward for a great discussion and feedback on the same. I'll keep KIP-486
> in discussion state until we reach to some good conclusion on how to allow
> custom key/trust stores after KIP-519 work is done. Based on that, we will
> update the KIP-486 appropriately.
>
> Thanks
> Maulin
>
>
> This email did not originate from inside Information Builders.
>

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi all

I created a KIP-519 for pluggability of SSLContext/Engine object. Looking
forward for a great discussion and feedback on the same. I'll keep KIP-486
in discussion state until we reach to some good conclusion on how to allow
custom key/trust stores after KIP-519 work is done. Based on that, we will
update the KIP-486 appropriately.

Thanks
Maulin

On Sun, Sep 8, 2019 at 11:08 PM Maulin Vasavada 
wrote:

> Thank you Clement and Colin for shading light on the process for me. I
> will probably start working on a new kip for pluggable SslEngine.
>
> Side note: The pull request is from my fork to my fork to make it easier
> to see what changes I am making. I have no intention to getting it reviewed
> as a pull request.
>
> Okay, let me start with a new KIP and we will go from there. I'll keep
> this KIP-486 since I feel ultimately we need that kip.
>
> Thanks
> Maulin
>
> On Fri, Sep 6, 2019 at 2:52 PM Colin McCabe  wrote:
>
>> Sorry, I meant to write "removing an API typically requires a new major
>> release of Kafka".  Deprecating an API can be done in a minor release.
>>
>> regards,
>> Colin
>>
>>
>> On Fri, Sep 6, 2019, at 14:49, Colin McCabe wrote:
>> > Hi Maulin,
>> >
>> > Clement brought up a good point, which is that we should have agreement
>> > on the overall design before we start looking at PRs.  In Kafka, that
>> > means starting a discussion on a KIP, and then getting that KIP voted
>> > in.
>> >
>> > There is more information about the KIP process here:
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> > Feel free to ask more detailed questions about the process on the
>> > mailing list as well :)
>> >
>> > It sounds like Clement is proposing that you create a new KIP rather
>> > than reusing KIP-383.  So you should probably either repurpose KIP-486
>> > for this, or choose a new number.  Both options seem reasonable here.
>> > As far as I can see, once we implement a pluggable SslEngineBuilder,
>> > KIP-383 would no longer be needed, and so we would put that one into
>> > "rejected KIPs."
>> >
>> > In general, coming up with good APIs can be harder than it might seem
>> > at first.  Once we add an API, we have to assume that we're going to
>> > support it forever.  Deprecating an API can take a very long time, and
>> > typically requires a new major release of Kafka.  Therefore, sometimes
>> > it's worth making users go through a little bit of copying and pasting
>> > in order to avoid creating an API that would constrain the project in
>> > the future.  Hopefully, we can come up with something good here that
>> > will be useful to everyone.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Fri, Sep 6, 2019, at 07:48, Pellerin, Clement wrote:
>> > > This is the way I see it, which is in no way authoritative.
>> > >
>> > > First you have to find a complete solution, then you need to document
>> > > it in a KIP and that KIP needs to pass a vote. Any votes before the
>> KIP
>> > > vote starts is meaningless.
>> > >
>> > > As for the ownership and authorship of the KIPs, I don't plan to work
>> > > on this, so KIP-383 is better left the way it is. I would prefer if
>> you
>> > > would update your KIP or maybe create a new one. We can mark KIP-383
>> as
>> > > obsolete after your KIP passes its vote.
>> > >
>> > > -Original Message-
>> > > From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
>> > > Sent: Friday, September 6, 2019 2:36 AM
>> > > To: dev@kafka.apache.org
>> > > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
>> TrustStore
>> > >
>> > > Hi all
>> > >
>> > > Any input on my last question about the process?
>> > >
>> > > Also, I have updated the code based on the feedback so far at:
>> > > https://github.com/maulin-vasavada/kafka/pull/2/files. Still have to
>> figure
>> > > out how to plug keys/certs loading while using
>> DefaultSslEngineFactory's
>> > > implementation (still need to make it final).
>> > >
>> > > Thanks
>> > > Maulin
>> > >
>> > >
>> > > On Thu, Sep 5, 2019 at 4:34 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> > > wrote:
>> > >
>> > >

[jira] [Created] (KAFKA-8890) KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada (Jira)]

Maulin Vasavada created KAFKA-8890:
--

 Summary: KIP-519: Make SSL context/engine configuration extensible
 Key: KAFKA-8890
 URL: https://issues.apache.org/jira/browse/KAFKA-8890
 Project: Kafka
  Issue Type: New Feature
  Components: security
Reporter: Maulin Vasavada


This is to track changes for KIP-519: Make SSL context/engine configuration 
extensible 
([https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952])



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Hi all

Based on longer discussion on another KIP-486 we are opening this KIP-519 (
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952).

Please help us review this and provide your suggestions.

Thanks
Maulin

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Thank you Clement and Colin for shading light on the process for me. I will
probably start working on a new kip for pluggable SslEngine.

Side note: The pull request is from my fork to my fork to make it easier to
see what changes I am making. I have no intention to getting it reviewed as
a pull request.

Okay, let me start with a new KIP and we will go from there. I'll keep this
KIP-486 since I feel ultimately we need that kip.

Thanks
Maulin

On Fri, Sep 6, 2019 at 2:52 PM Colin McCabe  wrote:

> Sorry, I meant to write "removing an API typically requires a new major
> release of Kafka".  Deprecating an API can be done in a minor release.
>
> regards,
> Colin
>
>
> On Fri, Sep 6, 2019, at 14:49, Colin McCabe wrote:
> > Hi Maulin,
> >
> > Clement brought up a good point, which is that we should have agreement
> > on the overall design before we start looking at PRs.  In Kafka, that
> > means starting a discussion on a KIP, and then getting that KIP voted
> > in.
> >
> > There is more information about the KIP process here:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > Feel free to ask more detailed questions about the process on the
> > mailing list as well :)
> >
> > It sounds like Clement is proposing that you create a new KIP rather
> > than reusing KIP-383.  So you should probably either repurpose KIP-486
> > for this, or choose a new number.  Both options seem reasonable here.
> > As far as I can see, once we implement a pluggable SslEngineBuilder,
> > KIP-383 would no longer be needed, and so we would put that one into
> > "rejected KIPs."
> >
> > In general, coming up with good APIs can be harder than it might seem
> > at first.  Once we add an API, we have to assume that we're going to
> > support it forever.  Deprecating an API can take a very long time, and
> > typically requires a new major release of Kafka.  Therefore, sometimes
> > it's worth making users go through a little bit of copying and pasting
> > in order to avoid creating an API that would constrain the project in
> > the future.  Hopefully, we can come up with something good here that
> > will be useful to everyone.
> >
> > best,
> > Colin
> >
> >
> > On Fri, Sep 6, 2019, at 07:48, Pellerin, Clement wrote:
> > > This is the way I see it, which is in no way authoritative.
> > >
> > > First you have to find a complete solution, then you need to document
> > > it in a KIP and that KIP needs to pass a vote. Any votes before the
> KIP
> > > vote starts is meaningless.
> > >
> > > As for the ownership and authorship of the KIPs, I don't plan to work
> > > on this, so KIP-383 is better left the way it is. I would prefer if
> you
> > > would update your KIP or maybe create a new one. We can mark KIP-383
> as
> > > obsolete after your KIP passes its vote.
> > >
> > > -Original Message-
> > > From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> > > Sent: Friday, September 6, 2019 2:36 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
> > >
> > > Hi all
> > >
> > > Any input on my last question about the process?
> > >
> > > Also, I have updated the code based on the feedback so far at:
> > > https://github.com/maulin-vasavada/kafka/pull/2/files. Still have to
> figure
> > > out how to plug keys/certs loading while using
> DefaultSslEngineFactory's
> > > implementation (still need to make it final).
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > > On Thu, Sep 5, 2019 at 4:34 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > > wrote:
> > >
> > > > Hi all
> > > >
> > > > It seems we are in some sort of agreement so far apart from code
> > > > review/comments. However, I have a fundamental question - going
> forward how
> > > > this work from the process standpoint? What do we do with this
> KIP-486 vs
> > > > KIP-383? I feel that both needs to come together since in order to
> make
> > > > Pluggable key/trust store on top of making SslEngineBuilder
> pluggable we
> > > > will need changes suggested by KIP-486 with some differences to the
> > > > original proposal. It would great if someone can help us clarify the
> next
> > > > steps.
> > > >
> > > > Thanks
> > > > Maulin
> > > &

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi all

Any input on my last question about the process?

Also, I have updated the code based on the feedback so far at:
https://github.com/maulin-vasavada/kafka/pull/2/files. Still have to figure
out how to plug keys/certs loading while using DefaultSslEngineFactory's
implementation (still need to make it final).

Thanks
Maulin


On Thu, Sep 5, 2019 at 4:34 PM Maulin Vasavada 
wrote:

> Hi all
>
> It seems we are in some sort of agreement so far apart from code
> review/comments. However, I have a fundamental question - going forward how
> this work from the process standpoint? What do we do with this KIP-486 vs
> KIP-383? I feel that both needs to come together since in order to make
> Pluggable key/trust store on top of making SslEngineBuilder pluggable we
> will need changes suggested by KIP-486 with some differences to the
> original proposal. It would great if someone can help us clarify the next
> steps.
>
> Thanks
> Maulin
>
> On Wed, Sep 4, 2019 at 1:54 PM Maulin Vasavada 
> wrote:
>
>> Do you guys think it would be easier if you can provide comments on
>> GitHub and we can continue there and summarize the conclusion here?
>>
>> We should not lose addressing any comments.
>>
>> On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement <
>> clement_pelle...@ibi.com> wrote:
>>
>>> The proposed interface does not look like the Builder pattern I am used
>>> to.
>>> Should SslEngineBuilder be called SslEngineFactory instead?
>>>
>>> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
>>> > I would expect SslEngineBuilder interface to look something like this,
>>> > perhaps with some tweaking:
>>> >
>>> > public interface SslEngineBuilder extends Configurable, Closeable {
>>> >
>>> > Set reconfigurableConfigs();
>>> >
>>> > boolean shouldBeRebuilt(Map nextConfigs);
>>> >
>>> > SSLEngine createSslEngine(Mode mode, String peerHost, int
>>> > peerPort, String endpointIdentification);
>>> >
>>> > }
>>>
>>

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi all

It seems we are in some sort of agreement so far apart from code
review/comments. However, I have a fundamental question - going forward how
this work from the process standpoint? What do we do with this KIP-486 vs
KIP-383? I feel that both needs to come together since in order to make
Pluggable key/trust store on top of making SslEngineBuilder pluggable we
will need changes suggested by KIP-486 with some differences to the
original proposal. It would great if someone can help us clarify the next
steps.

Thanks
Maulin

On Wed, Sep 4, 2019 at 1:54 PM Maulin Vasavada 
wrote:

> Do you guys think it would be easier if you can provide comments on GitHub
> and we can continue there and summarize the conclusion here?
>
> We should not lose addressing any comments.
>
> On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement <
> clement_pelle...@ibi.com> wrote:
>
>> The proposed interface does not look like the Builder pattern I am used
>> to.
>> Should SslEngineBuilder be called SslEngineFactory instead?
>>
>> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
>> > I would expect SslEngineBuilder interface to look something like this,
>> > perhaps with some tweaking:
>> >
>> > public interface SslEngineBuilder extends Configurable, Closeable {
>> >
>> > Set reconfigurableConfigs();
>> >
>> > boolean shouldBeRebuilt(Map nextConfigs);
>> >
>> > SSLEngine createSslEngine(Mode mode, String peerHost, int
>> > peerPort, String endpointIdentification);
>> >
>> > }
>>
>

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Do you guys think it would be easier if you can provide comments on GitHub
and we can continue there and summarize the conclusion here?

We should not lose addressing any comments.

On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement 
wrote:

> The proposed interface does not look like the Builder pattern I am used to.
> Should SslEngineBuilder be called SslEngineFactory instead?
>
> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> > I would expect SslEngineBuilder interface to look something like this,
> > perhaps with some tweaking:
> >
> > public interface SslEngineBuilder extends Configurable, Closeable {
> >
> > Set reconfigurableConfigs();
> >
> > boolean shouldBeRebuilt(Map nextConfigs);
> >
> > SSLEngine createSslEngine(Mode mode, String peerHost, int
> > peerPort, String endpointIdentification);
> >
> > }
>

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Colin

Thanks for your comments. I agree with most of them. This is not a
pull-request ready code yet :)

If we want to make DefaultSslEngineBuilder final then what do you propose
to address our requirements to be able to plugin custom way for loading
keys/certs? (The original challenge we have documented with this KIP). What
I gather from your comment on that is - we will have to standardize those
requirements as public APIs like interfaces documented in the KIP-486 for
KeyStoreLoader/TrustStoreLoader, correct? We have those pluggable APIs to
provide custom source for keys/certs and have SslEngineBuilder Interface
somehow take those?





On Wed, Sep 4, 2019 at 10:58 AM Colin McCabe  wrote:

> On Tue, Sep 3, 2019, at 22:56, Maulin Vasavada wrote:
> > Hi all
> >
> > Please check
> >
> https://github.com/maulin-vasavada/kafka/commit/44f86395b1ba3fe4bd87de89029d72da77995ff8
> >
> >
> > This is just the first cut obviously. There are few call outs I would
> like
> > to make,
> >
> > 1. So far I kept the old SslEngineBuilder hence I had to name the
> interface
> > with "I" (that can change later)
>
> Hi Maulin,
>
> Thanks for working on this.
>
> We don't use Hungarian notation in Kafka.  The interface should probably
> just be SslEngineBuilder.  The default implementation can be
> DefaultSslEngineBuilder.
>
> >
> > 2. I did not yet add the creation of SslEngineBuilder via loading the
> > configuration like 'ssl.engine.builder.class'. Hence you see direct
> > creation of DefaultSslEngineBuilder class
> >
> > 3. Due to validation logic in the current SslFactory I had to add more
> > methods in ISslEngineBuilder interface (like keystore(), truststore()
> etc).
> > Due to other classes like EchoServer depending upon needing SSLContext, I
> > had to add getSSLContext() also in the interface.
>
> Hmm.  I don't think we want to expose this stuff.  EchoServer is just used
> for testing, so it can cast the SslEngineBuilder to DefaultEngineBuilder
> (the only one that it will use during JUnit tests) and get what it needs
> that way.
>
> >
> > 4. With these changes and with existing old SslEngineBuilder, the
> > clients/core projects builds with tests successfully but I didn't add any
> > additional tests yet
> >
> > 5. I wanted to have DefaultSslEngineBuilder in such a way that if
> somebody
> > wants to implement custom SslEngineBuilder they can extend and override
> > only key required methods without repeating any logic.
>
> No, DefaultSslEngineBuilder should be final.  We should not allow people
> to extend the default engine builder, since then it becomes a public API.
> If there are utility functions that we feel would be useful to everyone, we
> can spell those out explicitly and standardize them as public APIs that
> can't be changed.
>
> >
> > 6. For reconfigurable interface I kept the way suggested by Rajini -
> > meaning SslFactory really is reconfigurable BUT it relies on the
> > ISslEngineBuilder to define the reconfigurable options. This means that
> > ISslEngineBuilder dictates based on which reconfigurable params the
> > SslFactory should try to reconfigure the SSLEngine.
>
> Each SslEngineBuilder should define its own set of reconfigurable
> configurations.  We don't know ahead of time what they will need.  We want
> to be flexible.  People might want to fetch certificates from a central
> location via HTTPS, for example.  Or maybe they want to use a native
> library of some kind.  And so on.
>
> best,
> Colin
>
> >
> > With this - open to all the suggestions and further improvements.
> >
> > Thanks
> > Maulin
> >
> >
> > On Tue, Sep 3, 2019 at 10:07 AM Colin McCabe  wrote:
> >
> > > On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> > > > I would expect SslEngineBuilder interface to look something like
> this,
> > > > perhaps with some tweaking:
> > > >
> > > > public interface SslEngineBuilder extends Configurable, Closeable {
> > > >
> > > > Set reconfigurableConfigs();
> > > >
> > > > boolean shouldBeRebuilt(Map nextConfigs);
> > > >
> > > > SSLEngine createSslEngine(Mode mode, String peerHost, int
> > > > peerPort, String endpointIdentification);
> > > >
> > > > }
> > > >
> > > > The existing SslEngineBuilder class would be renamed and will
> implement
> > > > this interface. Loading of keystore/truststore will be in
> > > SslEngineBuilder
> > > > as it is now.  The

Re: [VOTE] KIP-512:Adding headers to RecordMetaData

[Maulin Vasavada]

+1 (non-binding)

On Tue, Sep 3, 2019 at 3:38 PM Renuka M  wrote:

> Hi All,
>
> After good discussion for KIP-512
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3AAdding+headers+to+RecordMetaData
> ,
> am starting thread for voting.
>
> Thanks
> Renuka M
>

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi all

Please check
https://github.com/maulin-vasavada/kafka/commit/44f86395b1ba3fe4bd87de89029d72da77995ff8


This is just the first cut obviously. There are few call outs I would like
to make,

1. So far I kept the old SslEngineBuilder hence I had to name the interface
with "I" (that can change later)

2. I did not yet add the creation of SslEngineBuilder via loading the
configuration like 'ssl.engine.builder.class'. Hence you see direct
creation of DefaultSslEngineBuilder class

3. Due to validation logic in the current SslFactory I had to add more
methods in ISslEngineBuilder interface (like keystore(), truststore() etc).
Due to other classes like EchoServer depending upon needing SSLContext, I
had to add getSSLContext() also in the interface.

4. With these changes and with existing old SslEngineBuilder, the
clients/core projects builds with tests successfully but I didn't add any
additional tests yet

5. I wanted to have DefaultSslEngineBuilder in such a way that if somebody
wants to implement custom SslEngineBuilder they can extend and override
only key required methods without repeating any logic.

6. For reconfigurable interface I kept the way suggested by Rajini -
meaning SslFactory really is reconfigurable BUT it relies on the
ISslEngineBuilder to define the reconfigurable options. This means that
ISslEngineBuilder dictates based on which reconfigurable params the
SslFactory should try to reconfigure the SSLEngine.

With this - open to all the suggestions and further improvements.

Thanks
Maulin


On Tue, Sep 3, 2019 at 10:07 AM Colin McCabe  wrote:

> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> > I would expect SslEngineBuilder interface to look something like this,
> > perhaps with some tweaking:
> >
> > public interface SslEngineBuilder extends Configurable, Closeable {
> >
> > Set reconfigurableConfigs();
> >
> > boolean shouldBeRebuilt(Map nextConfigs);
> >
> > SSLEngine createSslEngine(Mode mode, String peerHost, int
> > peerPort, String endpointIdentification);
> >
> > }
> >
> > The existing SslEngineBuilder class would be renamed and will implement
> > this interface. Loading of keystore/truststore will be in
> SslEngineBuilder
> > as it is now.  The method `shouldBeRebuilt()` will validate configs
> during
> > reconfiguration and decide if reconfiguration is required because
> keystore
> > or truststore changed. SslFactory.reconfigurableConfigs() will return
> > SslEngineBuilder.reconfigurableConfigs() as well including any custom
> > configs of SslEngineBuilder, so no other changes will be required when we
> > eventually support custom SSL configs.
> >
> > We don't want to make SslFactory the pluggable class since that contains
> > validation logic for SSL engines. Everything that we want to customise is
> > contained in SslEngineBuilder. Basically custom SslEngineBuilder will
> > validate custom configs during reconfiguration and create SSLEngine.
> > SslFactory will continue to perform validation of SSLEngines and this
> will
> > not be customizable. SslEngineBuilder will not be reconfigurable, instead
> > we create a new builder as we do now to avoid having to deal with
> > thread-safety and atomicity of updates. We could consider using a public
> > Reconfigurable interface as the pluggable interface for consistency, but
> I
> > think we would still want to create a new Builder on reconfiguration and
> > retain non-pluggable SSL engine validation in SslFactory.
>
> +1
>
> C.
>
> >
> >
> > On Fri, Aug 30, 2019 at 10:21 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > wrote:
> >
> > > Looking at SslFactory and SslEngineBuilder I feel the responsibilities
> are
> > > not clear. Both has public method for createSSLEngine for example. I
> feel
> > > the SslEngineBuilder was created to move out lot of code but it is not
> > > necessarily a public class (e.g. I don't think anybody calling
> > > SslEngineBuilder separately without SslFactory in between). I am
> currently
> > > inclined toward what Celement is suggesting - having pluggable
> SslFactory.
> > >
> > > Let me do this - let me refactor SslFactory and SslEngineBuilder and
> review
> > > what I can come up with you guys. Let us see if we can address all the
> > > objections raised previously for KIP-383's iterations. I'll need
> sometime
> > > though. Let me try to do it by next of next week.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Fri, Aug 30, 2019 at 12:25 PM Pellerin, Clement <
> > > clement_pelle...@ibi.com>
> > > wrote:
>

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Looking at SslFactory and SslEngineBuilder I feel the responsibilities are
not clear. Both has public method for createSSLEngine for example. I feel
the SslEngineBuilder was created to move out lot of code but it is not
necessarily a public class (e.g. I don't think anybody calling
SslEngineBuilder separately without SslFactory in between). I am currently
inclined toward what Celement is suggesting - having pluggable SslFactory.

Let me do this - let me refactor SslFactory and SslEngineBuilder and review
what I can come up with you guys. Let us see if we can address all the
objections raised previously for KIP-383's iterations. I'll need sometime
though. Let me try to do it by next of next week.

Thanks
Maulin

On Fri, Aug 30, 2019 at 12:25 PM Pellerin, Clement 
wrote:

> What is your solution to the objection that killed the second iteration of
> KIP-383?
> Mainly, how do you support validation of reconfiguration requests that
> involve new custom properties implemented by the pluggable factory?
> Custom properties do not exist yet, but that is very legitimate thing to
> design for the future.
>
> That's why I favor a pluggable SslFactory instead of an SslEngineBuilder
> factory.
>
> -----Original Message-
> From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com]
> Sent: Friday, August 30, 2019 3:07 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
>
> +1 for making SslEngineBuilder configurable upon more thoughts.
>
> However, in the abstraction and default implementation we should make sure
> when we do have a requirement to plugin custom key/trust store people don't
> have to write lot more code which may not be related to it.
>
> Having said that, does this mean, we resurrect KIP-383 and update it with
> latest context and go from there?
>
> We are willing to take up that work for making it configurable.
>
> Thanks
> Maulin
>
>
>
>
>
> On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> wrote:
>
> > Why don't we make SSLEngineBuilder code delegate the whole Key/Trust
> store
> > initialization to the interfaces we are proposing? Default implementation
> > for those key/trust store loader interfaces will be "file based" loading
> vs
> > if somebody wants to customize any of it they can.
> >
> > Would that make sense?
> >
> > On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe 
> wrote:
> >
> >> +1 for making SslEngineBuilder configurable.  This would give
> >> implementers a lot more flexibility-- to use key distribution methods
> that
> >> were not files, for example.
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> >> > Just to make sure we are on the same page - KIP-383 was written before
> >> > the
> >> > code was refactored. The refactoring addressed some of the concerns of
> >> > KIP-383. My suggestion was to make SslEngineBuilder configurable. The
> >> > default implementation of this pluggable class would be
> >> >
> >>
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
> >> .
> >> > That should give an idea of the size of the configurable part that a
> >> > custom
> >> > class needs to implement. A large part of that is about loading
> >> > keystore/truststore. I agree it has slightly more code than KIP-486
> >> > proposes, but since it lets you customize creation of SSLEngine, it
> >> > would
> >> > address every possible scenario.
> >> >
> >> > Thoughts?
> >> >
> >> >
> >> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
> >> maulin.vasav...@gmail.com>
> >> > wrote:
> >> >
> >> > > I thought about it more. I feel that no matter how we refactor the
> >> code
> >> > > (with or without KIP-383 integrated), ultimately the need of
> >> customizing
> >> > > loading for keys and certs will still remain. Whenever that need
> >> arises we
> >> > > might end up thinking about the solution suggested by our KIP-486.
> >> Hence
> >> > > regardless of the other KIPs and configurations "if we do need to
> >> customize
> >> > > loading of keys/certs, we will need the code changes suggested by
> this
> >> > > KIP".
> >> > >
> >> > > Let me know

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

+1 for making SslEngineBuilder configurable upon more thoughts.

However, in the abstraction and default implementation we should make sure
when we do have a requirement to plugin custom key/trust store people don't
have to write lot more code which may not be related to it.

Having said that, does this mean, we resurrect KIP-383 and update it with
latest context and go from there?

We are willing to take up that work for making it configurable.

Thanks
Maulin





On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada 
wrote:

> Why don't we make SSLEngineBuilder code delegate the whole Key/Trust store
> initialization to the interfaces we are proposing? Default implementation
> for those key/trust store loader interfaces will be "file based" loading vs
> if somebody wants to customize any of it they can.
>
> Would that make sense?
>
> On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe  wrote:
>
>> +1 for making SslEngineBuilder configurable.  This would give
>> implementers a lot more flexibility-- to use key distribution methods that
>> were not files, for example.
>>
>> best,
>> Colin
>>
>>
>> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
>> > Just to make sure we are on the same page - KIP-383 was written before
>> > the
>> > code was refactored. The refactoring addressed some of the concerns of
>> > KIP-383. My suggestion was to make SslEngineBuilder configurable. The
>> > default implementation of this pluggable class would be
>> >
>> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
>> .
>> > That should give an idea of the size of the configurable part that a
>> > custom
>> > class needs to implement. A large part of that is about loading
>> > keystore/truststore. I agree it has slightly more code than KIP-486
>> > proposes, but since it lets you customize creation of SSLEngine, it
>> > would
>> > address every possible scenario.
>> >
>> > Thoughts?
>> >
>> >
>> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> > wrote:
>> >
>> > > I thought about it more. I feel that no matter how we refactor the
>> code
>> > > (with or without KIP-383 integrated), ultimately the need of
>> customizing
>> > > loading for keys and certs will still remain. Whenever that need
>> arises we
>> > > might end up thinking about the solution suggested by our KIP-486.
>> Hence
>> > > regardless of the other KIPs and configurations "if we do need to
>> customize
>> > > loading of keys/certs, we will need the code changes suggested by this
>> > > KIP".
>> > >
>> > > Let me know what you guys think.
>> > >
>> > > Harsha, we are working on changing the interfaces for key/trust store
>> > > loaders with Certificate and PrivateKey objects. Will probably be
>> able to
>> > > update it later today or tomorrow.
>> > >
>> > > Thanks
>> > > Maulin
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > On that, I actually looked at KIP-383 before briefly. However,  that
>> > > > sounded like lot of changes suggested.
>> > > >
>> > > > One "key" thing we have to keep in mind is - IF we need lot of
>> > > > customization Kafka already allows you to use your SslProvider via
>> > > > ssl.providers or the changes done by KIP-492 and
>> > > > SSLContext.getInstance(protocol, provider) call allows us to return
>> the
>> > > > SSLContext with "ALL" the details we would like to customize. Hence
>> I am
>> > > > not sure that customization suggested by KIP-383 would be worth the
>> > > effort.
>> > > > We also have similar SSLContext customization outside of Kafka.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
>> > > > clement_pelle...@ibi.com> wrote:
>> > > >
>> > > >> KIP-383 in its present form was vetoed becau

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Why don't we make SSLEngineBuilder code delegate the whole Key/Trust store
initialization to the interfaces we are proposing? Default implementation
for those key/trust store loader interfaces will be "file based" loading vs
if somebody wants to customize any of it they can.

Would that make sense?

On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe  wrote:

> +1 for making SslEngineBuilder configurable.  This would give implementers
> a lot more flexibility-- to use key distribution methods that were not
> files, for example.
>
> best,
> Colin
>
>
> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> > Just to make sure we are on the same page - KIP-383 was written before
> > the
> > code was refactored. The refactoring addressed some of the concerns of
> > KIP-383. My suggestion was to make SslEngineBuilder configurable. The
> > default implementation of this pluggable class would be
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
> .
> > That should give an idea of the size of the configurable part that a
> > custom
> > class needs to implement. A large part of that is about loading
> > keystore/truststore. I agree it has slightly more code than KIP-486
> > proposes, but since it lets you customize creation of SSLEngine, it
> > would
> > address every possible scenario.
> >
> > Thoughts?
> >
> >
> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > wrote:
> >
> > > I thought about it more. I feel that no matter how we refactor the code
> > > (with or without KIP-383 integrated), ultimately the need of
> customizing
> > > loading for keys and certs will still remain. Whenever that need
> arises we
> > > might end up thinking about the solution suggested by our KIP-486.
> Hence
> > > regardless of the other KIPs and configurations "if we do need to
> customize
> > > loading of keys/certs, we will need the code changes suggested by this
> > > KIP".
> > >
> > > Let me know what you guys think.
> > >
> > > Harsha, we are working on changing the interfaces for key/trust store
> > > loaders with Certificate and PrivateKey objects. Will probably be able
> to
> > > update it later today or tomorrow.
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
> maulin.vasav...@gmail.com
> > > >
> > > wrote:
> > >
> > > > On that, I actually looked at KIP-383 before briefly. However,  that
> > > > sounded like lot of changes suggested.
> > > >
> > > > One "key" thing we have to keep in mind is - IF we need lot of
> > > > customization Kafka already allows you to use your SslProvider via
> > > > ssl.providers or the changes done by KIP-492 and
> > > > SSLContext.getInstance(protocol, provider) call allows us to return
> the
> > > > SSLContext with "ALL" the details we would like to customize. Hence
> I am
> > > > not sure that customization suggested by KIP-383 would be worth the
> > > effort.
> > > > We also have similar SSLContext customization outside of Kafka.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> > > > clement_pelle...@ibi.com> wrote:
> > > >
> > > >> KIP-383 in its present form was vetoed because it was not possible
> to
> > > add
> > > >> validation of custom properties in a future KIP. The solution to
> that is
> > > >> the first proposal I wrote for KIP-383 which made the whole
> SslFactory
> > > >> pluggable. That first solution was also vetoed hence the deadlock.
> > > >>
> > > >> Replacing the whole factory was a much nicer solution. It was vetoed
> > > >> because doing this almost invariably meant the replacement lost all
> the
> > > >> complex validation code in the default SslFactory.
> > > >>
> > > >> My current idea is to extract the validation code into another
> public
> > > API
> > > >> that SslFactory would call. I did not look at the newly refactored
> code
> > > and
> > > >>

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

I thought about it more. I feel that no matter how we refactor the code
(with or without KIP-383 integrated), ultimately the need of customizing
loading for keys and certs will still remain. Whenever that need arises we
might end up thinking about the solution suggested by our KIP-486. Hence
regardless of the other KIPs and configurations "if we do need to customize
loading of keys/certs, we will need the code changes suggested by this KIP".

Let me know what you guys think.

Harsha, we are working on changing the interfaces for key/trust store
loaders with Certificate and PrivateKey objects. Will probably be able to
update it later today or tomorrow.

Thanks
Maulin





On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada 
wrote:

> On that, I actually looked at KIP-383 before briefly. However,  that
> sounded like lot of changes suggested.
>
> One "key" thing we have to keep in mind is - IF we need lot of
> customization Kafka already allows you to use your SslProvider via
> ssl.providers or the changes done by KIP-492 and
> SSLContext.getInstance(protocol, provider) call allows us to return the
> SSLContext with "ALL" the details we would like to customize. Hence I am
> not sure that customization suggested by KIP-383 would be worth the effort.
> We also have similar SSLContext customization outside of Kafka.
>
> Thanks
> Maulin
>
>
>
>
>
> On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> clement_pelle...@ibi.com> wrote:
>
>> KIP-383 in its present form was vetoed because it was not possible to add
>> validation of custom properties in a future KIP. The solution to that is
>> the first proposal I wrote for KIP-383 which made the whole SslFactory
>> pluggable. That first solution was also vetoed hence the deadlock.
>>
>> Replacing the whole factory was a much nicer solution. It was vetoed
>> because doing this almost invariably meant the replacement lost all the
>> complex validation code in the default SslFactory.
>>
>> My current idea is to extract the validation code into another public API
>> that SslFactory would call. I did not look at the newly refactored code and
>> I did not study how to do this yet. KIP-383 was not popular at the time and
>> designing a new solution is a lot of work.
>>
>> Is there interest from 3 binding voters for something like this?
>>
>> -Original Message-
>> From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
>> Sent: Thursday, August 29, 2019 2:57 PM
>> To: dev
>> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
>> TrustStore
>>
>> Hi Maulin,
>>
>> In SSL scenarios, I imagine security providers introduced by KIP-492 are
>> likely to be most useful when you want to use third party providers. The
>> biggest advantage of the config from that KIP is that you don't need to
>> write much code to integrate existing security providers into Kafka
>> brokers
>> or clients. As I understand it, KIP-486 is a more convenient option for
>> the
>> specific problem of loading keystores/truststores differently. It can be
>> achieved in theory with KIP-492, but KIP-486 is a much simpler option for
>> this case.
>>
>> My concern about KIP-486 is that it introduces yet another interface into
>> our already complex security code, while only solving one particular use
>> case. Have you looked at
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
>> ?
>> The goal was to make
>> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
>> The code has already been refactored by Colin after that KIP was written,
>> making it easier to implement KIP-383. This should enable you to load your
>> keystores and truststores differently. Using a pluggable SslEngineBuilder
>> will also solve several other use cases at the same time. KIP-383 hasn't
>> been voted through yet, but perhaps you could take a look and we could
>> revive that instead if it solves your use case as well?
>>
>> Regards,
>>
>> Rajini
>>
>>
>> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> wrote:
>>
>> > Hi Harsha
>> >
>> > Thank you. Appreciate your time and support on this. Let me go back do
>> some
>> > more research and get back to you on the KeyStore interface part.
>> > Basically, if we return certs and keys in the interface then Kafka code
>> > will have to build KeyStore object - which is also reasonable.
>> >
>> > Thanks
>> > Maulin
>> >
>> > 

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

On that, I actually looked at KIP-383 before briefly. However,  that
sounded like lot of changes suggested.

One "key" thing we have to keep in mind is - IF we need lot of
customization Kafka already allows you to use your SslProvider via
ssl.providers or the changes done by KIP-492 and
SSLContext.getInstance(protocol, provider) call allows us to return the
SSLContext with "ALL" the details we would like to customize. Hence I am
not sure that customization suggested by KIP-383 would be worth the effort.
We also have similar SSLContext customization outside of Kafka.

Thanks
Maulin





On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement 
wrote:

> KIP-383 in its present form was vetoed because it was not possible to add
> validation of custom properties in a future KIP. The solution to that is
> the first proposal I wrote for KIP-383 which made the whole SslFactory
> pluggable. That first solution was also vetoed hence the deadlock.
>
> Replacing the whole factory was a much nicer solution. It was vetoed
> because doing this almost invariably meant the replacement lost all the
> complex validation code in the default SslFactory.
>
> My current idea is to extract the validation code into another public API
> that SslFactory would call. I did not look at the newly refactored code and
> I did not study how to do this yet. KIP-383 was not popular at the time and
> designing a new solution is a lot of work.
>
> Is there interest from 3 binding voters for something like this?
>
> -Original Message-
> From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
> Sent: Thursday, August 29, 2019 2:57 PM
> To: dev
> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
>
> Hi Maulin,
>
> In SSL scenarios, I imagine security providers introduced by KIP-492 are
> likely to be most useful when you want to use third party providers. The
> biggest advantage of the config from that KIP is that you don't need to
> write much code to integrate existing security providers into Kafka brokers
> or clients. As I understand it, KIP-486 is a more convenient option for the
> specific problem of loading keystores/truststores differently. It can be
> achieved in theory with KIP-492, but KIP-486 is a much simpler option for
> this case.
>
> My concern about KIP-486 is that it introduces yet another interface into
> our already complex security code, while only solving one particular use
> case. Have you looked at
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> ?
> The goal was to make
> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
> The code has already been refactored by Colin after that KIP was written,
> making it easier to implement KIP-383. This should enable you to load your
> keystores and truststores differently. Using a pluggable SslEngineBuilder
> will also solve several other use cases at the same time. KIP-383 hasn't
> been voted through yet, but perhaps you could take a look and we could
> revive that instead if it solves your use case as well?
>
> Regards,
>
> Rajini
>
>
> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada  >
> wrote:
>
> > Hi Harsha
> >
> > Thank you. Appreciate your time and support on this. Let me go back do
> some
> > more research and get back to you on the KeyStore interface part.
> > Basically, if we return certs and keys in the interface then Kafka code
> > will have to build KeyStore object - which is also reasonable.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani 
> > wrote:
> >
> > > Hi Maulin,
> > > Use cases are clear now. I am +1 for moving forward
> > > with the discussions on having such configurable option for users. But
> > the
> > > interfaces is proposed doesn't look right to me. We are still talking
> > about
> > > keystore interfaces.  Given keystore's are used as filebased way of
> > > transporting certificates I am not sure it will help the rest of the
> > > user-base.
> > >   In short, I am +1 on the KIP's motivation and only
> have
> > > questions around returning keystores instead of returning certs,
> private
> > > keys etc. . If others in the community are ok with such interface we
> can
> > > move forward.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > wrote:
> > >
> > > > Hi Harsha
> > > >
> > >

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Harsha

Thank you. Appreciate your time and support on this. Let me go back do some
more research and get back to you on the KeyStore interface part.
Basically, if we return certs and keys in the interface then Kafka code
will have to build KeyStore object - which is also reasonable.

Thanks
Maulin

On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani 
wrote:

> Hi Maulin,
> Use cases are clear now. I am +1 for moving forward
> with the discussions on having such configurable option for users. But the
> interfaces is proposed doesn't look right to me. We are still talking about
> keystore interfaces.  Given keystore's are used as filebased way of
> transporting certificates I am not sure it will help the rest of the
> user-base.
>   In short, I am +1 on the KIP's motivation and only have
> questions around returning keystores instead of returning certs, private
> keys etc. . If others in the community are ok with such interface we can
> move forward.
>
> Thanks,
> Harsha
>
>
> On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> maulin.vasav...@gmail.com>
> wrote:
>
> > Hi Harsha
> >
> > As we synced-up offline on this topic, we hope you don't have any more
> > clarifications that you are seeking. If that is the case, can you please
> > help us move this forward and discuss what changes you would expect on
> the
> > KIP design in order to make it valuable contribution?
> >
> > Just FYI - we verified our primary design change with the author of Sun's
> > X509 Trustmanager implementation and the outcome is that what we are
> > proposing makes sense at the heart of it - "Instead of writing
> TrustManager
> > just plugin the Trust store". We are open to discuss additional changes
> > that you/anybody else would like to see on the functionality however.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > wrote:
> >
> > Hi Harsha
> >
> > Any response on my question? I feel this KIP is worth accommodating. Your
> > help is much appreciated.
> >
> > Thanks
> > Maulin
> >
> > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada < maulin.vasavada@gmail.
> > com> wrote:
> >
> > Hi Harsha
> >
> > I've examined the SPIFFE provider more and have one question -
> >
> > If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> > location, would you really still write the Provider? *OR* Would you just
> > use TrustManagerFactory.init(KeyStore) signature to pass the KeyStore
> from
> > set of certs returned by spiffeIdManager. getTrustedCerts()?
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > provider/CertificateUtils.java#L100
> >
> > /**
> >
> > * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> > configured in
> > * the java.security property ssl.spiffe.accept
> > *
> > * If the authorized spiffe ids list is empty any spiffe id is authorized
> > *
> > * @param chain an array of X509Certificate that contains the Peer's SVID
> > to be validated
> > * @throws CertificateException when either the certificates doesn't have
> a
> > SPIFFE ID or the SPIFFE ID is not authorized
> > */
> > static void checkSpiffeId(X509Certificate[] chain) throws
> > CertificateException {
> >
> > Thanks
> > Maulin
> >
> > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani 
> > wrote:
> >
> > Maulin,
> > The code parts you are pointing are specific for Spiffe and if
> > you are talking about validate method which uses PKIX check like any
> other
> > provider does.
> > If you want to default to SunJSSE everywhere you can do so by delegating
> > the calls in these methods to SunJSSE provider.
> >
> > TrustManagerFactory tmf = TrustManagerFactory
> > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> > tmf.chekServerTrusted()
> > or use
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > TrustManagerFactory.html#getInstance(java.lang.String)if you want a
> > specific provider.
> >
> > -Harsha
> >
> > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada < maulin.vasavada@gmail.
> > com>
> > wrote:
> >
> > Okay, so I take that you guys agree that I have to write a 'custom'
> > algorithm and a provider to make it work , correct?
> >
> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> > implem

[VOTE] KIP-486: Support custom way to load KeyStore and TrustStore

[Maulin Vasavada]

Hi all

After a good discussion on the KIP at
https://www.mail-archive.com/dev@kafka.apache.org/msg99126.html I think we
are ready to start voting.

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-486%3A+Support+custom+way+to+load+KeyStore+and+TrustStore

The KIP proposes - supporting custom way to load key/trust store instead of
just having ability to have key/trust store on the file system.

Thanks
Maulin

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Rajan

Your email format doesn't show up correctly. Can you repost to make it more
readable?

Thanks
Maulin

On Wed, Aug 28, 2019 at 2:32 PM Rajan Dhabalia  wrote:

> *Hi Harsha/Maulin,I am following  KIP-486 and KIP-492 and it seems
> https://github.com/apache/kafka/pull/7090
> <https://github.com/apache/kafka/pull/7090> is the right solution when one
> wants to register custom factory class for KeyManager and TrustManager.
> User can easily configure custom implementation of TrustManager and
> KeyManager using factory Provider class.Configuration of the provider is
> also simple and straightforward: 1. Create custom provider which defines
> factory classes for KeyManager and TrustManagerpublic class CustomProvider
> extends Provider {public CustomProvider()
> {super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> TrustStore");super.put("KeyManagerFactory.CUSTOM",
> "customKeyManagerFactory");
> super.put("TrustManagerFactory.CUSTOM","customTrustManagerFactory");
>  }}
> 1. Register provider at broker
> startupjava.security.Security.addProvider(new CustomProvider());However,
> this approach is useful when one wants to implement custom TrustManager for
> X509 certs by extending X509ExtendedKeyManager and implement various
> abstract methods such as: checkClientTrusted, checkServerTrusted, etc..In
> JDK, default implementation class of X509ExtendedKeyManager is
> X509TrustManagerImpl and one can’t extend or delegate calls to this class
> because it’s final and same is applicable for other available providers
> such as : BouncyCastleProviderTurstManager/KeyManager mainly serve two
> purposes: 1. Provide certs/key2. Perform validationX509TrustManagerImpl
> performs various RFC specified validations.#7090 can be helpful when user
> has both above asks. However, problem defined at KIP-486 has different ask
> where user wants to provide certs/key without copying/implementing Manager
> class because all the available Manager classes are final and can’t be
> extended or delegated. And security team in most of the companies don’t
> allow custom/copying provider in order to get up to date with various RFC
> validations provided into standard jdk provider.Many users manage keys and
> certs into KMS and sometimes it’s not feasible to copy them to file-system
> instead directly use them from the memory. So, KIP-486 provides a custom
> way to load keys/certs without implementing security-provider.Thanks,Rajan*
>
> On Wed, Aug 28, 2019 at 2:18 PM Maulin Vasavada  >
> wrote:
>
> >
> >
> > Hi Harsha
> >
> > As we synced-up offline on this topic, we hope you don't have any more
> > clarifications that you are seeking. If that is the case, can you please
> > help us move this forward and discuss what changes you would expect on
> the
> > KIP design in order to make it valuable contribution?
> >
> > Just FYI - we verified our primary design change with the author of Sun's
> > X509 Trustmanager implementation and the outcome is that what we are
> > proposing makes sense at the heart of it - "Instead of writing
> TrustManager
> > just plugin the Trust store". We are open to discuss additional changes
> > that you/anybody else would like to see on the functionality however.
> >
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > wrote:
> >
> >> Hi Harsha
> >>
> >> Any response on my question? I feel this KIP is worth accommodating.
> Your
> >> help is much appreciated.
> >>
> >> Thanks
> >> Maulin
> >>
> >> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> >> maulin.vasav...@gmail.com> wrote:
> >>
> >>> Hi Harsha
> >>>
> >>> I've examined the SPIFFE provider more and have one question -
> >>>
> >>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> >>> location, would you really still write the Provider? *OR*
> >>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass
> >>> the KeyStore from set of certs returned by spiffeIdManager.
> >>> getTrustedCerts()?
> >>>
> >>>
> >>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
> >>>
> >>>
> >>> /**
> >>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> >>>> configured in
> >>>> * the java.security proper

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Harsha

As we synced-up offline on this topic, we hope you don't have any more
clarifications that you are seeking. If that is the case, can you please
help us move this forward and discuss what changes you would expect on the
KIP design in order to make it valuable contribution?

Just FYI - we verified our primary design change with the author of Sun's
X509 Trustmanager implementation and the outcome is that what we are
proposing makes sense at the heart of it - "Instead of writing TrustManager
just plugin the Trust store". We are open to discuss additional changes
that you/anybody else would like to see on the functionality however.


Thanks
Maulin

On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada 
wrote:

> Hi Harsha
>
> Any response on my question? I feel this KIP is worth accommodating. Your
> help is much appreciated.
>
> Thanks
> Maulin
>
> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> maulin.vasav...@gmail.com> wrote:
>
>> Hi Harsha
>>
>> I've examined the SPIFFE provider more and have one question -
>>
>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
>> location, would you really still write the Provider? *OR*
>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass
>> the KeyStore from set of certs returned by spiffeIdManager.
>> getTrustedCerts()?
>>
>>
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
>>
>>
>> /**
>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
>>> configured in
>>> * the java.security property ssl.spiffe.accept
>>> *
>>> * If the authorized spiffe ids list is empty any spiffe id is authorized
>>> *
>>> * @param chain an array of X509Certificate that contains the Peer's
>>> SVID to be validated
>>> * @throws CertificateException when either the certificates doesn't
>>> have a SPIFFE ID or the SPIFFE ID is not authorized
>>> */
>>> static void checkSpiffeId(X509Certificate[] chain) throws
>>> CertificateException {
>>
>>
>>
>> Thanks
>> Maulin
>>
>>
>> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani 
>> wrote:
>>
>>> Maulin,
>>>  The code parts you are pointing are specific for Spiffe and
>>> if
>>> you are talking about validate method  which uses  PKIX check like any
>>> other provider does.
>>> If you want to default to SunJSSE everywhere you can do so by delegating
>>> the calls in these methods to SunJSSE provider.
>>>
>>> TrustManagerFactory tmf = TrustManagerFactory
>>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
>>> tmf.chekServerTrusted()
>>> or use
>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
>>> you want a specific provider.
>>>
>>> -Harsha
>>>
>>>
>>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
>>> maulin.vasav...@gmail.com>
>>> wrote:
>>>
>>> > Okay, so I take that you guys agree that I have to write a 'custom'
>>> > algorithm and a provider to make it work , correct?
>>> >
>>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
>>> > implementation per say , ..." , I diagree. You can refer to https://
>>> >
>>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>>> > SpiffeTrustManager.java#L91 and
>>> >
>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>> > provider/CertificateUtils.java#L100
>>> >
>>> > "that code" is the customization you have for the custom way to check
>>> > something on top of regular checks. That method is NOT doing custom
>>> > truststore loading. It is validating/verifying something in the
>>> "custom"
>>> > way with spiffeId.
>>> > I bet that without that you won't have a need of the custom algorithm
>>> in
>>> > the first place.
>>> >
>>> > Let me know if you agree to this.
>>> >
>>> > Thanks
>>> > Maulin
>>> >
>>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri 
>>> wrote:
>>> >
>>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
>>> > KIP492, you need to create a new provider, register a *new* 

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Harsha

Any response on my question? I feel this KIP is worth accommodating. Your
help is much appreciated.

Thanks
Maulin

On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada 
wrote:

> Hi Harsha
>
> I've examined the SPIFFE provider more and have one question -
>
> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> location, would you really still write the Provider? *OR*
> Would you just use TrustManagerFactory.init(KeyStore) signature to pass
> the KeyStore from set of certs returned by spiffeIdManager.
> getTrustedCerts()?
>
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
>
>
> /**
>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
>> configured in
>> * the java.security property ssl.spiffe.accept
>> *
>> * If the authorized spiffe ids list is empty any spiffe id is authorized
>> *
>> * @param chain an array of X509Certificate that contains the Peer's SVID
>> to be validated
>> * @throws CertificateException when either the certificates doesn't have
>> a SPIFFE ID or the SPIFFE ID is not authorized
>> */
>> static void checkSpiffeId(X509Certificate[] chain) throws
>> CertificateException {
>
>
>
> Thanks
> Maulin
>
>
> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani 
> wrote:
>
>> Maulin,
>>  The code parts you are pointing are specific for Spiffe and
>> if
>> you are talking about validate method  which uses  PKIX check like any
>> other provider does.
>> If you want to default to SunJSSE everywhere you can do so by delegating
>> the calls in these methods to SunJSSE provider.
>>
>> TrustManagerFactory tmf = TrustManagerFactory
>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
>> tmf.chekServerTrusted()
>> or use
>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
>> you want a specific provider.
>>
>> -Harsha
>>
>>
>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> wrote:
>>
>> > Okay, so I take that you guys agree that I have to write a 'custom'
>> > algorithm and a provider to make it work , correct?
>> >
>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
>> > implementation per say , ..." , I diagree. You can refer to https://
>> >
>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>> > SpiffeTrustManager.java#L91 and
>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > provider/CertificateUtils.java#L100
>> >
>> > "that code" is the customization you have for the custom way to check
>> > something on top of regular checks. That method is NOT doing custom
>> > truststore loading. It is validating/verifying something in the "custom"
>> > way with spiffeId.
>> > I bet that without that you won't have a need of the custom algorithm in
>> > the first place.
>> >
>> > Let me know if you agree to this.
>> >
>> > Thanks
>> > Maulin
>> >
>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri 
>> wrote:
>> >
>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
>> > KIP492, you need to create a new provider, register a *new* custom
>> > algorithm for your keymanager and trustmanager factory implementations.
>> > After this, the kafka server configuration can be done as given below
>> >
>> > # Register the provider class with custom algorithm, say CUSTOM
>> security.
>> > provider.classes=com.company.security.CustomProvider
>> > <http://security.provider.classes=com.company.security.customprovider/>
>> >
>> > # Register the keymanager and trustmanager algorithms
>> > # These algorithms indicate that the Keymanager and Trustmanagers
>> > registered under the algorithm "CUSTOM" needs to be used
>> > ssl.trustmanager.algorithm=CUSTOM
>> > ssl.keymanager.algorithm=CUSTOM
>> >
>> > And the customprovider looks like this...
>> >
>> > public class CustomProvider extends Provider {
>> > public CustomProvider() {
>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
>> > super.put("Trus

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Harsha

I've examined the SPIFFE provider more and have one question -

If SPIFFE didn't have a need to do checkSpiffeId() call at the below
location, would you really still write the Provider? *OR*
Would you just use TrustManagerFactory.init(KeyStore) signature to pass the
KeyStore from set of certs returned by spiffeIdManager.getTrustedCerts()?

https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100


/**
> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> configured in
> * the java.security property ssl.spiffe.accept
> *
> * If the authorized spiffe ids list is empty any spiffe id is authorized
> *
> * @param chain an array of X509Certificate that contains the Peer's SVID
> to be validated
> * @throws CertificateException when either the certificates doesn't have
> a SPIFFE ID or the SPIFFE ID is not authorized
> */
> static void checkSpiffeId(X509Certificate[] chain) throws
> CertificateException {



Thanks
Maulin


On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani  wrote:

> Maulin,
>  The code parts you are pointing are specific for Spiffe and if
> you are talking about validate method  which uses  PKIX check like any
> other provider does.
> If you want to default to SunJSSE everywhere you can do so by delegating
> the calls in these methods to SunJSSE provider.
>
> TrustManagerFactory tmf = TrustManagerFactory
> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> tmf.chekServerTrusted()
> or use
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
> you want a specific provider.
>
> -Harsha
>
>
> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> maulin.vasav...@gmail.com>
> wrote:
>
> > Okay, so I take that you guys agree that I have to write a 'custom'
> > algorithm and a provider to make it work , correct?
> >
> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> > implementation per say , ..." , I diagree. You can refer to https://
> > github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > SpiffeTrustManager.java#L91 and
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > provider/CertificateUtils.java#L100
> >
> > "that code" is the customization you have for the custom way to check
> > something on top of regular checks. That method is NOT doing custom
> > truststore loading. It is validating/verifying something in the "custom"
> > way with spiffeId.
> > I bet that without that you won't have a need of the custom algorithm in
> > the first place.
> >
> > Let me know if you agree to this.
> >
> > Thanks
> > Maulin
> >
> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri  wrote:
> >
> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
> > KIP492, you need to create a new provider, register a *new* custom
> > algorithm for your keymanager and trustmanager factory implementations.
> > After this, the kafka server configuration can be done as given below
> >
> > # Register the provider class with custom algorithm, say CUSTOM security.
> > provider.classes=com.company.security.CustomProvider
> > <http://security.provider.classes=com.company.security.customprovider/>
> >
> > # Register the keymanager and trustmanager algorithms
> > # These algorithms indicate that the Keymanager and Trustmanagers
> > registered under the algorithm "CUSTOM" needs to be used
> > ssl.trustmanager.algorithm=CUSTOM
> > ssl.keymanager.algorithm=CUSTOM
> >
> > And the customprovider looks like this...
> >
> > public class CustomProvider extends Provider {
> > public CustomProvider() {
> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> > super.put("TrustManagerFactory.CUSTOM",
> > "customTrustManagerFactory");
> > }
> > }
> >
> > The PR for this is in review and can be found here: https://github.com/
> > apache/kafka/pull/7090
> > This PR includes the fixed insertProviderAt function call.
> >
> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani 
> > wrote:
> >
> > Answers are in-line
> >
> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> maulin.vasavada@gmail.
> > com
> >
> > wrote:
> >
> > Hi Colin
> >
> > When I refer to "standard" or "cu

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Okay, so I take that you guys agree that I have to write a 'custom'
algorithm and a provider to make it work , correct?

Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
implementation per say , ..." , I diagree. You can refer to
https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeTrustManager.java#L91
 and
https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100


"that code" is the customization you have for the custom way to check
something on top of regular checks. That method is NOT doing custom
truststore loading. It is validating/verifying something in the "custom"
way with spiffeId.
I bet that without that you won't have a need of the custom algorithm in
the first place.

Let me know if you agree to this.

Thanks
Maulin



On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri  wrote:

> Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
> KIP492, you need to create a new provider, register a *new* custom
> algorithm for your keymanager and trustmanager factory implementations.
> After this, the kafka server configuration can be done as given below
>
> # Register the provider class with custom algorithm, say CUSTOM
> security.provider.classes=com.company.security.CustomProvider
>
> # Register the keymanager and trustmanager algorithms
> # These algorithms indicate that the Keymanager and Trustmanagers
> registered under the algorithm "CUSTOM" needs to be used
> ssl.trustmanager.algorithm=CUSTOM
> ssl.keymanager.algorithm=CUSTOM
>
>
> And the customprovider looks like this...
>
> public class CustomProvider extends Provider {
> public CustomProvider() {
> super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> TrustStore");
> super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> super.put("TrustManagerFactory.CUSTOM",
> "customTrustManagerFactory");
> }
> }
>
>
> The PR for this is in review and can be found here:
> https://github.com/apache/kafka/pull/7090
> This PR includes the fixed insertProviderAt function call.
>
> On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani 
> wrote:
>
> > Answers are in-line
> >
> >
> >
> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > maulin.vasav...@gmail.com
> > > wrote:
> >
> > > Hi Colin
> > >
> > >
> > > When I refer to "standard" or "custom" algorithms I am following Java
> > > security Provider Terminology. You can refer to
> > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > StandardNames.html#TrustManagerFactory link I provided earlier in the
> > > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
> > >
> > >
> > > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> > > custom algorithms' because the following file clearly indicates that it
> > > does use custom algorithm-
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > provider/SpiffeProvider.java#L17
> > >
> > >
> > > Algorithm value:
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > provider/SpiffeProviderConstants.java#L6
> > >
> > >
> > > @Harsha do you want to chime in since you use that provider?
> > >
> >
> > Here the "Custom" Algorithm is not an implementation per say , rather
> used
> > to invoke the custom trust store factory and key manager factory. You are
> > not moving away from "standard" alogrithms that are available.
> >
> >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeTrustManager.java
> > As you can see it delegates all the calls of verification of certificate
> to
> > the default implementation available.
> > So in our implementation we still use PKIX to verify the certificate
> > chain.  So you are not losing anything here and Spiffe is not
> > reimplementing the verification process.
> >
> >
> > 2. I already mentioned in my 3rd point, in my previous post, why using
> > > ssl.provider does NOT work. I updated KIP-486 in "rejected
> alternatives"
> > > also why ssl.provider does not work.
> > >
> >
> > As mentioned before , provider is the right way to go. I am still not
> > understanding the gap is.
> > If I understand correctly your argument is , 

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Instead of keep diverging in different directions, it would be helpful if
you guys take my detailed posts with 1st to 4th points I mentioned and
start referring/commenting on each of those if you agree with them or not.

On Mon, Aug 19, 2019 at 10:45 PM Maulin Vasavada 
wrote:

> Hi Colin
>
> When I refer to "standard" or "custom" algorithms I am following Java
> security Provider Terminology. You can refer to
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html#TrustManagerFactory
> link I provided earlier in the emails. It says PKIX is the default
> Algorithm for TrustManagerFactory.
>
> 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> custom algorithms' because the following file clearly indicates that it
> does use custom algorithm-
>
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeProvider.java#L17
>
>
> Algorithm value:
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeProviderConstants.java#L6
>
> @Harsha do you want to chime in since you use that provider?
>
> 2. I already mentioned in my 3rd point, in my previous post, why using
> ssl.provider does NOT work. I updated KIP-486 in "rejected alternatives"
> also why ssl.provider does not work.
>
> 3. Security.insertProviderAt() comments were based on assumption if
> KIP-492 changes are done and we use that mechanism to configure providers
> instead of ssl.provider configuration.
>
> Can you read my all the points, I mentioned in my previous post, very
> carefully? I am covering all the aspects in explaining. I am open to still
> discuss more to clarify any doubts.
>
> Thanks
> Maulin
>
>
>
> On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe  wrote:
>
>> Hi Maulin,
>>
>> A lot of JSSE providers don't implement custom algorithms.  Spire is a
>> good example of a JSSE provider that doesn't, and yet is still useful to
>> many people.  Your JSSE provider can work fine even if it doesn't implement
>> a custom algorithm.
>>
>> Maybe I'm missing something, but I don't understand the discussion of
>> Security.insertProviderAt() that you included.  SslEngineBuilder doesn't
>> use that API to get the security provider.  Instead, it calls
>> "SSLContext.getInstance(protocol, provider)", where provider is the name of
>> the provider.
>>
>> best,
>> Colin
>>
>>
>> On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>> > On top of everything above I feel strongly to add the 4th point which is
>> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
>> >
>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#init(java.security.KeyStore)
>> )
>> > and KeyManagerFactory.init(KeyStore, char[]) (
>> >
>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.html#init(java.security.KeyStore,%20char[])
>> > ).
>> >
>> > 4. The above APIs are intended to support providing "trust/key material"
>> > from the user without having to write their own
>> TrustManager/KeyManagers.
>> > To quote from the TrustManagerFactory.init()'s documentation
>> "Initializes
>> > this factory with a source of certificate authorities and related trust
>> > material."
>> > To quote from the KeyManagerFactory.init()'s documentation "Initializes
>> > this factory with a source of key material."
>> >
>> > Based on this it is clear that there is a flexibility provided by Java
>> to
>> > to enable developers to provide the required trust/key material loaded
>> from
>> > "anywhere" without requiring them to write custom provider OR trust/key
>> > managers. This same flexibility is reflected in Kafka code also where it
>> > loads the trust/keys from a local file and doesn't require writing a
>> > Provider necessarily. If we do NOT have a custom algorithm, it makes
>> less
>> > sense to write a Provider.
>> >
>> > Thanks
>> > Maulin
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> > wrote:
>> >
>> > > Hi Harsha/Colin
>> > >
>> > > I did the sample with a custom Provider for TrustStoreManager and
>> tried
>> > > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
>> > > adding

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Colin

When I refer to "standard" or "custom" algorithms I am following Java
security Provider Terminology. You can refer to
https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html#TrustManagerFactory
link I provided earlier in the emails. It says PKIX is the default
Algorithm for TrustManagerFactory.

1. For SPIFFE, I am not sure why you are saying 'it does not implement
custom algorithms' because the following file clearly indicates that it
does use custom algorithm-

https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeProvider.java#L17


Algorithm value:
https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeProviderConstants.java#L6

@Harsha do you want to chime in since you use that provider?

2. I already mentioned in my 3rd point, in my previous post, why using
ssl.provider does NOT work. I updated KIP-486 in "rejected alternatives"
also why ssl.provider does not work.

3. Security.insertProviderAt() comments were based on assumption if KIP-492
changes are done and we use that mechanism to configure providers instead
of ssl.provider configuration.

Can you read my all the points, I mentioned in my previous post, very
carefully? I am covering all the aspects in explaining. I am open to still
discuss more to clarify any doubts.

Thanks
Maulin



On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe  wrote:

> Hi Maulin,
>
> A lot of JSSE providers don't implement custom algorithms.  Spire is a
> good example of a JSSE provider that doesn't, and yet is still useful to
> many people.  Your JSSE provider can work fine even if it doesn't implement
> a custom algorithm.
>
> Maybe I'm missing something, but I don't understand the discussion of
> Security.insertProviderAt() that you included.  SslEngineBuilder doesn't
> use that API to get the security provider.  Instead, it calls
> "SSLContext.getInstance(protocol, provider)", where provider is the name of
> the provider.
>
> best,
> Colin
>
>
> On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> > On top of everything above I feel strongly to add the 4th point which is
> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#init(java.security.KeyStore)
> )
> > and KeyManagerFactory.init(KeyStore, char[]) (
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.html#init(java.security.KeyStore,%20char[])
> > ).
> >
> > 4. The above APIs are intended to support providing "trust/key material"
> > from the user without having to write their own TrustManager/KeyManagers.
> > To quote from the TrustManagerFactory.init()'s documentation "Initializes
> > this factory with a source of certificate authorities and related trust
> > material."
> > To quote from the KeyManagerFactory.init()'s documentation "Initializes
> > this factory with a source of key material."
> >
> > Based on this it is clear that there is a flexibility provided by Java to
> > to enable developers to provide the required trust/key material loaded
> from
> > "anywhere" without requiring them to write custom provider OR trust/key
> > managers. This same flexibility is reflected in Kafka code also where it
> > loads the trust/keys from a local file and doesn't require writing a
> > Provider necessarily. If we do NOT have a custom algorithm, it makes less
> > sense to write a Provider.
> >
> > Thanks
> > Maulin
> >
> >
> >
> >
> >
> >
> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > wrote:
> >
> > > Hi Harsha/Colin
> > >
> > > I did the sample with a custom Provider for TrustStoreManager and tried
> > > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > > adding Provider programmatically instead of relying on
> > > ssl.provider+java.security. The below sample is followed by my detailed
> > > findings. I'll appreciate if you can go through it carefully and see
> if you
> > > see my point.
> > >
> > > package providertest;
> > >
> > > import java.security.Provider;
> > >
> > > public class MyProvider extends Provider {
> > >
> > > private static final String name = "MyProvider";
> > > private static double version = 1.0d;
> > > private static String info = "Maulin's SSL Provider v"+version;
> > >
> > > public MyProvider() {
> > > super(name, ve

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

On top of everything above I feel strongly to add the 4th point which is
based on Java APIs for TrustManagerFactory.init(KeyStore) (
https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#init(java.security.KeyStore))
and KeyManagerFactory.init(KeyStore, char[]) (
https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.html#init(java.security.KeyStore,%20char[])
).

4. The above APIs are intended to support providing "trust/key material"
from the user without having to write their own TrustManager/KeyManagers.
To quote from the TrustManagerFactory.init()'s documentation "Initializes
this factory with a source of certificate authorities and related trust
material."
To quote from the KeyManagerFactory.init()'s documentation "Initializes
this factory with a source of key material."

Based on this it is clear that there is a flexibility provided by Java to
to enable developers to provide the required trust/key material loaded from
"anywhere" without requiring them to write custom provider OR trust/key
managers. This same flexibility is reflected in Kafka code also where it
loads the trust/keys from a local file and doesn't require writing a
Provider necessarily. If we do NOT have a custom algorithm, it makes less
sense to write a Provider.

Thanks
Maulin






On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada 
wrote:

> Hi Harsha/Colin
>
> I did the sample with a custom Provider for TrustStoreManager and tried
> using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> adding Provider programmatically instead of relying on
> ssl.provider+java.security. The below sample is followed by my detailed
> findings. I'll appreciate if you can go through it carefully and see if you
> see my point.
>
> package providertest;
>
> import java.security.Provider;
>
> public class MyProvider extends Provider {
>
> private static final String name = "MyProvider";
> private static double version = 1.0d;
> private static String info = "Maulin's SSL Provider v"+version;
>
> public MyProvider() {
> super(name, version, info);
> this.put("TrustManagerFactory.PKIX", 
> "providertest.MyTrustManagerFactory");
> }
> }
>
>
>
> *Details:*
>
> KIP-492 documents that it will use Security.addProvider() assuming it will
> add it as position '0' which is not a correct assumption. The
> addProvider()'s documentation says it will add it to the last available
> position. You may want to correct that to say
> Security.insertProviderAt(provider, 1).
>
> Now coming back to our specific discussion,
>
> 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add that
> provider in the provider list via Security.addProvider() the position where
> it gets added doesn't matter (even if you don't end up adding it as first
> entry) since that is the ONLY provider for SPIFFE specific algorithm you
> might have.
>
> We do *not* have custom algorithm for Key/Trust StoreMangers. Which means
> we have to use X509, PKIX etc "Standard Algorithms" ((
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html))
> in our provider to override the TrustStoreManager (see my sample code) and
> KeyStoreManger and KeyManager. This creates another challenge mentioned in
> the below point.
>
> 2. In order to make our Provider for loading custom TrustStore work, we
> have to add the provider as 'first' in the list since there are others with
> the same algorithm.
>
> However, the programatic way of adding provider
> (Security.insertProviderAt()) is *not* deterministic for ordering since
> different code can call the method for a different provider and depending
> upon the order of the call our provider can be first or pushed down the
> list. This can happen very well in any client application using Kafka. This
> is specially problematic for a case when you want to guarantee order for a
> Provider having "Standard Algorithms".
>
> If we add our provider in java.security file that definitely guarantees
> the order(unless somebody calls removeProvider() which is less likely). But
> if we add our provider in java.security file it will defeat the purpose of
> the KIP-492.
>
> In the gist - Apache Kafka must not rely on "un-deterministic" method to
> rely on Provider ordering.
>
> 3. If we just use existing ssl.provider kafka configuration then our
> provider will be used in SSLContext.getInstance(protocol, provider) call in
> SslFactory.java and if our provider does not have implementation for
> SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested it). Example: In
> MyProvider sample above you see that I didn't add S

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Harsha/Colin

I did the sample with a custom Provider for TrustStoreManager and tried
using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
adding Provider programmatically instead of relying on
ssl.provider+java.security. The below sample is followed by my detailed
findings. I'll appreciate if you can go through it carefully and see if you
see my point.

package providertest;

import java.security.Provider;

public class MyProvider extends Provider {

private static final String name = "MyProvider";
private static double version = 1.0d;
private static String info = "Maulin's SSL Provider v"+version;

public MyProvider() {
super(name, version, info);
this.put("TrustManagerFactory.PKIX",
"providertest.MyTrustManagerFactory");
}
}



*Details:*

KIP-492 documents that it will use Security.addProvider() assuming it will
add it as position '0' which is not a correct assumption. The
addProvider()'s documentation says it will add it to the last available
position. You may want to correct that to say
Security.insertProviderAt(provider, 1).

Now coming back to our specific discussion,

1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add that
provider in the provider list via Security.addProvider() the position where
it gets added doesn't matter (even if you don't end up adding it as first
entry) since that is the ONLY provider for SPIFFE specific algorithm you
might have.

We do *not* have custom algorithm for Key/Trust StoreMangers. Which means
we have to use X509, PKIX etc "Standard Algorithms" ((
https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html))
in our provider to override the TrustStoreManager (see my sample code) and
KeyStoreManger and KeyManager. This creates another challenge mentioned in
the below point.

2. In order to make our Provider for loading custom TrustStore work, we
have to add the provider as 'first' in the list since there are others with
the same algorithm.

However, the programatic way of adding provider
(Security.insertProviderAt()) is *not* deterministic for ordering since
different code can call the method for a different provider and depending
upon the order of the call our provider can be first or pushed down the
list. This can happen very well in any client application using Kafka. This
is specially problematic for a case when you want to guarantee order for a
Provider having "Standard Algorithms".

If we add our provider in java.security file that definitely guarantees the
order(unless somebody calls removeProvider() which is less likely). But if
we add our provider in java.security file it will defeat the purpose of the
KIP-492.

In the gist - Apache Kafka must not rely on "un-deterministic" method to
rely on Provider ordering.

3. If we just use existing ssl.provider kafka configuration then our
provider will be used in SSLContext.getInstance(protocol, provider) call in
SslFactory.java and if our provider does not have implementation for
SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested it). Example: In
MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
"Service+Algorithm" and that didn't work for me. In SPIFFE provider you
don't have this challenge since you are planning to bypass ssl.provider as
you mention in the KIP-492.


*Overall summary,*

1. Any provider based mechanisms- a) existing ssl.provider and b)KIP-492,
for loading key/trust store using "Standard Algorithms" do not work

2. Approach suggested in our KIP-486 works without any issue and it is *not*
our context specific solve

3. Based on above we feel KIP-492 and KIP-486 are complimentary changes and
not contradicting or redundent.

If you want we can do a joint session somehow to walk through the sample I
have and various experiments I did. I would encourage you to do similar
exercise by writing a Provider for "Standard Algorithm" for
TrustStoreManager (like our needs) and see what you find since only writing
samples can bring out the complexity/challenges we face.

Thanks
Maulin

On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada 
wrote:

> Just to update - still working on it. Get to work only on and off on it :(
>
> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada 
> wrote:
>
>> Hi Harsha
>>
>> Let me try to write samples and will let you know.
>>
>> Thanks
>> Maulin
>>
>> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch  wrote:
>>
>>> Hi Maulin,
>>>  With java security providers can be as custom you would like it
>>> to
>>> be. If you only want to to implement a custom way of loading the
>>> keystore and truststore and not implement any protocol/encryption
>>> handling
>>> you can leave them empty and no need to implement.
>>> Have you looked into the

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Just to update - still working on it. Get to work only on and off on it :(

On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada 
wrote:

> Hi Harsha
>
> Let me try to write samples and will let you know.
>
> Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch  wrote:
>
>> Hi Maulin,
>>  With java security providers can be as custom you would like it
>> to
>> be. If you only want to to implement a custom way of loading the
>> keystore and truststore and not implement any protocol/encryption handling
>> you can leave them empty and no need to implement.
>> Have you looked into the links I pasted before?
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>
>> Can you please tell me which methods are too complex in above to implement
>> or unnecessary? You are changing anything in SSL/TLS implementations
>> provided by
>>
>> All of the implementations delegating the checks to the default
>> implementation anyway.
>> Spire agent is an example, its nothing but a GRPC server listening on a
>> unix domain socket . Above code is making a RPC call to the local daemon
>> to
>> get the certificate and keys. The mechanics are pretty much same as what
>> you are asking for.
>>
>> Thanks,
>> Harsha
>>
>> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada > >
>> wrote:
>>
>> > Imagine a scenario like - We know we have a custom KMS and as a Kafka
>> owner
>> > we want to comply to using that KMS source to load keys/certs. As a
>> Kafka
>> > owner we know how to integrate with KMS but doesn't necessarily have to
>> > know anything about cipher suites, algorithms, and SSL/TLS
>> implementation.
>> > Going the Provider way requires to know lot more than we should, isn't
>> it?
>> > Not that we would have concern/shy-away knowing those details - but if
>> we
>> > don't have to - why should we?
>> >
>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> > wrote:
>> >
>> > > Hi Harsha
>> > >
>> > > We don't have spire (or similar) agents and we do not have keys/certs
>> > > locally on any brokers. To elaborate more on my previous email,
>> > >
>> > > I agree that Java security Providers are used in much broader sense -
>> to
>> > > have a particular implementation of an algorithm, use specific cipher
>> > > suites for SSL , OR  in our current team's case have a particular way
>> to
>> > > leverage pre-generated SSL sessions. However, the scope of our KIP
>> (486)
>> > is
>> > > much restricted than that. We merely intend to provide a custom
>> > > keystore/truststore for our SSL connections and not really worry about
>> > > underlying specific SSL/TLS implementation.  This simplifies it a lot
>> for
>> > > us to keep the concerns separate and clear.
>> > >
>> > > I feel our approach is more complimentary such that it allows for
>> using
>> > > keystores of choice while retaining the flexibility to use any
>> > > underlying/available Provider for actually making the SSL call.
>> > >
>> > > We agree with KIP-492's approach based on Providers (and Java's
>> > > recommendation), but also strongly believe that our approach can
>> > compliment
>> > > it very effectively for reasons explained above.
>> > >
>> > > Thanks
>> > > Maulin
>> > >
>> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani 
>> > > wrote:
>> > >
>> > >> Hi Maulin,
>> > >>
>> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>> > >> maulin.vasav...@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > Hi Harsha
>> > >> >
>> > >> > The reason we rejected the SslProvider route is that - we only
>> needed
>> > a
>> > >> > custom way to load keys/certs. Not touch any policy that existing
>> > >> Providers

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Harsha

Let me try to write samples and will let you know.

Thanks
Maulin

On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch  wrote:

> Hi Maulin,
>  With java security providers can be as custom you would like it to
> be. If you only want to to implement a custom way of loading the
> keystore and truststore and not implement any protocol/encryption handling
> you can leave them empty and no need to implement.
> Have you looked into the links I pasted before?
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>
> Can you please tell me which methods are too complex in above to implement
> or unnecessary? You are changing anything in SSL/TLS implementations
> provided by
>
> All of the implementations delegating the checks to the default
> implementation anyway.
> Spire agent is an example, its nothing but a GRPC server listening on a
> unix domain socket . Above code is making a RPC call to the local daemon to
> get the certificate and keys. The mechanics are pretty much same as what
> you are asking for.
>
> Thanks,
> Harsha
>
> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada 
> wrote:
>
> > Imagine a scenario like - We know we have a custom KMS and as a Kafka
> owner
> > we want to comply to using that KMS source to load keys/certs. As a Kafka
> > owner we know how to integrate with KMS but doesn't necessarily have to
> > know anything about cipher suites, algorithms, and SSL/TLS
> implementation.
> > Going the Provider way requires to know lot more than we should, isn't
> it?
> > Not that we would have concern/shy-away knowing those details - but if we
> > don't have to - why should we?
> >
> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > wrote:
> >
> > > Hi Harsha
> > >
> > > We don't have spire (or similar) agents and we do not have keys/certs
> > > locally on any brokers. To elaborate more on my previous email,
> > >
> > > I agree that Java security Providers are used in much broader sense -
> to
> > > have a particular implementation of an algorithm, use specific cipher
> > > suites for SSL , OR  in our current team's case have a particular way
> to
> > > leverage pre-generated SSL sessions. However, the scope of our KIP
> (486)
> > is
> > > much restricted than that. We merely intend to provide a custom
> > > keystore/truststore for our SSL connections and not really worry about
> > > underlying specific SSL/TLS implementation.  This simplifies it a lot
> for
> > > us to keep the concerns separate and clear.
> > >
> > > I feel our approach is more complimentary such that it allows for using
> > > keystores of choice while retaining the flexibility to use any
> > > underlying/available Provider for actually making the SSL call.
> > >
> > > We agree with KIP-492's approach based on Providers (and Java's
> > > recommendation), but also strongly believe that our approach can
> > compliment
> > > it very effectively for reasons explained above.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani 
> > > wrote:
> > >
> > >> Hi Maulin,
> > >>
> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > >> maulin.vasav...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Harsha
> > >> >
> > >> > The reason we rejected the SslProvider route is that - we only
> needed
> > a
> > >> > custom way to load keys/certs. Not touch any policy that existing
> > >> Providers
> > >> > govern like SunJSSE Provider.
> > >> >
> > >>
> > >> We have exactly the same requirements to load certs and keys through
> > spire
> > >> agent. We used security.provider to do that exactly. I am not sure why
> > you
> > >> would be modifying any policies provided by default SunJSSE provider.
> > Can
> > >> you give me an example of having custom provider that will override an
> > >> existing policy in  SunJSSE provider.
> > >>
> > &

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Colin,

To your point - "Also, it seems like most people who want a custom
truststore / keystore will also want a custom SSL provider," I don't think
so. Take our own example. We have a fairly large Kafka eco-system (500B+
messages a day flowing through with poly-glot client-base) with strict
InfoSec requirements but we still do not need Custom SSL Provider for Kafka.

Thanks
Maulin

On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada 
wrote:

> Imagine a scenario like - We know we have a custom KMS and as a Kafka
> owner we want to comply to using that KMS source to load keys/certs. As a
> Kafka owner we know how to integrate with KMS but doesn't necessarily have
> to know anything about cipher suites, algorithms, and SSL/TLS
> implementation. Going the Provider way requires to know lot more than we
> should, isn't it? Not that we would have concern/shy-away knowing those
> details - but if we don't have to - why should we?
>
> On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada 
> wrote:
>
>> Hi Harsha
>>
>> We don't have spire (or similar) agents and we do not have keys/certs
>> locally on any brokers. To elaborate more on my previous email,
>>
>> I agree that Java security Providers are used in much broader sense - to
>> have a particular implementation of an algorithm, use specific cipher
>> suites for SSL , OR  in our current team's case have a particular way to
>> leverage pre-generated SSL sessions. However, the scope of our KIP (486) is
>> much restricted than that. We merely intend to provide a custom
>> keystore/truststore for our SSL connections and not really worry about
>> underlying specific SSL/TLS implementation.  This simplifies it a lot for
>> us to keep the concerns separate and clear.
>>
>> I feel our approach is more complimentary such that it allows for using
>> keystores of choice while retaining the flexibility to use any
>> underlying/available Provider for actually making the SSL call.
>>
>> We agree with KIP-492's approach based on Providers (and Java's
>> recommendation), but also strongly believe that our approach can compliment
>> it very effectively for reasons explained above.
>>
>> Thanks
>> Maulin
>>
>> On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani 
>> wrote:
>>
>>> Hi Maulin,
>>>
>>> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>>> maulin.vasav...@gmail.com>
>>> wrote:
>>>
>>> > Hi Harsha
>>> >
>>> > The reason we rejected the SslProvider route is that - we only needed a
>>> > custom way to load keys/certs. Not touch any policy that existing
>>> Providers
>>> > govern like SunJSSE Provider.
>>> >
>>>
>>> We have exactly the same requirements to load certs and keys through
>>> spire
>>> agent. We used security.provider to do that exactly. I am not sure why
>>> you
>>> would be modifying any policies provided by default SunJSSE provider.
>>> Can
>>> you give me an example of having custom provider that will override an
>>> existing policy in  SunJSSE provider.
>>>
>>> As pointed out earlier, this kip
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>> allows
>>> you to  load security.provider through config.
>>> Take a look at the examples I gave before
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>> It registers KeyManagerFactory and TrustManagerFactory and Keystore
>>> algorithm.
>>> Implement your custom way of loading Keystore in here
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>>
>>> and Trust manager like here
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>>
>>> In your Kafka client  you can set the security.provider to your custom
>>> implementation and with this fix
>>> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>>> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>>
>>> All of this is in your clients and broker side and do not need to touch
>>> any
>>> policy changes at JVM level. You'll register the providers in the
>>> priority
>>> order and can still have 

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Imagine a scenario like - We know we have a custom KMS and as a Kafka owner
we want to comply to using that KMS source to load keys/certs. As a Kafka
owner we know how to integrate with KMS but doesn't necessarily have to
know anything about cipher suites, algorithms, and SSL/TLS implementation.
Going the Provider way requires to know lot more than we should, isn't it?
Not that we would have concern/shy-away knowing those details - but if we
don't have to - why should we?

On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada 
wrote:

> Hi Harsha
>
> We don't have spire (or similar) agents and we do not have keys/certs
> locally on any brokers. To elaborate more on my previous email,
>
> I agree that Java security Providers are used in much broader sense - to
> have a particular implementation of an algorithm, use specific cipher
> suites for SSL , OR  in our current team's case have a particular way to
> leverage pre-generated SSL sessions. However, the scope of our KIP (486) is
> much restricted than that. We merely intend to provide a custom
> keystore/truststore for our SSL connections and not really worry about
> underlying specific SSL/TLS implementation.  This simplifies it a lot for
> us to keep the concerns separate and clear.
>
> I feel our approach is more complimentary such that it allows for using
> keystores of choice while retaining the flexibility to use any
> underlying/available Provider for actually making the SSL call.
>
> We agree with KIP-492's approach based on Providers (and Java's
> recommendation), but also strongly believe that our approach can compliment
> it very effectively for reasons explained above.
>
> Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani 
> wrote:
>
>> Hi Maulin,
>>
>> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> wrote:
>>
>> > Hi Harsha
>> >
>> > The reason we rejected the SslProvider route is that - we only needed a
>> > custom way to load keys/certs. Not touch any policy that existing
>> Providers
>> > govern like SunJSSE Provider.
>> >
>>
>> We have exactly the same requirements to load certs and keys through spire
>> agent. We used security.provider to do that exactly. I am not sure why you
>> would be modifying any policies provided by default SunJSSE provider.  Can
>> you give me an example of having custom provider that will override an
>> existing policy in  SunJSSE provider.
>>
>> As pointed out earlier, this kip
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> allows
>> you to  load security.provider through config.
>> Take a look at the examples I gave before
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>> It registers KeyManagerFactory and TrustManagerFactory and Keystore
>> algorithm.
>> Implement your custom way of loading Keystore in here
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>
>> and Trust manager like here
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>
>> In your Kafka client  you can set the security.provider to your custom
>> implementation and with this fix
>> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>
>> All of this is in your clients and broker side and do not need to touch
>> any
>> policy changes at JVM level. You'll register the providers in the priority
>> order and can still have SunJSSE provider and have your custom provider to
>> implement the key and trust managers.
>>
>>
>>
>> The ask here is different than KIP-492. We don't have any need to
>> > modify/specify the algorithm parameter. Does that make sense?
>> >
>>
>> The ask in KIP is introducing new interfaces where the KIP's
>> goal/motivation can be achieved through the security.provider and we
>> worked
>> on similar goal without touching any Keystore or Truststore interfaces.
>> My advise is against changing or introducing new interfaces when it can
>> work through security.provider.
>>
>> Thanks,
>> Harsha
>>
>>
>> Thanks
>> > Maulin
>> >
>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani 
>> > wr

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Harsha

We don't have spire (or similar) agents and we do not have keys/certs
locally on any brokers. To elaborate more on my previous email,

I agree that Java security Providers are used in much broader sense - to
have a particular implementation of an algorithm, use specific cipher
suites for SSL , OR  in our current team's case have a particular way to
leverage pre-generated SSL sessions. However, the scope of our KIP (486) is
much restricted than that. We merely intend to provide a custom
keystore/truststore for our SSL connections and not really worry about
underlying specific SSL/TLS implementation.  This simplifies it a lot for
us to keep the concerns separate and clear.

I feel our approach is more complimentary such that it allows for using
keystores of choice while retaining the flexibility to use any
underlying/available Provider for actually making the SSL call.

We agree with KIP-492's approach based on Providers (and Java's
recommendation), but also strongly believe that our approach can compliment
it very effectively for reasons explained above.

Thanks
Maulin

On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani  wrote:

> Hi Maulin,
>
> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> maulin.vasav...@gmail.com>
> wrote:
>
> > Hi Harsha
> >
> > The reason we rejected the SslProvider route is that - we only needed a
> > custom way to load keys/certs. Not touch any policy that existing
> Providers
> > govern like SunJSSE Provider.
> >
>
> We have exactly the same requirements to load certs and keys through spire
> agent. We used security.provider to do that exactly. I am not sure why you
> would be modifying any policies provided by default SunJSSE provider.  Can
> you give me an example of having custom provider that will override an
> existing policy in  SunJSSE provider.
>
> As pointed out earlier, this kip
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> allows
> you to  load security.provider through config.
> Take a look at the examples I gave before
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> It registers KeyManagerFactory and TrustManagerFactory and Keystore
> algorithm.
> Implement your custom way of loading Keystore in here
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>
> and Trust manager like here
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>
> In your Kafka client  you can set the security.provider to your custom
> implementation and with this fix
> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>
> All of this is in your clients and broker side and do not need to touch any
> policy changes at JVM level. You'll register the providers in the priority
> order and can still have SunJSSE provider and have your custom provider to
> implement the key and trust managers.
>
>
>
> The ask here is different than KIP-492. We don't have any need to
> > modify/specify the algorithm parameter. Does that make sense?
> >
>
> The ask in KIP is introducing new interfaces where the KIP's
> goal/motivation can be achieved through the security.provider and we worked
> on similar goal without touching any Keystore or Truststore interfaces.
> My advise is against changing or introducing new interfaces when it can
> work through security.provider.
>
> Thanks,
> Harsha
>
>
> Thanks
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani 
> > wrote:
> >
> > In your KIP you added security. provider as rejected alternative and
> > specified "its not the correct way". Do you mind explaining why its not?
> I
> > didn't find any evidence in Java docs to say so. Contrary to your
> statement
> > it does say in the java docs
> > " However, please note that a provider can be used to implement any
> > security service in Java that uses a pluggable architecture with a choice
> > of implementations that fit underneath."
> >
> > Java Security Providers have been used by other projects to provide such
> > integration . I am not sure if you looked into Spiffe project to
> > efficiently distribute certificates but here is an example of Java
> provider
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> spiffe-security-provider/src/main/java/spiffe/api/provid

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Hi Harsha

The reason we rejected the SslProvider route is that - we only needed a
custom way to load keys/certs. Not touch any policy that existing Providers
govern like SunJSSE Provider.

The ask here is different than KIP-492. We don't have any need to
modify/specify the algorithm parameter. Does that make sense?

Thanks
Maulin

On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani  wrote:

> In your KIP you added security. provider as rejected alternative and
> specified "its not the correct way". Do you mind explaining why its not? I
> didn't find any evidence in Java docs to say so. Contrary to your statement
> it does say in the java docs
> " However, please note that a provider can be used to implement any
> security service in Java that uses a pluggable architecture with a choice
> of implementations that fit underneath."
>
> Java Security Providers have been used by other projects to provide such
> integration . I am not sure if you looked into Spiffe project to
> efficiently distribute certificates but here is an example of Java provider
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> which
> obtains certificates from local daemons.
> These integrations are being used in Tomcat, Jetty etc..  We are also using
> Security provider to do the same in our Kafka clusters. So unless I see
> more evidence why security.provider doesn't work for you
> adding new interfaces while there exists more cleaner way of  achieving the
> goals of this KIP  is unnecessary and breaks the well known security
> interfaces provided by Java itself.
>
> Thanks,
> Harsha
>
>
> On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani 
> wrote:
>
> > Hi Maulin,
> >Not sure if you looked at my previous replies. This
> changes
> > are not required as there is already security Provider to do what you are
> > proposing.  This KIP https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also
> > addresses easy registration of such providers.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada  > com> wrote:
> >
> > Bump! Can somebody please review this?
> >
> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > wrote:
> >
> > Bump! Can somebody please review this?
> >
> >
>

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Bump! Can somebody please review this?

On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada 
wrote:

> Bump! Can somebody please review this?
>

Re: [DISCUSS] KIP-317: Transparent Data Encryption

[Maulin Vasavada]

Hi Sönke Liebau


Thanks for the great detailed documentation. However, I feel by leaving the
KMS outside of Kafka might simplify the whole thing to a great extent. If
the broker is not going to touch the encrypted messages, why would we put
any dependency of KMS interfaces on the Broker. We have experimented doing
end-to-end message encryption and we used topic level keys and message
encryption with serializer wrapper which encrypts each message before
serializing. The serializer wrapper have to integrate with required KMS we
use internally and that was all.

However one key observation we had was - if we could do encryption at
'batch' level instead of 'per-message' it can perform much better
(depending upon batch sizing). We didn't experiment with that though.

Thanks
Maulin

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Bump! Can somebody please review this?

[jira] [Created] (KAFKA-8621) KIP-486: Support for pluggable KeyStore and TrustStore

[MAULIN VASAVADA (JIRA)]

MAULIN VASAVADA created KAFKA-8621:
--

 Summary: KIP-486: Support for pluggable KeyStore and TrustStore
 Key: KAFKA-8621
 URL: https://issues.apache.org/jira/browse/KAFKA-8621
 Project: Kafka
  Issue Type: New Feature
Reporter: MAULIN VASAVADA


[https://cwiki.apache.org/confluence/display/KAFKA/KIP-486%3A+Support+for+pluggable+KeyStore+and+TrustStore]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[maulin . vasavada]

Hi all,

Starting this thread to discuss about KIP 486 - 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-486%3A+Support+for+pluggable+KeyStore+and+TrustStore

Thanks
Maulin

Request for Permission to Create KIP

[Maulin Vasavada]

Hi

Can you please give me permission to Create KIP?

My username: maulin.vasavada

Thank you.
Maulin