Hi Colin.
<<< It seems like this [--zk-tls-config-file information] could just appear
in a configuration file, which all of these tools already accept (I think)
ZkSecurityMigrator has no such property file facility; adding a
"--zk-tls-config-file" parameter is exactly for this purpose. If we add
that to ZkSecurityMigrator then it is trivial to add it to other commands
(the same code is simply reused; it ends up being just a few extra lines).
I do not see any parameter in the other two commands to adjust the ZK
connection; ConfigCommand accepts a "--command-config" flag, but according
to the code "This is used only with --bootstrap-server option for
describing and altering broker configs."
I do agree there would be no need to add "--zk-tls-config-file" to
ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
for the next release.
ConfigCommand supports the "--bootstrap-server" option and will have its
direct ZooKeeper access formally deprecated as per KIP-555, but the special
use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
prior to starting Kafka will still exist, so it feels like while
"--zk-tls-config-file" would never be used except for this use case it
could probably still be added for this particular situation.
Ron
P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
sent) did not go through. I am trying to get emails through now to move
this discussion forward.
On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cmcc...@apache.org> wrote:
> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > Hi everyone. I would like to make the following changes to the KIP.
> >
> > MOTIVATION:
> > Include a statement that it will be difficult in the short term to
> > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> (which
> > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > with encrypted passwords in Zookeeper is an explicitly-supported use
> case;
> > therefore it is in scope to be able to securely configure the CLI tools
> > that still leverage non-deprecated direct Zookeeper communication for TLS
> > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > zookeeper-security-migration.sh).
>
> Hi Ron,
>
> Thanks for the KIP.
>
> About deprecations:
>
> * zookeeper-security-migration: clearly, deprecating ZK access in this one
> would not make sense, since it would defeat the whole point of the tool :)
>
> * kafka-reassign-partitions: ZK access should be deprecated here. KIP-455
> implementation has dragged a bit, but this should get done soon. Certainly
> before the next release.
>
> * kafka-configs: I think ZK access should be deprecated here as well. I
> agree there is a super-special bootstrapping case here, but that should
> have its own tool, not use kafka-configs.
>
> I will post a separate KIP for this, though.
>
> >
> > GOALS:
> > Support the secure configuration of TLS-encrypted communication between
> > Zookeeper and:
> > a) Kafka brokers
> > b) The three CLI tools mentioned above that still support direct,
> > non-deprecated communication to Zookeeper
> > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > communication in CLI tools as part of this KIP; such work will occur in
> > future KIPs instead.
> >
> > PUBLIC INTERFACES:
> > 1) The following new broker configurations will be recognized.
> > zookeeper.client.secure (default value = false, for backwards
> > compatibility)
> > zookeeper.clientCnxnSocket
> > zookeeper.ssl.keyStore.location
> > zookeeper.ssl.keyStore.password
> > zookeeper.ssl.trustStore.location
> > zookeeper.ssl.trustStore.password
> > It will be an error for any of the last 5 values to be left unspecified
> if
> > zookeeper.client.secure is explicitly set to true.
> >
> > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> supports
> > the ability to connect to a different Zookeeper instance than the one the
> > brokers use. We therefore also add the following optional configs, which
> > override the corresponding ones from above when present:
> > authorizer.zookeeper.client.secure
> > authorizer.zookeeper.clientCnxnSocket
> > authorizer.zookeeper.ssl.keyStore.location
> > authorizer.zookeeper.ssl.keyStore.password
> > authorizer.zookeeper.ssl.trustStore.location
> > authorizer.zookeeper.ssl.trustStore.password
> >
> > 3) The three CLI tools mentioned above will support a new
> --zk-tls-config-file
> > <String: Zookeeper TLS configuration file path>" option. The following
> > properties will be recognized in that file, and unrecognized properties
> > will be ignored to allow the possibility of pointing zk-tls-config-file
> at
> > the broker's config file.
> > zookeeper.client.secure (default value = false)
> > zookeeper.clientCnxnSocket
> > zookeeper.ssl.keyStore.location
> > zookeeper.ssl.keyStore.password
> > zookeeper.ssl.trustStore.location
> > zookeeper.ssl.trustStore.password
> > It will be an error for any of the last 5 values to be left unspecified
> if
> > zookeeper.client.secure is explicitly set to true.
>
> Do we really need a --zk-tls-config-file flag? It seems like this could
> just appear in a configuration file, which all of these tools already
> accept (I think).
>
> best,
> Colin
>
>
> >
> > Ron
> >
> > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rndg...@gmail.com> wrote:
> >
> > > Hi everyone. Let's get this discussion going again now that Kafka 2.4
> > > with Zookeeper 3.5.6 is out.
> > >
> > > First, regarding the KIP number, the other KIP that was using this
> number
> > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > discussion. I've updated the Kafka Improvement Proposal page to list
> this
> > > KIP in the 515 slot, so we're all set there.
> > >
> > > Regarding the actual issues under discussion, I think there are some
> > > things we should clarify.
> > >
> > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> Kafka
> > > 2.4 -- the problem is that configuration information has to be passed
> via
> > > system properties as "-D" command line options on the java invocation
> of
> > > the broker, and those are not secure (anyone with access to the box
> can see
> > > the command line used to invoke the broker); the configuration includes
> > > sensitive information (e.g. a keystore password), so we need a secure
> > > mechanism for passing the configuration values. I believe the real
> > > motivation for this KIP is to harden the configuration mechanism for
> > > Zookeeper TLS connectivity.
> > >
> > > 2) I believe the list of CLI tools that continue to use direct
> Zookeeper
> > > connectivity in a non-deprecated fashion is:
> > > a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > b)
> > >
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > >
> > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
> > > explicitly states in a comment that a supported use case is
> bootstrapping a
> > > Kafka cluster with encrypted passwords in Zookeeper (see
> > >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> ).
> > > This means it will be especially difficult to deprecate this particular
> > > direct Zookeeper connectivity without a different storage mechanism for
> > > dynamic configuration values being available (i.e. the self-managed
> quorum
> > > referred to in KIP-500).
> > >
> > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > configuration in both Kafka and in CLI tools compared to trying to
> > > deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
> > > especially when ConfigCommand has no obvious short-term path to
> deprecation.
> > >
> > > Regarding how to harden the TLS configuration, adding the appropriate
> > > configs to Kafka is an obvious choice for the brokers. Not that these
> > > values cannot be dynamically reconfigurable because the values would be
> > > required to connect to Zookeeper via TLS in the first place.
> > >
> > > Regarding how to harden the TLS configuration for the 3 remaining CLI
> > > tools, it would be relatively straightforward to add support for a
> > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > properties file path>" option to each one.
> > >
> > > Thoughts?
> > >
> > > Ron
> > >
> > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <pere.ur...@gmail.com
> >
> > > wrote:
> > >
> > >> True, the number is duplicated.
> > >>
> > >> do you know how can we get the problem fix? I was not aware, I
> missed,
> > >> sorry the need to add the KIP to the table.
> > >>
> > >> -- Pere
> > >>
> > >> Missatge de Jorge Esteban Quilcate Otoya <quilcate.jo...@gmail.com>
> del
> > >> dia
> > >> dt., 3 de set. 2019 a les 13:43:
> > >>
> > >> > Hi Pere,
> > >> >
> > >> > Have you add your KIP to the list here
> > >> >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >> ?
> > >> >
> > >> > I found the KIP number assigned to another.
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> pere.ur...@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> Thanks for your time Harsha,
> > >> >> anyone else with comments? looking forward to hearing from you.
> > >> >>
> > >> >> Stupid question: when do you move from discussion to vote?
> > >> >>
> > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
> > >> d’ag.
> > >> >> 2019 a les 21:59:
> > >> >>
> > >> >> > Thanks Pere. KIP looks good to me.
> > >> >> > -Harsha
> > >> >> >
> > >> >> >
> > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > >> >> pere.ur...@gmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> >> Not really,
> > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> major
> > >> >> >> changes. But if you pass a properties file, then this takes
> > >> precedence
> > >> >> over
> > >> >> >> the other, with the idea that you can do sasl as well with the
> > >> >> properties
> > >> >> >> files.
> > >> >> >>
> > >> >> >> Makes sense?
> > >> >> >>
> > >> >> >> -- Pere
> > >> >> >>
> > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv.,
> 30
> > >> >> d’ag.
> > >> >> >> 2019 a les 19:00:
> > >> >> >>
> > >> >> >>> Hi Pere,
> > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for
> Kafka
> > >> >> makes
> > >> >> >>> sense.
> > >> >> >>> "The changes are planned to be introduced in a compatible way,
> by
> > >> >> >>> keeping the current JAAS variable precedence."
> > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> file
> > >> with
> > >> >> >>> Client section it will take precedence over zookeeper SSL
> configs?
> > >> >> >>>
> > >> >> >>> Thanks,
> > >> >> >>> Harsha
> > >> >> >>>
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > >> >> pere.ur...@gmail.com>
> > >> >> >>> wrote:
> > >> >> >>>
> > >> >> >>>> Hi,
> > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> planned
> > >> for
> > >> >> >>>> September. I think it would be really awesome to have this for
> > >> this
> > >> >> >>>> release, do you think we can make it?
> > >> >> >>>>
> > >> >> >>>> -- Pere
> > >> >> >>>>
> > >> >> >>>> Missatge de Pere Urbón Bayes <pere.ur...@gmail.com> del dia
> dj.,
> > >> 29
> > >> >> >>>> d’ag. 2019 a les 20:10:
> > >> >> >>>>
> > >> >> >>>> Hi,
> > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> really
> > >> need
> > >> >> >>>> to the process. Looking forward to hearing from you and learn
> the
> > >> >> best
> > >> >> >>>> ropes here.
> > >> >> >>>>
> > >> >> >>>> I would like to propose this KIP-515 to enable the
> > >> ZookeeperClients
> > >> >> to
> > >> >> >>>> take full advantage of the TLS communication in the new
> Zookeeper
> > >> >> 3.5.5.
> > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> that
> > >> >> without
> > >> >> >>>> this change will not work with TLS, disabling users to use
> ACLs
> > >> when
> > >> >> the
> > >> >> >>>> Zookeeper cluster use TLS.
> > >> >> >>>>
> > >> >> >>>> link:
> > >> >> >>>>
> > >> >> >>>>
> > >> >>
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > >> >> >>>>
> > >> >> >>>> Looking forward to hearing from you on this,
> > >> >> >>>>
> > >> >> >>>> /cheers
> > >> >> >>>>
> > >> >> >>>> --
> > >> >> >>>> Pere Urbon-Bayes
> > >> >> >>>> Software Architect
> > >> >> >>>> http://www.purbon.com
> > >> >> >>>> https://twitter.com/purbon
> > >> >> >>>> https://www.linkedin.com/in/purbon/
> > >> >> >>>>
> > >> >> >>>> --
> > >> >> >>>> Pere Urbon-Bayes
> > >> >> >>>> Software Architect
> > >> >> >>>> http://www.purbon.com
> > >> >> >>>> https://twitter.com/purbon
> > >> >> >>>> https://www.linkedin.com/in/purbon/
> > >> >> >>>>
> > >> >> >>>
> > >> >> >>>
> > >> >> >>
> > >> >> >> --
> > >> >> >> Pere Urbon-Bayes
> > >> >> >> Software Architect
> > >> >> >> http://www.purbon.com
> > >> >> >> https://twitter.com/purbon
> > >> >> >> https://www.linkedin.com/in/purbon/
> > >> >> >>
> > >> >> >
> > >> >> >
> > >> >>
> > >> >> --
> > >> >> Pere Urbon-Bayes
> > >> >> Software Architect
> > >> >> http://www.purbon.com
> > >> >> https://twitter.com/purbon
> > >> >> https://www.linkedin.com/in/purbon/
> > >> >>
> > >> >
> > >>
> > >> --
> > >> Pere Urbon-Bayes
> > >> Software Architect
> > >> http://www.purbon.com
> > >> https://twitter.com/purbon
> > >> https://www.linkedin.com/in/purbon/
> > >>
> > >
> >
>
