Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Rajini. Thanks. Yes, I’ll update the KIP, and then I’ll start the vote. Ron > On Jan 20, 2020, at 6:30 AM, Rajini Sivaram wrote: > > Hi Ron, > > Good point. Keystore and truststore configs (type/location/password) are > dynamically reconfigurable. A few of them like protocol, enabled.protocols, > cipher.suites and endpoint.identification.algorithm are not. But to keep it > simple, it may be better not to inherit any of the SSL configs. Perhaps > just document that in Rejected Alternatives? > >> On Sat, Jan 18, 2020 at 12:43 AM Ron Dagostino wrote: >> >> Thanks, Rajini. Still not sure what the answer is, but I thought of a >> couple more issues with config inheritance that I wanted to raise. >> The first is a minor issue, but just to document it (and I will add it >> to the KIP as well), ZooKeeper does not support a key password that >> differs from the keystore password. So inheriting from Kafka's TLS >> configs would only work if those configs use the same key and keystore >> password. Again, not a big deal -- just something to document. >> >> However... >> >> I think there may be a fundamental problem with inheriting broker >> configs for ZooKeeper access. The problem is that the broker's TLS >> configs are dynamically reconfigurable, and the dynamic values >> themselves are stored in ZooKeeper. It would be impossible to know >> those values without connecting to ZooKeeper, and it would be >> impossible to connect to ZooKeeper without the values. Given this, I >> wonder if inheritance is impossible in this particular case. >> >> Ron >> >>> On Jan 17, 2020, at 5:39 PM, Rajini Sivaram >> wrote: >>> >>> Hi Ron, >>> >>> For Kafka, we moved from hostname verification disabled-by-default to >>> enabled-by-default under >>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default >> . >>> So we have tested empty String for disabling hostname verification and we >>> know that it works, not only in test environments, but also in actual >>> deployments. Basically, "ssl.endpoint.identification.algorithm=" in the >>> file results in a config value of empty String and not setting it results >>> in the default value of HTTPS since the value is null.I would probably do >>> the same for the ZK config. If you are creating certificates for >> ZooKeeper >>> without hostnames, then you may be doing the same for Kafka too and in >> both >>> cases, the config would be an empty string. This is insecure anyway and >> we >>> would really expect this only in test environments. Not sure if it is >> worth >>> making an exception for this in terms of consistent config names. >>> On Fri, Jan 17, 2020 at 2:37 PM Ron Dagostino >> wrote: Hi again, Rajini. I've updated the KIP, and while doing it I became concerned about using zookeeper.ssl.endpoint.identification.algorithm for enabling/disabling hostname verification. The KIP reflects what we decided, but upon reading it, I wonder if it is workable. Here's what it says for this config: "Specifies whether to enable hostname verification in the ZooKeeper TLS negotiation process, with (case-insensitively) "https" meaning ZooKeeper hostname verification is enabled and an explicit blank value meaning it is disabled (disabling it is only recommended for testing purposes). Overrides any explicit "true" or "false" value set via the zookeeper.ssl.hostnameVerification system property (true implying https and false implying blank) and inherits the value of ssl.endpoint.identification.algorithm (if any) if no explicit value or non-value is set both here and via the system property." I wonder if someone sets an explicit empty value (zookeeper.ssl.endpoint.identification.algorithm=) in the config file, will we be able to distinguish between that and a config file that doesn't have it at all? And even if we can, this feels pretty forced to me, like we are bending over backwards for consistency when "zookeeper.ssl.hostname.verification.enable" is a natural fit. I doubt this config is going to be used very much, so it really does feel to me like we should just go with the natural fit and ditch the compatibility and inheritance for this one config. Ron > On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino >> wrote: > > Hi Rajini. I’ll submit a documentation PR to clarify the Kafka >> behavior of #1 and will adopt the same config for ZK. I agree now we should >> inherit for AclAuthorizer too — I was just stuck on the “no inheritance” idea >> more than I realized, and while the ZK quorums are different that doesn’t necessarily mean the same client identity wouldn’t be valid on both >> (and if not then the configs can still be overridden). > > I’ll update the KIP today and let this version soak for a day or two, and the
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Ron, Good point. Keystore and truststore configs (type/location/password) are dynamically reconfigurable. A few of them like protocol, enabled.protocols, cipher.suites and endpoint.identification.algorithm are not. But to keep it simple, it may be better not to inherit any of the SSL configs. Perhaps just document that in Rejected Alternatives? On Sat, Jan 18, 2020 at 12:43 AM Ron Dagostino wrote: > Thanks, Rajini. Still not sure what the answer is, but I thought of a > couple more issues with config inheritance that I wanted to raise. > The first is a minor issue, but just to document it (and I will add it > to the KIP as well), ZooKeeper does not support a key password that > differs from the keystore password. So inheriting from Kafka's TLS > configs would only work if those configs use the same key and keystore > password. Again, not a big deal -- just something to document. > > However... > > I think there may be a fundamental problem with inheriting broker > configs for ZooKeeper access. The problem is that the broker's TLS > configs are dynamically reconfigurable, and the dynamic values > themselves are stored in ZooKeeper. It would be impossible to know > those values without connecting to ZooKeeper, and it would be > impossible to connect to ZooKeeper without the values. Given this, I > wonder if inheritance is impossible in this particular case. > > Ron > > > On Jan 17, 2020, at 5:39 PM, Rajini Sivaram > wrote: > > > > Hi Ron, > > > > For Kafka, we moved from hostname verification disabled-by-default to > > enabled-by-default under > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default > . > > So we have tested empty String for disabling hostname verification and we > > know that it works, not only in test environments, but also in actual > > deployments. Basically, "ssl.endpoint.identification.algorithm=" in the > > file results in a config value of empty String and not setting it results > > in the default value of HTTPS since the value is null.I would probably do > > the same for the ZK config. If you are creating certificates for > ZooKeeper > > without hostnames, then you may be doing the same for Kafka too and in > both > > cases, the config would be an empty string. This is insecure anyway and > we > > would really expect this only in test environments. Not sure if it is > worth > > making an exception for this in terms of consistent config names. > > > >> On Fri, Jan 17, 2020 at 2:37 PM Ron Dagostino > wrote: > >> > >> Hi again, Rajini. I've updated the KIP, and while doing it I became > >> concerned about using zookeeper.ssl.endpoint.identification.algorithm > >> for enabling/disabling hostname verification. The KIP reflects what > >> we decided, but upon reading it, I wonder if it is workable. Here's > >> what it says for this config: > >> > >> "Specifies whether to enable hostname verification in the ZooKeeper > >> TLS negotiation process, with (case-insensitively) "https" meaning > >> ZooKeeper hostname verification is enabled and an explicit blank value > >> meaning it is disabled (disabling it is only recommended for testing > >> purposes). Overrides any explicit "true" or "false" value set via the > >> zookeeper.ssl.hostnameVerification system property (true > >> implying https and false implying blank) and inherits the value of > >> ssl.endpoint.identification.algorithm (if any) if no > >> explicit value or non-value is set both here and via the system > >> property." > >> > >> I wonder if someone sets an explicit empty value > >> (zookeeper.ssl.endpoint.identification.algorithm=) in the config file, > >> will we be able to distinguish between that and a config file that > >> doesn't have it at all? And even if we can, this feels pretty forced > >> to me, like we are bending over backwards for consistency when > >> "zookeeper.ssl.hostname.verification.enable" is a natural fit. I > >> doubt this config is going to be used very much, so it really does > >> feel to me like we should just go with the natural fit and ditch the > >> compatibility and inheritance for this one config. > >> > >> Ron > >> > >>> On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino > wrote: > >>> > >>> Hi Rajini. I’ll submit a documentation PR to clarify the Kafka > behavior > >> of #1 and will adopt the same config for ZK. I agree now we should > inherit > >> for AclAuthorizer too — I was just stuck on the “no inheritance” idea > more > >> than I realized, and while the ZK quorums are different that doesn’t > >> necessarily mean the same client identity wouldn’t be valid on both > (and if > >> not then the configs can still be overridden). > >>> > >>> I’ll update the KIP today and let this version soak for a day or two, > >> and then I’ll start a vote if no other issues/comments arise. > >>> > >>> Ron > >>> > On Jan 17, 2020, at 6:08 AM, Rajini Sivaram > >> wrote: > > Hi Ron, > > Unresolved item #1: Yes, Kafka di
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Thanks, Rajini. Still not sure what the answer is, but I thought of a couple more issues with config inheritance that I wanted to raise. The first is a minor issue, but just to document it (and I will add it to the KIP as well), ZooKeeper does not support a key password that differs from the keystore password. So inheriting from Kafka's TLS configs would only work if those configs use the same key and keystore password. Again, not a big deal -- just something to document. However... I think there may be a fundamental problem with inheriting broker configs for ZooKeeper access. The problem is that the broker's TLS configs are dynamically reconfigurable, and the dynamic values themselves are stored in ZooKeeper. It would be impossible to know those values without connecting to ZooKeeper, and it would be impossible to connect to ZooKeeper without the values. Given this, I wonder if inheritance is impossible in this particular case. Ron > On Jan 17, 2020, at 5:39 PM, Rajini Sivaram wrote: > > Hi Ron, > > For Kafka, we moved from hostname verification disabled-by-default to > enabled-by-default under > https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default. > So we have tested empty String for disabling hostname verification and we > know that it works, not only in test environments, but also in actual > deployments. Basically, "ssl.endpoint.identification.algorithm=" in the > file results in a config value of empty String and not setting it results > in the default value of HTTPS since the value is null.I would probably do > the same for the ZK config. If you are creating certificates for ZooKeeper > without hostnames, then you may be doing the same for Kafka too and in both > cases, the config would be an empty string. This is insecure anyway and we > would really expect this only in test environments. Not sure if it is worth > making an exception for this in terms of consistent config names. > >> On Fri, Jan 17, 2020 at 2:37 PM Ron Dagostino wrote: >> >> Hi again, Rajini. I've updated the KIP, and while doing it I became >> concerned about using zookeeper.ssl.endpoint.identification.algorithm >> for enabling/disabling hostname verification. The KIP reflects what >> we decided, but upon reading it, I wonder if it is workable. Here's >> what it says for this config: >> >> "Specifies whether to enable hostname verification in the ZooKeeper >> TLS negotiation process, with (case-insensitively) "https" meaning >> ZooKeeper hostname verification is enabled and an explicit blank value >> meaning it is disabled (disabling it is only recommended for testing >> purposes). Overrides any explicit "true" or "false" value set via the >> zookeeper.ssl.hostnameVerification system property (true >> implying https and false implying blank) and inherits the value of >> ssl.endpoint.identification.algorithm (if any) if no >> explicit value or non-value is set both here and via the system >> property." >> >> I wonder if someone sets an explicit empty value >> (zookeeper.ssl.endpoint.identification.algorithm=) in the config file, >> will we be able to distinguish between that and a config file that >> doesn't have it at all? And even if we can, this feels pretty forced >> to me, like we are bending over backwards for consistency when >> "zookeeper.ssl.hostname.verification.enable" is a natural fit. I >> doubt this config is going to be used very much, so it really does >> feel to me like we should just go with the natural fit and ditch the >> compatibility and inheritance for this one config. >> >> Ron >> >>> On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino wrote: >>> >>> Hi Rajini. I’ll submit a documentation PR to clarify the Kafka behavior >> of #1 and will adopt the same config for ZK. I agree now we should inherit >> for AclAuthorizer too — I was just stuck on the “no inheritance” idea more >> than I realized, and while the ZK quorums are different that doesn’t >> necessarily mean the same client identity wouldn’t be valid on both (and if >> not then the configs can still be overridden). >>> >>> I’ll update the KIP today and let this version soak for a day or two, >> and then I’ll start a vote if no other issues/comments arise. >>> >>> Ron >>> On Jan 17, 2020, at 6:08 AM, Rajini Sivaram >> wrote: Hi Ron, Unresolved item #1: Yes, Kafka disables hostname verification if "ssl.endpoint.identification.algorithm" is an empty string. Unresolved item #2: Yes, those 9 plus hostname verification. If we do decide to inherit Kafka configs, wouldn't we inherit these 10 in AclAuthorizer as well? Unresolved item #3: Agree, following up on the discussion thread for KIP-553. > On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino >> wrote: > > Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration > value that turns it on is zookeeper.ssl.client.enable (which used to > be ca
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Ron, For Kafka, we moved from hostname verification disabled-by-default to enabled-by-default under https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default. So we have tested empty String for disabling hostname verification and we know that it works, not only in test environments, but also in actual deployments. Basically, "ssl.endpoint.identification.algorithm=" in the file results in a config value of empty String and not setting it results in the default value of HTTPS since the value is null.I would probably do the same for the ZK config. If you are creating certificates for ZooKeeper without hostnames, then you may be doing the same for Kafka too and in both cases, the config would be an empty string. This is insecure anyway and we would really expect this only in test environments. Not sure if it is worth making an exception for this in terms of consistent config names. On Fri, Jan 17, 2020 at 2:37 PM Ron Dagostino wrote: > Hi again, Rajini. I've updated the KIP, and while doing it I became > concerned about using zookeeper.ssl.endpoint.identification.algorithm > for enabling/disabling hostname verification. The KIP reflects what > we decided, but upon reading it, I wonder if it is workable. Here's > what it says for this config: > > "Specifies whether to enable hostname verification in the ZooKeeper > TLS negotiation process, with (case-insensitively) "https" meaning > ZooKeeper hostname verification is enabled and an explicit blank value > meaning it is disabled (disabling it is only recommended for testing > purposes). Overrides any explicit "true" or "false" value set via the > zookeeper.ssl.hostnameVerification system property (true > implying https and false implying blank) and inherits the value of > ssl.endpoint.identification.algorithm (if any) if no > explicit value or non-value is set both here and via the system > property." > > I wonder if someone sets an explicit empty value > (zookeeper.ssl.endpoint.identification.algorithm=) in the config file, > will we be able to distinguish between that and a config file that > doesn't have it at all? And even if we can, this feels pretty forced > to me, like we are bending over backwards for consistency when > "zookeeper.ssl.hostname.verification.enable" is a natural fit. I > doubt this config is going to be used very much, so it really does > feel to me like we should just go with the natural fit and ditch the > compatibility and inheritance for this one config. > > Ron > > On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino wrote: > > > > Hi Rajini. I’ll submit a documentation PR to clarify the Kafka behavior > of #1 and will adopt the same config for ZK. I agree now we should inherit > for AclAuthorizer too — I was just stuck on the “no inheritance” idea more > than I realized, and while the ZK quorums are different that doesn’t > necessarily mean the same client identity wouldn’t be valid on both (and if > not then the configs can still be overridden). > > > > I’ll update the KIP today and let this version soak for a day or two, > and then I’ll start a vote if no other issues/comments arise. > > > > Ron > > > > > On Jan 17, 2020, at 6:08 AM, Rajini Sivaram > wrote: > > > > > > Hi Ron, > > > > > > Unresolved item #1: Yes, Kafka disables hostname verification if > > > "ssl.endpoint.identification.algorithm" is an empty string. > > > > > > Unresolved item #2: Yes, those 9 plus hostname verification. If we do > > > decide to inherit Kafka configs, wouldn't we inherit these 10 in > > > AclAuthorizer as well? > > > > > > Unresolved item #3: Agree, following up on the discussion thread for > > > KIP-553. > > > > > > > > >> On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino > wrote: > > >> > > >> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration > > >> value that turns it on is zookeeper.ssl.client.enable (which used to > > >> be called zookeeper.client.secure before we renamed everything to be > > >> consistent with Kafka's config naming style and eliminate confusion). > > >> The default value for this is false. Everything else related to TLS > > >> has no effect unless this is explicitly set to true, and this value > > >> does not inherit from anywhere, > > >> > > >> Ron > > >> > > >>> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe > wrote: > > >>> > > On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote: > > Hi Ron/Colin, > > > > Thanks for the responses. I think we need to consider these cases: > > > > 1) The recommended approach: This is the one we document. Configs > > consistent with Kafka make sense here. It seems a lot more > confusing to > > configure `ssl.cipher.suites`, `ssl.enabled.protocols`, ` > > zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` > with > > multiple naming inconsistencies in server.properties. For users who > > >> have > > never used ZK system properties, this is clearly simpl
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi again, Rajini. I've updated the KIP, and while doing it I became concerned about using zookeeper.ssl.endpoint.identification.algorithm for enabling/disabling hostname verification. The KIP reflects what we decided, but upon reading it, I wonder if it is workable. Here's what it says for this config: "Specifies whether to enable hostname verification in the ZooKeeper TLS negotiation process, with (case-insensitively) "https" meaning ZooKeeper hostname verification is enabled and an explicit blank value meaning it is disabled (disabling it is only recommended for testing purposes). Overrides any explicit "true" or "false" value set via the zookeeper.ssl.hostnameVerification system property (true implying https and false implying blank) and inherits the value of ssl.endpoint.identification.algorithm (if any) if no explicit value or non-value is set both here and via the system property." I wonder if someone sets an explicit empty value (zookeeper.ssl.endpoint.identification.algorithm=) in the config file, will we be able to distinguish between that and a config file that doesn't have it at all? And even if we can, this feels pretty forced to me, like we are bending over backwards for consistency when "zookeeper.ssl.hostname.verification.enable" is a natural fit. I doubt this config is going to be used very much, so it really does feel to me like we should just go with the natural fit and ditch the compatibility and inheritance for this one config. Ron On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino wrote: > > Hi Rajini. I’ll submit a documentation PR to clarify the Kafka behavior of > #1 and will adopt the same config for ZK. I agree now we should inherit for > AclAuthorizer too — I was just stuck on the “no inheritance” idea more than I > realized, and while the ZK quorums are different that doesn’t necessarily > mean the same client identity wouldn’t be valid on both (and if not then the > configs can still be overridden). > > I’ll update the KIP today and let this version soak for a day or two, and > then I’ll start a vote if no other issues/comments arise. > > Ron > > > On Jan 17, 2020, at 6:08 AM, Rajini Sivaram wrote: > > > > Hi Ron, > > > > Unresolved item #1: Yes, Kafka disables hostname verification if > > "ssl.endpoint.identification.algorithm" is an empty string. > > > > Unresolved item #2: Yes, those 9 plus hostname verification. If we do > > decide to inherit Kafka configs, wouldn't we inherit these 10 in > > AclAuthorizer as well? > > > > Unresolved item #3: Agree, following up on the discussion thread for > > KIP-553. > > > > > >> On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino wrote: > >> > >> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration > >> value that turns it on is zookeeper.ssl.client.enable (which used to > >> be called zookeeper.client.secure before we renamed everything to be > >> consistent with Kafka's config naming style and eliminate confusion). > >> The default value for this is false. Everything else related to TLS > >> has no effect unless this is explicitly set to true, and this value > >> does not inherit from anywhere, > >> > >> Ron > >> > >>> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe wrote: > >>> > On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote: > Hi Ron/Colin, > > Thanks for the responses. I think we need to consider these cases: > > 1) The recommended approach: This is the one we document. Configs > consistent with Kafka make sense here. It seems a lot more confusing to > configure `ssl.cipher.suites`, `ssl.enabled.protocols`, ` > zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with > multiple naming inconsistencies in server.properties. For users who > >> have > never used ZK system properties, this is clearly simpler. Another > >> config to > consider is `ssl.endpoint.identification.algorithm` which does the same > thing as `zookeeper.ssl.hostnameVerification` - if we are staying > consistent, we should use the Kafka format for this too. I would also > consider using something like `zookeeper.client.ssl.enabled` instead > >> of ` > zookeeper.client.secure` since it is difficult to tell what `secure` > >> means > when you have options to configure Kerberos, ACLs and/or SSL. > > 2) Inheritance using prefixed configs as we do for other prefixed > >> configs > in Kafka: I think it may be useful for configs like `ssl.protocol` and > >> ` > ssl.cipher.suites`. > > 3) Default values: Do we want to use Kafka defaults? The inconsistency > >> we > have is `ssl.protocol/ssl.enabled.protocols` since we still have > >> insecure > protocols enabled for Kafka. We have KIP-553 ( > > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956 > >> ) > open to disable these. It is currently blocked since it also talks > >> about > enabling TLSv1.3 by default and
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Rajini. I’ll submit a documentation PR to clarify the Kafka behavior of #1 and will adopt the same config for ZK. I agree now we should inherit for AclAuthorizer too — I was just stuck on the “no inheritance” idea more than I realized, and while the ZK quorums are different that doesn’t necessarily mean the same client identity wouldn’t be valid on both (and if not then the configs can still be overridden). I’ll update the KIP today and let this version soak for a day or two, and then I’ll start a vote if no other issues/comments arise. Ron > On Jan 17, 2020, at 6:08 AM, Rajini Sivaram wrote: > > Hi Ron, > > Unresolved item #1: Yes, Kafka disables hostname verification if > "ssl.endpoint.identification.algorithm" is an empty string. > > Unresolved item #2: Yes, those 9 plus hostname verification. If we do > decide to inherit Kafka configs, wouldn't we inherit these 10 in > AclAuthorizer as well? > > Unresolved item #3: Agree, following up on the discussion thread for > KIP-553. > > >> On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino wrote: >> >> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration >> value that turns it on is zookeeper.ssl.client.enable (which used to >> be called zookeeper.client.secure before we renamed everything to be >> consistent with Kafka's config naming style and eliminate confusion). >> The default value for this is false. Everything else related to TLS >> has no effect unless this is explicitly set to true, and this value >> does not inherit from anywhere, >> >> Ron >> >>> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe wrote: >>> On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote: Hi Ron/Colin, Thanks for the responses. I think we need to consider these cases: 1) The recommended approach: This is the one we document. Configs consistent with Kafka make sense here. It seems a lot more confusing to configure `ssl.cipher.suites`, `ssl.enabled.protocols`, ` zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with multiple naming inconsistencies in server.properties. For users who >> have never used ZK system properties, this is clearly simpler. Another >> config to consider is `ssl.endpoint.identification.algorithm` which does the same thing as `zookeeper.ssl.hostnameVerification` - if we are staying consistent, we should use the Kafka format for this too. I would also consider using something like `zookeeper.client.ssl.enabled` instead >> of ` zookeeper.client.secure` since it is difficult to tell what `secure` >> means when you have options to configure Kerberos, ACLs and/or SSL. 2) Inheritance using prefixed configs as we do for other prefixed >> configs in Kafka: I think it may be useful for configs like `ssl.protocol` and >> ` ssl.cipher.suites`. 3) Default values: Do we want to use Kafka defaults? The inconsistency >> we have is `ssl.protocol/ssl.enabled.protocols` since we still have >> insecure protocols enabled for Kafka. We have KIP-553 ( >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956 >> ) open to disable these. It is currently blocked since it also talks >> about enabling TLSv1.3 by default and we haven't done sufficient testing for that. Since it will be good to disable the older insecure protocols >> anyway, perhaps we could do that without enabling TLSv1.3 by default in 2.5? >>> >>> Hi Rajini, >>> >>> Inheriting the SSL settings from Kafka would have the effect of changing >> the defaults for almost everyone using SSL from non-SSL ZK to SSL-based ZK, >> right? This seems to expand the scope of the KIP greatly, since it moves >> it from being opt-in to opt-out. Are there any potential negative >> implications of this? >>> >>> best, >>> Colin >>> >>> 4) Backward compatibility for deployments which are using system properties. We shouldn't override system properties with Kafka >> defaults or inherited values. But we do want to override if user configures >> properties explicitly. In the current approach, this was simpler since we just >> had to set the configured values. But if we decide to inherit Kafka configs, >> then we will need to check system properties and update accordingly. But >> that should be doable? On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe >> wrote: >> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote: >> Hi Colin. Two things come to mind with respect to ZooKeeper >> camelCase >> style vs Kafka-style config names for ZooKeeper. First, I think it >> would be desirable for the client configs and broker configs to be >> interoperable. For example, it feels like it would be convenient >> to >> be able to pass the broker's config file to the ZK Security >> Migrator >> tool. I think whatever we use (ZooKeeper camelCase style
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Ron, Unresolved item #1: Yes, Kafka disables hostname verification if "ssl.endpoint.identification.algorithm" is an empty string. Unresolved item #2: Yes, those 9 plus hostname verification. If we do decide to inherit Kafka configs, wouldn't we inherit these 10 in AclAuthorizer as well? Unresolved item #3: Agree, following up on the discussion thread for KIP-553. On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino wrote: > Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration > value that turns it on is zookeeper.ssl.client.enable (which used to > be called zookeeper.client.secure before we renamed everything to be > consistent with Kafka's config naming style and eliminate confusion). > The default value for this is false. Everything else related to TLS > has no effect unless this is explicitly set to true, and this value > does not inherit from anywhere, > > Ron > > On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe wrote: > > > > On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote: > > > Hi Ron/Colin, > > > > > > Thanks for the responses. I think we need to consider these cases: > > > > > > 1) The recommended approach: This is the one we document. Configs > > > consistent with Kafka make sense here. It seems a lot more confusing to > > > configure `ssl.cipher.suites`, `ssl.enabled.protocols`, ` > > > zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with > > > multiple naming inconsistencies in server.properties. For users who > have > > > never used ZK system properties, this is clearly simpler. Another > config to > > > consider is `ssl.endpoint.identification.algorithm` which does the same > > > thing as `zookeeper.ssl.hostnameVerification` - if we are staying > > > consistent, we should use the Kafka format for this too. I would also > > > consider using something like `zookeeper.client.ssl.enabled` instead > of ` > > > zookeeper.client.secure` since it is difficult to tell what `secure` > means > > > when you have options to configure Kerberos, ACLs and/or SSL. > > > > > > 2) Inheritance using prefixed configs as we do for other prefixed > configs > > > in Kafka: I think it may be useful for configs like `ssl.protocol` and > ` > > > ssl.cipher.suites`. > > > > > > 3) Default values: Do we want to use Kafka defaults? The inconsistency > we > > > have is `ssl.protocol/ssl.enabled.protocols` since we still have > insecure > > > protocols enabled for Kafka. We have KIP-553 ( > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956 > ) > > > open to disable these. It is currently blocked since it also talks > about > > > enabling TLSv1.3 by default and we haven't done sufficient testing for > > > that. Since it will be good to disable the older insecure protocols > anyway, > > > perhaps we could do that without enabling TLSv1.3 by default in 2.5? > > > > Hi Rajini, > > > > Inheriting the SSL settings from Kafka would have the effect of changing > the defaults for almost everyone using SSL from non-SSL ZK to SSL-based ZK, > right? This seems to expand the scope of the KIP greatly, since it moves > it from being opt-in to opt-out. Are there any potential negative > implications of this? > > > > best, > > Colin > > > > > > > > > > 4) Backward compatibility for deployments which are using system > > > properties. We shouldn't override system properties with Kafka > defaults or > > > inherited values. But we do want to override if user configures > properties > > > explicitly. In the current approach, this was simpler since we just > had to > > > set the configured values. But if we decide to inherit Kafka configs, > then > > > we will need to check system properties and update accordingly. But > that > > > should be doable? > > > > > > On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe > wrote: > > > > > > > On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote: > > > > > Hi Colin. Two things come to mind with respect to ZooKeeper > camelCase > > > > > style vs Kafka-style config names for ZooKeeper. First, I think it > > > > > would be desirable for the client configs and broker configs to be > > > > > interoperable. For example, it feels like it would be convenient > to > > > > > be able to pass the broker's config file to the ZK Security > Migrator > > > > > tool. I think whatever we use (ZooKeeper camelCase style or Kafka > > > > > style), I think we should use the same for both. > > > > > > > > > > The second thing to think about is the fact that ZooKeeper > > > > > configuration inherently uses Java system properties as a first > pass. > > > > > So while we might switch to Kafka-style configs in the config file > > > > > (e.g. zookeeper.ssl.trust.store.location), the > > > > > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS > > > > > configs are set) is going to look for the camelCase > > > > > zookeeper.ssl.trustStore.location system property and use any value > > > > > there. I think this could be very confusing for people
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration value that turns it on is zookeeper.ssl.client.enable (which used to be called zookeeper.client.secure before we renamed everything to be consistent with Kafka's config naming style and eliminate confusion). The default value for this is false. Everything else related to TLS has no effect unless this is explicitly set to true, and this value does not inherit from anywhere, Ron On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe wrote: > > On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote: > > Hi Ron/Colin, > > > > Thanks for the responses. I think we need to consider these cases: > > > > 1) The recommended approach: This is the one we document. Configs > > consistent with Kafka make sense here. It seems a lot more confusing to > > configure `ssl.cipher.suites`, `ssl.enabled.protocols`, ` > > zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with > > multiple naming inconsistencies in server.properties. For users who have > > never used ZK system properties, this is clearly simpler. Another config to > > consider is `ssl.endpoint.identification.algorithm` which does the same > > thing as `zookeeper.ssl.hostnameVerification` - if we are staying > > consistent, we should use the Kafka format for this too. I would also > > consider using something like `zookeeper.client.ssl.enabled` instead of ` > > zookeeper.client.secure` since it is difficult to tell what `secure` means > > when you have options to configure Kerberos, ACLs and/or SSL. > > > > 2) Inheritance using prefixed configs as we do for other prefixed configs > > in Kafka: I think it may be useful for configs like `ssl.protocol` and ` > > ssl.cipher.suites`. > > > > 3) Default values: Do we want to use Kafka defaults? The inconsistency we > > have is `ssl.protocol/ssl.enabled.protocols` since we still have insecure > > protocols enabled for Kafka. We have KIP-553 ( > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956) > > open to disable these. It is currently blocked since it also talks about > > enabling TLSv1.3 by default and we haven't done sufficient testing for > > that. Since it will be good to disable the older insecure protocols anyway, > > perhaps we could do that without enabling TLSv1.3 by default in 2.5? > > Hi Rajini, > > Inheriting the SSL settings from Kafka would have the effect of changing the > defaults for almost everyone using SSL from non-SSL ZK to SSL-based ZK, > right? This seems to expand the scope of the KIP greatly, since it moves it > from being opt-in to opt-out. Are there any potential negative implications > of this? > > best, > Colin > > > > > > 4) Backward compatibility for deployments which are using system > > properties. We shouldn't override system properties with Kafka defaults or > > inherited values. But we do want to override if user configures properties > > explicitly. In the current approach, this was simpler since we just had to > > set the configured values. But if we decide to inherit Kafka configs, then > > we will need to check system properties and update accordingly. But that > > should be doable? > > > > On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe wrote: > > > > > On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote: > > > > Hi Colin. Two things come to mind with respect to ZooKeeper camelCase > > > > style vs Kafka-style config names for ZooKeeper. First, I think it > > > > would be desirable for the client configs and broker configs to be > > > > interoperable. For example, it feels like it would be convenient to > > > > be able to pass the broker's config file to the ZK Security Migrator > > > > tool. I think whatever we use (ZooKeeper camelCase style or Kafka > > > > style), I think we should use the same for both. > > > > > > > > The second thing to think about is the fact that ZooKeeper > > > > configuration inherently uses Java system properties as a first pass. > > > > So while we might switch to Kafka-style configs in the config file > > > > (e.g. zookeeper.ssl.trust.store.location), the > > > > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS > > > > configs are set) is going to look for the camelCase > > > > zookeeper.ssl.trustStore.location system property and use any value > > > > there. I think this could be very confusing for people. Granted, we > > > > are doing this so that people don't have to use system properties, but > > > > there is no way to turn off the use of system properties, so I think > > > > there would be pretty substantial potential for confusion. > > > > > > I don't think we document or recommend that anyone use system properties > > > to configure Zookeeper usage within Kafka. And I think the reason is > > > exactly the issue you pointed out -- it wouldn't be consistent with our > > > other configurations. > > > > > > best, > > > Colin > > > > > > > > > > > One idea is to migrate the system properties -- i.e. look to see if
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote: > Hi Ron/Colin, > > Thanks for the responses. I think we need to consider these cases: > > 1) The recommended approach: This is the one we document. Configs > consistent with Kafka make sense here. It seems a lot more confusing to > configure `ssl.cipher.suites`, `ssl.enabled.protocols`, ` > zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with > multiple naming inconsistencies in server.properties. For users who have > never used ZK system properties, this is clearly simpler. Another config to > consider is `ssl.endpoint.identification.algorithm` which does the same > thing as `zookeeper.ssl.hostnameVerification` - if we are staying > consistent, we should use the Kafka format for this too. I would also > consider using something like `zookeeper.client.ssl.enabled` instead of ` > zookeeper.client.secure` since it is difficult to tell what `secure` means > when you have options to configure Kerberos, ACLs and/or SSL. > > 2) Inheritance using prefixed configs as we do for other prefixed configs > in Kafka: I think it may be useful for configs like `ssl.protocol` and ` > ssl.cipher.suites`. > > 3) Default values: Do we want to use Kafka defaults? The inconsistency we > have is `ssl.protocol/ssl.enabled.protocols` since we still have insecure > protocols enabled for Kafka. We have KIP-553 ( > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956) > open to disable these. It is currently blocked since it also talks about > enabling TLSv1.3 by default and we haven't done sufficient testing for > that. Since it will be good to disable the older insecure protocols anyway, > perhaps we could do that without enabling TLSv1.3 by default in 2.5? Hi Rajini, Inheriting the SSL settings from Kafka would have the effect of changing the defaults for almost everyone using SSL from non-SSL ZK to SSL-based ZK, right? This seems to expand the scope of the KIP greatly, since it moves it from being opt-in to opt-out. Are there any potential negative implications of this? best, Colin > > 4) Backward compatibility for deployments which are using system > properties. We shouldn't override system properties with Kafka defaults or > inherited values. But we do want to override if user configures properties > explicitly. In the current approach, this was simpler since we just had to > set the configured values. But if we decide to inherit Kafka configs, then > we will need to check system properties and update accordingly. But that > should be doable? > > On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe wrote: > > > On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote: > > > Hi Colin. Two things come to mind with respect to ZooKeeper camelCase > > > style vs Kafka-style config names for ZooKeeper. First, I think it > > > would be desirable for the client configs and broker configs to be > > > interoperable. For example, it feels like it would be convenient to > > > be able to pass the broker's config file to the ZK Security Migrator > > > tool. I think whatever we use (ZooKeeper camelCase style or Kafka > > > style), I think we should use the same for both. > > > > > > The second thing to think about is the fact that ZooKeeper > > > configuration inherently uses Java system properties as a first pass. > > > So while we might switch to Kafka-style configs in the config file > > > (e.g. zookeeper.ssl.trust.store.location), the > > > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS > > > configs are set) is going to look for the camelCase > > > zookeeper.ssl.trustStore.location system property and use any value > > > there. I think this could be very confusing for people. Granted, we > > > are doing this so that people don't have to use system properties, but > > > there is no way to turn off the use of system properties, so I think > > > there would be pretty substantial potential for confusion. > > > > I don't think we document or recommend that anyone use system properties > > to configure Zookeeper usage within Kafka. And I think the reason is > > exactly the issue you pointed out -- it wouldn't be consistent with our > > other configurations. > > > > best, > > Colin > > > > > > > > One idea is to migrate the system properties -- i.e. look to see if > > > zookeeper.ssl.trustStore.location is set and if it is clear out the > > > value there and move it under the zookeeper.ssl.trust.store.location > > > system property. (I said it was an idea -- not necessarily a good one > > > :-) > > > > > > Ron > > > > > > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe wrote: > > > > > > > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote: > > > > > Thanks Colin and Rajini. > > > > > > > > > > I've updated the KIP to say that KIP-421 functionality is available > > to > > > > > encrypt sensitive configs like the ZK key store and trust store > > > > > passwords. (I've also made it clear that the configs are not > > >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Rajini. Thanks for the feedback. I have now come around to the
idea of abandoning the ZooKeeper camelCase style in favor of being
consistent with the existing Kafka broker configs. Here's what I've
done:
#1) ZK style vs. Kafka style
I've updated the KIP to convert all of the following configs from ZK
camelCase style to Kafka style:
zookeeper.ssl.keyStore.location => zookeeper.ssl.keystore.location
zookeeper.ssl.keyStore.password => zookeeper.ssl.keystore.password
zookeeper.ssl.keyStore.type => zookeeper.ssl.keystore.type
zookeeper.ssl.trustStore.location => zookeeper.ssl.truststore.location
zookeeper.ssl.trustStore.password => zookeeper.ssl.truststore.password
zookeeper.ssl.trustStore.type => zookeeper.ssl.truststore.type
zookeeper.ssl.ciphersuites => zookeeper.ssl.cipher.suites
zookeeper.ssl.enabledProtocols => zookeeper.ssl.enabled.protocols
I've also renamed the following as you suggested (all similar broker
configs use the suffix ".enable" instead of ".enabled"):
zookeeper.client.secure => zookeeper.ssl.client.enable
Finally, to be consistent with Kafka's ".enable" suffix convention,
I've adjusted these two (we didn't discuss this, but it makes sense to
me; I'll assume it's fine if there are no further comments about it):
zookeeper.ssl.crl => zookeeper.ssl.crl.enable
zookeeper.ssl.ocsp => zookeeper.ssl.ocsp.enable
UNRESOLVED ITEM #1:
Regarding zookeeper.ssl.hostnameVerification and
ssl.endpoint.identification.algorithm, I'm not sure what the right
solution is here. "zookeeper.ssl.endpoint.identification.algorithm"
seems weird to me -- we aren't trying to distinguish between HTTPS and
LDAPS (which are the two valid endpoint identification algorithms
according to
https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html).
Does Kafka provide the ability to disable hostname checking by
clearing this property from the default value of (case-insensitively)
HTTPS? If so then maybe there is a doc bug here to make that
explicit. So maybe we can explicitly clarify if that is how it works
(and I would submit a PR to amend the docs to state it if so); then we
can perhaps decide what to do here. I would think the two choices are
zookeeper.ssl.endpoint.identification.algorithm (string with the
default https or blank) or zookeeper.ssl.hostname.verify.enable (no
camel-case, boolean with the default being true).
#2) Config Inheritance
UNRESOLVED ITEM #2:
I think what you are saying is that for the following 9 configs, if
the "zookeeper." prefixed version is not provided then the
non-prefixed version should be used instead. Can you confirm? The
default values for the last two are an issue, but see #3 below. I
haven't updated the KIP yet -- I will wait for confirmation that these
9 configs are the ones to be inherited and the default value issue is
resolved.
[zookeeper.]ssl.keystore.{location,password,type}
[zookeeper.]ssl.truststore.{location,password,type}
[zookeeper.]ssl.cipher.suites
[zookeeper.]ssl.protocol
[zookeeper.]ssl.enabled.protocols
Note: The AclAuthorizer configs on the broker side do not use
inheritance at all -- if AclAuthorizer is pointed at a separate ZK
quorum then the configs must specify everything.
#3) Default values
UNRESOLVED ITEM #3:
I agree it is probably best to decouple disabling older TLS protocols
from making TLSv1.3 as the default if both can't be done --
independent of anything else, I think think in this case it's best to
tighten security a little bit now and then a bit more later on than to
wait until both changes can happen at the same time. However, as you
point out, if the defaults for Kafka's ssl.protocol and
ssl.enabled.protocols were to be amended to TLSv1.2 then it would also
allow the default values for ZK and Kafka to be the same for
ssl.protocol/ssl.enabled.protocols, and then inheritance works well
for these configs -- we probably shouldn't do inheritance for them
otherwise. (BTW, just for completeness, I will state that the
allowable values are defined in the "SSLContext Algorithms" section of
https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html).
#4) System Properties
It is not a problem to start with ZK-style system properties when the
user explicitly defines them. The evaluation algorithm will be:
1) Start with the ZK-style system property if it is defined
2) Use the explicitly-prefixed "zookeeper." property value if it is
defined (overriding any system property value)
3) Inherit any non-prefixed Kafka config value if #1 and #2 did not
result in a value
4) Use the default value if there is still no value
I've updated the KIP accordingly to reflect this evaluation order..
Ron
On Thu, Jan 16, 2020 at 6:06 AM Rajini Sivaram wrote:
>
> Forgot to mention:
>
> Overriding system property with a differently named Kafka configuration
> option is something we already do for JAAS configs - we default to System
> property `java.security.auth.login.config` if `sasl.jaas.config` is not
> configu
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Forgot to mention: Overriding system property with a differently named Kafka configuration option is something we already do for JAAS configs - we default to System property `java.security.auth.login.config` if `sasl.jaas.config` is not configured. On Thu, Jan 16, 2020 at 10:58 AM Rajini Sivaram wrote: > Hi Ron/Colin, > > Thanks for the responses. I think we need to consider these cases: > > 1) The recommended approach: This is the one we document. Configs > consistent with Kafka make sense here. It seems a lot more confusing to > configure `ssl.cipher.suites`, `ssl.enabled.protocols`, ` > zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with > multiple naming inconsistencies in server.properties. For users who have > never used ZK system properties, this is clearly simpler. Another config > to consider is `ssl.endpoint.identification.algorithm` which does the > same thing as `zookeeper.ssl.hostnameVerification` - if we are staying > consistent, we should use the Kafka format for this too. I would also > consider using something like `zookeeper.client.ssl.enabled` instead of ` > zookeeper.client.secure` since it is difficult to tell what `secure` > means when you have options to configure Kerberos, ACLs and/or SSL. > > 2) Inheritance using prefixed configs as we do for other prefixed configs > in Kafka: I think it may be useful for configs like `ssl.protocol` and ` > ssl.cipher.suites`. > > 3) Default values: Do we want to use Kafka defaults? The inconsistency we > have is `ssl.protocol/ssl.enabled.protocols` since we still have insecure > protocols enabled for Kafka. We have KIP-553 ( > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956) > open to disable these. It is currently blocked since it also talks about > enabling TLSv1.3 by default and we haven't done sufficient testing for > that. Since it will be good to disable the older insecure protocols anyway, > perhaps we could do that without enabling TLSv1.3 by default in 2.5? > > 4) Backward compatibility for deployments which are using system > properties. We shouldn't override system properties with Kafka defaults or > inherited values. But we do want to override if user configures properties > explicitly. In the current approach, this was simpler since we just had to > set the configured values. But if we decide to inherit Kafka configs, then > we will need to check system properties and update accordingly. But that > should be doable? > > On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe wrote: > >> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote: >> > Hi Colin. Two things come to mind with respect to ZooKeeper camelCase >> > style vs Kafka-style config names for ZooKeeper. First, I think it >> > would be desirable for the client configs and broker configs to be >> > interoperable. For example, it feels like it would be convenient to >> > be able to pass the broker's config file to the ZK Security Migrator >> > tool. I think whatever we use (ZooKeeper camelCase style or Kafka >> > style), I think we should use the same for both. >> > >> > The second thing to think about is the fact that ZooKeeper >> > configuration inherently uses Java system properties as a first pass. >> > So while we might switch to Kafka-style configs in the config file >> > (e.g. zookeeper.ssl.trust.store.location), the >> > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS >> > configs are set) is going to look for the camelCase >> > zookeeper.ssl.trustStore.location system property and use any value >> > there. I think this could be very confusing for people. Granted, we >> > are doing this so that people don't have to use system properties, but >> > there is no way to turn off the use of system properties, so I think >> > there would be pretty substantial potential for confusion. >> >> I don't think we document or recommend that anyone use system properties >> to configure Zookeeper usage within Kafka. And I think the reason is >> exactly the issue you pointed out -- it wouldn't be consistent with our >> other configurations. >> >> best, >> Colin >> >> > >> > One idea is to migrate the system properties -- i.e. look to see if >> > zookeeper.ssl.trustStore.location is set and if it is clear out the >> > value there and move it under the zookeeper.ssl.trust.store.location >> > system property. (I said it was an idea -- not necessarily a good one >> > :-) >> > >> > Ron >> > >> > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe >> wrote: >> > > >> > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote: >> > > > Thanks Colin and Rajini. >> > > > >> > > > I've updated the KIP to say that KIP-421 functionality is available >> to >> > > > encrypt sensitive configs like the ZK key store and trust store >> > > > passwords. (I've also made it clear that the configs are not >> > > > dynamically reconfigurable since dynamic values are stored in ZK and >> > > > the data is needed to get to ZK in the first place). Coli
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Ron/Colin, Thanks for the responses. I think we need to consider these cases: 1) The recommended approach: This is the one we document. Configs consistent with Kafka make sense here. It seems a lot more confusing to configure `ssl.cipher.suites`, `ssl.enabled.protocols`, ` zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with multiple naming inconsistencies in server.properties. For users who have never used ZK system properties, this is clearly simpler. Another config to consider is `ssl.endpoint.identification.algorithm` which does the same thing as `zookeeper.ssl.hostnameVerification` - if we are staying consistent, we should use the Kafka format for this too. I would also consider using something like `zookeeper.client.ssl.enabled` instead of ` zookeeper.client.secure` since it is difficult to tell what `secure` means when you have options to configure Kerberos, ACLs and/or SSL. 2) Inheritance using prefixed configs as we do for other prefixed configs in Kafka: I think it may be useful for configs like `ssl.protocol` and ` ssl.cipher.suites`. 3) Default values: Do we want to use Kafka defaults? The inconsistency we have is `ssl.protocol/ssl.enabled.protocols` since we still have insecure protocols enabled for Kafka. We have KIP-553 ( https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956) open to disable these. It is currently blocked since it also talks about enabling TLSv1.3 by default and we haven't done sufficient testing for that. Since it will be good to disable the older insecure protocols anyway, perhaps we could do that without enabling TLSv1.3 by default in 2.5? 4) Backward compatibility for deployments which are using system properties. We shouldn't override system properties with Kafka defaults or inherited values. But we do want to override if user configures properties explicitly. In the current approach, this was simpler since we just had to set the configured values. But if we decide to inherit Kafka configs, then we will need to check system properties and update accordingly. But that should be doable? On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe wrote: > On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote: > > Hi Colin. Two things come to mind with respect to ZooKeeper camelCase > > style vs Kafka-style config names for ZooKeeper. First, I think it > > would be desirable for the client configs and broker configs to be > > interoperable. For example, it feels like it would be convenient to > > be able to pass the broker's config file to the ZK Security Migrator > > tool. I think whatever we use (ZooKeeper camelCase style or Kafka > > style), I think we should use the same for both. > > > > The second thing to think about is the fact that ZooKeeper > > configuration inherently uses Java system properties as a first pass. > > So while we might switch to Kafka-style configs in the config file > > (e.g. zookeeper.ssl.trust.store.location), the > > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS > > configs are set) is going to look for the camelCase > > zookeeper.ssl.trustStore.location system property and use any value > > there. I think this could be very confusing for people. Granted, we > > are doing this so that people don't have to use system properties, but > > there is no way to turn off the use of system properties, so I think > > there would be pretty substantial potential for confusion. > > I don't think we document or recommend that anyone use system properties > to configure Zookeeper usage within Kafka. And I think the reason is > exactly the issue you pointed out -- it wouldn't be consistent with our > other configurations. > > best, > Colin > > > > > One idea is to migrate the system properties -- i.e. look to see if > > zookeeper.ssl.trustStore.location is set and if it is clear out the > > value there and move it under the zookeeper.ssl.trust.store.location > > system property. (I said it was an idea -- not necessarily a good one > > :-) > > > > Ron > > > > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe wrote: > > > > > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote: > > > > Thanks Colin and Rajini. > > > > > > > > I've updated the KIP to say that KIP-421 functionality is available > to > > > > encrypt sensitive configs like the ZK key store and trust store > > > > passwords. (I've also made it clear that the configs are not > > > > dynamically reconfigurable since dynamic values are stored in ZK and > > > > the data is needed to get to ZK in the first place). Colin, let me > > > > know if you think anything else needs to be said/done about it -- I > > > > wasn't sure if your comment above implies that there are additional > > > > direct actions that we need to take aside from this. > > > > > > > > I agree that making the brokers' ZooKeeper clients dynamic with > > > > respect to key and trust stores (e.g. via > > > > zookeeper.ssl.context.supplier.class) may increase the risk that this >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote: > Hi Colin. Two things come to mind with respect to ZooKeeper camelCase > style vs Kafka-style config names for ZooKeeper. First, I think it > would be desirable for the client configs and broker configs to be > interoperable. For example, it feels like it would be convenient to > be able to pass the broker's config file to the ZK Security Migrator > tool. I think whatever we use (ZooKeeper camelCase style or Kafka > style), I think we should use the same for both. > > The second thing to think about is the fact that ZooKeeper > configuration inherently uses Java system properties as a first pass. > So while we might switch to Kafka-style configs in the config file > (e.g. zookeeper.ssl.trust.store.location), the > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS > configs are set) is going to look for the camelCase > zookeeper.ssl.trustStore.location system property and use any value > there. I think this could be very confusing for people. Granted, we > are doing this so that people don't have to use system properties, but > there is no way to turn off the use of system properties, so I think > there would be pretty substantial potential for confusion. I don't think we document or recommend that anyone use system properties to configure Zookeeper usage within Kafka. And I think the reason is exactly the issue you pointed out -- it wouldn't be consistent with our other configurations. best, Colin > > One idea is to migrate the system properties -- i.e. look to see if > zookeeper.ssl.trustStore.location is set and if it is clear out the > value there and move it under the zookeeper.ssl.trust.store.location > system property. (I said it was an idea -- not necessarily a good one > :-) > > Ron > > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe wrote: > > > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote: > > > Thanks Colin and Rajini. > > > > > > I've updated the KIP to say that KIP-421 functionality is available to > > > encrypt sensitive configs like the ZK key store and trust store > > > passwords. (I've also made it clear that the configs are not > > > dynamically reconfigurable since dynamic values are stored in ZK and > > > the data is needed to get to ZK in the first place). Colin, let me > > > know if you think anything else needs to be said/done about it -- I > > > wasn't sure if your comment above implies that there are additional > > > direct actions that we need to take aside from this. > > > > > > I agree that making the brokers' ZooKeeper clients dynamic with > > > respect to key and trust stores (e.g. via > > > zookeeper.ssl.context.supplier.class) may increase the risk that this > > > KIP may not land in time for 2.5. > > > > > > Regarding the config names for ZK being different than the ZooKeeper > > > configs (e.g. camel-case keyStore/trustStore instead of > > > keystore/truststore, ciphersuites instead of cipher.suites, > > > enabledProtocols instead of enabled.protocols), I agree it is an > > > issue. I tried to mitigate it by putting warning text in the config > > > docs for these. Regarding configuration inheritance, I think what you > > > are saying is that for any configs where the broker has a clear > > > semantic equivalent, the broker could fall back to the broker value if > > > the ZK value is not given. The potential list is: > > > > > > zookeeper.ssl.keyStore.location => ssl.keystore.location > > > zookeeper.ssl.keyStore.password => ssl.keystore.password > > > zookeeper.ssl.keyStore.type => ssl.keystore.type > > > zookeeper.ssl.trustStore.location => ssl.truststore.location > > > zookeeper.ssl.trustStore.password => ssl.truststore.password > > > zookeeper.ssl.trustStore.type => ssl.truststore.type > > > zookeeper.ssl.ciphersuites => ssl.cipher.suites > > > > > > Note that there are two configs that look the same based on their key > > > names but whose allowable values differ: > > > > > > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS) > > > zookeeper.ssl.enabledProtocols(Default: value of protocol property) = > > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1) > > > > > > Not sure what the right answer is, but hopefully this detail will help > > > get us there. > > > > > > Ron > > > > I think, on balance, I agree with Rajini that it would be nice to make the > > configs look more like Kafka configs. We don't have camel-case in > > configuration keys, so we should avoid it here. > > > > best, > > Colin > > > > > > > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe wrote: > > > > > > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote: > > > > > Hi Ron, > > > > > > > > > > Thanks for the detailed explanation, sounds good to me. > > > > > > > > > > A few more questions: > > > > > > > > > > 1) At the moment, all sensitive broker configs including > > > > > keystore/truststore passwords can be stored encrypted in ZooKeeper > > > > > prior to > >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Colin. Two things come to mind with respect to ZooKeeper camelCase style vs Kafka-style config names for ZooKeeper. First, I think it would be desirable for the client configs and broker configs to be interoperable. For example, it feels like it would be convenient to be able to pass the broker's config file to the ZK Security Migrator tool. I think whatever we use (ZooKeeper camelCase style or Kafka style), I think we should use the same for both. The second thing to think about is the fact that ZooKeeper configuration inherently uses Java system properties as a first pass. So while we might switch to Kafka-style configs in the config file (e.g. zookeeper.ssl.trust.store.location), the org.apache.zookeeper.client.ZKClientConfig class (which is how TLS configs are set) is going to look for the camelCase zookeeper.ssl.trustStore.location system property and use any value there. I think this could be very confusing for people. Granted, we are doing this so that people don't have to use system properties, but there is no way to turn off the use of system properties, so I think there would be pretty substantial potential for confusion. One idea is to migrate the system properties -- i.e. look to see if zookeeper.ssl.trustStore.location is set and if it is clear out the value there and move it under the zookeeper.ssl.trust.store.location system property. (I said it was an idea -- not necessarily a good one :-) Ron On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe wrote: > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote: > > Thanks Colin and Rajini. > > > > I've updated the KIP to say that KIP-421 functionality is available to > > encrypt sensitive configs like the ZK key store and trust store > > passwords. (I've also made it clear that the configs are not > > dynamically reconfigurable since dynamic values are stored in ZK and > > the data is needed to get to ZK in the first place). Colin, let me > > know if you think anything else needs to be said/done about it -- I > > wasn't sure if your comment above implies that there are additional > > direct actions that we need to take aside from this. > > > > I agree that making the brokers' ZooKeeper clients dynamic with > > respect to key and trust stores (e.g. via > > zookeeper.ssl.context.supplier.class) may increase the risk that this > > KIP may not land in time for 2.5. > > > > Regarding the config names for ZK being different than the ZooKeeper > > configs (e.g. camel-case keyStore/trustStore instead of > > keystore/truststore, ciphersuites instead of cipher.suites, > > enabledProtocols instead of enabled.protocols), I agree it is an > > issue. I tried to mitigate it by putting warning text in the config > > docs for these. Regarding configuration inheritance, I think what you > > are saying is that for any configs where the broker has a clear > > semantic equivalent, the broker could fall back to the broker value if > > the ZK value is not given. The potential list is: > > > > zookeeper.ssl.keyStore.location => ssl.keystore.location > > zookeeper.ssl.keyStore.password => ssl.keystore.password > > zookeeper.ssl.keyStore.type => ssl.keystore.type > > zookeeper.ssl.trustStore.location => ssl.truststore.location > > zookeeper.ssl.trustStore.password => ssl.truststore.password > > zookeeper.ssl.trustStore.type => ssl.truststore.type > > zookeeper.ssl.ciphersuites => ssl.cipher.suites > > > > Note that there are two configs that look the same based on their key > > names but whose allowable values differ: > > > > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS) > > zookeeper.ssl.enabledProtocols(Default: value of protocol property) = > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1) > > > > Not sure what the right answer is, but hopefully this detail will help > > get us there. > > > > Ron > > I think, on balance, I agree with Rajini that it would be nice to make the > configs look more like Kafka configs. We don't have camel-case in > configuration keys, so we should avoid it here. > > best, > Colin > > > > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe wrote: > > > > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote: > > > > Hi Ron, > > > > > > > > Thanks for the detailed explanation, sounds good to me. > > > > > > > > A few more questions: > > > > > > > > 1) At the moment, all sensitive broker configs including > > > > keystore/truststore passwords can be stored encrypted in ZooKeeper > > > > prior to > > > > starting up brokers. We are now adding SSL keystore/truststore passwords > > > > for ZooKeeper client that cannot be stored in ZooKeeper since you need > > > > these to connect to ZK. We should perhaps document that these configs > > > > can > > > > be encrypted using KIP-421. > > > > > > That's a good point. Previously, we didn't have ZK on-the-wire security > > > support. However, now that we do, sending sensitive keystore and > > > truststore passwords to ZK in cleartext should be depre
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote: > Thanks Colin and Rajini. > > I've updated the KIP to say that KIP-421 functionality is available to > encrypt sensitive configs like the ZK key store and trust store > passwords. (I've also made it clear that the configs are not > dynamically reconfigurable since dynamic values are stored in ZK and > the data is needed to get to ZK in the first place). Colin, let me > know if you think anything else needs to be said/done about it -- I > wasn't sure if your comment above implies that there are additional > direct actions that we need to take aside from this. > > I agree that making the brokers' ZooKeeper clients dynamic with > respect to key and trust stores (e.g. via > zookeeper.ssl.context.supplier.class) may increase the risk that this > KIP may not land in time for 2.5. > > Regarding the config names for ZK being different than the ZooKeeper > configs (e.g. camel-case keyStore/trustStore instead of > keystore/truststore, ciphersuites instead of cipher.suites, > enabledProtocols instead of enabled.protocols), I agree it is an > issue. I tried to mitigate it by putting warning text in the config > docs for these. Regarding configuration inheritance, I think what you > are saying is that for any configs where the broker has a clear > semantic equivalent, the broker could fall back to the broker value if > the ZK value is not given. The potential list is: > > zookeeper.ssl.keyStore.location => ssl.keystore.location > zookeeper.ssl.keyStore.password => ssl.keystore.password > zookeeper.ssl.keyStore.type => ssl.keystore.type > zookeeper.ssl.trustStore.location => ssl.truststore.location > zookeeper.ssl.trustStore.password => ssl.truststore.password > zookeeper.ssl.trustStore.type => ssl.truststore.type > zookeeper.ssl.ciphersuites => ssl.cipher.suites > > Note that there are two configs that look the same based on their key > names but whose allowable values differ: > > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS) > zookeeper.ssl.enabledProtocols(Default: value of protocol property) = > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1) > > Not sure what the right answer is, but hopefully this detail will help > get us there. > > Ron I think, on balance, I agree with Rajini that it would be nice to make the configs look more like Kafka configs. We don't have camel-case in configuration keys, so we should avoid it here. best, Colin > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe wrote: > > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote: > > > Hi Ron, > > > > > > Thanks for the detailed explanation, sounds good to me. > > > > > > A few more questions: > > > > > > 1) At the moment, all sensitive broker configs including > > > keystore/truststore passwords can be stored encrypted in ZooKeeper prior > > > to > > > starting up brokers. We are now adding SSL keystore/truststore passwords > > > for ZooKeeper client that cannot be stored in ZooKeeper since you need > > > these to connect to ZK. We should perhaps document that these configs can > > > be encrypted using KIP-421. > > > > That's a good point. Previously, we didn't have ZK on-the-wire security > > support. However, now that we do, sending sensitive keystore and > > truststore passwords to ZK in cleartext should be deprecated in favor of > > using KIP-421. > > > > > > > > 2) We can dynamically update keystores and trust stores used by brokers > > > without restarting the broker. Can we support this easily for ZK clients > > > created by the broker, for example by adding our own > > > `zookeeper.ssl.context.supplier.class`? > > > > > > > Hmm. That might be better to consider in a follow-up, since the deadline > > for 2.5 KIPs is coming up? > > > > best, > > Colin > > > > > 3) Looks like we are using config names that directly map to ZK configs. > > > Have we considered using equivalent Kafka config names with prefixes, > > > perhaps with inheritance from the non-prefixed names? Not sure if this is > > > a > > > good idea, but perhaps worth documenting in Rejected Alternatives at > > > least. > > > > > > > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe wrote: > > > > > > > Hi Ron, > > > > > > > > Thanks for the explanation. I guess thinking about it a little bit > > > > more, > > > > we should just add --zk-tls-config-file to all of these commands. > > > > > > > > We will be removing this option (plus ZK in general) from these commands > > > > in the next major release, but ZK is still supported in 2.5, so we > > > > should > > > > just do the logical thing. (The exception is ZkSecurityMigrator, which > > > > will stay). > > > > > > > > best, > > > > Colin > > > > > > > > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote: > > > > > Hi Colin. > > > > > > > > > > <<< It seems like this [--zk-tls-config-file information] could just > > > > appear > > > > > in a configuration file, which all of these tools already accept
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Thanks Colin and Rajini. I've updated the KIP to say that KIP-421 functionality is available to encrypt sensitive configs like the ZK key store and trust store passwords. (I've also made it clear that the configs are not dynamically reconfigurable since dynamic values are stored in ZK and the data is needed to get to ZK in the first place). Colin, let me know if you think anything else needs to be said/done about it -- I wasn't sure if your comment above implies that there are additional direct actions that we need to take aside from this. I agree that making the brokers' ZooKeeper clients dynamic with respect to key and trust stores (e.g. via zookeeper.ssl.context.supplier.class) may increase the risk that this KIP may not land in time for 2.5. Regarding the config names for ZK being different than the ZooKeeper configs (e.g. camel-case keyStore/trustStore instead of keystore/truststore, ciphersuites instead of cipher.suites, enabledProtocols instead of enabled.protocols), I agree it is an issue. I tried to mitigate it by putting warning text in the config docs for these. Regarding configuration inheritance, I think what you are saying is that for any configs where the broker has a clear semantic equivalent, the broker could fall back to the broker value if the ZK value is not given. The potential list is: zookeeper.ssl.keyStore.location => ssl.keystore.location zookeeper.ssl.keyStore.password => ssl.keystore.password zookeeper.ssl.keyStore.type => ssl.keystore.type zookeeper.ssl.trustStore.location => ssl.truststore.location zookeeper.ssl.trustStore.password => ssl.truststore.password zookeeper.ssl.trustStore.type => ssl.truststore.type zookeeper.ssl.ciphersuites => ssl.cipher.suites Note that there are two configs that look the same based on their key names but whose allowable values differ: zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS) zookeeper.ssl.enabledProtocols(Default: value of protocol property) = ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1) Not sure what the right answer is, but hopefully this detail will help get us there. Ron On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe wrote: > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote: > > Hi Ron, > > > > Thanks for the detailed explanation, sounds good to me. > > > > A few more questions: > > > > 1) At the moment, all sensitive broker configs including > > keystore/truststore passwords can be stored encrypted in ZooKeeper prior to > > starting up brokers. We are now adding SSL keystore/truststore passwords > > for ZooKeeper client that cannot be stored in ZooKeeper since you need > > these to connect to ZK. We should perhaps document that these configs can > > be encrypted using KIP-421. > > That's a good point. Previously, we didn't have ZK on-the-wire security > support. However, now that we do, sending sensitive keystore and truststore > passwords to ZK in cleartext should be deprecated in favor of using KIP-421. > > > > > 2) We can dynamically update keystores and trust stores used by brokers > > without restarting the broker. Can we support this easily for ZK clients > > created by the broker, for example by adding our own > > `zookeeper.ssl.context.supplier.class`? > > > > Hmm. That might be better to consider in a follow-up, since the deadline for > 2.5 KIPs is coming up? > > best, > Colin > > > 3) Looks like we are using config names that directly map to ZK configs. > > Have we considered using equivalent Kafka config names with prefixes, > > perhaps with inheritance from the non-prefixed names? Not sure if this is a > > good idea, but perhaps worth documenting in Rejected Alternatives at least. > > > > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe wrote: > > > > > Hi Ron, > > > > > > Thanks for the explanation. I guess thinking about it a little bit more, > > > we should just add --zk-tls-config-file to all of these commands. > > > > > > We will be removing this option (plus ZK in general) from these commands > > > in the next major release, but ZK is still supported in 2.5, so we should > > > just do the logical thing. (The exception is ZkSecurityMigrator, which > > > will stay). > > > > > > best, > > > Colin > > > > > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote: > > > > Hi Colin. > > > > > > > > <<< It seems like this [--zk-tls-config-file information] could just > > > appear > > > > in a configuration file, which all of these tools already accept (I > > > think) > > > > > > > > ZkSecurityMigrator has no such property file facility; adding a > > > > "--zk-tls-config-file" parameter is exactly for this purpose. If we add > > > > that to ZkSecurityMigrator then it is trivial to add it to other > > > > commands > > > > (the same code is simply reused; it ends up being just a few extra > > > lines). > > > > I do not see any parameter in the other two commands to adjust the ZK > > > > connection; ConfigCommand accepts a "--command-config" flag, but > > > acco
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> Hi Ron,
>
> Thanks for the detailed explanation, sounds good to me.
>
> A few more questions:
>
> 1) At the moment, all sensitive broker configs including
> keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> starting up brokers. We are now adding SSL keystore/truststore passwords
> for ZooKeeper client that cannot be stored in ZooKeeper since you need
> these to connect to ZK. We should perhaps document that these configs can
> be encrypted using KIP-421.
That's a good point. Previously, we didn't have ZK on-the-wire security
support. However, now that we do, sending sensitive keystore and truststore
passwords to ZK in cleartext should be deprecated in favor of using KIP-421.
>
> 2) We can dynamically update keystores and trust stores used by brokers
> without restarting the broker. Can we support this easily for ZK clients
> created by the broker, for example by adding our own
> `zookeeper.ssl.context.supplier.class`?
>
Hmm. That might be better to consider in a follow-up, since the deadline for
2.5 KIPs is coming up?
best,
Colin
> 3) Looks like we are using config names that directly map to ZK configs.
> Have we considered using equivalent Kafka config names with prefixes,
> perhaps with inheritance from the non-prefixed names? Not sure if this is a
> good idea, but perhaps worth documenting in Rejected Alternatives at least.
>
>
> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe wrote:
>
> > Hi Ron,
> >
> > Thanks for the explanation. I guess thinking about it a little bit more,
> > we should just add --zk-tls-config-file to all of these commands.
> >
> > We will be removing this option (plus ZK in general) from these commands
> > in the next major release, but ZK is still supported in 2.5, so we should
> > just do the logical thing. (The exception is ZkSecurityMigrator, which
> > will stay).
> >
> > best,
> > Colin
> >
> >
> > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > Hi Colin.
> > >
> > > <<< It seems like this [--zk-tls-config-file information] could just
> > appear
> > > in a configuration file, which all of these tools already accept (I
> > think)
> > >
> > > ZkSecurityMigrator has no such property file facility; adding a
> > > "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> > > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > > (the same code is simply reused; it ends up being just a few extra
> > lines).
> > > I do not see any parameter in the other two commands to adjust the ZK
> > > connection; ConfigCommand accepts a "--command-config" flag, but
> > according
> > > to the code "This is used only with --bootstrap-server option for
> > > describing and altering broker configs."
> > >
> > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > time
> > > for the next release.
> > >
> > > ConfigCommand supports the "--bootstrap-server" option and will have its
> > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > special
> > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > > prior to starting Kafka will still exist, so it feels like while
> > > "--zk-tls-config-file" would never be used except for this use case it
> > > could probably still be added for this particular situation.
> > >
> > > Ron
> > >
> > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > > sent) did not go through. I am trying to get emails through now to move
> > > this discussion forward.
> > >
> > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe wrote:
> > >
> > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > Hi everyone. I would like to make the following changes to the KIP.
> > > > >
> > > > > MOTIVATION:
> > > > > Include a statement that it will be difficult in the short term to
> > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > > (which
> > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > cluster
> > > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > > case;
> > > > > therefore it is in scope to be able to securely configure the CLI
> > tools
> > > > > that still leverage non-deprecated direct Zookeeper communication
> > for TLS
> > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > zookeeper-security-migration.sh).
> > > >
> > > > Hi Ron,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > About deprecations:
> > > >
> > > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> > one
> > > > would not make sense, since it would defeat the whole point of the
> > tool :)
> > > >
> > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > KIP-455
> > > > implementation has dragged a bit, but this should get done
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Manikumar. Yes, you are correct: if the ZK Security Migrator session
authenticates to ZooKeeper with multiple identities — SASL and certificate —
then ACLs are applied authorizing both the SASL principal and the certificate
DN.
Ron
> On Jan 15, 2020, at 6:33 AM, Manikumar wrote:
>
> Hi Ron,
>
> Thanks for the KIP. KIP looks good to me.
>
> Am I correct in understanding that, when we run ZkSecurityMigrator with
> SASL + SSL, multiple identities will be added to the ACLs?
>
> Thanks,
>
> On Wed, Jan 15, 2020 at 1:19 AM Rajini Sivaram
> wrote:
>
>> Hi Ron,
>>
>> Thanks for the detailed explanation, sounds good to me.
>>
>> A few more questions:
>>
>> 1) At the moment, all sensitive broker configs including
>> keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
>> starting up brokers. We are now adding SSL keystore/truststore passwords
>> for ZooKeeper client that cannot be stored in ZooKeeper since you need
>> these to connect to ZK. We should perhaps document that these configs can
>> be encrypted using KIP-421.
>>
>> 2) We can dynamically update keystores and trust stores used by brokers
>> without restarting the broker. Can we support this easily for ZK clients
>> created by the broker, for example by adding our own
>> `zookeeper.ssl.context.supplier.class`?
>>
>> 3) Looks like we are using config names that directly map to ZK configs.
>> Have we considered using equivalent Kafka config names with prefixes,
>> perhaps with inheritance from the non-prefixed names? Not sure if this is a
>> good idea, but perhaps worth documenting in Rejected Alternatives at least.
>>
>>
>>> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe wrote:
>>>
>>> Hi Ron,
>>>
>>> Thanks for the explanation. I guess thinking about it a little bit more,
>>> we should just add --zk-tls-config-file to all of these commands.
>>>
>>> We will be removing this option (plus ZK in general) from these commands
>>> in the next major release, but ZK is still supported in 2.5, so we should
>>> just do the logical thing. (The exception is ZkSecurityMigrator, which
>>> will stay).
>>>
>>> best,
>>> Colin
>>>
>>>
On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
Hi Colin.
<<< It seems like this [--zk-tls-config-file information] could just
>>> appear
in a configuration file, which all of these tools already accept (I
>>> think)
ZkSecurityMigrator has no such property file facility; adding a
"--zk-tls-config-file" parameter is exactly for this purpose. If we
>> add
that to ZkSecurityMigrator then it is trivial to add it to other
>> commands
(the same code is simply reused; it ends up being just a few extra
>>> lines).
I do not see any parameter in the other two commands to adjust the ZK
connection; ConfigCommand accepts a "--command-config" flag, but
>>> according
to the code "This is used only with --bootstrap-server option for
describing and altering broker configs."
I do agree there would be no need to add "--zk-tls-config-file" to
ReassignPartitionsCommand if its direct ZK connectivity is replaced in
>>> time
for the next release.
ConfigCommand supports the "--bootstrap-server" option and will have
>> its
direct ZooKeeper access formally deprecated as per KIP-555, but the
>>> special
use case of bootstrapping a ZooKeeper ensemble with encrypted
>> credentials
prior to starting Kafka will still exist, so it feels like while
"--zk-tls-config-file" would never be used except for this use case it
could probably still be added for this particular situation.
Ron
P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
sent) did not go through. I am trying to get emails through now to
>> move
this discussion forward.
On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe
>> wrote:
>> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
>> Hi everyone. I would like to make the following changes to the
>> KIP.
>>
>> MOTIVATION:
>> Include a statement that it will be difficult in the short term to
>> deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> (which
>> invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
>>> cluster
>> with encrypted passwords in Zookeeper is an explicitly-supported
>> use
> case;
>> therefore it is in scope to be able to securely configure the CLI
>>> tools
>> that still leverage non-deprecated direct Zookeeper communication
>>> for TLS
>> (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
>> zookeeper-security-migration.sh).
>
> Hi Ron,
>
> Thanks for the KIP.
>
> About deprecations:
>
> * zookeeper-security-migration: clearly, deprecating ZK access in
>> this
>>> one
> would not make sense, since it would defeat the whole point of the
>>> tool :)
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Ron,
Thanks for the KIP. KIP looks good to me.
Am I correct in understanding that, when we run ZkSecurityMigrator with
SASL + SSL, multiple identities will be added to the ACLs?
Thanks,
On Wed, Jan 15, 2020 at 1:19 AM Rajini Sivaram
wrote:
> Hi Ron,
>
> Thanks for the detailed explanation, sounds good to me.
>
> A few more questions:
>
> 1) At the moment, all sensitive broker configs including
> keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> starting up brokers. We are now adding SSL keystore/truststore passwords
> for ZooKeeper client that cannot be stored in ZooKeeper since you need
> these to connect to ZK. We should perhaps document that these configs can
> be encrypted using KIP-421.
>
> 2) We can dynamically update keystores and trust stores used by brokers
> without restarting the broker. Can we support this easily for ZK clients
> created by the broker, for example by adding our own
> `zookeeper.ssl.context.supplier.class`?
>
> 3) Looks like we are using config names that directly map to ZK configs.
> Have we considered using equivalent Kafka config names with prefixes,
> perhaps with inheritance from the non-prefixed names? Not sure if this is a
> good idea, but perhaps worth documenting in Rejected Alternatives at least.
>
>
> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe wrote:
>
> > Hi Ron,
> >
> > Thanks for the explanation. I guess thinking about it a little bit more,
> > we should just add --zk-tls-config-file to all of these commands.
> >
> > We will be removing this option (plus ZK in general) from these commands
> > in the next major release, but ZK is still supported in 2.5, so we should
> > just do the logical thing. (The exception is ZkSecurityMigrator, which
> > will stay).
> >
> > best,
> > Colin
> >
> >
> > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > Hi Colin.
> > >
> > > <<< It seems like this [--zk-tls-config-file information] could just
> > appear
> > > in a configuration file, which all of these tools already accept (I
> > think)
> > >
> > > ZkSecurityMigrator has no such property file facility; adding a
> > > "--zk-tls-config-file" parameter is exactly for this purpose. If we
> add
> > > that to ZkSecurityMigrator then it is trivial to add it to other
> commands
> > > (the same code is simply reused; it ends up being just a few extra
> > lines).
> > > I do not see any parameter in the other two commands to adjust the ZK
> > > connection; ConfigCommand accepts a "--command-config" flag, but
> > according
> > > to the code "This is used only with --bootstrap-server option for
> > > describing and altering broker configs."
> > >
> > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > time
> > > for the next release.
> > >
> > > ConfigCommand supports the "--bootstrap-server" option and will have
> its
> > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > special
> > > use case of bootstrapping a ZooKeeper ensemble with encrypted
> credentials
> > > prior to starting Kafka will still exist, so it feels like while
> > > "--zk-tls-config-file" would never be used except for this use case it
> > > could probably still be added for this particular situation.
> > >
> > > Ron
> > >
> > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > > sent) did not go through. I am trying to get emails through now to
> move
> > > this discussion forward.
> > >
> > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe
> wrote:
> > >
> > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > Hi everyone. I would like to make the following changes to the
> KIP.
> > > > >
> > > > > MOTIVATION:
> > > > > Include a statement that it will be difficult in the short term to
> > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > > (which
> > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > cluster
> > > > > with encrypted passwords in Zookeeper is an explicitly-supported
> use
> > > > case;
> > > > > therefore it is in scope to be able to securely configure the CLI
> > tools
> > > > > that still leverage non-deprecated direct Zookeeper communication
> > for TLS
> > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > zookeeper-security-migration.sh).
> > > >
> > > > Hi Ron,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > About deprecations:
> > > >
> > > > * zookeeper-security-migration: clearly, deprecating ZK access in
> this
> > one
> > > > would not make sense, since it would defeat the whole point of the
> > tool :)
> > > >
> > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > KIP-455
> > > > implementation has dragged a bit, but this should get done soon.
> > Certainly
> > > > before the next release.
> > > >
> > > > * kafka-configs: I think ZK access should be deprecated here as well.
> >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Ron,
Thanks for the detailed explanation, sounds good to me.
A few more questions:
1) At the moment, all sensitive broker configs including
keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
starting up brokers. We are now adding SSL keystore/truststore passwords
for ZooKeeper client that cannot be stored in ZooKeeper since you need
these to connect to ZK. We should perhaps document that these configs can
be encrypted using KIP-421.
2) We can dynamically update keystores and trust stores used by brokers
without restarting the broker. Can we support this easily for ZK clients
created by the broker, for example by adding our own
`zookeeper.ssl.context.supplier.class`?
3) Looks like we are using config names that directly map to ZK configs.
Have we considered using equivalent Kafka config names with prefixes,
perhaps with inheritance from the non-prefixed names? Not sure if this is a
good idea, but perhaps worth documenting in Rejected Alternatives at least.
On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe wrote:
> Hi Ron,
>
> Thanks for the explanation. I guess thinking about it a little bit more,
> we should just add --zk-tls-config-file to all of these commands.
>
> We will be removing this option (plus ZK in general) from these commands
> in the next major release, but ZK is still supported in 2.5, so we should
> just do the logical thing. (The exception is ZkSecurityMigrator, which
> will stay).
>
> best,
> Colin
>
>
> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > Hi Colin.
> >
> > <<< It seems like this [--zk-tls-config-file information] could just
> appear
> > in a configuration file, which all of these tools already accept (I
> think)
> >
> > ZkSecurityMigrator has no such property file facility; adding a
> > "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > (the same code is simply reused; it ends up being just a few extra
> lines).
> > I do not see any parameter in the other two commands to adjust the ZK
> > connection; ConfigCommand accepts a "--command-config" flag, but
> according
> > to the code "This is used only with --bootstrap-server option for
> > describing and altering broker configs."
> >
> > I do agree there would be no need to add "--zk-tls-config-file" to
> > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> time
> > for the next release.
> >
> > ConfigCommand supports the "--bootstrap-server" option and will have its
> > direct ZooKeeper access formally deprecated as per KIP-555, but the
> special
> > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > prior to starting Kafka will still exist, so it feels like while
> > "--zk-tls-config-file" would never be used except for this use case it
> > could probably still be added for this particular situation.
> >
> > Ron
> >
> > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > sent) did not go through. I am trying to get emails through now to move
> > this discussion forward.
> >
> > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe wrote:
> >
> > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > Hi everyone. I would like to make the following changes to the KIP.
> > > >
> > > > MOTIVATION:
> > > > Include a statement that it will be difficult in the short term to
> > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > (which
> > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> cluster
> > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > case;
> > > > therefore it is in scope to be able to securely configure the CLI
> tools
> > > > that still leverage non-deprecated direct Zookeeper communication
> for TLS
> > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > zookeeper-security-migration.sh).
> > >
> > > Hi Ron,
> > >
> > > Thanks for the KIP.
> > >
> > > About deprecations:
> > >
> > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> one
> > > would not make sense, since it would defeat the whole point of the
> tool :)
> > >
> > > * kafka-reassign-partitions: ZK access should be deprecated here.
> KIP-455
> > > implementation has dragged a bit, but this should get done soon.
> Certainly
> > > before the next release.
> > >
> > > * kafka-configs: I think ZK access should be deprecated here as well.
> I
> > > agree there is a super-special bootstrapping case here, but that should
> > > have its own tool, not use kafka-configs.
> > >
> > > I will post a separate KIP for this, though.
> > >
> > > >
> > > > GOALS:
> > > > Support the secure configuration of TLS-encrypted communication
> between
> > > > Zookeeper and:
> > > > a) Kafka brokers
> > > > b) The three CLI tools mentioned above that still support direct,
> > > > non-deprecated communication to Zookeeper
> > > > It is explic
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Ron,
Thanks for the explanation. I guess thinking about it a little bit more, we
should just add --zk-tls-config-file to all of these commands.
We will be removing this option (plus ZK in general) from these commands in the
next major release, but ZK is still supported in 2.5, so we should just do the
logical thing. (The exception is ZkSecurityMigrator, which will stay).
best,
Colin
On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> Hi Colin.
>
> <<< It seems like this [--zk-tls-config-file information] could just appear
> in a configuration file, which all of these tools already accept (I think)
>
> ZkSecurityMigrator has no such property file facility; adding a
> "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> that to ZkSecurityMigrator then it is trivial to add it to other commands
> (the same code is simply reused; it ends up being just a few extra lines).
> I do not see any parameter in the other two commands to adjust the ZK
> connection; ConfigCommand accepts a "--command-config" flag, but according
> to the code "This is used only with --bootstrap-server option for
> describing and altering broker configs."
>
> I do agree there would be no need to add "--zk-tls-config-file" to
> ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
> for the next release.
>
> ConfigCommand supports the "--bootstrap-server" option and will have its
> direct ZooKeeper access formally deprecated as per KIP-555, but the special
> use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> prior to starting Kafka will still exist, so it feels like while
> "--zk-tls-config-file" would never be used except for this use case it
> could probably still be added for this particular situation.
>
> Ron
>
> P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> sent) did not go through. I am trying to get emails through now to move
> this discussion forward.
>
> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe wrote:
>
> > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > Hi everyone. I would like to make the following changes to the KIP.
> > >
> > > MOTIVATION:
> > > Include a statement that it will be difficult in the short term to
> > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > (which
> > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > case;
> > > therefore it is in scope to be able to securely configure the CLI tools
> > > that still leverage non-deprecated direct Zookeeper communication for TLS
> > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > zookeeper-security-migration.sh).
> >
> > Hi Ron,
> >
> > Thanks for the KIP.
> >
> > About deprecations:
> >
> > * zookeeper-security-migration: clearly, deprecating ZK access in this one
> > would not make sense, since it would defeat the whole point of the tool :)
> >
> > * kafka-reassign-partitions: ZK access should be deprecated here. KIP-455
> > implementation has dragged a bit, but this should get done soon. Certainly
> > before the next release.
> >
> > * kafka-configs: I think ZK access should be deprecated here as well. I
> > agree there is a super-special bootstrapping case here, but that should
> > have its own tool, not use kafka-configs.
> >
> > I will post a separate KIP for this, though.
> >
> > >
> > > GOALS:
> > > Support the secure configuration of TLS-encrypted communication between
> > > Zookeeper and:
> > > a) Kafka brokers
> > > b) The three CLI tools mentioned above that still support direct,
> > > non-deprecated communication to Zookeeper
> > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > communication in CLI tools as part of this KIP; such work will occur in
> > > future KIPs instead.
> > >
> > > PUBLIC INTERFACES:
> > > 1) The following new broker configurations will be recognized.
> > > zookeeper.client.secure (default value = false, for backwards
> > > compatibility)
> > > zookeeper.clientCnxnSocket
> > > zookeeper.ssl.keyStore.location
> > > zookeeper.ssl.keyStore.password
> > > zookeeper.ssl.trustStore.location
> > > zookeeper.ssl.trustStore.password
> > > It will be an error for any of the last 5 values to be left unspecified
> > if
> > > zookeeper.client.secure is explicitly set to true.
> > >
> > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > supports
> > > the ability to connect to a different Zookeeper instance than the one the
> > > brokers use. We therefore also add the following optional configs, which
> > > override the corresponding ones from above when present:
> > > authorizer.zookeeper.client.secure
> > > authorizer.zookeeper.clientCnxnSocket
> > > authorizer.zookeeper.ssl.keyStore.location
> > > authorizer.zookeeper.ssl.keyStore.password
> > > authorizer.zookeeper.ssl.trustStore.location
> >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Rajini. Thanks for pointing out kafka-acls as another bootstrap use
case. I will update the KIP to mention it. (BTW, I sent an email on 1/8
stating that I had updated the KIP to reflect the current understanding and
discussion, but that email did not go through -- the KIP has undergone a
major rewrite, so interested parties should definitely take a look).
It is easy to support the bootstrap use case as I described previously, so
I think we should support it in both kafka-acls and kafka-configs.
Regarding SASL + SSL simultaneously, yes, the two can be used together in
ZooKeeper. I will be sure to state this in any documentation, and I will
update the KIP to fully describe how this works, including the principal
that is used and whether there is any way to transform the DN in the
certificate. Here is a quick summary:
1) ZooKeeper 3.5.6 requires a client certificate. There is a sever-side
config "ssl.clientAuth" that the code recognizes (case-insensitively:
want/need/none are the valid options), but this config has no effect in
3.5.6 (https://issues.apache.org/jira/browse/ZOOKEEPER-3674). I built from
source recently and confirmed that it worked in the 3.6 SNAPSHOT, but that
version is not yet released.
2) ZooKeeper will associate multiple identities with any session that
authenticates multiple ways (e.g. SSL and SASL). A client is authorized if
it has at least 1 of the identities present in an ACL.
3) The SSL identity is the DN, and that can only be changed by implementing
and using a custom ZK authentication provider that overrides the
method protected
String getClientId(X509Certificate clientCert).
Given the above, if clients want to use client certificate authentication
with SASL, then there really isn't any difficulty -- brokers and ZK
security migrator will be identified by both their respective DNs and the
SASL identity, and at least the SASL identity will be common to all clients
as it is today, so it continues to work fine.
If clients want to use client certificate authentication without SASL
(which is I believe what Pere intended in the original KIP discussion),
then either all brokers and ZK security migrator will have to present the
same DN (with different SANs for the different hostnames) or ZooKeeper will
have to be running with a custom authProvider that maps the different DNs
to a single, common identity (this second approach is more difficult and
seems to be less secure than the first one that uses SANs since it creates
the possibility of the DN mapping being leveraged in a malicious way).
Since I am catching up on emails I sent that did not go through, I'll also
include the following two comments here:
1) The ZooKeeper project's command line interface has no facility for
accepting client TLS configuration except via Java system properties. I've
opened ZooKeeper issue https://issues.apache.org/jira/browse/ZOOKEEPER-3689 for
this, but in the meantime we can address it in Kafka's
zookeeper-shell.{bat,sh} scripts by invoking a new, Kafka-specific subclass
of org.apache.zookeeper.ZooKeeperMain as opposed
to org.apache.zookeeper.ZooKeeperMain itself. We would simply look for a
"-zk-tls-config-file " argument pair, create the TLS client config if
specified, and pass the arguments without the config information to the
superclass; then we would override the connectToZK(String) method to use
any created TLS client config. While the subclass unfortunately needs to
be in the same org.apache.zookeeper package due to the package-private
run() method in ZooKeeperMain that it has to call, it is working fine
(confirmed via a system test).
2) Regarding Thomas Zhou's question on 1/10 about changing the way client
certificates are loaded (
https://lists.apache.org/thread.html/r3bf6c58b052bf47bc99c360dff86a8bd38c35a89a7387ba2a5f16dee%40%3Cdev.kafka.apache.org%3E),
when I updated the KIP I included all of the ZooKeeper-supported
client configurations related to TLS (not just keystore/trsustore locations
and passwords). There is no intention to support anything beyond what
ZooKeeper supports, but the "zookeeper.ssl.context.supplier.class"
configuration is included and is useful for this particular use case.
Ron
On Tue, Jan 14, 2020 at 10:48 AM Rajini Sivaram
wrote:
> Resending this note which wasn't delivered when it was sent on Jan 6th:
>
> Hi Ron,
>
> Thanks for taking over this KIP. Is it possible to use SASL/GSSAPI with SSL
> for ZooKeeper like we do in Kafka? If not, we need to describe that in the
> migration section. We should also document the principals used for
> authorization when SSL is used and any additional configs that may be used
> for transforming the DN from the certificate.
>
> For the CLI tools, the two that use ZooKeeper for bootstrapping are
> kafka-configs.sh and kafka-acls.sh. In both cases, we support direct
> connectivity to ZooKeeper in order to store configs/ACLs prior to starting
> brokers. Are we going to support SSL for both?
>
> Regards,
>
> Rajini
>
> On
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Resending this note which wasn't delivered when it was sent on Jan 6th:
Hi Ron,
Thanks for taking over this KIP. Is it possible to use SASL/GSSAPI with SSL
for ZooKeeper like we do in Kafka? If not, we need to describe that in the
migration section. We should also document the principals used for
authorization when SSL is used and any additional configs that may be used
for transforming the DN from the certificate.
For the CLI tools, the two that use ZooKeeper for bootstrapping are
kafka-configs.sh and kafka-acls.sh. In both cases, we support direct
connectivity to ZooKeeper in order to store configs/ACLs prior to starting
brokers. Are we going to support SSL for both?
Regards,
Rajini
On Tue, Jan 14, 2020 at 3:39 PM Ron Dagostino wrote:
> Hi Colin.
>
> <<< It seems like this [--zk-tls-config-file information] could just appear
> in a configuration file, which all of these tools already accept (I think)
>
> ZkSecurityMigrator has no such property file facility; adding a
> "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> that to ZkSecurityMigrator then it is trivial to add it to other commands
> (the same code is simply reused; it ends up being just a few extra lines).
> I do not see any parameter in the other two commands to adjust the ZK
> connection; ConfigCommand accepts a "--command-config" flag, but according
> to the code "This is used only with --bootstrap-server option for
> describing and altering broker configs."
>
> I do agree there would be no need to add "--zk-tls-config-file" to
> ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
> for the next release.
>
> ConfigCommand supports the "--bootstrap-server" option and will have its
> direct ZooKeeper access formally deprecated as per KIP-555, but the special
> use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> prior to starting Kafka will still exist, so it feels like while
> "--zk-tls-config-file" would never be used except for this use case it
> could probably still be added for this particular situation.
>
> Ron
>
> P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> sent) did not go through. I am trying to get emails through now to move
> this discussion forward.
>
> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe wrote:
>
> > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > Hi everyone. I would like to make the following changes to the KIP.
> > >
> > > MOTIVATION:
> > > Include a statement that it will be difficult in the short term to
> > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > (which
> > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > case;
> > > therefore it is in scope to be able to securely configure the CLI tools
> > > that still leverage non-deprecated direct Zookeeper communication for
> TLS
> > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > zookeeper-security-migration.sh).
> >
> > Hi Ron,
> >
> > Thanks for the KIP.
> >
> > About deprecations:
> >
> > * zookeeper-security-migration: clearly, deprecating ZK access in this
> one
> > would not make sense, since it would defeat the whole point of the tool
> :)
> >
> > * kafka-reassign-partitions: ZK access should be deprecated here.
> KIP-455
> > implementation has dragged a bit, but this should get done soon.
> Certainly
> > before the next release.
> >
> > * kafka-configs: I think ZK access should be deprecated here as well. I
> > agree there is a super-special bootstrapping case here, but that should
> > have its own tool, not use kafka-configs.
> >
> > I will post a separate KIP for this, though.
> >
> > >
> > > GOALS:
> > > Support the secure configuration of TLS-encrypted communication between
> > > Zookeeper and:
> > > a) Kafka brokers
> > > b) The three CLI tools mentioned above that still support direct,
> > > non-deprecated communication to Zookeeper
> > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > communication in CLI tools as part of this KIP; such work will occur in
> > > future KIPs instead.
> > >
> > > PUBLIC INTERFACES:
> > > 1) The following new broker configurations will be recognized.
> > > zookeeper.client.secure (default value = false, for backwards
> > > compatibility)
> > > zookeeper.clientCnxnSocket
> > > zookeeper.ssl.keyStore.location
> > > zookeeper.ssl.keyStore.password
> > > zookeeper.ssl.trustStore.location
> > > zookeeper.ssl.trustStore.password
> > > It will be an error for any of the last 5 values to be left unspecified
> > if
> > > zookeeper.client.secure is explicitly set to true.
> > >
> > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > supports
> > > the ability to connect to a different Zookeeper instance than the one
> the
> > > brokers use. We therefore also add the following optional configs,
> which
> >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Colin.
<<< It seems like this [--zk-tls-config-file information] could just appear
in a configuration file, which all of these tools already accept (I think)
ZkSecurityMigrator has no such property file facility; adding a
"--zk-tls-config-file" parameter is exactly for this purpose. If we add
that to ZkSecurityMigrator then it is trivial to add it to other commands
(the same code is simply reused; it ends up being just a few extra lines).
I do not see any parameter in the other two commands to adjust the ZK
connection; ConfigCommand accepts a "--command-config" flag, but according
to the code "This is used only with --bootstrap-server option for
describing and altering broker configs."
I do agree there would be no need to add "--zk-tls-config-file" to
ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
for the next release.
ConfigCommand supports the "--bootstrap-server" option and will have its
direct ZooKeeper access formally deprecated as per KIP-555, but the special
use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
prior to starting Kafka will still exist, so it feels like while
"--zk-tls-config-file" would never be used except for this use case it
could probably still be added for this particular situation.
Ron
P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
sent) did not go through. I am trying to get emails through now to move
this discussion forward.
On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe wrote:
> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > Hi everyone. I would like to make the following changes to the KIP.
> >
> > MOTIVATION:
> > Include a statement that it will be difficult in the short term to
> > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> (which
> > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > with encrypted passwords in Zookeeper is an explicitly-supported use
> case;
> > therefore it is in scope to be able to securely configure the CLI tools
> > that still leverage non-deprecated direct Zookeeper communication for TLS
> > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > zookeeper-security-migration.sh).
>
> Hi Ron,
>
> Thanks for the KIP.
>
> About deprecations:
>
> * zookeeper-security-migration: clearly, deprecating ZK access in this one
> would not make sense, since it would defeat the whole point of the tool :)
>
> * kafka-reassign-partitions: ZK access should be deprecated here. KIP-455
> implementation has dragged a bit, but this should get done soon. Certainly
> before the next release.
>
> * kafka-configs: I think ZK access should be deprecated here as well. I
> agree there is a super-special bootstrapping case here, but that should
> have its own tool, not use kafka-configs.
>
> I will post a separate KIP for this, though.
>
> >
> > GOALS:
> > Support the secure configuration of TLS-encrypted communication between
> > Zookeeper and:
> > a) Kafka brokers
> > b) The three CLI tools mentioned above that still support direct,
> > non-deprecated communication to Zookeeper
> > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > communication in CLI tools as part of this KIP; such work will occur in
> > future KIPs instead.
> >
> > PUBLIC INTERFACES:
> > 1) The following new broker configurations will be recognized.
> > zookeeper.client.secure (default value = false, for backwards
> > compatibility)
> > zookeeper.clientCnxnSocket
> > zookeeper.ssl.keyStore.location
> > zookeeper.ssl.keyStore.password
> > zookeeper.ssl.trustStore.location
> > zookeeper.ssl.trustStore.password
> > It will be an error for any of the last 5 values to be left unspecified
> if
> > zookeeper.client.secure is explicitly set to true.
> >
> > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> supports
> > the ability to connect to a different Zookeeper instance than the one the
> > brokers use. We therefore also add the following optional configs, which
> > override the corresponding ones from above when present:
> > authorizer.zookeeper.client.secure
> > authorizer.zookeeper.clientCnxnSocket
> > authorizer.zookeeper.ssl.keyStore.location
> > authorizer.zookeeper.ssl.keyStore.password
> > authorizer.zookeeper.ssl.trustStore.location
> > authorizer.zookeeper.ssl.trustStore.password
> >
> > 3) The three CLI tools mentioned above will support a new
> --zk-tls-config-file
> > " option. The following
> > properties will be recognized in that file, and unrecognized properties
> > will be ignored to allow the possibility of pointing zk-tls-config-file
> at
> > the broker's config file.
> > zookeeper.client.secure (default value = false)
> > zookeeper.clientCnxnSocket
> > zookeeper.ssl.keyStore.location
> > zookeeper.ssl.keyStore.password
> > zookeeper.ssl.trustStore.location
> > zookeeper.ssl.trustStore.password
> > It will be an error for any of the last 5 values to be
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Pere, It is a very meaningful KIP to make kafka broker -> ZK connection secured. In the meanwhile, there is another KIP under discussion talking about making SSLContext pluggle on broker side - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952. Instead of putting credentials on file, this can load credentials in a custom way into cache. I think for Zookeeper this feature can also be valid. Could you please kindly take a look at that KIP and take the idea into consideration? Thanks, Thomas On 9/2/19, 5:23 AM, "Pere Urbón Bayes" wrote: Thanks for your time Harsha, anyone else with comments? looking forward to hearing from you. Stupid question: when do you move from discussion to vote? Missatge de Harsha Chintalapani del dia dv., 30 d’ag. 2019 a les 21:59: > Thanks Pere. KIP looks good to me. > -Harsha > > > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes > wrote: > >> Not really, >> my idea is to keep the JAAS parameter, so people don't see major >> changes. But if you pass a properties file, then this takes precedence over >> the other, with the idea that you can do sasl as well with the properties >> files. >> >> Makes sense? >> >> -- Pere >> >> Missatge de Harsha Chintalapani del dia dv., 30 d’ag. >> 2019 a les 19:00: >> >>> Hi Pere, >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes >>> sense. >>> "The changes are planned to be introduced in a compatible way, by >>> keeping the current JAAS variable precedence." >>> Can you elaborate a bit here. If the user configures a JAAS file with >>> Client section it will take precedence over zookeeper SSL configs? >>> >>> Thanks, >>> Harsha >>> >>> >>> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes >>> wrote: >>> Hi, quick question, I saw in another mail that 2.4 release is planned for September. I think it would be really awesome to have this for this release, do you think we can make it? -- Pere Missatge de Pere Urbón Bayes del dia dj., 29 d’ag. 2019 a les 20:10: Hi, this is my first KIP for a change in Apache Kafka, so I'm really need to the process. Looking forward to hearing from you and learn the best ropes here. I would like to propose this KIP-515 to enable the ZookeeperClients to take full advantage of the TLS communication in the new Zookeeper 3.5.5. Specially interesting it the Zookeeper Security Migration, that without this change will not work with TLS, disabling users to use ACLs when the Zookeeper cluster use TLS. link: https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-515%253A%2BEnable%2BZK%2Bclient%2Bto%2Buse%2Bthe%2Bnew%2BTLS%2Bsupported%2Bauthentication&data=01%7C01%7Cthzhou%40paypal.com%7C5d7c9fe278a44846502e08d72fa05dab%7Cfb00791460204374977e21bac5f3f4c8%7C1&sdata=PrNRY3teOpZ4cvmI%2FIGofhZhOs5lb2b7b5Hif9jTYH0%3D&reserved=0 Looking forward to hearing from you on this, /cheers -- Pere Urbon-Bayes Software Architect https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.purbon.com&data=01%7C01%7Cthzhou%40paypal.com%7C5d7c9fe278a44846502e08d72fa05dab%7Cfb00791460204374977e21bac5f3f4c8%7C1&sdata=tXdEiq2%2BeivI2Xo9a3r2c6v9LRK4eXp6sFovEzZ7NEY%3D&reserved=0 https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fpurbon&data=01%7C01%7Cthzhou%40paypal.com%7C5d7c9fe278a44846502e08d72fa05dab%7Cfb00791460204374977e21bac5f3f4c8%7C1&sdata=hht3hwCEu0kS4feTn58HO36Rw2rgF7wSrfn8VRyzzU4%3D&reserved=0 https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fpurbon%2F&data=01%7C01%7Cthzhou%40paypal.com%7C5d7c9fe278a44846502e08d72fa05dab%7Cfb00791460204374977e21bac5f3f4c8%7C1&sdata=XOPFUsKlAT7TxfF%2Ff%2BAKdN1r4lFg5reE8%2F7mbvWq5UI%3D&reserved=0 -- Pere Urbon-Bayes Software Architect https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.purbon.com&data=01%7C01%7Cthzhou%40paypal.com%7C5d7c9fe278a44846502e08d72fa05dab%7Cfb00791460204374977e21bac5f3f4c8%7C1&sdata=tXdEiq2%2BeivI2Xo9a3r2c6v9LRK4eXp6sFovEzZ7NEY%3D&reserved=0 https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fpurbon&data=01%7C01%7Cthzhou%40paypal.com%7C5d7c9fe278a44846502e08d72fa05dab%7Cfb00791460204374977e21bac5f3f4c8%7C1&sdata=hht3hwCEu0kS4feTn58HO36Rw2rgF7wSrfn8VRyzzU4%3D&reserved=0 https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fpurbon%2F&d
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> Hi everyone. I would like to make the following changes to the KIP.
>
> MOTIVATION:
> Include a statement that it will be difficult in the short term to
> deprecate direct Zookeeper communication in kafka-configs.{sh, bat} (which
> invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> with encrypted passwords in Zookeeper is an explicitly-supported use case;
> therefore it is in scope to be able to securely configure the CLI tools
> that still leverage non-deprecated direct Zookeeper communication for TLS
> (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> zookeeper-security-migration.sh).
Hi Ron,
Thanks for the KIP.
About deprecations:
* zookeeper-security-migration: clearly, deprecating ZK access in this one
would not make sense, since it would defeat the whole point of the tool :)
* kafka-reassign-partitions: ZK access should be deprecated here. KIP-455
implementation has dragged a bit, but this should get done soon. Certainly
before the next release.
* kafka-configs: I think ZK access should be deprecated here as well. I agree
there is a super-special bootstrapping case here, but that should have its own
tool, not use kafka-configs.
I will post a separate KIP for this, though.
>
> GOALS:
> Support the secure configuration of TLS-encrypted communication between
> Zookeeper and:
> a) Kafka brokers
> b) The three CLI tools mentioned above that still support direct,
> non-deprecated communication to Zookeeper
> It is explicitly out-of-scope to deprecate any direct Zookeeper
> communication in CLI tools as part of this KIP; such work will occur in
> future KIPs instead.
>
> PUBLIC INTERFACES:
> 1) The following new broker configurations will be recognized.
> zookeeper.client.secure (default value = false, for backwards
> compatibility)
> zookeeper.clientCnxnSocket
> zookeeper.ssl.keyStore.location
> zookeeper.ssl.keyStore.password
> zookeeper.ssl.trustStore.location
> zookeeper.ssl.trustStore.password
> It will be an error for any of the last 5 values to be left unspecified if
> zookeeper.client.secure is explicitly set to true.
>
> 2) In addition, the kafka.security.authorizer.AclAuthorizer class supports
> the ability to connect to a different Zookeeper instance than the one the
> brokers use. We therefore also add the following optional configs, which
> override the corresponding ones from above when present:
> authorizer.zookeeper.client.secure
> authorizer.zookeeper.clientCnxnSocket
> authorizer.zookeeper.ssl.keyStore.location
> authorizer.zookeeper.ssl.keyStore.password
> authorizer.zookeeper.ssl.trustStore.location
> authorizer.zookeeper.ssl.trustStore.password
>
> 3) The three CLI tools mentioned above will support a new --zk-tls-config-file
> " option. The following
> properties will be recognized in that file, and unrecognized properties
> will be ignored to allow the possibility of pointing zk-tls-config-file at
> the broker's config file.
> zookeeper.client.secure (default value = false)
> zookeeper.clientCnxnSocket
> zookeeper.ssl.keyStore.location
> zookeeper.ssl.keyStore.password
> zookeeper.ssl.trustStore.location
> zookeeper.ssl.trustStore.password
> It will be an error for any of the last 5 values to be left unspecified if
> zookeeper.client.secure is explicitly set to true.
Do we really need a --zk-tls-config-file flag? It seems like this could just
appear in a configuration file, which all of these tools already accept (I
think).
best,
Colin
>
> Ron
>
> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino wrote:
>
> > Hi everyone. Let's get this discussion going again now that Kafka 2.4
> > with Zookeeper 3.5.6 is out.
> >
> > First, regarding the KIP number, the other KIP that was using this number
> > moved to KIP 534, so KIP 515 remains the correct number for this
> > discussion. I've updated the Kafka Improvement Proposal page to list this
> > KIP in the 515 slot, so we're all set there.
> >
> > Regarding the actual issues under discussion, I think there are some
> > things we should clarify.
> >
> > 1) It is possible to use TLS connectivity to Zookeeper from Apache Kafka
> > 2.4 -- the problem is that configuration information has to be passed via
> > system properties as "-D" command line options on the java invocation of
> > the broker, and those are not secure (anyone with access to the box can see
> > the command line used to invoke the broker); the configuration includes
> > sensitive information (e.g. a keystore password), so we need a secure
> > mechanism for passing the configuration values. I believe the real
> > motivation for this KIP is to harden the configuration mechanism for
> > Zookeeper TLS connectivity.
> >
> > 2) I believe the list of CLI tools that continue to use direct Zookeeper
> > connectivity in a non-deprecated fashion is:
> > a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi everyone. I would like to make the following changes to the KIP.
MOTIVATION:
Include a statement that it will be difficult in the short term to
deprecate direct Zookeeper communication in kafka-configs.{sh, bat} (which
invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
with encrypted passwords in Zookeeper is an explicitly-supported use case;
therefore it is in scope to be able to securely configure the CLI tools
that still leverage non-deprecated direct Zookeeper communication for TLS
(the other 2 tools are kafka-reassign-partitions.{sh, bat} and
zookeeper-security-migration.sh).
GOALS:
Support the secure configuration of TLS-encrypted communication between
Zookeeper and:
a) Kafka brokers
b) The three CLI tools mentioned above that still support direct,
non-deprecated communication to Zookeeper
It is explicitly out-of-scope to deprecate any direct Zookeeper
communication in CLI tools as part of this KIP; such work will occur in
future KIPs instead.
PUBLIC INTERFACES:
1) The following new broker configurations will be recognized.
zookeeper.client.secure (default value = false, for backwards
compatibility)
zookeeper.clientCnxnSocket
zookeeper.ssl.keyStore.location
zookeeper.ssl.keyStore.password
zookeeper.ssl.trustStore.location
zookeeper.ssl.trustStore.password
It will be an error for any of the last 5 values to be left unspecified if
zookeeper.client.secure is explicitly set to true.
2) In addition, the kafka.security.authorizer.AclAuthorizer class supports
the ability to connect to a different Zookeeper instance than the one the
brokers use. We therefore also add the following optional configs, which
override the corresponding ones from above when present:
authorizer.zookeeper.client.secure
authorizer.zookeeper.clientCnxnSocket
authorizer.zookeeper.ssl.keyStore.location
authorizer.zookeeper.ssl.keyStore.password
authorizer.zookeeper.ssl.trustStore.location
authorizer.zookeeper.ssl.trustStore.password
3) The three CLI tools mentioned above will support a new --zk-tls-config-file
" option. The following
properties will be recognized in that file, and unrecognized properties
will be ignored to allow the possibility of pointing zk-tls-config-file at
the broker's config file.
zookeeper.client.secure (default value = false)
zookeeper.clientCnxnSocket
zookeeper.ssl.keyStore.location
zookeeper.ssl.keyStore.password
zookeeper.ssl.trustStore.location
zookeeper.ssl.trustStore.password
It will be an error for any of the last 5 values to be left unspecified if
zookeeper.client.secure is explicitly set to true.
Ron
On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino wrote:
> Hi everyone. Let's get this discussion going again now that Kafka 2.4
> with Zookeeper 3.5.6 is out.
>
> First, regarding the KIP number, the other KIP that was using this number
> moved to KIP 534, so KIP 515 remains the correct number for this
> discussion. I've updated the Kafka Improvement Proposal page to list this
> KIP in the 515 slot, so we're all set there.
>
> Regarding the actual issues under discussion, I think there are some
> things we should clarify.
>
> 1) It is possible to use TLS connectivity to Zookeeper from Apache Kafka
> 2.4 -- the problem is that configuration information has to be passed via
> system properties as "-D" command line options on the java invocation of
> the broker, and those are not secure (anyone with access to the box can see
> the command line used to invoke the broker); the configuration includes
> sensitive information (e.g. a keystore password), so we need a secure
> mechanism for passing the configuration values. I believe the real
> motivation for this KIP is to harden the configuration mechanism for
> Zookeeper TLS connectivity.
>
> 2) I believe the list of CLI tools that continue to use direct Zookeeper
> connectivity in a non-deprecated fashion is:
> a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> b)
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
>
> 3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
> explicitly states in a comment that a supported use case is bootstrapping a
> Kafka cluster with encrypted passwords in Zookeeper (see
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65).
> This means it will be especially difficult to deprecate this particular
> direct Zookeeper connectivity without a different storage mechanism for
> dynamic configuration values being available (i.e. the self-managed quorum
> referred to in KIP-500).
>
> I think it would be easier and simpler to harden the Zookeeper TLS
> configuration in both Kafka and in CLI tools compared to trying to
> deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
> especially when ConfigCommand has no obvious short-term path to deprecation.
>
> Regarding how to harden the
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi everyone. Let's get this discussion going again now that Kafka 2.4 with
Zookeeper 3.5.6 is out.
First, regarding the KIP number, the other KIP that was using this number
moved to KIP 534, so KIP 515 remains the correct number for this
discussion. I've updated the Kafka Improvement Proposal page to list this
KIP in the 515 slot, so we're all set there.
Regarding the actual issues under discussion, I think there are some things
we should clarify.
1) It is possible to use TLS connectivity to Zookeeper from Apache Kafka
2.4 -- the problem is that configuration information has to be passed via
system properties as "-D" command line options on the java invocation of
the broker, and those are not secure (anyone with access to the box can see
the command line used to invoke the broker); the configuration includes
sensitive information (e.g. a keystore password), so we need a secure
mechanism for passing the configuration values. I believe the real
motivation for this KIP is to harden the configuration mechanism for
Zookeeper TLS connectivity.
2) I believe the list of CLI tools that continue to use direct Zookeeper
connectivity in a non-deprecated fashion is:
a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
b)
kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
explicitly states in a comment that a supported use case is bootstrapping a
Kafka cluster with encrypted passwords in Zookeeper (see
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65).
This means it will be especially difficult to deprecate this particular
direct Zookeeper connectivity without a different storage mechanism for
dynamic configuration values being available (i.e. the self-managed quorum
referred to in KIP-500).
I think it would be easier and simpler to harden the Zookeeper TLS
configuration in both Kafka and in CLI tools compared to trying to
deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
especially when ConfigCommand has no obvious short-term path to deprecation.
Regarding how to harden the TLS configuration, adding the appropriate
configs to Kafka is an obvious choice for the brokers. Not that these
values cannot be dynamically reconfigurable because the values would be
required to connect to Zookeeper via TLS in the first place.
Regarding how to harden the TLS configuration for the 3 remaining CLI
tools, it would be relatively straightforward to add support for a
"--zk-tls-config-file " option to each one.
Thoughts?
Ron
On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes
wrote:
> True, the number is duplicated.
>
> do you know how can we get the problem fix? I was not aware, I missed,
> sorry the need to add the KIP to the table.
>
> -- Pere
>
> Missatge de Jorge Esteban Quilcate Otoya del
> dia
> dt., 3 de set. 2019 a les 13:43:
>
> > Hi Pere,
> >
> > Have you add your KIP to the list here
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> ?
> >
> > I found the KIP number assigned to another.
> >
> >
> >
> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes
> > wrote:
> >
> >> Thanks for your time Harsha,
> >>anyone else with comments? looking forward to hearing from you.
> >>
> >> Stupid question: when do you move from discussion to vote?
> >>
> >> Missatge de Harsha Chintalapani del dia dv., 30 d’ag.
> >> 2019 a les 21:59:
> >>
> >> > Thanks Pere. KIP looks good to me.
> >> > -Harsha
> >> >
> >> >
> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> >> pere.ur...@gmail.com>
> >> > wrote:
> >> >
> >> >> Not really,
> >> >> my idea is to keep the JAAS parameter, so people don't see major
> >> >> changes. But if you pass a properties file, then this takes
> precedence
> >> over
> >> >> the other, with the idea that you can do sasl as well with the
> >> properties
> >> >> files.
> >> >>
> >> >> Makes sense?
> >> >>
> >> >> -- Pere
> >> >>
> >> >> Missatge de Harsha Chintalapani del dia dv., 30
> >> d’ag.
> >> >> 2019 a les 19:00:
> >> >>
> >> >>> Hi Pere,
> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka
> >> makes
> >> >>> sense.
> >> >>> "The changes are planned to be introduced in a compatible way, by
> >> >>> keeping the current JAAS variable precedence."
> >> >>> Can you elaborate a bit here. If the user configures a JAAS file
> with
> >> >>> Client section it will take precedence over zookeeper SSL configs?
> >> >>>
> >> >>> Thanks,
> >> >>> Harsha
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> >> pere.ur...@gmail.com>
> >> >>> wrote:
> >> >>>
> >> Hi,
> >> quick question, I saw in another mail that 2.4 release is planned
> for
> >> September. I think it would be really awesome to have this for this
> >> release, do you think we can make it?
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
True, the number is duplicated. do you know how can we get the problem fix? I was not aware, I missed, sorry the need to add the KIP to the table. -- Pere Missatge de Jorge Esteban Quilcate Otoya del dia dt., 3 de set. 2019 a les 13:43: > Hi Pere, > > Have you add your KIP to the list here > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals? > > I found the KIP number assigned to another. > > > > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes > wrote: > >> Thanks for your time Harsha, >>anyone else with comments? looking forward to hearing from you. >> >> Stupid question: when do you move from discussion to vote? >> >> Missatge de Harsha Chintalapani del dia dv., 30 d’ag. >> 2019 a les 21:59: >> >> > Thanks Pere. KIP looks good to me. >> > -Harsha >> > >> > >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes < >> pere.ur...@gmail.com> >> > wrote: >> > >> >> Not really, >> >> my idea is to keep the JAAS parameter, so people don't see major >> >> changes. But if you pass a properties file, then this takes precedence >> over >> >> the other, with the idea that you can do sasl as well with the >> properties >> >> files. >> >> >> >> Makes sense? >> >> >> >> -- Pere >> >> >> >> Missatge de Harsha Chintalapani del dia dv., 30 >> d’ag. >> >> 2019 a les 19:00: >> >> >> >>> Hi Pere, >> >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka >> makes >> >>> sense. >> >>> "The changes are planned to be introduced in a compatible way, by >> >>> keeping the current JAAS variable precedence." >> >>> Can you elaborate a bit here. If the user configures a JAAS file with >> >>> Client section it will take precedence over zookeeper SSL configs? >> >>> >> >>> Thanks, >> >>> Harsha >> >>> >> >>> >> >>> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes < >> pere.ur...@gmail.com> >> >>> wrote: >> >>> >> Hi, >> quick question, I saw in another mail that 2.4 release is planned for >> September. I think it would be really awesome to have this for this >> release, do you think we can make it? >> >> -- Pere >> >> Missatge de Pere Urbón Bayes del dia dj., 29 >> d’ag. 2019 a les 20:10: >> >> Hi, >> this is my first KIP for a change in Apache Kafka, so I'm really need >> to the process. Looking forward to hearing from you and learn the >> best >> ropes here. >> >> I would like to propose this KIP-515 to enable the ZookeeperClients >> to >> take full advantage of the TLS communication in the new Zookeeper >> 3.5.5. >> Specially interesting it the Zookeeper Security Migration, that >> without >> this change will not work with TLS, disabling users to use ACLs when >> the >> Zookeeper cluster use TLS. >> >> link: >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication >> >> Looking forward to hearing from you on this, >> >> /cheers >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> >> >>> >> >>> >> >> >> >> -- >> >> Pere Urbon-Bayes >> >> Software Architect >> >> http://www.purbon.com >> >> https://twitter.com/purbon >> >> https://www.linkedin.com/in/purbon/ >> >> >> > >> > >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Ismael, fair point and totally reasonable point. I added the CLI part to bring compatibility, but I can agree with not doing it right now per compatibility reasons. However, I do have a question for you: You say most of the CLI tools, is there anyone in your mind that you recall that would break if the cluster uses a TLS activated Zookeeper cluster? Only one I can probably think right now is the Zookeeper Security Migration tool, right? If I clean the others, would you agree to move the KIP forward? -- Pere Missatge de Ismael Juma del dia dt., 3 de set. 2019 a les 14:53: > Hi Pere, > > Thanks for the KIP. With regards to the CLI tools, most of them support > direct access to ZK for compatibility reasons and we encourage usage of the > Kafka protocol instead. I am not sure we should be extending them as > described in the KIP. What are your thoughts on that? > > Ismael > > On Thu, Aug 29, 2019 at 11:11 AM Pere Urbón Bayes > wrote: > >> Hi, >> this is my first KIP for a change in Apache Kafka, so I'm really need to >> the process. Looking forward to hearing from you and learn the best ropes >> here. >> >> I would like to propose this KIP-515 to enable the ZookeeperClients to >> take >> full advantage of the TLS communication in the new Zookeeper 3.5.5. >> Specially interesting it the Zookeeper Security Migration, that without >> this change will not work with TLS, disabling users to use ACLs when the >> Zookeeper cluster use TLS. >> >> link: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication >> >> Looking forward to hearing from you on this, >> >> /cheers >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Pere, Thanks for the KIP. With regards to the CLI tools, most of them support direct access to ZK for compatibility reasons and we encourage usage of the Kafka protocol instead. I am not sure we should be extending them as described in the KIP. What are your thoughts on that? Ismael On Thu, Aug 29, 2019 at 11:11 AM Pere Urbón Bayes wrote: > Hi, > this is my first KIP for a change in Apache Kafka, so I'm really need to > the process. Looking forward to hearing from you and learn the best ropes > here. > > I would like to propose this KIP-515 to enable the ZookeeperClients to take > full advantage of the TLS communication in the new Zookeeper 3.5.5. > Specially interesting it the Zookeeper Security Migration, that without > this change will not work with TLS, disabling users to use ACLs when the > Zookeeper cluster use TLS. > > link: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > Looking forward to hearing from you on this, > > /cheers > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Pere, Have you add your KIP to the list here https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals? I found the KIP number assigned to another. On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes wrote: > Thanks for your time Harsha, >anyone else with comments? looking forward to hearing from you. > > Stupid question: when do you move from discussion to vote? > > Missatge de Harsha Chintalapani del dia dv., 30 d’ag. > 2019 a les 21:59: > > > Thanks Pere. KIP looks good to me. > > -Harsha > > > > > > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes > > > wrote: > > > >> Not really, > >> my idea is to keep the JAAS parameter, so people don't see major > >> changes. But if you pass a properties file, then this takes precedence > over > >> the other, with the idea that you can do sasl as well with the > properties > >> files. > >> > >> Makes sense? > >> > >> -- Pere > >> > >> Missatge de Harsha Chintalapani del dia dv., 30 d’ag. > >> 2019 a les 19:00: > >> > >>> Hi Pere, > >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka > makes > >>> sense. > >>> "The changes are planned to be introduced in a compatible way, by > >>> keeping the current JAAS variable precedence." > >>> Can you elaborate a bit here. If the user configures a JAAS file with > >>> Client section it will take precedence over zookeeper SSL configs? > >>> > >>> Thanks, > >>> Harsha > >>> > >>> > >>> > >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes < > pere.ur...@gmail.com> > >>> wrote: > >>> > Hi, > quick question, I saw in another mail that 2.4 release is planned for > September. I think it would be really awesome to have this for this > release, do you think we can make it? > > -- Pere > > Missatge de Pere Urbón Bayes del dia dj., 29 > d’ag. 2019 a les 20:10: > > Hi, > this is my first KIP for a change in Apache Kafka, so I'm really need > to the process. Looking forward to hearing from you and learn the best > ropes here. > > I would like to propose this KIP-515 to enable the ZookeeperClients to > take full advantage of the TLS communication in the new Zookeeper > 3.5.5. > Specially interesting it the Zookeeper Security Migration, that > without > this change will not work with TLS, disabling users to use ACLs when > the > Zookeeper cluster use TLS. > > link: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > Looking forward to hearing from you on this, > > /cheers > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ > > >>> > >>> > >> > >> -- > >> Pere Urbon-Bayes > >> Software Architect > >> http://www.purbon.com > >> https://twitter.com/purbon > >> https://www.linkedin.com/in/purbon/ > >> > > > > > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Thanks for your time Harsha, anyone else with comments? looking forward to hearing from you. Stupid question: when do you move from discussion to vote? Missatge de Harsha Chintalapani del dia dv., 30 d’ag. 2019 a les 21:59: > Thanks Pere. KIP looks good to me. > -Harsha > > > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes > wrote: > >> Not really, >> my idea is to keep the JAAS parameter, so people don't see major >> changes. But if you pass a properties file, then this takes precedence over >> the other, with the idea that you can do sasl as well with the properties >> files. >> >> Makes sense? >> >> -- Pere >> >> Missatge de Harsha Chintalapani del dia dv., 30 d’ag. >> 2019 a les 19:00: >> >>> Hi Pere, >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes >>> sense. >>> "The changes are planned to be introduced in a compatible way, by >>> keeping the current JAAS variable precedence." >>> Can you elaborate a bit here. If the user configures a JAAS file with >>> Client section it will take precedence over zookeeper SSL configs? >>> >>> Thanks, >>> Harsha >>> >>> >>> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes >>> wrote: >>> Hi, quick question, I saw in another mail that 2.4 release is planned for September. I think it would be really awesome to have this for this release, do you think we can make it? -- Pere Missatge de Pere Urbón Bayes del dia dj., 29 d’ag. 2019 a les 20:10: Hi, this is my first KIP for a change in Apache Kafka, so I'm really need to the process. Looking forward to hearing from you and learn the best ropes here. I would like to propose this KIP-515 to enable the ZookeeperClients to take full advantage of the TLS communication in the new Zookeeper 3.5.5. Specially interesting it the Zookeeper Security Migration, that without this change will not work with TLS, disabling users to use ACLs when the Zookeeper cluster use TLS. link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication Looking forward to hearing from you on this, /cheers -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/ -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/ >>> >>> >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> > > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Thanks Pere. KIP looks good to me. -Harsha On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes wrote: > Not really, > my idea is to keep the JAAS parameter, so people don't see major > changes. But if you pass a properties file, then this takes precedence over > the other, with the idea that you can do sasl as well with the properties > files. > > Makes sense? > > -- Pere > > Missatge de Harsha Chintalapani del dia dv., 30 d’ag. > 2019 a les 19:00: > > Hi Pere, > Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes > sense. > "The changes are planned to be introduced in a compatible way, by keeping > the current JAAS variable precedence." > Can you elaborate a bit here. If the user configures a JAAS file with > Client section it will take precedence over zookeeper SSL configs? > > Thanks, > Harsha > > > > On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes > wrote: > > Hi, > quick question, I saw in another mail that 2.4 release is planned for > September. I think it would be really awesome to have this for this > release, do you think we can make it? > > -- Pere > > Missatge de Pere Urbón Bayes del dia dj., 29 d’ag. > 2019 a les 20:10: > > Hi, > this is my first KIP for a change in Apache Kafka, so I'm really need to > the process. Looking forward to hearing from you and learn the best ropes > here. > > I would like to propose this KIP-515 to enable the ZookeeperClients to > take full advantage of the TLS communication in the new Zookeeper 3.5.5. > Specially interesting it the Zookeeper Security Migration, that without > this change will not work with TLS, disabling users to use ACLs when the > Zookeeper cluster use TLS. > > link: > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > Looking forward to hearing from you on this, > > /cheers > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ > > > > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Not really, my idea is to keep the JAAS parameter, so people don't see major changes. But if you pass a properties file, then this takes precedence over the other, with the idea that you can do sasl as well with the properties files. Makes sense? -- Pere Missatge de Harsha Chintalapani del dia dv., 30 d’ag. 2019 a les 19:00: > Hi Pere, > Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes > sense. > "The changes are planned to be introduced in a compatible way, by keeping > the current JAAS variable precedence." > Can you elaborate a bit here. If the user configures a JAAS file with > Client section it will take precedence over zookeeper SSL configs? > > Thanks, > Harsha > > > > On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes > wrote: > >> Hi, >> quick question, I saw in another mail that 2.4 release is planned for >> September. I think it would be really awesome to have this for this >> release, do you think we can make it? >> >> -- Pere >> >> Missatge de Pere Urbón Bayes del dia dj., 29 >> d’ag. 2019 a les 20:10: >> >> Hi, >> this is my first KIP for a change in Apache Kafka, so I'm really need to >> the process. Looking forward to hearing from you and learn the best ropes >> here. >> >> I would like to propose this KIP-515 to enable the ZookeeperClients to >> take full advantage of the TLS communication in the new Zookeeper 3.5.5. >> Specially interesting it the Zookeeper Security Migration, that without >> this change will not work with TLS, disabling users to use ACLs when the >> Zookeeper cluster use TLS. >> >> link: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication >> >> Looking forward to hearing from you on this, >> >> /cheers >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> > > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Pere, Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes sense. "The changes are planned to be introduced in a compatible way, by keeping the current JAAS variable precedence." Can you elaborate a bit here. If the user configures a JAAS file with Client section it will take precedence over zookeeper SSL configs? Thanks, Harsha On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes wrote: > Hi, > quick question, I saw in another mail that 2.4 release is planned for > September. I think it would be really awesome to have this for this > release, do you think we can make it? > > -- Pere > > Missatge de Pere Urbón Bayes del dia dj., 29 d’ag. > 2019 a les 20:10: > > Hi, > this is my first KIP for a change in Apache Kafka, so I'm really need to > the process. Looking forward to hearing from you and learn the best ropes > here. > > I would like to propose this KIP-515 to enable the ZookeeperClients to > take full advantage of the TLS communication in the new Zookeeper 3.5.5. > Specially interesting it the Zookeeper Security Migration, that without > this change will not work with TLS, disabling users to use ACLs when the > Zookeeper cluster use TLS. > > link: > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > Looking forward to hearing from you on this, > > /cheers > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
It helps as a beginner! thanks for sharing! any tip for getting reviewers to take a look? :pray:, :pray: ;-) -- Pere Missatge de Colin McCabe del dia dv., 30 d’ag. 2019 a les 18:57: > Hi Pere, > > It will depend on how quickly the KIP can be voted in, and then > implemented. > > Manikumar Reddy is going to be the release manager for Apache Kafka > 2.4.0. Right now, the KIP freeze is scheduled for Sep 25, 2019. A KIP > must be accepted by this date in order to be part of 2.4.0. (Being > accepted doesn't mean it will be part of 2.4.0, though-- it also has to be > implemented.) Hope this helps. > > cheers, > Colin > > > On Fri, Aug 30, 2019, at 07:50, Pere Urbón Bayes wrote: > > Hi, > > quick question, I saw in another mail that 2.4 release is planned for > > September. I think it would be really awesome to have this for this > > release, do you think we can make it? > > > > -- Pere > > > > Missatge de Pere Urbón Bayes del dia dj., 29 > d’ag. > > 2019 a les 20:10: > > > > > Hi, > > > this is my first KIP for a change in Apache Kafka, so I'm really need > to > > > the process. Looking forward to hearing from you and learn the best > ropes > > > here. > > > > > > I would like to propose this KIP-515 to enable the ZookeeperClients to > > > take full advantage of the TLS communication in the new Zookeeper > 3.5.5. > > > Specially interesting it the Zookeeper Security Migration, that without > > > this change will not work with TLS, disabling users to use ACLs when > the > > > Zookeeper cluster use TLS. > > > > > > link: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > > > > > Looking forward to hearing from you on this, > > > > > > /cheers > > > > > > -- > > > Pere Urbon-Bayes > > > Software Architect > > > http://www.purbon.com > > > https://twitter.com/purbon > > > https://www.linkedin.com/in/purbon/ > > > > > > > > > -- > > Pere Urbon-Bayes > > Software Architect > > http://www.purbon.com > > https://twitter.com/purbon > > https://www.linkedin.com/in/purbon/ > > > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Pere, It will depend on how quickly the KIP can be voted in, and then implemented. Manikumar Reddy is going to be the release manager for Apache Kafka 2.4.0. Right now, the KIP freeze is scheduled for Sep 25, 2019. A KIP must be accepted by this date in order to be part of 2.4.0. (Being accepted doesn't mean it will be part of 2.4.0, though-- it also has to be implemented.) Hope this helps. cheers, Colin On Fri, Aug 30, 2019, at 07:50, Pere Urbón Bayes wrote: > Hi, > quick question, I saw in another mail that 2.4 release is planned for > September. I think it would be really awesome to have this for this > release, do you think we can make it? > > -- Pere > > Missatge de Pere Urbón Bayes del dia dj., 29 d’ag. > 2019 a les 20:10: > > > Hi, > > this is my first KIP for a change in Apache Kafka, so I'm really need to > > the process. Looking forward to hearing from you and learn the best ropes > > here. > > > > I would like to propose this KIP-515 to enable the ZookeeperClients to > > take full advantage of the TLS communication in the new Zookeeper 3.5.5. > > Specially interesting it the Zookeeper Security Migration, that without > > this change will not work with TLS, disabling users to use ACLs when the > > Zookeeper cluster use TLS. > > > > link: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > > > Looking forward to hearing from you on this, > > > > /cheers > > > > -- > > Pere Urbon-Bayes > > Software Architect > > http://www.purbon.com > > https://twitter.com/purbon > > https://www.linkedin.com/in/purbon/ > > > > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi, quick question, I saw in another mail that 2.4 release is planned for September. I think it would be really awesome to have this for this release, do you think we can make it? -- Pere Missatge de Pere Urbón Bayes del dia dj., 29 d’ag. 2019 a les 20:10: > Hi, > this is my first KIP for a change in Apache Kafka, so I'm really need to > the process. Looking forward to hearing from you and learn the best ropes > here. > > I would like to propose this KIP-515 to enable the ZookeeperClients to > take full advantage of the TLS communication in the new Zookeeper 3.5.5. > Specially interesting it the Zookeeper Security Migration, that without > this change will not work with TLS, disabling users to use ACLs when the > Zookeeper cluster use TLS. > > link: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > Looking forward to hearing from you on this, > > /cheers > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
[DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi, this is my first KIP for a change in Apache Kafka, so I'm really need to the process. Looking forward to hearing from you and learn the best ropes here. I would like to propose this KIP-515 to enable the ZookeeperClients to take full advantage of the TLS communication in the new Zookeeper 3.5.5. Specially interesting it the Zookeeper Security Migration, that without this change will not work with TLS, disabling users to use ACLs when the Zookeeper cluster use TLS. link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication Looking forward to hearing from you on this, /cheers -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
