[
https://issues.apache.org/jira/browse/ZOOKEEPER-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18036231#comment-18036231
]
Tero Saarni edited comment on ZOOKEEPER-4990 at 11/7/25 1:00 PM:
-----------------------------------------------------------------
Hi,
How does the client configuration for zkCli.sh look like? It seemed to me like
the exception is from client and it would rather indicate that the client was
not configured with the correct CA certificates, or it was not configured
correctly.
I tested and multiple CAs work on both the server and client sides and they can
be configured either as a truststore file or as a PEM bundle:
* Server configured with multiple CAs accepted clients that presented client
certificate issued by two separate root CAs
* Client configured with multiple CAs accepted server that presented server
certificate issued by two separate root CAs
The configuration for client would look like this when using keystore and
truststore
{{export CLIENT_JVMFLAGS="}}
{{-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty}}
{{-Dzookeeper.client.secure=true}}
{{-Dzookeeper.ssl.keyStore.location=/path/server-cert-keystore.jks}}
{{-Dzookeeper.ssl.keyStore.password=my-password}}
{{-Dzookeeper.ssl.trustStore.location=/path/server-ca-truststore.jks}}
{{-Dzookeeper.ssl.trustStore.password=my-password}}
{{"}}
or in case of PEM bundles
{{export CLIENT_JVMFLAGS="}}
{{-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty}}
{{-Dzookeeper.client.secure=true}}
{{-Dzookeeper.ssl.keyStore.location=/path/client-bundle.pem}}
{{-Dzookeeper.ssl.trustStore.location=/path/ca-bundle.pem}}
{{"}}
A PEM bundle is a file that contains multiple PEM entries concatenated
together, separated by the standard PEM markers.
For trusted CAs, concatenate multiple CA certificates:
{{cat cacert1.pem cacert2.pem > ca-bundle.pem}}
For server or client credentials, concatenate the certificate and private key
e.g.:
{{cat client.pem client-key.pem > client-bundle.pem}}
was (Author: JIRAUSER285792):
Hi,
How does the client configuration for zkCli.sh look like? It seemed to me like
the exception is from client and it would rather indicate that the client was
not configured with the correct CA certificates, or it was not configured
correctly.
I tested and multiple CAs wotk on both the server and client sides and they can
be configured either as a truststore file or as a PEM bundle:
* Server configured with multiple CAs accepted clients that presented client
certificate issued by two separate root CAs
* Client configured with multiple CAs accepted server that presented server
certificate issued by two separate root CAs
The configuration for client would look like this when using keystore and
truststore
{{export CLIENT_JVMFLAGS="}}
{{-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty}}
{{-Dzookeeper.client.secure=true}}
{{-Dzookeeper.ssl.keyStore.location=/path/server-cert-keystore.jks}}
{{-Dzookeeper.ssl.keyStore.password=my-password}}
{{-Dzookeeper.ssl.trustStore.location=/path/server-ca-truststore.jks}}
{{-Dzookeeper.ssl.trustStore.password=my-password}}
{{"}}
or in case of PEM bundles
{{export CLIENT_JVMFLAGS="}}
{{-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty}}
{{-Dzookeeper.client.secure=true}}
{{-Dzookeeper.ssl.keyStore.location=/path/client-bundle.pem}}
{{-Dzookeeper.ssl.trustStore.location=/path/ca-bundle.pem}}
{{"}}
A PEM bundle is a file that contains multiple PEM entries concatenated
together, separated by the standard PEM markers.
For trusted CAs, concatenate multiple CA certificates:
{{cat cacert1.pem cacert2.pem > ca-bundle.pem}}
For server or client credentials, concatenate the certificate and private key
e.g.:
{{cat client.pem client-key.pem > client-bundle.pem}}
> Zookeeper doesn't support multiple ca into truststore
> -----------------------------------------------------
>
> Key: ZOOKEEPER-4990
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4990
> Project: ZooKeeper
> Issue Type: Bug
> Components: java client, other
> Affects Versions: 3.8.4
> Environment: Kubernetes environment using jdk 17
> Reporter: Santosh Kumar Sahu
> Priority: Blocker
>
> We have a microservice zookeeper based, For zookeeper configuration, we have
> configured with below parameters
> ssl.keyStore.location=/var/lib/zookeeper/secrets/server/zk-server-keystore.jks
> ssl.quorum.keyStore.password=xxxxxxxxxxxxxxxxx
> ssl.quorum.trustStore.password=xxxxxxxxxxxxxxxxx
> ssl.quorum.keyStore.location=/var/lib/zookeeper/secrets/server/zk-server-keystore.jks
> ssl.quorum.trustStore.location=/var/lib/zookeeper/secrets/server/zk-server-truststore.jks
> ssl.trustStore.password=xxxxxxxxxxxxxxxxx
> ssl.keyStore.password=xxxxxxxxxxxxxxxxx
> Where multiple CA's is being imported into trustStore as alias in jks format,
> so when the client tries to connect with CA signed but it's not working as
> expected.
> -----
> keytool -list -keystore
> /var/lib/zookeeper/secrets/server/zk-server-keystore.jks
> Enter keystore password:
> Keystore type: PKCS12
> Keystore provider: SUN
> Your keystore contains 1 entry
> zookeeper, Nov 6, 2025, PrivateKeyEntry,
> Certificate fingerprint (SHA-256):
> 74:30:24:28:52:09:F5:07:6F:AD:39:97:43:5A:CF:A6:53:AF:44:1C:3B:34:11:5A:B1:86:AD:A4:2F:AC:06:EA
> -------
> bash-4.4$ keytool -list -keystore
> /var/lib/zookeeper/secrets/server/zk-server-truststore.jks
> Enter keystore password:
> Keystore type: PKCS12
> Keystore provider: SUN
> Your keystore contains 3 entries
> zkserverca_cert, Nov 6, 2025, trustedCertEntry,
> Certificate fingerprint (SHA-256):
> 88:82:EA:2C:AD:A5:A9:DB:13:2C:B6:12:89:7A:B5:52:AF:1D:58:96:83:00:C2:7F:95:C0:C6:A1:E6:4F:45:2C
> zkserverca_cert1, Nov 6, 2025, trustedCertEntry,
> Certificate fingerprint (SHA-256):
> 14:95:7E:DA:07:C0:C9:08:01:A3:3D:3C:AF:FD:F8:43:06:E2:CA:D8:DC:1A:20:50:C1:0A:B4:82:5E:45:77:9C
> zkserverca_cert2, Nov 6, 2025, trustedCertEntry,
> Certificate fingerprint (SHA-256):
> 9D:5C:95:F6:ED:5D:67:94:96:A5:91:E4:3D:CB:65:34:DB:32:1B:52:B7:A5:28:F0:B0:A2:87:B0:B3:7E:CD:0B
> ------
> keytool -list -keystore
> /var/lib/zookeeper/secrets/server/zk-client-truststore.jks
> Enter keystore password:
> Keystore type: PKCS12
> Keystore provider: SUN
> Your keystore contains 3 entries
> zkclientca_cert, Nov 6, 2025, trustedCertEntry,
> Certificate fingerprint (SHA-256):
> 11:F6:BB:D1:36:0C:C0:4E:15:C6:3A:A2:9A:DF:DA:50:06:4E:50:01:B2:54:24:57:8E:DE:1D:02:8B:38:28:8D
> zkclientca_cert1, Nov 6, 2025, trustedCertEntry,
> Certificate fingerprint (SHA-256):
> 02:D3:BF:49:6F:38:CC:F7:7D:A7:83:64:A2:EB:5B:4B:40:15:4E:08:8A:25:4E:AC:5E:15:6A:B0:ED:6E:FF:D7
> zkclientca_cert2, Nov 6, 2025, trustedCertEntry,
> Certificate fingerprint (SHA-256):
> A2:B8:5E:FB:CB:B2:C3:59:0C:BA:E0:86:D1:DB:15:2C:5C:03:6E:22:CB:2D:33:03:3E:9E:BE:FE:0E:C1:7E:C4
> ---------
> Does zookeeper support multiple CAs in the truststore?
> Are there any limitations I should be aware off?
> When trying to connect with zkcli.sh with getting the below error
> - Unexpected throwable
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException:
> PKIX path validation failed: java.security.cert.CertPathValidatorException:
> Path does not chain with any of the trust anchors
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed:
> java.security.cert.CertPathValidatorException: Path does not chain with any
> of the trust anchors
> at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
> at
> java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
> at
> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
> at
> java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
> at
> java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
> at
> java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
> at
> java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
> at
> io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1695)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1541)
> at
> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428)
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
> ... 17 common frames omitted
> Caused by: sun.security.validator.ValidatorException: PKIX path validation
> failed: java.security.cert.CertPathValidatorException: Path does not chain
> with any of the trust anchors
> at
> java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369)
> at
> java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275)
> at
> java.base/sun.security.validator.Validator.validate(Validator.java:264)
> at
> java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
> at
> java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
> at
> org.apache.zookeeper.common.ZKTrustManager.checkServerTrusted(ZKTrustManager.java:135)
> at
> java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
> ... 31 common frames omitted
> Caused by: java.security.cert.CertPathValidatorException: Path does not chain
> with any of the trust anchors
> at
> java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157)
> at
> java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83)
> at
> java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309)
> at
> java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
