Hi there,
Iam trying to use parameter implemented in
https://github.com/apache/zookeeper/pull/680
I have vault+vault agent installed on each zookeeper node, certs a generated
and packed to keystore/trustore.
TTL of certificate is 3 mins, and keystores are recreated each 3 mins with new
certs/keys.
Initially clusters starts good, I can zkCli into it securely, but if I will try
to connect to secure port of zookeeper server after 3 mins it fails with
-certificate expired exception.
So for me it looks like the certificates are not reloaded properly from
keystore.
Also I see exceptions in zookeeper.out log file:
2021-05-26 12:54:30,738 [myid:1] - ERROR
[FileChangeWatcher:FileChangeWatcher$WatcherThread@223] - Error from callback
java.lang.RuntimeException:
org.apache.zookeeper.common.X509Exception$SSLContextException: Failed to create
KeyManager
at
org.apache.zookeeper.common.X509Util.handleWatchEvent(X509Util.java:638)
at
org.apache.zookeeper.common.X509Util.lambda$newFileChangeWatcher$0(X509Util.java:562)
at
org.apache.zookeeper.common.FileChangeWatcher$WatcherThread.runLoop(FileChangeWatcher.java:221)
at
org.apache.zookeeper.common.FileChangeWatcher$WatcherThread.run(FileChangeWatcher.java:194)
Caused by: org.apache.zookeeper.common.X509Exception$SSLContextException:
Failed to create KeyManager
at
org.apache.zookeeper.common.X509Util.createSSLContextAndOptionsFromConfig(X509Util.java:350)
at
org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:328)
at
org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:282)
at
org.apache.zookeeper.common.X509Util.resetDefaultSSLContextAndOptions(X509Util.java:272)
at
org.apache.zookeeper.common.X509Util.handleWatchEvent(X509Util.java:636)
... 3 more
Caused by: org.apache.zookeeper.common.X509Exception$KeyManagerException:
java.io.EOFException
at
org.apache.zookeeper.common.X509Util.createKeyManager(X509Util.java:447)
at
org.apache.zookeeper.common.X509Util.createSSLContextAndOptionsFromConfig(X509Util.java:348)
... 7 more
Caused by: java.io.EOFException
at java.base/java.io.DataInputStream.readInt(DataInputStream.java:397)
at
java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:662)
at
java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
at java.base/java.security.KeyStore.load(KeyStore.java:1479)
at
org.apache.zookeeper.common.StandardTypeFileKeyStoreLoader.loadKeyStore(StandardTypeFileKeyStoreLoader.java:55)
at org.apache.zookeeper.common.X509Util.loadKeyStore(X509Util.java:400)
at
org.apache.zookeeper.common.X509Util.createKeyManager(X509Util.java:436)
... 8 more
Here is my zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/zookeeper/data
clientPort=2181
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpPort=7000
metricsProvider.exportJvmInfo=true
server.1=1.1.1.1:2888:3888
server.2=1.1.1.2:2888:3888
server.3=1.1.1.3:2888:3888
secureClientPort=2281
sslQuorum=true
sslQuorumReloadCertFiles=true
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.quorum.keyStore.location=/opt/vault/agent/zookeeper/zk_keystore.jks
ssl.quorum.keyStore.password=test
ssl.quorum.trustStore.location=/opt/vault/agent/zookeeper/zk_trustore.jks
ssl.quorum.trustStore.password=test
ssl.keyStore.location=/opt/vault/agent/zookeeper/zk_keystore.jks
ssl.keyStore.password=test
ssl.trustStore.location=/opt/vault/agent/zookeeper/zk_trustore.jks
ssl.trustStore.password=test
