Dharani created ZOOKEEPER-4887:
----------------------------------
Summary: Zookeeper quorum formation fails when TLS is enabled in
k8s env
Key: ZOOKEEPER-4887
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4887
Project: ZooKeeper
Issue Type: Bug
Affects Versions: 3.8.3
Reporter: Dharani
We have three(3) node zookeeper cluster running as a pod on Kubernetes cluster,
zookeeper quorum formation fails with TLS handshake error, as the server name
in the https request does not match with any of the SANs in the certificate
configured for zookeeper server. Server name in the request is of the form
"x-x-x-x.kubernetes.default.svc.cluster.local" (where x-x-x-x is the IP address
of the POD), and I am unable to understand the reason behind pre-pending FQDN
with a IP address.
Please find below the extract of the error logs from the zookeeper POD
{code:java}
[myid:] - ERROR
[LearnerHandler-/192.168.220.10:46516:o.a.z.c.ZKTrustManager@191] - Failed to
verify host address: 192.168.220.10
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <192.168.220.10>
doesn't match any of the subject alternative names: [eric-data-coordinator-zk,
eric-data-coordinator-zk.zdhagxx1, eric-data-coordinator-zk.zdhagxx1.svc,
eric-data-coordinator-zk.zdhagxx1.svc.cluster.local,
*.eric-data-coordinator-zk-ensemble-service.zdhagxx1.svc.cluster.local,
certified-scrape-target]
org.apache.zookeeper.common.ZKHostnameVerifier.matchIPAddress(ZKHostnameVerifier.java:197)
org.apache.zookeeper.common.ZKHostnameVerifier.verify(ZKHostnameVerifier.java:165)
org.apache.zookeeper.common.ZKTrustManager.performHostVerification(ZKTrustManager.java:180)
org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:93)
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1285)
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1204)
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181)
java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1511)
java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:926)
java.base/sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:372)
org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:269)
org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.getSocket(UnifiedServerSocket.java:298)
org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.access$400(UnifiedServerSocket.java:172)
org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699)
org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:271)
java.base/java.io.DataInputStream.readInt(DataInputStream.java:392)
org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:96)
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:134)
org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:472)[myid:]
- ERROR [LearnerHandler-/192.168.220.10:46516:o.a.z.c.ZKTrustManager@192] -
Failed to verify hostname:
192-168-220-10.eric-data-coordinator-zk.zdhagxx1.svc.cluster.local
javax.net.ssl.SSLPeerUnverifiedException: Certificate for
<192-168-220-10.eric-data-coordinator-zk.zdhagxx1.svc.cluster.local> doesn't
match any of the subject alternative names: [eric-data-coordinator-zk,
eric-data-coordinator-zk.zdhagxx1, eric-data-coordinator-zk.zdhagxx1.svc,
eric-data-coordinator-zk.zdhagxx1.svc.cluster.local,
*.eric-data-coordinator-zk-ensemble-service.zdhagxx1.svc.cluster.local,
certified-scrape-target]
org.apache.zookeeper.common.ZKHostnameVerifier.matchDNSName(ZKHostnameVerifier.java:230)
org.apache.zookeeper.common.ZKHostnameVerifier.verify(ZKHostnameVerifier.java:171)
org.apache.zookeeper.common.ZKTrustManager.performHostVerification(ZKTrustManager.java:189)
org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:93)
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1285)
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1204)
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181)
java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1511)
java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:926)
java.base/sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:372)
org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:269)
org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.getSocket(UnifiedServerSocket.java:298)
org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.access$400(UnifiedServerSocket.java:172)
org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699)
org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:271)
java.base/java.io.DataInputStream.readInt(DataInputStream.java:392)
org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:96)
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:134)
org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:472)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
