Marc Richter created ZOOKEEPER-4403:
---------------------------------------
Summary: Quorum TLS certificate validation uses wrong name
Key: ZOOKEEPER-4403
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4403
Project: ZooKeeper
Issue Type: Bug
Components: quorum, security
Affects Versions: 3.5.9
Reporter: Marc Richter
When using SSL certificates for public IPv6 DNS endpoints as received from some
public Service like "Let's Encrypt" for Quorum Encryption, Zookeeper validates
the SNAs of that certificate for the IP Address instead of the DNS name, as
configured.
As a Result, these certificates can't be used, since no certificates for IPv6
IPs issued.
This has been observed with Zookeeper Version 3.5.9, which is the one bundled
in the most recent release of Kafka (2.8.1).
In the affected environment, there is a 3-node-Zookeeper-Cluster, which is
configured as this in {{zookeeper.properties}} (mind the DNS name!):
{code}
server.1=zookeeper1.ourdomain.cloud:2888:3888
server.2=zookeeper2.ourdomain.cloud:2888:3888
server.3=zookeeper3.ourdomain.cloud:2888:3888
{code}
All these records do have a public IPv6 entry only.
The SSL certificates from Let's Encrypt are requested and added to the
Quorum-Keystores like this:
# Using https://github.com/acmesh-official/acme.sh
# Requesting the cert from Let's Encrypt using:
{code}./acme.sh --issue --dns dns_nsupdate -d
zookeeper1.ourdomain.cloud{code}
for each system.
# Merge fullchain- and certificate-file to a single PKCS12 file using:
{code}openssl pkcs12 -export -in <certfile> -inkey <keyfile> -out
<pkcs12_file> -name zookeeper1.ourdomain.cloud \
-CAfile <fullchainfile> -password <JKS_password>{code}
# Adding the resulting PKCS12 file to the Quorum Keystore:
{code}keytool -importkeystore -deststorepass <JKS_password> -destkeypass
<JKS_password> -deststoretype pkcs12 \
-srckeystore <pkcs12_file> -srcstoretype PKCS12 -srcstorepass
<JKS_password> -destkeystore <quorum_jks> \
-alias zookeeper1.ourdomain.cloud{code}
When any of the systems tries to initiate the quorum-connect, their logs state
that the remote's Certificates could not be verified, since the SNA-List does
not contain the IPv6 address.
For example: This is the log from {{zookeeper2.ourdomain.cloud}} when
connecting {{zookeeper3.ourdomain.cloud}}:
{code}
[2021-10-13 15:13:49,960] INFO Received connection request from
/2a01:--CUT--:750:47566 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-10-13 15:13:50,094] ERROR Failed to verify host address: 2a01:--CUT--:750
(org.apache.zookeeper.common.ZKTrustManager)
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <2a01:--CUT--:750>
doesn't match any of the subject alternative names: [zookeeper3.ourdomain.cloud]
{code}
I think the log lines cited clearly show:
# Zookeeper is picking up the correct certificate from the quorum Keystore,
since it states that the request does not match any SNA and lists
{{zookeeper3.ourdomain.cloud}} only, which it can only know from the
certificate itself.
# Zookeeper is validating the wrong thing here: Even though the config clearly
states to use a DNS name, the certificates SNAs alre validated against the
IPv6 address that record belongs to instead of the DNS name configured ({{ERROR
Failed to verify host address: 2a01:--CUT--:750}})
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
