Hello *
In order to get familiar with Zookeeper / Kafka I'm in the process of
creating a docker-compose based setup. The aim is that all communications
are secured using mTLS. Thereby I run into the following issue with the
host name verification for the zookeeper quorum.
In short, I currently plan to have 3 zookeeper nodes. The nodes have the
hostnames zookeeper, zookeeper2 and zookeeper3. The core issue is, that
while it is possible to ping the container using these names, the reverse
DNS lookup differs:
E.g.:
$ ping zookeeper2
PING zookeeper2 (172.21.0.4) 56(84) bytes of data.
64 bytes from kafka_ssl-zookeeper2-1.kafka_ssl_default (172.21.0.4):
icmp_seq=1 ttl=64 time=0.069 ms
$ dig -x 172.21.0.4
; <<>> DiG 9.11.5-P4-5.1+deb10u7-Debian <<>> -x 172.21.0.4
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63658
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;4.0.21.172.in-addr.arpa. IN PTR
;; ANSWER SECTION:
4.0.21.172.in-addr.arpa. 600 IN PTR
kafka_ssl-zookeeper2-1.kafka_ssl_default.
;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Wed Apr 06 18:53:41 UTC 2022
;; MSG SIZE rcvd: 118
kafka_ssl-zookeeper2-1.kafka_ssl_default seems to represent <docker-compose
project name>-<container-name>-<id>.<docker-network-name>
When I use zookeeper, zookeeper2 or zookeeper3 as the common name in the
certificates, the zookeeper reports the following errors:
javax.net.ssl.SSLPeerUnverifiedException: Certificate for
<kafka_ssl-zookeeper2-1.kafka_ssl_default> doesn't match common name of the
certificate subject: zookeeper2
java.security.cert.CertificateException: Failed to verify both host address
and host name
When disabling host name verification using ssl.quorum.hostnameVerification
the setup seems to work however.
When using kafka_ssl-zookeeper2-1.kafka_ssl_default as common names in the
certificates, these names can not be resolved: java.net.SocketException:
Unresolved address
A fully working example to reproducing this issue (including the full logs,
script to generate the certificates etc.) can be found here:
https://github.com/rbuffat/kafka_playground/tree/main/kafka_ssl for the
zookeeper* as common name case and
https://github.com/rbuffat/kafka_playground/tree/main/kafka_ssl_v2 for the
case when using kafka_ssl-zookeeper2-1.kafka_ssl_defaultas the common name
case.
Any insights into what to do to resolve this issue is much appreciated.
lg rene
