@Jose >9092 is as well SSL protocol? Yes, it is. As you see in the config snippet from my initial email.
> Zookeeper is connecting over SSL? Yes, at least as far as I can tell. It's set up there too but neither of those are making verifying that easy... >So then I would review all certificates to check if valid. I did that after your first response. >As well there is a Kafka broker property 'advertised.host.name' you could set with same hostname in the certificate. I added this property but didn't change anything, I get the exact same error messages. Thanks On Mon, Oct 28, 2019 at 2:08 PM Jose Manuel Vega Monroy < jose.mon...@williamhill.com> wrote: > @Peter > > 9092 is as well SSL protocol? Zookeeper is connecting over SSL? > > So then I would review all certificates to check if valid. > > As well there is a Kafka broker property 'advertised.host.name' you could > set with same hostname in the certificate. > > Thanks > > <http://www.williamhill.com/> > <http://www.whenthefunstops.co.uk/> > Jose Manuel Vega Monroy > Java Developer / Software Developer Engineer in Test > Direct: +0035 0 2008038 (Ext. 8038) > Email: jose.mon...@williamhill.com > William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA > > > > > On 28/10/2019, 14:03, "Péter Nagykátai" <st4r.f1...@gmail.com> wrote: > > Sorry, if I was unclear before. I'm absolutely new to Kafka and how it > works. > > @Jose > >That happening when clients trying to SSL connect? > There are no clients at the moment just one Kafka broker which spews > the > errors in the server.log. To be specific, there is a ZooKeeper client > which > has no issues: > >> INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient) > > @Manna > >Are you talking about local network loopback? > No, at least I don't think so. I'm simply trying to have the broker in > a > stable running state but after it starts, it tries to connect to the > listed > internal broker, which is itself since at the moment there aren't other > brokers in the cluster. > > >Also, have you tried ssl debug using openssl? What did you observe? > Per Jose's advice I checked the certificates I generated last week and > everything checked out on the 'rootCA' node. > > >How have you setup your signed certificates? > I have a secured node where I generate certificates for every server > in the > cluster (with an intermediate CA). Here are the commands I used: > > `openssl genrsa -out kafka-1.key.pem 2048` > `openssl req -config openssl_intermediate.cnf -key kafka-1.key.pem -new > -sha256 -out kafka-1.csr.pem` > `openssl ca -config openssl_intermediate.cnf -extensions server_cert > -days > 375 -notext -md sha256 -in kafka-1.csr.pem -out kafka-1.cert.pem` > `openssl pkcs12 -export -in kafka-1.cert.pem -inkey kafka-1.key.pem > -out > kafka-1.p12 -name kafka-1` > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12 > -alias kafka-1 -destkeystore kafka-1.jks` > > Also, for the root+intermediate chain: > `keytool -importcert -alias ca-root -keystore truststore.jks -file > ca-chain.cert.pem` > > >Does your CN/SAN matches with your advertised.listeners setup? > Yes. > > >Have you setup hostname verification correctly? > My Kafka configuration file only have the settings I pasted before, the > rest aren't network specific. > > > My (beginner) opinion is that Kafka tries to authenticate itself as a > client and gets confused when getting 'server_hello' message. > ("Unexpected > handshake message: server_hello") > > Thanks! > > On Mon, Oct 28, 2019 at 12:25 PM M. Manna <manme...@gmail.com> wrote: > > > Hi, > > > > not sure what it means "Tries to communicate with itself". Are you > talking > > about local network loopback? > > > > Also, have you tried ssl debug using openssl? What did you observe? > > > > The exception is handshake exception. This is quite common when your > cert > > validation fails. How have you setup your signed certificates? Does > your > > CN/SAN matches with your advertised.listeners setup? Have you setup > > hostname verification correctly? > > > > Thanks, > > > > On Mon, 28 Oct 2019 at 11:11, Péter Nagykátai <st4r.f1...@gmail.com> > > wrote: > > > > > @Jose > > > > > > >It looks like communication problem between brokers. > > > As I mentioned, "I can't get the first broker started". The > message above > > > is from when the broker tries to communicate with "itself": > [Controller > > > id=1001, targetBrokerId=1001]). > > > > > > Nevertheless, I went through the checklist and everything is in > order. > > For > > > the first couple of tries, I got different SSL errors but I could > work > > > those out (that time I messed up the certificates), but now the > problem > > is: > > > >> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected > handshake > > > **message: > > > server_hello* > > > > > > Peter > > > > > > On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy < > > > jose.mon...@williamhill.com> wrote: > > > > > > > @Peter > > > > > > > > It looks like communication problem between brokers. But ensure: > > > > > > > > 1) Crtificates are valid and properly signed by root CA or > intermediate > > > > one in the chain > > > > 2) Clients and brokers having private key and certificate in > their > > > > keystore and properly configured to point to its path > > > > 3) Clients and brokers having CA certificates in the truststore > and > > > > properly configured to point to its path > > > > 4) Clients and brokersbroker having root CA certificate in their > > keystore > > > > and properly configured to.point to its path > > > > 5) Permissions are right ones fro trustore and keystore > > > > > > > > Thanks > > > > > > > > Get Outlook for Android < > https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=LbcoJotfup6doDvIgqyHQISw77V3SVM4PTr_r6Tbn-Y&s=M2vFRWGBcJ0-qtIHW1BhAMr8yANCnvghejnBHBuTeJA&e= > > > > > > > > > > ------------------------------ > > > > *From:* Péter Nagykátai <st4r.f1...@gmail.com> > > > > *Sent:* Monday, 28 October 2019, 00:13 > > > > *To:* users@kafka.apache.org > > > > *Subject:* [EXTERNAL] SSL setup failing > > > > > > > > Hi! > > > > > > > > I'm experimenting with setting up a log ingesting cluster and > Kafka > > would > > > > be part of it. Unfortunately, I can't get the first broker > started. I > > > need > > > > to secure the communication between a dozen nodes and Kaquiafka > would > > > only > > > > be > > > > one part of it. I have a secured node where I generate > certificates for > > > > every server in the cluster (with an intermediate CA). AFAIK, I > need to > > > use > > > > '.jks' files for Kafka, so I've generated a '.p12' file from the > > openssl > > > > certificate and key then used `keytool` to generate a keystore: > > > > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype > PKCS12 > > > > -alias kafka-1 -destkeystore kafka-1.jks` > > > > I generated a truststore for the root and intermediate chain as > well: > > > > `keytool -importcert -alias ca-root -keystore truststore.jks > -file > > > > ca-chain.cert.pem > > > > > > > > Relevant part of the 'server.properties' configuration: > > > > #### > > > > listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093 > > > > > advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093 > > > > inter.broker.listener.name=INTERNAL > > > > listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL > > > > security.protocol=SSL > > > > ssl.client.auth=required > > > > ssl.truststore.location=/*******/truststore.jks > > > > ssl.truststore.password=************* > > > > ssl.keystore.location=/*******/kafka-1.jks > > > > ssl.keystore.password=************* > > > > #### > > > > > > > > After starting Kafka (as a service) I get the the following in > the > > > > 'server.log': > > > > >>... > > > > >> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer) > > > > >> INFO [SocketServer brokerId=1001] Failed authentication with > > > > /XXX.XXX.XXX.XXX (SSL handshake failed) > > > > (org.apache.kafka.common.network.Selector) > > > > >> INFO [Controller id=1001, targetBrokerId=1001] Failed > authentication > > > > with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed) > > > > (org.apache.kafka.common.network.Selector) > > > > >> ERROR [Controller id=1001, targetBrokerId=1001] Connection to > node > > > 1001 > > > > (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL > > > handshake > > > > failed (org.apache.kafka.clients.NetworkClient) > > > > >>... > > > > >> WARN SSL handshake failed (kafka.utils.CoreUtils$) > > > > >> org.apache.kafka.common.errors.SslAuthenticationException: SSL > > > handshake > > > > failed > > > > >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected > handshake > > > > message: server_hello > > > > >>... > > > > > > > > I couldn't find any lead with that error message and got stuck. > Any > > ideas > > > > what that error message means and how to solve it? > > > > > > > > Specs: > > > > - Ubuntu 18.04.3 LTS > > > > - OpenJDK Runtime Environment (build > > > 11.0.4+11-post-Ubuntu-1ubuntu218.04.3) > > > > - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz) > > > > - OpenSSL 1.1.1 > > > > > > > > Thank you! > > > > Peter > > > > > > > > Confidentiality: The contents of this e-mail and any attachments > > > > transmitted with it are intended to be confidential to the > intended > > > > recipient; and may be privileged or otherwise protected from > > disclosure. > > > If > > > > you are not an intended recipient of this e-mail, do not > duplicate or > > > > redistribute it by any means. Please delete it and any > attachments and > > > > notify the sender that you have received it in error. This > e-mail is > > sent > > > > by a William Hill PLC group company. The William Hill group > companies > > > > include, among others, William Hill PLC (registered number > 4212563), > > > > William Hill Organization Limited (registered number 278208), > William > > > Hill > > > > US HoldCo Inc, WHG (International) Limited (registered number > 99191) > > and > > > Mr > > > > Green Limited (registered number C43260). Each of William Hill > PLC and > > > > William Hill Organization Limited is registered in England and > Wales > > and > > > > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, > UK. > > > > William Hill U.S. HoldCo, Inc. is registered in Delaware and has > its > > > > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, > New > > > Castle > > > > County DE 19801 Delaware, United States of America. WHG > (International) > > > > Limited is registered in Gibraltar and has its registered office > at 6/1 > > > > Waterport Place, Gibraltar. Mr Green Limited is registered in > Malta and > > > has > > > > its registered office at Tagliaferro Business Centre, Level 7, > 14 High > > > > Street, Sliema SLM 1549, Malta. Unless specifically indicated > > otherwise, > > > > the contents of this e-mail are subject to contract; and are not > an > > > > official statement, and do not necessarily represent the views, > of > > > William > > > > Hill PLC, its subsidiaries or affiliated companies. Please note > that > > > > neither William Hill PLC, nor its subsidiaries and affiliated > companies > > > can > > > > accept any responsibility for any viruses contained within this > e-mail > > > and > > > > it is your responsibility to scan any emails and their > attachments. > > > William > > > > Hill PLC, its subsidiaries and affiliated companies may monitor > e-mail > > > > traffic data and also the content of e-mails for effective > operation of > > > the > > > > e-mail system, or for security, purposes. > > > > > > > > > > > > Confidentiality: The contents of this e-mail and any attachments > transmitted with it are intended to be confidential to the intended > recipient; and may be privileged or otherwise protected from disclosure. If > you are not an intended recipient of this e-mail, do not duplicate or > redistribute it by any means. Please delete it and any attachments and > notify the sender that you have received it in error. This e-mail is sent > by a William Hill PLC group company. The William Hill group companies > include, among others, William Hill PLC (registered number 4212563), > William Hill Organization Limited (registered number 278208), William Hill > US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr > Green Limited (registered number C43260). Each of William Hill PLC and > William Hill Organization Limited is registered in England and Wales and > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle > County DE 19801 Delaware, United States of America. WHG (International) > Limited is registered in Gibraltar and has its registered office at 6/1 > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has > its registered office at Tagliaferro Business Centre, Level 7, 14 High > Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, > the contents of this e-mail are subject to contract; and are not an > official statement, and do not necessarily represent the views, of William > Hill PLC, its subsidiaries or affiliated companies. Please note that > neither William Hill PLC, nor its subsidiaries and affiliated companies can > accept any responsibility for any viruses contained within this e-mail and > it is your responsibility to scan any emails and their attachments. William > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail > traffic data and also the content of e-mails for effective operation of the > e-mail system, or for security, purposes. >
