@Peter 9092 is as well SSL protocol? Zookeeper is connecting over SSL?
So then I would review all certificates to check if valid. As well there is a Kafka broker property 'advertised.host.name' you could set with same hostname in the certificate. Thanks <http://www.williamhill.com/> <http://www.whenthefunstops.co.uk/> Jose Manuel Vega Monroy Java Developer / Software Developer Engineer in Test Direct: +0035 0 2008038 (Ext. 8038) Email: jose.mon...@williamhill.com William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA On 28/10/2019, 14:03, "Péter Nagykátai" <st4r.f1...@gmail.com> wrote: Sorry, if I was unclear before. I'm absolutely new to Kafka and how it works. @Jose >That happening when clients trying to SSL connect? There are no clients at the moment just one Kafka broker which spews the errors in the server.log. To be specific, there is a ZooKeeper client which has no issues: >> INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient) @Manna >Are you talking about local network loopback? No, at least I don't think so. I'm simply trying to have the broker in a stable running state but after it starts, it tries to connect to the listed internal broker, which is itself since at the moment there aren't other brokers in the cluster. >Also, have you tried ssl debug using openssl? What did you observe? Per Jose's advice I checked the certificates I generated last week and everything checked out on the 'rootCA' node. >How have you setup your signed certificates? I have a secured node where I generate certificates for every server in the cluster (with an intermediate CA). Here are the commands I used: `openssl genrsa -out kafka-1.key.pem 2048` `openssl req -config openssl_intermediate.cnf -key kafka-1.key.pem -new -sha256 -out kafka-1.csr.pem` `openssl ca -config openssl_intermediate.cnf -extensions server_cert -days 375 -notext -md sha256 -in kafka-1.csr.pem -out kafka-1.cert.pem` `openssl pkcs12 -export -in kafka-1.cert.pem -inkey kafka-1.key.pem -out kafka-1.p12 -name kafka-1` `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12 -alias kafka-1 -destkeystore kafka-1.jks` Also, for the root+intermediate chain: `keytool -importcert -alias ca-root -keystore truststore.jks -file ca-chain.cert.pem` >Does your CN/SAN matches with your advertised.listeners setup? Yes. >Have you setup hostname verification correctly? My Kafka configuration file only have the settings I pasted before, the rest aren't network specific. My (beginner) opinion is that Kafka tries to authenticate itself as a client and gets confused when getting 'server_hello' message. ("Unexpected handshake message: server_hello") Thanks! On Mon, Oct 28, 2019 at 12:25 PM M. Manna <manme...@gmail.com> wrote: > Hi, > > not sure what it means "Tries to communicate with itself". Are you talking > about local network loopback? > > Also, have you tried ssl debug using openssl? What did you observe? > > The exception is handshake exception. This is quite common when your cert > validation fails. How have you setup your signed certificates? Does your > CN/SAN matches with your advertised.listeners setup? Have you setup > hostname verification correctly? > > Thanks, > > On Mon, 28 Oct 2019 at 11:11, Péter Nagykátai <st4r.f1...@gmail.com> > wrote: > > > @Jose > > > > >It looks like communication problem between brokers. > > As I mentioned, "I can't get the first broker started". The message above > > is from when the broker tries to communicate with "itself": [Controller > > id=1001, targetBrokerId=1001]). > > > > Nevertheless, I went through the checklist and everything is in order. > For > > the first couple of tries, I got different SSL errors but I could work > > those out (that time I messed up the certificates), but now the problem > is: > > >> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected handshake > > **message: > > server_hello* > > > > Peter > > > > On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy < > > jose.mon...@williamhill.com> wrote: > > > > > @Peter > > > > > > It looks like communication problem between brokers. But ensure: > > > > > > 1) Crtificates are valid and properly signed by root CA or intermediate > > > one in the chain > > > 2) Clients and brokers having private key and certificate in their > > > keystore and properly configured to point to its path > > > 3) Clients and brokers having CA certificates in the truststore and > > > properly configured to point to its path > > > 4) Clients and brokersbroker having root CA certificate in their > keystore > > > and properly configured to.point to its path > > > 5) Permissions are right ones fro trustore and keystore > > > > > > Thanks > > > > > > Get Outlook for Android <https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=LbcoJotfup6doDvIgqyHQISw77V3SVM4PTr_r6Tbn-Y&s=M2vFRWGBcJ0-qtIHW1BhAMr8yANCnvghejnBHBuTeJA&e= > > > > > > > ------------------------------ > > > *From:* Péter Nagykátai <st4r.f1...@gmail.com> > > > *Sent:* Monday, 28 October 2019, 00:13 > > > *To:* users@kafka.apache.org > > > *Subject:* [EXTERNAL] SSL setup failing > > > > > > Hi! > > > > > > I'm experimenting with setting up a log ingesting cluster and Kafka > would > > > be part of it. Unfortunately, I can't get the first broker started. I > > need > > > to secure the communication between a dozen nodes and Kaquiafka would > > only > > > be > > > one part of it. I have a secured node where I generate certificates for > > > every server in the cluster (with an intermediate CA). AFAIK, I need to > > use > > > '.jks' files for Kafka, so I've generated a '.p12' file from the > openssl > > > certificate and key then used `keytool` to generate a keystore: > > > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12 > > > -alias kafka-1 -destkeystore kafka-1.jks` > > > I generated a truststore for the root and intermediate chain as well: > > > `keytool -importcert -alias ca-root -keystore truststore.jks -file > > > ca-chain.cert.pem > > > > > > Relevant part of the 'server.properties' configuration: > > > #### > > > listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093 > > > advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093 > > > inter.broker.listener.name=INTERNAL > > > listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL > > > security.protocol=SSL > > > ssl.client.auth=required > > > ssl.truststore.location=/*******/truststore.jks > > > ssl.truststore.password=************* > > > ssl.keystore.location=/*******/kafka-1.jks > > > ssl.keystore.password=************* > > > #### > > > > > > After starting Kafka (as a service) I get the the following in the > > > 'server.log': > > > >>... > > > >> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer) > > > >> INFO [SocketServer brokerId=1001] Failed authentication with > > > /XXX.XXX.XXX.XXX (SSL handshake failed) > > > (org.apache.kafka.common.network.Selector) > > > >> INFO [Controller id=1001, targetBrokerId=1001] Failed authentication > > > with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed) > > > (org.apache.kafka.common.network.Selector) > > > >> ERROR [Controller id=1001, targetBrokerId=1001] Connection to node > > 1001 > > > (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL > > handshake > > > failed (org.apache.kafka.clients.NetworkClient) > > > >>... > > > >> WARN SSL handshake failed (kafka.utils.CoreUtils$) > > > >> org.apache.kafka.common.errors.SslAuthenticationException: SSL > > handshake > > > failed > > > >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake > > > message: server_hello > > > >>... > > > > > > I couldn't find any lead with that error message and got stuck. Any > ideas > > > what that error message means and how to solve it? > > > > > > Specs: > > > - Ubuntu 18.04.3 LTS > > > - OpenJDK Runtime Environment (build > > 11.0.4+11-post-Ubuntu-1ubuntu218.04.3) > > > - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz) > > > - OpenSSL 1.1.1 > > > > > > Thank you! > > > Peter > > > > > > Confidentiality: The contents of this e-mail and any attachments > > > transmitted with it are intended to be confidential to the intended > > > recipient; and may be privileged or otherwise protected from > disclosure. > > If > > > you are not an intended recipient of this e-mail, do not duplicate or > > > redistribute it by any means. Please delete it and any attachments and > > > notify the sender that you have received it in error. This e-mail is > sent > > > by a William Hill PLC group company. The William Hill group companies > > > include, among others, William Hill PLC (registered number 4212563), > > > William Hill Organization Limited (registered number 278208), William > > Hill > > > US HoldCo Inc, WHG (International) Limited (registered number 99191) > and > > Mr > > > Green Limited (registered number C43260). Each of William Hill PLC and > > > William Hill Organization Limited is registered in England and Wales > and > > > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. > > > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its > > > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New > > Castle > > > County DE 19801 Delaware, United States of America. WHG (International) > > > Limited is registered in Gibraltar and has its registered office at 6/1 > > > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and > > has > > > its registered office at Tagliaferro Business Centre, Level 7, 14 High > > > Street, Sliema SLM 1549, Malta. Unless specifically indicated > otherwise, > > > the contents of this e-mail are subject to contract; and are not an > > > official statement, and do not necessarily represent the views, of > > William > > > Hill PLC, its subsidiaries or affiliated companies. Please note that > > > neither William Hill PLC, nor its subsidiaries and affiliated companies > > can > > > accept any responsibility for any viruses contained within this e-mail > > and > > > it is your responsibility to scan any emails and their attachments. > > William > > > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail > > > traffic data and also the content of e-mails for effective operation of > > the > > > e-mail system, or for security, purposes. > > > > > > Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr Green Limited (registered number C43260). Each of William Hill PLC and William Hill Organization Limited is registered in England and Wales and has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 Delaware, United States of America. WHG (International) Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has its registered office at Tagliaferro Business Centre, Level 7, 14 High Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes.
