Yu Yang created KAFKA-7450:
------------------------------
Summary: kafka controller RequestSendThread stuck in infinite loop
after SSL handshake failure with peer brokers
Key: KAFKA-7450
URL: https://issues.apache.org/jira/browse/KAFKA-7450
Project: Kafka
Issue Type: Bug
Components: controller
Affects Versions: 2.0.0
Reporter: Yu Yang
After updating security.inter.broker.protocol to SSL for our cluster, we
observed that the controller can get into almost 100% cpu usage.
{code}
listeners=PLAINTEXT://:9092,SSL://:9093
security.inter.broker.protocol=SSL
{code}
There is no obvious error in server.log. But in controller.log, there is
repetitive SSL handshare failure error as below:
{code}
[2018-09-28 05:53:10,821] WARN [RequestSendThread controllerId=6042] Controller
6042's connection to broker datakafka06176.ec2.pin220.com:9093 (id: 6176 rack:
null) was unsuccessful (kafka.controller.RequestSendThread)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence
violation, 2
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1487)
at
sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at
org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:468)
at
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
at
org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
at
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:125)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at
org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73)
at
kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
at
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence
violation, 2
at
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:196)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
at
org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
at
org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
... 10 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
