little brother ma created KAFKA-7238: ----------------------------------------
Summary: Producer can't update Metadata when two brokers break down and restart the first one Key: KAFKA-7238 URL: https://issues.apache.org/jira/browse/KAFKA-7238 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.11.0.3, 0.11.0.1 Environment: os: 14.04.1-Ubuntu java: 1.8.0_151-b12 Reporter: little brother ma There is a kafka cluster with two brokers( broker ids are 70 and 67),I create a topic named topic2 with 2 partitions and 1 replica, and partition 0 is on the broker 70,and partition 1 is on the broker 67. While sending datas, I stop broker 70 first, and wait 5 ms, stop broker 67 . And then ,I restart broker 70, the producer client can't update metadata and don't switch to connet broker 70,otherwise, it always try to connect broker 67. The log before stop the broker: ================send The message size: 12, key: null, partition:0, offset:29 [2018-08-02 19:59:10,180] DEBUG Sending metadata request (type=MetadataRequest, topics=topic2) to node 10.93.132.67:9092 (id: 67 rack: null) (org.apache.kafka.clients.NetworkClient) [2018-08-02 19:59:10,184] DEBUG Updated cluster metadata version 11 to Cluster(id = 3fL1MXQtRo6Ujmxa9dS3eA, nodes = [10.93.132.70:9092 (id: 70 rack: null), 10.93.132.67:9092 (id: 67 rack: null)], partitions = [Partition(topic = topic2, partition = 1, leader = 70, replicas = [70], isr = [70]), Partition(topic = topic2, partition = 0, leader = 67, replicas = [67], isr = [67])]) (org.apache.kafka.clients.Metadata) ================send The message size: 12, key: null, partition:1, offset:69 ================send The message size: 12, key: null, partition:0, offset:30 ================send The message size: 12, key: null, partition:1, offset:70 ================send The message size: 12, key: null, partition:0, offset:31 ================send The message size: 12, key: null, partition:1, offset:71 ================send The message size: 12, key: null, partition:0, offset:32 The log after stop the broker 70 : [2018-08-02 20:02:50,049] DEBUG Node 70 disconnected. (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:50,049] WARN Connection to node 70 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:50,049] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:50,100] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:50,151] DEBUG Initialize connection to node 10.93.132.70:9092 (id: 70 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:50,151] DEBUG Initiating connection to node 10.93.132.70:9092 (id: 70 rack: null) (org.apache.kafka.clients.NetworkClient) ================send The message size: 12, key: null, partition:0, offset:95 [2018-08-02 20:02:51,156] DEBUG Connection with /10.93.132.70 disconnected (org.apache.kafka.common.network.Selector) java.net.ConnectException: Connection refused: no further information at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50) at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:361) at org.apache.kafka.common.network.Selector.poll(Selector.java:326) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:954) at java.lang.Thread.run(Thread.java:745) [2018-08-02 20:02:51,157] DEBUG Node 70 disconnected. (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:51,157] WARN Connection to node 70 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:51,157] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:51,208] DEBUG Initialize connection to node 10.93.132.70:9092 (id: 70 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:51,208] DEBUG Initiating connection to node 10.93.132.70:9092 (id: 70 rack: null) (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:52,216] DEBUG Connection with /10.93.132.70 disconnected (org.apache.kafka.common.network.Selector) java.net.ConnectException: Connection refused: no further information at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50) at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:361) at org.apache.kafka.common.network.Selector.poll(Selector.java:326) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:954) at java.lang.Thread.run(Thread.java:745) [2018-08-02 20:02:52,216] DEBUG Node 70 disconnected. (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:52,216] WARN Connection to node 70 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:52,216] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:52,267] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:52,318] DEBUG Initialize connection to node 10.93.132.70:9092 (id: 70 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:02:52,318] DEBUG Initiating connection to node 10.93.132.70:9092 (id: 70 rack: null) (org.apache.kafka.clients.NetworkClient) ================send The message size: 12, key: null, partition:0, offset:96 The broker 67 is working! The log after stop the broker 70 ,broker 67 ,and restart broker 70 : [2018-08-02 20:05:15,136] DEBUG Node 67 disconnected. (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:05:15,136] WARN Connection to node 67 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:05:15,137] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:05:15,177] DEBUG Initiating connection to node 10.93.132.67:9092 (id: 67 rack: null) (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:05:16,183] DEBUG Connection with /10.93.132.67 disconnected (org.apache.kafka.common.network.Selector) java.net.ConnectException: Connection refused: no further information at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50) at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:361) at org.apache.kafka.common.network.Selector.poll(Selector.java:326) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162) at java.lang.Thread.run(Thread.java:745) [2018-08-02 20:05:16,183] DEBUG Node 67 disconnected. (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:05:16,183] WARN Connection to node 67 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:05:16,183] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:05:16,233] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:05:16,242] DEBUG Initiating connection to node 10.93.132.67:9092 (id: 67 rack: null) (org.apache.kafka.clients.NetworkClient) [2018-08-02 20:05:17,247] DEBUG Connection with /10.93.132.67 disconnected (org.apache.kafka.common.network.Selector) java.net.ConnectException: Connection refused: no further information at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50) at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:361) at org.apache.kafka.common.network.Selector.poll(Selector.java:326) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005)