And the solution was:
Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the output is:
11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker
id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1 topic(s)
Set(inputTopic)
11893 [main] INFO kafka.producer.SyncProducer - Connected to
192.168.59.103:9092 for producing
12045 [main] INFO kafka.producer.SyncProducer - Disconnecting from
192.168.59.103:9092
12062 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching
metadata [{TopicMetadata for topic inputTopic ->
No partition metadata for topic inputTopic due to
kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
kafka.common.LeaderNotAvailableException
12066 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker
id:0,host:192.168.59.103,port:9092 with correlation id 1 for 1 topic(s)
Set(inputTopic)
12067 [main] INFO kafka.producer.SyncProducer - Connected to
192.168.59.103:9092 for producing
12097 [main] INFO kafka.producer.SyncProducer - Disconnecting from
192.168.59.103:9092
12098 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching
metadata [{TopicMetadata for topic inputTopic ->
No partition metadata for topic inputTopic due to
kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
kafka.common.LeaderNotAvailableException
12098 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to collate
messages by topic, partition due to: Failed to fetch topic metadata for topic:
inputTopic
12099 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for 1000
ms before retrying send. Remaining retries = 3
13104 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker
id:0,host:192.168.59.103,port:9092 with correlation id 2 for 1 topic(s)
Set(inputTopic)
13111 [main] INFO kafka.producer.SyncProducer - Connected to
192.168.59.103:9092 for producing
13137 [main] INFO kafka.producer.SyncProducer - Disconnecting from
192.168.59.103:9092
13161 [main] INFO kafka.producer.SyncProducer - Connected to
192.168.59.103:9092 for producing
In case this would be not enough for you, you can try to change the values of
- message.send.max.retries from the default 5 to e.g. 10 and
- topic.metadata.refresh.interval.ms to 0
It is still unclear, why it takes 2 tries i.e. 2 seconds to fetch the topic
metadata…
Cheers,
Andras
> On 26 Sep 2014, at 14:43, Andras Hatvani <[email protected]>
> wrote:
>
> Hi,
>
> What does the exception mean in the following context?
>
> I've got two Kafka brokers out which the second one will be elected as leader:
>
> kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2], started
> (kafka.server.KafkaServer)
> kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
>
> Then, when I try to produce a message, I only get the following error:
>
> 23182 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for
> 100 ms before retrying send. Remaining retries = 0
> 23286 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker
> id:0,host:192.168.59.103,port:9092 with correlation id 8 for 1 topic(s)
> Set(inputTopic)
> 23287 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> 23300 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> 23301 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching
> metadata [{TopicMetadata for topic inputTopic ->
> No partition metadata for topic inputTopic due to
> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
> kafka.common.LeaderNotAvailableException
> 23303 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to send
> requests for topics inputTopic with correlation ids in [0,8]
>
> The property metadata.broker.list contains the host:port values of both
> brokers.
>
> What can be wrong/missing?
> Thanks,
> Andras
