> > kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2],
> > started (kafka.server.KafkaServer)
> > kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2
> > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
The above logs are for controller election. Not leader election for
the topic you are producing to.
When you producer starts it will issue a metadata request and it
should auto-create the topic (provided auto-create is on which is
default). The first metadata request for a non-existent topic always
returns LeaderNotAvailable because the controller then has to elect a
leader for the new topic.
Joel
On Fri, Sep 26, 2014 at 04:07:58PM +0200, Andras Hatvani wrote:
> And the solution was:
> Increase retry.backoff.ms from the default 100 ms to 1000 ms, so the output
> is:
>
> 11891 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker
> id:0,host:192.168.59.103,port:9092 with correlation id 0 for 1 topic(s)
> Set(inputTopic)
> 11893 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> 12045 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> 12062 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching
> metadata [{TopicMetadata for topic inputTopic ->
> No partition metadata for topic inputTopic due to
> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
> kafka.common.LeaderNotAvailableException
> 12066 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker
> id:0,host:192.168.59.103,port:9092 with correlation id 1 for 1 topic(s)
> Set(inputTopic)
> 12067 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> 12097 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> 12098 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching
> metadata [{TopicMetadata for topic inputTopic ->
> No partition metadata for topic inputTopic due to
> kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
> kafka.common.LeaderNotAvailableException
> 12098 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to
> collate messages by topic, partition due to: Failed to fetch topic metadata
> for topic: inputTopic
> 12099 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for
> 1000 ms before retrying send. Remaining retries = 3
> 13104 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from broker
> id:0,host:192.168.59.103,port:9092 with correlation id 2 for 1 topic(s)
> Set(inputTopic)
> 13111 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
> 13137 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> 192.168.59.103:9092
> 13161 [main] INFO kafka.producer.SyncProducer - Connected to
> 192.168.59.103:9092 for producing
>
> In case this would be not enough for you, you can try to change the values of
> - message.send.max.retries from the default 5 to e.g. 10 and
> - topic.metadata.refresh.interval.ms to 0
>
>
> It is still unclear, why it takes 2 tries i.e. 2 seconds to fetch the topic
> metadata…
>
>
> Cheers,
> Andras
>
> > On 26 Sep 2014, at 14:43, Andras Hatvani <[email protected]>
> > wrote:
> >
> > Hi,
> >
> > What does the exception mean in the following context?
> >
> > I've got two Kafka brokers out which the second one will be elected as
> > leader:
> >
> > kafka2_1 | [2014-09-26 12:35:07,289] INFO [Kafka Server 2],
> > started (kafka.server.KafkaServer)
> > kafka2_1 | [2014-09-26 12:35:07,394] INFO New leader is 2
> > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> >
> > Then, when I try to produce a message, I only get the following error:
> >
> > 23182 [main] INFO kafka.producer.async.DefaultEventHandler - Back off for
> > 100 ms before retrying send. Remaining retries = 0
> > 23286 [main] INFO kafka.client.ClientUtils$ - Fetching metadata from
> > broker id:0,host:192.168.59.103,port:9092 with correlation id 8 for 1
> > topic(s) Set(inputTopic)
> > 23287 [main] INFO kafka.producer.SyncProducer - Connected to
> > 192.168.59.103:9092 for producing
> > 23300 [main] INFO kafka.producer.SyncProducer - Disconnecting from
> > 192.168.59.103:9092
> > 23301 [main] WARN kafka.producer.BrokerPartitionInfo - Error while
> > fetching metadata [{TopicMetadata for topic inputTopic ->
> > No partition metadata for topic inputTopic due to
> > kafka.common.LeaderNotAvailableException}] for topic [inputTopic]: class
> > kafka.common.LeaderNotAvailableException
> > 23303 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to
> > send requests for topics inputTopic with correlation ids in [0,8]
> >
> > The property metadata.broker.list contains the host:port values of both
> > brokers.
> >
> > What can be wrong/missing?
> > Thanks,
> > Andras
>
