Re: LeaderNotAvailableException in 0.8.1.1
yes, I gave it several minutes. On Sat, Jun 14, 2014 at 2:18 PM, Michael G. Noll mich...@michael-noll.com wrote: Have you given Kafka some time to re-elect a new leader for the missing partition when you re-try steps 1-5? See here: If you do, you should be able to go through steps 1-8 without seeing LeaderNotAvailableExceptions (you may need to give Kafka some time to re-elect the remaining, second broker as the new leader for the first broker's partitions though). Best, Michael On 06/12/2014 08:43 PM, Prakash Gowri Shankor wrote: So if we go back to the 2 broker case, I tried your suggestion with replication-factor 2 ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper localhost:2181 --replication-factor When i repeat steps 1-5 i still see the exception. When i go to step 8 ( back to 2 brokers ), I dont see it. Here is my topic description: ./kafka-topics.sh --describe --topic test2 --zookeeper localhost:2181 Topic:test2 PartitionCount:3 ReplicationFactor:2 Configs: Topic: test2 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: test2 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: test2 Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0 On Wed, Jun 11, 2014 at 3:20 PM, Michael G. Noll michael+st...@michael-noll.com wrote: In your second case (1-broker cluster and putting your laptop to sleep) these exceptions should be transient and disappear after a while. In the logs you should see ZK session expirations (hence the initial/transient exceptions, which in this case are expected and ok), followed by new ZK sessions being established. So this case is (should?) be very different from your case number 1. --Michael On 11.06.2014, at 23:13, Prakash Gowri Shankor prakash.shan...@gmail.com wrote: Thanks for your response Michael. In step 3, I am actually stopping the entire cluster and restarting it without the 2nd broker. But I see your point. When i look in /tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it holds test2-1 ( ie 1st partition of test2 topic ). For /tmp/kafka-logs ( which is the log dir for the first broker ) I see it holds test2-0 and test2-2 ( 0th and 2nd partition of test2 topic ). So it would seem that kafka is missing the leader for partition 1 and hence throwing the exception on the producer side. Let me try your replication suggestion. While all of the above might explain the exception in the case of 2 brokers, there are still times when I see it with just a single broker. In this case, I start from a normal working cluster with 1 broker only. Then I either put my machine into sleep/hibernation. On wake, I do shutdown the cluster ( for sanity ) and restart. On restart, I start seeing this exception. In this case i only have one broker. I still create the topic the way i described earlier. I understand this is not the ideal production topology, but its annoying to see it during development. Thanks On Wed, Jun 11, 2014 at 1:40 PM, Michael G. Noll mich...@michael-noll.com wrote: Prakash, you are configure the topic with a replication factor of only 1, i.e. no additional replica beyond the original one. This replication setting of 1 means that only one of the two brokers will ever host the (single) replica -- which is implied to also be the leader in-sync replica -- of a given partition. In step 3 you are disabling one of the two brokers. Because this stopped broker is the only broker that hosts one or more of the 3 partitions you configured (I can't tell which partition(s) it is, but you can find out by --describe'ing the topic), your Kafka cluster -- which is now running in degraded state -- will miss the leader of those affected partitions. And because you set the replication factor to 1, the remaining, second broker will not and will never take over the leadership of those partitions from the stopped broker. Hence you will keep getting the LeaderNotAvailableException's until you restart the stopped broker in step 7. So to me it looks as if the behavior of Kafka is actually correct and as expected. If you want to rectify your test setup, try increasing the replication factor from 1 to 2. If you do, you should be able to go through steps 1-8 without seeing LeaderNotAvailableExceptions (you may need to give Kafka some time to re-elect the remaining, second broker as the new leader for the first broker's partitions though). Hope this helps, Michael On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote: yes, here are the steps: Create topic as : ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper localhost:2181 --replication-factor 1 1) Start cluster with 2 brokers, 3 consumers. 2) Dont start any producer 3) Shutdown cluster and disable one broker from starting 4) restart
Re: LeaderNotAvailableException in 0.8.1.1
Have you given Kafka some time to re-elect a new leader for the missing partition when you re-try steps 1-5? See here: If you do, you should be able to go through steps 1-8 without seeing LeaderNotAvailableExceptions (you may need to give Kafka some time to re-elect the remaining, second broker as the new leader for the first broker's partitions though). Best, Michael On 06/12/2014 08:43 PM, Prakash Gowri Shankor wrote: So if we go back to the 2 broker case, I tried your suggestion with replication-factor 2 ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper localhost:2181 --replication-factor When i repeat steps 1-5 i still see the exception. When i go to step 8 ( back to 2 brokers ), I dont see it. Here is my topic description: ./kafka-topics.sh --describe --topic test2 --zookeeper localhost:2181 Topic:test2 PartitionCount:3 ReplicationFactor:2 Configs: Topic: test2 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: test2 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: test2 Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0 On Wed, Jun 11, 2014 at 3:20 PM, Michael G. Noll michael+st...@michael-noll.com wrote: In your second case (1-broker cluster and putting your laptop to sleep) these exceptions should be transient and disappear after a while. In the logs you should see ZK session expirations (hence the initial/transient exceptions, which in this case are expected and ok), followed by new ZK sessions being established. So this case is (should?) be very different from your case number 1. --Michael On 11.06.2014, at 23:13, Prakash Gowri Shankor prakash.shan...@gmail.com wrote: Thanks for your response Michael. In step 3, I am actually stopping the entire cluster and restarting it without the 2nd broker. But I see your point. When i look in /tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it holds test2-1 ( ie 1st partition of test2 topic ). For /tmp/kafka-logs ( which is the log dir for the first broker ) I see it holds test2-0 and test2-2 ( 0th and 2nd partition of test2 topic ). So it would seem that kafka is missing the leader for partition 1 and hence throwing the exception on the producer side. Let me try your replication suggestion. While all of the above might explain the exception in the case of 2 brokers, there are still times when I see it with just a single broker. In this case, I start from a normal working cluster with 1 broker only. Then I either put my machine into sleep/hibernation. On wake, I do shutdown the cluster ( for sanity ) and restart. On restart, I start seeing this exception. In this case i only have one broker. I still create the topic the way i described earlier. I understand this is not the ideal production topology, but its annoying to see it during development. Thanks On Wed, Jun 11, 2014 at 1:40 PM, Michael G. Noll mich...@michael-noll.com wrote: Prakash, you are configure the topic with a replication factor of only 1, i.e. no additional replica beyond the original one. This replication setting of 1 means that only one of the two brokers will ever host the (single) replica -- which is implied to also be the leader in-sync replica -- of a given partition. In step 3 you are disabling one of the two brokers. Because this stopped broker is the only broker that hosts one or more of the 3 partitions you configured (I can't tell which partition(s) it is, but you can find out by --describe'ing the topic), your Kafka cluster -- which is now running in degraded state -- will miss the leader of those affected partitions. And because you set the replication factor to 1, the remaining, second broker will not and will never take over the leadership of those partitions from the stopped broker. Hence you will keep getting the LeaderNotAvailableException's until you restart the stopped broker in step 7. So to me it looks as if the behavior of Kafka is actually correct and as expected. If you want to rectify your test setup, try increasing the replication factor from 1 to 2. If you do, you should be able to go through steps 1-8 without seeing LeaderNotAvailableExceptions (you may need to give Kafka some time to re-elect the remaining, second broker as the new leader for the first broker's partitions though). Hope this helps, Michael On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote: yes, here are the steps: Create topic as : ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper localhost:2181 --replication-factor 1 1) Start cluster with 2 brokers, 3 consumers. 2) Dont start any producer 3) Shutdown cluster and disable one broker from starting 4) restart cluster with 1 broker, 3 consumers 5) Start producer and send messages. I see this exception 6) Shutdown cluster. 7) Enable 2nd broker. 8) Restart cluster with 2 brokers, 3 consumer and the one producer and send messages. Now I dont see
Re: LeaderNotAvailableException in 0.8.1.1
So if we go back to the 2 broker case, I tried your suggestion with replication-factor 2 ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper localhost:2181 --replication-factor When i repeat steps 1-5 i still see the exception. When i go to step 8 ( back to 2 brokers ), I dont see it. Here is my topic description: ./kafka-topics.sh --describe --topic test2 --zookeeper localhost:2181 Topic:test2 PartitionCount:3 ReplicationFactor:2 Configs: Topic: test2 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: test2 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: test2 Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0 On Wed, Jun 11, 2014 at 3:20 PM, Michael G. Noll michael+st...@michael-noll.com wrote: In your second case (1-broker cluster and putting your laptop to sleep) these exceptions should be transient and disappear after a while. In the logs you should see ZK session expirations (hence the initial/transient exceptions, which in this case are expected and ok), followed by new ZK sessions being established. So this case is (should?) be very different from your case number 1. --Michael On 11.06.2014, at 23:13, Prakash Gowri Shankor prakash.shan...@gmail.com wrote: Thanks for your response Michael. In step 3, I am actually stopping the entire cluster and restarting it without the 2nd broker. But I see your point. When i look in /tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it holds test2-1 ( ie 1st partition of test2 topic ). For /tmp/kafka-logs ( which is the log dir for the first broker ) I see it holds test2-0 and test2-2 ( 0th and 2nd partition of test2 topic ). So it would seem that kafka is missing the leader for partition 1 and hence throwing the exception on the producer side. Let me try your replication suggestion. While all of the above might explain the exception in the case of 2 brokers, there are still times when I see it with just a single broker. In this case, I start from a normal working cluster with 1 broker only. Then I either put my machine into sleep/hibernation. On wake, I do shutdown the cluster ( for sanity ) and restart. On restart, I start seeing this exception. In this case i only have one broker. I still create the topic the way i described earlier. I understand this is not the ideal production topology, but its annoying to see it during development. Thanks On Wed, Jun 11, 2014 at 1:40 PM, Michael G. Noll mich...@michael-noll.com wrote: Prakash, you are configure the topic with a replication factor of only 1, i.e. no additional replica beyond the original one. This replication setting of 1 means that only one of the two brokers will ever host the (single) replica -- which is implied to also be the leader in-sync replica -- of a given partition. In step 3 you are disabling one of the two brokers. Because this stopped broker is the only broker that hosts one or more of the 3 partitions you configured (I can't tell which partition(s) it is, but you can find out by --describe'ing the topic), your Kafka cluster -- which is now running in degraded state -- will miss the leader of those affected partitions. And because you set the replication factor to 1, the remaining, second broker will not and will never take over the leadership of those partitions from the stopped broker. Hence you will keep getting the LeaderNotAvailableException's until you restart the stopped broker in step 7. So to me it looks as if the behavior of Kafka is actually correct and as expected. If you want to rectify your test setup, try increasing the replication factor from 1 to 2. If you do, you should be able to go through steps 1-8 without seeing LeaderNotAvailableExceptions (you may need to give Kafka some time to re-elect the remaining, second broker as the new leader for the first broker's partitions though). Hope this helps, Michael On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote: yes, here are the steps: Create topic as : ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper localhost:2181 --replication-factor 1 1) Start cluster with 2 brokers, 3 consumers. 2) Dont start any producer 3) Shutdown cluster and disable one broker from starting 4) restart cluster with 1 broker, 3 consumers 5) Start producer and send messages. I see this exception 6) Shutdown cluster. 7) Enable 2nd broker. 8) Restart cluster with 2 brokers, 3 consumer and the one producer and send messages. Now I dont see the exception.
Re: LeaderNotAvailableException in 0.8.1.1
Is this what you want from kafka-topics ? I took this script dump now when
the exception is occuring.
./kafka-topics.sh --describe test2 --zookeeper localhost:2181
Topic:test2 PartitionCount:3 ReplicationFactor:1 Configs:
Topic: test2 Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: test2 Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: test2 Partition: 2 Leader: 0 Replicas: 0 Isr: 0
On Tue, Jun 10, 2014 at 9:22 PM, Jun Rao jun...@gmail.com wrote:
Could you use the kafka-topic command to describe test2 and see if the
leader is available?
Thanks,
Jun
On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor
prakash.shan...@gmail.com wrote:
Hi,
I am running a cluster with a single broker, the performance producer
script and 3 consumers.
On a fresh start of the cluster , the producer throws this exception.
I was able to run this cluster successfully on the same topic ( test2 )
successfully the first time.
The solution( from stackoverflow ) seems to be to delete the topic data
in
the broker and in zookeeper. This doesnt seem to be a viable production
solution to me. Is there a way to solve this without losing topic data ?
Also does the incidence of this problem decrease if I run more
brokers/servers ?
I see logs like these in the server.log
[2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,0] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,1] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,2] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
The exception trace is:
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
with correlation ids in [41,48]
(kafka.producer.async.DefaultEventHandler)
[2014-06-10 10:45:32,464] WARN Error while fetching metadata
[{TopicMetadata for topic test2 -
No partition metadata for topic test2 due to
kafka.common.LeaderNotAvailableException}] for topic [test2]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
Re: LeaderNotAvailableException in 0.8.1.1
Prakash, you are configure the topic with a replication factor of only 1, i.e. no additional replica beyond the original one. This replication setting of 1 means that only one of the two brokers will ever host the (single) replica -- which is implied to also be the leader in-sync replica -- of a given partition. In step 3 you are disabling one of the two brokers. Because this stopped broker is the only broker that hosts one or more of the 3 partitions you configured (I can't tell which partition(s) it is, but you can find out by --describe'ing the topic), your Kafka cluster -- which is now running in degraded state -- will miss the leader of those affected partitions. And because you set the replication factor to 1, the remaining, second broker will not and will never take over the leadership of those partitions from the stopped broker. Hence you will keep getting the LeaderNotAvailableException's until you restart the stopped broker in step 7. So to me it looks as if the behavior of Kafka is actually correct and as expected. If you want to rectify your test setup, try increasing the replication factor from 1 to 2. If you do, you should be able to go through steps 1-8 without seeing LeaderNotAvailableExceptions (you may need to give Kafka some time to re-elect the remaining, second broker as the new leader for the first broker's partitions though). Hope this helps, Michael On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote: yes, here are the steps: Create topic as : ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper localhost:2181 --replication-factor 1 1) Start cluster with 2 brokers, 3 consumers. 2) Dont start any producer 3) Shutdown cluster and disable one broker from starting 4) restart cluster with 1 broker, 3 consumers 5) Start producer and send messages. I see this exception 6) Shutdown cluster. 7) Enable 2nd broker. 8) Restart cluster with 2 brokers, 3 consumer and the one producer and send messages. Now I dont see the exception.
Re: LeaderNotAvailableException in 0.8.1.1
Thanks for your response Michael. In step 3, I am actually stopping the entire cluster and restarting it without the 2nd broker. But I see your point. When i look in /tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it holds test2-1 ( ie 1st partition of test2 topic ). For /tmp/kafka-logs ( which is the log dir for the first broker ) I see it holds test2-0 and test2-2 ( 0th and 2nd partition of test2 topic ). So it would seem that kafka is missing the leader for partition 1 and hence throwing the exception on the producer side. Let me try your replication suggestion. While all of the above might explain the exception in the case of 2 brokers, there are still times when I see it with just a single broker. In this case, I start from a normal working cluster with 1 broker only. Then I either put my machine into sleep/hibernation. On wake, I do shutdown the cluster ( for sanity ) and restart. On restart, I start seeing this exception. In this case i only have one broker. I still create the topic the way i described earlier. I understand this is not the ideal production topology, but its annoying to see it during development. Thanks On Wed, Jun 11, 2014 at 1:40 PM, Michael G. Noll mich...@michael-noll.com wrote: Prakash, you are configure the topic with a replication factor of only 1, i.e. no additional replica beyond the original one. This replication setting of 1 means that only one of the two brokers will ever host the (single) replica -- which is implied to also be the leader in-sync replica -- of a given partition. In step 3 you are disabling one of the two brokers. Because this stopped broker is the only broker that hosts one or more of the 3 partitions you configured (I can't tell which partition(s) it is, but you can find out by --describe'ing the topic), your Kafka cluster -- which is now running in degraded state -- will miss the leader of those affected partitions. And because you set the replication factor to 1, the remaining, second broker will not and will never take over the leadership of those partitions from the stopped broker. Hence you will keep getting the LeaderNotAvailableException's until you restart the stopped broker in step 7. So to me it looks as if the behavior of Kafka is actually correct and as expected. If you want to rectify your test setup, try increasing the replication factor from 1 to 2. If you do, you should be able to go through steps 1-8 without seeing LeaderNotAvailableExceptions (you may need to give Kafka some time to re-elect the remaining, second broker as the new leader for the first broker's partitions though). Hope this helps, Michael On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote: yes, here are the steps: Create topic as : ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper localhost:2181 --replication-factor 1 1) Start cluster with 2 brokers, 3 consumers. 2) Dont start any producer 3) Shutdown cluster and disable one broker from starting 4) restart cluster with 1 broker, 3 consumers 5) Start producer and send messages. I see this exception 6) Shutdown cluster. 7) Enable 2nd broker. 8) Restart cluster with 2 brokers, 3 consumer and the one producer and send messages. Now I dont see the exception.
Re: LeaderNotAvailableException in 0.8.1.1
In your second case (1-broker cluster and putting your laptop to sleep) these exceptions should be transient and disappear after a while. In the logs you should see ZK session expirations (hence the initial/transient exceptions, which in this case are expected and ok), followed by new ZK sessions being established. So this case is (should?) be very different from your case number 1. --Michael On 11.06.2014, at 23:13, Prakash Gowri Shankor prakash.shan...@gmail.com wrote: Thanks for your response Michael. In step 3, I am actually stopping the entire cluster and restarting it without the 2nd broker. But I see your point. When i look in /tmp/kafka-logs-2 ( which is the log dir for the 2nd broker ) I see it holds test2-1 ( ie 1st partition of test2 topic ). For /tmp/kafka-logs ( which is the log dir for the first broker ) I see it holds test2-0 and test2-2 ( 0th and 2nd partition of test2 topic ). So it would seem that kafka is missing the leader for partition 1 and hence throwing the exception on the producer side. Let me try your replication suggestion. While all of the above might explain the exception in the case of 2 brokers, there are still times when I see it with just a single broker. In this case, I start from a normal working cluster with 1 broker only. Then I either put my machine into sleep/hibernation. On wake, I do shutdown the cluster ( for sanity ) and restart. On restart, I start seeing this exception. In this case i only have one broker. I still create the topic the way i described earlier. I understand this is not the ideal production topology, but its annoying to see it during development. Thanks On Wed, Jun 11, 2014 at 1:40 PM, Michael G. Noll mich...@michael-noll.com wrote: Prakash, you are configure the topic with a replication factor of only 1, i.e. no additional replica beyond the original one. This replication setting of 1 means that only one of the two brokers will ever host the (single) replica -- which is implied to also be the leader in-sync replica -- of a given partition. In step 3 you are disabling one of the two brokers. Because this stopped broker is the only broker that hosts one or more of the 3 partitions you configured (I can't tell which partition(s) it is, but you can find out by --describe'ing the topic), your Kafka cluster -- which is now running in degraded state -- will miss the leader of those affected partitions. And because you set the replication factor to 1, the remaining, second broker will not and will never take over the leadership of those partitions from the stopped broker. Hence you will keep getting the LeaderNotAvailableException's until you restart the stopped broker in step 7. So to me it looks as if the behavior of Kafka is actually correct and as expected. If you want to rectify your test setup, try increasing the replication factor from 1 to 2. If you do, you should be able to go through steps 1-8 without seeing LeaderNotAvailableExceptions (you may need to give Kafka some time to re-elect the remaining, second broker as the new leader for the first broker's partitions though). Hope this helps, Michael On 06/11/2014 07:49 PM, Prakash Gowri Shankor wrote: yes, here are the steps: Create topic as : ./kafka-topics.sh --topic test2 --create --partitions 3 --zookeeper localhost:2181 --replication-factor 1 1) Start cluster with 2 brokers, 3 consumers. 2) Dont start any producer 3) Shutdown cluster and disable one broker from starting 4) restart cluster with 1 broker, 3 consumers 5) Start producer and send messages. I see this exception 6) Shutdown cluster. 7) Enable 2nd broker. 8) Restart cluster with 2 brokers, 3 consumer and the one producer and send messages. Now I dont see the exception.
Re: LeaderNotAvailableException in 0.8.1.1
Hello Prakash,
Is this exception transient or persistent on broker startup?
Guozhang
On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor
prakash.shan...@gmail.com wrote:
Hi,
I am running a cluster with a single broker, the performance producer
script and 3 consumers.
On a fresh start of the cluster , the producer throws this exception.
I was able to run this cluster successfully on the same topic ( test2 )
successfully the first time.
The solution( from stackoverflow ) seems to be to delete the topic data in
the broker and in zookeeper. This doesnt seem to be a viable production
solution to me. Is there a way to solve this without losing topic data ?
Also does the incidence of this problem decrease if I run more
brokers/servers ?
I see logs like these in the server.log
[2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with correlation
id 0 from client on partition [test2,0] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with correlation
id 0 from client on partition [test2,1] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with correlation
id 0 from client on partition [test2,2] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
The exception trace is:
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
with correlation ids in [41,48] (kafka.producer.async.DefaultEventHandler)
[2014-06-10 10:45:32,464] WARN Error while fetching metadata
[{TopicMetadata for topic test2 -
No partition metadata for topic test2 due to
kafka.common.LeaderNotAvailableException}] for topic [test2]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
--
-- Guozhang
Re: LeaderNotAvailableException in 0.8.1.1
From the moment it starts occuring, it is persistent. Restarts dont seem to
make it go away. The only thing that makes it go away is following the
steps listed in this stackoverflow thread.
http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang wangg...@gmail.com wrote:
Hello Prakash,
Is this exception transient or persistent on broker startup?
Guozhang
On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor
prakash.shan...@gmail.com wrote:
Hi,
I am running a cluster with a single broker, the performance producer
script and 3 consumers.
On a fresh start of the cluster , the producer throws this exception.
I was able to run this cluster successfully on the same topic ( test2 )
successfully the first time.
The solution( from stackoverflow ) seems to be to delete the topic data
in
the broker and in zookeeper. This doesnt seem to be a viable production
solution to me. Is there a way to solve this without losing topic data ?
Also does the incidence of this problem decrease if I run more
brokers/servers ?
I see logs like these in the server.log
[2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,0] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,1] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,2] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
The exception trace is:
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
with correlation ids in [41,48]
(kafka.producer.async.DefaultEventHandler)
[2014-06-10 10:45:32,464] WARN Error while fetching metadata
[{TopicMetadata for topic test2 -
No partition metadata for topic test2 due to
kafka.common.LeaderNotAvailableException}] for topic [test2]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
--
-- Guozhang
Re: LeaderNotAvailableException in 0.8.1.1
Did you use the delete topic command?
That was an experimental feature in the 0.8.1 release with several
bugs. The fixes are all on trunk, but those fixes did not make it into
0.8.1.1 - except for a config option to disable delete-topic support
on the broker.
Joel
On Tue, Jun 10, 2014 at 01:07:45PM -0700, Prakash Gowri Shankor wrote:
From the moment it starts occuring, it is persistent. Restarts dont seem to
make it go away. The only thing that makes it go away is following the
steps listed in this stackoverflow thread.
http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang wangg...@gmail.com wrote:
Hello Prakash,
Is this exception transient or persistent on broker startup?
Guozhang
On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor
prakash.shan...@gmail.com wrote:
Hi,
I am running a cluster with a single broker, the performance producer
script and 3 consumers.
On a fresh start of the cluster , the producer throws this exception.
I was able to run this cluster successfully on the same topic ( test2 )
successfully the first time.
The solution( from stackoverflow ) seems to be to delete the topic data
in
the broker and in zookeeper. This doesnt seem to be a viable production
solution to me. Is there a way to solve this without losing topic data ?
Also does the incidence of this problem decrease if I run more
brokers/servers ?
I see logs like these in the server.log
[2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,0] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,1] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,2] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
The exception trace is:
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
with correlation ids in [41,48]
(kafka.producer.async.DefaultEventHandler)
[2014-06-10 10:45:32,464] WARN Error while fetching metadata
[{TopicMetadata for topic test2 -
No partition metadata for topic test2 due to
kafka.common.LeaderNotAvailableException}] for topic [test2]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
--
-- Guozhang
Re: LeaderNotAvailableException in 0.8.1.1
No i have not used the delete topic feature. I have been manually deleting
the topics from zookeeper and removing the topic from the kafka and zk logs.
I've experimented a bit more. It seems like this occurs when I have a
single broker running. When i restart with 2 brokers, the problem goes away.
On Tue, Jun 10, 2014 at 2:09 PM, Joel Koshy jjkosh...@gmail.com wrote:
Did you use the delete topic command?
That was an experimental feature in the 0.8.1 release with several
bugs. The fixes are all on trunk, but those fixes did not make it into
0.8.1.1 - except for a config option to disable delete-topic support
on the broker.
Joel
On Tue, Jun 10, 2014 at 01:07:45PM -0700, Prakash Gowri Shankor wrote:
From the moment it starts occuring, it is persistent. Restarts dont seem
to
make it go away. The only thing that makes it go away is following the
steps listed in this stackoverflow thread.
http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang wangg...@gmail.com
wrote:
Hello Prakash,
Is this exception transient or persistent on broker startup?
Guozhang
On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor
prakash.shan...@gmail.com wrote:
Hi,
I am running a cluster with a single broker, the performance producer
script and 3 consumers.
On a fresh start of the cluster , the producer throws this exception.
I was able to run this cluster successfully on the same topic (
test2 )
successfully the first time.
The solution( from stackoverflow ) seems to be to delete the topic
data
in
the broker and in zookeeper. This doesnt seem to be a viable
production
solution to me. Is there a way to solve this without losing topic
data ?
Also does the incidence of this problem decrease if I run more
brokers/servers ?
I see logs like these in the server.log
[2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,0] failed due to Topic test2
either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,1] failed due to Topic test2
either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,2] failed due to Topic test2
either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
The exception trace is:
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages
after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2014-06-10 10:45:32,464] ERROR Failed to send requests for topics
test2
with correlation ids in [41,48]
(kafka.producer.async.DefaultEventHandler)
[2014-06-10 10:45:32,464] WARN Error while fetching metadata
[{TopicMetadata for topic test2 -
No partition metadata for topic test2 due to
kafka.common.LeaderNotAvailableException}] for topic [test2]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages
after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
Re: LeaderNotAvailableException in 0.8.1.1
Any exceptions you saw on the controller log?
On Tue, Jun 10, 2014 at 2:23 PM, Prakash Gowri Shankor
prakash.shan...@gmail.com wrote:
No i have not used the delete topic feature. I have been manually deleting
the topics from zookeeper and removing the topic from the kafka and zk
logs.
I've experimented a bit more. It seems like this occurs when I have a
single broker running. When i restart with 2 brokers, the problem goes
away.
On Tue, Jun 10, 2014 at 2:09 PM, Joel Koshy jjkosh...@gmail.com wrote:
Did you use the delete topic command?
That was an experimental feature in the 0.8.1 release with several
bugs. The fixes are all on trunk, but those fixes did not make it into
0.8.1.1 - except for a config option to disable delete-topic support
on the broker.
Joel
On Tue, Jun 10, 2014 at 01:07:45PM -0700, Prakash Gowri Shankor wrote:
From the moment it starts occuring, it is persistent. Restarts dont
seem
to
make it go away. The only thing that makes it go away is following the
steps listed in this stackoverflow thread.
http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang wangg...@gmail.com
wrote:
Hello Prakash,
Is this exception transient or persistent on broker startup?
Guozhang
On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor
prakash.shan...@gmail.com wrote:
Hi,
I am running a cluster with a single broker, the performance
producer
script and 3 consumers.
On a fresh start of the cluster , the producer throws this
exception.
I was able to run this cluster successfully on the same topic (
test2 )
successfully the first time.
The solution( from stackoverflow ) seems to be to delete the topic
data
in
the broker and in zookeeper. This doesnt seem to be a viable
production
solution to me. Is there a way to solve this without losing topic
data ?
Also does the incidence of this problem decrease if I run more
brokers/servers ?
I see logs like these in the server.log
[2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,0] failed due to Topic test2
either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,1] failed due to Topic test2
either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,2] failed due to Topic test2
either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
The exception trace is:
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200
events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages
after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2014-06-10 10:45:32,464] ERROR Failed to send requests for topics
test2
with correlation ids in [41,48]
(kafka.producer.async.DefaultEventHandler)
[2014-06-10 10:45:32,464] WARN Error while fetching metadata
[{TopicMetadata for topic test2 -
No partition metadata for topic test2 due to
kafka.common.LeaderNotAvailableException}] for topic [test2]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200
events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages
after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
Re: LeaderNotAvailableException in 0.8.1.1
Do you by any chance have steps to reproduce this issue easily?
On Tue, Jun 10, 2014 at 02:23:20PM -0700, Prakash Gowri Shankor wrote:
No i have not used the delete topic feature. I have been manually deleting
the topics from zookeeper and removing the topic from the kafka and zk logs.
I've experimented a bit more. It seems like this occurs when I have a
single broker running. When i restart with 2 brokers, the problem goes away.
On Tue, Jun 10, 2014 at 2:09 PM, Joel Koshy jjkosh...@gmail.com wrote:
Did you use the delete topic command?
That was an experimental feature in the 0.8.1 release with several
bugs. The fixes are all on trunk, but those fixes did not make it into
0.8.1.1 - except for a config option to disable delete-topic support
on the broker.
Joel
On Tue, Jun 10, 2014 at 01:07:45PM -0700, Prakash Gowri Shankor wrote:
From the moment it starts occuring, it is persistent. Restarts dont seem
to
make it go away. The only thing that makes it go away is following the
steps listed in this stackoverflow thread.
http://stackoverflow.com/questions/23228222/running-into-leadernotavailableexception-when-using-kafka-0-8-1-with-zookeeper-3
On Tue, Jun 10, 2014 at 12:47 PM, Guozhang Wang wangg...@gmail.com
wrote:
Hello Prakash,
Is this exception transient or persistent on broker startup?
Guozhang
On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor
prakash.shan...@gmail.com wrote:
Hi,
I am running a cluster with a single broker, the performance producer
script and 3 consumers.
On a fresh start of the cluster , the producer throws this exception.
I was able to run this cluster successfully on the same topic (
test2 )
successfully the first time.
The solution( from stackoverflow ) seems to be to delete the topic
data
in
the broker and in zookeeper. This doesnt seem to be a viable
production
solution to me. Is there a way to solve this without losing topic
data ?
Also does the incidence of this problem decrease if I run more
brokers/servers ?
I see logs like these in the server.log
[2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,0] failed due to Topic test2
either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,1] failed due to Topic test2
either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with
correlation
id 0 from client on partition [test2,2] failed due to Topic test2
either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
The exception trace is:
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages
after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2014-06-10 10:45:32,464] ERROR Failed to send requests for topics
test2
with correlation ids in [41,48]
(kafka.producer.async.DefaultEventHandler)
[2014-06-10 10:45:32,464] WARN Error while fetching metadata
[{TopicMetadata for topic test2 -
No partition metadata for topic test2 due to
kafka.common.LeaderNotAvailableException}] for topic [test2]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages
after
3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
Re: LeaderNotAvailableException in 0.8.1.1
Could you use the kafka-topic command to describe test2 and see if the
leader is available?
Thanks,
Jun
On Tue, Jun 10, 2014 at 11:04 AM, Prakash Gowri Shankor
prakash.shan...@gmail.com wrote:
Hi,
I am running a cluster with a single broker, the performance producer
script and 3 consumers.
On a fresh start of the cluster , the producer throws this exception.
I was able to run this cluster successfully on the same topic ( test2 )
successfully the first time.
The solution( from stackoverflow ) seems to be to delete the topic data in
the broker and in zookeeper. This doesnt seem to be a viable production
solution to me. Is there a way to solve this without losing topic data ?
Also does the incidence of this problem decrease if I run more
brokers/servers ?
I see logs like these in the server.log
[2014-06-10 10:45:35,194] WARN [KafkaApi-0] Offset request with correlation
id 0 from client on partition [test2,0] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,211] WARN [KafkaApi-0] Offset request with correlation
id 0 from client on partition [test2,1] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
[2014-06-10 10:45:35,221] WARN [KafkaApi-0] Offset request with correlation
id 0 from client on partition [test2,2] failed due to Topic test2 either
doesn't exist or is in the process of being deleted
(kafka.server.KafkaApis)
The exception trace is:
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2014-06-10 10:45:32,464] ERROR Failed to send requests for topics test2
with correlation ids in [41,48] (kafka.producer.async.DefaultEventHandler)
[2014-06-10 10:45:32,464] WARN Error while fetching metadata
[{TopicMetadata for topic test2 -
No partition metadata for topic test2 due to
kafka.common.LeaderNotAvailableException}] for topic [test2]: class
kafka.common.LeaderNotAvailableException
(kafka.producer.BrokerPartitionInfo)
[2014-06-10 10:45:32,464] ERROR Error in handling batch of 200 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
