Re: Leader doesn't get assigned for new topics

2013-09-18 Thread Neha Narkhede
When new topics are created, it takes some time for the controller to communicate the partition assignment for the new topic to all brokers. Until that happens, attempts to send/receive data to the brokers fails with the following kind of error - [2013-09-17 19:21:36,531] WARN [KafkaApi-2] Produce

Re: Leader doesn't get assigned for new topics

2013-09-18 Thread Rajasekar Elango
>From the output of StateChangeLogMerger tool, I see only this error repeated; [2013-09-18 14:16:48,358] ERROR [KafkaApi-1] Error while fetching metadata for partition [FunnelProto,0] (kafka.server.KafkaApis) On the state-change.log itself, I see this error: [2013-09-18 14:22:48,954] ERROR Condi

Re: Leader doesn't get assigned for new topics

2013-09-17 Thread Neha Narkhede
Raja, Could you run the StateChangeLogMerger tool and give it one topic-partition that has the above mentioned problem. This tool is documented here - https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-7.StateChangeLogMergerTool . Let me know if you run into any

Re: Leader doesn't get assigned for new topics

2013-09-17 Thread Rajasekar Elango
Neha/Jun, The same problem started happening again although now our zookeeper cluster is configured correctly. The produce always failed with LeaderNotAvailableException and list topics shows topic is created with leader "none". In the controller and stage-change log, I am seeing lot of these fail

Re: Leader doesn't get assigned for new topics

2013-09-13 Thread Neha Narkhede
Ah ok. Thanks for sharing that. On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Elango wrote: > We have 3 zookeeper node in the cluster with a hardware load balancer . In > one of the zookeeper, we did not configure ensemble correctly (server.n > property in zoo.cfg) . So it ended up as like 2 node

Re: Leader doesn't get assigned for new topics

2013-09-13 Thread Rajasekar Elango
We have 3 zookeeper node in the cluster with a hardware load balancer . In one of the zookeeper, we did not configure ensemble correctly (server.n property in zoo.cfg) . So it ended up as like 2 nodes in one cluster, one node in other cluster. The load balancer is randomly hitting one of 2 zookeep

Re: Leader doesn't get assigned for new topics

2013-09-13 Thread Neha Narkhede
Just curious to know, what was the misconfiguration? On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango wrote: > Thanks Neha and Jun, It turned out to be miss configuration in our > zookeeper cluster. After correcting it everything looks good. > > Thanks, > Raja. > > > On Fri, Sep 13, 2013 at 10

Re: Leader doesn't get assigned for new topics

2013-09-13 Thread Rajasekar Elango
Thanks Neha and Jun, It turned out to be miss configuration in our zookeeper cluster. After correcting it everything looks good. Thanks, Raja. On Fri, Sep 13, 2013 at 10:13 AM, Jun Rao wrote: > Any error in the controller and the state-change log? Are brokers 2,3,4 > alive? > > Thanks, > > Jun

Re: Leader doesn't get assigned for new topics

2013-09-13 Thread Neha Narkhede
Do you see any errors in the controller.log or the state change log? Thanks, Neha On Sep 12, 2013 10:47 PM, "Rajasekar Elango" wrote: > We are seeing a problem that we we try to send messages to new topic it > fails kafka.common.LeaderNotAvailableException. But usually this problem > will be tra

Re: Leader doesn't get assigned for new topics

2013-09-13 Thread Jun Rao
Any error in the controller and the state-change log? Are brokers 2,3,4 alive? Thanks, Jun On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango wrote: > We are seeing a problem that we we try to send messages to new topic it > fails kafka.common.LeaderNotAvailableException. But usually this probl

Leader doesn't get assigned for new topics

2013-09-12 Thread Rajasekar Elango
We are seeing a problem that we we try to send messages to new topic it fails kafka.common.LeaderNotAvailableException. But usually this problem will be transient and if we re-send messages to same topic will work. But now we tried rending message to same topic several time, but still fails with sa