Re: Leader: -1 on Kafka

No it is not common/expected. It will be difficult to give you any useful advise without your logs - please send over your state change logs and broker logs. On Mon, Apr 21, 2014 at 11:36:34PM -0500, Kashyap Mhaisekar wrote: > No. Is this error common? How to overcome this? > > Regards, > Kashyap

Re: Leader: -1 on Kafka

No. Is this error common? How to overcome this? Regards, Kashyap On Mon, Apr 21, 2014 at 4:02 PM, Joel Koshy wrote: > So only one of the brokers have the data? i.e., with one partition. > > Do you see errors in your broker log around that time? What do you see > in the kafka-state-change.log f

Re: Too many replicas after partition reassignment

Which version of Kafka are you using? Thanks, Jun On Mon, Apr 21, 2014 at 11:41 AM, Ryan Berdeen wrote: > After doing some partition reassignments, I've ended up with some > partitions that have both the old and new brokers assigned. > > The output of kafka-topics.sh --describe looks like thi

Re: Too many replicas after partition reassignment

Hi Ryan, Also KAFKA-1317 should be fixed in both trunk and latest 0.8.1 branch, are you running with either or just with one of the previous released versions? Tim On Mon, Apr 21, 2014 at 5:00 PM, Guozhang Wang wrote: > Hi Ryan, > > Did you see any error logs on the new controller's controller

Re: Too many replicas after partition reassignment

Hi Ryan, Did you see any error logs on the new controller's controller log and state-change log? Guozhang On Mon, Apr 21, 2014 at 11:41 AM, Ryan Berdeen wrote: > After doing some partition reassignments, I've ended up with some > partitions that have both the old and new brokers assigned. > >

Re: how to measure consumer time for x number of messages

There are message-rate mbeans that you can read. You can hook up something like jconsole/visualvm to your consumer instance and explore the mbeans. E.g., we have a messages/sec meter for each topic that I think would help in your case. Joel On Mon, Apr 21, 2014 at 03:14:38PM -0700, Kazue Watanabe

how to measure consumer time for x number of messages

Hi, I am looking for a way to measure time for a consumer to consumer x number of messages. Basically I would like to compare how much it takes the consumer to consume messages and compare them against the number of messages. Is there any tool in Kafka to do this? Please let me know. Thank you so

Re: Leader: -1 on Kafka

So only one of the brokers have the data? i.e., with one partition. Do you see errors in your broker log around that time? What do you see in the kafka-state-change.log for that topic/partition? On Mon, Apr 21, 2014 at 03:36:52PM -0500, Kashyap Mhaisekar wrote: > Yes. > I have 3 kafka brokers and

Re: Leader: -1 on Kafka

Yes. I have 3 kafka brokers and I have created one topic with 1 partition and 1 replication (default options). All params are default. I was testing it with heavy load and had a storm reading from this and suddenly the messages stop. kafka's list topics script files then show the leader as -1. I g

Too many replicas after partition reassignment

After doing some partition reassignments, I've ended up with some partitions that have both the old and new brokers assigned. The output of kafka-topics.sh --describe looks like this: Topic:cs-es-indexer-a PartitionCount:30 ReplicationFactor:2 Configs: retention.ms=1080 ... Topic: cs-es-ind

Re: Leader: -1 on Kafka

Can you describe your set up in more detail and also if you can reproduce this easily? This can happen when none of the replicas for a partition are available, but cannot comment further without details. On Mon, Apr 21, 2014 at 10:54:59AM -0500, Kashyap Mhaisekar wrote: > Hi, > At times, some of k

Leader: -1 on Kafka

Hi, At times, some of kafka topics end up showing the leader as -1. After this, the messages dont get added to the topic nor consumed. I tried digging into why the leader turns -1 *(leader: -1)* Is there a reason why this happens and how it can be resolved? Regards, kashyap

Re: Cluster design distribution and JBOD vs RAID

> If any drive fails the broker is > down. Oh interesting. Maybe it should only remove partitions on that drive from the ISR…feature request! :) On Apr 19, 2014, at 2:12 PM, Jay Kreps wrote: > I think we are saying the same thing. If any drive fails the broker is > down. But when the drive

Re: Kafka with multiple zookeper

you can give zkconnect as host1:port1,host2:port2,hostn:portn... If you use paths, then host1:port1,host2:port2,hostn:portn/ Arya On Mon, Apr 21, 2014 at 6:46 PM, Dhiraj Thakur < dhiraj.tha...@northgate-is.com> wrote: > Hi, > As per the documentation we can have multiple broker of kafka on mul

Kafka with multiple zookeper

Hi, As per the documentation we can have multiple broker of kafka on multiple node talking to single Zookeeper instance. Is it possible to use multiple instance of zookeeper with multiple instance of kafka broker? Such that if one instance of zookeeper goes down then another one can take its respon