Re: frequent periods of ~1500 replicas not in sync

2014-10-22 Thread Neha Narkhede
Neil, We fixed a bug related to the BadVersion problem in 0.8.1.1. Would you mind repeating your test on 0.8.1.1 and if you can still reproduce this issue, then send around the thread dump and attach the logs to KAFKA-1407? Thanks, Neha On Tue, Oct 21, 2014 at 11:56 AM, Neil Harkins

frequent periods of ~1500 replicas not in sync

2014-10-21 Thread Neil Harkins
Hi. I've got a 5 node cluster running Kafka 0.8.1, with 4697 partitions (2 replicas each) across 564 topics. I'm sending it about 1% of our total messaging load now, and several times a day there is a period where 1~1500 partitions have one replica not in sync. Is this normal? If a consumer is

Re: frequent periods of ~1500 replicas not in sync

2014-10-21 Thread Gwen Shapira
Consumers always read from the leader replica, which is always in sync by definition. So you are good there. The concern would be if the leader crashes during this period. On Tue, Oct 21, 2014 at 2:56 PM, Neil Harkins nhark...@gmail.com wrote: Hi. I've got a 5 node cluster running Kafka 0.8.1,

Re: frequent periods of ~1500 replicas not in sync

2014-10-21 Thread Guozhang Wang
Neil, what you are seeing could probably be KAFKA-1407 https://issues.apache.org/jira/browse/KAFKA-1407. On Tue, Oct 21, 2014 at 12:03 PM, Gwen Shapira gshap...@cloudera.com wrote: Consumers always read from the leader replica, which is always in sync by definition. So you are good there. The