Kafka crashed after multiple topics were added

We have 3 node kafka cluster. I initially created 4 topics. I wrote small shell script to create 150 topics. TOPICS=$(< $1) for topic in $TOPICS do echo "/usr/local/kafka/bin/kafka-create-topic.sh --replica 3 --topic $topic --zookeeper $2:2181/kafka --partition 36" /usr/local/kafka/bin/kafka

Re: Very low volume topic

Not sure if there is an easy way to do what you want. One thing I can think of is to reduce queuedchunks.max and fetch.size to reduce the # of queued messages. Thanks, Jun On Tue, Aug 13, 2013 at 6:47 PM, Eric Sites wrote: > Hello everyone, > > I have a very low volume topic that has 2 consume

Re: Kafka Consumer Threads Stalled

If you don't want to see ConsumerTimeoutException, just set consumer.timeout.ms to -1. If you do need consumer.timeout.ms larger than 0, make sure that on ConsumerTimeoutException, your consumer thread loops back and calls hasNext() on the iterator to resume the consumption. Thanks, Jun On Tue

Re: Very low volume topic

Responses inline On 8/13/13 9:57 PM, "Philip O'Toole" wrote: >My experience is solely with 0.72. More inline. I am currently using 0.8. > >On Tue, Aug 13, 2013 at 6:47 PM, Eric Sites >wrote: >> Hello everyone, >> >> I have a very low volume topic that has 2 consumers in the same group. >>How

Re: Very low volume topic

My experience is solely with 0.72. More inline. On Tue, Aug 13, 2013 at 6:47 PM, Eric Sites wrote: > Hello everyone, > > I have a very low volume topic that has 2 consumers in the same group. How do > I get each consumer to only consume 1 message at a time and if the the first > consumer is bus

Very low volume topic

Hello everyone, I have a very low volume topic that has 2 consumers in the same group. How do I get each consumer to only consume 1 message at a time and if the the first consumer is busy get the other consumer to consume the message? Currently what I am doing is: First consumer connects to Ka

Kafka Consumer Threads Stalled

Hi, We are using zookeeper 3.3.6 with kafka 0.7.2. We have a topic with 8 partitions on each of 3 brokers that we are consuming with a consumer group with multiple threads. We are using the following settings for our consumers: zk.connectiontimeout.ms=1200 fetch_size=52428800 queuedchunks.m

Re: Kafka/Hadoop consumers and producers

> may merge a bit of your work into bigtop-989 if that's ok with you. Merge away! Happy to help. :) > I'll ask on bigtop regarding the .deb requirement - it seems they don't abide > by this. Yeah, there seems to be a constant struggle between the 'java way' of doing things, e.g. Maven downlo

Re: Kafka 0.8 Maven and IntelliJ

Does anybody on the Kafka team have any insight into this issue? I'm stuck, so any help is very much appreciated. Thanks! Florin On 8/7/13 12:07 AM, "Florin Trofin" wrote: >An update on this issue: I still can't build the 0.8 branch using Maven. >My automated build system uses Maven, so I need

Re: Kafka/Hadoop consumers and producers

Thanks - I'll ask on bigtop regarding the .deb requirement - it seems they don't abide by this. I may merge a bit of your work into bigtop-989 if that's ok with you. I do know the bigtop folks  would like to see sbt support. From: Andrew Otto To: Kam Kasravi

Re: cluster reboot and leader selection

You can run the preferred replica leader election tool after the restarts (https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools)

cluster reboot and leader selection

Hi, We have three hosts (1, 2, 3) in a kafka cluster. When we restart the cluster, If broker on host 2 restarts first, then most of partitions will use broker 2 as leader. Ideally, we want to see one third topics use broker 1 as leader, other one third topics use broker 2 and the remaining one thi

Re: Kafka/Hadoop consumers and producers

> What installs all the kafka dependencies under /usr/share/java? The debian/ work was done mostly by another WMF staffer. We tried and tried to make sbt behave with debian standards, most importantly the one that requires that .debs can be created without needing to connect to the internet, a

Re: Kafka/Hadoop consumers and producers

Thanks Andrew - I like the shell wrapper - very clean and simple.  What installs all the kafka dependencies under /usr/share/java? From: Andrew Otto To: Kam Kasravi Cc: "d...@kafka.apache.org" ; Ken Goodhope ; Andrew Psaltis ; "dibyendu.bhattacha...@pearson.

Re: Kafka/Hadoop consumers and producers

Thanks Andrew. I upgrade it to use the high level consumer. From: Andrew Psaltis To: "users@kafka.apache.org" ; Kam Kasravi ; "d...@kafka.apache.org" ; Ken Goodhope Cc: Andrew Psaltis ; "dibyendu.bhattacha...@pearson.com" ; "camus_...@googlegroups.com" ;

Re: Kafka/Hadoop consumers and producers

Andrew, I'm about to dive into figuring out how to use Camus without Avro. Perhaps we should join forces? (Be warned thought! My java fu is low at the moment. :) ). -Ao On Aug 12, 2013, at 11:20 PM, Andrew Psaltis wrote: > Kam, > I am perfectly fine if you pick this up. After thinking abo