Re: can producers(from same system) send messages to separate broker systems?

Hi Neha No i haven't experienced any noticeable latency as of now, the high priority data is too critical for any sort of latency, that's why i wanted to optimize everything before deployment. I'm using 0.7.2 since my consumer is a storm spout, and i read that storm is most compatible with

Re: can producers(from same system) send messages to separate broker systems?

The feature I mentioned is only available on 0.8. In 0.7.2, you can tweak producer batch size and the flush interval on the broker for the high priority topics. Note that setting those too low will have performance implications. Thanks, Neha On May 17, 2013 2:25 PM, Neha Narkhede

Re: What happens if one broker goes down

0.7.2 does not support replication. So when a broker goes down, there can be some data loss. If you are ok with duplicates, you can configure the producer side retries to be higher. Thanks, Neha On May 19, 2013 11:32 PM, Chitra Raveendran chitra.raveend...@fluturasolutions.com wrote: HI When

Re: hi plz reple me

You can do that using Kafka. Please read the design details here - http://kafka.apache.org/07/design.html Thanks, Neha On May 20, 2013 6:57 AM, satya prakash satyacusa...@gmail.com wrote: i am using kafka .i need to create one message on producer side and send to multiple

RE: are commitOffsets botched to zookeeper?

Hi Neha, Is moving to zookeeper 3.4.x is a big change ?. Can you please explain parts it affects consumer API for example ?. Thanks, Balaji -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Friday, May 17, 2013 7:35 AM To: users@kafka.apache.org Subject:

RE: are commitOffsets botched to zookeeper?

Zookeeper 3.4.x is API compatible. However to get full benefits, we will have to change kafka code to use the batch API that zookeeper 3.4.x provides. Also, we use zkclient library to interface with zookeeper. We might have to patch that to use zookeeper 3.4.x APIs. Thanks, Neha On May 20, 2013

Re: Relationship between Zookeeper and Kafka

My guess, EBS is likely your bottleneck. Try running on instance local disks, and compare your results. Is this 0.8? What replication factor are you using? On Mon, May 20, 2013 at 8:11 AM, Jason Weiss jason_we...@rapid7.com wrote: I'm trying to maximize my throughput and seem to have hit a

Re: Relationship between Zookeeper and Kafka

Ahh, yeah, piops is definitely faster than standard EBS, but still much slower than local disk. you could try benchmarking local disk to see what the instances you are using are capable of, then try tweaking iops etc to see where you get. M1.Larges arent super fast so your macbook beating them

Re: Update: RE: are commitOffsets botched to zookeeper?

Did so. The proposal looks perfectly sensible on first reading. I understand that the patches in https://issues.apache.org/jira/browse/KAFKA-657 are already in the trunk and scheduled for 0.8.1? Are they going out with 0.8? If not, what's ETA for 0.8.1? Either way, I'm going to try my hand at

Re: Relationship between Zookeeper and Kafka

Hi Jason, On May 20, 2013, at 10:01am, Jason Weiss wrote: Hi Scott. I'm using Kafka 0.7.2. I am using the default replication factor, since I don't recall changing that configuration at all. I'm using provisioned IOPS, which from attending the AWS event in NYC a few weeks ago was

About Kafka Users Group around Hadoop Summit

Jun and Neha, Is there any plan for Kafka Users group meeting around Hadoop Summit? It was done last year. It really works well for people like me who don't live in SF Bay Area. A session on 0.8 would be great. Regards, Vaibhav Puranik GumGum

Re: About Kafka Users Group around Hadoop Summit

Great idea Vaibhav! I would also be interested in this as I live in Denver and don't get to the Bay area too often. -Jonathan On Mon, May 20, 2013 at 2:35 PM, Vaibhav Puranik vpura...@gmail.com wrote: Jun and Neha, Is there any plan for Kafka Users group meeting around Hadoop Summit? It

RE: Update: RE: are commitOffsets botched to zookeeper?

Yes, it looks spot on. Thanks, rob -Original Message- From: Alex Zuzin [mailto:carna...@gmail.com] Sent: Monday, May 20, 2013 11:37 AM To: users@kafka.apache.org Subject: Re: Update: RE: are commitOffsets botched to zookeeper? Did so. The proposal looks perfectly sensible on

RE: only-once consumer groups

That page is packed full of super design! Many of these features we would find useful, I think. One thing I found myself not knowing is what a consumer rebalance actually is. Is a rebalance when the thread that is consuming a particular partition dies, this is detected and the partition is

RE: only-once consumer groups

Rob, A consumer rebalances whenever a consumer process dies or a new consumer process joins the group. The details of the algorithm can be found here http://kafka.apache.org/07/design.html Thanks, Neha On May 20, 2013 6:45 PM, Rob Withers reefed...@gmail.com wrote: That page is packed full of

RE: only-once consumer groups

Right, Neha, consumer groups are from 7, while replicas are in 8. Does this mean the simple consumer in 8 can recognize a leader change? Thanks, rob -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Monday, May 20, 2013 8:00 PM To: users@kafka.apache.org

RE: only-once consumer groups

Sorry for being unclear Neha. I meant that I had forgotten that the introduction of replicas is happening in 0.8 and I was confusing the two. Thanks, rob -Original Message- From: Rob Withers [mailto:reefed...@gmail.com] Sent: Monday, May 20, 2013 8:21 PM To:

Re: About Kafka Users Group around Hadoop Summit

Yes, we can have a Kafka user group meeting then. We could do this in one of the evenings (Tue, Wed, or Thu). What will people prefer? Also, there will be a Kafka talk in Hadoop summit too. Thanks, Jun On Mon, May 20, 2013 at 1:35 PM, Vaibhav Puranik vpura...@gmail.com wrote: Jun and Neha,