Re: Separate broker replication traffic from producer/consumer traffic

2014-03-26 Thread Joris VanRemoortere
Hi Otto, We've separated our traffic for a couple of reasons: 1. We wanted to protect our producer bandwidth to maintain a low latency pipeline 2. We expected consumers to sometimes pick up from an older offset and clog the pipe, causing latency for other services 3. When an out of sync replica co

Re: Messages from producer are immediately going to /tmp/logs in kafka

2013-10-14 Thread Joris VanRemoortere
I also filed this bug + patch ( https://issues.apache.org/jira/browse/KAFKA-1042) where the config variables you mention are not actually enforced within an active segment. It is related, but the reverse of what you are looking for. The flush intervals are not enforced properly; however, the filesy

Re: How to force producer and high level consumer to use different Ethernet cards on the kafka broker node

2014-01-22 Thread Joris VanRemoortere
Sorry for the confusion. We implemented this by re-mapping the hostname to the ip on eth1 on our consumers; zookeeper can keep publishing the same hostname while different clients (producers / brokers / consumers) can map that hostname to different physical interfaces. This allows us to have separa

Re: Jira issues

2014-01-23 Thread Joris VanRemoortere
Hello, Last evening Jira was having all kinds of "issues" and errors. It seems while I was trying to create an issue for rack-aware replication, I created issues 1215-1225. Can you please delete all the duplicates? I will attach my patch to 1215. Sorry for the inconvenience, Joris

Re: Jira issues

2014-01-23 Thread Joris VanRemoortere
ta Open Source Security LLC > > http://www.stealth.ly > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > / > > > > > > On Thu, Jan 23, 2014 at 12:25 PM, Joris VanRemoortere < > > jvanremoort...@tagged.com> wrote: > > > &