[heka] Slow throughput using Kafka as Input

2015-05-05 Thread Mathieu Chouquet-Stringer
Hello, I'm facing a somewhat similar issue than Antonin described in https://mail.mozilla.org/pipermail/heka/2015-April/000522.html where Heka is slow getting stuff from Kafka (around 1,600 messages/s). I have setup a Kafka instance where I dump a truck load of log messages. Without tuni

Re: [heka] Slow throughput using Kafka as Input

2015-05-05 Thread Antonin Kral
Hi Mathieu, glad I am not alone :) * Mathieu Chouquet-Stringer [2015-05-05 15:15] wrote: > Wait, I just discovered something, if I switch the offset_method to > oldest, I get around 5,000 messages/s instead of 1,600? Very in-line with what I have seen as well. I have observer quite a lot of IO

Re: [heka] Slow throughput using Kafka as Input

2015-05-05 Thread Michael Trinkala
Yeah, we should address that: https://github.com/mozilla-services/heka/issues/1514 On Tue, May 5, 2015 at 9:48 AM, Antonin Kral wrote: > Hi Mathieu, > > glad I am not alone :) > > * Mathieu Chouquet-Stringer [2015-05-05 15:15] wrote: > > Wait, I just discovered something, if I switch the offset