There was a bug that could lead to the fetch request from the consumer hitting it's timeout instead of being immediately triggered by the produce request. To see if you are effected by that set you consumer max wait time to 1 ms and see if the latency drops to 1 ms (or, alternately, try with trunk and see if that fixes the problem).
The reason I suspect this problem is because the default timeout in the java consumer is 100ms. -Jay On Tue, Oct 21, 2014 at 11:06 AM, Mohit Anchlia <mohitanch...@gmail.com> wrote: > This is the version I am using: kafka_2.10-0.8.1.1 > > I think this is fairly recent version > On Tue, Oct 21, 2014 at 10:57 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > What version of Kafka is this? Can you try the same test against trunk? > We > > fixed a couple of latency related bugs which may be the cause. > > > > -Jay > > > > On Tue, Oct 21, 2014 at 10:50 AM, Mohit Anchlia <mohitanch...@gmail.com> > > wrote: > > > > > It's consistently close to 100ms which makes me believe that there are > > some > > > settings that I might have to tweak, however, I am not sure how to > > confirm > > > that assumption :) > > > On Tue, Oct 21, 2014 at 8:53 AM, Mohit Anchlia <mohitanch...@gmail.com > > > > > wrote: > > > > > > > I have a java test that produces messages and then consumer consumers > > it. > > > > Consumers are active all the time. There is 1 consumer for 1 > producer. > > I > > > am > > > > measuring the time between the message is successfully written to the > > > queue > > > > and the time consumer picks it up. > > > > > > > > On Tue, Oct 21, 2014 at 8:32 AM, Neha Narkhede < > > neha.narkh...@gmail.com> > > > > wrote: > > > > > > > >> Can you give more information about the performance test? Which > test? > > > >> Which > > > >> queue? How did you measure the dequeue latency. > > > >> > > > >> On Mon, Oct 20, 2014 at 5:09 PM, Mohit Anchlia < > > mohitanch...@gmail.com> > > > >> wrote: > > > >> > > > >> > I am running a performance test and from what I am seeing is that > > > >> messages > > > >> > are taking about 100ms to pop from the queue itself and hence > making > > > the > > > >> > test slow. I am looking for pointers of how I can troubleshoot > this > > > >> issue. > > > >> > > > > >> > There seems to be plenty of CPU and IO available. I am running 22 > > > >> producers > > > >> > and 22 consumers in the same group. > > > >> > > > > >> > > > > > > > > > > > > > >
