what are the latency numbers you observed, avg as well as worst case? Here is a blog that we did recently which should reflect latest performance metrics for latency - http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
On Wed, Jun 18, 2014 at 11:01 AM, Supun Kamburugamuva <supu...@gmail.com> wrote: > I've found this performance test. > > http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/ > > This performance test has mentioned about the same issue at the end. > > Thanks, > Supun.. > > > On Wed, Jun 18, 2014 at 12:43 PM, Supun Kamburugamuva <supu...@gmail.com> > wrote: > > > The spikes happens without any correlation with the > log.flush.interval.message. > > They happen more frequently. > > > > I'm using the latest version. I'm sending the messages to Kafka, then > > there is a message receiver, it sends the same messages back through > kafka > > to original sender. The round trip latency is measured. > > > > Thanks, > > Supun.. > > > > > > On Wed, Jun 18, 2014 at 12:02 PM, Neha Narkhede <neha.narkh...@gmail.com > > > > wrote: > > > >> Which version of Kafka did you use? When you say latency, do you mean > the > >> latency between the producer and consumer? If so, are you using a > >> timestamp > >> within the message to compute this latency? > >> > >> > >> On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill <mag...@edenhill.se> > >> wrote: > >> > >> > Hi, > >> > > >> > do these spikes happen to correlate with log.flush.interval.messages > or > >> > log.flush.interval.ms? > >> > If so it's the file system sync blockage you are seeing. > >> > > >> > /Magnus > >> > > >> > > >> > 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva <supu...@gmail.com>: > >> > > >> > > Hi, > >> > > > >> > > We are trying to evaluate Kafka for a real time application. We are > >> > sending > >> > > 50 Kb messages at a fixed rate. The normal messages have a > reasonable > >> > > latency. But then there are these outliers that takes unpredictable > >> > amount > >> > > of time. This causes the average latency to increase dramatically. > We > >> are > >> > > running with basically the default configuration. Any suggestions > for > >> > > improving the latency? > >> > > > >> > > Thanks in advance, > >> > > Supun.. > >> > > > >> > > -- > >> > > Supun Kamburugamuva > >> > > Member, Apache Software Foundation; http://www.apache.org > >> > > E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 > >> > > Blog: http://supunk.blogspot.com > >> > > > >> > > >> > > > > > > > > -- > > Supun Kamburugamuva > > Member, Apache Software Foundation; http://www.apache.org > > E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 > > Blog: http://supunk.blogspot.com > > > > > > > -- > Supun Kamburugamuva > Member, Apache Software Foundation; http://www.apache.org > E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 > Blog: http://supunk.blogspot.com >
