Re: Kafka latency measures
There were actually several patches against trunk since 0.8.1.1 that may impact latency however, especially when using acks=-1. So those results in the blog may be a bit better than what you would see in 0.8.1.1. -Jay On Wed, Jun 18, 2014 at 7:58 PM, Supun Kamburugamuva supu...@gmail.com wrote: My machine configuration is not very high. The average one way latency we observe is around 10 ~ 15 ms for 50k messages. The outliers doesn't occur for small messages. For small messages we observe around 6 ms latency. Thanks, Supun.. On Wed, Jun 18, 2014 at 10:18 PM, Neha Narkhede neha.narkh...@gmail.com wrote: what are the latency numbers you observed, avg as well as worst case? Here is a blog that we did recently which should reflect latest performance metrics for latency - http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines On Wed, Jun 18, 2014 at 11:01 AM, Supun Kamburugamuva supu...@gmail.com wrote: I've found this performance test. http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/ This performance test has mentioned about the same issue at the end. Thanks, Supun.. On Wed, Jun 18, 2014 at 12:43 PM, Supun Kamburugamuva supu...@gmail.com wrote: The spikes happens without any correlation with the log.flush.interval.message. They happen more frequently. I'm using the latest version. I'm sending the messages to Kafka, then there is a message receiver, it sends the same messages back through kafka to original sender. The round trip latency is measured. Thanks, Supun.. On Wed, Jun 18, 2014 at 12:02 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Which version of Kafka did you use? When you say latency, do you mean the latency between the producer and consumer? If so, are you using a timestamp within the message to compute this latency? On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill mag...@edenhill.se wrote: Hi, do these spikes happen to correlate with log.flush.interval.messages or log.flush.interval.ms? If so it's the file system sync blockage you are seeing. /Magnus 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva supu...@gmail.com : Hi, We are trying to evaluate Kafka for a real time application. We are sending 50 Kb messages at a fixed rate. The normal messages have a reasonable latency. But then there are these outliers that takes unpredictable amount of time. This causes the average latency to increase dramatically. We are running with basically the default configuration. Any suggestions for improving the latency? Thanks in advance, Supun.. -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com
Re: Kafka latency measures
Hi, do these spikes happen to correlate with log.flush.interval.messages or log.flush.interval.ms? If so it's the file system sync blockage you are seeing. /Magnus 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva supu...@gmail.com: Hi, We are trying to evaluate Kafka for a real time application. We are sending 50 Kb messages at a fixed rate. The normal messages have a reasonable latency. But then there are these outliers that takes unpredictable amount of time. This causes the average latency to increase dramatically. We are running with basically the default configuration. Any suggestions for improving the latency? Thanks in advance, Supun.. -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com
Re: Kafka latency measures
Which version of Kafka did you use? When you say latency, do you mean the latency between the producer and consumer? If so, are you using a timestamp within the message to compute this latency? On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill mag...@edenhill.se wrote: Hi, do these spikes happen to correlate with log.flush.interval.messages or log.flush.interval.ms? If so it's the file system sync blockage you are seeing. /Magnus 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva supu...@gmail.com: Hi, We are trying to evaluate Kafka for a real time application. We are sending 50 Kb messages at a fixed rate. The normal messages have a reasonable latency. But then there are these outliers that takes unpredictable amount of time. This causes the average latency to increase dramatically. We are running with basically the default configuration. Any suggestions for improving the latency? Thanks in advance, Supun.. -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com
Re: Kafka latency measures
The spikes happens without any correlation with the log.flush.interval.message. They happen more frequently. I'm using the latest version. I'm sending the messages to Kafka, then there is a message receiver, it sends the same messages back through kafka to original sender. The round trip latency is measured. Thanks, Supun.. On Wed, Jun 18, 2014 at 12:02 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Which version of Kafka did you use? When you say latency, do you mean the latency between the producer and consumer? If so, are you using a timestamp within the message to compute this latency? On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill mag...@edenhill.se wrote: Hi, do these spikes happen to correlate with log.flush.interval.messages or log.flush.interval.ms? If so it's the file system sync blockage you are seeing. /Magnus 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva supu...@gmail.com: Hi, We are trying to evaluate Kafka for a real time application. We are sending 50 Kb messages at a fixed rate. The normal messages have a reasonable latency. But then there are these outliers that takes unpredictable amount of time. This causes the average latency to increase dramatically. We are running with basically the default configuration. Any suggestions for improving the latency? Thanks in advance, Supun.. -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com
Re: Kafka latency measures
what are the latency numbers you observed, avg as well as worst case? Here is a blog that we did recently which should reflect latest performance metrics for latency - http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines On Wed, Jun 18, 2014 at 11:01 AM, Supun Kamburugamuva supu...@gmail.com wrote: I've found this performance test. http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/ This performance test has mentioned about the same issue at the end. Thanks, Supun.. On Wed, Jun 18, 2014 at 12:43 PM, Supun Kamburugamuva supu...@gmail.com wrote: The spikes happens without any correlation with the log.flush.interval.message. They happen more frequently. I'm using the latest version. I'm sending the messages to Kafka, then there is a message receiver, it sends the same messages back through kafka to original sender. The round trip latency is measured. Thanks, Supun.. On Wed, Jun 18, 2014 at 12:02 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Which version of Kafka did you use? When you say latency, do you mean the latency between the producer and consumer? If so, are you using a timestamp within the message to compute this latency? On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill mag...@edenhill.se wrote: Hi, do these spikes happen to correlate with log.flush.interval.messages or log.flush.interval.ms? If so it's the file system sync blockage you are seeing. /Magnus 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva supu...@gmail.com: Hi, We are trying to evaluate Kafka for a real time application. We are sending 50 Kb messages at a fixed rate. The normal messages have a reasonable latency. But then there are these outliers that takes unpredictable amount of time. This causes the average latency to increase dramatically. We are running with basically the default configuration. Any suggestions for improving the latency? Thanks in advance, Supun.. -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com
Re: Kafka latency measures
My machine configuration is not very high. The average one way latency we observe is around 10 ~ 15 ms for 50k messages. The outliers doesn't occur for small messages. For small messages we observe around 6 ms latency. Thanks, Supun.. On Wed, Jun 18, 2014 at 10:18 PM, Neha Narkhede neha.narkh...@gmail.com wrote: what are the latency numbers you observed, avg as well as worst case? Here is a blog that we did recently which should reflect latest performance metrics for latency - http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines On Wed, Jun 18, 2014 at 11:01 AM, Supun Kamburugamuva supu...@gmail.com wrote: I've found this performance test. http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/ This performance test has mentioned about the same issue at the end. Thanks, Supun.. On Wed, Jun 18, 2014 at 12:43 PM, Supun Kamburugamuva supu...@gmail.com wrote: The spikes happens without any correlation with the log.flush.interval.message. They happen more frequently. I'm using the latest version. I'm sending the messages to Kafka, then there is a message receiver, it sends the same messages back through kafka to original sender. The round trip latency is measured. Thanks, Supun.. On Wed, Jun 18, 2014 at 12:02 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Which version of Kafka did you use? When you say latency, do you mean the latency between the producer and consumer? If so, are you using a timestamp within the message to compute this latency? On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill mag...@edenhill.se wrote: Hi, do these spikes happen to correlate with log.flush.interval.messages or log.flush.interval.ms? If so it's the file system sync blockage you are seeing. /Magnus 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva supu...@gmail.com: Hi, We are trying to evaluate Kafka for a real time application. We are sending 50 Kb messages at a fixed rate. The normal messages have a reasonable latency. But then there are these outliers that takes unpredictable amount of time. This causes the average latency to increase dramatically. We are running with basically the default configuration. Any suggestions for improving the latency? Thanks in advance, Supun.. -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com
