How to avoid Kafka latency spikes caused by log segment flush

I originally asked on StackOverflow (https://stackoverflow.com/questions/73555649/how-to-avoid-kafka-latency-spikes-caused-by-log-segment-flush) and I'm really trying to get help. Thanks, Jiri

Re: kafka latency for large message

that's very good information from the slides, thanks. Our design to use kafka has 2 purpose. one is use it as a cache, we use ktable for that purpose, second purpose is use as message delivery mechanism to send it to other system. Because we very much care the latency, the ktable with a compact top

Re: kafka latency for large message

Hi Nan, Would you consider other approaches that may actually be a more efficient solution for you? There is a slide deck Handle Large Messages In Apache Kafka . For messages this large, one of the approaches su

Re: kafka latency for large message

It takes time to send that much data over the network. Why would you expect a smaller latency? On Mon, Mar 18, 2019 at 8:05 AM Nan Xu wrote: > anyone can give some suggestion? or an explanation why kafka give a big > latency for large payload. > > Thanks, > Nan > > On Thu, Mar 14, 2019 at 3:53 P

Re: kafka latency for large message

anyone can give some suggestion? or an explanation why kafka give a big latency for large payload. Thanks, Nan On Thu, Mar 14, 2019 at 3:53 PM Xu, Nan wrote: > Hi, > > We are using kafka to send messages and there is less than 1% of > message is very big, close to 30M. understanding kafka i

kafka latency for large message

Hi, We are using kafka to send messages and there is less than 1% of message is very big, close to 30M. understanding kafka is not ideal for sending big messages, because the large message rate is very low, we just want let kafka do it anyway. But still want to get a reasonable latency.

Kafka latency optimization

Hi, everyone My kafka version is 0.10.2.1. My service have really low qps (1msg/sec). And our requirement for rtt is really strict. ( 99.9% < 30ms) Currently I've encounter a problem, when kafka run for a long time, 15 days or so, performance start to go down. 2017-10-21 was like Time .

Re: kafka latency

Hello Chao, How did you measure latency? See also https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines Kind regards, Stevo Slavic. On Tue, Jul 26, 2016 at 9:52 PM, Luo, Chao wrote: > Dear Kafka guys, > > I measured the latency of kafka

kafka latency

Dear Kafka guys, I measured the latency of kafka system. The producer, kafka servers, and consumer were running on different machines (AWS EC2 instances). The producer-to-consumer latency is around 250 milliseconds. Is it a normal value for kafka system. Can I do better and how? Any comments or

Re: Trying to figure out kafka latency issues

14 at 10:22 AM, Rajiv Kurian < > > > > ra...@signalfuse.com> > > > > > > >> wrote: > > > > > > >> > > > > > > >>> Thanks Jay. Will check (1) and (2) and get back to you. The > > test > > > is > > > > >

Re: Trying to figure out kafka latency issues

> > > > > > >>> On Mon, Dec 29, 2014 at 9:45 AM, Jay Kreps > > > wrote: > > > > > >>> > > > > > >>>> Hey Rajiv, > > > > > >>>> > > > > > >>&g

Re: Trying to figure out kafka latency issues

gt; > >>> executable. It might take me a bit of time to get that going. >> > > > >>> >> > > > >>> On Mon, Dec 29, 2014 at 9:45 AM, Jay Kreps >> > wrote: >> > > > >>> >> > > > >>>> Hey Rajiv, >&

Re: Trying to figure out kafka latency issues

gt; 2. It would be worth also getting the jmx stats on the producer > as > > > > they > > > > >>>> will show things like what percentage of time it is waiting for > > > buffer > > > > >>>> space etc. > > > > >>>> >

Re: Trying to figure out kafka latency issues

le a > > > JIRA > > > >>>> and > > > >>>> attach the test code and the findings you already have so someone > > can > > > >>>> dig > > > >>>> into what is going on. > > > >>>> > &g

Re: Trying to figure out kafka latency issues

gt; > > >>>> > Hi all, > > >>>> > > > >>>> > Bumping this up, in case some one has any ideas. I did yet another > > >>>> > experiment where I create 4 producers and stripe the send requests > > >>>

Re: Trying to figure out kafka latency issues

cross > >>>> > them in a manner such that any one producer only sees 256 partitions > >>>> > instead of the entire 1024. This seems to have helped a bit, and > >>>> though I > >>>> > still see crazy high 99th (25-30 seconds), the medi

Re: Trying to figure out kafka latency issues

n, mean, 75th and >>>> 95th >>>> > percentile have all gone down. >>>> > >>>> > Thanks! >>>> > >>>> > On Sun, Dec 21, 2014 at 12:27 PM, Thunder Stumpges < >>>> tstump...@ntent.com> >>>&g

Re: Trying to figure out kafka latency issues

gt;> > On Sun, Dec 21, 2014 at 12:27 PM, Thunder Stumpges < >>> tstump...@ntent.com> >>> > wrote: >>> > >>> > > Ah I thought it was restarting the broker that made things better :) >>> > > >>> > > Yeah I have n

Re: Trying to figure out kafka latency issues

21, 2014 at 12:27 PM, Thunder Stumpges > > >> > wrote: >> > >> > > Ah I thought it was restarting the broker that made things better :) >> > > >> > > Yeah I have no experience with the Java client so can't really help >> > there.

Re: Trying to figure out kafka latency issues

t; > > > Good luck! > > > > > > -Original Message- > > > From: Rajiv Kurian [ra...@signalfuse.com] > > > Received: Sunday, 21 Dec 2014, 12:25PM > > > To: users@kafka.apache.org [users@kafka.apache.org] > > > Subject: Re: Tryin

Re: Trying to figure out kafka latency issues

21 Dec 2014, 12:25PM > > To: users@kafka.apache.org [users@kafka.apache.org] > > Subject: Re: Trying to figure out kafka latency issues > > > > I'll take a look at the GC profile of the brokers Right now I keep a tab > on > > the CPU, Messages in, Bytes in, Bytes

Re: Trying to figure out kafka latency issues

client so can't really help there. > > Good luck! > > -Original Message- > From: Rajiv Kurian [ra...@signalfuse.com] > Received: Sunday, 21 Dec 2014, 12:25PM > To: users@kafka.apache.org [users@kafka.apache.org] > Subject: Re: Trying to figure out kafka latency

RE: Trying to figure out kafka latency issues

e.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab on the CPU, Messages in, Bytes in, Bytes out, free memory (on the machine not JVM heap) free disk space on the broker. I'll need to tak

Re: Trying to figure out kafka latency issues

deas on what jmx values to watch. > > Best, > Thunder > > > -Original Message- > From: Rajiv Kurian [ra...@signalfuse.com] > Received: Saturday, 20 Dec 2014, 10:24PM > To: users@kafka.apache.org [users@kafka.apache.org] > Subject: Re: Trying to figure out kafka latency i

RE: Trying to figure out kafka latency issues

users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues Some more work tells me that the end to end latency numbers vary with the number of partitions I am writing to. I did an experiment, where based on a run time flag I would dynamically select how many of the

Re: Trying to figure out kafka latency issues

Some more work tells me that the end to end latency numbers vary with the number of partitions I am writing to. I did an experiment, where based on a run time flag I would dynamically select how many of the *1024 partitions* I write to. So say I decide I'll write to at most 256 partitions I mod wha

Re: Trying to figure out kafka latency issues

I've done some more measurements. I've also started measuring the latency between when I ask my producer to send a message and when I get an acknowledgement via the callback. Here is my code: // This function is called on every producer once every 30 seconds. public void addLagMarkers(final Histo

Re: Trying to figure out kafka latency issues

On Sat, Dec 20, 2014 at 3:49 PM, Rajiv Kurian wrote: > > I am trying to replace a Thrift peer to peer API with kafka for a > particular work flow. I am finding the 99th percentile latency to be > unacceptable at this time. This entire work load runs in an Amazon VPC. I'd > greatly appreciate it if

RE: Trying to figure out kafka latency issues

ka.apache.org [users@kafka.apache.org] Subject: Trying to figure out kafka latency issues I am trying to replace a Thrift peer to peer API with kafka for a particular work flow. I am finding the 99th percentile latency to be unacceptable at this time. This entire work load runs in an Amazon VPC. I'd greatly

Trying to figure out kafka latency issues

I am trying to replace a Thrift peer to peer API with kafka for a particular work flow. I am finding the 99th percentile latency to be unacceptable at this time. This entire work load runs in an Amazon VPC. I'd greatly appreciate it if some one has any insights on why I am seeing such poor numbers.

Re: Kafka latency measures

There were actually several patches against trunk since 0.8.1.1 that may impact latency however, especially when using acks=-1. So those results in the blog may be a bit better than what you would see in 0.8.1.1. -Jay On Wed, Jun 18, 2014 at 7:58 PM, Supun Kamburugamuva wrote: > My machine con

Re: Kafka latency measures

My machine configuration is not very high. The average one way latency we observe is around 10 ~ 15 ms for 50k messages. The outliers doesn't occur for small messages. For small messages we observe around 6 ms latency. Thanks, Supun.. On Wed, Jun 18, 2014 at 10:18 PM, Neha Narkhede wrote: > wh

Re: Kafka latency measures

what are the latency numbers you observed, avg as well as worst case? Here is a blog that we did recently which should reflect latest performance metrics for latency - http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines On Wed, Jun 18, 2014

Re: Kafka latency measures

I've found this performance test. http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/ This performance test has mentioned about the same issue at the end. Thanks, Supun.. On Wed, Jun 18, 2014 at 12:43 PM, Supun Kamburugamuva wrote: > The spikes happens without any correlati

Re: Kafka latency measures

The spikes happens without any correlation with the log.flush.interval.message. They happen more frequently. I'm using the latest version. I'm sending the messages to Kafka, then there is a message receiver, it sends the same messages back through kafka to original sender. The round trip latency i

Re: Kafka latency measures

Which version of Kafka did you use? When you say latency, do you mean the latency between the producer and consumer? If so, are you using a timestamp within the message to compute this latency? On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill wrote: > Hi, > > do these spikes happen to correlate

Re: Kafka latency measures

Hi, do these spikes happen to correlate with log.flush.interval.messages or log.flush.interval.ms? If so it's the file system sync blockage you are seeing. /Magnus 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva : > Hi, > > We are trying to evaluate Kafka for a real time application. We are sen

Kafka latency measures

Hi, We are trying to evaluate Kafka for a real time application. We are sending 50 Kb messages at a fixed rate. The normal messages have a reasonable latency. But then there are these outliers that takes unpredictable amount of time. This causes the average latency to increase dramatically. We are