Re: Measure End-to-End latency/delay for each record

Ok thanks Michael for all your help! -- Dhruv Kumar PhD Candidate Department of Computer Science and Engineering University of Minnesota www.dhruvkumar.me > On Apr 26, 2018, at 19:24, TechnoMage wrote: > > Yes, Kafka for

Re: Measure End-to-End latency/delay for each record

Yes, Kafka for source and sink which makes monitoring the Flink in/out easy. Michael > On Apr 26, 2018, at 5:27 PM, Dhruv Kumar wrote: > > Ok that answers my questions. > > What are you keeping the source and sink as? Is it Kafka for both? > >

Re: Measure End-to-End latency/delay for each record

Ok that answers my questions. What are you keeping the source and sink as? Is it Kafka for both? -- Dhruv Kumar PhD Candidate Department of Computer Science and Engineering University of Minnesota www.dhruvkumar.me > On Apr 26, 2018, at 16:37,

Re: Measure End-to-End latency/delay for each record

Yes NTP can still have skew. It may be measured in fractions of a second, but with Flink that can be significant if you care about sub-second latency accuracy. Since I have a 20 stage stream with 0.002 second latency it can matter. Back pressure is the limiting of input due to the inability

Re: Measure End-to-End latency/delay for each record

What do you mean by the time skew from one machine(source) to another(sink)? Do you mean the system time clocks of the source and sink may not be in sync. If I regularly use NTP to keep the system clocks in sync, will time skew still happen? Could you also elaborate on what do you mean by back

Re: Measure End-to-End latency/delay for each record

In a single machine system this may work ok. In a multi-machine system this is not as reliable as the time skew from one machine (source) to another (sink) can impact the measurements. This also does not account for back presure on the source. We are using an external process to in parallel

Measure End-to-End latency/delay for each record

Hi I was trying to compute the end-to-end-latency for each record processed by Flink. By end-to-end latency, I mean the difference between the time at which the record entered the Flink system (came at source) and the time at which the record is finally emitted into the sink. What is the best