Latency Monitoring in Flink application

2019-06-13 Thread Halfon, Roey
Hi All, I'm looking for help regarding latency monitoring. Let's say I have a simple streaming data flow with the following operators: FlinkKafkaConsumer -> Map -> print. In case I want to measure a latency of records processing in my dataflow, what would be the best opportunity? I want to get t

Re: Latency Monitoring in Flink application

2019-06-13 Thread Konstantin Knauf
Hi Roey, with Latency Tracking you will get a distribution of the time it took for LatencyMarkers to travel from each source operator to each downstream operator (per default one histogram per source operator in each non-source operator, see metrics.latency.granularity). LatencyMarkers are inject

Re: Latency Monitoring in Flink application

2019-06-13 Thread Timothy Victor
Thanks for the insight. I was also interested in this topic. One thought occurred to me is what about the queuing delay when sending to your message bus (e.g. kafka). I am guessing the probe will be before the message is added to the send queue? Thanks again Tim On Thu, Jun 13, 2019, 6:08 AM