Hi All, I'm looking for help regarding latency monitoring. Let's say I have a simple streaming data flow with the following operators: FlinkKafkaConsumer -> Map -> print.
In case I want to measure a latency of records processing in my dataflow, what would be the best opportunity? I want to get the duration of processing input received in the source until it received by the sink/finished sink operation. I've added my code: env.getConfig().setLatencyTrackingInterval(100); And then, the following latency metrics are available: [cid:image001.png@01D521EA.9F2873F0] But I don't understand what exactly they are measuring? Also latency avg values are not seem to be related to latency as I see it. I've tried also to use codahale metrics to get duration of some methods but it's not helping me to get a latency of record that processed in my whole pipeline. Is the solution related to LatencyMarker? If yes, how can I reach it in my sink operation in order to retrieve it? Thanks, Roey.
