Below are some examples. They might be old, but the concept is the same - essentially you are implementing an interface. You have to be conscious of not to do heavy or high latency processing to be non-impactful to the data flow.
https://www.ashishpaliwal.com/blog/2013/06/flume-cookbook-implementing-custom-interceptors/ https://medium.com/@bkvarda/building-a-custom-flume-interceptor-8c7a55070038 From: antonio saldivar <[email protected]> Date: Friday, July 20, 2018 at 9:57 AM To: "Thakrar, Jayesh" <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: Measure latency from Source to Sink Hi Actually I am running the app I a single node this is for a POC, I was not aware of the custom interceptors do you have an example of this? Best Regards El vie., 20 jul. 2018 a las 9:20, Thakrar, Jayesh (<[email protected]<mailto:[email protected]>>) escribió: See if you can use a custom interceptors for this. The only fuzzy thing is that the clocks would be different so I would be a little skeptical of its accuracy. I have heard of some companies who have a special topic in which they insert test msgs and then read them back - using the same machine for producer and consumer, thereby doing away with this dilemma. The assumption is that any delay in the cluster would be felt everywhere. Another option is to create topics with one replica on all brokers (i.e. # of topics = # of brokers) - so you can be sensitive to a single broker having an issue. On 7/19/18, 3:09 PM, "antonio saldivar" <[email protected]<mailto:[email protected]>> wrote: Hello I am developing an application using kafka and flink, I need to be able to measure the latency from the producer and when it comes out to the sink. I can append the timestamp in Millisecond when I send the trxn from the producer but at the end how to append the timestamp when it comes out the sink. Someone can help me with an example Thank you Best Regards
