Re: Modify in-flight messages

2016-11-02 Thread Eno Thereska
Hi Dominik, Not sure if this is 100% relevant, but since I noticed you saying that you are benchmarking stream processing engines, one way to modify a message would be to use the Kafka Streams library, where you consume a message from a topic, modify it as needed/do some processing, and then pr

Re: Modify in-flight messages

2016-11-01 Thread Dominik Safaric
Dear Michael, Thanks for sharing this information with me. I am aware of the fact that each message has a timestamp indicating either the log append or creation time. But in my case, this is not enough since I want to derive the consumption time of messages. The reason for this is because we

Re: Modify in-flight messages

2016-11-01 Thread Mickael Maison
Hi Dominik, On both the consumer and producer you can use the "interceptor.classes" config to specify classes that intercept and can modify records when they are sent/received. Also as of Kafka 0.10, messages have a timestamp field. See http://kafka.apache.org/0101/javadoc/org/apache/kafka/clients

Modify in-flight messages

2016-11-01 Thread Dominik Safaric
Is it possible to somehow modify the Kafka message payload before being sent to the consumer for consumption? Such as for example adding a timestamp to the current message payload indicating the time of message consumption. Dominik Šafarić