Hello- We receive events from external source ( for example, facebook status update events). These events are pushed to kafka queue when received There is possibility of duplicate event ( multiple facebook status update events for same account in quick intervals ) coming again and gets pushed into kafka queue. At consumer end, we do not want to process duplicate events ( connect to facebook and fetch the status). We would prefer not to have another data structure to single instance the events received. There is time limit in which whatever events received, we want to do single instancing ( at once fetch all the status update events received in five minutes for single account ). In async producer, events are anyways not written to broker synchronously. They are batched and then pushed after predefined time interval. That's perfect for us. We just want to look at the batch and delete duplicate events from it and then push it broker.
Possible? thanks On Wed, Jul 3, 2013 at 3:07 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > Callback handlers are no longer supported in 0.8. Can you go into why > the filtering needs to be done at this stage as opposed to before > actually sending to the producer? > > Thanks, > > Joel > > On Tue, Jul 2, 2013 at 10:41 AM, Nitin Supekar <ni...@ecinity.com> wrote: > > Hello- > > > > Is CallbackHandler supported in Kafka 0.8 for async producers? > > > > If yes, can I use it to alter the batched messages before they are pushed > > to broker? For example, I may want to delete some of the messages in the > > batch based on some business logic in my application? > > > > If no, is there any alternate way? I want to do some kind of single > > instancing on messages pushed in kafka in last X minutes. > > > > thanks >
