Thanks Jay. Any idea, when this feature can be expected? Are there any patches for it?
Thanks Arun -----Original Message----- From: Jay Kreps [mailto:jay.kr...@gmail.com] Sent: Tuesday, February 17, 2015 10:16 PM To: users@kafka.apache.org Subject: Re: Producer duplicates There are some details to get this right, the lookup table has to survive failures. But yes this is exactly what we would like to add: https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer -Jay On Tue, Feb 17, 2015 at 12:44 AM, Arunkumar Srambikkal (asrambik) < asram...@cisco.com> wrote: > Hi, > > I guess message production duplicate scenario in Kafka is, when a > producer commits the data but does not get an ack (broker or network > fails AFTER > commit) and retries. > > I got thinking that the retry can be caught by the broker which could > then identify the previous message with a unique message ID before > committing them again but it requires a small lookup table for the > previous commit ID for each producer. > > Can I please know what is wrong about this picture, since Kafka does > not do this? > > Thanks > Arun >
