Hi all, I was wondering if someone could list some of the causes which may lead to Kafka delivering the same messages more than once.
We've looked around and we see no errors to notice, yet intermittently, we see messages being delivered more than once. Kafka documentation talks about the below delivery modes: - *At most once*—Messages may be lost but are never redelivered. - *At least once*—Messages are never lost but may be redelivered. - *Exactly once*—this is what people actually want, each message is delivered once and only once. So the default is 'at least once' and that is what we're running with (we don't want to do "at most once" as that appears to yield some potential for message loss). We had not seen duplicated deliveries for a while previously but just started seeing them quite frequently in our test cluster. What are some of the possible causes for this? What are some of the available tools for troubleshooting this issue? What are some of the possible fixes folks have developed or instrumented for this issue? Also, is there an effort underway on Kafka side to provide support for the "exactly once" semantic? That is exactly the semantic we want and we're wondering how that may be achieved. Thanks, - Dmitry
