It also seems you are using "at least one" strategy (maybe with auto-commit, or commiting after sending the email) Maybe a "at most once" could be a valid business strategy here ?
- at least once (you will deliver all the emails, but you could deliver duplicates) consumeMessages sendEmails commitOffsets - at most once (you will never deliver duplicates, but you might never deliver an given email) consumeMessages commitOffsets sendEmails Ideally, you could do "exactly once", but it is hard to achieve in the scenario, Kafka -> External system. The usual strategy here is to have an idempotent operation in combination with a "at least once" strategy Best, Vincent On Fri, May 24, 2019 at 10:39 AM Liam Clarke <[email protected]> wrote: > Consumers will rebalance if you add partitions, add consumers to the group > or if a consumer leaves the group. > > Consumers will leave the group after not communicating with the server for > a period set by session.timeout.ms. This is usually due to an exception in > the code polling with the consumer, or message processing code taking too > long. > > If your consumers are reprocessing messages thus causing emails to send, it > implies that they weren't able to commit their offsets before > failing/timing out. > > We had a similar issue in a database sink that consumed from Kafka and > duplicated data because it took too long, and hit the session timeout, and > then wasn't able to commits its offsets. > > So I'd look closely at your consuming code and log every possible source of > exceptions. > > Kind regards, > > Liam Clarke > > On Fri, 24 May 2019, 7:37 pm ASHOK MACHERLA, <[email protected]> wrote: > > > Dear Team Member > > > > Currently we are using Kafka 0.10.1, zookeeper 3.4.6 versions. In our > > project we have to send bulk emails to customers for this purpose we are > > using Kafka cluster setup. > > > > But customers are getting same emails for roughly 30-40 times. This is > > very worst thing. In this situation our consumer group is showing > > rebalancing. Might be its could be reason ????? > > Currently one topic we are using for this. We have 10 partitions and 10 > > consumers. > > I hope we have enough partitions and consumer as well. > > But I don’t know exactly number of partitions & consumer are required to > > overcome this issue. > > > > Can you please suggest us to fix this issue. > > > > If anything changes required in Kafka side as well as consumer side?? > > How to stop rebalancing issue?????? > > Please suggest us, Thanks > > > > > > > > Sent from Outlook. > > > > >
