Hi WIlliam, I see from your example that you close the kafka producer in the send loop, based on the content of sendException that is used in the callback of the KafkaProducer send. Since your send loop is a different thread than the KafkaProducer uses to send you will encounter race conditions on this close logic.
I actually had a similar requirement as yours and solved it by using a sendException like you do, but I close the KafkaProducer inside the send callback. The send callback is executed as part of the produce thread, and closing the consumer there will stop all subsequent batches of processing, as the current batch isn't finished yet. Combined with idempotence enabled and max inflight set to 5 (the maximum for idempotence tracking) it gave me relatively good performance. Kind regards, Richard Bosch Developer Advocate Axual BV https://axual.com/
