Thanks Matthias, I would like to start a discussion thread on KIP-1215.
I'm proposing add a new option to ProcessingExceptionHandler where the application before fail, commit all previous records to avoid duplicated records if streams process restart automatically. Please take a look and feel free to share any thoughts. https://cwiki.apache.org/confluence/display/KAFKA/KIP+1215%3A+Graceful+stop+on+Kafka+Streams+exceptions+processing+handlers Thanks, Victor Osório -----Original Message----- From: Matthias J. Sax <[email protected]> Sent: Wednesday, September 3, 2025 10:00 PM To: [email protected] Subject: Re: Graceful Shutdown on Kafka Streams exception handler [You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] CAUTION: This email is from an external source. Please don't open any unknown links or attachments. Hello Victor, thanks for reaching out. I am not sure if it would be easily possible to implement what you propose. The problem is really, that records might be partially processed when an error happens, and before we can figure out what the correct offset for committing is, KS needs to flush, but after we hit an error, flushing cleanly might not be possible any longer. Beside the above issue, w/o EOS (ie, Kafka transactions), there is all kind of other scenarios that could lead to duplicate output, or reprocessing an input record a second time. So even if we can implement what you propose (we would need to investigate in more details if it might be possible or not), it could only reduce duplicates, but not fully eliminate them. Just want to make sure you are aware of this. So if you are really interested to contribute such a feature, and you can figure out how to do this correctly, yes, please write a KIP about it :) -- We can also try to help you with this investigation (it might require some POC PR...); however, not sure how much time we can find atm to support you TBH. However, I am wondering what kind of performance you need, and why EOS would not be able to deliver. There is many configs, so maybe there is a way to tune your app to make it work with EOS? -Matthias On 9/3/25 10:50 AM, Victor Osorio wrote: > Hello everyone, > > We're currently using Kafka Streams to process transactional data with > *exactly-once semantics (EOS)*. However, for some of our workloads, we > require higher throughput, which makes EOS impractical. > > To ensure data integrity, we rely > on UncaughtExceptionHandler and ProductionExceptionHandler to halt > stream processing upon any exception. This prevents data loss but > introduces a new challenge: when a thread stops due to an exception, > it doesn't commit the records that were already successfully > processed. As a result, when the stream restarts, those records are > reprocessed, leading to duplication. > > While reviewing the discussion around KIP-1033, I noticed the > suggestion to avoid exposing commit functionality in the Kafka Streams > API > (https://lis/ > ts.apache.org%2Fthread%2Fk4v0737tqjdnq5vl3yp9rjr4qzqoo306&data=05%7C02 > %7CVictor.Osorio%40amdocs.com%7C49feb9f09642470bfed308ddeb4e6349%7Cc8e > ca3ca127646d59d9da0f2a028920f%7C0%7C0%7C638925444117688366%7CUnknown%7 > CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zM > iIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=uQGZx2cHVjWZ%2F > KZaX4z2sXSprz5d6%2BuoN83XntX%2FT9g%3D&reserved=0 > <https://lists.apache.org/thread/k4v0737tqjdnq5vl3yp9rjr4qzqoo306>). > That makes sense in many contexts, but I'd like to revisit a related idea: > > *Could we introduce a new shutdown mechanism, perhaps a "Graceful > Shutdown" API, that commits all successfully processed records while > skipping the one that caused the failure?* > > This would allow us to maintain data integrity without sacrificing > throughput or introducing duplicates. I'm curious to hear your thoughts: > > * Would this be possible to implement with current Kafka Streams APIs? > * Is that possible, or desired, to be added as a Kafka Streams feature > in further releases? If yes, we can open a KIP. > > Looking forward to your insights and feedback. > > Best regards, > Victor Osório > > amdocs-2017-brand-mark-rgb > > *This email and the information contained herein is proprietary and > confidential and subject to the Amdocs Email Terms of Service, which > you may review > at**https://www.amdocs.com/about/email-terms-of-service* > <https://www.amdocs.com/about/email-terms-of-service> > This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
