Kafka Streams Handling uncaught exceptions REPLACE_THREAD
"REPLACE_THREAD - Replaces the thread receiving the exception and processing continues with the same number of configured threads. (Note: this can result in duplicate records depending on the application’s processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)" If an application gets an uncaught exception, then the failed thread will be replaced with another thread and it will continue processing messages, skipping that failed message?
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Hi Yoda, For your question: > If an application gets an uncaught exception, then the failed thread will be replaced with another thread and it will continue processing messages, skipping that failed message? --> Yes, if everything goes well after `replace thread`, you can ignore this failed message. Just one reminder that you should check the failed message to avoid this `uncaught exception` thrown again, because if this happens frequently, it'll impact application performance. Thank you. Luke On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master wrote: > "REPLACE_THREAD - Replaces the thread receiving the exception and > processing continues with the same number of configured threads. (Note: > this can result in duplicate records depending on the application’s > processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)" > > If an application gets an uncaught exception, then the failed thread will > be replaced with another thread and it will continue processing messages, > skipping that failed message? >
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Hi Yoda, What do you mean exactly with "skipping that failed message"? Do you mean a record consumed from a topic that caused an exception that killed the stream thread? If the record killed the stream thread due to an exception, for example, a deserialization exception, it will probably also kill the next stream thread that will read that record. Replacing a stream thread does not skip records but it can result in duplicate records depending on the application’s processing mode determined by the PROCESSING_GUARANTEE_CONFIG value as stated in the docs you cited. Best, Bruno On 10.08.21 11:15, Luke Chen wrote: Hi Yoda, For your question: If an application gets an uncaught exception, then the failed thread will be replaced with another thread and it will continue processing messages, skipping that failed message? --> Yes, if everything goes well after `replace thread`, you can ignore this failed message. Just one reminder that you should check the failed message to avoid this `uncaught exception` thrown again, because if this happens frequently, it'll impact application performance. Thank you. Luke On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master wrote: "REPLACE_THREAD - Replaces the thread receiving the exception and processing continues with the same number of configured threads. (Note: this can result in duplicate records depending on the application’s processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)" If an application gets an uncaught exception, then the failed thread will be replaced with another thread and it will continue processing messages, skipping that failed message?
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Hi Luke, thank you for your answer. I will try it, I think I will set an alert if there are too many messages. To ignore the message should I simply return "replace_thread" in the handler? On Tue, Aug 10, 2021 at 12:16 PM Luke Chen wrote: > Hi Yoda, > For your question: > > If an application gets an uncaught exception, then the failed thread will > be replaced with another thread and it will continue processing messages, > skipping that failed message? > > --> Yes, if everything goes well after `replace thread`, you can ignore > this failed message. Just one reminder that you should check the failed > message to avoid this `uncaught exception` thrown again, because if this > happens frequently, it'll impact application performance. > > Thank you. > Luke > > On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master > wrote: > > > "REPLACE_THREAD - Replaces the thread receiving the exception and > > processing continues with the same number of configured threads. (Note: > > this can result in duplicate records depending on the application’s > > processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)" > > > > If an application gets an uncaught exception, then the failed thread will > > be replaced with another thread and it will continue processing messages, > > skipping that failed message? > > >
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Hi Bruno, thank you for your answer. I mean that the message that caused the exception was consumed and replaced thread will continue from the next message. How then does it handle uncaught exceptions, if it will fail again? On Tue, Aug 10, 2021 at 12:33 PM Bruno Cadonna wrote: > Hi Yoda, > > What do you mean exactly with "skipping that failed message"? > > Do you mean a record consumed from a topic that caused an exception that > killed the stream thread? > > If the record killed the stream thread due to an exception, for example, > a deserialization exception, it will probably also kill the next stream > thread that will read that record. Replacing a stream thread does not > skip records but it can result in duplicate records depending on the > application’s processing mode determined by the > PROCESSING_GUARANTEE_CONFIG value as stated in the docs you cited. > > Best, > Bruno > > > > On 10.08.21 11:15, Luke Chen wrote: > > Hi Yoda, > > For your question: > >> If an application gets an uncaught exception, then the failed thread > will > > be replaced with another thread and it will continue processing messages, > > skipping that failed message? > > > > --> Yes, if everything goes well after `replace thread`, you can ignore > > this failed message. Just one reminder that you should check the failed > > message to avoid this `uncaught exception` thrown again, because if this > > happens frequently, it'll impact application performance. > > > > Thank you. > > Luke > > > > On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master > wrote: > > > >> "REPLACE_THREAD - Replaces the thread receiving the exception and > >> processing continues with the same number of configured threads. (Note: > >> this can result in duplicate records depending on the application’s > >> processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)" > >> > >> If an application gets an uncaught exception, then the failed thread > will > >> be replaced with another thread and it will continue processing > messages, > >> skipping that failed message? > >> > > >
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Hi Yoda, for certain cases, Kafka Streams allows you to specify handlers that skip the problematic record. Those handlers are: 1. deserialization exception handler configured in default.deserialization.exception.handler 2. time extractor set in default.timestamp.extractor and in the Consumed object 3. production exception handler configured in default.production.exception.handler Kafka Streams provides implementations for handlers 1 and 2 to skip the problematic records, that are LogAndContinueExceptionHandler and LogAndSkipOnInvalidTimestamp, respectively. For some more details have a look at https://docs.confluent.io/platform/current/streams/faq.html#failure-and-exception-handling If problematic records cause an exception in user code, the user code needs to provide functionality to skip the problematic record. Best, Bruno On 10.08.21 13:26, Yoda Jedi Master wrote: Hi Bruno, thank you for your answer. I mean that the message that caused the exception was consumed and replaced thread will continue from the next message. How then does it handle uncaught exceptions, if it will fail again? On Tue, Aug 10, 2021 at 12:33 PM Bruno Cadonna wrote: Hi Yoda, What do you mean exactly with "skipping that failed message"? Do you mean a record consumed from a topic that caused an exception that killed the stream thread? If the record killed the stream thread due to an exception, for example, a deserialization exception, it will probably also kill the next stream thread that will read that record. Replacing a stream thread does not skip records but it can result in duplicate records depending on the application’s processing mode determined by the PROCESSING_GUARANTEE_CONFIG value as stated in the docs you cited. Best, Bruno On 10.08.21 11:15, Luke Chen wrote: Hi Yoda, For your question: If an application gets an uncaught exception, then the failed thread will be replaced with another thread and it will continue processing messages, skipping that failed message? --> Yes, if everything goes well after `replace thread`, you can ignore this failed message. Just one reminder that you should check the failed message to avoid this `uncaught exception` thrown again, because if this happens frequently, it'll impact application performance. Thank you. Luke On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master wrote: "REPLACE_THREAD - Replaces the thread receiving the exception and processing continues with the same number of configured threads. (Note: this can result in duplicate records depending on the application’s processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)" If an application gets an uncaught exception, then the failed thread will be replaced with another thread and it will continue processing messages, skipping that failed message?
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Thank you for your help, I will check it and try it :-) On Mon, Aug 16, 2021 at 11:45 AM Bruno Cadonna wrote: > Hi Yoda, > > for certain cases, Kafka Streams allows you to specify handlers that > skip the problematic record. Those handlers are: > > 1. deserialization exception handler configured in > default.deserialization.exception.handler > 2. time extractor set in default.timestamp.extractor and in the Consumed > object > 3. production exception handler configured in > default.production.exception.handler > > Kafka Streams provides implementations for handlers 1 and 2 to skip the > problematic records, that are LogAndContinueExceptionHandler and > LogAndSkipOnInvalidTimestamp, respectively. > > For some more details have a look at > > https://docs.confluent.io/platform/current/streams/faq.html#failure-and-exception-handling > > If problematic records cause an exception in user code, the user code > needs to provide functionality to skip the problematic record. > > Best, > Bruno > > On 10.08.21 13:26, Yoda Jedi Master wrote: > > Hi Bruno, thank you for your answer. > > I mean that the message that caused the exception was consumed and > replaced > > thread will continue from the next message. How then does it handle > > uncaught exceptions, if it will fail again? > > > > > > On Tue, Aug 10, 2021 at 12:33 PM Bruno Cadonna > wrote: > > > >> Hi Yoda, > >> > >> What do you mean exactly with "skipping that failed message"? > >> > >> Do you mean a record consumed from a topic that caused an exception that > >> killed the stream thread? > >> > >> If the record killed the stream thread due to an exception, for example, > >> a deserialization exception, it will probably also kill the next stream > >> thread that will read that record. Replacing a stream thread does not > >> skip records but it can result in duplicate records depending on the > >> application’s processing mode determined by the > >> PROCESSING_GUARANTEE_CONFIG value as stated in the docs you cited. > >> > >> Best, > >> Bruno > >> > >> > >> > >> On 10.08.21 11:15, Luke Chen wrote: > >>> Hi Yoda, > >>> For your question: > If an application gets an uncaught exception, then the failed thread > >> will > >>> be replaced with another thread and it will continue processing > messages, > >>> skipping that failed message? > >>> > >>> --> Yes, if everything goes well after `replace thread`, you can ignore > >>> this failed message. Just one reminder that you should check the failed > >>> message to avoid this `uncaught exception` thrown again, because if > this > >>> happens frequently, it'll impact application performance. > >>> > >>> Thank you. > >>> Luke > >>> > >>> On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master > >> wrote: > >>> > "REPLACE_THREAD - Replaces the thread receiving the exception and > processing continues with the same number of configured threads. > (Note: > this can result in duplicate records depending on the application’s > processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)" > > If an application gets an uncaught exception, then the failed thread > >> will > be replaced with another thread and it will continue processing > >> messages, > skipping that failed message? > > >>> > >> > > >