Re: Understanding default.deserialization.exception.handler
Thanks for reporting! It might have been an unknown issue. The community relies on reports like this, so we really appreciate that you reached out! -Matthias On 9/19/18 1:08 AM, Tim Ward wrote: > Thanks. No, it's not that big a deal now that I understand it, but as I'd had > to spend a fair amount of time working out what was going on I thought I'd > flag it up in case it wasn't known about. > > Tim Ward > > -Original Message- > From: Matthias J. Sax > Sent: 14 September 2018 19:57 > To: users@kafka.apache.org > Subject: Re: Understanding default.deserialization.exception.handler > > Your observation is correct. It's a known bug: > https://issues.apache.org/jira/browse/KAFKA-6502 > > In practice, it should not be a big issue though. > - you would only hit this bug if you don't process a "good message" > afterwards > - even if you hit this bug, you would just skip the message again > > Thus, the only drawback I see is an additional log message. As long as > you don't have many _consecutive_ corrupted messages it should be impact > you much. > > Hope this helps. > > > -Matthias > > On 9/13/18 6:09 AM, Tim Ward wrote: >> With >> >> >> props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, >> LogAndContinueExceptionHandler.class); >> >> Scenario A: >> >> Run application. Feed a message into the topic that will fail >> deserialization. >> Application logs exception and keeps running. >> Shut down application. Restart application. >> Application re-reads broken message, logs exception again (and keeps >> running). >> >> Scenario B: >> >> Run application. Feed a message into the topic that will fail >> deserialization. >> Application logs exception and keeps running. >> Feed a good message into deserialization. >> Application processes it normally. >> Shut down application. Restart application. >> Application does *not* re-reads broken message. >> >> So it looks like LogAndContinueExceptionHandler does not seem to commit() >> the incoming "poison pill" message(s), and these will be re-read if the >> application is restarted without any good messages having been read after >> the bad ones. >> >> Have I understood this correctly? If so, is this correct behaviour as >> designed? Is it documented to that level of detail? >> >> Tim Ward >> >> The contents of this email and any attachment are confidential to the >> intended recipient(s). If you are not an intended recipient: (i) do not use, >> disclose, distribute, copy or publish this email or its contents; (ii) >> please contact the sender immediately; and (iii) delete this email. Our >> privacy policy is available here: https://origamienergy.com/privacy-policy/. >> Origami Energy Limited (company number 8619644); Origami Storage Limited >> (company number 10436515) and OSSPV001 Limited (company number 10933403), >> each registered in England and each with a registered office at: Ashcombe >> Court, Woolsack Way, Godalming, GU7 1LQ. >> > > The contents of this email and any attachment are confidential to the > intended recipient(s). If you are not an intended recipient: (i) do not use, > disclose, distribute, copy or publish this email or its contents; (ii) please > contact the sender immediately; and (iii) delete this email. Our privacy > policy is available here: https://origamienergy.com/privacy-policy/. Origami > Energy Limited (company number 8619644); Origami Storage Limited (company > number 10436515) and OSSPV001 Limited (company number 10933403), each > registered in England and each with a registered office at: Ashcombe Court, > Woolsack Way, Godalming, GU7 1LQ. > signature.asc Description: OpenPGP digital signature
RE: Understanding default.deserialization.exception.handler
Thanks. No, it's not that big a deal now that I understand it, but as I'd had to spend a fair amount of time working out what was going on I thought I'd flag it up in case it wasn't known about. Tim Ward -Original Message- From: Matthias J. Sax Sent: 14 September 2018 19:57 To: users@kafka.apache.org Subject: Re: Understanding default.deserialization.exception.handler Your observation is correct. It's a known bug: https://issues.apache.org/jira/browse/KAFKA-6502 In practice, it should not be a big issue though. - you would only hit this bug if you don't process a "good message" afterwards - even if you hit this bug, you would just skip the message again Thus, the only drawback I see is an additional log message. As long as you don't have many _consecutive_ corrupted messages it should be impact you much. Hope this helps. -Matthias On 9/13/18 6:09 AM, Tim Ward wrote: > With > > > props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, > LogAndContinueExceptionHandler.class); > > Scenario A: > > Run application. Feed a message into the topic that will fail deserialization. > Application logs exception and keeps running. > Shut down application. Restart application. > Application re-reads broken message, logs exception again (and keeps running). > > Scenario B: > > Run application. Feed a message into the topic that will fail deserialization. > Application logs exception and keeps running. > Feed a good message into deserialization. > Application processes it normally. > Shut down application. Restart application. > Application does *not* re-reads broken message. > > So it looks like LogAndContinueExceptionHandler does not seem to commit() the > incoming "poison pill" message(s), and these will be re-read if the > application is restarted without any good messages having been read after the > bad ones. > > Have I understood this correctly? If so, is this correct behaviour as > designed? Is it documented to that level of detail? > > Tim Ward > > The contents of this email and any attachment are confidential to the > intended recipient(s). If you are not an intended recipient: (i) do not use, > disclose, distribute, copy or publish this email or its contents; (ii) please > contact the sender immediately; and (iii) delete this email. Our privacy > policy is available here: https://origamienergy.com/privacy-policy/. Origami > Energy Limited (company number 8619644); Origami Storage Limited (company > number 10436515) and OSSPV001 Limited (company number 10933403), each > registered in England and each with a registered office at: Ashcombe Court, > Woolsack Way, Godalming, GU7 1LQ. > The contents of this email and any attachment are confidential to the intended recipient(s). If you are not an intended recipient: (i) do not use, disclose, distribute, copy or publish this email or its contents; (ii) please contact the sender immediately; and (iii) delete this email. Our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy Limited (company number 8619644); Origami Storage Limited (company number 10436515) and OSSPV001 Limited (company number 10933403), each registered in England and each with a registered office at: Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
Re: Understanding default.deserialization.exception.handler
Your observation is correct. It's a known bug: https://issues.apache.org/jira/browse/KAFKA-6502 In practice, it should not be a big issue though. - you would only hit this bug if you don't process a "good message" afterwards - even if you hit this bug, you would just skip the message again Thus, the only drawback I see is an additional log message. As long as you don't have many _consecutive_ corrupted messages it should be impact you much. Hope this helps. -Matthias On 9/13/18 6:09 AM, Tim Ward wrote: > With > > > props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, > LogAndContinueExceptionHandler.class); > > Scenario A: > > Run application. Feed a message into the topic that will fail deserialization. > Application logs exception and keeps running. > Shut down application. Restart application. > Application re-reads broken message, logs exception again (and keeps running). > > Scenario B: > > Run application. Feed a message into the topic that will fail deserialization. > Application logs exception and keeps running. > Feed a good message into deserialization. > Application processes it normally. > Shut down application. Restart application. > Application does *not* re-reads broken message. > > So it looks like LogAndContinueExceptionHandler does not seem to commit() the > incoming "poison pill" message(s), and these will be re-read if the > application is restarted without any good messages having been read after the > bad ones. > > Have I understood this correctly? If so, is this correct behaviour as > designed? Is it documented to that level of detail? > > Tim Ward > > The contents of this email and any attachment are confidential to the > intended recipient(s). If you are not an intended recipient: (i) do not use, > disclose, distribute, copy or publish this email or its contents; (ii) please > contact the sender immediately; and (iii) delete this email. Our privacy > policy is available here: https://origamienergy.com/privacy-policy/. Origami > Energy Limited (company number 8619644); Origami Storage Limited (company > number 10436515) and OSSPV001 Limited (company number 10933403), each > registered in England and each with a registered office at: Ashcombe Court, > Woolsack Way, Godalming, GU7 1LQ. > signature.asc Description: OpenPGP digital signature
Understanding default.deserialization.exception.handler
With props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, LogAndContinueExceptionHandler.class); Scenario A: Run application. Feed a message into the topic that will fail deserialization. Application logs exception and keeps running. Shut down application. Restart application. Application re-reads broken message, logs exception again (and keeps running). Scenario B: Run application. Feed a message into the topic that will fail deserialization. Application logs exception and keeps running. Feed a good message into deserialization. Application processes it normally. Shut down application. Restart application. Application does *not* re-reads broken message. So it looks like LogAndContinueExceptionHandler does not seem to commit() the incoming "poison pill" message(s), and these will be re-read if the application is restarted without any good messages having been read after the bad ones. Have I understood this correctly? If so, is this correct behaviour as designed? Is it documented to that level of detail? Tim Ward The contents of this email and any attachment are confidential to the intended recipient(s). If you are not an intended recipient: (i) do not use, disclose, distribute, copy or publish this email or its contents; (ii) please contact the sender immediately; and (iii) delete this email. Our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy Limited (company number 8619644); Origami Storage Limited (company number 10436515) and OSSPV001 Limited (company number 10933403), each registered in England and each with a registered office at: Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
