Just a heads up that we're tracking this and other improvements in exception handling at https://issues.apache.org/jira/browse/KAFKA-5156 <https://issues.apache.org/jira/browse/KAFKA-5156>.
Thanks Eno > On 23 May 2017, at 17:31, Mike Gould <mikeyg...@gmail.com> wrote: > > That's great for the value but not the key > > On Thu, 13 Apr 2017 at 18:27, Sachin Mittal <sjmit...@gmail.com> wrote: > >> We are also catching the exception in serde and returning null and then >> filtering out null values downstream so as they are not included. >> >> Thanks >> Sachin >> >> >> On Thu, Apr 13, 2017 at 9:13 PM, Mike Gould <mikeyg...@gmail.com> wrote: >> >>> Great to know I've not gone off in the wrong direction >>> Thanks >>> >>> On Thu, 13 Apr 2017 at 16:34, Matthias J. Sax <matth...@confluent.io> >>> wrote: >>> >>>> Mike, >>>> >>>> thanks for your feedback. You are absolutely right that Streams API >> does >>>> not have great support for this atm. And it's very valuable that you >>>> report this (you are not the first person). It helps us prioritizing :) >>>> >>>> For now, there is no better solution as the one you described in your >>>> email, but its on our roadmap to improve the API -- and its priority >> got >>>> just increase by your request. >>>> >>>> I am sorry, that I can't give you a better answer right now :( >>>> >>>> >>>> -Matthias >>>> >>>> >>>> On 4/13/17 8:16 AM, Mike Gould wrote: >>>>> Hi >>>>> Are there any better error handling options for Kafka streams in >> java. >>>>> >>>>> Any errors in the serdes will break the stream. The suggested >>>>> implementation is to use the byte[] serde and do the deserialisation >>> in a >>>>> map operation. However this isn't ideal either as there's no great >> way >>>> to >>>>> handle exceptions. >>>>> My current tactics are to use flatMap in place of map everywhere and >>>> return >>>>> empySet on error. Unfortunately this means the error has to be >> handled >>>>> directly in the function where it happened and can only be handled >> as a >>>>> side effect. >>>>> >>>>> It seems to me that this could be done better. Maybe the *Mapper >>>> interfaces >>>>> could allow specific checked exceptions. These could be handled by >>>> specific >>>>> downstream KStream.mapException() steps which might e.g. Put an error >>>>> response on another stream branch. >>>>> Alternatively could it be made easier to return something like an >>> Either >>>>> from the Mappers with a the addition of few extra mapError or mapLeft >>>>> mapRight methods on KStream? >>>>> >>>>> Unless there's a better error handling pattern which I've entirely >>>> missed? >>>>> >>>>> Thanks >>>>> MIkeG >>>>> >>>> >>>> -- >>> - MikeG >>> http://en.wikipedia.org/wiki/Common_misconceptions >>> <http://en.wikipedia.org/wiki/Special:Random> >>> >> > -- > - MikeG > http://en.wikipedia.org/wiki/Common_misconceptions > <http://en.wikipedia.org/wiki/Special:Random>
