Watching it with interest thanks Not sure where appropriate to add suggestions but I'd vote for exceptions being passed along the stream in something like a hidden Either wrapper. Most of the KStream methods would ignore this but new onException() or similar methods would be able to examine the error with the key/value prior to the error and handle it - possibly by replacing the message, sending a message to a new stream, or even putting it back on the original stream for retry.
Regards MikeG On Wed, 24 May 2017 at 10:09, Eno Thereska <eno.there...@gmail.com> wrote: > Just a heads up that we're tracking this and other improvements in > exception handling at https://issues.apache.org/jira/browse/KAFKA-5156 < > https://issues.apache.org/jira/browse/KAFKA-5156>. > > Thanks > Eno > > On 23 May 2017, at 17:31, Mike Gould <mikeyg...@gmail.com> wrote: > > > > That's great for the value but not the key > > > > On Thu, 13 Apr 2017 at 18:27, Sachin Mittal <sjmit...@gmail.com> wrote: > > > >> We are also catching the exception in serde and returning null and then > >> filtering out null values downstream so as they are not included. > >> > >> Thanks > >> Sachin > >> > >> > >> On Thu, Apr 13, 2017 at 9:13 PM, Mike Gould <mikeyg...@gmail.com> > wrote: > >> > >>> Great to know I've not gone off in the wrong direction > >>> Thanks > >>> > >>> On Thu, 13 Apr 2017 at 16:34, Matthias J. Sax <matth...@confluent.io> > >>> wrote: > >>> > >>>> Mike, > >>>> > >>>> thanks for your feedback. You are absolutely right that Streams API > >> does > >>>> not have great support for this atm. And it's very valuable that you > >>>> report this (you are not the first person). It helps us prioritizing > :) > >>>> > >>>> For now, there is no better solution as the one you described in your > >>>> email, but its on our roadmap to improve the API -- and its priority > >> got > >>>> just increase by your request. > >>>> > >>>> I am sorry, that I can't give you a better answer right now :( > >>>> > >>>> > >>>> -Matthias > >>>> > >>>> > >>>> On 4/13/17 8:16 AM, Mike Gould wrote: > >>>>> Hi > >>>>> Are there any better error handling options for Kafka streams in > >> java. > >>>>> > >>>>> Any errors in the serdes will break the stream. The suggested > >>>>> implementation is to use the byte[] serde and do the deserialisation > >>> in a > >>>>> map operation. However this isn't ideal either as there's no great > >> way > >>>> to > >>>>> handle exceptions. > >>>>> My current tactics are to use flatMap in place of map everywhere and > >>>> return > >>>>> empySet on error. Unfortunately this means the error has to be > >> handled > >>>>> directly in the function where it happened and can only be handled > >> as a > >>>>> side effect. > >>>>> > >>>>> It seems to me that this could be done better. Maybe the *Mapper > >>>> interfaces > >>>>> could allow specific checked exceptions. These could be handled by > >>>> specific > >>>>> downstream KStream.mapException() steps which might e.g. Put an error > >>>>> response on another stream branch. > >>>>> Alternatively could it be made easier to return something like an > >>> Either > >>>>> from the Mappers with a the addition of few extra mapError or mapLeft > >>>>> mapRight methods on KStream? > >>>>> > >>>>> Unless there's a better error handling pattern which I've entirely > >>>> missed? > >>>>> > >>>>> Thanks > >>>>> MIkeG > >>>>> > >>>> > >>>> -- > >>> - MikeG > >>> http://en.wikipedia.org/wiki/Common_misconceptions > >>> <http://en.wikipedia.org/wiki/Special:Random> > >>> > >> > > -- > > - MikeG > > http://en.wikipedia.org/wiki/Common_misconceptions > > <http://en.wikipedia.org/wiki/Special:Random> > > -- - MikeG http://en.wikipedia.org/wiki/Common_misconceptions <http://en.wikipedia.org/wiki/Special:Random>
