Thanks Mike, Most of the JIRAs there are internal cleanups, but for the user-facing ones we're planning on creating a wiki and collecting feedback like yours, and a KIP, so stay tuned (your current feedback already noted, thanks).
Eno > On 24 May 2017, at 15:35, Mike Gould <mikeyg...@gmail.com> wrote: > > Watching it with interest thanks > > Not sure where appropriate to add suggestions but I'd vote for exceptions > being passed along the stream in something like a hidden Either wrapper. > Most of the KStream methods would ignore this but new onException() or > similar methods would be able to examine the error with the key/value prior > to the error and handle it - possibly by replacing the message, sending a > message to a new stream, or even putting it back on the original stream for > retry. > > Regards > MikeG > > On Wed, 24 May 2017 at 10:09, Eno Thereska <eno.there...@gmail.com > <mailto:eno.there...@gmail.com>> wrote: > >> Just a heads up that we're tracking this and other improvements in >> exception handling at https://issues.apache.org/jira/browse/KAFKA-5156 >> <https://issues.apache.org/jira/browse/KAFKA-5156> < >> https://issues.apache.org/jira/browse/KAFKA-5156 >> <https://issues.apache.org/jira/browse/KAFKA-5156>>. >> >> Thanks >> Eno >>> On 23 May 2017, at 17:31, Mike Gould <mikeyg...@gmail.com> wrote: >>> >>> That's great for the value but not the key >>> >>> On Thu, 13 Apr 2017 at 18:27, Sachin Mittal <sjmit...@gmail.com> wrote: >>> >>>> We are also catching the exception in serde and returning null and then >>>> filtering out null values downstream so as they are not included. >>>> >>>> Thanks >>>> Sachin >>>> >>>> >>>> On Thu, Apr 13, 2017 at 9:13 PM, Mike Gould <mikeyg...@gmail.com> >> wrote: >>>> >>>>> Great to know I've not gone off in the wrong direction >>>>> Thanks >>>>> >>>>> On Thu, 13 Apr 2017 at 16:34, Matthias J. Sax <matth...@confluent.io> >>>>> wrote: >>>>> >>>>>> Mike, >>>>>> >>>>>> thanks for your feedback. You are absolutely right that Streams API >>>> does >>>>>> not have great support for this atm. And it's very valuable that you >>>>>> report this (you are not the first person). It helps us prioritizing >> :) >>>>>> >>>>>> For now, there is no better solution as the one you described in your >>>>>> email, but its on our roadmap to improve the API -- and its priority >>>> got >>>>>> just increase by your request. >>>>>> >>>>>> I am sorry, that I can't give you a better answer right now :( >>>>>> >>>>>> >>>>>> -Matthias >>>>>> >>>>>> >>>>>> On 4/13/17 8:16 AM, Mike Gould wrote: >>>>>>> Hi >>>>>>> Are there any better error handling options for Kafka streams in >>>> java. >>>>>>> >>>>>>> Any errors in the serdes will break the stream. The suggested >>>>>>> implementation is to use the byte[] serde and do the deserialisation >>>>> in a >>>>>>> map operation. However this isn't ideal either as there's no great >>>> way >>>>>> to >>>>>>> handle exceptions. >>>>>>> My current tactics are to use flatMap in place of map everywhere and >>>>>> return >>>>>>> empySet on error. Unfortunately this means the error has to be >>>> handled >>>>>>> directly in the function where it happened and can only be handled >>>> as a >>>>>>> side effect. >>>>>>> >>>>>>> It seems to me that this could be done better. Maybe the *Mapper >>>>>> interfaces >>>>>>> could allow specific checked exceptions. These could be handled by >>>>>> specific >>>>>>> downstream KStream.mapException() steps which might e.g. Put an error >>>>>>> response on another stream branch. >>>>>>> Alternatively could it be made easier to return something like an >>>>> Either >>>>>>> from the Mappers with a the addition of few extra mapError or mapLeft >>>>>>> mapRight methods on KStream? >>>>>>> >>>>>>> Unless there's a better error handling pattern which I've entirely >>>>>> missed? >>>>>>> >>>>>>> Thanks >>>>>>> MIkeG >>>>>>> >>>>>> >>>>>> -- >>>>> - MikeG >>>>> http://en.wikipedia.org/wiki/Common_misconceptions >>>>> <http://en.wikipedia.org/wiki/Special:Random> >>>>> >>>> >>> -- >>> - MikeG >>> http://en.wikipedia.org/wiki/Common_misconceptions >>> <http://en.wikipedia.org/wiki/Special:Random> >> >> -- > - MikeG > http://en.wikipedia.org/wiki/Common_misconceptions > <http://en.wikipedia.org/wiki/Common_misconceptions> > <http://en.wikipedia.org/wiki/Special:Random > <http://en.wikipedia.org/wiki/Special:Random>>
