Re: Recommended pattern for implementing a DLQ with Flink+Kafka

2020-07-30 Thread Arvid Heise
;>>> def process(ele) { >>>> >>>>try{ >>>> >>>> biz >>>> >>>> } catch { >>>> >>>>Sideout( ele + exception context) >>>> >>>> } >>>> >>>> }

Re: Recommended pattern for implementing a DLQ with Flink+Kafka

2020-07-24 Thread Tom Fennelly
gt; In general side output can do something like were >>> >>> >>> >>> def process(ele) { >>> >>>try{ >>> >>> biz >>> >>> } catch { >>> >>> Sideout( ele + exception context) >>&g

Re: Recommended pattern for implementing a DLQ with Flink+Kafka

2020-07-24 Thread Robert Metzger
; } >> >> } >> >> >> >> process(func).sideoutput(tag).addSink(kafkasink) >> >> >> >> Thanks, >> >> Chen >> >> >> >> >> >> >> >> *From: *Eleanore Jin >> *Sent: *Wednesday, J

Re: Recommended pattern for implementing a DLQ with Flink+Kafka

2020-07-22 Thread Tom Fennelly
; > Thanks, > > Chen > > > > > > > > *From: *Eleanore Jin > *Sent: *Wednesday, July 22, 2020 9:25 AM > *To: *Tom Fennelly > *Cc: *user > *Subject: *Re: Recommended pattern for implementing a DLQ with Flink+Kafka > > > > +1 we have a similar us

RE: Recommended pattern for implementing a DLQ with Flink+Kafka

2020-07-22 Thread Chen Qin
22, 2020 9:25 AMTo: Tom FennellyCc: userSubject: Re: Recommended pattern for implementing a DLQ with Flink+Kafka +1 we have a similar use case for message schema validation. Eleanore On Wed, Jul 22, 2020 at 4:12 AM Tom Fennelly <tfenne...@cloudbees.com> wrote:Hi. I've been searchin

Re: Recommended pattern for implementing a DLQ with Flink+Kafka

2020-07-22 Thread Eleanore Jin
+1 we have a similar use case for message schema validation. Eleanore On Wed, Jul 22, 2020 at 4:12 AM Tom Fennelly wrote: > Hi. > > I've been searching blogs etc trying to see if there are > established patterns/mechanisms for reprocessing of failed messages via > something like a DLQ. I've rea

Recommended pattern for implementing a DLQ with Flink+Kafka

2020-07-22 Thread Tom Fennelly
Hi. I've been searching blogs etc trying to see if there are established patterns/mechanisms for reprocessing of failed messages via something like a DLQ. I've read about using checkpointing and restarting tasks (not what we want because we want to keep processing forward) and then also how some u