Hi Tom,
using side outputs is actually the established Flink pattern in that
regard. The advantage of side output is that you do not depend on the DLQ
concept of the source system, which is incredibly useful if you read from
multiple systems.
Most commonly, the side-output is then outputted to
Thanks Robert.
On Fri, Jul 24, 2020 at 2:32 PM Robert Metzger wrote:
> Hey Tom,
>
> I'm not aware of any patterns for this problem, sorry. Intuitively, I
> would send dead letters to a separate Kafka topic.
>
> Best,
> Robert
>
>
> On Wed, Jul 22, 2020 at 7:22 PM Tom Fennelly
> wrote:
>
>>
Hey Tom,
I'm not aware of any patterns for this problem, sorry. Intuitively, I would
send dead letters to a separate Kafka topic.
Best,
Robert
On Wed, Jul 22, 2020 at 7:22 PM Tom Fennelly
wrote:
> Thanks Chen.
>
> I'm thinking about errors that occur while processing a record/message
> that
Thanks Chen.
I'm thinking about errors that occur while processing a record/message that
shouldn't be retried until after some "action" has been taken Vs flooding
the system with pointless retries e.g.
- A side output step might involve an API call to an external system and
that system is
Could you more specific on what “failed message” means here?In general side output can do something like were def process(ele) { try{ biz} catch { Sideout( ele + exception context)}} process(func).sideoutput(tag).addSink(kafkasink) Thanks,Chen From: Eleanore JinSent: Wednesday, July
+1 we have a similar use case for message schema validation.
Eleanore
On Wed, Jul 22, 2020 at 4:12 AM Tom Fennelly
wrote:
> Hi.
>
> I've been searching blogs etc trying to see if there are
> established patterns/mechanisms for reprocessing of failed messages via
> something like a DLQ. I've
