RE: Recommended pattern for implementing a DLQ with Flink+Kafka

[Chen Qin]

Could you more specific on what “failed message” means here? In general side output can do something like were   def process(ele) {    try{         biz } catch {    Sideout( ele + exception context) } }   process(func).sideoutput(tag).addSink(kafkasink)   Thanks, Chen       From: Eleanore Jin Sent: Wednesday, July 22, 2020 9:25 AM To: Tom Fennelly Cc: user Subject: Re: Recommended pattern for implementing a DLQ with Flink+Kafka   +1 we have a similar use case for message schema validation.   Eleanore   On Wed, Jul 22, 2020 at 4:12 AM Tom Fennelly <tfenne...@cloudbees.com> wrote: Hi.   I've been searching blogs etc trying to see if there are established patterns/mechanisms for reprocessing of failed messages via something like a DLQ. I've read about using checkpointing and restarting tasks (not what we want because we want to keep processing forward) and then also how some use side outputs to filter "bad" data to a DLQ style topic. Kafka has dead letter topic configs too but it seems that can't really be used from inside Flink (from what I can see).   We're running Flink+Kafka. I'm new to Flink+Kafka so not sure if there just isn't a defined pattern for it, or if I'm just not asking the right questions in my searches. I searched the archives here and don't see anything either, which obviously makes me think that I'm not thinking about this in the "Flink way" :-|   Regards,   Tom.