If so , then you can implement your own deserializer[1] with costume logic and error handling
1. https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/streaming/util/serialization/KeyedDeserializationSchemaWrapper.html On Thu, Nov 22, 2018 at 8:57 AM Scott Sue <scott....@celer-tech.com> wrote: > Json is sent into Kafka > > > Regards, > Scott > > SCOTT SUE > CHIEF TECHNOLOGY OFFICER > > Support Line : +44(0) 2031 371 603 > Mobile : +852 9611 3969 > > 9/F, 33 Lockhart Road, Wan Chai, Hong Kong > www.celer-tech.com > > > > > > > > On 22 Nov 2018, at 14:55, miki haiat <miko5...@gmail.com> wrote: > > Which data format is sent to kafka ? > Json Avro Other ? > > > > On Thu, Nov 22, 2018 at 7:36 AM Scott Sue <scott....@celer-tech.com> > wrote: > >> Unexpected data meaning business level data that I didn’t expect to >> receive. So business level data that doesn’t quite conform >> >> On Thu, 22 Nov 2018 at 13:30, miki haiat <miko5...@gmail.com> wrote: >> >>> Unexpected data you mean parsing error ? >>> Which format is sent to Kafka ? >>> >>> >>> >>> On Thu, 22 Nov 2018, 6:59 Scott Sue <scott....@celer-tech.com wrote: >>> >>>> Hi all, >>>> >>>> When I'm running my jobs I am consuming data from Kafka to process in my >>>> job. Unfortunately my job receives unexpected data from time to time >>>> which >>>> I'm trying to find the root cause of the issue. >>>> >>>> Ideally, I want to be able to have a way to know when the job has >>>> failed due >>>> to an exception, to then log to file the last message that it was >>>> consuming >>>> at the time to help track down the offending message consumed. How is >>>> this >>>> possible within Flink? >>>> >>>> Thinking about this more, it may not be a consumed message that killed >>>> the >>>> job, but maybe a transformation within the job itself and it died in a >>>> downstream Operator. In this case, is there a way to log to file the >>>> message that an Operator was processing at the time that caused the >>>> exception? >>>> >>>> >>>> Thanks in advance! >>>> >>>> >>>> >>>> -- >>>> Sent from: >>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >>>> >>> -- >> >> >> Regards, >> Scott >> >> SCOTT SUE >> CHIEF TECHNOLOGY OFFICER >> >> Support Line : +44(0) 2031 371 603 <+44%2020%203137%201603> >> Mobile : +852 9611 3969 <9611%203969> >> >> 9/F, 33 Lockhart Road, Wanchai, Hong Kong >> www.celer-tech.com >> >> *This message, including any attachments, may include private, privileged >> and confidential information and is intended only for the personal and >> confidential use of the intended recipient(s). If the reader of this >> message is not an intended recipient, you are hereby notified that any >> review, use, dissemination, distribution, printing or copying of this >> message or its contents is strictly prohibited and may be unlawful. If you >> are not an intended recipient or have received this communication in error, >> please immediately notify the sender by telephone and/or a reply email and >> permanently delete the original message, including any attachments, without >> making a copy.* >> > > > *This message, including any attachments, may include private, privileged > and confidential information and is intended only for the personal and > confidential use of the intended recipient(s). If the reader of this > message is not an intended recipient, you are hereby notified that any > review, use, dissemination, distribution, printing or copying of this > message or its contents is strictly prohibited and may be unlawful. If you > are not an intended recipient or have received this communication in error, > please immediately notify the sender by telephone and/or a reply email and > permanently delete the original message, including any attachments, without > making a copy.* >
