[ https://issues.apache.org/jira/browse/FLINK-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855478#comment-15855478 ]
Haohui Mai commented on FLINK-5583: ----------------------------------- The interface looks good to me. It looks like that you want to have the deserialization out of the lock. Is it okay to assume that collector is thread-safe? > Support flexible error handling in the Kafka consumer > ----------------------------------------------------- > > Key: FLINK-5583 > URL: https://issues.apache.org/jira/browse/FLINK-5583 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector > Reporter: Haohui Mai > Assignee: Haohui Mai > > We found that it is valuable to allow the applications to handle errors and > exceptions in the Kafka consumer in order to build a robust application in > production. > The context is the following: > (1) We have schematized, Avro records flowing through Kafka. > (2) The decoder implements the DeserializationSchema to decode the records. > (3) Occasionally there are corrupted records (e.g., schema issues). The > streaming pipeline might want to bail out (which is the current behavior) or > to skip the corrupted records depending on the applications. > Two options are available: > (1) Have a variant of DeserializationSchema to return a FlatMap like > structure as suggested in FLINK-3679. > (2) Allow the applications to catch and handle the exception by exposing some > APIs that are similar to the {{ExceptionProxy}}. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.15#6346)