Hi Ilya, I have not got any experience of doing this, but wonder if we could use the Flink Metrics <https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/> . I wonder: - There could be hook point at that part of the code to discover some custom code that implements the metrics. - A better way might be to throw a new unable-to-parse Exception that could be caught in non-protobuf code and then a count metric be incremented. I guess the metrics would be scoped to a Job – like the watermark metrics. - I wondered if you might also want to count how many were successfully parsed in a non-protobuf layer (Dynamic table sort of level) - do you know why some messages do not parse? It would interesting to know the types of non-parsable messages, I assume with the idea to understand and potentially fix the source / parser limitations.
Kind regards, David. From: Ilya Karpov <idkf...@gmail.com> Date: Monday, 17 June 2024 at 12:39 To: user <user@flink.apache.org> Subject: [EXTERNAL] A way to meter number of deserialization errors Hi all, we are planning to use flink as a connector between kafka and external systems. We use protobuf as a message format in kafka. If non-backward compatible changes occur we want to skip those messages ('protobuf. ignore-parse-errors' Hi all, we are planning to use flink as a connector between kafka and external systems. We use protobuf as a message format in kafka. If non-backward compatible changes occur we want to skip those messages ('protobuf.ignore-parse-errors' = 'true') but record an error and raise an alert. I didn't find any way to record deserialization errors (https://github.com/apache/flink/blob/3a15d1ce69ac21d619f60033ec45cae303489c8f/flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/PbRowDataDeserializationSchema.java#L73), does it exist? Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU