Hi Ilya,
I have not got any experience of doing this, but wonder if we could use the
Flink Metrics
<https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/> . I
wonder:
- There could be hook point at that part of the code to discover some custom
code that implements the metrics.
- A better way might be to throw a new unable-to-parse Exception that could be
caught in non-protobuf code and then a count metric be incremented. I guess the
metrics would be scoped to a Job – like the watermark metrics.
- I wondered if you might also want to count how many were successfully parsed
in a non-protobuf layer (Dynamic table sort of level)
- do you know why some messages do not parse? It would interesting to know the
types of non-parsable messages, I assume with the idea to understand and
potentially fix the source / parser limitations.
Kind regards, David.
From: Ilya Karpov <idkf...@gmail.com>
Date: Monday, 17 June 2024 at 12:39
To: user <user@flink.apache.org>
Subject: [EXTERNAL] A way to meter number of deserialization errors
Hi all, we are planning to use flink as a connector between kafka and external
systems. We use protobuf as a message format in kafka. If non-backward
compatible changes occur we want to skip those messages ('protobuf.
ignore-parse-errors'
Hi all,
we are planning to use flink as a connector between kafka and external systems.
We use protobuf as a message format in kafka. If non-backward compatible
changes occur we want to skip those messages ('protobuf.ignore-parse-errors' =
'true') but record an error and raise an alert. I didn't find any way to record
deserialization errors
(https://github.com/apache/flink/blob/3a15d1ce69ac21d619f60033ec45cae303489c8f/flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/PbRowDataDeserializationSchema.java#L73),
does it exist?
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
