Hi community,
I'd like to expand my previous question to also cover the Flink runner.
Is there any way to track the number of elements dropped due to
lateness? Either using the native Samza/Flink metrics or with Beam metrics?
Thanks
Sören
Am 02.12.22 um 16:45 schrieb Sören Henning:
Hey community,
I'm interested in how we can monitor the number of elements dropped
that arrive too late for a windowed aggregation (i.e., allowed
lateness passed). I configured a JMX reporter for Samza (and export
these JMX metrics to Prometheus). However, while I get a number of
both Samza and Beam metrics, I neither find a Samza nor a Beam metric
reporting on the number of dropped/skipped messages.
In the Beam sources, I found that a "droppedDueToLateness" metric
(https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java)
exists and the Samza documentation reports on "window-skipped"
(https://samza.apache.org/learn/documentation/latest/container/metrics.html).
However, both metrics are not reported for my pipeline and I don't
find any further explanation for them.
Thanks
Sören
