Hey community,
I'm interested in how we can monitor the number of elements dropped that
arrive too late for a windowed aggregation (i.e., allowed lateness
passed). I configured a JMX reporter for Samza (and export these JMX
metrics to Prometheus). However, while I get a number of both Samza and
Beam metrics, I neither find a Samza nor a Beam metric reporting on the
number of dropped/skipped messages.
In the Beam sources, I found that a "droppedDueToLateness" metric
(https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java)
exists and the Samza documentation reports on "window-skipped"
(https://samza.apache.org/learn/documentation/latest/container/metrics.html).
However, both metrics are not reported for my pipeline and I don't find
any further explanation for them.
Thanks
Sören
- Monitor number of dropped elements due to lateness in Samza... Sören Henning
-