Hey community,

I'm interested in how we can monitor the number of elements dropped that arrive too late for a windowed aggregation (i.e., allowed lateness passed). I configured a JMX reporter for Samza (and export these JMX metrics to Prometheus). However, while I get a number of both Samza and Beam metrics, I neither find a Samza nor a Beam metric reporting on the number of dropped/skipped messages.

In the Beam sources, I found that a "droppedDueToLateness" metric (https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java) exists and the Samza documentation reports on "window-skipped" (https://samza.apache.org/learn/documentation/latest/container/metrics.html). However, both metrics are not reported for my pipeline and I don't find any further explanation for them.

Thanks
Sören

Reply via email to