Using Prometheus Client Metrics in Beam with FlinkRunner

[Rion Williams]

Is anyone aware of a way to get Prometheus metrics to output via Beam (i.e.
using them over the Beam abstractions)? I seem to be able to define them as
expected, however I don’t see them being emitted with the rest of my
Prometheus metrics:

private class RouteEvent(): DoFn<...>() {
        private lateinit var totalMessages: Counter

        @StartBundle
        fun startBundle(context: StartBundleContext){
            totalMessages = RoutingMetrics.getTotalMessageCounter()
            // This is the equivalent of grabbing the following registered
metric
            // val totalMessages = Counter
            //   .build()
            //    .name("messages_routed_total")
            //    .namespace("event_router")
            //    .help("The total number of messages routed")
            //    .register()
        }

        @ProcessElement
        fun processElement(context: ProcessContext){
            // Omitted for brevity

            totalMessages.inc()
        }
}

I’m using the FlinkRunner along with the PrometheusReporter, but it seems
like a piece is missing that will ensure that these same metrics are
included with those that are emitted by default. It’s worth noting that if
I use the Beam abstractions, this works as expected, however in this case I
need the ability to add/update labels, which doesn’t appear to be
supported, so I figured that Prometheus might be more appropriate.

Any ideas? It seems like it's a configuration issue or something missing
that connects the dots between Prometheus here but I'm not entirely sure
where that connection lies.

Thanks!

Rion