Hello devs, I recently took a look at how Dataflow is retrieving metrics from the Beam SDK harnesses, and noticed something. As you may (or may not) remember, the portability API currently has two ways of reporting metrics. Namely, the newer MonitoringInfo API[1], and the older Metrics one[2].
This is somewhat troublesome because now we have two things that do the same thing. The SDKs report double the amount of metrics[3][4], and I bet it's confusing for runner implementers. Luckily, it seems like the Flink and Spark runners do use the new API [5][6] - yay! : ) - so I guess then the only runner that uses the old API is Dataflow? (internally) Which way does the Samza runner use? +Hai Lu? How about the Go SDK +Robert Burke <r...@google.com> ? - Ah I bet this uses the old API? If they all use the MonitoringInfos, we may be able to clean up the old api, and move to the new one (somewhat)soon : ) [1] https://github.com/apache/beam/blob/v2.15.0/model/fn-execution/src/main/proto/beam_fn_api.proto#L395 [2] https://github.com/apache/beam/blob/v2.15.0/model/fn-execution/src/main/proto/beam_fn_api.proto#L391 [3] https://github.com/apache/beam/blob/c1007b678a00ea85671872236edef940a8e56adc/sdks/python/apache_beam/runners/worker/sdk_worker.py#L406-L414 [4] https://github.com/apache/beam/blob/c1007b678a00ea85671872236edef940a8e56adc/sdks/python/apache_beam/runners/worker/sdk_worker.py#L378-L384 [5] https://github.com/apache/beam/blob/44fa33e6518574cb9561f47774e218e0910093fe/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L94-L97 [6] https://github.com/apache/beam/blob/932bd80a17171bd2d8157820ffe09e8389a52b9b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L219-L226
