[ https://issues.apache.org/jira/browse/BEAM-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599032#comment-16599032 ]
Jozef Vilcek edited comment on BEAM-5246 at 8/31/18 5:15 PM: ------------------------------------------------------------- I was looking around a bit more ... Does the attached PR make sense? To forward metrics from accumulator to metricGroup only for actual stepName. was (Author: jozovilcek): I was looking around a bit more ... Does the attached PR make sense, to forward metrics from accumulator to metricGroup only for actual stepName? > Beam metrics exported as flink metrics are not correct > ------------------------------------------------------ > > Key: BEAM-5246 > URL: https://issues.apache.org/jira/browse/BEAM-5246 > Project: Beam > Issue Type: Bug > Components: runner-flink > Affects Versions: 2.6.0 > Reporter: Jozef Vilcek > Assignee: Aljoscha Krettek > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In Flink UI and fink native MetricReported, I am seeing too many instances of > my Beam metric counter. It looks like the counter is materialised for every > operator running within the task, although is is emitter from only one beam > step (which should map to one operator?). This produces double counting. > A bit debugging I noticed this is happening for stream jobs. In batch I was > not able to reproduce it. Problem might be in FlinkMetricContainer. > [https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L86] > The update seems to be called from operators after finishing the bundle. Data > from accumulator are flushed to `runtimeContext.getMetricGroup()`. The scope > of accumulator seems to be different than metricGroup as in there with > different call the scope components change, especially for operatorID. It > seems like during the run, `metricResult.getStep()` does not match > operatorName of metricGroup where metric is being pushed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)