[jira] [Commented] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712689#comment-16712689
]
Maximilian Michels commented on BEAM-6172:
--
I believe that should be fixed in the pending PR
https://github.com/apache/beam/pull/7207. Would you mind giving it a try?
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712160#comment-16712160
]
Micah Wylde commented on BEAM-6172:
---
In addition to those two changes, there's an extra "group" in the name. For
example, a group-specific native flink metric (as output from the slf4j
reporter with default configs) looks like
{code}
10.100.208.242.taskmanager.4f13adf64e7315b7198911465ca44119.BeamApp-mwylde-1206234804-16211aec.group.0.numRecordsIn:
41
{code}
while a beam one looks like
{code}
10.100.208.242.taskmanager.4f13adf64e7315b7198911465ca44119.BeamApp-mwylde-1206234804-16211aec.group.0.__counter__group__org.apache.beam.runners.core.ReduceFnRunner__droppedDueToClosedWindow:
41
{code}
I think ideally the beam metric would be
{code}
10.100.208.242.taskmanager.4f13adf64e7315b7198911465ca44119.BeamApp-mwylde-1206234804-16211aec.group.0.org.apache.beam.runners.core.ReduceFnRunner.droppedDueToClosedWindow:
41
{code}
Our high-level goal is to be able to use the same metric parsing logic as we
use for existing flink metrics (both internal and application-specific) and get
reasonable results back.
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709924#comment-16709924
]
Maximilian Michels commented on BEAM-6172:
--
Beam on Flink is not the same as native Flink. Metrics are expected to look
different. Nevertheless, it makes sense to 1) respect the delimiter and 2)
remove the metric type. Perhaps we could make the latter configurable.
Is there anything else that you would like to change?
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
