[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=174036&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174036
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 11/Dec/18 11:37
Start Date: 11/Dec/18 11:37
Worklog Time Spent: 10m
Work Description: mxm closed pull request #7207: [BEAM-6172] Adjust Flink
metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
index 73fe62e219fc..f4857d18a587 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
@@ -19,12 +19,14 @@
import static
org.apache.beam.runners.core.metrics.MetricsContainerStepMap.asAttemptedOnlyMetricResults;
+import com.google.common.annotations.VisibleForTesting;
import java.util.HashMap;
import java.util.Map;
import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
import org.apache.beam.runners.core.metrics.MetricsContainerStepMap;
import org.apache.beam.sdk.metrics.DistributionResult;
import org.apache.beam.sdk.metrics.GaugeResult;
+import org.apache.beam.sdk.metrics.MetricName;
import org.apache.beam.sdk.metrics.MetricQueryResults;
import org.apache.beam.sdk.metrics.MetricResult;
import org.apache.beam.sdk.metrics.MetricResults;
@@ -32,6 +34,8 @@
import org.apache.beam.sdk.metrics.MetricsFilter;
import org.apache.flink.api.common.accumulators.Accumulator;
import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.configuration.GlobalConfiguration;
+import org.apache.flink.configuration.MetricOptions;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
import org.slf4j.Logger;
@@ -47,10 +51,8 @@
private static final Logger LOG =
LoggerFactory.getLogger(FlinkMetricContainer.class);
- private static final String METRIC_KEY_SEPARATOR = "__";
- private static final String COUNTER_PREFIX = "__counter";
- private static final String DISTRIBUTION_PREFIX = "__distribution";
- private static final String GAUGE_PREFIX = "__gauge";
+ private static final String METRIC_KEY_SEPARATOR =
+
GlobalConfiguration.loadConfiguration().getString(MetricOptions.SCOPE_DELIMITER);
private final RuntimeContext runtimeContext;
private final Map flinkCounterCache;
@@ -94,7 +96,7 @@ void updateMetrics(String stepName) {
private void updateCounters(Iterable> counters) {
for (MetricResult metricResult : counters) {
- String flinkMetricName = getFlinkMetricNameString(COUNTER_PREFIX,
metricResult);
+ String flinkMetricName = getFlinkMetricNameString(metricResult);
Long update = metricResult.getAttempted();
@@ -109,7 +111,7 @@ private void updateCounters(Iterable>
counters) {
private void updateDistributions(Iterable>
distributions) {
for (MetricResult metricResult : distributions) {
- String flinkMetricName = getFlinkMetricNameString(DISTRIBUTION_PREFIX,
metricResult);
+ String flinkMetricName = getFlinkMetricNameString(metricResult);
DistributionResult update = metricResult.getAttempted();
@@ -129,7 +131,7 @@ private void
updateDistributions(Iterable> dist
private void updateGauge(Iterable> gauges) {
for (MetricResult metricResult : gauges) {
- String flinkMetricName = getFlinkMetricNameString(GAUGE_PREFIX,
metricResult);
+ String flinkMetricName = getFlinkMetricNameString(metricResult);
GaugeResult update = metricResult.getAttempted();
@@ -144,14 +146,13 @@ private void
updateGauge(Iterable> gauges) {
}
}
- private static String getFlinkMetricNameString(String prefix,
MetricResult metricResult) {
-return prefix
-+ METRIC_KEY_SEPARATOR
-+ metricResult.getStep()
-+ METRIC_KEY_SEPARATOR
-+ metricResult.getName().getNamespace()
-+ METRIC_KEY_SEPARATOR
-+ metricResult.getName().getName();
+ @VisibleForTesting
+ static String getFlinkMetricNameString(MetricResult metricResult) {
+MetricName metricName = metricResult.getName();
+// We use only the MetricName here, the step name is already contained
+// in the operator name which is passed to Flink's MetricGroup to which
+// the metric with the following name will be added.
+return metricName.getNamespace() + METRIC_K
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=174035&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174035
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 11/Dec/18 11:36
Start Date: 11/Dec/18 11:36
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7207: [BEAM-6172] Adjust Flink
metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#issuecomment-446171930
Merging this, Python PreCommit failures unrelated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 174035)
Time Spent: 2h 20m (was: 2h 10m)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=174011&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174011
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 11/Dec/18 10:37
Start Date: 11/Dec/18 10:37
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7207: [BEAM-6172] Adjust Flink
metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#issuecomment-446154463
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 174011)
Time Spent: 2h 10m (was: 2h)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=173903&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173903
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 11/Dec/18 04:49
Start Date: 11/Dec/18 04:49
Worklog Time Spent: 10m
Work Description: mwylde commented on issue #7207: [BEAM-6172] Adjust
Flink metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#issuecomment-446071429
LGTM! Thank you!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 173903)
Time Spent: 1h 50m (was: 1h 40m)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=173904&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173904
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 11/Dec/18 04:49
Start Date: 11/Dec/18 04:49
Worklog Time Spent: 10m
Work Description: mwylde commented on issue #7207: [BEAM-6172] Adjust
Flink metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#issuecomment-446071484
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 173904)
Time Spent: 2h (was: 1h 50m)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=173688&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173688
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 10/Dec/18 16:51
Start Date: 10/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7207: [BEAM-6172] Adjust Flink
metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#issuecomment-445887045
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 173688)
Time Spent: 1.5h (was: 1h 20m)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=173689&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173689
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 10/Dec/18 16:51
Start Date: 10/Dec/18 16:51
Worklog Time Spent: 10m
Work Description: mxm removed a comment on issue #7207: [BEAM-6172]
Adjust Flink metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#issuecomment-445887045
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 173689)
Time Spent: 1h 40m (was: 1.5h)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=173680&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173680
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 10/Dec/18 16:26
Start Date: 10/Dec/18 16:26
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7207: [BEAM-6172] Adjust Flink
metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#issuecomment-445877680
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 173680)
Time Spent: 1h 20m (was: 1h 10m)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=173614&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173614
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 10/Dec/18 14:33
Start Date: 10/Dec/18 14:33
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7207: [BEAM-6172] Adjust Flink
metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#issuecomment-445835659
Should be good to go @mwylde.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 173614)
Time Spent: 1h 10m (was: 1h)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=173613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173613
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 10/Dec/18 14:32
Start Date: 10/Dec/18 14:32
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #7207:
[BEAM-6172] Adjust Flink metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#discussion_r240231499
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
##
@@ -144,9 +145,9 @@ private void
updateGauge(Iterable> gauges) {
}
}
- private static String getFlinkMetricNameString(String prefix,
MetricResult metricResult) {
-return prefix
-+ METRIC_KEY_SEPARATOR
+ @VisibleForTesting
+ static String getFlinkMetricNameString(MetricResult metricResult) {
+return METRIC_KEY_SEPARATOR
Review comment:
Flink generates its MetricGroup from the operator name which contains the
step id. I think it is fine to omit the step name from the generated name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 173613)
Time Spent: 1h (was: 50m)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=173042&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173042
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 07/Dec/18 19:22
Start Date: 07/Dec/18 19:22
Worklog Time Spent: 10m
Work Description: mwylde commented on a change in pull request #7207:
[BEAM-6172] Adjust Flink metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#discussion_r239917917
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
##
@@ -144,9 +145,9 @@ private void
updateGauge(Iterable> gauges) {
}
}
- private static String getFlinkMetricNameString(String prefix,
MetricResult metricResult) {
-return prefix
-+ METRIC_KEY_SEPARATOR
+ @VisibleForTesting
+ static String getFlinkMetricNameString(MetricResult metricResult) {
+return METRIC_KEY_SEPARATOR
Review comment:
With this I get
```
10.100.208.242.taskmanager.4f13adf64e7315b7198911465ca44119.BeamApp-mwylde-1207183134-fd6cc74c.group.0..group.org.apache.beam.runners.core.ReduceFnRunner.droppedDueToClosedWindow
```
It appears that Flink already includes the step, so in the cases I've seen
this can be
```
return
metricResult.getName().getNamespace()
+ METRIC_KEY_SEPARATOR
+ metricResult.getName().getName();
```
which gives you
```
10.100.208.242.taskmanager.4f13adf64e7315b7198911465ca44119.BeamApp-mwylde-1207185706-ce63a421.group.0.org.apache.beam.runners.core.ReduceFnRunner.droppedDueToClosedWindow:
112
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 173042)
Time Spent: 40m (was: 0.5h)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=173043&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173043
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 07/Dec/18 19:22
Start Date: 07/Dec/18 19:22
Worklog Time Spent: 10m
Work Description: mwylde commented on a change in pull request #7207:
[BEAM-6172] Adjust Flink metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#discussion_r239917917
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
##
@@ -144,9 +145,9 @@ private void
updateGauge(Iterable> gauges) {
}
}
- private static String getFlinkMetricNameString(String prefix,
MetricResult metricResult) {
-return prefix
-+ METRIC_KEY_SEPARATOR
+ @VisibleForTesting
+ static String getFlinkMetricNameString(MetricResult metricResult) {
+return METRIC_KEY_SEPARATOR
Review comment:
With this I get
```
10.100.208.242.taskmanager.4f13adf64e7315b7198911465ca44119.BeamApp-mwylde-1207183134-fd6cc74c.group.0..group.org.apache.beam.runners.core.ReduceFnRunner.droppedDueToClosedWindow
```
It appears that Flink already includes the step, so in the cases I've seen
this can be
```
return
metricResult.getName().getNamespace()
+ METRIC_KEY_SEPARATOR
+ metricResult.getName().getName();
```
which gives you
```
10.100.208.242.taskmanager.4f13adf64e7315b7198911465ca44119.BeamApp-mwylde-1207185706-ce63a421.group.0.org.apache.beam.runners.core.ReduceFnRunner.droppedDueToClosedWindow
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 173043)
Time Spent: 50m (was: 40m)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=172935&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172935
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 07/Dec/18 09:19
Start Date: 07/Dec/18 09:19
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7207: [BEAM-6172] Adjust Flink
metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#issuecomment-445171435
Please review @mwylde since you requested this change in the JIRA issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172935)
Time Spent: 0.5h (was: 20m)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[
https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=172310&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172310
]
ASF GitHub Bot logged work on BEAM-6172:
Author: ASF GitHub Bot
Created on: 05/Dec/18 15:02
Start Date: 05/Dec/18 15:02
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7207: [BEAM-6172] Adjust Flink
metric names / Add metric reporting tests
URL: https://github.com/apache/beam/pull/7207#issuecomment-444515725
CC @ryan-williams
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 172310)
Time Spent: 20m (was: 10m)
> Flink metrics are not generated in standard format
> --
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Micah Wylde
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6172) Flink metrics are not generated in standard format
[ https://issues.apache.org/jira/browse/BEAM-6172?focusedWorklogId=172269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172269 ] ASF GitHub Bot logged work on BEAM-6172: Author: ASF GitHub Bot Created on: 05/Dec/18 13:20 Start Date: 05/Dec/18 13:20 Worklog Time Spent: 10m Work Description: mxm opened a new pull request #7207: [BEAM-6172] Adjust Flink metric names / Add metric reporting tests URL: https://github.com/apache/beam/pull/7207 This picks up the metrics delimiter from the Flink config and removes the metric type prefix. This improves consistency with Flink's defaults and makes metric names more readable. R @mwylde CC @tweise Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 172269) Time Spent: 10m Remaining Estimate: 0h > Flink metrics are not generated in standard format > -- > > Key: BEAM-6172 > URL: https://issues.apache.org/jira/browse/BEAM-6172 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Affects Versions: 2.8.0 >Reporter: Micah Wylde >Assignee: Maximilian Michels >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The metrics that the flink runner exports do not follow the standard format > use
