[
https://issues.apache.org/jira/browse/BEAM-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106702#comment-17106702
]
Kyle Weaver commented on BEAM-7219:
-----------------------------------
For the test_metrics failure, I found that metrics are being passed to Python.
The test breaks because no metrics match the filter [1], because the step names
are transformed somehow such that the filter logic is too strict to recognize
[2]:
Spark Runner:
MetricKey(step=ref_AppliedPTransform_count1_17, metric=MetricName(namespace=ns,
name=counter), labels={}): 2
MetricKey(step=ref_AppliedPTransform_count2_18, metric=MetricName(namespace=ns,
name=counter), labels={}): 4
...
Fn API Runner:
MetricKey(step=count1, metric=MetricName(namespace=ns, name=counter),
labels={}): 2,
MetricKey(step=count2, metric=MetricName(namespace=ns, name=counter),
labels={}): 4
Also, note that Flink has its own, completely different implementation of
test_metrics [3].
[1]
https://github.com/apache/beam/blob/2ef7b9db8af015dcba544b93df00a4e54cd8caf2/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner_test.py#L744
[2]
https://github.com/apache/beam/blob/2ef7b9db8af015dcba544b93df00a4e54cd8caf2/sdks/python/apache_beam/metrics/metric.py#L151-L155
[3]
https://github.com/apache/beam/blob/2ef7b9db8af015dcba544b93df00a4e54cd8caf2/sdks/python/apache_beam/runners/portability/flink_runner_test.py#L251
> Spark portable runner: support Spark metrics
> --------------------------------------------
>
> Key: BEAM-7219
> URL: https://issues.apache.org/jira/browse/BEAM-7219
> Project: Beam
> Issue Type: New Feature
> Components: runner-spark
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Major
> Labels: portability-spark
>
> The Spark portable runner already updates and collects metrics from the SDK
> harness, but it needs to send the collected metrics on to Spark itself,
> similar to FlinkMetricContainer
> ([https://github.com/apache/beam/blob/5c31efbae5fa40c024a47a44af3e6bc8a079cc2a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java]).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
