[
https://issues.apache.org/jira/browse/BEAM-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Etienne Chauchot updated BEAM-3310:
-----------------------------------
Description:
The idea is to avoid relying on the runners to provide access to the metrics
(either at the end of the pipeline or while it runs) because they don't have
all the same capabilities towards metrics (e.g. spark runner configures sinks
like csv, graphite or in memory sinks using the spark engine conf). The target
is to push the metrics in the common runner code so that no matter the chosen
runner, a user can get his metrics out of beam.
Here is the link to the discussion thread on the dev ML:
https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
And the design doc:
https://s.apache.org/runner_independent_metrics_extraction
was:
The idea is to avoid relying on the runners to provide access to the metrics
(either at the end of the pipeline or while it runs) because they don't have
all the same capabilities towards metrics (e.g. spark runner configures sinks
like csv, graphite or in memory sinks using the spark engine conf). The aim is
to push metrics to a sink when they are updated. The target is to push the
metrics in the common sdk code so that no matter the chosen runner, a user can
get his metrics out of beam.
Here is the link to the discussion thread on the dev ML:
https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
And the design doc:
https://s.apache.org/runner_independent_metrics_extraction
> Push metrics to a backend in an runner agnostic way
> ---------------------------------------------------
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
> Reporter: Etienne Chauchot
>
> The idea is to avoid relying on the runners to provide access to the metrics
> (either at the end of the pipeline or while it runs) because they don't have
> all the same capabilities towards metrics (e.g. spark runner configures sinks
> like csv, graphite or in memory sinks using the spark engine conf). The
> target is to push the metrics in the common runner code so that no matter the
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML:
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
