Hi all,
Etienne forgot to mention that we started a PoC about that.
What I started is to wrap the Pipeline creation to include a thread that polls
periodically the metrics in the pipeline result (it's what I proposed when I
compared with Karaf Decanter some time ago).
Then, this thread marshalls the collected metrics and send to a sink. At the
end, it means that the harvested metrics data will be store in a backend (for
instance elasticsearch).
The pro of this approach is that it doesn't require any change in the core, it's
up to the user to use the PipelineWithMetric wrapper.
The cons is that the user needs to explicitly use the PipelineWithMetric
wrapper.
IMHO, it's good enough as user can decide to poll metrics for some pipelines and
not for others.
Regards
JB
On 11/27/2017 04:56 PM, Etienne Chauchot wrote:
Hi all,
I came by this ticket https://issues.apache.org/jira/browse/BEAM-2456. I know
that the metrics subject has already been discussed a lot, but I would like to
revive the discussion.
The aim in this ticket is to avoid relying on the runner to provide the metrics
because they don't have all the same capabilities towards metrics. The idea in
the ticket is to still use beam metrics API (and not others like codahale as it
has been discussed some time ago) and provide a way to extract the metrics with
a polling thread that would be forked by a PipelineWithMetrics (so, almost
invisible to the end user) and then to push to a sink (such as a Http rest sink
for example or Graphite sink or anything else...). Nevertheless, a polling
thread might not work for all the runners because some might not make the
metrics available before the end of the pipeline. Also, forking a thread would
be a bit unconventional, so it could be provided as a beam sdk extension.
Another way, to avoid polling, would be to push metrics values to a sink when
they are updated but I don't know if it is feasible in a runner independent way.
WDYT about the ideas in this ticket?
Best,
Etienne
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
