Hi guys,
In this thread, we discussed about two main topics:
1. The metrics we should collect and where (IO, PTransform, ...).
2. How we use/dump the metrics.
Today, for instance, with the Spark runner, it's already possible to use
a sink by providing a metrics.properties containing something like:
driver.sink.graphite.class=org.apache.beam.runners.spark.metrics.sink.GraphiteSink
driver.sink.graphite.host=localhost
driver.sink.graphite.port=2003
driver.sink.graphite.prefix=spark
driver.sink.graphite.period=1
driver.sink.graphite.unit=SECONDS
I think this is great but, as stated before, runner specific. So, if I
run my pipeline on another runner, I have to update the
configuration/tools in use, and it could not be possible (depending of
the runner).
Maybe we could have a metric-extension that start a thread periodically
polling the MetricResults (from the PipelineResult).
I will start a PoC of this metric-extension and gonna test with the
different runners.
Thoughts ?
Regards
JB
On 02/22/2017 09:14 PM, Jean-Baptiste Onofré wrote:
Hi Ismaël,
You did a great summary and you put the emphasis on the valid point.
I full agree with you about the fact we should provide something
agnostic for end-users to get the metrics.
Let's see what the others are thinking. I'm ready to start a PoC to
bring part of Decanter in Beam ;)
Thanks again
Regards
JB
On 02/22/2017 07:55 PM, Ismaël Mejía wrote:
Hello,
Thanks everyone for giving your points of view. I was waiting to see how
the conversation evolved to summarize it and continue on the open points.
Points where mostly everybody agrees (please correct me if somebody still
disagrees):
- Default metrics should not affect performance, for that reason they
should be calculated by default. (in this case disabling them should
matter
less and probably we can evaluate if we make this configurable per IO in
the future).
- There is a clear distinction between metrics that are useful for the
user
and metrics that are useful for the runner. The user metrics are the most
important for user experience. (we will focus on these).
- We should support metrics for IOs in both APIs: Source/Sink based
IOs and
SDF.
- IO metrics should focus on what's relevant to the Pipeline. Relevant
metrics should be discussed in a per IO basis. (It is hard to generalize,
but probably we will make progress faster just creating metrics for each
one and then consolidating the common ones).
Points where consensus is not yet achieved
- Should IOs expose metrics that are useful for the runners? And if so
How?
I think this is important but not relevant for the current discussion
so we
should probably open a different conversation for this. My only comment
around this is that we must prevent the interference of mixing runner and
user oriented metrics (probably with a namespace).
- Where is the frontier of the responsibilities of Beam for metrics?
Should
we have a runner-agnostic way to recollect metrics (different from result
polling)?
We can offer a plugin-like system to push metrics into given sinks, JB
proposed an approach similar to Karaf’s Decanter. There is also the issue
of pull-based metrics like those of Codehale.
As a user I think having something like what JB proposed is nice, even a
REST service to query stuff about pipelines in a runner-agnostic way
would
make me happy too, but again it is up to the community to decide how
can we
we implement this and if this should be part of Beam.
What do you guys think about the pending issues? Did I miss something
else ?
Ismaël
On Sat, Feb 18, 2017 at 9:02 PM, Jean-Baptiste Onofré <[email protected]>
wrote:
Yes, agree Ben.
More than the collected metrics, my question is more how to
"expose"/"push" those metrics.
Imagine, I have a pipeline executed using the Spark runner on a Spark
cluster. Now, I change this pipeline to use the dataflow runner on
Google
Cloud Dataflow service. I have to completely change the way of getting
metrics.
Optionally, if I'm able to define some like --metric-appender=foo.cfg
containing additionally to the execution engine specific layer, a target
where I can push the metric (like elasticsearch or kafka), I can
implement
a simple and generic way of harvesting metrics.
It's totally fine to have some metric specific to Spark when using the
Spark runner and others specific to Dataflow when using the dataflow
runner, my point is more: how I can send the metrics to my global
monitoring/reporting layer.
Somehow, we can see the metrics like meta side output of the pipeline,
send to a target sink.
I don't want to change or bypass the execution engine specific, I mean
provide a way for the user to target his system (optional).
Regards
JB
On 02/18/2017 06:15 PM, Ben Chambers wrote:
The question is how much of metrics has to be in the runner and how
much
can be shared. So far we share the API the user uses to report
metrics -
this is the most important part since it is required for pipelines
to be
portable.
The next piece that could be shared is something related to reporting.
But,
implementing logically correct metrics requires the execution engine
to be
involved, since it depends on how and which bundles are retried.
What I'm
not sure about is how much can be shared and/or made available for
these
runners vs. how much is tied to the execution engine.
On Sat, Feb 18, 2017, 8:52 AM Jean-Baptiste Onofré <[email protected]>
wrote:
For Spark, I fully agree. My point is more when the execution engine or
runner doesn't provide anything or we have to provide a generic way of
harvesting/pushing metrics.
It could at least be a documentation point. Actually, I'm
evaluation the
monitoring capabilities of the different runners.
Regards
JB
On 02/18/2017 05:47 PM, Amit Sela wrote:
That's what I don't understand - why would we want that ?
Taking on responsibilities in the "stack" should have a good reason.
Someone choosing to run Beam on Spark/Flink/Apex would have to
take care
of
installing those clusters, right ? perhaps providing them a resilient
underlying FS ? and if he wants, setup monitoring (which even with
the
API
proposed he'd have to do).
I just don't see why it should be a part of the runner and/or Metrics
API.
On Sat, Feb 18, 2017 at 6:35 PM Jean-Baptiste Onofré
<[email protected]>
wrote:
Good point.
In Decanter, it's what I named a "scheduled collector". So, yes, the
adapter will periodically harvest metric to push.
Regards
JB
On 02/18/2017 05:30 PM, Amit Sela wrote:
First issue with "push" metrics plugin - what if the runner's
underlying
reporting mechanism is "pull" ? Codahale ScheduledReporter will
sample
the
values every X and send to ...
So any runner using a "pull-like" would use an adapter ?
On Sat, Feb 18, 2017 at 6:27 PM Jean-Baptiste Onofré
<[email protected]
wrote:
Hi Ben,
ok it's what I thought. Thanks for the clarification.
+1 for the plugin-like "push" API (it's what I have in mind too
;)).
I will start a PoC for discussion next week.
Regards
JB
On 02/18/2017 05:17 PM, Ben Chambers wrote:
The runner can already report metrics during pipeline
execution so
it
is
usable for monitoring.
The pipeline result can be used to query metrics during pipeline
execution,
so a first version of reporting to other systems is to
periodically
pulls
metrics from the runner with that API.
We may eventually want to provide a plugin-like API to get the
runner
to
push metrics more directly to other metrics stores. This layer
needs
some
thought since it has to handle the complexity of
attempted/committed
metrics to be consistent with the model.
On Sat, Feb 18, 2017, 5:44 AM Jean-Baptiste Onofré
<[email protected]
wrote:
Hi Amit,
before Beam, I didn't mind about portability ;) So I used the
Spark
approach.
But, now, as a Beam user, I would expect a generic way to deal
with
metric whatever the runner would be.
Today, you are right: I'm using the solution provided by the
execution
engine. That's the current approach and it works fine. And it's up to
me
to leverage (for intance Accumulators) it with my own system.
My thought is more to provide a generic way. It's only a
discussion
for
now ;)
Regards
JB
On 02/18/2017 02:38 PM, Amit Sela wrote:
On Sat, Feb 18, 2017 at 10:16 AM Jean-Baptiste Onofré <
[email protected]>
wrote:
Hi Amit,
my point is: how do we provide metric today to end user and how
can
they
use it to monitor a running pipeline ?
Clearly the runner is involved, but, it should behave the
same way
for
all runners. Let me take an example.
On my ecosystem, I'm using both Flink and Spark with Beam, some
pipelines on each. I would like to get the metrics for all
pipelines
to
my monitoring backend. If I can "poll" from the execution engine
metric
backend to my system that's acceptable, but it's an overhead of
work.
Having a generic metric reporting layer would allow us to have a
more
common way. If the user doesn't provide any reporting sink, then we
use
the execution backend metric layer. If provided, we use the
reporting
sink.
How did you do it before Beam ? I that for Spark you
reported it's
native
metrics via Codahale Reporter and Accumulators were visible in
the
UI,
and
the Spark runner took it a step forward to make it all visible
via
Codahale. Assuming Flink does something similar, it all
belongs to
runner
setup/configuration.
About your question: you are right, it's possible to update a
collector
or appender without impacting anything else.
Regards
JB
On 02/17/2017 10:38 PM, Amit Sela wrote:
@JB I think what you're suggesting is that Beam should
provide a
"Metrics
Reporting" API as well, and I used to think like you, but the
more I
thought of that the more I tend to disagree now.
The SDK is for users to author pipelines, so Metrics are for
user-defined
metrics (in contrast to runner metrics).
The Runner API is supposed to help different backends to
integrate
with
Beam to allow users to execute those pipeline on their favourite
backend. I
believe the Runner API has to provide restrictions/demands
that
are
just
enough so a runner could execute a Beam pipeline as best it can,
and
I'm
afraid that this would demand runner authors to do work that is
unnecessary.
This is also sort of "crossing the line" into the runner's
domain
and
"telling it how to do" instead of what, and I don't think we want
that.
I do believe however that runner's should integrate the
Metrics
into
their
own metrics reporting system - but that's for the runner
author
to
decide.
Stas did this for the Spark runner because Spark doesn't
report
back
user-defined Accumulators (Spark's Aggregators) to it's Metrics
system.
On a curious note though, did you use an OSGi service per
event-type ?
so
you can upgrade specific event-handlers without taking down the
entire
reporter ? but that's really unrelated to this thread :-) .
On Fri, Feb 17, 2017 at 8:36 PM Ben Chambers
<[email protected]>
wrote:
It don't think it is possible for there to be a general
mechanism
for
pushing metrics out during the execution of a pipeline. The
Metrics
API
suggests that metrics should be reported as values across all
attempts
and
values across only successful attempts. The latter requires
runner
involvement to ensure that a given metric value is atomically
incremented
(or checkpointed) when the bundle it was reported in is
committed.
Aviem has already implemented Metrics support for the Spark
runner. I
am
working on support for the Dataflow runner.
On Fri, Feb 17, 2017 at 7:50 AM Jean-Baptiste Onofré <
[email protected]>
wrote:
Hi guys,
As I'm back from vacation, I'm back on this topic ;)
It's a great discussion, and I think about the Metric IO
coverage,
it's
good.
However, there's a point that we discussed very fast in the
thread
and
I
think it's an important one (maybe more important than the
provided
metrics actually in term of roadmap ;)).
Assuming we have pipelines, PTransforms, IOs, ... using the
Metric
API,
how do we expose the metrics for the end-users ?
A first approach would be to bind a JMX MBean server by the
pipeline
and
expose the metrics via MBeans. I don't think it's a good idea
for
the
following reasons:
1. It's not easy to know where the pipeline is actually
executed,
and
so, not easy to find the MBean server URI.
2. For the same reason, we can have port binding error.
3. If it could work for unbounded/streaming pipelines (as
they
are
always "running"), it's not really applicable for bounded/batch
pipelines as their lifetime is "limited" ;)
So, I think the "push" approach is better: during the
execution,
a
pipeline "internally" collects and pushes the metric to a
backend.
The "push" could a kind of sink. For instance, the metric
"records"
can
be sent to a Kafka topic, or directly to Elasticsearch or
whatever.
The metric backend can deal with alerting, reporting, etc.
Basically, we have to define two things:
1. The "appender" where the metrics have to be sent (and the
corresponding configuration to connect, like Kafka or
Elasticsearch
location)
2. The format of the metric data (for instance, json format).
In Apache Karaf, I created something similar named Decanter:
http://blog.nanthrax.net/2015/07/monitoring-and-alerting-wit
h-apache-karaf-decanter/
http://karaf.apache.org/manual/decanter/latest-1/
Decanter provides collectors that harvest the metrics
(like JMX
MBean
attributes, log messages, ...). Basically, for Beam, it would be
directly the Metric API used by pipeline parts.
Then, the metric record are send to a dispatcher which
send the
metric
records to an appender. The appenders store or send the metric
records
to a backend (elasticsearc, cassandra, kafka, jms, reddis, ...).
I think it would make sense to provide the configuration and
Metric
"appender" via the pipeline options.
As it's not really runner specific, it could be part of the
metric
API
(or SPI in that case).
WDYT ?
Regards
JB
On 02/15/2017 09:22 AM, Stas Levin wrote:
+1 to making the IO metrics (e.g. producers, consumers)
available
as
part
of the Beam pipeline metrics tree for debugging and
visibility.
As it has already been mentioned, many IO clients have a
metrics
mechanism
in place, so in these cases I think it could be
beneficial to
mirror
their
metrics under the relevant subtree of the Beam metrics tree.
On Wed, Feb 15, 2017 at 12:04 AM Amit Sela <
[email protected]>
wrote:
I think this is a great discussion and I'd like to relate to
some
of
the
points raised here, and raise some of my own.
First of all I think we should be careful here not to cross
boundaries.
IOs
naturally have many metrics, and Beam should avoid "taking
over"
those.
IO
metrics should focus on what's relevant to the Pipeline:
input/output
rate,
backlog (for UnboundedSources, which exists in bytes but for
monitoring
purposes we might want to consider #messages).
I don't agree that we should not invest in doing this in
Sources/Sinks
and
going directly to SplittableDoFn because the IO API is
familiar
and
known,
and as long as we keep it should be treated as a first class
citizen.
As for enable/disable - if IOs consider focusing on
pipeline-related
metrics I think we should be fine, though this could also
change
between
runners as well.
Finally, considering "split-metrics" is interesting
because on
one
hand
it
affects the pipeline directly (unbalanced partitions in
Kafka
that
may
cause backlog) but this is that fine-line of responsibilities
(Kafka
monitoring would probably be able to tell you that partitions
are
not
balanced).
My 2 cents, cheers!
On Tue, Feb 14, 2017 at 8:46 PM Raghu Angadi
<[email protected]
wrote:
On Tue, Feb 14, 2017 at 9:21 AM, Ben Chambers
<[email protected]
wrote:
* I also think there are data source specific metrics
that a
given
IO
will
want to expose (ie, things like kafka backlog for a
topic.)
UnboundedSource has API for backlog. It is better for
beam/runners
to
handle backlog as well.
Of course there will be some source specific metrics too
(errors,
i/o
ops
etc).
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
