@JB I think what you're suggesting is that Beam should provide a "Metrics Reporting" API as well, and I used to think like you, but the more I thought of that the more I tend to disagree now.
The SDK is for users to author pipelines, so Metrics are for user-defined metrics (in contrast to runner metrics). The Runner API is supposed to help different backends to integrate with Beam to allow users to execute those pipeline on their favourite backend. I believe the Runner API has to provide restrictions/demands that are just enough so a runner could execute a Beam pipeline as best it can, and I'm afraid that this would demand runner authors to do work that is unnecessary. This is also sort of "crossing the line" into the runner's domain and "telling it how to do" instead of what, and I don't think we want that. I do believe however that runner's should integrate the Metrics into their own metrics reporting system - but that's for the runner author to decide. Stas did this for the Spark runner because Spark doesn't report back user-defined Accumulators (Spark's Aggregators) to it's Metrics system. On a curious note though, did you use an OSGi service per event-type ? so you can upgrade specific event-handlers without taking down the entire reporter ? but that's really unrelated to this thread :-) . On Fri, Feb 17, 2017 at 8:36 PM Ben Chambers <[email protected]> wrote: > It don't think it is possible for there to be a general mechanism for > pushing metrics out during the execution of a pipeline. The Metrics API > suggests that metrics should be reported as values across all attempts and > values across only successful attempts. The latter requires runner > involvement to ensure that a given metric value is atomically incremented > (or checkpointed) when the bundle it was reported in is committed. > > Aviem has already implemented Metrics support for the Spark runner. I am > working on support for the Dataflow runner. > > On Fri, Feb 17, 2017 at 7:50 AM Jean-Baptiste Onofré <[email protected]> > wrote: > > Hi guys, > > As I'm back from vacation, I'm back on this topic ;) > > It's a great discussion, and I think about the Metric IO coverage, it's > good. > > However, there's a point that we discussed very fast in the thread and I > think it's an important one (maybe more important than the provided > metrics actually in term of roadmap ;)). > > Assuming we have pipelines, PTransforms, IOs, ... using the Metric API, > how do we expose the metrics for the end-users ? > > A first approach would be to bind a JMX MBean server by the pipeline and > expose the metrics via MBeans. I don't think it's a good idea for the > following reasons: > 1. It's not easy to know where the pipeline is actually executed, and > so, not easy to find the MBean server URI. > 2. For the same reason, we can have port binding error. > 3. If it could work for unbounded/streaming pipelines (as they are > always "running"), it's not really applicable for bounded/batch > pipelines as their lifetime is "limited" ;) > > So, I think the "push" approach is better: during the execution, a > pipeline "internally" collects and pushes the metric to a backend. > The "push" could a kind of sink. For instance, the metric "records" can > be sent to a Kafka topic, or directly to Elasticsearch or whatever. > The metric backend can deal with alerting, reporting, etc. > > Basically, we have to define two things: > 1. The "appender" where the metrics have to be sent (and the > corresponding configuration to connect, like Kafka or Elasticsearch > location) > 2. The format of the metric data (for instance, json format). > > In Apache Karaf, I created something similar named Decanter: > > > http://blog.nanthrax.net/2015/07/monitoring-and-alerting-with-apache-karaf-decanter/ > > http://karaf.apache.org/manual/decanter/latest-1/ > > Decanter provides collectors that harvest the metrics (like JMX MBean > attributes, log messages, ...). Basically, for Beam, it would be > directly the Metric API used by pipeline parts. > Then, the metric record are send to a dispatcher which send the metric > records to an appender. The appenders store or send the metric records > to a backend (elasticsearc, cassandra, kafka, jms, reddis, ...). > > I think it would make sense to provide the configuration and Metric > "appender" via the pipeline options. > As it's not really runner specific, it could be part of the metric API > (or SPI in that case). > > WDYT ? > > Regards > JB > > On 02/15/2017 09:22 AM, Stas Levin wrote: > > +1 to making the IO metrics (e.g. producers, consumers) available as part > > of the Beam pipeline metrics tree for debugging and visibility. > > > > As it has already been mentioned, many IO clients have a metrics > mechanism > > in place, so in these cases I think it could be beneficial to mirror > their > > metrics under the relevant subtree of the Beam metrics tree. > > > > On Wed, Feb 15, 2017 at 12:04 AM Amit Sela <[email protected]> wrote: > > > >> I think this is a great discussion and I'd like to relate to some of the > >> points raised here, and raise some of my own. > >> > >> First of all I think we should be careful here not to cross boundaries. > IOs > >> naturally have many metrics, and Beam should avoid "taking over" those. > IO > >> metrics should focus on what's relevant to the Pipeline: input/output > rate, > >> backlog (for UnboundedSources, which exists in bytes but for monitoring > >> purposes we might want to consider #messages). > >> > >> I don't agree that we should not invest in doing this in Sources/Sinks > and > >> going directly to SplittableDoFn because the IO API is familiar and > known, > >> and as long as we keep it should be treated as a first class citizen. > >> > >> As for enable/disable - if IOs consider focusing on pipeline-related > >> metrics I think we should be fine, though this could also change between > >> runners as well. > >> > >> Finally, considering "split-metrics" is interesting because on one hand > it > >> affects the pipeline directly (unbalanced partitions in Kafka that may > >> cause backlog) but this is that fine-line of responsibilities (Kafka > >> monitoring would probably be able to tell you that partitions are not > >> balanced). > >> > >> My 2 cents, cheers! > >> > >> On Tue, Feb 14, 2017 at 8:46 PM Raghu Angadi <[email protected] > > > >> wrote: > >> > >>> On Tue, Feb 14, 2017 at 9:21 AM, Ben Chambers > >> <[email protected] > >>>> > >>> wrote: > >>> > >>>> > >>>>> * I also think there are data source specific metrics that a given IO > >>>> will > >>>>> want to expose (ie, things like kafka backlog for a topic.) > >>> > >>> > >>> UnboundedSource has API for backlog. It is better for beam/runners to > >>> handle backlog as well. > >>> Of course there will be some source specific metrics too (errors, i/o > ops > >>> etc). > >>> > >> > > > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
