I’ve been thinking about metrics lately, specifically the fact that people tend to struggle with implementing a metrics consumer. (Like this one [1]).
The IMetricsConsumer interface is pretty low level, and common aggregations, calculations, etc. are left up to each individual implementation. That seems like an area where further abstraction would make it easier to support different back ends (Graphite, JMX, Splunk, etc.). My thought is to create an abstract IMetricsConsumer implementation that does common aggregations and calculations, and then delegates to a plugable “metrics sink” implementation (e.g. “IMetricsSink”, etc.). That would greatly simplify the effort required to integrate with various external metrics systems. I know of at least a few users that would be interested, one is currently scraping the logs from LoggingMetricsConsumer and polling the Storm REST API for their metrics. -Taylor [1] http://twocentsonsoftware.blogspot.co.il/2014/12/sending-out-storm-metrics.html > On Oct 10, 2016, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote: > > First of all the server exposes essentially the same interface that the > IMetricsConsumer exposes. It mostly just adds a bunch of overhead in the > middle to serialize out the objects send them over http to another process > which then has to deserialize them and process them. If you really don't > need the metrics to show up on a special known box you can have that exact > same code running inside the metrics consumer without all of the overhead. > The server/client are insecure, have to deal with thread issues that a normal > IMetricsConsumer does not, and are not written to be robust (If the HTTP > server is down the consumer crashes and continues to crash until the server > is brought back up). It was written very quickly for a test situation and > it honestly never crossed my mind that anyone would want to use it in > production. > > - Bobby > > On Monday, October 10, 2016 10:59 AM, S G <sg.online.em...@gmail.com> > wrote: > > > Thanks Bobby. > > If we write our own metrics consumer, how do we ensure that it is better > than HttpForwardingMetricsServer? In other words, what aspects of the > HttpForwardingMetricsServer > should we avoid to make our own metrics consumer better and ready for > production? > > Is versign/storm-graphite <https://github.com/verisign/storm-graphite> > production > ready? > > Also, we should add a line about production-readiness of > HttpForwardingMetricsServer > in the documentation at http://storm.apache.org/releases/1.0.2/Metrics.html > (We were just about to think seriously on using this for production as we > thought this to be the standard solution for metrics' consumption in 1.0+ > version). > > -SG > > On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans <ev...@yahoo-inc.com> wrote: > >> First of all there really are two different sets of metrics. One set is >> the topology metrics and the other set is the daemon metrics (metrics for >> things like the ui and nimbus). The JmxPreparableReporter plugin only >> exposes daemon metrics not the topology metrics through JMX. Exposing >> topology metrics through JMX is a non trivial task. The current metrics >> feature was not designed for this. We are in the process of trying to >> redesign the metrics system to allow for features like this, but it is >> still a ways off. >> >> - Bobby >> >> >> On Saturday, October 8, 2016 11:39 AM, S G <sg.online.em...@gmail.com> >> wrote: >> >> >> Thanks Bobby, >> >> We will need some kind of IMetricsConsumer to talk to telegraf. >> Many other softwares like Solr, Elastic-Search, Cassandra etc. provide >> metrics through a URL making it very easy to consume by tools like telegraf. >> How about a IMetricsConsumer that will run on storm-ui and provide the >> metrics through a URL such as <storm-ui-host>/metrics ? >> >> Also, I see the following option in defaults.yaml: >> #default storm daemon metrics reporter plugins >> storm.daemon.metrics.reporter.plugins: >> - "org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter" >> >> Is this a good option to use for converting metrics into JMX ? >> >> Thanks >> SG >> >> >> >> >> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans <ev...@yahoo-inc.com.invalid> >> wrote: >> >> HttpForwardingMetricsServer is a real hack intended really for tests. I >> know I wrote it :). Please don't use it in production. You can write your >> own IMetricesConsumer to do whatever you want to with the metrics. >> https://github.com/apache/ storm/blob/master/storm-core/ >> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer. java >> <https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java> >> >> That is the correct way to get the data out. If you want to write a >> bridge to JMX for this that might work, but going directly to telegraph >> would probably be better. - Bobby >> >> On Thursday, October 6, 2016 1:43 PM, S G <sg.online.em...@gmail.com> >> wrote: >> >> >> Hi, >> >> We want to use Telegraf ( >> https://github.com/influxdata/ telegraf/tree/master/plugins >> <https://github.com/influxdata/telegraf/tree/master/plugins>) for getting >> storm's metrics. >> >> But we do not want to add a HttpForwardingMetricsServer just to get the >> metrics and send them to telegraf. >> >> Other option is to use Jolokia (https://jolokia.org/) that can read JMX >> and >> write into telegraf. >> >> Does storm report all its metrics (including those of custom spouts/bolts) >> into JMX? >> Or spawning a HttpForwardingMetricsServer is the only option? >> >> Thanks >> SG >> >> >> >> >> >> >> >> > >
signature.asc
Description: Message signed with OpenPGP using GPGMail
