sorry, hopefully the link goes through now:
http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite
Sending Metrics from Storm to Graphite - Michael G. Noll
By Michael G. Noll
Sending application-level metrics from Storm topologies to the Graphite
monitoring system
On Tuesday, October 11, 2016 1:13 PM, Alessandro Bellina
<abell...@yahoo-inc.com.INVALID> wrote:
I think what Bobby is referring to is that the metrics consumer is another
bolt, so stats are flowing through storm.
What does changing the model to polling buy us? I could see cases were we'd
need more error handling for instance slow/busy workers.
If we think that writing a new system is the way to go (say with codahale
throughout), would working on an abstraction layer that is used by the daemons
but also by end-users be a good place to start? With codahale as the
implementation?
Looks like Michael Noll has done a lot work with codahale, for instance:
Sending Metrics from Storm to Graphite - Michael G. Noll.
|
|
|
| | |
|
|
|
| |
Sending Metrics from Storm to Graphite - Michael G. Noll
By Michael G. Noll Sending application-level metrics from Storm topologies to
the Graphite monitoring system | |
|
|
Thanks,
Alessandro
On Tuesday, October 11, 2016 11:07 AM, S G <sg.online.em...@gmail.com>
wrote:
"Dropwizard has solved all of these problems already and I don't see a
reason to reinvent the wheel" - I love dropwizard too and many of the other
tools have switched to using the same too.
"I don't personally see a lot of value in trying to send all of the metrics
through storm itself" - How about every node reporting its own metrics by a
URL ? That ways there is no need for a metrics-consumer that can bottleneck
the whole topology. We can then provide a separate server that can query
all nodes to get those metrics and aggregate them. Only cluster wide
metrics should be reported by the storm-UI's REST API (assuming there are
not too many of those).
On Tue, Oct 11, 2016 at 7:15 AM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> I agree that IMetricsConsumer is not good, but the reality is that all of
> the metrics system needs to be redone. The problem is that we ship an
> object as a metric. If I get an object I have no idea what it is hand
> hence no idea how to report it or what to do with it. What is more the
> common types we use in the metrics we provide are really not enough. For
> example CountMetric sends a Long. Well when I get it in the metrics
> consumer I have no idea if I should report it like a counter or if I should
> report it like a gauge (something that every metrics system I have used
> wants to know). But then we support pre-aggregation of the metrics with
> IReducer so the number I get might be an average instead of either a gauge
> or a counter, which no good metrics system will want to collect because I
> cannot aggregate it with anything else, the math just does not work.
> The proposal I have said before and I still believe is that we need to put
> in place a parallel metrics API/system. We will deprecate all of
> https://git.corp.yahoo.com/storm/storm/tree/master-
> security/storm-core/src/jvm/backtype/storm/metric/api and create a new
> parallel one that provides an API similar to http://metrics.dropwizard.
> io/3.1.0/. I would even be fine in just using their API and exposing
> that to end users. Dropwizard has solved all of these problems already and
> I don't see a reason to reinvent the wheel. I don't personally see a lot
> of value in trying to send all of the metrics through storm itslef. I am
> fine if we are able to support that, but it is far from a requirement. -
> Bobby
>
> On Monday, October 10, 2016 10:47 PM, S G <sg.online.em...@gmail.com>
> wrote:
>
>
> +1
> We can probably start by opening a JIRA for this and adding a design
> approach for the same?
> I would like to help in the coding-effort for this.
>
> -SG
>
> On Mon, Oct 10, 2016 at 1:51 PM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
>
> > I’ve been thinking about metrics lately, specifically the fact that
> people
> > tend to struggle with implementing a metrics consumer. (Like this one
> [1]).
> >
> > The IMetricsConsumer interface is pretty low level, and common
> > aggregations, calculations, etc. are left up to each individual
> > implementation. That seems like an area where further abstraction would
> > make it easier to support different back ends (Graphite, JMX, Splunk,
> etc.).
> >
> > My thought is to create an abstract IMetricsConsumer implementation that
> > does common aggregations and calculations, and then delegates to a
> plugable
> > “metrics sink” implementation (e.g. “IMetricsSink”, etc.). That would
> > greatly simplify the effort required to integrate with various external
> > metrics systems. I know of at least a few users that would be interested,
> > one is currently scraping the logs from LoggingMetricsConsumer and
> polling
> > the Storm REST API for their metrics.
> >
> > -Taylor
> >
> > [1] http://twocentsonsoftware.blogspot.co.il/2014/12/
> > sending-out-storm-metrics.html
> >
> >
> > > On Oct 10, 2016, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID
> >
> > wrote:
> > >
> > > First of all the server exposes essentially the same interface that the
> > IMetricsConsumer exposes. It mostly just adds a bunch of overhead in the
> > middle to serialize out the objects send them over http to another
> process
> > which then has to deserialize them and process them. If you really don't
> > need the metrics to show up on a special known box you can have that
> exact
> > same code running inside the metrics consumer without all of the
> overhead.
> > > The server/client are insecure, have to deal with thread issues that a
> > normal IMetricsConsumer does not, and are not written to be robust (If
> the
> > HTTP server is down the consumer crashes and continues to crash until the
> > server is brought back up). It was written very quickly for a test
> > situation and it honestly never crossed my mind that anyone would want to
> > use it in production.
> > >
> > > - Bobby
> > >
> > > On Monday, October 10, 2016 10:59 AM, S G <
> sg.online.em...@gmail.com>
> > wrote:
> > >
> > >
> > > Thanks Bobby.
> > >
> > > If we write our own metrics consumer, how do we ensure that it is
> better
> > > than HttpForwardingMetricsServer? In other words, what aspects of the
> > > HttpForwardingMetricsServer
> > > should we avoid to make our own metrics consumer better and ready for
> > > production?
> > >
> > > Is versign/storm-graphite <https://github.com/verisign/storm-graphite>
> > > production
> > > ready?
> > >
> > > Also, we should add a line about production-readiness of
> > > HttpForwardingMetricsServer
> > > in the documentation at http://storm.apache.org/
> > releases/1.0.2/Metrics.html
> > > (We were just about to think seriously on using this for production as
> we
> > > thought this to be the standard solution for metrics' consumption in
> 1.0+
> > > version).
> > >
> > > -SG
> > >
> > > On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans <ev...@yahoo-inc.com>
> > wrote:
> > >
> > >> First of all there really are two different sets of metrics. One set
> is
> > >> the topology metrics and the other set is the daemon metrics (metrics
> > for
> > >> things like the ui and nimbus). The JmxPreparableReporter plugin only
> > >> exposes daemon metrics not the topology metrics through JMX. Exposing
> > >> topology metrics through JMX is a non trivial task. The current
> metrics
> > >> feature was not designed for this. We are in the process of trying to
> > >> redesign the metrics system to allow for features like this, but it is
> > >> still a ways off.
> > >>
> > >> - Bobby
> > >>
> > >>
> > >> On Saturday, October 8, 2016 11:39 AM, S G <sg.online.em...@gmail.com
> >
> > >> wrote:
> > >>
> > >>
> > >> Thanks Bobby,
> > >>
> > >> We will need some kind of IMetricsConsumer to talk to telegraf.
> > >> Many other softwares like Solr, Elastic-Search, Cassandra etc. provide
> > >> metrics through a URL making it very easy to consume by tools like
> > telegraf.
> > >> How about a IMetricsConsumer that will run on storm-ui and provide the
> > >> metrics through a URL such as <storm-ui-host>/metrics ?
> > >>
> > >> Also, I see the following option in defaults.yaml:
> > >> #default storm daemon metrics reporter plugins
> > >> storm.daemon.metrics.reporter.plugins:
> > >> - "org.apache.storm.daemon.metrics.reporters.
> > JmxPreparableReporter"
> > >>
> > >> Is this a good option to use for converting metrics into JMX ?
> > >>
> > >> Thanks
> > >> SG
> > >>
> > >>
> > >>
> > >>
> > >> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans
> <ev...@yahoo-inc.com.invalid
> > >
> > >> wrote:
> > >>
> > >> HttpForwardingMetricsServer is a real hack intended really for
> tests. I
> > >> know I wrote it :). Please don't use it in production. You can write
> > your
> > >> own IMetricesConsumer to do whatever you want to with the metrics.
> > >> https://github.com/apache/ storm/blob/master/storm-core/
> > >> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer. java
> > >> <https://github.com/apache/storm/blob/master/storm-core/
> > src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java>
> > >>
> > >> That is the correct way to get the data out. If you want to write a
> > >> bridge to JMX for this that might work, but going directly to
> telegraph
> > >> would probably be better. - Bobby
> > >>
> > >> On Thursday, October 6, 2016 1:43 PM, S G <
> > sg.online.em...@gmail.com>
> > >> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> We want to use Telegraf (
> > >> https://github.com/influxdata/ telegraf/tree/master/plugins
> > >> <https://github.com/influxdata/telegraf/tree/master/plugins>) for
> > getting
> > >> storm's metrics.
> > >>
> > >> But we do not want to add a HttpForwardingMetricsServer just to get
> the
> > >> metrics and send them to telegraf.
> > >>
> > >> Other option is to use Jolokia (https://jolokia.org/) that can read
> JMX
> > >> and
> > >> write into telegraf.
> > >>
> > >> Does storm report all its metrics (including those of custom
> > spouts/bolts)
> > >> into JMX?
> > >> Or spawning a HttpForwardingMetricsServer is the only option?
> > >>
> > >> Thanks
> > >> SG
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> >
> >
>
>
>
