Alessandro, that was before Verisign introduced storm-graphite. In result he followed Storm's metric system.
I initiated the discussion around metrics several times (mostly first half of this year), and from many times the results were that all the metrics interfaces are not good and we need to rework. Finding a way with resorting current interface doesn't help. How we renew the metrics system is the matter. I was waiting on phase of JStorm merger so that we can evaluate JStorm's new metrics system. I guess adopting that is enough changes but I saw that Alibaba met memory issue. How they handle this seems not exposed. For me the thing is JStorm merger is going really slow (nearly stopped), so going to phase 2 might not happen on this year. For seeking other ways, I guess Kafka and Flink renews their metrics (KafkaStream for Kafka) so we may want to take a look. Anyway, someone volunteer to renew metrics system via Dropwizard or JStorm metrics it would be awesome. I'm focused to improve Storm SQL and there's no other active contributor on Storm SQL yet so I couldn't look at the other side. - Jungtaek Lim (HeartSaVioR) 2016년 10월 12일 (수) 오전 3:15, Alessandro Bellina <abell...@yahoo-inc.com.invalid>님이 작성: > sorry, hopefully the link goes through now: > http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite > > > Sending Metrics from Storm to Graphite - Michael G. Noll > By Michael G. Noll > Sending application-level metrics from Storm topologies to the Graphite > monitoring system > > > > On Tuesday, October 11, 2016 1:13 PM, Alessandro Bellina < > abell...@yahoo-inc.com.INVALID> wrote: > > > > I think what Bobby is referring to is that the metrics consumer is another > bolt, so stats are flowing through storm. > What does changing the model to polling buy us? I could see cases were > we'd need more error handling for instance slow/busy workers. > If we think that writing a new system is the way to go (say with codahale > throughout), would working on an abstraction layer that is used by the > daemons but also by end-users be a good place to start? With codahale as > the implementation? > Looks like Michael Noll has done a lot work with codahale, for instance: > Sending Metrics from Storm to Graphite - Michael G. Noll. > > | > | > | > | | | > > | > > | > | > | | > Sending Metrics from Storm to Graphite - Michael G. Noll > By Michael G. Noll Sending application-level metrics from Storm topologies > to the Graphite monitoring system | | > > | > > | > > > > Thanks, > Alessandro > On Tuesday, October 11, 2016 11:07 AM, S G <sg.online.em...@gmail.com> > wrote: > > > > "Dropwizard has solved all of these problems already and I don't see a > reason to reinvent the wheel" - I love dropwizard too and many of the other > tools have switched to using the same too. > > "I don't personally see a lot of value in trying to send all of the metrics > through storm itself" - How about every node reporting its own metrics by a > URL ? That ways there is no need for a metrics-consumer that can bottleneck > the whole topology. We can then provide a separate server that can query > all nodes to get those metrics and aggregate them. Only cluster wide > metrics should be reported by the storm-UI's REST API (assuming there are > not too many of those). > > On Tue, Oct 11, 2016 at 7:15 AM, Bobby Evans <ev...@yahoo-inc.com.invalid> > wrote: > > > I agree that IMetricsConsumer is not good, but the reality is that all of > > the metrics system needs to be redone. The problem is that we ship an > > object as a metric. If I get an object I have no idea what it is hand > > hence no idea how to report it or what to do with it. What is more the > > common types we use in the metrics we provide are really not enough. For > > example CountMetric sends a Long. Well when I get it in the metrics > > consumer I have no idea if I should report it like a counter or if I > should > > report it like a gauge (something that every metrics system I have used > > wants to know). But then we support pre-aggregation of the metrics with > > IReducer so the number I get might be an average instead of either a > gauge > > or a counter, which no good metrics system will want to collect because I > > cannot aggregate it with anything else, the math just does not work. > > The proposal I have said before and I still believe is that we need to > put > > in place a parallel metrics API/system. We will deprecate all of > > https://git.corp.yahoo.com/storm/storm/tree/master- > > security/storm-core/src/jvm/backtype/storm/metric/api and create a new > > parallel one that provides an API similar to http://metrics.dropwizard. > > io/3.1.0/. I would even be fine in just using their API and exposing > > that to end users. Dropwizard has solved all of these problems already > and > > I don't see a reason to reinvent the wheel. I don't personally see a lot > > of value in trying to send all of the metrics through storm itslef. I am > > fine if we are able to support that, but it is far from a requirement. - > > Bobby > > > > On Monday, October 10, 2016 10:47 PM, S G <sg.online.em...@gmail.com> > > wrote: > > > > > > +1 > > We can probably start by opening a JIRA for this and adding a design > > approach for the same? > > I would like to help in the coding-effort for this. > > > > -SG > > > > On Mon, Oct 10, 2016 at 1:51 PM, P. Taylor Goetz <ptgo...@gmail.com> > > wrote: > > > > > I’ve been thinking about metrics lately, specifically the fact that > > people > > > tend to struggle with implementing a metrics consumer. (Like this one > > [1]). > > > > > > The IMetricsConsumer interface is pretty low level, and common > > > aggregations, calculations, etc. are left up to each individual > > > implementation. That seems like an area where further abstraction would > > > make it easier to support different back ends (Graphite, JMX, Splunk, > > etc.). > > > > > > My thought is to create an abstract IMetricsConsumer implementation > that > > > does common aggregations and calculations, and then delegates to a > > plugable > > > “metrics sink” implementation (e.g. “IMetricsSink”, etc.). That would > > > greatly simplify the effort required to integrate with various external > > > metrics systems. I know of at least a few users that would be > interested, > > > one is currently scraping the logs from LoggingMetricsConsumer and > > polling > > > the Storm REST API for their metrics. > > > > > > -Taylor > > > > > > [1] http://twocentsonsoftware.blogspot.co.il/2014/12/ > > > sending-out-storm-metrics.html > > > > > > > > > > On Oct 10, 2016, at 12:14 PM, Bobby Evans > <ev...@yahoo-inc.com.INVALID > > > > > > wrote: > > > > > > > > First of all the server exposes essentially the same interface that > the > > > IMetricsConsumer exposes. It mostly just adds a bunch of overhead in > the > > > middle to serialize out the objects send them over http to another > > process > > > which then has to deserialize them and process them. If you really > don't > > > need the metrics to show up on a special known box you can have that > > exact > > > same code running inside the metrics consumer without all of the > > overhead. > > > > The server/client are insecure, have to deal with thread issues that > a > > > normal IMetricsConsumer does not, and are not written to be robust (If > > the > > > HTTP server is down the consumer crashes and continues to crash until > the > > > server is brought back up). It was written very quickly for a test > > > situation and it honestly never crossed my mind that anyone would want > to > > > use it in production. > > > > > > > > - Bobby > > > > > > > > On Monday, October 10, 2016 10:59 AM, S G < > > sg.online.em...@gmail.com> > > > wrote: > > > > > > > > > > > > Thanks Bobby. > > > > > > > > If we write our own metrics consumer, how do we ensure that it is > > better > > > > than HttpForwardingMetricsServer? In other words, what aspects of the > > > > HttpForwardingMetricsServer > > > > should we avoid to make our own metrics consumer better and ready for > > > > production? > > > > > > > > Is versign/storm-graphite < > https://github.com/verisign/storm-graphite> > > > > production > > > > ready? > > > > > > > > Also, we should add a line about production-readiness of > > > > HttpForwardingMetricsServer > > > > in the documentation at http://storm.apache.org/ > > > releases/1.0.2/Metrics.html > > > > (We were just about to think seriously on using this for production > as > > we > > > > thought this to be the standard solution for metrics' consumption in > > 1.0+ > > > > version). > > > > > > > > -SG > > > > > > > > On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans <ev...@yahoo-inc.com> > > > wrote: > > > > > > > >> First of all there really are two different sets of metrics. One > set > > is > > > >> the topology metrics and the other set is the daemon metrics > (metrics > > > for > > > >> things like the ui and nimbus). The JmxPreparableReporter plugin > only > > > >> exposes daemon metrics not the topology metrics through JMX. > Exposing > > > >> topology metrics through JMX is a non trivial task. The current > > metrics > > > >> feature was not designed for this. We are in the process of trying > to > > > >> redesign the metrics system to allow for features like this, but it > is > > > >> still a ways off. > > > >> > > > >> - Bobby > > > >> > > > >> > > > >> On Saturday, October 8, 2016 11:39 AM, S G < > sg.online.em...@gmail.com > > > > > > >> wrote: > > > >> > > > >> > > > >> Thanks Bobby, > > > >> > > > >> We will need some kind of IMetricsConsumer to talk to telegraf. > > > >> Many other softwares like Solr, Elastic-Search, Cassandra etc. > provide > > > >> metrics through a URL making it very easy to consume by tools like > > > telegraf. > > > >> How about a IMetricsConsumer that will run on storm-ui and provide > the > > > >> metrics through a URL such as <storm-ui-host>/metrics ? > > > >> > > > >> Also, I see the following option in defaults.yaml: > > > >> #default storm daemon metrics reporter plugins > > > >> storm.daemon.metrics.reporter.plugins: > > > >> - "org.apache.storm.daemon.metrics.reporters. > > > JmxPreparableReporter" > > > >> > > > >> Is this a good option to use for converting metrics into JMX ? > > > >> > > > >> Thanks > > > >> SG > > > >> > > > >> > > > >> > > > >> > > > >> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans > > <ev...@yahoo-inc.com.invalid > > > > > > > >> wrote: > > > >> > > > >> HttpForwardingMetricsServer is a real hack intended really for > > tests. I > > > >> know I wrote it :). Please don't use it in production. You can > write > > > your > > > >> own IMetricesConsumer to do whatever you want to with the metrics. > > > >> https://github.com/apache/ storm/blob/master/storm-core/ > > > >> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer. java > > > >> <https://github.com/apache/storm/blob/master/storm-core/ > > > src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java> > > > >> > > > >> That is the correct way to get the data out. If you want to write a > > > >> bridge to JMX for this that might work, but going directly to > > telegraph > > > >> would probably be better. - Bobby > > > >> > > > >> On Thursday, October 6, 2016 1:43 PM, S G < > > > sg.online.em...@gmail.com> > > > >> wrote: > > > >> > > > >> > > > >> Hi, > > > >> > > > >> We want to use Telegraf ( > > > >> https://github.com/influxdata/ telegraf/tree/master/plugins > > > >> <https://github.com/influxdata/telegraf/tree/master/plugins>) for > > > getting > > > >> storm's metrics. > > > >> > > > >> But we do not want to add a HttpForwardingMetricsServer just to get > > the > > > >> metrics and send them to telegraf. > > > >> > > > >> Other option is to use Jolokia (https://jolokia.org/) that can read > > JMX > > > >> and > > > >> write into telegraf. > > > >> > > > >> Does storm report all its metrics (including those of custom > > > spouts/bolts) > > > >> into JMX? > > > >> Or spawning a HttpForwardingMetricsServer is the only option? > > > >> > > > >> Thanks > > > >> SG > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > >
