S G, I am slowly making progress and plan to share something this week, specifically for hooking up a default codahale reporter to the workers. Also the U of I students for the open source class have made progress on their rocksdb implementation for the default store (the stuff I am doing should feed into their store). I don't think any of this is going to be in 1.1.0 given that it hasn't really been looked at by others, let alone reviewed. Thanks Alessandro On Monday, November 28, 2016, 4:51:34 PM CST, S G <[email protected]> wrote:Hi,
I see https://issues.apache.org/jira/browse/STORM-2153 being open for this and a lot of good discussion happening too. There is also a talk for releasing 1.1.0 version soon. So I just wanted to check if any metric related improvements will be available in 1.1.0. It would be great to try out these new improvements. Thanks SG On Tue, Oct 18, 2016 at 8:57 PM, Abhishek Nigam <[email protected]> wrote: > Hi all, > > I'm Abhishek Nigam, one of the students from the group at the University of > Illinois; I'm really looking forward to working on this issue! We'll be > sure to keep you all updated as to how we progress. > > Cheers, > -- > Abhishek > > On Tue, Oct 18, 2016 at 1:23 PM Alessandro Bellina <[email protected] > > > wrote: > > > + Naren, Austin, and Abhishek, the students from the University of > > Illinois Open Source class. > > > > > > On Tuesday, October 11, 2016 11:48 PM, S G <[email protected]> > > wrote: > > > > > > Response on this important issue is pretty good. I am happily surprised > :) > > > > I want to mention our strategy for extracting metrics from other > products. > > We use jolokia_proxy (https://jolokia.org/features/proxy.html) to get > JMX > > beans from several softwares and feed them to telegraf. That way, we > avoid > > writing custom processors for all these different products. > > > > Telegraf is quickly becoming a standard for metrics data. (Just see the > > list of input plugins here: > > https://github.com/influxdata/telegraf/tree/master/plugins/inputs). And > it > > integrates well with several outputs too ( > > https://github.com/influxdata/telegraf/tree/master/plugins/outputs). > > > > Also since the metrics in JMX, they can be queried by jolokia-agent > > installed per node. This avoids the extra metrics-consumer bolt which can > > hit the topology throughtput too. > > > > So I cast my vote in favor of JMX-implementation of metrics. > > Other approaches are welcome to be discussed. > > > > On Tue, Oct 11, 2016 at 7:30 PM, Alessandro Bellina < > > [email protected]> wrote: > > > > > blockquote, div.yahoo_quoted { margin-left: 0 !important; > > border-left:1px > > > #715FFA solid !important; padding-left:1ex !important; > > > background-color:white !important; } Yeap that's a requirement from our > > > perspective (working through this list). > > > Sure I think as usual we can start with master with an eye for what > would > > > need to be back ported. > > > Sent from Yahoo Mail for iPhone > > > > > > > > > On Tuesday, October 11, 2016, 8:50 PM, P. Taylor Goetz < > > [email protected]> > > > wrote: > > > > > > I hope I didn't come across as overly critical. You did the best with > > what > > > you had to work with. Which isn't pretty. > > > > > > We could potentially do a parallel metrics API in 1.1, 1.2, or master > and > > > still stay close to semantic versioning...? > > > > > > -Taylor > > > > > > > On Oct 11, 2016, at 9:28 PM, Jungtaek Lim <[email protected]> wrote: > > > > > > > > Yeah I admit that configuration flag was bad for me also, but I have > no > > > > alternatives. Only way to avoid struggling with design limitation is > > > revamp > > > > / redesign. > > > > Thanks S G for exposing willingness of volunteer and great news > > > Alessandro > > > > for that project. > > > > Alessandro, could you forward the upcoming news for the project to > dev@ > > > > list? > > > > > > > > - Jungtaek Lim (HeartSaVioR) > > > > > > > > 2016년 10월 12일 (수) 오전 10:22, P. Taylor Goetz <[email protected]>님이 > 작성: > > > > > > > >> I was thinking on a smaller scale in terms of effort, but the more I > > > think > > > >> about it, the more supportive I would be of a full revamp (new API) > > for > > > >> metrics based on Coda Hale's metrics library. It's proven and > stable. > > > I've > > > >> used it many times. I think either approach would be roughly the > same > > > >> amount of work. > > > >> > > > >> Some of the metrics API improvements in the 1.1.x branch are nice, > but > > > >> IMHO are lipstick on a pig. > > > >> > > > >> With apologies to Jungtaek, who has done amazing work all across the > > > >> codebase, I'm a little squeamish about the proposed change to > metrics > > > that > > > >> changes the consumer API based on a configuration flag (don't know > the > > > PR > > > >> number offhand). > > > >> > > > >> I'm +1 for moving in this direction (revamped metrics). Let's end > the > > > >> metrics pain. > > > >> > > > >> -Taylor > > > >> > > > >>> On Oct 11, 2016, at 10:15 AM, Bobby Evans < > > [email protected] > > > > > > > >> wrote: > > > >>> > > > >>> I agree that IMetricsConsumer is not good, but the reality is that > > all > > > >> of the metrics system needs to be redone. The problem is that we > ship > > > an > > > >> object as a metric. If I get an object I have no idea what it is > hand > > > >> hence no idea how to report it or what to do with it. What is more > > the > > > >> common types we use in the metrics we provide are really not enough. > > > For > > > >> example CountMetric sends a Long. Well when I get it in the metrics > > > >> consumer I have no idea if I should report it like a counter or if I > > > should > > > >> report it like a gauge (something that every metrics system I have > > used > > > >> wants to know). But then we support pre-aggregation of the metrics > > with > > > >> IReducer so the number I get might be an average instead of either a > > > gauge > > > >> or a counter, which no good metrics system will want to collect > > because > > > I > > > >> cannot aggregate it with anything else, the math just does not work. > > > >>> The proposal I have said before and I still believe is that we need > > to > > > >> put in place a parallel metrics API/system. We will deprecate all > of > > > >> https://git.corp.yahoo.com/storm/storm/tree/master- > > > security/storm-core/src/jvm/backtype/storm/metric/api > > > >> and create a new parallel one that provides an API similar to > > > >> http://metrics.dropwizard.io/3.1.0/. > > <http://metrics.dropwizard.io/3.1.0/> I would even be fine in just > > > using > > > >> their API and exposing that to end users. Dropwizard has solved all > > of > > > >> these problems already and I don't see a reason to reinvent the > wheel. > > > I > > > >> don't personally see a lot of value in trying to send all of the > > metrics > > > >> through storm itslef. I am fine if we are able to support that, but > > it > > > is > > > >> far from a requirement. - Bobby > > > >>> > > > >>> On Monday, October 10, 2016 10:47 PM, S G < > > [email protected]> > > > >> wrote: > > > >>> > > > >>> > > > >>> +1 > > > >>> We can probably start by opening a JIRA for this and adding a > design > > > >>> approach for the same? > > > >>> I would like to help in the coding-effort for this. > > > >>> > > > >>> -SG > > > >>> > > > >>>> On Mon, Oct 10, 2016 at 1:51 PM, P. Taylor Goetz < > [email protected] > > > > > > >> wrote: > > > >>>> > > > >>>> I’ve been thinking about metrics lately, specifically the fact > that > > > >> people > > > >>>> tend to struggle with implementing a metrics consumer. (Like this > > one > > > >> [1]). > > > >>>> > > > >>>> The IMetricsConsumer interface is pretty low level, and common > > > >>>> aggregations, calculations, etc. are left up to each individual > > > >>>> implementation. That seems like an area where further abstraction > > > would > > > >>>> make it easier to support different back ends (Graphite, JMX, > > Splunk, > > > >> etc.). > > > >>>> > > > >>>> My thought is to create an abstract IMetricsConsumer > implementation > > > that > > > >>>> does common aggregations and calculations, and then delegates to a > > > >> plugable > > > >>>> “metrics sink” implementation (e.g. “IMetricsSink”, etc.). That > > would > > > >>>> greatly simplify the effort required to integrate with various > > > external > > > >>>> metrics systems. I know of at least a few users that would be > > > >> interested, > > > >>>> one is currently scraping the logs from LoggingMetricsConsumer and > > > >> polling > > > >>>> the Storm REST API for their metrics. > > > >>>> > > > >>>> -Taylor > > > >>>> > > > >>>> [1] http://twocentsonsoftware.blogspot.co.il/2014/12/ > > > >>>> sending-out-storm-metrics.html > > > >>>> > > > >>>> > > > >>>>> On Oct 10, 2016, at 12:14 PM, Bobby Evans > > > <[email protected] > > > >>> > > > >>>> wrote: > > > >>>>> > > > >>>>> First of all the server exposes essentially the same interface > that > > > the > > > >>>> IMetricsConsumer exposes. It mostly just adds a bunch of overhead > > in > > > >> the > > > >>>> middle to serialize out the objects send them over http to another > > > >> process > > > >>>> which then has to deserialize them and process them. If you > really > > > >> don't > > > >>>> need the metrics to show up on a special known box you can have > that > > > >> exact > > > >>>> same code running inside the metrics consumer without all of the > > > >> overhead. > > > >>>>> The server/client are insecure, have to deal with thread issues > > that > > > a > > > >>>> normal IMetricsConsumer does not, and are not written to be robust > > (If > > > >> the > > > >>>> HTTP server is down the consumer crashes and continues to crash > > until > > > >> the > > > >>>> server is brought back up). It was written very quickly for a > test > > > >>>> situation and it honestly never crossed my mind that anyone would > > want > > > >> to > > > >>>> use it in production. > > > >>>>> > > > >>>>> - Bobby > > > >>>>> > > > >>>>> On Monday, October 10, 2016 10:59 AM, S G < > > > >> [email protected]> > > > >>>> wrote: > > > >>>>> > > > >>>>> > > > >>>>> Thanks Bobby. > > > >>>>> > > > >>>>> If we write our own metrics consumer, how do we ensure that it is > > > >> better > > > >>>>> than HttpForwardingMetricsServer? In other words, what aspects of > > the > > > >>>>> HttpForwardingMetricsServer > > > >>>>> should we avoid to make our own metrics consumer better and ready > > for > > > >>>>> production? > > > >>>>> > > > >>>>> Is versign/storm-graphite <https://github.com/verisign/ > > > storm-graphite> > > > >>>>> production > > > >>>>> ready? > > > >>>>> > > > >>>>> Also, we should add a line about production-readiness of > > > >>>>> HttpForwardingMetricsServer > > > >>>>> in the documentation at http://storm.apache.org/ > > > >>>> releases/1.0.2/Metrics.html > > > >>>>> (We were just about to think seriously on using this for > production > > > as > > > >> we > > > >>>>> thought this to be the standard solution for metrics' consumption > > in > > > >> 1.0+ > > > >>>>> version). > > > >>>>> > > > >>>>> -SG > > > >>>>> > > > >>>>> On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans < > [email protected]> > > > >>>> wrote: > > > >>>>> > > > >>>>>> First of all there really are two different sets of metrics. > One > > > set > > > >> is > > > >>>>>> the topology metrics and the other set is the daemon metrics > > > (metrics > > > >>>> for > > > >>>>>> things like the ui and nimbus). The JmxPreparableReporter > plugin > > > only > > > >>>>>> exposes daemon metrics not the topology metrics through JMX. > > > Exposing > > > >>>>>> topology metrics through JMX is a non trivial task. The current > > > >> metrics > > > >>>>>> feature was not designed for this. We are in the process of > > trying > > > to > > > >>>>>> redesign the metrics system to allow for features like this, but > > it > > > is > > > >>>>>> still a ways off. > > > >>>>>> > > > >>>>>> - Bobby > > > >>>>>> > > > >>>>>> > > > >>>>>> On Saturday, October 8, 2016 11:39 AM, S G < > > > [email protected] > > > >>> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>> > > > >>>>>> Thanks Bobby, > > > >>>>>> > > > >>>>>> We will need some kind of IMetricsConsumer to talk to telegraf. > > > >>>>>> Many other softwares like Solr, Elastic-Search, Cassandra etc. > > > provide > > > >>>>>> metrics through a URL making it very easy to consume by tools > like > > > >>>> telegraf. > > > >>>>>> How about a IMetricsConsumer that will run on storm-ui and > provide > > > the > > > >>>>>> metrics through a URL such as <storm-ui-host>/metrics ? > > > >>>>>> > > > >>>>>> Also, I see the following option in defaults.yaml: > > > >>>>>> #default storm daemon metrics reporter plugins > > > >>>>>> storm.daemon.metrics.reporter.plugins: > > > >>>>>> - "org.apache.storm.daemon.metrics.reporters. > > > >>>> JmxPreparableReporter" > > > >>>>>> > > > >>>>>> Is this a good option to use for converting metrics into JMX ? > > > >>>>>> > > > >>>>>> Thanks > > > >>>>>> SG > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans > > > >> <[email protected] > > > >>>>> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>> HttpForwardingMetricsServer is a real hack intended really for > > > >> tests. I > > > >>>>>> know I wrote it :). Please don't use it in production. You can > > > write > > > >>>> your > > > >>>>>> own IMetricesConsumer to do whatever you want to with the > metrics. > > > >>>>>> https://github.com/apache/ storm/blob/master/storm-core/ > > > >>>>>> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer. java > > > >>>>>> <https://github.com/apache/storm/blob/master/storm-core/ > > > >>>> src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java> > > > >>>>>> > > > >>>>>> That is the correct way to get the data out. If you want to > > write a > > > >>>>>> bridge to JMX for this that might work, but going directly to > > > >> telegraph > > > >>>>>> would probably be better. - Bobby > > > >>>>>> > > > >>>>>> On Thursday, October 6, 2016 1:43 PM, S G < > > > >>>> [email protected]> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>> > > > >>>>>> Hi, > > > >>>>>> > > > >>>>>> We want to use Telegraf ( > > > >>>>>> https://github.com/influxdata/ telegraf/tree/master/plugins > > > >>>>>> <https://github.com/influxdata/telegraf/tree/master/plugins>) > for > > > >>>> getting > > > >>>>>> storm's metrics. > > > >>>>>> > > > >>>>>> But we do not want to add a HttpForwardingMetricsServer just to > > get > > > >> the > > > >>>>>> metrics and send them to telegraf. > > > >>>>>> > > > >>>>>> Other option is to use Jolokia (https://jolokia.org/) that can > > read > > > >> JMX > > > >>>>>> and > > > >>>>>> write into telegraf. > > > >>>>>> > > > >>>>>> Does storm report all its metrics (including those of custom > > > >>>> spouts/bolts) > > > >>>>>> into JMX? > > > >>>>>> Or spawning a HttpForwardingMetricsServer is the only option? > > > >>>>>> > > > >>>>>> Thanks > > > >>>>>> SG > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>>> > > > >>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > > > > > > > >
