Jungtaek, Probably a filter config to whitelist and blacklist certain metrics. So that it will scale if there are too many workers and users can turn off certain metrics. Thanks, Harsha On Mon, May 2, 2016, at 06:19 AM, Stephen Powis wrote: > Oooh I'd love this as well! I really dig the ease of the metric > framework in storm and have all the metrics go thru one centralized > config. But as the number of storm hosts and number of tasks grow, > I've found that Graphite/Grafana has a hard time collecting up all the > relevant metrics across a lot of wildcarded keys for things like > hostnames and taskIds to properly display my graphs. > > On Sun, May 1, 2016 at 8:17 AM, Kevin Conaway > <kevin.a.cona...@gmail.com> wrote: >> One thing I would like to see added (if not already present) is the >> ability to register metrics that are not tied to a component. >> >> As of now, the only non-component metrics are reported by the >> SystemBolt pseudo-component which feels like a work-around. It >> reports JVM level metrics like GC time, heap size and other things >> that aren't associated with a given component. >> >> It would be great if application developers could expose similar >> metrics like this for things like connection pools and other JVM wide >> objects that aren't unique to a specific component. >> >> I don't think this is possible now, is it? >> >> On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim >> <kabh...@gmail.com> wrote: >>> Let me start sharing my thought. :) >>> >>> 1. Need to enrich docs about metrics / stats. >>> >>> In fact, I couldn't see the fact - topology stats are sampled by >>> default and sample rate is 0.05 - from the docs when I was newbie of >>> Apache Storm. It made me misleading and made me saying "Why there're >>> difference between the counts?". I also saw some mails from user@ >>> about same question. If we include this to guide doc that would be >>> better. >>> >>> And Metrics document page[1] seems not well written. I think it has >>> appropriate headings but lacks contents on each heading. >>> It should be addressed, and introducing some external metrics >>> consumer plugins (like storm-graphite[2] from Verisign) would be >>> great, too. >>> >>> 2. Need to increase sample rate or (ideally) no sampling at all. >>> >>> Let's postpone considering performance hit at this time. >>> Ideally, we expect precision of metrics gets better when we increase >>> sample rate. It affects non-gauge kinds of metrics which are >>> counter, and latency, and so on. >>> >>> Btw, I would like to hear about opinions on latency since I'm not an >>> expert. >>> Storm provides only average latency and it's indeed based on sample >>> rate. Do we feel OK with this? If not how much having also >>> percentiles can help us? >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <kabh...@gmail.com>님이 작성: >>>> Hi Storm users, >>>> >>>> I'm Jungtaek Lim, committer and PMC member of Apache Storm. >>>> >>>> If you subscribed dev@ mailing list, you may have seen that >>>> recently we're addressing the metrics feature on Apache Storm. >>>> >>>> For now, improvements are going forward based on current metrics >>>> feature. >>>> >>>> - Improve (Topology) MetricsConsumer[3] >>>> - Provide topology metrics in detail (metrics per each stream)[4] >>>> - (WIP) Introduce Cluster Metrics Consumer >>>> >>>> As I don't maintain large cluster for myself, I really want to >>>> collect the any ideas for improving, any inconveniences, use cases >>>> of Metrics with community members, so we're on the right way to go >>>> forward. >>>> >>>> Let's talk! >>>> >>>> Thanks in advance, >>>> Jungtaek Lim (HeartSaVioR) >> >> >> >> -- >> Kevin Conaway http://www.linkedin.com/pub/kevin-conaway/7/107/580/ >> https://github.com/kevinconaway >>
Links: 1. http://storm.apache.org/releases/1.0.0/Metrics.html 2. https://github.com/verisign/storm-graphite 3. https://issues.apache.org/jira/browse/STORM-1699 4. https://issues.apache.org/jira/browse/STORM-1719
