Thanks Cody and Bobby for the explanation. Cody, I took a look at design doc and looks promising, especially it doesn't do sampling when metric type is 'counter'. As far as I heard (I didn't try it) it becomes huge performance hit in Apache Storm when we change sample rate to 1.0. Could you guide the entry point of metric feature in JStorm to dig into?
And just a curiosity, did you consider extracting metric feature (which is done with TopologyMasters and Nimbuses) into separate component? I understood your mention to 'metrics server' as separate component, but after seeing design doc, feature seems to be implemented on Nimbus. Thanks, Jungtaek Lim (HeartSaVioR) 2016년 3월 19일 (토) 오전 1:25, Cody Innowhere <e.neve...@gmail.com>님이 작성: > JStorm has provided a MetricUploader interface, which is similar to > IMetricsConsumer in storm, and the underlying implementation is pluggable, > you can use HBase, or any other KV store that supports timeline queries or > even a database(maybe for it's a small cluster). We provide model classes > in jstorm-core, as to what kinds of metrics data need to be stored, it's > totally up to the detailed implementation. Our internal implementation uses > OTS, which is a product of aliyun (https://www.aliyun.com/product/ots/), > but it's easy to adapt to other implementations. > > On Fri, Mar 18, 2016 at 11:52 PM, Bobby Evans <ev...@yahoo-inc.com.invalid > > > wrote: > > > Yes we originally wanted to try and use the Hadoop Timeline Server for > > storm metrics feedback to nimbus + UI + history like server. But it was > > not stable at the time, so we stopped. For the sake of playing nicely > with > > the rest of the big data ecosystem I would like to see us support it as > an > > option for metrics collection/query, but until the timeline server v2 is > > ready and released. For me the important thing is that we have a decent > > time series DB that comes with storm by default and is pluggable so we > can > > replace it with something else that has similar capabilities in the > future. > > - Bobby > > > > On Friday, March 18, 2016 10:39 AM, Cody Innowhere < > > e.neve...@gmail.com> wrote: > > > > > > It's actually in Phase 2 of porting JStorm, but I'm absolutely ok to > > discuss this in advance. > > > > On Fri, Mar 18, 2016 at 11:31 PM, Cody Innowhere <e.neve...@gmail.com> > > wrote: > > > > > Yes it's already in production. > > > The implementation basically follows the design document in > > > https://issues.apache.org/jira/browse/STORM-1329, you can take a look > > > first and feel free to ask questions. > > > > > > On Fri, Mar 18, 2016 at 10:19 PM, Jungtaek Lim <kabh...@gmail.com> > > wrote: > > > > > >> Hi, > > >> > > >> I got something to do with metrics so I'm seeking the pull requests > > which > > >> addresses metrics. > > >> And at #753 <https://github.com/apache/storm/pull/753> I found Cody > > said > > >> we > > >> (maybe it means Alibaba team) are currently working on Metrics Server. > > >> (I also found comment which said there was some talk while ago around > > >> integrating Hadoop timeline server. Seems like no one came up with the > > >> result, and I prefer to avoid big dependency so I'm in favor of > Metrics > > >> Server for now.) > > >> > > >> I think that would improve metrics feature of Storm much better, so > I'd > > >> like to see how the work is going. Sure it's only when there's no > issue > > >> for > > >> you to work transparently. I just would like to prevent duplication of > > >> work, and would like to help if needed and possible. > > >> > > >> Thanks, > > >> Jungtaek Lim (HeartSaVioR) > > >> > > > > > > > > > > > > > > >
