Hi Thomas, Could you find '__transfer' under '__system' component? '__system' denotes SystemBolt which will be launched for each worker, and worker transfer queue is registered to publish metrics in '__system' component. You can also get metrics from '__recv-iconnection' under '__system' component which is a set of simple metrics for Netty server. There's no explicit disruptor queue for receive queue on worker. Receive callback function will route received messages to executors' receive queues. (They're disruptor queues.)
Hope this helps. Thanks, Jungtaek Lim (HeartSaVioR) 2017년 10월 26일 (목) 오후 9:12, Thomas Cooper (PGR) <t.coo...@newcastle.ac.uk>님이 작성: > Hi, > > > I'm working on modelling the performance of storm topologies and I am > interested in the worker processes (WP) send / receive queues and threads. > > > My aim is to model a proposed physical plan (WP and Executor assignment) > based on performance metrics from a currently running plan. > > > Storm provides detailed metrics for the spouts and bolts. However, to > accurately model the effect of a new physical plan I need to understand the > performance of the WP send and receive systems. Obviously if a new plan > puts a lot of high traffic executors in a given WP I want to be able to > model the effect of this potential bottleneck. > > > As I understand it, in the v1.1.1 code base, the WP receive queue and > transfer (send) queues are both instances of > *org.apache.storm.utils.DisruptorQueue*. These like the executor receive > and send queues can produce metrics. However, as far as I can tell, the WP > queue metrics are not reported (I have my own metrics consumer that records > all available metrics). Is this right? I assume this is because the WP > queues are not part of a component with a link to a metrics consumer? > > > My question is weather it would be possible to obtain metrics from the WP > receive and transfer queues? If this possible, then also how difficult it > would be to obtain them (would this entail modifying the worker class). I > can infer some of the metrics via other means, the send queue arrival rates > for executors within the WP for example. However, it would be more accurate > to have the actual metrics. > > > Also, I am assuming metrics on processing times for the actual receive and > send threads (moving tuple off queues) will not be available as this > functionality seems to be distributed over several different classes? > > > Thanks in advance, > > > Thomas Cooper > PhD Student > Newcastle University, School of Computer Science > W: http://www.tomcooper.org.uk | Twitter: @tomncooper > <https://twitter.com/tomncooper> >
