Hi,
I'm working on modelling the performance of storm topologies and I am interested in the worker processes (WP) send / receive queues and threads. My aim is to model a proposed physical plan (WP and Executor assignment) based on performance metrics from a currently running plan. Storm provides detailed metrics for the spouts and bolts. However, to accurately model the effect of a new physical plan I need to understand the performance of the WP send and receive systems. Obviously if a new plan puts a lot of high traffic executors in a given WP I want to be able to model the effect of this potential bottleneck. As I understand it, in the v1.1.1 code base, the WP receive queue and transfer (send) queues are both instances of org.apache.storm.utils.DisruptorQueue. These like the executor receive and send queues can produce metrics. However, as far as I can tell, the WP queue metrics are not reported (I have my own metrics consumer that records all available metrics). Is this right? I assume this is because the WP queues are not part of a component with a link to a metrics consumer? My question is weather it would be possible to obtain metrics from the WP receive and transfer queues? If this possible, then also how difficult it would be to obtain them (would this entail modifying the worker class). I can infer some of the metrics via other means, the send queue arrival rates for executors within the WP for example. However, it would be more accurate to have the actual metrics. Also, I am assuming metrics on processing times for the actual receive and send threads (moving tuple off queues) will not be available as this functionality seems to be distributed over several different classes? Thanks in advance, Thomas Cooper PhD Student Newcastle University, School of Computer Science W: http://www.tomcooper.org.uk | Twitter: @tomncooper<https://twitter.com/tomncooper>
