Hello and thank you for your answers, @Sushant: Thank you for your suggestion. I will try to see if the values make more sense with nano-time.
@Ohidiost: Thank you for explaining the way metrics work in Nimbus. I will try to apply your code and set up ganglia to my cluster. However, I wanted to ask if you know how does Nimbus retrieve those stats in the first place. If those stats exist in each Bolt, I want to first check if I can get them, before I ask them from Nimbus. Thank you all again for your responses. Cheers, Nick 2015-04-22 22:49 GMT-04:00 Ohidiost Nal <[email protected]>: > Storm UI retrieves those stats from Nimbus. They are maintained for 10 > minutes, 1hr, 3 hr etc. windows. The bolts or any other code can access the > same metrics using the Nimbus Thrift API. > I recently added a ganglia plugin to collect historical Storm topology > stats from Nimbus (as the sliding window stats are not really very helpful > for measuring the performance over time). > https://github.com/ganglia/gmond_python_modules/tree/master/storm_topology > > That may be useful to understand how to get the process, execute latency > etc. for Storm spout/bolt etc. > > On Thu, Apr 23, 2015 at 3:10 AM, Nick R. Katsipoulakis < > [email protected]> wrote: > >> Hello all, >> >> I have been trying to calculate, as precisely as possible, how much time >> a tuple spends waiting in the input queue of a Bolt until it is provided in >> the BaseRichBolt.execute() function. >> >> The way I try to get this wait time (latency) is by adding a timestamp >> attribute in the tuple, the time it gets produced (I call Java's >> System.currentTimeInMillis()). When it gets picked up from a downstream >> Bolt (call of execute() function), I retrieve the timestamp attribute and >> calculate the difference in milliseconds from the current timestamp. >> However, due to clock skewness in my local cluster (I guess that this is >> the cause), I usually retrieve difference values that do not make sense >> (sometimes even negative wait times). I have previously tried NTP but I >> still do not get values that make sense. >> >> When I check the Storm UI reported metrics, I see that there two latency >> metrics: Process latency and one more. Is there any way I can retrieve >> those from the java code inside a Bolt's execute method? Is there a better >> way to get a more precise latency metric from inside the execute() function >> of a Bolt? >> >> Thanks, >> Nick >> >> -- >> Nikolaos Romanos Katsipoulakis, >> University of Pittsburgh, PhD candidate >> > > > > -- > Regards, > Nipun Talukdar > Bangalore > -- Nikolaos Romanos Katsipoulakis, University of Pittsburgh, PhD candidate
