Re: LoggingMetricsConsumer
Thanks for your response John, Could you explain to me how this would work when I am using Trident? Since with Trident bolts are abstracted away from the user, how might I configure my own MetricsConsumerBolt / debug it to figure out why it isn't calling handleDataPoints() ? my metricsConsumer's prepare() and cleanup() methods are called, but never the handleDataPoints() function. Thanks On Thu, Sep 25, 2014 at 1:33 PM, John Reilly j...@inconspicuous.org wrote: It is called by the MetricsConsumerBolt which is created by storm when a worker is starting up. When you define a metrics consumer, you should see metrics output every 60 seconds. Also, I think the metrics code was only introduced in 0.9.0 so you would need to be running at least that version. One other issue I ran into when registering a metrics consumer was that the config args I was passing initially because of serialization issues. When I used a Map instead of a serializable class that I created, it worked fine. For the packaged LoggingMetricsConsumer there is no config though. I think I did run into an issue when trying to configure both LoggingMetricsConsumer and my own metrics consumer. IIRC, if initialization of my consumer failed, the LoggingMetricsConsumer would also failit may have depended on the order that I was registering them in, but I don't remember exactly. Cheers, John On Thu, Sep 25, 2014 at 10:07 AM, Raphael Hsieh raffihs...@gmail.com wrote: Hi, I've been trying to figure out why registerinfg a LoggingMetricsConsumer isn't working for me. I've been able to figure out that it is indeed running, however the handleDataPoints() function is never called. Can someone explain to me how this class is used by Storm in order to log metrics? When is the handleDataPoints function called? Thanks -- Raphael Hsieh -- Raphael Hsieh
Re: LoggingMetricsConsumer
I have not used Trident, but I think the metrics should be handled the same for trident bolts as normal rich bolts. You can confirm that the metrics bolt is there by looking in the storm ui. Click on the topology and go down to the bottom of the page and click on Show System Stats and it will show you the metrics consumers under the list of bolts. Executed, Acked, Process Latency and Execute Latency are non-zero values for me for my 3 metrics consumers. Hopefully that helps a little. Even if you did not register any metrics you should see the built in metrics for each bolt and spout so I'm not sure why you don't see any (assuming you are using the default 60s interval for the built in metrics). On Fri, Sep 26, 2014 at 8:44 AM, Raphael Hsieh raffihs...@gmail.com wrote: Thanks for your response John, Could you explain to me how this would work when I am using Trident? Since with Trident bolts are abstracted away from the user, how might I configure my own MetricsConsumerBolt / debug it to figure out why it isn't calling handleDataPoints() ? my metricsConsumer's prepare() and cleanup() methods are called, but never the handleDataPoints() function. Thanks On Thu, Sep 25, 2014 at 1:33 PM, John Reilly j...@inconspicuous.org wrote: It is called by the MetricsConsumerBolt which is created by storm when a worker is starting up. When you define a metrics consumer, you should see metrics output every 60 seconds. Also, I think the metrics code was only introduced in 0.9.0 so you would need to be running at least that version. One other issue I ran into when registering a metrics consumer was that the config args I was passing initially because of serialization issues. When I used a Map instead of a serializable class that I created, it worked fine. For the packaged LoggingMetricsConsumer there is no config though. I think I did run into an issue when trying to configure both LoggingMetricsConsumer and my own metrics consumer. IIRC, if initialization of my consumer failed, the LoggingMetricsConsumer would also failit may have depended on the order that I was registering them in, but I don't remember exactly. Cheers, John On Thu, Sep 25, 2014 at 10:07 AM, Raphael Hsieh raffihs...@gmail.com wrote: Hi, I've been trying to figure out why registerinfg a LoggingMetricsConsumer isn't working for me. I've been able to figure out that it is indeed running, however the handleDataPoints() function is never called. Can someone explain to me how this class is used by Storm in order to log metrics? When is the handleDataPoints function called? Thanks -- Raphael Hsieh -- Raphael Hsieh
LoggingMetricsConsumer
Hi, I've been trying to figure out why registerinfg a LoggingMetricsConsumer isn't working for me. I've been able to figure out that it is indeed running, however the handleDataPoints() function is never called. Can someone explain to me how this class is used by Storm in order to log metrics? When is the handleDataPoints function called? Thanks -- Raphael Hsieh
Re: LoggingMetricsConsumer
It is called by the MetricsConsumerBolt which is created by storm when a worker is starting up. When you define a metrics consumer, you should see metrics output every 60 seconds. Also, I think the metrics code was only introduced in 0.9.0 so you would need to be running at least that version. One other issue I ran into when registering a metrics consumer was that the config args I was passing initially because of serialization issues. When I used a Map instead of a serializable class that I created, it worked fine. For the packaged LoggingMetricsConsumer there is no config though. I think I did run into an issue when trying to configure both LoggingMetricsConsumer and my own metrics consumer. IIRC, if initialization of my consumer failed, the LoggingMetricsConsumer would also failit may have depended on the order that I was registering them in, but I don't remember exactly. Cheers, John On Thu, Sep 25, 2014 at 10:07 AM, Raphael Hsieh raffihs...@gmail.com wrote: Hi, I've been trying to figure out why registerinfg a LoggingMetricsConsumer isn't working for me. I've been able to figure out that it is indeed running, however the handleDataPoints() function is never called. Can someone explain to me how this class is used by Storm in order to log metrics? When is the handleDataPoints function called? Thanks -- Raphael Hsieh
