Re: How to monitor a Trident/Kafka Topology?

[Otis Gospodnetic]

Hi Guillaume,

We use our own SPM  to monitor our Kafka clusters
(producers, brokers, and consumers  you can actually see all their
metrics at [0]) and, while we are currently not using Storm, SPM itself can
monitor it (see [1]), which means you can have your Kafka and Storm metrics
and alerts side by side, on the same dashboard(s).  We do this with other
parts of our stack (e.g. Kafka + HBase or Kafka + Elasticsearch) and it
works nicely.  Oh, and you could still capture other metrics with Coda
Hale's Metrics and graph that (see [2]).

[0] https://apps.sematext.com/demo
[1]
http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
[2]
https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics#CustomMetrics-CodaHale(akaYammer)MetricsReporter

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/



On Mon, Sep 15, 2014 at 9:53 AM, Guillaume Boesch 
wrote:

> Thank you for your response, I was thinking about using CodaHale's Metrics
> for my Trident application but I wanted to be sure that Storm UI and the
> REST API were useless in my case.
>
> 2014-09-13 6:08 GMT+02:00 Cyrille Karmann :
>
>> Metrics are a facility designed for Storm, and has no knowledge of
>> Trident, which is an abstraction on top of Storm. So messages will appear
>> that are used by Trident under the hood.
>> To have statistics for Trident, you are a bit on your own. But it is easy
>> to use a general metrics library like
>> https://dropwizard.github.io/metrics to obtain information from a worker
>> via JMX for example.
>>
>> Cyrille
>>
>
>

Re: How to monitor a Trident/Kafka Topology?

[Guillaume Boesch]

Thank you for your response, I was thinking about using CodaHale's Metrics
for my Trident application but I wanted to be sure that Storm UI and the
REST API were useless in my case.

2014-09-13 6:08 GMT+02:00 Cyrille Karmann :

> Metrics are a facility designed for Storm, and has no knowledge of
> Trident, which is an abstraction on top of Storm. So messages will appear
> that are used by Trident under the hood.
> To have statistics for Trident, you are a bit on your own. But it is easy
> to use a general metrics library like https://dropwizard.github.io/metrics
> to obtain information from a worker via JMX for example.
>
> Cyrille
>

Re: How to monitor a Trident/Kafka Topology?

[Cyrille Karmann]

Metrics are a facility designed for Storm, and has no knowledge of Trident,
which is an abstraction on top of Storm. So messages will appear that are
used by Trident under the hood.
To have statistics for Trident, you are a bit on your own. But it is easy
to use a general metrics library like https://dropwizard.github.io/metrics
to obtain information from a worker via JMX for example.

Cyrille