Alex, In the above mentioned case I think monitoring the consumer lag will be helpful "kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager". Another approach I used before was to use metrics library to send consumer stats to ganglia and use nagios alert on those stats, If the consumed messages drops below certain threshold . -Harsha
On Fri, Oct 17, 2014, at 01:08 AM, Alex Objelean wrote: > @Neha Narkede > > Though monitoring the health of Kafka & Zookeeper clusters directly is > useful, it might not be enough. > Consider the following scenario: > > You have a client consuming messages. Zookeeper and kafka nodes are > healthy. The monitor says everything is ok. For some reason, a connection > between client and zookeeper fails (new firewall rule or connectivity > issues between different data centers, doesn't really matter). Since the > zookeeper and kafka nodes are still healthy, the monitor is still > reporting > OK. The client retries to reconnect indefinitely, but unless a human is > watching the logs, the problem is not visible by anybody. > > Having a listener which is invoked when the connection state is changed > is > something that is provided by other messaging frameworks. Having this > feature added to kafka would be a nice addition. > > Thanks, > Alex > > On 17 October 2014 10:55, Alex Objelean <alex.objel...@gmail.com> wrote: > > > @Otis thanks for your answer. > > > > Of course it is not about manually watching connection. We already have > > monitoring tools in place. But I would expect the kafka client to provide > > means for get notified about potential connectivity issues. > > Actual behavior is this: the client retries to reconnect indefinitely (I > > see errors in logs). What I need, is a kind of listener which would be > > invoked when the connection state change. In other words, once the client > > notice the problem and is trying to reconnect, our monitoring tools would > > reflect the connection state as soon as the problem appears. > > > > Thanks, > > Alex > > > > > > > > On 16 October 2014 13:09, Alex Objelean <alex.objel...@gmail.com> wrote: > > > >> Hi, > >> > >> I'm trying to monitor the kafka connection on the consumer side. In other > >> words, if the broker cluster is unavailable (or zookeer dies), I would like > >> to know about that problem as soon as possible. > >> Unfortunately, I didn't find anything useful to achieve that when using > >> kafka library. > >> Are there any suggestions about how to fix this issue? > >> > >> Thanks, > >> Alex > >> > > > >