The short answer to the problem you saw is monitor the disk space. Also monitor client side logs for errors. Running out of commit log space does not stop the node from doing reads, so it can still be considered up.
One nodes view of it's own UP'ness is not as important as the other nodes (or clients) view of it. For example... A node will appear UP in the ring view of another node if it is participating in gossip messages and it's application state is normal. But a node will appear UP in it's own view of the ring most of time (assuming not bootstrap, leaving etc and it has joined the ring). This applies even if it's gossip service has been disabled. To a client a node will appear down if it is not responding to RPC requests. But it could still be part of the cluster, appear UP to other nodes and be responding to read and/or write. So to monitor that a node is running in some form you can... - you should be monitoring the TP stats anyway, so you know the node is in some running state - check that you can connect as a client to each node and do some simple call. Either read/write or describe_ring() which will exec locallay or describe_schema_versions() which will call all live nodes. A read/write will only verify that the node can act as a coordinator, not that it can read/write it's self. - monitor the other nodes view of each node using nodetool ring. Now that i've written that I'm not 100% sold on it, but it will do for now :) Cheers ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 17 Jun 2011, at 10:25, Suan Aik Yeo wrote: > > Having a ping column can work if every key is replicated to every node. It > > would tell you the cluster is working, sort of. Once the number of nodes is > > greater than the RF, it tells you a subset of the nodes works. > > The way our check works is that each node checks itself, so in this context > we're not concerned about whether the cluster is "up", but that each > individual node is "up". > > So the symptoms I saw, the node actually going "down" etc, were probably due > to many different events happening at the time, and will be very hard to > recreate? > > On Thu, Jun 16, 2011 at 6:16 AM, aaron morton <aa...@thelastpickle.com> wrote: > > DEBUG 14:36:55,546 ... timed out > > Is logged when the coordinator times out waiting for the replicas to respond, > the timeout setting is rpc_timeout in the yaml file. This results in the > client getting a TimedOutException. > > AFAIK There is no global everything is good / bad flags to check. e.g. AFAIK > I node will not mark its self down if it runs out of disk space. So you need > to monitor the free disk space and alert on that. > > Having a ping column can work if every key is replicated to every node. It > would tell you the cluster is working, sort of. Once the number of nodes is > greater than the RF, it tells you a subset of the nodes works. > > If you google around you'll find discussions about monitoring with munin, > ganglia, cloud kick and Ops Centre. > > If you install mx4j you can access the JMX metrics via HTTP, > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 16 Jun 2011, at 10:38, Suan Aik Yeo wrote: > > > Here's a weird one... what's the best way to get a Cassandra node into a > > "half-crashed" state? > > > > We have a 3-node cluster running 0.7.5. A few days ago this happened > > organically to node1 - the partition the commitlog was on was 100% full and > > there was a "No space left on device" error, and after a while, although > > the cluster and node1 was still up, to the other nodes it was down, and > > messages like: > > DEBUG 14:36:55,546 ... timed out > > started to show up in its debug logs. > > > > We have a tool to indicate to the load balancer that a Cassandra node is > > down, but it didn't detect it that time. Now I'm having trouble > > purposefully getting the node back to that state, so that I can try other > > monitoring methods. I've tried to fill up the commitlog partition with > > other files, and although I get the "No space left on device" error, the > > node still doesn't go down and show the other symptoms it showed before. > > > > Also, if anyone could recommend a good way for a node itself to detect that > > its in such a state I'd be interested in that too. Currently what we're > > doing is making a "describe_cluster_name()" thrift call, but that still > > worked when the node was "down". I'm thinking of something like > > reading/writing to a fixed value in a keyspace as a check... Unfortunately > > Java-based solutions are out of the question. > > > > > > Thanks, > > Suan > >
