ghost node?

2011-05-26 Thread jonathan . colby
A node with IP 10.46.108.102 was removed from the cluster several days ago but the cassandra logs are full of these messages! Anyone know how to permanently remove this information? I\m beginning to think it is affecting the throughput of the live ndes. INFO [FlushWriter:1] 2011-05-27 04:28

Ghost node showing up in the ring

2011-03-22 Thread Alexis Lê-Quôc
Hi, I've seen some strange occurrence of a deleted node reappearing all of a sudden in the ring, which leads to my question: where is the ring structure maintained (memory with local copies?) and what prompts it to change? I appreciate any thoughts on the events below. I'm running 0.7.4 on 4 EC2

Re: Ghost node showing up in the ring

2011-03-23 Thread aaron morton
When the node starts it reads the stored token information from the LocationInfo CF in the System KS. It looks like the log message "is now part of the cluster" is only logged when an endpoint is added to a nodes view of the ring via gossip It is not logged when the endpoint is added during st

Re: Ghost node showing up in the ring

2011-03-23 Thread Alexis Lê-Quôc
I'm going to take a stab at a hypothesis: Sunday: I drain and decommission 2.3.4.193 *but* I forget to run node cleanup on the rest of the nodes. The ring looks clean but I did not see "Annoucing that ..." in the logs. Tuesday: the ghost node reappears on the ring for all nodes.

Re: Ghost node showing up in the ring

2011-03-23 Thread Brandon Williams
t; in the logs. > > Tuesday: the ghost node reappears on the ring for all nodes. > > Could this be caused by old hinted handoffs for 2.3.4.193 that were > processed at that time, causing the rest of the nodes to think that the > 2.3.4.193 is still present (albeit down)?