So I have looked at the cluster from - Cassandra-client - describe cluster => shows correctly - 3 nodes - used the StorageService - JMX bean =>UnreachableNodes - shows 0
If all these show the correct ring state, why are hints being maintained, looks like that is the only way to find out about "phantom" nodes. On Wed, Aug 24, 2011 at 8:01 AM, Anand Somani <meatfor...@gmail.com> wrote: > So, I restarted the cluster (not rolling), but it is still maintaining > hints for the IP's that are no longer part of the ring. nodetool ring shows > things correctly (as only 3 nodes). > When I check thru the jmx hintedhandoff manager, it shows it is maintaining > the hints for those non existent IP's. So the question is > - How can I remove these IP permanently, so hints do not get saved? > - Not all nodes see the same list of IP's > > > > > On Sun, Aug 21, 2011 at 3:10 PM, aaron morton <aa...@thelastpickle.com>wrote: > >> Yup, you can check the what HH is doing via JMX. >> >> there is a bug in 0.7 that can result in log files not been deleted >> https://issues.apache.org/jira/browse/CASSANDRA-2829 >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 22/08/2011, at 4:56 AM, Anand Somani wrote: >> >> We have a lot of space on /data, and looks like it was flushing data fine >> from file timestamps. >> >> We did have a bit of goofup with IP's when bringing up a down node (and >> the commit files have been around since then). Wonder if that is what >> triggered it and we have a bunch of hinted handoff's being backed up. >> >> For hinted hand off - how do I check if the nodes are collecting hints ( I >> do have it turned on). I noticed console bean HintedHandManager, is that the >> only way to find out. >> >> On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller < >> peter.schul...@infidyne.com> wrote: >> >>> > When does the actual commit-data file get deleted. >>> > >>> > The flush interval on all my memtables is 60 minutes >>> >>> They *should* be getting deleted when they no longer contain any data >>> that has not been flushed to disk. Are flushes definitely still >>> happening? Is it possible flushing has started failing (e.g. out of >>> disk)? >>> >>> The only way I can think of over nodes directly affecting the commit >>> log size on your node would be e.g. hinted handoff resulting in burst >>> of writes. >>> >>> -- >>> / Peter Schuller (@scode on twitter) >>> >> >> >> >
