So if you kill a joining node, it will remain as a Fat Client (as of 0.7.x) in gossip. That is innocuous on its own. However if a node was decommissioned while that was joining, then the lingering Fat Client will also gossip about the node that was decommissioned. Then it will try to hint for that decommissioned node. That was the origin of our out of control hints.
So we killed the joining node, cleared it out to start fresh, and did a full cluster restart to clear out the gossip of the Fat Clients and the hints. That worked. On Aug 18, 2011, at 7:56 PM, Jonathan Ellis wrote: > I would assume it's because it thinks some node is down and is > creating hints for it. > > On Thu, Aug 18, 2011 at 6:31 PM, Jeremy Hanna > <jeremy.hanna1...@gmail.com> wrote: >> We're trying to bootstrap some new nodes and it appears when adding a new >> node that there is a lot of logging on hints being flushed and compacted. >> It's been taking about 75 minutes thus far to bootstrap for only about 10 GB >> of data. It's ballooned up to over 40 GB on the new node. I do 'ls -l >> $CASSANDRA/data/system | grep -i hint | wc -l' and it's over 2000. >> >> Any idea why this would be happening and why it's going crazy flushing these >> files? It's also trying to keep up and compacting them as well. >> >> Completed flushing $CASSANDRA/data/system/HintsColumnFamily-f-1740-Data.db >> >> Version 0.7.8 (upgrading from 0.7.6 right now) >> Number of nodes: 20 > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com
