Here is our cluster which has 10 billion rows on 6 nodes and about 1.2TB [root@sdi-ci ~]# clush -g datanodes du -sh /opt/datastore/commitlog a5: 1.1G /opt/datastore/commitlog a3: 1.1G /opt/datastore/commitlog a1: 1.1G /opt/datastore/commitlog a2: 1006M /opt/datastore/commitlog a4: 1.1G /opt/datastore/commitlog a6: 1.1G /opt/datastore/commitlog
If you run nodetool drain on a node, you can wipe the commit logs after that(we QA tested this but don't take my word for it). We also found out drain was moving data to the sstables but did not seem to delete from the commit log at all as commit log space used remained the same after a drain. We did not fully test removing the commit log files so you should try to do that yourself with a test in QA. Later, Dean From: a k <kumaramit.ex...@gmail.com<mailto:kumaramit.ex...@gmail.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Tuesday, March 12, 2013 10:46 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: commitlog -deleted keyspaces. We are running a 4 node cluster version 1.1.0 and our commit logs seem to be ever growing. We have a total about 250 GB per node in the keyspaces/column families and the commit logs are at about 30 GB.There have been several deletions of keyspaces in our setup and I am concerned about a few things. First being the size of the commit logs, we have not modified the "commitlog_total_space_in_mb" in yaml so I assume it is the default 4092MB, We don't have "memtable_flush_after_mins" setting either. Is this sane? Why would the size of the commit logs be so large (I am worried about the startup time, when it has to replay the commit logs) and why would the default size of 4092 MB not be enforced. Would it cause us trouble when we upgrade to 1.2? Another thing I have noticed is that upon restarts, the old keyspaces that were deleted re-appear although with less data, I would imagine that has nothing to do with the commit logs. Can I safely delete the commitlogs after the nodetool flush? Many thanks Amit
