Hi,
I have a keyspace which is using about 19GB on most nodes in the cluster.
However on one node the data directory for the only table in that keyspace
now uses 52GB. It seems that old sstables are not removed after compaction.
Here is a view of the data directory Data files before compaction (I didn't
think to check sizes at the time):
$ ls /var/lib/cassandra/data/trackcontent/track_content/*Data.db
trackcontent-track_content-jb-30852-Data.db
trackcontent-track_content-jb-30866-Data.db
trackcontent-track_content-jb-37136-Data.db
trackcontent-track_content-jb-43371-Data.db
trackcontent-track_content-jb-44676-Data.db
trackcontent-track_content-jb-51845-Data.db
trackcontent-track_content-jb-55339-Data.db
trackcontent-track_content-jb-57353-Data.db
trackcontent-track_content-jb-57366-Data.db
trackcontent-track_content-jb-57367-Data.db
trackcontent-track_content-jb-57368-Data.db
Then when compaction was run I got the following in the logs:
INFO [CompactionExecutor:3546] 2015-07-06 12:01:04,246 CompactionTask.java
(line 120) Compacting [
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30852-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30866-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-37136-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-43371-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-44676-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-51845-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-55339-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57353-Data.db')
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57366-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57371-Data.db'),
]
INFO [CompactionExecutor:3546] 2015-07-06 13:34:31,458 CompactionTask.java
(line 296) Compacted 10 sstables to
[/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57372,].
36,182,766,725 bytes to 16,916,868,412 (~46% of original) in 5,607,211ms =
2.877221MB/s. 84,600,535 total partitions merged to 40,835,912. Partition
merge counts were {1:864528, 2:36318841, 3:3523852, 4:124091, 5:6051, 6:25,
}
But after compaction completes, the data directory still looks like:
$ ls -l /var/lib/cassandra/data/trackcontent/track_content/*Data.db
-rw-r--r-- 1 cassandra cassandra 16642164891 Jun 29 06:55
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30852-Data.db
-rw-r--r-- 1 cassandra cassandra 17216513377 Jun 30 08:36
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30866-Data.db
-rw-r--r-- 1 cassandra cassandra 813683923 Jun 30 12:03
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-37136-Data.db
-rw-r--r-- 1 cassandra cassandra 855070477 Jun 30 13:15
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-43371-Data.db
-rw-r--r-- 1 cassandra cassandra 209921645 Jun 30 13:28
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-44676-Data.db
-rw-r--r-- 1 cassandra cassandra 213532360 Jul 2 03:16
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-51845-Data.db
-rw-r--r-- 1 cassandra cassandra 14763933 Jul 2 07:58
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-52921-Data.db
-rw-r--r-- 1 cassandra cassandra 16955005 Jul 2 08:09
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-53037-Data.db
-rw-r--r-- 1 cassandra cassandra 61322156 Jul 2 23:34
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-55339-Data.db
-rw-r--r-- 1 cassandra cassandra 61898096 Jul 4 01:02
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57353-Data.db
-rw-r--r-- 1 cassandra cassandra 75912668 Jul 5 21:23
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57366-Data.db
-rw-r--r-- 1 cassandra cassandra 32747132 Jul 6 11:01
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57371-Data.db
-rw-r--r-- 1 cassandra cassandra 16916868412 Jul 6 13:34
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57372-Data.db
-rw-r--r-- 1 cassandra cassandra 8392714 Jul 6 13:24
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57373-Data.db
So it appears that the 10 sstables that were compacted
to
/var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57372
are still sitting around, with a couple of them being very large!
Any ideas for what I can check? Can I just delete the old sstables? (I'm
guessing not).
Jeff
