[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zach Aller updated CASSANDRA-7042:
----------------------------------
Attachment: tabledump_before_restart.txt
tabledump_after_restart.txt
This is a dump of the directory for the cf that has the issues as you can see
file count increases. It seems like files are not being deleted when they
should be. The before file is taken when cassandra was in a stopped state the
after is when cassandra had been started up again.
> Disk space growth until restart
> -------------------------------
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
> Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.<init>(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
--
This message was sent by Atlassian JIRA
(v6.2#6252)
