[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106836#comment-14106836
]
Benedict commented on CASSANDRA-7042:
-------------------------------------
The default concurrent_compactors is most likely too high for machines with
many CPUs and modest disk throughput. What disk layout do you have?
> Disk space growth until restart
> -------------------------------
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
> Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.<init>(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
