[ https://issues.apache.org/jira/browse/CASSANDRA-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Stupp updated CASSANDRA-9036: ------------------------------------ Fix Version/s: 2.0.14 2.1.4 3.0 > "disk full" when running cleanup (on a far from full disk) > ---------------------------------------------------------- > > Key: CASSANDRA-9036 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9036 > Project: Cassandra > Issue Type: Bug > Reporter: Erik Forsberg > Assignee: Robert Stupp > Fix For: 3.0, 2.1.4, 2.0.14 > > Attachments: 9036-2.0.txt, 9036-2.1.txt, 9036-3.0.txt > > > I'm trying to run cleanup, but get this: > {noformat} > INFO [CompactionExecutor:18] 2015-03-25 10:29:16,355 CompactionManager.java > (line 564) Cleaning up > SSTableReader(path='/cassandra/production/Data_daily/production-Data_daily-jb-4345750-Data.db') > ERROR [CompactionExecutor:18] 2015-03-25 10:29:16,664 CassandraDaemon.java > (line 199) Exception in thread Thread[CompactionExecutor:18,1,main] > java.io.IOException: disk full > at > org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:567) > at > org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63) > at > org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > Now that's odd, since: > * Disk has some 680G left > * The sstable it's trying to cleanup is far less than 680G: > {noformat} > # ls -lh *4345750* > -rw-r--r-- 1 cassandra cassandra 64M Mar 21 04:42 > production-Data_daily-jb-4345750-CompressionInfo.db > -rw-r--r-- 1 cassandra cassandra 219G Mar 21 04:42 > production-Data_daily-jb-4345750-Data.db > -rw-r--r-- 1 cassandra cassandra 503M Mar 21 04:42 > production-Data_daily-jb-4345750-Filter.db > -rw-r--r-- 1 cassandra cassandra 42G Mar 21 04:42 > production-Data_daily-jb-4345750-Index.db > -rw-r--r-- 1 cassandra cassandra 5.9K Mar 21 04:42 > production-Data_daily-jb-4345750-Statistics.db > -rw-r--r-- 1 cassandra cassandra 81M Mar 21 04:42 > production-Data_daily-jb-4345750-Summary.db > -rw-r--r-- 1 cassandra cassandra 79 Mar 21 04:42 > production-Data_daily-jb-4345750-TOC.txt > {noformat} > Sure, it's large, but it's not 680G. > No other compactions are running on that server. I'm getting this on 12 / 56 > servers right now. > Could it be some bug in the calculation of the expected size of the new > sstable, perhaps? -- This message was sent by Atlassian JIRA (v6.3.4#6332)