[ https://issues.apache.org/jira/browse/CASSANDRA-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273754#comment-14273754 ]
Benedict commented on CASSANDRA-8580: ------------------------------------- I managed to reproduce this with a slight bug consistently locally todat with ColumnFaimlyStoreTest. Unfortunately it seems to have been a wild goose chase tracking down the cause; testRemoveUnfinishedCompactionLeftovers deletes a stats file, and this file was then consistently missing. Strangely, though, it only occurred if DataTracker.releaseReferences did NOT release the final reference to sstables, but I have stopped looking further after spending quite a bit of time on it, since this was a deliberate corruption of the files for the unit test. I post here only in case it helps understand the cause, given the reference release interplay, although I expect it won't. > AssertionErrors after activating unchecked_tombstone_compaction with leveled > compaction > --------------------------------------------------------------------------------------- > > Key: CASSANDRA-8580 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8580 > Project: Cassandra > Issue Type: Bug > Reporter: Björn Hachmann > Assignee: Marcus Eriksson > Fix For: 2.1.3 > > Attachments: system.log > > > During our upgrade of Cassandra from version 2.0.7 to 2.1.2 we experienced a > serious problem regarding the setting unchecked_tombstone_compaction in > combination with leveled compaction strategy. > In order to prevent tombstone-threshold-warnings we activated the setting for > a specific table after the upgrade. Some time after that we observed new > errors in our log files: > {code} > INFO [CompactionExecutor:184] 2014-12-11 12:36:06,597 > CompactionTask.java:136 - Compacting > [SSTableReader(path='/data/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ka-1848-Data.db'), > SSTableReader(path='/ > data/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ka-1847-Data.db'), > > SSTableReader(path='/data/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ka-1845-Data.db'), > SSTableReader > (path='/data/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ka-1846-Data.db')] > ERROR [CompactionExecutor:183] 2014-12-11 12:36:06,613 > CassandraDaemon.java:153 - Exception in thread > Thread[CompactionExecutor:183,1,main] > java.lang.AssertionError: > /data/cassandra/data/metrigo_prod/new_user_data/metrigo_prod-new_user_data-tmplink-ka-705732-Data.db > at > org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:243) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:146) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_45] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > {code} > Obviously that error aborted the compaction and after some time the number of > pending compactions became very high on every node. Of course, this in turn > had a negative impact on several other metrics. > After reverting the setting we had to restart all nodes. After that > compactions could finish again and the pending compactions could be worked > off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)