[ https://issues.apache.org/jira/browse/CASSANDRA-8190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcus Eriksson resolved CASSANDRA-8190. ---------------------------------------- Resolution: Won't Fix The compactions probably stop since we don't call 'submitBackground()' if the CompactionTask throws an exception and there are no writes to the node so no new compaction tasks are triggered This could be 'fixed' by doing submitBackground() in a finally block after the compaction task is executed, but I think that might be a bad idea since we could end up in weird infinite loop situations if we keep throwing exceptions in the compaction tasks. And, if we do throw, the node might need some manual intervention anyway Created CASSANDRA-8211 for the cause of the exception in this issue > Compactions stop completely because of RuntimeException in CompactionExecutor > ----------------------------------------------------------------------------- > > Key: CASSANDRA-8190 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8190 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: DSE 4.5.2 (Cassandra 2.0.10) > Reporter: Nikolai Grigoriev > Assignee: Marcus Eriksson > Attachments: cassandra-env.sh, cassandra.yaml, jstack.txt.gz, > system.log.gz, system.log.gz > > > I have a cluster that is recovering from being overloaded with writes. I am > using the workaround from CASSANDRA-6621 to prevent the STCS fallback (which > is killing the cluster - see CASSANDRA-7949). > I have observed that after one or more exceptions like this > {code} > ERROR [CompactionExecutor:4087] 2014-10-26 22:50:05,016 CassandraDaemon.java > (line 199) Exception in thread Thread[CompactionExecutor:4087,1,main] > java.lang.RuntimeException: Last written key DecoratedKey(425124616570337476, > 0010000000001111000000000000033523da00001000000000033523da000000001111000000001000000000 > 00004000000000000000000100) >= current key DecoratedKey(-8778432288598355336, > 0010000000001111000000000000040c7a8f00001000000000040c7a8f000000001111000000001000000000 > 00004000000000000000000100) writing into > /cassandra-data/disk2/myks/mytable/myks-mytable-tmp-jb-130379-Data.db > at > org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:142) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:165) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:160) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {code} > the node completely stops the compactions and I end up in the state like this: > {code} > # nodetool compactionstats > pending tasks: 1288 > compaction type keyspace table completed > total unit progress > Active compaction remaining time : n/a > {code} > The node recovers if restarted and starts compactions - until getting more > exceptions like this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)