[ https://issues.apache.org/jira/browse/CASSANDRA-8190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188353#comment-14188353 ]
Marcus Eriksson edited comment on CASSANDRA-8190 at 10/29/14 1:51 PM: ---------------------------------------------------------------------- [~ngrigoriev] you seem to be hitting a bunch of weird stuff lately, could you post your config and more details about your setup? and perhaps a full log (start node -> this happens)? And, how do you write to the cluster? was (Author: krummas): [~ngrigoriev] you seem to be hitting a bunch of weird stuff lately, could you post your config and more details about your setup? and perhaps a full log (start node -> this happens)? > Compactions stop completely because of RuntimeException in CompactionExecutor > ----------------------------------------------------------------------------- > > Key: CASSANDRA-8190 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8190 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: DSE 4.5.2 (Cassandra 2.0.10) > Reporter: Nikolai Grigoriev > Assignee: Marcus Eriksson > Attachments: jstack.txt.gz, system.log.gz > > > I have a cluster that is recovering from being overloaded with writes. I am > using the workaround from CASSANDRA-6621 to prevent the STCS fallback (which > is killing the cluster - see CASSANDRA-7949). > I have observed that after one or more exceptions like this > {code} > ERROR [CompactionExecutor:4087] 2014-10-26 22:50:05,016 CassandraDaemon.java > (line 199) Exception in thread Thread[CompactionExecutor:4087,1,main] > java.lang.RuntimeException: Last written key DecoratedKey(425124616570337476, > 0010000000001111000000000000033523da00001000000000033523da000000001111000000001000000000 > 00004000000000000000000100) >= current key DecoratedKey(-8778432288598355336, > 0010000000001111000000000000040c7a8f00001000000000040c7a8f000000001111000000001000000000 > 00004000000000000000000100) writing into > /cassandra-data/disk2/myks/mytable/myks-mytable-tmp-jb-130379-Data.db > at > org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:142) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:165) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:160) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {code} > the node completely stops the compactions and I end up in the state like this: > {code} > # nodetool compactionstats > pending tasks: 1288 > compaction type keyspace table completed > total unit progress > Active compaction remaining time : n/a > {code} > The node recovers if restarted and starts compactions - until getting more > exceptions like this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)