I've been using nodetool move to rebalance my cluster. Most of the moves take under an hour, or a few hours at most. The current move has taken 4+ days so I'm afraid it will never complete. What's the best way to cancel it and try again?
I'm running a cluster of 12 nodes at AWS. Each node runs Cassandra 1.2.5 on an m1.xlarge EC2 instance, and they are spread across 3 availability zones within a single region. I've seen some of these errors in the log. I'm not sure if it's related or not: ERROR [CompactionExecutor:4092] 2013-09-10 01:31:49,783 CassandraDaemon.java (line 175) Exception in thread Thread[CompactionExecutor:4092,1,main] java.lang.IndexOutOfBoundsException: index (1) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45) at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94) at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76) at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31) at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:128) at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114) at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109) at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:219) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114) at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:98) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:134) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Here's the status of the cluster as reported by nodetool ring, showing the one node in "Moving" state: Datacenter: us-east ========== Address Rack Status State Load Owns Token 127605887595351923798765477786913079290 10.xxx.xxx.xxx 1c Up Normal 224.53 GB 25.00% 0 10.xxx.xxx.xxx 1d Up Moving 297.46 GB 2.44% 4150051970709140963435425752946440221 10.xxx.xxx.xxx 1d Up Normal 107.75 GB 5.89% 14178431955039102644307275309657008810 10.xxx.xxx.xxx 1e Up Normal 82.75 GB 8.33% 28356863910078205288614550619314017620 10.xxx.xxx.xxx 1e Up Normal 173.83 GB 2.99% 33451586107772559423309548485325625873 10.xxx.xxx.xxx 1c Up Normal 64.4 GB 5.34% 42535295865117307932921825928971026430 10.xxx.xxx.xxx 1d Up Normal 56.46 GB 8.33% 56713727820156410577229101238628035240 10.xxx.xxx.xxx 1e Up Normal 57.88 GB 8.33% 70892159775195513221536376548285044050 10.xxx.xxx.xxx 1c Up Normal 95.18 GB 8.33% 85070591730234615865843651857942052860 10.xxx.xxx.xxx 1d Up Normal 169.37 GB 8.33% 99249023685273718510150927167599061670 10.xxx.xxx.xxx 1e Up Normal 109.99 GB 8.33% 113427455640312821154458202477256070480 10.xxx.xxx.xxx 1c Up Normal 98.64 GB 8.33% 127605887595351923798765477786913079290