Johan Bjork created CASSANDRA-7892:
--------------------------------------
Summary: Hashmap IllegalStateException in anticompaction
Key: CASSANDRA-7892
URL: https://issues.apache.org/jira/browse/CASSANDRA-7892
Project: Cassandra
Issue Type: Bug
Environment: Cassandra 2.1 rc 5
Reporter: Johan Bjork
{noformat}
ERROR [AntiEntropyStage:4] 2014-09-05 19:55:06,439 CassandraDaemon.java:166 -
Exception in thread Thread[AntiEntropyStage:4,5,main]
java.lang.RuntimeException: java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.IllegalStateException
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:117)
~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.IllegalStateException
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411)
~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400)
~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:113)
~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
... 4 common frames omitted
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalStateException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
~[na:1.7.0_60]
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
~[na:1.7.0_60]
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407)
~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
... 6 common frames omitted
Caused by: java.lang.IllegalStateException: null
at java.util.HashMap$HashIterator.remove(HashMap.java:938)
~[na:1.7.0_60]
at
org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425)
~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381)
~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_60]
... 3 common frames omitted
{noformat}
And on a different host:
{noformat}
ERROR [CompactionExecutor:43] 2014-09-05 19:55:06,376 CassandraDaemon.java:166
- Exception in thread Thread[CompactionExecutor:4
3,1,main]
java.lang.IllegalStateException: null
at java.util.HashMap$HashIterator.remove(HashMap.java:938)
~[na:1.7.0_60]
at
org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:425)
~[apache-cassa
ndra-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:381)
~[apache-cassandra-2.1
.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.0-rc5.jar:2.1.0-rc5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
{noformat}
The exceptions happened in our test environment after I added a bunch of
new-nodes at the same time that was in a different (previously non-existing)
datacenter. The datacenter is determined using GossipingPropertyFileSnitch and
started repairs on a bunch of nodes at the same time using
{noformat}
nodetool -h bfstor4.aoa repair -inc -par
{noformat}
The nodes in the old datacenter had previously been repaired using the -inc
option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
