[ https://issues.apache.org/jira/browse/CASSANDRA-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161943#comment-14161943 ]
Marcus Eriksson commented on CASSANDRA-8065: -------------------------------------------- no, I need to know what happened *before* the exceptions started > Mutationstage error, ThreadPoolExecutor has shut down > ----------------------------------------------------- > > Key: CASSANDRA-8065 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8065 > Project: Cassandra > Issue Type: Bug > Environment: Casssandra Version: 2.0.9 > Total Clusters/Datacenters: 3 > Nodes in each cluster: 4 > Total # of Nodes: 12 > Replication Factor:3 for each data center > Node Size of each cluster: C3.xlarge in EC2 > Reporter: Vishal Mehta > > 1. Writes running at about 120 qps w/ TTL at row level for 60 minutes (for 3 > column families) > 2. GC set to run at every 10 minutes (so that we claim our space back from > tombstones) > 3. Other settings set as default. > After some time (1 hour 30 minutes approximately of tests execution) 11 nodes > across datacenters are Down. And write tests fails, with following exception > in /var/log/cassandra/system.log. > {noformat} > ERROR [MutationStage:18098] 2014-10-04 01:05:36,525 CassandraDaemon.java > (line 199) Exception in thread Thread[MutationStage:18098,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:145) > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) > at > org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:893) > at > org.apache.cassandra.service.StorageProxy.access$300(StorageProxy.java:68) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:1958) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > WARN [StorageServiceShutdownHook] 2014-10-04 01:05:36,525 StorageProxy.java > (line 2031) Some hints were not written before shutdown. This is not > supposed to happen. You should (a) run repair, and (b) file a bug report > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)