[jira] [Comment Edited] (CASSANDRA-6774) Cleanup fails with assertion error after stopping previous run
[ https://issues.apache.org/jira/browse/CASSANDRA-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13937541#comment-13937541 ] Marcus Eriksson edited comment on CASSANDRA-6774 at 3/17/14 8:31 AM: - hmm, i actually think we should throw an exception if we fail, so that the nodetool user gets some feedback without having to check logs, attached was (Author: krummas): hmm, i think we should throw an exception if we fail, so that the nodetool user gets some feedback without having to check logs, attached > Cleanup fails with assertion error after stopping previous run > -- > > Key: CASSANDRA-6774 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6774 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.0.5 >Reporter: Keith Wright >Assignee: Marcus Eriksson > Fix For: 2.0.7 > > Attachments: 0001-6774-wip.patch, > 0001-Dont-continue-after-failing-to-cancel-in-progress-co.patch, 6774-v2.txt > > > I am stress testing a new 2.0.5 cluster and did the following: > - start decommission during heavy write, moderate read load > - trigger cleanup on non-decommissioning node (nodetool cleanup) > - Started to see higher GC load stop stopped cleanup via nodetool stop CLEANUP > - attempt to launch cleanup now fails with the following message in console. > Cassandra log shows: > http://aep.appspot.com/display/cKmlMcDuKD72iYAcBykDuVZkRWY/ -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-6774) Cleanup fails with assertion error after stopping previous run
[
https://issues.apache.org/jira/browse/CASSANDRA-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931563#comment-13931563
]
Dmitrij Koniajev edited comment on CASSANDRA-6774 at 3/12/14 6:36 PM:
--
Same issue here on LCS - after decreasing RF cleanup fails with the same
stacktrace. Restart didn't help, after running "nodetool disableautocompaction"
command and waiting for compaction get finished cleanup fails with the
following stacktrace:
{quote}
Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 1, Size: 1
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:227)
at
org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:265)
at
org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
at
org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.cassandra.db.compaction.CompactionManager.needsCleanup(CompactionManager.java:502)
at
org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:540)
at
org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:274)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
{quot
[jira] [Comment Edited] (CASSANDRA-6774) Cleanup fails with assertion error after stopping previous run
[
https://issues.apache.org/jira/browse/CASSANDRA-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913311#comment-13913311
]
Keith Wright edited comment on CASSANDRA-6774 at 2/26/14 6:55 PM:
--
Additional info:
cqlsh:system> select * from compactions_in_progress limit 10;
id | columnfamily_name | inputs
| keyspace_name
--++--+---
9a20d760-9f15-11e3-bfe7-fb46089edbb0 |global_user | {16796, 17068,
17073, 17077, 17118, 17135, 17164, 17193, 17219, 17281, 17293, 17365, 17553,
17575, 17606, 17668, 17695, 17701, 17906, 17983, 18016, 18017, 18069, 18089,
18098, 18099, 18108, 18114, 18119, 18123, 18188, 18449} | users
93778f70-9f16-11e3-bfe7-fb46089edbb0 | client_end_user_lookup |
{3280, 3281, 3283, 3284, 3286, 3293, 3297, 3298, 3301, 3302, 3303, 3306, 3323,
3324, 3326, 3328, 3329, 3330, 3331, 3332, , 3334} | users
96d23e10-9f14-11e3-bfe7-fb46089edbb0 | cookie_user_lookup |
{9072, 9078, 9083, 9088, 9094, 9100, 9106, 9112, 9118,
9124, 9131, 9146, 9369, 9370, 9371, 9372, 9373, 9393} | users
(3 rows)
was (Author: keithwrightbos):
Additional info:
cqlsh:system> select * from compactions_in_progress limit 10;
id | columnfamily_name | inputs
| keyspace_name
--++--+---
d56eeb80-9f16-11e3-8115-2981d742 | cookie_user_lookup |
{8203, 8206,
9990, 9991, 9992, 9993, 9994, 9995, 9996, 10409} | users
33807e50-9f17-11e3-8115-2981d742 | shard_user_lookup |
{6684,
6687, 6690, 6693, 6697, 6700, 7109, 7110, 7486} | users
576eefd0-9f13-11e3-8115-2981d742 |global_user | {17107, 17115,
17123, 17132, 17142, 17150, 17209, 17218, 17409, 17417, 18276, 18322, 18334,
18335, 18337, 18351, 18369, 18371, 18379, 18439, 18445, 18446, 18451, 19515,
19519, 19538, 19555, 19580, 19624, 19753, 19754, 19755} | users
(3 rows)
[kwright@castest06 ~]$ nodetool compactionstats
pending tasks: 33
compaction typekeyspace table completed
total unit progress
Compaction users global_user 14478355576
17165883943 bytes84.34%
Compaction usersshard_user_lookup 705875461
3249369496 bytes21.72%
Compaction userscookie_user_lookup 1723865787
3489228069 bytes49.41%
Active compaction remaining time :n/a
> Cleanup fails with assertion error after stopping previous run
> --
>
> Key: CASSANDRA-6774
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6774
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 2.0.5
>Reporter: Keith Wright
>
> I am stress testing a new 2.0.5 cluster and did the following:
> - start decommission during heavy write, moderate read load
> - trigger cleanup on non-decommissioning node (nodetool cleanup)
> - Started to see higher GC load stop stopped cleanup via nodetool stop CLEANUP
> - attempt to launch cleanup now fails with the following message in console.
> Cassandra log shows:
> WARN 18:45:38,420 Unable to cancel in-progress compactions for
>
