[ https://issues.apache.org/jira/browse/CASSANDRA-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000585#comment-15000585 ]
Jeff Griffith commented on CASSANDRA-10579: ------------------------------------------- (i'll try to move things to 2.1.11 to simplify this) > IndexOutOfBoundsException during memtable flushing at startup (with > offheap_objects) > ------------------------------------------------------------------------------------ > > Key: CASSANDRA-10579 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10579 > Project: Cassandra > Issue Type: Bug > Environment: 2.1.10 on linux > Reporter: Jeff Griffith > Assignee: Benedict > Fix For: 2.1.12, 2.2.4 > > > Sometimes we have problems at startup where memtable flushes with an index > out of bounds exception as seen below. Cassandra is then dead in the water > until we track down the corresponding commit log via the segment ID and > remove it: > {code} > INFO [main] 2015-10-23 14:43:36,440 CommitLogReplayer.java:267 - Replaying > /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log > INFO [main] 2015-10-23 14:43:36,440 CommitLogReplayer.java:270 - Replaying > /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log (CL version 4, > messaging version 8) > INFO [main] 2015-10-23 14:43:36,594 CommitLogReplayer.java:478 - Finished > reading /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log > INFO [main] 2015-10-23 14:43:36,594 CommitLogReplayer.java:267 - Replaying > /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log > INFO [main] 2015-10-23 14:43:36,595 CommitLogReplayer.java:270 - Replaying > /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log (CL version 4, > messaging version 8) > INFO [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:478 - Finished > reading /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log > INFO [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:267 - Replaying > /home/y/var/cassandra/commitlog/CommitLog-4-1445474832694.log > INFO [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:270 - Replaying > /home/y/var/cassandra/commitlog/CommitLog-4-1445474832694.log (CL version 4, > messaging version 8) > WARN [SharedPool-Worker-5] 2015-10-23 14:43:36,747 > AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-5,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 6 > at > org.apache.cassandra.db.AbstractNativeCell.nametype(AbstractNativeCell.java:204) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at > org.apache.cassandra.db.AbstractNativeCell.isStatic(AbstractNativeCell.java:199) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at > org.apache.cassandra.db.composites.AbstractCType.compare(AbstractCType.java:166) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at > org.apache.cassandra.db.composites.AbstractCellNameType$1.compare(AbstractCellNameType.java:61) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at > org.apache.cassandra.db.composites.AbstractCellNameType$1.compare(AbstractCellNameType.java:58) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.utils.btree.BTree.find(BTree.java:277) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at > org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:154) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.utils.btree.Builder.update(Builder.java:74) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.utils.btree.BTree.update(BTree.java:186) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at > org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:225) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.Memtable.put(Memtable.java:210) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1225) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:455) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_31] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)