[ https://issues.apache.org/jira/browse/CASSANDRA-7775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126967#comment-14126967 ]
Pavel Pletnev edited comment on CASSANDRA-7775 at 9/9/14 1:26 PM: ------------------------------------------------------------------ After trying to flush corrupted memtable MemtablePostFlusher thread stuck and start to pill up. This is my trace: INFO [FlushWriter:4] 2014-09-09 14:55:03,405 Memtable.java (line 355) Writing Memtable-ucc_ttl16318@1568409896(230/2300 serialized/live bytes, 2 ops) INFO [OptionalTasks:1] 2014-09-09 14:55:03,410 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-sstable_activity@1141428663(38070/1495207 serialized/live bytes, 2800 ops) ERROR [FlushWriter:4] 2014-09-09 14:55:03,610 CassandraDaemon.java (line 199) Exception in thread Thread[FlushWriter:4,5,main] java.lang.RuntimeException: Cannot get comparator 1 in org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type). This might due to a mismatch between the schema and the data read at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:133) at org.apache.cassandra.db.marshal.AbstractCompositeType.split(AbstractCompositeType.java:100) at org.apache.cassandra.io.sstable.ColumnNameHelper.minComponents(ColumnNameHelper.java:103) at org.apache.cassandra.db.ColumnFamily.getColumnStats(ColumnFamily.java:443) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:194) at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:389) at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:342) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IndexOutOfBoundsException: index (1) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:306) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:285) at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45) at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:124) ... 11 more Version: 2.0.10 was (Author: merentafo): After trying to flush corrupted memtable MemtablePostFlusher thread stuck and start to pill up. This is my trace: INFO [FlushWriter:4] 2014-09-09 14:55:03,405 Memtable.java (line 355) Writing Memtable-ucc_ttl16318@1568409896(230/2300 serialized/live bytes, 2 ops) INFO [OptionalTasks:1] 2014-09-09 14:55:03,410 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-sstable_activity@1141428663(38070/1495207 serialized/live bytes, 2800 ops) ERROR [FlushWriter:4] 2014-09-09 14:55:03,610 CassandraDaemon.java (line 199) Exception in thread Thread[FlushWriter:4,5,main] java.lang.RuntimeException: Cannot get comparator 1 in org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type). This might due to a mismatch between the schema and the data read at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:133) at org.apache.cassandra.db.marshal.AbstractCompositeType.split(AbstractCompositeType.java:100) at org.apache.cassandra.io.sstable.ColumnNameHelper.minComponents(ColumnNameHelper.java:103) at org.apache.cassandra.db.ColumnFamily.getColumnStats(ColumnFamily.java:443) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:194) at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:389) at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:342) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IndexOutOfBoundsException: index (1) must be less than size (1) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:306) at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:285) at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45) at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:124) ... 11 more > Cassandra attempts to flush an empty memtable into disk and fails > ----------------------------------------------------------------- > > Key: CASSANDRA-7775 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7775 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: $ nodetool version > ReleaseVersion: 2.0.6 > $ java -version > java version "1.7.0_51" > Java(TM) SE Runtime Environment (build 1.7.0_51-b13) > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) > Reporter: Omri Bahumi > > I'm not sure what triggers this flush, but when it happens the following > appears in our logs: > {code} > INFO [OptionalTasks:1] 2014-08-15 02:24:20,115 ColumnFamilyStore.java (line > 785) Enqueuing flush of Memtable-app_recs_best_in_expr_prefix2@1219170646(0/0 > serialized/live bytes, 0 ops) > INFO [FlushWriter:34] 2014-08-15 02:24:20,116 Memtable.java (line 331) > Writing Memtable-app_recs_best_in_expr_prefix2@1219170646(0/0 serialized/live > bytes, 0 ops) > ERROR [FlushWriter:34] 2014-08-15 02:24:20,127 CassandraDaemon.java (line > 196) Exception in thread Thread[FlushWriter:34,5,main] > java.lang.RuntimeException: Cannot get comparator 1 in > org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type). > This might due to a mismatch between the schema and the data read > at > org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:133) > at > org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:140) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:96) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) > at > org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:125) > at > org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:122) > at java.util.TreeMap.compare(TreeMap.java:1188) > at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1264) > at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1699) > at java.util.TreeMap.tailMap(TreeMap.java:905) > at java.util.TreeSet.tailSet(TreeSet.java:350) > at java.util.TreeSet.tailSet(TreeSet.java:383) > at > org.apache.cassandra.db.RangeTombstone$Tracker.update(RangeTombstone.java:203) > at > org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:192) > at > org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:138) > at > org.apache.cassandra.io.sstable.SSTableWriter.rawAppend(SSTableWriter.java:202) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:187) > at > org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:365) > at > org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:318) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.lang.IndexOutOfBoundsException: index (1) must be less than > size (1) > at > com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:306) > at > com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:285) > at > com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45) > at > org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:124) > ... 23 more > {code} > After this happens, the MemtablePostFlusher thread pool starts piling up. > When trying to restart the cluster, a similar exception occurs when trying to > replay the commit log. > Our way of recovering from this is to delete all commit logs in the faulty > node, start it and issue a repair. -- This message was sent by Atlassian JIRA (v6.3.4#6332)