[
https://issues.apache.org/jira/browse/CASSANDRA-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833624#comment-13833624
]
Andriy Yevsyukov commented on CASSANDRA-5761:
---------------------------------------------
Hi,
Not sure whether it fixes the problem, but that's what we found.
1. Problem happens when often reset colum families- we are in active
development stage so very often reset colum families(drop, create, load test
data)
2. invalidate cache doesn't help.
Our solution:
We dropped keyspaces and deleted ALL files in data folder
We stopped to reset column families
For now everything is fine
> Issue with secondary index sstable.
> -----------------------------------
>
> Key: CASSANDRA-5761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5761
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.5
> Reporter: Andriy Yevsyukov
> Priority: Critical
>
> With Cassandra 1.2.5 having issue very similar to
> [CASSANDRA-5225|https://issues.apache.org/jira/browse/CASSANDRA-5225] but for
> secondary index sstable. Every query that uses this index fails in Hector
> with ConnectionTimeout but cassandra log says that reason is:
> {noformat}
> ERROR [ReadStage:55803] 2013-07-15 12:11:35,392 CassandraDaemon.java (line
> 175) Exception in thread Thread[ReadStage:55803,5,main]
> java.lang.RuntimeException:
> org.apache.cassandra.io.sstable.CorruptSSTableException:
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
> (/data/cassandra/data/betting/events/betting-events.events_sport_type_idx-ic-1-Data.db,
> 19658 bytes remaining)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
> (/data/cassandra/data/betting/events/betting-events.events_sport_type_idx-ic-1-Data.db,
> 19658 bytes remaining)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:108)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:90)
> at
> org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:171)
> at
> org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:154)
> at
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:143)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:86)
> at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:45)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:134)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:293)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at
> org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:140)
> at
> org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:109)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1466)
> at
> org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:82)
> at
> org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:548)
> at
> org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1454)
> at
> org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:44)
> at
> org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1076)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
> ... 3 more
> Caused by: org.apache.cassandra.db.ColumnSerializer$CorruptColumnException:
> invalid column name length 0
> (/data/cassandra/data/betting/events/betting-events.events_sport_type_idx-ic-1-Data.db,
> 19658 bytes remaining)
> at
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException.create(ColumnSerializer.java:148)
> at
> org.apache.cassandra.db.RangeTombstone$Serializer.deserializeBody(RangeTombstone.java:279)
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:90)
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:73)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:104)
> ... 30 more
> {noformat}
> We cleared CF but we have a lot of insertion and in few hours the issue
> happened again. Are there any advices?
--
This message was sent by Atlassian JIRA
(v6.1#6144)
