Hi All,
When running a high load on a MB Cluster we have observed Cassandra
throwing the following error.
Test Scenario:
1 publisher publishing 1,000,000 messages
3 consumers, consuming
after some time 15-20 mins the consumers stop consuming and we have
observed the following error in Cassandra. We also observed
*me.prettyprint.hector.api.exceptions.HTimedOutException:
TimedOutException()* thrown in MB
Cassandra Version: 1.2.13
cassandra system.log:
----------------------------------
INFO [ScheduledTasks:1] 2014-02-27 21:56:59,928 GCInspector.java (line
119) GC for ParNew: 241 ms for 1 collections, 1191010416 used; max is
8375238656
INFO [MemoryMeter:1] 2014-02-27 21:57:38,322 Memtable.java (line 516)
CFS(Keyspace='QpidKeySpace', ColumnFamily='QueueEntries') liveRatio is
49.411764705882355 (just-counted was 49.411764705882355). calculation took
0ms for 1 columns
ERROR [ReadStage:923] 2014-02-27 22:01:42,284 CassandraDaemon.java (line
191) Exception in thread Thread[ReadStage:923,5,main]
java.lang.RuntimeException:
org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.IOException: Corrupt (negative) value length encountered
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1614)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.IOException: Corrupt (negative) value length encountered
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:357)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:166)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:50)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:90)
at
org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:171)
at
org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:154)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:143)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:160)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:291)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1397)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1213)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1129)
at org.apache.cassandra.db.Table.getRow(Table.java:344)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1058)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1610)
... 3 more
Caused by: java.io.IOException: Corrupt (negative) value length encountered
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:352)
at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:102)
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92)
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:73)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:398)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:353)
... 27 more
WSO2 MB:
-----------------
[2014-02-27 22:18:12,335] WARN
{me.prettyprint.cassandra.connection.HConnectionManager} - Could not
fullfill request on this host CassandraClient<204.13.85.2:9160-49>
[2014-02-27 22:18:12,335] WARN
{me.prettyprint.cassandra.connection.HConnectionManager} - Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$7.execute(KeyspaceServiceImpl.java:285)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$7.execute(KeyspaceServiceImpl.java:268)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.getSlice(KeyspaceServiceImpl.java:289)
at
me.prettyprint.cassandra.model.thrift.ThriftSliceQuery$1.doInKeyspace(ThriftSliceQuery.java:53)
at
me.prettyprint.cassandra.model.thrift.ThriftSliceQuery$1.doInKeyspace(ThriftSliceQuery.java:49)
at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
at
me.prettyprint.cassandra.model.thrift.ThriftSliceQuery.execute(ThriftSliceQuery.java:48)
at
org.wso2.andes.server.store.util.CassandraDataAccessHelper.getMessagesFromQueue(CassandraDataAccessHelper.java:489)
at
org.wso2.andes.server.store.CassandraMessageStore.getMessagesFromGlobalQueue(CassandraMessageStore.java:1252)
at
org.wso2.andes.server.cluster.GlobalQueueWorker.run(GlobalQueueWorker.java:73)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7296)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:552)
at
org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:536)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$7.execute(KeyspaceServiceImpl.java:273)
... 16 more
Thanks,
NuwanS.
--
*Nuwan Silva*
*Senior Software Engineer - QA*
Mobile: +94779804543
WSO2 Inc.
lean . enterprise . middlewear.
http://www.wso2.com
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
