Hi, I have a small cluster of 1.2.6 and after some config changes I started seeing errors int the logs.
Not sure that's related, but the changes I performed were to disable hinted handoff and disable auto snapshot. I'll try to reverte these, see if the picture changes. But anyway, that seems like a bug, right? I see this across many nodes, not only one. ERROR [ReplicateOnWriteStage:105] 2013-10-06 16:13:13,799 CassandraDaemon.java (line 192) Exception in thread Thread[ReplicateOnWriteStage:105,5,main] java.lang.AssertionError: DecoratedKey(-9223372036854775808, ) != DecoratedKey(-1854619418400985942, 000338393900000a4769676f707469782d3100) in /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119) at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60) at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68) at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) at org.apache.cassandra.db.Table.getRow(Table.java:347) at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64) at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90) at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [ReplicateOnWriteStage:82] 2013-10-06 16:13:14,249 CassandraDaemon.java (line 192) Exception in thread Thread[ReplicateOnWriteStage:82,5,main] java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to seek to position 2171332 in /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db (1250125 bytes) in read-only mode at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1597) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalArgumentException: unable to seek to position 2171332 in /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db (1250125 bytes) in read-only mode at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306) at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42) at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1054) at org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:94) at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:112) at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60) at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68) at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) at org.apache.cassandra.db.Table.getRow(Table.java:347) at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64) at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90) at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593) -- /Ran http://tavory.com
