Update: I've reverted hinted_handoff_enabled back to its default value of true and the errors stopped. Is this just a coincidence, or could be related?
On Sun, Oct 6, 2013 at 7:23 PM, Ran Tavory <ran...@gmail.com> wrote: > Pardon me, now with the appropriate subject line... > > Hi, I have a small cluster of 1.2.6 and after some config changes I > started seeing errors int the logs. > > Not sure that's related, but the changes I performed were to disable > hinted handoff and disable auto snapshot. I'll try to reverte these, see if > the picture changes. > > But anyway, that seems like a bug, right? > > I see this across many nodes, not only one. > > ERROR [ReplicateOnWriteStage:105] 2013-10-06 16:13:13,799 > CassandraDaemon.java (line 192) Exception in thread > Thread[ReplicateOnWriteStage:105,5,main] > java.lang.AssertionError: DecoratedKey(-9223372036854775808, ) != > DecoratedKey(-1854619418400985942, 000338393900000a4769676f707469782d3100) > in > /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60) > at > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68) > at > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272) > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) > at org.apache.cassandra.db.Table.getRow(Table.java:347) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64) > at > org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90) > at > org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [ReplicateOnWriteStage:82] 2013-10-06 16:13:14,249 > CassandraDaemon.java (line 192) Exception in thread > Thread[ReplicateOnWriteStage:82,5,main] > java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to > seek to position 2171332 in > /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db > (1250125 bytes) in read-only mode > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1597) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.IllegalArgumentException: unable to seek to position > 2171332 in > /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db > (1250125 bytes) in read-only mode > at > org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306) > at > org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42) > at > org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1054) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:94) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:112) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60) > at > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68) > at > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272) > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) > at org.apache.cassandra.db.Table.getRow(Table.java:347) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64) > at > org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90) > at > org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593) > > -- > /Ran > http://tavory.com > -- /Ran http://tavory.com
