Re: AssertionError: DecoratedKey(... ) != DecoratedKey (...)

2013-10-06 Thread Ran Tavory 
Update: I've reverted hinted_handoff_enabled back to its default value of
true and the errors stopped. Is this just a coincidence, or could be
related?



On Sun, Oct 6, 2013 at 7:23 PM, Ran Tavory  wrote:

> Pardon me, now with the appropriate subject line...
>
> Hi, I have a small cluster of 1.2.6 and after some config changes I
> started seeing errors int the logs.
>
> Not sure that's related, but the changes I performed were to disable
> hinted handoff and disable auto snapshot. I'll try to reverte these, see if
> the picture changes.
>
> But anyway, that seems like a bug, right?
>
> I see this across many nodes, not only one.
>
> ERROR [ReplicateOnWriteStage:105] 2013-10-06 16:13:13,799
> CassandraDaemon.java (line 192) Exception in thread
> Thread[ReplicateOnWriteStage:105,5,main]
> java.lang.AssertionError: DecoratedKey(-9223372036854775808, ) !=
> DecoratedKey(-1854619418400985942, 00033839390a4769676f707469782d3100)
> in
> /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db
>  at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.(SSTableNamesIterator.java:60)
>  at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
>  at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
>  at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
> at
> org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
>  at
> org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
> ERROR [ReplicateOnWriteStage:82] 2013-10-06 16:13:14,249
> CassandraDaemon.java (line 192) Exception in thread
> Thread[ReplicateOnWriteStage:82,5,main]
> java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to
> seek to position 2171332 in
> /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db
> (1250125 bytes) in read-only mode
>  at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1597)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.IllegalArgumentException: unable to seek to position
> 2171332 in
> /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db
> (1250125 bytes) in read-only mode
>  at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306)
> at
> org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1054)
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:94)
>  at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:112)
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.(SSTableNamesIterator.java:60)
>  at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
>  at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
>  at
> org.apache.cassandra.db.SliceByNamesReadCommand.

Re: AssertionError: DecoratedKey(...) != DecoratedKey(...)

2010-04-15 Thread Gary Dusbabek 
No data corruption.  There was a bug in the way that the index was
scanned that was manifesting itself when when the index got bigger
than 2GB.

Gary.


On Thu, Apr 15, 2010 at 08:03, Ran Tavory  wrote:
> yes, this looks like the same issue, thanks Gary.
> Other than seeing the errors in the log I haven't seen any other
> irregularities. (maybe there are, but they haven't surfaced). Does this
> assertion mean data corruption or something else that's worth waiting to
> 0.6.1 for?
>
> On Thu, Apr 15, 2010 at 2:00 PM, Gary Dusbabek  wrote:
>>
>> Ran,
>>
>> It looks like you're seeing
>> https://issues.apache.org/jira/browse/CASSANDRA-866.  It's fixed in
>> 0.6.1.
>>
>> Gary
>>
>> On Thu, Apr 15, 2010 at 04:06, Ran Tavory  wrote:
>> > When restarting one of the nodes in my cluster I found this error in the
>> > log. What does this mean?
>> >
>> >  INFO [GC inspection] 2010-04-15 05:03:04,898 GCInspector.java (line
>> > 110) GC
>> > for ConcurrentMarkSweep: 712 ms, 11149016 reclaimed leaving 442336680
>> > used;
>> > max is 4432068608
>> > ERROR [HINTED-HANDOFF-POOL:1] 2010-04-15 05:03:17,948
>> > DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>> > java.util.concurrent.ExecutionException: java.lang.AssertionError:
>> > DecoratedKey(163143070370570938845670096830182058073,
>> > 1K2i35+B8RuuRDP7Gwz3Xw==) !=
>> > DecoratedKey(163143368384879375649994309361429628039,
>> > 4k54mGvj7JoT5rBH68K+9A==) in
>> > /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
>> >         at
>> > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> >         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> >         at
>> >
>> > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >         at java.lang.Thread.run(Thread.java:619)
>> > Caused by: java.lang.AssertionError:
>> > DecoratedKey(163143070370570938845670096830182058073,
>> > 1K2i35+B8RuuRDP7Gwz3Xw==) !=
>> > DecoratedKey(163143368384879375649994309361429628039,
>> > 4k54mGvj7JoT5rBH68K+9A==) in
>> > /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
>> >         at
>> >
>> > org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.(SSTableSliceIterator.java:127)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.SSTableSliceIterator.(SSTableSliceIterator.java:59)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:830)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
>> >         at
>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> >         at
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> >         at
>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >         ... 2 more
>> >
>
>

Re: AssertionError: DecoratedKey(...) != DecoratedKey(...)

2010-04-15 Thread Ran Tavory 
yes, this looks like the same issue, thanks Gary.

Other than seeing the errors in the log I haven't seen any other
irregularities. (maybe there are, but they haven't surfaced). Does this
assertion mean data corruption or something else that's worth waiting to
0.6.1 for?

On Thu, Apr 15, 2010 at 2:00 PM, Gary Dusbabek  wrote:

> Ran,
>
> It looks like you're seeing
> https://issues.apache.org/jira/browse/CASSANDRA-866.  It's fixed in
> 0.6.1.
>
> Gary
>
> On Thu, Apr 15, 2010 at 04:06, Ran Tavory  wrote:
> > When restarting one of the nodes in my cluster I found this error in the
> > log. What does this mean?
> >
> >  INFO [GC inspection] 2010-04-15 05:03:04,898 GCInspector.java (line 110)
> GC
> > for ConcurrentMarkSweep: 712 ms, 11149016 reclaimed leaving 442336680
> used;
> > max is 4432068608
> > ERROR [HINTED-HANDOFF-POOL:1] 2010-04-15 05:03:17,948
> > DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
> > java.util.concurrent.ExecutionException: java.lang.AssertionError:
> > DecoratedKey(163143070370570938845670096830182058073,
> > 1K2i35+B8RuuRDP7Gwz3Xw==) !=
> > DecoratedKey(163143368384879375649994309361429628039,
> > 4k54mGvj7JoT5rBH68K+9A==) in
> > /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
> > at
> > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> > at
> >
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:619)
> > Caused by: java.lang.AssertionError:
> > DecoratedKey(163143070370570938845670096830182058073,
> > 1K2i35+B8RuuRDP7Gwz3Xw==) !=
> > DecoratedKey(163143368384879375649994309361429628039,
> > 4k54mGvj7JoT5rBH68K+9A==) in
> > /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
> > at
> >
> org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.(SSTableSliceIterator.java:127)
> > at
> >
> org.apache.cassandra.db.filter.SSTableSliceIterator.(SSTableSliceIterator.java:59)
> > at
> >
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:830)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
> > at
> >
> org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
> > at
> >
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> > at
> >
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
> > at
> >
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
> > at
> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > ... 2 more
> >
>

Re: AssertionError: DecoratedKey(...) != DecoratedKey(...)

2010-04-15 Thread Gary Dusbabek 
Ran,

It looks like you're seeing
https://issues.apache.org/jira/browse/CASSANDRA-866.  It's fixed in
0.6.1.

Gary

On Thu, Apr 15, 2010 at 04:06, Ran Tavory  wrote:
> When restarting one of the nodes in my cluster I found this error in the
> log. What does this mean?
>
>  INFO [GC inspection] 2010-04-15 05:03:04,898 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 712 ms, 11149016 reclaimed leaving 442336680 used;
> max is 4432068608
> ERROR [HINTED-HANDOFF-POOL:1] 2010-04-15 05:03:17,948
> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
> java.util.concurrent.ExecutionException: java.lang.AssertionError:
> DecoratedKey(163143070370570938845670096830182058073,
> 1K2i35+B8RuuRDP7Gwz3Xw==) !=
> DecoratedKey(163143368384879375649994309361429628039,
> 4k54mGvj7JoT5rBH68K+9A==) in
> /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>         at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.AssertionError:
> DecoratedKey(163143070370570938845670096830182058073,
> 1K2i35+B8RuuRDP7Gwz3Xw==) !=
> DecoratedKey(163143368384879375649994309361429628039,
> 4k54mGvj7JoT5rBH68K+9A==) in
> /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
>         at
> org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.(SSTableSliceIterator.java:127)
>         at
> org.apache.cassandra.db.filter.SSTableSliceIterator.(SSTableSliceIterator.java:59)
>         at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:830)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
>         at
> org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
>         at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
>         at
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
>         at
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         ... 2 more
>