[
https://issues.apache.org/jira/browse/CASSANDRA-7465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046708#comment-14046708
]
T Jake Luciani commented on CASSANDRA-7465:
-------------------------------------------
Great detective work Jason! Well I recall we had this issue with memorable
arenas a while ago and cloned the keys before we put them in the cache. Doing
the same for queries should be straight fwd. but it would be nice as Benedict
says to find these systematically.
> DecoratedKey assertion error on reads
> -------------------------------------
>
> Key: CASSANDRA-7465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 3 nodes
> Oracle Linux Server 6.3
> kernel ver 2.6.39
> Reporter: Jason Brown
> Priority: Blocker
> Fix For: 2.1.0
>
>
> Getting the following exception when running read stress:
> {code}WARN [SharedPool-Worker-31] 2014-06-27 21:25:51,391
> AbstractTracingAwareExecutorService.java:166 - Uncaught exception on thread
> Thread[SharedPool-Worker-31,10,main]: {}
> java.lang.AssertionError: DecoratedKey(-5397116645141815707,
> 30303031393143364639) != DecoratedKey(-5397116645141815707,
> 30303031343439443233) in
> /u/sdd/cassandra-jasobrown/data/Keyspace1/Standard1-6ab9bd90fe3b11e385edff96c2ef2fd6/Keyspace1-Standard1-ka-73-Data.db
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:111)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:59)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:89)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:125)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:59)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:341)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1393)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2060)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_13]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:162)
> ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:103)
> [apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_13]
> {code}
> I have a three node cluster, which I populate with the following stress
> command:
> {code}
> cassandra-stress write n=60000000 -schema replication\(factor\=2\) -key
> populate=1..60000000 -rate threads=42 -mode native prepared cql3 -port
> native=9043 thrift=9161 -node athena06-a,athena06-b,athena06-c -col
> n=fixed\(21\) size=exp\(11..42\)
> {code}
> Then I run the read stress:
> {code}
> cassandra-stress read n=100000000 -key dist=extr\(1..600000000,2\) -mode
> native prepared cql3 -port native=9043 thrift=9161 -node
> athena06-b,athena06-c,athena06-a -col n=fixed\(21\) -rate threads=64
> {code}
> The above exception occurs semi-frequently (several to ~50 times a minute,
> but seems to depend on amount of data in cluster - anecdotal evidence only).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
