[ https://issues.apache.org/jira/browse/CASSANDRA-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13732897#comment-13732897 ]
Tyler Hobbs commented on CASSANDRA-5855: ---------------------------------------- At least with a 1.1.9 scrub, it's trying to look up the column value validator with the full composite column name, but CfMetadata is expecting to only see the last component (e.g. "created_on"). So, it ends up using the default validator, which is UTF8Type, to try to validate all of the column values. > Scrub does not understand compound primary key created in CQL 3 beta > -------------------------------------------------------------------- > > Key: CASSANDRA-5855 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5855 > Project: Cassandra > Issue Type: Bug > Components: Tools > Reporter: J.B. Langston > Assignee: Tyler Hobbs > > We have a customer who was using the beta version of CQL 3 in DSE 3.0 which > includes Cassandra 1.1.9 plus patches backported from later versions. > They've now upgraded to DSE 3.1, which includes Cassandra 1.2.6 plus patches. > When restarting for the first time after running upgradesstables, they > noticed the following error in the log: > {noformat} > Thread[SSTableBatchOpen:2,5,main] > java.lang.AssertionError > at > org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:401) > at > org.apache.cassandra.io.sstable.IndexSummary$IndexSummarySerializer.deserialize(IndexSummary.java:124) > at > org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:426) > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:360) > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:201) > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:154) > at > org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:241) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > {noformat} > This error was also reported on CASSANDRA-5703. The comments suggested it > was caused by an empty row key, so I had them run scrub on it. When they > did, scrub reported the following warning almost 4 million times: > {noformat} > WARN [CompactionExecutor:27] 2013-08-02 10:13:13,041 OutputHandler.java > (line 52) Row at 530332255 is unreadable; skipping to next > WARN [CompactionExecutor:27] 2013-08-02 10:13:13,041 OutputHandler.java > (line 57) Non-fatal error reading row (stacktrace follows) > java.lang.RuntimeException: Error validating row > DecoratedKey(139154446688383793922009760478335751546, > 735fc9da503b11e2844b123140ff209f) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:243) > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114) > at > org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:98) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:166) > at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:173) > at > org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:529) > at > org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:518) > at > org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:73) > at > org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:283) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:253) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.cassandra.db.marshal.MarshalException: String didn't > validate. > at org.apache.cassandra.db.marshal.UTF8Type.validate(UTF8Type.java:66) > at org.apache.cassandra.db.Column.validateFields(Column.java:292) > at > org.apache.cassandra.db.ColumnFamily.validateColumnFields(ColumnFamily.java:382) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:239) > ... 15 more > {noformat} > The customer did some testing and they've determined that the issue only > exists when taking a Cassandra 1.1 sstable with compound primary keys and > running scrub on it in either Cassandra 1.1 or 1.2. > It appears that the scrub does not understand the 1.1 compound primary key so > is invalidating the row. > The customer provided a cassandra data directory that's from DSE 3.0. Running > "nodetool scrub" in either DSE 3.0 or 3.1 generates all sorts of exceptions. > If you fire up cassandra before running the scrub, running this query: > {noformat} > select count(*) from b_projectsubscription_project; > {noformat} > will return 88. > After the scrub, it returns 0. > When I discussed this with Alexey Yeschenko, he said that if he recalls > correctly, the beta didn't have row markers, and did not use a composite > comparator for simple primary keys. Whereas CQL3 final used > CompositeType(UTF8Type), the beta would just use UTF8Type. I asked him if > this could cause these errors, and he said he didn't think so because after > upgrading your schema created under the beta would still have UTF8Type, so > Cassandra would know how to handle it correctly. > Based on the customer's investigation, it sounds like this may be true of the > normal read/write path but not for scrub. However, given the error that > occurred at startup, this may be causing some issues aside from just scrub. > My theory is that scrub is looking at what's just a UTF8 string and trying to > interpret the first few bytes as the sentinels for a composite type. When it > then tries to interpret the remaining bytes, if only part of a multi-byte > UTF8 character was in the remaining byte array, it might cause the UTF8 > validation errors above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira