[
https://issues.apache.org/jira/browse/CASSANDRA-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-5855:
-----------------------------------
Description:
We have a customer who was using the beta version of CQL 3 in DSE 3.0 which
includes Cassandra 1.1.9 plus patches backported from later versions. They've
now upgraded to DSE 3.1, which includes Cassandra 1.2.6 plus patches.
When restarting for the first time after running upgradesstables, they noticed
the following error in the log:
{noformat}
Thread[SSTableBatchOpen:2,5,main]
java.lang.AssertionError
at
org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:401)
at
org.apache.cassandra.io.sstable.IndexSummary$IndexSummarySerializer.deserialize(IndexSummary.java:124)
at
org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:426)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:360)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:201)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:154)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:241)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
This error was also reported on CASSANDRA-5703. The comments suggested it was
caused by an empty row key, so I had them run scrub on it. When they did,
scrub reported the following warning almost 4 million times:
{noformat}
WARN [CompactionExecutor:27] 2013-08-02 10:13:13,041 OutputHandler.java (line
52) Row at 530332255 is unreadable; skipping to next
WARN [CompactionExecutor:27] 2013-08-02 10:13:13,041 OutputHandler.java (line
57) Non-fatal error reading row (stacktrace follows)
java.lang.RuntimeException: Error validating row
DecoratedKey(139154446688383793922009760478335751546,
735fc9da503b11e2844b123140ff209f)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:243)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
at
org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:98)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:166)
at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:173)
at
org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:529)
at
org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:518)
at
org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:73)
at
org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:283)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:253)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.cassandra.db.marshal.MarshalException: String didn't
validate.
at org.apache.cassandra.db.marshal.UTF8Type.validate(UTF8Type.java:66)
at org.apache.cassandra.db.Column.validateFields(Column.java:292)
at
org.apache.cassandra.db.ColumnFamily.validateColumnFields(ColumnFamily.java:382)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:239)
... 15 more
{noformat}
The customer did some testing and they've determined that the issue only exists
when taking a Cassandra 1.1 sstable with compound primary keys and running
scrub on it in either Cassandra 1.1 or 1.2.
It appears that the scrub does not understand the 1.1 compound primary key so
is invalidating the row.
The customer provided a cassandra data directory that's from DSE 3.0. Running
"nodetool scrub" in either DSE 3.0 or 3.1 generates all sorts of exceptions.
If you fire up cassandra before running the scrub, running this query:
{noformat}
select count(*) from b_projectsubscription_project;
{noformat}
will return 88.
After the scrub, it returns 0.
When I discussed this with Alexey Yeschenko, he said that if he recalls
correctly, the beta didn't have row markers, and did not use a composite
comparator for simple primary keys. Whereas CQL3 final used
CompositeType(UTF8Type), the beta would just use UTF8Type. I asked him if this
could cause these errors, and he said he didn't think so because after
upgrading your schema created under the beta would still have UTF8Type, so
Cassandra would know how to handle it correctly.
Based on the customer's investigation, it sounds like this may be true of the
normal read/write path but not for scrub. However, given the error that
occurred at startup, this may be causing some issues aside from just scrub. My
theory is that scrub is looking at what's just a UTF8 string and trying to
interpret the first few bytes as the sentinels for a composite type. When it
then tries to interpret the remaining bytes, if only part of a multi-byte UTF8
character was in the remaining byte array, it might cause the UTF8 validation
errors above.
was:
We have a customer who was using the beta version of CQL 3 in DSE 3.0 which
includes Cassandra 1.1.9 plus patches backported from later versions. They've
now upgraded to DSE 3.1, which includes Cassandra 1.2.6 plus patches.
When restarting for the first time after running upgradesstables, they noticed
the following error in the log:
Thread[SSTableBatchOpen:2,5,main]
java.lang.AssertionError
at
org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:401)
at
org.apache.cassandra.io.sstable.IndexSummary$IndexSummarySerializer.deserialize(IndexSummary.java:124)
at
org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:426)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:360)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:201)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:154)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:241)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
This error was also reported on CASSANDRA-5703. The comments suggested it was
caused by an empty row key, so I had them run scrub on it. When they did,
scrub reported the following warning almost 4 million times:
WARN [CompactionExecutor:27] 2013-08-02 10:13:13,041 OutputHandler.java (line
52) Row at 530332255 is unreadable; skipping to next
WARN [CompactionExecutor:27] 2013-08-02 10:13:13,041 OutputHandler.java (line
57) Non-fatal error reading row (stacktrace follows)
java.lang.RuntimeException: Error validating row
DecoratedKey(139154446688383793922009760478335751546,
735fc9da503b11e2844b123140ff209f)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:243)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
at
org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:98)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:166)
at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:173)
at
org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:529)
at
org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:518)
at
org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:73)
at
org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:283)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:253)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.cassandra.db.marshal.MarshalException: String didn't
validate.
at org.apache.cassandra.db.marshal.UTF8Type.validate(UTF8Type.java:66)
at org.apache.cassandra.db.Column.validateFields(Column.java:292)
at
org.apache.cassandra.db.ColumnFamily.validateColumnFields(ColumnFamily.java:382)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:239)
... 15 more
The customer did some testing and they've determined that the issue only exists
when taking a Cassandra 1.1 sstable with compound primary keys and running
scrub on it in either Cassandra 1.1 or 1.2.
It appears that the scrub does not understand the 1.1 compound primary key so
is invalidating the row.
The customer provided a cassandra data directory that's from DSE 3.0. Running
"nodetool scrub" in either DSE 3.0 or 3.1 generates all sorts of exceptions.
If you fire up cassandra before running the scrub, running this query:
select count(*) from b_projectsubscription_project;
will return 88.
After the scrub, it returns 0.
When I discussed this with Alexey Yeschenko, he said that if he recalls
correctly, the beta didn't have row markers, and did not use a composite
comparator for simple primary keys. Whereas CQL3 final used
CompositeType(UTF8Type), the beta would just use UTF8Type. I asked him if this
could cause these errors, and he said he didn't think so because after
upgrading your schema created under the beta would still have UTF8Type, so
Cassandra would know how to handle it correctly.
Based on the customer's investigation, it sounds like this may be true of the
normal read/write path but not for scrub. However, given the error that
occurred at startup, this may be causing some issues aside from just scrub. My
theory is that scrub is looking at what's just a UTF8 string and trying to
interpret the first few bytes as the sentinels for a composite type. When it
then tries to interpret the remaining bytes, if only part of a multi-byte UTF8
character was in the remaining byte array, it might cause the UTF8 validation
errors above.
> Scrub does not understand compound primary key created in CQL 3 beta
> --------------------------------------------------------------------
>
> Key: CASSANDRA-5855
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5855
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: J.B. Langston
> Assignee: Tyler Hobbs
>
> We have a customer who was using the beta version of CQL 3 in DSE 3.0 which
> includes Cassandra 1.1.9 plus patches backported from later versions.
> They've now upgraded to DSE 3.1, which includes Cassandra 1.2.6 plus patches.
> When restarting for the first time after running upgradesstables, they
> noticed the following error in the log:
> {noformat}
> Thread[SSTableBatchOpen:2,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:401)
> at
> org.apache.cassandra.io.sstable.IndexSummary$IndexSummarySerializer.deserialize(IndexSummary.java:124)
> at
> org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:426)
> at
> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:360)
> at
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:201)
> at
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:154)
> at
> org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:241)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> This error was also reported on CASSANDRA-5703. The comments suggested it
> was caused by an empty row key, so I had them run scrub on it. When they
> did, scrub reported the following warning almost 4 million times:
> {noformat}
> WARN [CompactionExecutor:27] 2013-08-02 10:13:13,041 OutputHandler.java
> (line 52) Row at 530332255 is unreadable; skipping to next
> WARN [CompactionExecutor:27] 2013-08-02 10:13:13,041 OutputHandler.java
> (line 57) Non-fatal error reading row (stacktrace follows)
> java.lang.RuntimeException: Error validating row
> DecoratedKey(139154446688383793922009760478335751546,
> 735fc9da503b11e2844b123140ff209f)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:243)
> at
> org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
> at
> org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:98)
> at
> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160)
> at
> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:166)
> at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:173)
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:529)
> at
> org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:518)
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:73)
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:283)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:253)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.cassandra.db.marshal.MarshalException: String didn't
> validate.
> at org.apache.cassandra.db.marshal.UTF8Type.validate(UTF8Type.java:66)
> at org.apache.cassandra.db.Column.validateFields(Column.java:292)
> at
> org.apache.cassandra.db.ColumnFamily.validateColumnFields(ColumnFamily.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:239)
> ... 15 more
> {noformat}
> The customer did some testing and they've determined that the issue only
> exists when taking a Cassandra 1.1 sstable with compound primary keys and
> running scrub on it in either Cassandra 1.1 or 1.2.
> It appears that the scrub does not understand the 1.1 compound primary key so
> is invalidating the row.
> The customer provided a cassandra data directory that's from DSE 3.0. Running
> "nodetool scrub" in either DSE 3.0 or 3.1 generates all sorts of exceptions.
> If you fire up cassandra before running the scrub, running this query:
> {noformat}
> select count(*) from b_projectsubscription_project;
> {noformat}
> will return 88.
> After the scrub, it returns 0.
> When I discussed this with Alexey Yeschenko, he said that if he recalls
> correctly, the beta didn't have row markers, and did not use a composite
> comparator for simple primary keys. Whereas CQL3 final used
> CompositeType(UTF8Type), the beta would just use UTF8Type. I asked him if
> this could cause these errors, and he said he didn't think so because after
> upgrading your schema created under the beta would still have UTF8Type, so
> Cassandra would know how to handle it correctly.
> Based on the customer's investigation, it sounds like this may be true of the
> normal read/write path but not for scrub. However, given the error that
> occurred at startup, this may be causing some issues aside from just scrub.
> My theory is that scrub is looking at what's just a UTF8 string and trying to
> interpret the first few bytes as the sentinels for a composite type. When it
> then tries to interpret the remaining bytes, if only part of a multi-byte
> UTF8 character was in the remaining byte array, it might cause the UTF8
> validation errors above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
