Jeff Griffith created CASSANDRA-7408:
----------------------------------------

             Summary: System hints corruption - dataSize ... would be larger 
than file
                 Key: CASSANDRA-7408
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7408
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: RHEL 6.5
Cassandra 1.2.16
RF=3
Thrift
            Reporter: Jeff Griffith


I've found several unresolved JIRA tickets related to SSTable corruption but 
not sure if they apply to the case we are seeing in system/hints. We see 
periodic exceptions such as:

{noformat}
dataSize of 144115248479299639 starting at 17209 would be larger than file 
/home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length 35542
{noformat}

Is there something we could possibly be doing from the application to cause 
this sort of corruption? We also see it on some of our own column families also 
some *negative* lengths which are presumably a similar corruption.

{noformat}
ERROR [HintedHandoff:57] 2014-06-17 17:08:04,690 CassandraDaemon.java (line 
191) Exception in thread Thread[HintedHandoff:57,1,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: 
dataSize of 144115248479299639 starting at 17209 would be larger than file 
/home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length 35542
        at 
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:441)
        at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282)
        at 
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:90)
        at 
org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:508)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: 
dataSize of 144115248479299639 starting at 17209 would be larger than file 
/home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length 35542
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at 
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:437)
        ... 6 more
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: 
java.io.IOException: dataSize of 144115248479299639 starting at 17209 would be 
larger than file 
/home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length 35542
        at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:167)
        at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:83)
        at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69)
        at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:180)
        at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:155)
        at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:142)
        at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)
        at 
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
        at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122)
        at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96)
        at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at 
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145)
        at 
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
        at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
        at 
org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:442)
        at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        ... 3 more
Caused by: java.io.IOException: dataSize of 144115248479299639 starting at 
17209 would be larger than file 
/home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length 35542
        at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:123)
        ... 23 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Reply via email to