[
https://issues.apache.org/jira/browse/CASSANDRA-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262824#comment-14262824
]
Maxim Ivanov commented on CASSANDRA-7408:
-----------------------------------------
See same problem on 1.2.19, it seems to happening for system hints only.
While scrubbing it is reporting following errors:
{code}
WARN [CompactionExecutor:17] 2015-01-02 10:09:41,454 OutputHandler.java (line
57) Non-fatal error reading row (stacktrace follows)
java.io.IOError: java.io.IOException: Impossible row size 92971968326729728
at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:171)
at
org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:526)
at
org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:515)
at
org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:70)
at
org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:280)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:250)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Impossible row size 92971968326729728
... 10 more
WARN [CompactionExecutor:17] 2015-01-02 10:09:41,456 OutputHandler.java (line
52) Row at 299608 is unreadable; skipping to next
WARN [CompactionExecutor:17] 2015-01-02 10:09:41,457 OutputHandler.java (line
57) Non-fatal error reading row (stacktrace follows)
java.io.IOError: java.io.IOException: Impossible row size 7881707469400978738
at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:171)
at
org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:526)
at
org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:515)
at
org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:70)
at
org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:280)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:250)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Impossible row size 7881707469400978738
... 10 more
WARN [CompactionExecutor:17] 2015-01-02 10:09:41,459 OutputHandler.java (line
52) Row at 304082 is unreadable; skipping to next
{code}
> System hints corruption - dataSize ... would be larger than file
> ----------------------------------------------------------------
>
> Key: CASSANDRA-7408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7408
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: RHEL 6.5
> Cassandra 1.2.16
> RF=3
> Thrift
> Reporter: Jeff Griffith
>
> I've found several unresolved JIRA tickets related to SSTable corruption but
> not sure if they apply to the case we are seeing in system/hints. We see
> periodic exceptions such as:
> {noformat}
> dataSize of 144115248479299639 starting at 17209 would be larger than file
> /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length
> 35542
> {noformat}
> Is there something we could possibly be doing from the application to cause
> this sort of corruption? We also see it on some of our own column families
> also some *negative* lengths which are presumably a similar corruption.
> {noformat}
> ERROR [HintedHandoff:57] 2014-06-17 17:08:04,690 CassandraDaemon.java (line
> 191) Exception in thread Thread[HintedHandoff:57,1,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> dataSize of 144115248479299639 starting at 17209 would be larger than file
> /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length
> 35542
> at
> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:441)
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282)
> at
> org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:90)
> at
> org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:508)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> dataSize of 144115248479299639 starting at 17209 would be larger than file
> /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length
> 35542
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:437)
> ... 6 more
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.IOException: dataSize of 144115248479299639 starting at 17209 would
> be larger than file
> /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length
> 35542
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:167)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:83)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69)
> at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:180)
> at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:155)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:142)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)
> at
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:442)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ... 3 more
> Caused by: java.io.IOException: dataSize of 144115248479299639 starting at
> 17209 would be larger than file
> /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length
> 35542
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:123)
> ... 23 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
