[ https://issues.apache.org/jira/browse/CASSANDRA-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611567#comment-13611567 ]
Jonathan Ellis commented on CASSANDRA-5359: ------------------------------------------- This sounds like tombstones or TTLs expiring between the first and second pass, but we already check controller.gcBefore for the former, and SSTableIdentityIterator.expireBefore for the latter, so I don't know what to tell you. Presumably this will be fixed by CASSANDRA-4180. > AssertionError in HintedHandoff and CompactionExecutor > ------------------------------------------------------ > > Key: CASSANDRA-5359 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5359 > Project: Cassandra > Issue Type: Bug > Affects Versions: 1.2.2 > Environment: Centos 6.4 > 3 Nodes > RF 3 > Vnodes > Murmur3Partitioner > Reporter: Joey Imbasciano > > Seeing this on 2 of the 3 nodes pretty consistently. The stack traces seem to > happen consistently together, and I believe they are related. > HintedHandoff > {noformat} > ERROR [HintedHandoff:3] 2013-03-18 20:58:58,643 CassandraDaemon.java (line > 132) Exception in thread Thread[HintedHandoff:3,1,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.AssertionError: originally calculated column size of 4 > 53516860 but now it is 453517031 > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:406) > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:252) > at > org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:89) > at > org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:459) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: > originally calculated column size of 453516860 but now > it is 453517031 > at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) > at java.util.concurrent.FutureTask.get(Unknown Source) > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:402) > ... 7 more > Caused by: java.lang.AssertionError: originally calculated column size of > 453516860 but now it is 453517031 > at > org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62) > at > org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > ... 3 more > ERROR [CompactionExecutor:8] 2013-03-18 20:58:58,645 CassandraDaemon.java > (line 132) Exception in thread Thread[CompactionExecutor:8, > 1,main] > java.lang.AssertionError: originally calculated column size of 453516860 but > now it is 453517031 > at > org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62) > at > org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > {noformat} > CompactionExecutor > {noformat} > INFO [CompactionExecutor:9] 2013-03-18 21:02:54,486 > CompactionController.java (line 158) Compacting large row > system/hints:bcc4b89f- > a2ac-4abe-8420-4f5c2462a65b (453517091 bytes) incrementally > ERROR [CompactionExecutor:9] 2013-03-18 21:03:16,034 CassandraDaemon.java > (line 132) Exception in thread Thread[CompactionExecutor:9, > 1,main] > java.lang.AssertionError: originally calculated column size of 453516860 but > now it is 453517031 > at > org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62) > at > org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > ERROR [HintedHandoff:1] 2013-03-18 21:03:16,035 CassandraDaemon.java (line > 132) Exception in thread Thread[HintedHandoff:1,1,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.AssertionError: originally calculated column size of 4 > 53516860 but now it is 453517031 > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:406) > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:252) > at > org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:89) > at > org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:459) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: > originally calculated column size of 453516860 but now > it is 453517031 > at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) > at java.util.concurrent.FutureTask.get(Unknown Source) > at > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:402) > ... 7 more > Caused by: java.lang.AssertionError: originally calculated column size of > 453516860 but now it is 453517031 > at > org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62) > at > org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > ... 3 more > {noformat} > Please let me know any additional info needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira