[ https://issues.apache.org/jira/browse/CASSANDRA-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062954#comment-14062954 ]
Yuki Morishita commented on CASSANDRA-7543: ------------------------------------------- Thanks for your finding. Let me write simple Unit test case to verify. > Assertion error when compacting large row with map//list field or range > tombstone > --------------------------------------------------------------------------------- > > Key: CASSANDRA-7543 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7543 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: linux > Reporter: Matt Byrd > Assignee: Yuki Morishita > Labels: compaction, map > > Hi, > So in a couple of clusters we're hitting this problem when compacting large > rows with a schema which contains the map data-type. > Here is an example of the error: > {code} > java.lang.AssertionError: incorrect row data size 87776427 written to > /cassandra/X/Y/X-Y-tmp-ic-2381-Data.db; correct is 87845952 > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162) > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163) > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) > > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208) > {code} > I have a python script which reproduces the problem, by just writing lots of > data to a single partition key with a schema that contains the map data-type. > I added some debug logging and found that the difference in bytes seen in the > reproduction (255) was due to the following pieces of data being written: > {code} > DEBUG [CompactionExecutor:3] 2014-07-13 00:38:42,891 ColumnIndex.java (line > 168) DATASIZE writeOpenedMarker columnIndex: > org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn: > [java.nio.HeapByteBuffer[pos=0 lim=34 cap=34], java.nio.HeapByteBuffer[pos=0 > lim=34 cap=34]](deletedAt=1405237116014999, localDeletion=1405237116) > startPosition: 262476 endPosition: 262561 diff: 85 > DEBUG [CompactionExecutor:3] 2014-07-13 00:38:43,007 ColumnIndex.java (line > 168) DATASIZE writeOpenedMarker columnIndex: > org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn: > org.apache.cassandra.db.Column@3e5b5939 startPosition: 328157 endPosition: > 328242 diff: 85 > DEBUG [CompactionExecutor:3] 2014-07-13 00:38:44,159 ColumnIndex.java (line > 168) DATASIZE writeOpenedMarker columnIndex: > org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn: > org.apache.cassandra.db.Column@fc3299b startPosition: 984105 endPosition: > 984190 diff: 85 > {code} > So looking at the code you can see that there are extra range tombstones > written on the column index border (in ColumnIndex where > tombstoneTracker.writeOpenedMarker is called) which aren't accounted for in > LazilyCompactedRow.columnSerializedSize. > This is where the difference comes from in the assertion error, so the > solution is just to account for this data. > I have a patch which does just this, by keeping track of the extra data > written out via tombstoneTracker.writeOpenedMarker in ColumnIndex and adding > it back to the dataSize in LazilyCompactedRow.java, where it serialises out > the row size. > After applying the patch the reproduction stops producing the AssertionError. > I know this is not a problem in 2.0 + because of singe pass compaction, > however there are lots of 1.2 clusters out there still which might run into > this. > Please let me know if you've any questions. > Thanks, > Matt -- This message was sent by Atlassian JIRA (v6.2#6252)