[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107031#comment-14107031 ] Richard Low commented on CASSANDRA-4206: The root cause of this in 1.2 is CASSANDRA-7808. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) = (column=col2, value=8383582, timestamp=1335348630332000) --- RowKey:
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14073259#comment-14073259 ] mat gomes commented on CASSANDRA-4206: -- Same issues on v 1.2.12 and 1.2.18 Error occurred during compaction java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 116397997 but now it is 116398382 ... AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1,
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939247#comment-13939247 ] André Cruz commented on CASSANDRA-4206: --- We're seeing this with 1.2.15, and seems related with large rows being compacted incrementally and being changed at the same time. We upgraded from 1.1.5 and did not have this assertion error before. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1,
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939285#comment-13939285 ] Eric Falcao commented on CASSANDRA-4206: +1 just recently upgraded to 2.0.6 and all the errors went away. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) = (column=col2, value=8383582, timestamp=1335348630332000) --- RowKey:
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939357#comment-13939357 ] Robert Coli commented on CASSANDRA-4206: {quote} This was present in 1.1 and earlier releases as well. The fix is to upgrade to 2.0.x. {quote} Which change in the 2.0 series fixes this, and how? Do you have a JIRA number we could refer to? AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1,
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939361#comment-13939361 ] André Cruz commented on CASSANDRA-4206: --- I think the fix that is mentioned is the fact that 2.0 does only single-pass compactions, so it will never be in the situation of having calculated a previous value that has changed. It seems an architectural change, and not something easily backported. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php =
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872321#comment-13872321 ] Dan Hendry commented on CASSANDRA-4206: --- I am also seeing this happening during compaction with Cassandra 1.2.13 on a non-hints column family. I have multithreaded compaction disabled. Here is one new piece of information: based on the thread name the exception occurred on, I searched back through the Cassandra log and found a 'Compacting large row' message. I recognized the row key from *that* message because it has come up before within the context of a bug in our client code which resulted in runaway writes to Cassandra. Its entirely possible that that particular row had more than 2^31 columns written to it. This smells like an integer overflow bug to me although I have not had a chance to dig into the Cassandra code yet. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859902#comment-13859902 ] Eric Lubow commented on CASSANDRA-4206: --- We are seeing this as well with 1.2.11. As was mentioned above, knowing which CF would very useful here. It seems to be happening to hints. The only other major action we are seeing that is out of the ordinary is thousands of hint SSTables being transferred at a time. Here is the Java error: {quote} ERROR [HintedHandoff:6] 2014-01-01 16:45:19,914 CassandraDaemon.java (line 191) Exception in thread Thread[HintedHandoff:6,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 1028119265 but now it is 1028119453 at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:436) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:90) at org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:502) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 1028119265 but now it is 1028119453 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:432) ... 6 more Caused by: java.lang.AssertionError: originally calculated column size of 1028119265 but now it is 1028119453 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:442) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) ... 3 more {quote} Multithreaded compactions are set to false in our cluster on all nodes. We also don't have any pending compactions in the cluster. Just seeing this error a lot in the logs. The error seems to happen more frequently during bootstraps or repairs that have a lot of work to do. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855152#comment-13855152 ] Christian Spriegel commented on CASSANDRA-4206: --- I can confirm that this error also occurs with multithreaded_compaction=false. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) = (column=col2, value=8383582, timestamp=1335348630332000)
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855228#comment-13855228 ] Daniel Kador commented on CASSANDRA-4206: - We got desperate and tried multithreaded_compaction=false yesterday afternoon and it seems to have worked for us. Pending compactions across the cluster have dropped from over 20,000 to under 600 (we're not quite done yet). The numbers were so high because we added six new nodes and they were unable to compact quickly until we made this settings change. No idea why it worked for us but thought I'd share our experience. If there's any other useful information I can share, let me know. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854930#comment-13854930 ] Daniel Kador commented on CASSANDRA-4206: - We're seeing the AssertionError in LazilyCompactedRow a lot too on 1.2.10. Not a problem with hinted handoffs for us. We just have a large CF (close to 1TB on disk) that has had most of its row keys deleted at this point. Lots of pending compactions on many nodes and these errors seem to stymie progress. Hoping to get those compactions to finish to reclaim disk... Has anybody figured out a decent workaround? Should we try disabling multithreaded_compaction? Looks like folks are still seeing the errors with that off (it's on for us). How stupid would running Cassandra with assertions off be? Alternatively, has anybody who's had this problem attempted to upgrade to 2.0 and had the problem fixed? AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854934#comment-13854934 ] Daniel Kador commented on CASSANDRA-4206: - I'll add that this CF is using leveled compaction in case that's useful. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) = (column=col2, value=8383582, timestamp=1335348630332000) ---
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854938#comment-13854938 ] Jonathan Ellis commented on CASSANDRA-4206: --- Disabling assertions to fix this is a great way to corrupt your sstables and get errors at read time later on. This is definitely fixed in 2.0. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) =
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854940#comment-13854940 ] Daniel Kador commented on CASSANDRA-4206: - That's sort of what I figured. That's why I asked how stupid it would be. The answer is very, clearly. I understand that it's fixed in 2.0 but many of us are still on 1.2.x. Speaking for myself, upgrading to 2.0 is on the roadmap but I'd prefer to do that as part of a staged rollout and not as an attempt to fix what seems like a bug. Do you think disabling multithreaded_compaction would help? AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3]
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855010#comment-13855010 ] Jonathan Ellis commented on CASSANDRA-4206: --- Doubt it. Suspect only workaround is to increase in_memory_compaction limit. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) = (column=col2, value=8383582, timestamp=1335348630332000)
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841590#comment-13841590 ] Christian Spriegel commented on CASSANDRA-4206: --- Seen with 1.2.11: {code} java.lang.AssertionError: originally calculated column size of 44470356 but now it is 44470410 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) {code} AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841604#comment-13841604 ] Christian Spriegel commented on CASSANDRA-4206: --- For starters, could we add the SSTable filenames to the Exception? That way users could submit broken SStable files to this ticket in the future. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) =
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13831035#comment-13831035 ] Patricio Echague commented on CASSANDRA-4206: - Seen the same error in 1.2.10 with multithreaded_compaction disabled. It causes nodetool rebuild to fail making it really hard to rebuild a DC fully. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) =
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778837#comment-13778837 ] Dan Kogan commented on CASSANDRA-4206: -- We are also seeing the same error during compaction. We can provide the sstables if that helps resolving the issues. INFO [CompactionExecutor:74953] 2013-09-26 14:23:53,978 CompactionController.java (line 166) Compacting large row iqtell/mail_folder_data_subject_withdate_asc:97995 (131986528 bytes) incrementally ERROR [CompactionExecutor:74953] 2013-09-26 14:24:01,126 CassandraDaemon.java (line 174) Exception in thread Thread[CompactionExecutor:74953,1,main] java.lang.AssertionError: originally calculated column size of 131986437 but now it is 131986500 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:188) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751424#comment-13751424 ] Nate McCall commented on CASSANDRA-4206: FTR - seeing this currently in 1.2.8 on batchlog compaction attempt (unfortunately I can't easily modify the gc_grace for this). Stack trace: {code} ERROR [CompactionExecutor:105] 2013-08-27 17:54:39,942 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:105,1,main] java.lang.AssertionError: originally calculated column size of 17391408 but now it is 17391426 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:445) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) ERROR [OptionalTasks:1] 2013-08-27 17:54:39,942 CassandraDaemon.java (line 192) Exception in thread Thread[OptionalTasks:1,5,main] {code} AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751463#comment-13751463 ] Jonathan Ellis commented on CASSANDRA-4206: --- TBH I'm not sure we're going to fix this in 1.2.x. If you have a snapshot set of sstables that can reproduce it, then we can dig in, but eyeballing the code hasn't fixed it yet (despite multiple efforts) and probably won't. The good news is that 2.0 fixed it by always doing single-pass compaction. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php';
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746850#comment-13746850 ] Arya Goudarzi commented on CASSANDRA-4206: -- I don't remember which bug but there was one bug where Johnathan and Sylvain were arguing about discrepancy of column calculation logic in various places in the code which depended on gc_grace, deleted columns, etc. I took that as a hint and applied it to my scenario. I increased the gc_grace for the CF in question to a large number like 1 year. Then ran user defined compactions and they happily ran. Afther that I change the gc_grace setting back to 10 days and ran compactions again and my tombsones got cleaned up without seeing that exception. So, although, it didn't make sense to me what happened exactly, but at least this could be worth a try for those who are stuck. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13710440#comment-13710440 ] Arya Goudarzi commented on CASSANDRA-4206: -- Here is our exact scenario: Some bad code, resulted in a 600Mb row in one of our CFs. It is spanned across 2 SSTables. We've deleted it but it does not go away and nodetool cfstats still reports the max row size being the size of this row. It is causing problems because anytime this SSTable is part of a compaction or we do range queries on this CF, we ended up with slow 16 sec GCs. I have tried the following to try and clean it up, but it does not go away: 1. nodetool compact with sizeTiered; 2. LCS with large SSTables and wait till it decides to promote the 2 SSTables and merge them; 3. sizeTiered with forceUserDefined compaction trying to compact the 2 SSTable with the deleted key. Any love? AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index.
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13694104#comment-13694104 ] Arya Goudarzi commented on CASSANDRA-4206: -- Any clues? We are not able to get one of our critical CFs in good shape as we cannot run compactions on it and its performance is deteriorating. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php,
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685666#comment-13685666 ] Eric Falcao commented on CASSANDRA-4206: Seeing this on my 1.2.5 cluster since upgrading: ubuntu@c2-1d:~$ nodetool -h localhost compact TRProd Timelines [12/637] Error occurred during compaction java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 91281671 but now it is 91281729 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:334) at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1657) at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2146) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:235) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:250) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1447) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:89) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1292) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1380) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:812) at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.AssertionError: originally calculated column size of 91281671 but now it is 91281729 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:355) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685730#comment-13685730 ] Eric Falcao commented on CASSANDRA-4206: I should add that I was upgrading from a 1.1 cluster. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) = (column=col2, value=8383582, timestamp=1335348630332000) --- RowKey:
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675709#comment-13675709 ] Alain RODRIGUEZ commented on CASSANDRA-4206: Similar thing over here. C*1.2.2 I have a lot of hints (3 4 GB) on 2 nodes out of 12 due to an unknown issue while growing to multiple datacenter (all the nodes of the new datacenter saw themselves as UNREACHABLE from the cassandra-cli - nodetool ring is ok...). On these 2 nodes I see now : {code} ERROR 08:37:16,878 Exception in thread Thread[CompactionExecutor:1540,1,main] java.lang.AssertionError: originally calculated column size of 5469343266 but now it is 5469343506 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62) at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR 08:37:16,878 Exception in thread Thread[HintedHandoff:176,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 5469343266 but now it is 5469343506 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:406) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:252) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:89) at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:459) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 5469343266 but now it is 5469343506 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:402) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 5469343266 but now it is 5469343506 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) {code} AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13647886#comment-13647886 ] Arya Goudarzi commented on CASSANDRA-4206: -- I have 1600 compactions pending and no compaction is running, so they are all stuck. I also got this in the logs. And Jonathan, I have multi_threaded_compaction enabled. Why is that a problem? ERROR [CompactionExecutor:111] 2013-05-02 13:54:05,913 CassandraDaemon.java (line 174) Exception in thread Thread[CompactionExecutor:111,1,main]java.lang.AssertionError: originally calculated column size of 1337269150 but now it is 1337269195at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:188) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380)
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590564#comment-13590564 ] Joey Imbasciano commented on CASSANDRA-4206: I am seeing what seems like the same issue on 1.2.2 on nodes that had collected a lot of hints after a node had been down for a couple hours. {noformat} ERROR [CompactionExecutor:104] 2013-03-01 00:29:54,211 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:104,1,main] java.lang.AssertionError: originally calculated column size of 350273328 but now it is 350297055 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62) at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ERROR [HintedHandoff:1] 2013-03-01 00:29:54,211 CassandraDaemon.java (line 132) Exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 350273328 but now it is 350297055 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:406) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:252) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:89) at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:459) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 350273328 but now it is 350297055 at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:402) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 350273328 but now it is 350297055 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62) at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) ... 3 more {noformat} AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13590641#comment-13590641 ] Joey Imbasciano commented on CASSANDRA-4206: Version: 1.2.2 - 6 nodes, RF3 OS: Centos 6.3 Looking closer at the errors, I'm only seeing this for the system/hints: {noformat} INFO [CompactionExecutor:555] 2013-03-01 15:37:10,737 CompactionController.java (line 158) Compacting large row system/hints:384ac791 -d0d7-4f5e-b66b-57af885341d5 (350297131 bytes) incrementally ERROR [CompactionExecutor:555] 2013-03-01 15:37:28,377 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:5 55,1,main] {noformat} AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487371#comment-13487371 ] Rick Branson commented on CASSANDRA-4206: - Just had this pop up on a 1.1.6 node after doing a 'nodetool removetoken' for a node that had collected a lot of hints. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) = (column=col2,
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445572#comment-13445572 ] Tyler Hobbs commented on CASSANDRA-4206: I'm not sure how useful it is at this point, but just for the record I'm seeing what appears to be the same issue with 1.0.7 (which means it shouldn't be CASSANDRA-3579): {noformat} INFO [GossipTasks:1] 2012-08-27 14:22:50,242 Gossiper.java (line 818) InetAddress /xx.xx.178.59 is now dead. INFO [GossipStage:1] 2012-08-27 14:22:59,090 Gossiper.java (line 804) InetAddress /xx.xx.178.59 is now UP INFO [HintedHandoff:1] 2012-08-27 14:23:41,548 HintedHandOffManager.java (line 296) Started hinted handoff for token: 132332031580364958013534569556798748899 with IP: /xx.xx.178.59 INFO [HintedHandoff:1] 2012-08-27 14:23:41,870 ColumnFamilyStore.java (line 704) Enqueuing flush of Memtable-HintsColumnFamily@2081164539(597050/47764000 serialized/live bytes, 857 ops) INFO [FlushWriter:181] 2012-08-27 14:23:41,870 Memtable.java (line 246) Writing Memtable-HintsColumnFamily@2081164539(597050/47764000 serialized/live bytes, 857 ops) INFO [FlushWriter:181] 2012-08-27 14:23:41,959 Memtable.java (line 283) Completed flushing /xx/xx/xx/cassandra/datafile/system/HintsColumnFamily-hc-6730-Data.db (624946 bytes) INFO [CompactionExecutor:884] 2012-08-27 14:23:41,961 CompactionTask.java (line 113) Compacting [SSTableReader(path='/xx/xx/xx/cassandra/datafile/system/HintsColumnFamily-hc-6729-Data.db'), SSTableReader(path='/ngs/app/xcardp/cassandra/datafile/system/HintsColumnFamily-hc-6730-Data.db')] INFO [CompactionExecutor:884] 2012-08-27 14:23:41,987 CompactionController.java (line 133) Compacting large row system/HintsColumnFamily:31372e33342e3137382e3534 (274816343 bytes) incrementally ERROR [CompactionExecutor:884] 2012-08-27 14:23:56,322 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:884,1,main] java.lang.AssertionError: originally calculated column size of 197713629 but now it is 197711561 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159) at org.apache.cassandra.db.compaction.CompactionManager$6.call(CompactionManager.java:275) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [HintedHandoff:1] 2012-08-27 14:23:56,323 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 197713629 but now it is 197711561 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:369) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:248) at org.apache.cassandra.db.HintedHandOffManager.access$200(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:416) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 197713629 but now it is 197711561 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:365) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 197713629 but now it is 197711561 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159) at org.apache.cassandra.db.compaction.CompactionManager$6.call(CompactionManager.java:275) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more ERROR [HintedHandoff:1] 2012-08-27 14:23:56,345
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266895#comment-13266895 ] Jonathan Ellis commented on CASSANDRA-4206: --- Do you have {{multithreaded_compaction}} enabled? AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) = (column=col2, value=8383582, timestamp=1335348630332000) --- RowKey:
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266900#comment-13266900 ] Jonathan Ellis commented on CASSANDRA-4206: --- bq. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index That part is definitely a [separate] bug, assuming the index completed building. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey:
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[ https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266897#comment-13266897 ] Jonathan Ellis commented on CASSANDRA-4206: --- bq. It's the first time I've tried secondary indexes of other type than UTF8Type I think that's a red herring; the error occurs when compacting the internal hints columnfamily, which has no indexes on it. AssertionError: originally calculated column size of 629444349 but now it is 588008950 -- Key: CASSANDRA-4206 URL: https://issues.apache.org/jira/browse/CASSANDRA-4206 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9 Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin 6.26-0squeeze1 Reporter: Patrik Modesto I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: {noformat} ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more {noformat} Few lines regarding Hints from the output.log: {noformat} INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes) incrementally INFO 06:22:52,610 Compacting large row system/HintsColumnFamily:1000 (2616073981 bytes) incrementally INFO 06:22:59,111 flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') (estimated 305147360 bytes) INFO 06:22:59,813 Enqueuing flush of Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) INFO 06:22:59,814 Writing Memtable-HintsColumnFamily@833933926(3814342/305147360 serialized/live bytes, 7452 ops) {noformat} I think the problem may be somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: {noformat} [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1,