[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968696#comment-14968696 ] Sylvain Lebresne commented on CASSANDRA-10360: -- Ok, I've closed this issue then. Thanks! > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes > ccm start > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep 10 > for cver in 3.0 > do > echo "Draining all nodes" > ccm node1 nodetool drain > ccm stop > echo "switching to java 8" >
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967257#comment-14967257 ] Sylvain Lebresne commented on CASSANDRA-10360: -- Alright, committed with the last places having {{assert hasNext}} fixed. I'll mark the ticket for now just so we don't forget to include the tests I've linked above in our dtest suite. > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes > ccm start > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967606#comment-14967606 ] Jim Witschey commented on CASSANDRA-10360: -- I'll add the test -- I've filed CASSANDRA-10563 to track it. > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes > ccm start > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep 10 > for cver in 3.0 > do > echo "Draining all nodes" > ccm node1 nodetool drain > ccm stop > echo "switching to java
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966560#comment-14966560 ] Branimir Lambov commented on CASSANDRA-10360: - {{OldFormatDeserializer.skipNext()}} as well as {{UnfilteredIterator.next/peek}} and {{AtomIterator.next/peek}} still {{assert hasNext()}}. Other than that, the patch looks good to me. > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes > ccm start > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966539#comment-14966539 ] Sylvain Lebresne commented on CASSANDRA-10360: -- bq. {{assert hasNext()}} changes the code's behavior when asserts are on True. I did that because in practice, we never need this, the method that have those assert are always called after a {{hasNext()}} call. But you're right, there is not reason to take a risk, so added a simple commit to move the call out of an assert. bq. Is the code being tested already? Basic functionality of reading old sstable is tested by some of our existing upgrade tests. I have more focused tests for this (that include tests for statics and range tombstones) [here|https://github.com/pcmanus/cassandra-dtest/blob/8099_upgrade_tests/upgrade_8099_test.py], but those haven't been committed yet cause they use hard-coded path for now and I'll need to figure out how to modify that so cassci is happy with it. Actually, [~mambocab], could you look at including that test file in the upgrade tests in a way that works for CI? > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at >
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965172#comment-14965172 ] Branimir Lambov commented on CASSANDRA-10360: - It _is_ much clearer now. Thank you. One problem left in {{UnfilteredIterator.next/peek}} and {{AtomIterator.next/peek}}: {{assert hasNext()}} changes the code's behavior when asserts are on. Call {{hasNext()}} separately and then {{assert next != null}} or {{!isDone}}. Is the code being tested already? > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965103#comment-14965103 ] Sylvain Lebresne commented on CASSANDRA-10360: -- bq. hasNext is hard to follow. Agreed. I've actually gave a shot to a slightly more extensive refactor (pushed to the same branch) where I've separated some of the steps, creating first an iterator of atom from the input, then an iterator of {{Unfiltered}} from that, leaving mostly just the handling of the last special cases to {{hasNext}}. This takes a bit more lines of code, but I think this is overall easier to follow. The rest of the remarks were all on point and I included fixes for them in the new patch too. > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14960601#comment-14960601 ] Sylvain Lebresne commented on CASSANDRA-10360: -- bq. Does this solution satisfy the "no range tombstone markers between open and close" requirement? No, you're right, it doesn't, I've been sloppy. So I force-pushed [on the same branch|https://github.com/pcmanus/cassandra/commits/10360] a solution that should satisfy those requirements. It refactors the old format deserializer somewhat but that hopefully makes thing easier to follow/simpler. While testing this I also noticed a (simple to fix) regression from CASSANDRA-10136, so the branch has an additional commit for that. > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 rc2 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note --
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14960971#comment-14960971 ] Branimir Lambov commented on CASSANDRA-10360: - I think the code actually reads a bit better now, even if it has to do a bit more. [TombstoneTracker.isShadowed|https://github.com/pcmanus/cassandra/commit/618929034b045505368c4f92043f5d3da8e73ed5#diff-19cb83af11bbbc4d803ca33c8cdea57aR444]: Not sure if worth rewriting, but iterating over {{openTombstones.tailSet(atom.asRangeTombstone())}} instead would ensure the comparison is always true for tombstones. Perhaps change to: {code} SortedSet coveringTombstones = atom.isCell() ? openTombstones : openTombstones.tailSet(atom.asRangeTombstone()); return Iterables.any(coveringTombstones, tombstone -> tombstone.deletionTime.deletes(timestamp)); {code} It probably does not matter, but shouldn't the test above use {{isRow(atom)}}? [hasNext|https://github.com/pcmanus/cassandra/commit/618929034b045505368c4f92043f5d3da8e73ed5#diff-19cb83af11bbbc4d803ca33c8cdea57aR313] is hard to follow. One thing that may help is to get rid of {{atom}}, using {{next}} directly. I have a feeling that letting the RT/static row swap flow into the {{isRow}} option would make it a little clearer as well, but I may be wrong. It appears there is a situation when {{nextAtom}} may be lost: if static row pushes tombstone to {{nextAtom}} followed by grouping the static row which also will want to push something there. I wonder why [the assert|https://github.com/pcmanus/cassandra/commit/618929034b045505368c4f92043f5d3da8e73ed5#diff-19cb83af11bbbc4d803ca33c8cdea57aR298] doesn't fire (if my failure scenario is valid it would be good to have a test that makes it fire). I no longer see the code to drain the tombstone tracker after the input is exhausted. Don't we need to do that any more? [clearState|https://github.com/pcmanus/cassandra/commit/618929034b045505368c4f92043f5d3da8e73ed5#diff-19cb83af11bbbc4d803ca33c8cdea57aR404]: what about the tombstone tracker? > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 rc2 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955127#comment-14955127 ] Branimir Lambov commented on CASSANDRA-10360: - Does this solution satisfy the ["no range tombstone markers between open and close"|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/rows/RangeTombstoneMarker.java#L55] requirement? I believe we must convert to a boundary whenever there's overlap. > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 rc2 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953244#comment-14953244 ] Sylvain Lebresne commented on CASSANDRA-10360: -- There was indeed a small error in the logic of the next code leading to that assertion. I've force-pushed a fix, would you mind re-running your test and check if it works now? > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 rc2 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes > ccm start > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep 10 >
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953490#comment-14953490 ] Andrew Hust commented on CASSANDRA-10360: - Confirmed that the above upgrade test script now passes without exceptions. Ran with: apache/cassandra-2.1: {{4acc3a69d319b0e7e00cbd37b27e988ebfa4df4f}} apache/cassandra-2.2: {{0c051a46c54fd1a2f151e1a68f4556faca02be8d}} pcmanus/10360: {{6a56057a4eaab01480fba6292c5a59d4c3d62c49}} > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 rc2 > > Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, > system.log.2.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947144#comment-14947144 ] Andrew Hust commented on CASSANDRA-10360: - I ran your patch thru a 2.1->2.2->3.0 upgrade and it failed with a different exception. I've attached the system.log and size_estimates sstables from this run. Ran with: apache/cassandra-2.1 {{1b08cbd817dea379ea84175381d3ef151fe65681}} apache/cassandra-2.2: {{be89dae3ecfd98b2170732c45d7f95807d5c19af}} pcmanus/10360: {{8f7d524efcbf0157697e67a3b4ff4b883b441a55}} {code} java.lang.AssertionError at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.checkReady(UnfilteredDeserializer.java:346) at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.nextIsRow(UnfilteredDeserializer.java:367) at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:384) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369) at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:110) at org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52) at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:148) at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125) at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57) at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:112) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade >
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937461#comment-14937461 ] Andrew Hust commented on CASSANDRA-10360: - I have now reproduced this issue when upgrading from 2.1 to 3.0 when doing a rolling upgrade with cstar_perf. Attached site_estimates sstables and system log from one of the nodes. I'll work on getting a isolated script to reproduce. > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 3.0.0 rc2 > > Attachments: size_estimates.tar.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes > ccm start > ccm node1 stress write n=5K -rate threads=4 -mode native cql3 > sleep 10
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14791289#comment-14791289 ] Aleksey Yeschenko commented on CASSANDRA-10360: --- [~slebresne] Looks to me like another old format reading bug. Can you give it a quick look? > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug >Reporter: Andrew Hust > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes > ccm start > ccm node1 stress write n=5K -rate threads=4 -mode native cql3 > sleep 10 > for cver in 2.2 3.0 > do > echo "Draining all nodes" > ccm node1 nodetool drain > ccm stop > echo "switching to java 8" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) > echo "Upgrading to git:cassandra-$cver" > ccm setdir -v git:cassandra-$cver > ccm start > echo "Sleeping to