Andrew Hust created CASSANDRA-10360: ---------------------------------------
Summary: UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade Key: CASSANDRA-10360 URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 Project: Cassandra Issue Type: Bug Reporter: Andrew Hust When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck in a compaction loop throwing: {code} java.lang.UnsupportedOperationException at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not surface the issue. It can be reproduced with the following (note -- changing jdks will need to be altered if you aren't on OSX) {code} #!/bin/sh echo "using java7 for cassandra-2.1 instance" export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) ccm stop ccm remove upgrade_nodes ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes ccm start ccm node1 stress write n=5K -rate threads=4 -mode native cql3 sleep 10 for cver in 2.2 3.0 do echo "Draining all nodes" ccm node1 nodetool drain ccm stop echo "switching to java 8" export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) echo "Upgrading to git:cassandra-$cver" ccm setdir -v git:cassandra-$cver ccm start echo "Sleeping to all version migrations" sleep 30 echo "Upgrading sstables" ccm node1 nodetool upgradesstables ccm node1 stress write n=5K -rate threads=4 -mode native cql3 sleep 10 done echo "***********" echo "instead of creating churn to cause compaction naturally forcing compaction of system keyspace" echo "***********" ccm node1 nodetool compact system ccm stop {code} The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables system}}. I'm puzzled by that since the script runs {{upgradesstables}} on each iteration. Is the system keyspace not effected by the command without arguments? Ran against: 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}} 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}} 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)