[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14966539#comment-14966539
]
Sylvain Lebresne commented on CASSANDRA-10360:
----------------------------------------------
bq. {{assert hasNext()}} changes the code's behavior when asserts are on
True. I did that because in practice, we never need this, the method that have
those assert are always called after a {{hasNext()}} call. But you're right,
there is not reason to take a risk, so added a simple commit to move the call
out of an assert.
bq. Is the code being tested already?
Basic functionality of reading old sstable is tested by some of our existing
upgrade tests. I have more focused tests for this (that include tests for
statics and range tombstones)
[here|https://github.com/pcmanus/cassandra-dtest/blob/8099_upgrade_tests/upgrade_8099_test.py],
but those haven't been committed yet cause they use hard-coded path for now
and I'll need to figure out how to modify that so cassci is happy with it.
Actually, [~mambocab], could you look at including that test file in the
upgrade tests in a way that works for CI?
> UnsupportedOperationException when compacting system.size_estimates after 2.1
> -> 2.2 -> 3.0 upgrade
> ---------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-10360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
> Project: Cassandra
> Issue Type: Bug
> Reporter: Andrew Hust
> Assignee: Sylvain Lebresne
> Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2,
> system.log.2.bz2, system.log.bz2
>
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will
> get stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
> at
> org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
> at
> org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
> surface the issue. It can be reproduced with the following (note -- changing
> jdks will need to be altered if you aren't on OSX)
> {code}
> #!/bin/sh
> echo "using java7 for cassandra-2.1 instance"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
> ccm stop
> ccm remove upgrade_nodes
> ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
> ccm start
> ccm node1 stress write n=500K -rate threads=4 -mode native cql3
> sleep 10
> for cver in 3.0
> do
> echo "Draining all nodes"
> ccm node1 nodetool drain
> ccm stop
> echo "switching to java 8"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
> echo "Upgrading to git:cassandra-$cver"
> ccm setdir -v git:cassandra-$cver
> ccm start
> echo "Sleeping to all version migrations"
> sleep 30
> echo "Upgrading sstables"
> ccm node1 nodetool upgradesstables
> ccm node1 nodetool upgradesstables system
> ccm node1 nodetool compact system
> ccm node1 stress write n=500K -rate threads=4 -mode native cql3
> sleep 10
> done
> echo "***********"
> echo "instead of creating churn to cause compaction naturally forcing
> compaction of system keyspace"
> echo "***********"
> ccm node1 nodetool compact system
> ccm stop
> {code}
> The example uses {{nodetool compact system}} but it will also occur with
> {{nodetool upgradesstables system}}. I'm puzzled by that since the script
> runs {{upgradesstables}} on each iteration. Is the system keyspace not
> effected by the command without arguments?
> Ran against:
> 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
> 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}}
> 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}}
> //CC [~enigmacurry]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
