[ https://issues.apache.org/jira/browse/CASSANDRA-12171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375134#comment-15375134 ]
Aleksey Yeschenko commented on CASSANDRA-12171: ----------------------------------------------- Probably just a timeout unaccounted for, given that a node will be down during the upgrade. But I'll have a look. > counter mismatch during rolling upgrade from 2.2 to 3.0 > ------------------------------------------------------- > > Key: CASSANDRA-12171 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12171 > Project: Cassandra > Issue Type: Bug > Reporter: Russ Hatch > Assignee: Aleksey Yeschenko > > This may occur on other versions, but 3.0 is where I observed it recently. > N=RF=3, counter writes at quorum, reads at quorum. > This is being seen on some upgrade tests I'm currently repairing here: > https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix (this > branch is to resolve an issue where counters were not being properly tested > during rolling upgrade tests). > The test runs a continuous counter incrementing process, as well as a > continuous counter checking process. Once a counter value has been verified, > the test code makes it eligible to be incremented again. > The test is encountering the problem when trying to check an expected counter > value and not matching expectations, for example: > {noformat} > Traceback (most recent call last): > File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in > _bootstrap > self.run() > File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run > self._target(*self._args, **self._kwargs) > File > "/home/rhatch/git/cstar/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 210, in counter_checker > tester.assertEqual(expected_count, actual_count) > File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual > assertion_func(first, second, msg=msg) > File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual > raise self.failureException(msg) > AssertionError: 1 != 2 > ERROR > {noformat} > To check if something else could be going on, I did an experiment where I > changed the test to not upgrade nodes (just drain, stop, start) and the > mismatch didn't occur in several attempts. So it appears something about > upgrading is possibly the culprit. > To run the test and repro locally: > {noformat} > grab my dtest branch at > https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix > export UPGRADE_TEST_RUN=true > nosetests -v > upgrade_tests/upgrade_through_versions_test.py:TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_test > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)