[jira] [Commented] (CASSANDRA-12195) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402690#comment-15402690 ] Russ Hatch commented on CASSANDRA-12195: With this (or similar) test moved now to the dtest large job, I don't see any recent failure. In any case, if it recurs I think we'll get better debug output. I think this is safe to resolve for now and can be reopened later if it happens again. > dtest failure in > upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test > - > > Key: CASSANDRA-12195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12195 > Project: Cassandra > Issue Type: Test >Reporter: Sean McCarthy >Assignee: DS Test Eng > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_2_x_To_indev_3_0_x/rolling_upgrade_with_internode_ssl_test > Failed on CassCI build upgrade_tests-all #59 > {code} > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 295, in rolling_upgrade_with_internode_ssl_test > self.upgrade_scenario(rolling=True, internode_ssl=True) > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 352, in upgrade_scenario > self._check_on_subprocs(self.subprocs) > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 409, in _check_on_subprocs > raise RuntimeError(message) > "A subprocess has terminated early. Subprocess statuses: Process-13 > (is_alive: False), Process-14 (is_alive: True), Process-15 (is_alive: True), > Process-16 (is_alive: True), attempting to terminate remaining subprocesses > now. > {code} > node2_debug.log is too large to attach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12195) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382548#comment-15382548 ] Philip Thompson commented on CASSANDRA-12195: - [~rhatch], with no errors in the logs, do we have any idea why these subprocesses may have aborted? > dtest failure in > upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test > - > > Key: CASSANDRA-12195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12195 > Project: Cassandra > Issue Type: Test >Reporter: Sean McCarthy >Assignee: DS Test Eng > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_2_x_To_indev_3_0_x/rolling_upgrade_with_internode_ssl_test > Failed on CassCI build upgrade_tests-all #59 > {code} > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 295, in rolling_upgrade_with_internode_ssl_test > self.upgrade_scenario(rolling=True, internode_ssl=True) > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 352, in upgrade_scenario > self._check_on_subprocs(self.subprocs) > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 409, in _check_on_subprocs > raise RuntimeError(message) > "A subprocess has terminated early. Subprocess statuses: Process-13 > (is_alive: False), Process-14 (is_alive: True), Process-15 (is_alive: True), > Process-16 (is_alive: True), attempting to terminate remaining subprocesses > now. > {code} > node2_debug.log is too large to attach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12195) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382568#comment-15382568 ] Russ Hatch commented on CASSANDRA-12195: Most commonly this is an assertion error killing the process (unhandled exception), but that should in theory be visible on stderr/stdout. My work to transition upgrade tests over to regular test jobs will include giving the subprocesses descriptive names, so at the very least in the future we'll know what process died. > dtest failure in > upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test > - > > Key: CASSANDRA-12195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12195 > Project: Cassandra > Issue Type: Test >Reporter: Sean McCarthy >Assignee: DS Test Eng > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_2_x_To_indev_3_0_x/rolling_upgrade_with_internode_ssl_test > Failed on CassCI build upgrade_tests-all #59 > {code} > Stacktrace > File "/usr/lib/python2.7/unittest/case.py", line 329, in run > testMethod() > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 295, in rolling_upgrade_with_internode_ssl_test > self.upgrade_scenario(rolling=True, internode_ssl=True) > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 352, in upgrade_scenario > self._check_on_subprocs(self.subprocs) > File > "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", > line 409, in _check_on_subprocs > raise RuntimeError(message) > "A subprocess has terminated early. Subprocess statuses: Process-13 > (is_alive: False), Process-14 (is_alive: True), Process-15 (is_alive: True), > Process-16 (is_alive: True), attempting to terminate remaining subprocesses > now. > {code} > node2_debug.log is too large to attach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)