[ https://issues.apache.org/jira/browse/KAFKA-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Igor Soarez resolved KAFKA-16961. --------------------------------- Resolution: Fixed Verified test no longer failing with {code:java} TC_PATHS="tests/kafkatest/tests/core/kraft_upgrade_test.py::TestKRaftUpgrade" bash tests/docker/run_tests.sh {code} after KAFKA-16969 > TestKRaftUpgrade system tests fail in v3.7.1 RC1 > ------------------------------------------------ > > Key: KAFKA-16961 > URL: https://issues.apache.org/jira/browse/KAFKA-16961 > Project: Kafka > Issue Type: Test > Reporter: Luke Chen > Assignee: Igor Soarez > Priority: Blocker > Fix For: 3.8.0, 3.7.1 > > > > > {code:java} > ================================================================================ > SESSION REPORT (ALL TESTS) > ducktape version: 0.11.4 > session_id: 2024-06-14--003 > run time: 86 minutes 13.705 seconds > tests run: 24 > passed: 18 > flaky: 0 > failed: 6 > ignored: 0 > ================================================================================ > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 3 minutes 44.680 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 3 minutes 42.627 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 3 minutes 28.205 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 3 minutes 42.388 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 2 minutes 57.679 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 2 minutes 57.238 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 2 minutes 52.545 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 2 minutes 56.289 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 2 minutes 54.953 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 2 minutes 59.579 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=dev.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 3 minutes 21.016 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=dev.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT > status: PASS > run time: 2 minutes 56.175 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT > status: PASS > run time: 3 minutes 6.505 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT > status: PASS > run time: 3 minutes 8.059 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT > status: PASS > run time: 3 minutes 19.250 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT > status: PASS > run time: 3 minutes 33.163 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT > status: FAIL > run time: 4 minutes 19.037 seconds > TimeoutError("Kafka server didn't finish startup in 120 seconds") > Traceback (most recent call last): > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 186, in _do_run > data = self.run_test() > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 246, in run_test > return self.test_context.function(self.test) > File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line > 433, in wrapper > return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 116, in test_combined_mode_upgrade > self.run_upgrade(from_kafka_version) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in run_upgrade > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", > line 105, in run_produce_consume_validate > core_test_action(*args) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in <lambda> > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 63, in perform_version_change > self.kafka.controller_quorum.start_node(node) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in > start_node > self.wait_for_start(node, monitor, timeout_sec) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in > wait_for_start > monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, > backoff_sec=.25, > File > "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", > line 753, in wait_until > return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % > (self.offset + 1, self.log, pattern), > File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line > 58, in wait_until > raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from > last_exception > ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 > seconds-------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT > status: FAIL > run time: 4 minutes 51.680 seconds > TimeoutError("Kafka server didn't finish startup in 120 seconds") > Traceback (most recent call last): > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 186, in _do_run > data = self.run_test() > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 246, in run_test > return self.test_context.function(self.test) > File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line > 433, in wrapper > return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 116, in test_combined_mode_upgrade > self.run_upgrade(from_kafka_version) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in run_upgrade > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", > line 105, in run_produce_consume_validate > core_test_action(*args) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in <lambda> > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 63, in perform_version_change > self.kafka.controller_quorum.start_node(node) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in > start_node > self.wait_for_start(node, monitor, timeout_sec) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in > wait_for_start > monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, > backoff_sec=.25, > File > "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", > line 753, in wait_until > return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % > (self.offset + 1, self.log, pattern), > File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line > 58, in wait_until > raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from > last_exception > ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 > seconds-------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT > status: FAIL > run time: 5 minutes 1.860 seconds > TimeoutError("Kafka server didn't finish startup in 120 seconds") > Traceback (most recent call last): > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 186, in _do_run > data = self.run_test() > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 246, in run_test > return self.test_context.function(self.test) > File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line > 433, in wrapper > return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 116, in test_combined_mode_upgrade > self.run_upgrade(from_kafka_version) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in run_upgrade > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", > line 105, in run_produce_consume_validate > core_test_action(*args) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in <lambda> > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 63, in perform_version_change > self.kafka.controller_quorum.start_node(node) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in > start_node > self.wait_for_start(node, monitor, timeout_sec) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in > wait_for_start > monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, > backoff_sec=.25, > File > "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", > line 753, in wait_until > return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % > (self.offset + 1, self.log, pattern), > File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line > 58, in wait_until > raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from > last_exception > ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 > seconds-------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT > status: FAIL > run time: 4 minutes 50.405 seconds > TimeoutError("Kafka server didn't finish startup in 120 seconds") > Traceback (most recent call last): > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 186, in _do_run > data = self.run_test() > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 246, in run_test > return self.test_context.function(self.test) > File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line > 433, in wrapper > return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 116, in test_combined_mode_upgrade > self.run_upgrade(from_kafka_version) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in run_upgrade > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", > line 105, in run_produce_consume_validate > core_test_action(*args) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in <lambda> > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 63, in perform_version_change > self.kafka.controller_quorum.start_node(node) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in > start_node > self.wait_for_start(node, monitor, timeout_sec) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in > wait_for_start > monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, > backoff_sec=.25, > File > "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", > line 753, in wait_until > return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % > (self.offset + 1, self.log, pattern), > File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line > 58, in wait_until > raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from > last_exception > ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 > seconds-------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT > status: FAIL > run time: 4 minutes 48.846 seconds > TimeoutError("Kafka server didn't finish startup in 120 seconds") > Traceback (most recent call last): > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 186, in _do_run > data = self.run_test() > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 246, in run_test > return self.test_context.function(self.test) > File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line > 433, in wrapper > return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 116, in test_combined_mode_upgrade > self.run_upgrade(from_kafka_version) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in run_upgrade > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", > line 105, in run_produce_consume_validate > core_test_action(*args) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in <lambda> > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 63, in perform_version_change > self.kafka.controller_quorum.start_node(node) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in > start_node > self.wait_for_start(node, monitor, timeout_sec) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in > wait_for_start > monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, > backoff_sec=.25, > File > "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", > line 753, in wait_until > return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % > (self.offset + 1, self.log, pattern), > File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line > 58, in wait_until > raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from > last_exception > ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 > seconds-------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT > status: FAIL > run time: 4 minutes 19.672 seconds > TimeoutError("Kafka server didn't finish startup in 120 seconds") > Traceback (most recent call last): > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 186, in _do_run > data = self.run_test() > File > "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", > line 246, in run_test > return self.test_context.function(self.test) > File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line > 433, in wrapper > return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 116, in test_combined_mode_upgrade > self.run_upgrade(from_kafka_version) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in run_upgrade > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", > line 105, in run_produce_consume_validate > core_test_action(*args) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 105, in <lambda> > self.run_produce_consume_validate(core_test_action=lambda: > self.perform_version_change(from_kafka_version)) > File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", > line 63, in perform_version_change > self.kafka.controller_quorum.start_node(node) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in > start_node > self.wait_for_start(node, monitor, timeout_sec) > File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in > wait_for_start > monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, > backoff_sec=.25, > File > "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", > line 753, in wait_until > return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % > (self.offset + 1, self.log, pattern), > File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line > 58, in wait_until > raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from > last_exception > ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 > seconds-------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=dev.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT > status: PASS > run time: 3 minutes 11.294 seconds > -------------------------------------------------------------------------------- > test_id: > kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=dev.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT > status: PASS > run time: 3 minutes 9.933 seconds > -------------------------------------------------------------------------------- > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)