[
https://issues.apache.org/jira/browse/KAFKA-17978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
PoAn Yang resolved KAFKA-17978.
-------------------------------
Resolution: Fixed
> StreamsUpgradeTest#test_rolling_upgrade_with_2_bounces system tests fail
> ------------------------------------------------------------------------
>
> Key: KAFKA-17978
> URL: https://issues.apache.org/jira/browse/KAFKA-17978
> Project: Kafka
> Issue Type: Test
> Reporter: PoAn Yang
> Assignee: Nicholas Telford
> Priority: Major
>
> Run `TC_PATHS="tests/kafkatest/tests/streams/streams_upgrade_test.py"
> /bin/bash tests/docker/run_tests.sh` on trunk branch. The versions which can
> support fk_joins can't pass `test_rolling_upgrade_with_2_bounces`.
>
> {noformat}
> [INFO:2024-11-09 22:24:00,601]: Triggering test 10 of 19...
> [INFO:2024-11-09 22:24:00,611]: RunnerClient: Loading test {'directory':
> '/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
> 'streams_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name':
> 'test_rolling_upgrade_with_2_bounces', 'injected_args': {'from_version':
> '3.4.1', 'metadata_quorum': 'COMBINED_KRAFT'}}
> [INFO:2024-11-09 22:24:00,619]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
> on run 1/1
> [INFO:2024-11-09 22:24:00,621]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
> Setting up...
> [INFO:2024-11-09 22:24:00,623]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
> Running...
> [INFO:2024-11-09 22:26:26,343]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
> Tearing down...
> [INFO:2024-11-09 22:27:47,017]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
> FAIL: TimeoutError("Never saw output 'processed [0-9]* records from
> topic=data' on ducker@ducker07")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py",
> line 351, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py",
> line 411, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line
> 438, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line
> 137, in test_rolling_upgrade_with_2_bounces
> self.do_stop_start_bounce(p, from_version[:-2], to_version, counter,
> extra_properties)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line
> 402, in do_stop_start_bounce
> err_msg="Never saw output '%s' on " % self.processed_data_msg +
> str(node.account))
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py",
> line 754, in wait_until
> allow_fail=True) == 0, **kwargs)
> File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Never saw output 'processed [0-9]* records from
> topic=data' on ducker@ducker07
> [WARNING:2024-11-09 22:27:47,017]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
> Test requested 6 nodes, used only 5
> [INFO:2024-11-09 22:27:47,017]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
> Data: None
> [INFO:2024-11-09 22:27:47,124]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> [INFO:2024-11-09 22:27:47,125]: Triggering test 11 of 19...
> [INFO:2024-11-09 22:27:47,134]: RunnerClient: Loading test {'directory':
> '/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
> 'streams_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name':
> 'test_rolling_upgrade_with_2_bounces', 'injected_args': {'from_version':
> '3.5.2', 'metadata_quorum': 'COMBINED_KRAFT'}}
> [INFO:2024-11-09 22:27:47,142]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
> on run 1/1
> [INFO:2024-11-09 22:27:47,144]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
> Setting up...
> [INFO:2024-11-09 22:27:47,146]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
> Running...
> [INFO:2024-11-09 22:30:12,455]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
> Tearing down...
> [INFO:2024-11-09 22:31:34,101]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
> FAIL: TimeoutError("Never saw output 'processed [0-9]* records from
> topic=data' on ducker@ducker13")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py",
> line 351, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py",
> line 411, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line
> 438, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line
> 137, in test_rolling_upgrade_with_2_bounces
> self.do_stop_start_bounce(p, from_version[:-2], to_version, counter,
> extra_properties)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line
> 402, in do_stop_start_bounce
> err_msg="Never saw output '%s' on " % self.processed_data_msg +
> str(node.account))
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py",
> line 754, in wait_until
> allow_fail=True) == 0, **kwargs)
> File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Never saw output 'processed [0-9]* records from
> topic=data' on ducker@ducker13
> [WARNING - 2024-11-09 22:31:34,101 - runner_client - log - lineno:459]:
> RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
> Test requested 6 nodes, used only 5
> [WARNING:2024-11-09 22:31:34,101]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
> Test requested 6 nodes, used only 5
> [INFO:2024-11-09 22:31:34,101]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
> Data: None
> [INFO:2024-11-09 22:31:34,208]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> [INFO:2024-11-09 22:31:34,208]: Triggering test 12 of 19...
> [INFO:2024-11-09 22:31:34,218]: RunnerClient: Loading test {'directory':
> '/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
> 'streams_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name':
> 'test_rolling_upgrade_with_2_bounces', 'injected_args': {'from_version':
> '3.6.2', 'metadata_quorum': 'COMBINED_KRAFT'}}
> [INFO:2024-11-09 22:31:34,226]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
> on run 1/1
> [INFO:2024-11-09 22:31:34,228]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
> Setting up...
> [INFO:2024-11-09 22:31:34,229]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
> Running...
> [INFO:2024-11-09 22:33:58,400]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
> Tearing down...
> [INFO:2024-11-09 22:35:19,548]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
> FAIL: TimeoutError("Never saw output 'processed [0-9]* records from
> topic=data' on ducker@ducker07")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py",
> line 351, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py",
> line 411, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line
> 438, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line
> 137, in test_rolling_upgrade_with_2_bounces
> self.do_stop_start_bounce(p, from_version[:-2], to_version, counter,
> extra_properties)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line
> 402, in do_stop_start_bounce
> err_msg="Never saw output '%s' on " % self.processed_data_msg +
> str(node.account))
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py",
> line 754, in wait_until
> allow_fail=True) == 0, **kwargs)
> File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Never saw output 'processed [0-9]* records from
> topic=data' on ducker@ducker07
> [WARNING - 2024-11-09 22:35:19,549 - runner_client - log - lineno:459]:
> RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
> Test requested 6 nodes, used only 5
> [WARNING:2024-11-09 22:35:19,550]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
> Test requested 6 nodes, used only 5
> [INFO:2024-11-09 22:35:19,551]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
> Data: None
> [INFO:2024-11-09 22:35:19,660]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> [INFO:2024-11-09 22:35:19,660]: Triggering test 13 of 19...
> [INFO:2024-11-09 22:35:19,670]: RunnerClient: Loading test {'directory':
> '/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
> 'streams_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name':
> 'test_rolling_upgrade_with_2_bounces', 'injected_args': {'from_version':
> '3.7.1', 'metadata_quorum': 'COMBINED_KRAFT'}}
> [INFO:2024-11-09 22:35:19,678]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
> on run 1/1
> [INFO:2024-11-09 22:35:19,680]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
> Setting up...
> [INFO:2024-11-09 22:35:19,681]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
> Running...
> [INFO:2024-11-09 22:38:00,197]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
> Tearing down...
> [INFO:2024-11-09 22:39:21,823]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
> FAIL: TimeoutError("Never saw output 'processed [0-9]* records from
> topic=data' on ducker@ducker12")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py",
> line 351, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py",
> line 411, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line
> 438, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line
> 137, in test_rolling_upgrade_with_2_bounces
> self.do_stop_start_bounce(p, from_version[:-2], to_version, counter,
> extra_properties)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line
> 402, in do_stop_start_bounce
> err_msg="Never saw output '%s' on " % self.processed_data_msg +
> str(node.account))
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py",
> line 754, in wait_until
> allow_fail=True) == 0, **kwargs)
> File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Never saw output 'processed [0-9]* records from
> topic=data' on ducker@ducker12
> [WARNING - 2024-11-09 22:39:21,823 - runner_client - log - lineno:459]:
> RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
> Test requested 6 nodes, used only 5
> [WARNING:2024-11-09 22:39:21,823]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
> Test requested 6 nodes, used only 5
> [INFO:2024-11-09 22:39:21,824]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
> Data: None
> [INFO:2024-11-09 22:39:21,931]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> [INFO:2024-11-09 22:39:21,931]: Triggering test 14 of 19...
> [INFO:2024-11-09 22:39:21,940]: RunnerClient: Loading test {'directory':
> '/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
> 'streams_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name':
> 'test_rolling_upgrade_with_2_bounces', 'injected_args': {'from_version':
> '3.8.1', 'metadata_quorum': 'COMBINED_KRAFT'}}
> [INFO:2024-11-09 22:39:21,948]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
> on run 1/1
> [INFO:2024-11-09 22:39:21,950]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
> Setting up...
> [INFO:2024-11-09 22:39:21,952]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
> Running...
> [INFO:2024-11-09 22:41:49,029]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
> Tearing down...
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py",
> line 351, in _do_run
> data = self.run_test()
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py",
> line 411, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line
> 438, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line
> 137, in test_rolling_upgrade_with_2_bounces
> self.do_stop_start_bounce(p, from_version[:-2], to_version, counter,
> extra_properties)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line
> 402, in do_stop_start_bounce
> err_msg="Never saw output '%s' on " % self.processed_data_msg +
> str(node.account))
> File
> "/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py",
> line 754, in wait_until
> allow_fail=True) == 0, **kwargs)
> File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line
> 58, in wait_until
> raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
> last_exception
> ducktape.errors.TimeoutError: Never saw output 'processed [0-9]* records from
> topic=data' on ducker@ducker04
> [WARNING - 2024-11-09 22:43:10,413 - runner_client - log - lineno:459]:
> RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
> Test requested 6 nodes, used only 5
> [WARNING:2024-11-09 22:43:10,414]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
> Test requested 6 nodes, used only 5
> [INFO:2024-11-09 22:43:10,414]: RunnerClient:
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
> Data: None
> [INFO:2024-11-09 22:43:10,521]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
