Josep Prat created KAFKA-17083:
----------------------------------
Summary: KRaft Upgrade Failures in SystemTests
Key: KAFKA-17083
URL: https://issues.apache.org/jira/browse/KAFKA-17083
Project: Kafka
Issue Type: Bug
Components: system tests
Affects Versions: 3.8.0
Reporter: Josep Prat
2 System tests for "TestKRaftUpgrade are consistently failing on 3.8 in the
system tests.
{noformat}
Module: kafkatest.tests.core.kraft_upgrade_test
Class: TestKRaftUpgrade
Method: test_isolated_mode_upgrade
Arguments:
{
"from_kafka_version": "dev",
"metadata_quorum": "ISOLATED_KRAFT"
}
{noformat}
and
{code:java}
Module: kafkatest.tests.core.kraft_upgrade_test
Class: TestKRaftUpgrade
Method: test_combined_mode_upgrade
Arguments:
{
"from_kafka_version": "dev",
"metadata_quorum": "COMBINED_KRAFT"
}
{code}
Failure for Isolated is:
{noformat}
RemoteCommandError({'ssh_config': {'host': 'worker15', 'hostname':
'10.140.39.207', 'user': 'ubuntu', 'port': 22, 'password': None,
'identityfile': '/home/semaphore/kafka-overlay/semaphore-muckrake.pem'},
'hostname': 'worker15', 'ssh_hostname': '10.140.39.207', 'user': 'ubuntu',
'externally_routable_ip': '10.140.39.207', '_logger': <Logger
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=dev.metadata_quorum=ISOLATED_KRAFT-674
(DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at
0x7f07e2e800a0>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at
0x7f07e2f3d070>, '_custom_ssh_exception_checks': None},
'/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server
worker15:9092,worker16:9092,worker17:9092 upgrade --metadata 3.7', 1, b'SLF4J:
Class path contains multiple SLF4J bindings.\nSLF4J: Found binding in
[jar:file:/vagrant/tools/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
Found binding in
[jar:file:/vagrant/trogdor/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.\nSLF4J: Actual binding is of type
[org.slf4j.impl.Reload4jLoggerFactory]\n1 out of 1 operation(s) failed.\n')
Traceback (most recent call last):
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 121, in test_isolated_mode_upgrade
self.run_upgrade(from_kafka_version)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 105, in run_upgrade
self.run_produce_consume_validate(core_test_action=lambda:
self.perform_version_change(from_kafka_version))
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/produce_consume_validate.py",
line 105, in run_produce_consume_validate
core_test_action(*args)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 105, in <lambda>
self.run_produce_consume_validate(core_test_action=lambda:
self.perform_version_change(from_kafka_version))
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 75, in perform_version_change
self.kafka.upgrade_metadata_version(LATEST_STABLE_METADATA_VERSION)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/services/kafka/kafka.py",
line 920, in upgrade_metadata_version
self.run_features_command("upgrade", new_version)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/services/kafka/kafka.py",
line 930, in run_features_command
self.nodes[0].account.ssh(cmd)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 35, in wrapper
return method(self, *args, **kwargs)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 293, in ssh
raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker15: Command
'/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server
worker15:9092,worker16:9092,worker17:9092 upgrade --metadata 3.7' returned
non-zero exit status 1. Remote error message: b'SLF4J: Class path contains
multiple SLF4J bindings.\nSLF4J: Found binding in
[jar:file:/vagrant/tools/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
Found binding in
[jar:file:/vagrant/trogdor/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.\nSLF4J: Actual binding is of type
[org.slf4j.impl.Reload4jLoggerFactory]\n1 out of 1 operation(s) failed.\n'
{noformat}
Failure for combined is:
{noformat}
RemoteCommandError({'ssh_config': {'host': 'worker29', 'hostname':
'10.140.50.210', 'user': 'ubuntu', 'port': 22, 'password': None,
'identityfile': '/home/semaphore/kafka-overlay/semaphore-muckrake.pem'},
'hostname': 'worker29', 'ssh_hostname': '10.140.50.210', 'user': 'ubuntu',
'externally_routable_ip': '10.140.50.210', '_logger': <Logger
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=dev.metadata_quorum=COMBINED_KRAFT-1789
(DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at
0x7f07e2a616a0>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at
0x7f07e25ac7c0>, '_custom_ssh_exception_checks': None},
'/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server
worker29:9092,worker4:9092,worker1:9092 upgrade --metadata 3.7', 1, b'SLF4J:
Class path contains multiple SLF4J bindings.\nSLF4J: Found binding in
[jar:file:/vagrant/tools/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
Found binding in
[jar:file:/vagrant/trogdor/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.\nSLF4J: Actual binding is of type
[org.slf4j.impl.Reload4jLoggerFactory]\n1 out of 1 operation(s) failed.\n')
Traceback (most recent call last):
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 115, in test_combined_mode_upgrade
self.run_upgrade(from_kafka_version)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 105, in run_upgrade
self.run_produce_consume_validate(core_test_action=lambda:
self.perform_version_change(from_kafka_version))
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/produce_consume_validate.py",
line 105, in run_produce_consume_validate
core_test_action(*args)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 105, in <lambda>
self.run_produce_consume_validate(core_test_action=lambda:
self.perform_version_change(from_kafka_version))
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 75, in perform_version_change
self.kafka.upgrade_metadata_version(LATEST_STABLE_METADATA_VERSION)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/services/kafka/kafka.py",
line 920, in upgrade_metadata_version
self.run_features_command("upgrade", new_version)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/services/kafka/kafka.py",
line 930, in run_features_command
self.nodes[0].account.ssh(cmd)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 35, in wrapper
return method(self, *args, **kwargs)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 293, in ssh
raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker29: Command
'/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server
worker29:9092,worker4:9092,worker1:9092 upgrade --metadata 3.7' returned
non-zero exit status 1. Remote error message: b'SLF4J: Class path contains
multiple SLF4J bindings.\nSLF4J: Found binding in
[jar:file:/vagrant/tools/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
Found binding in
[jar:file:/vagrant/trogdor/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.\nSLF4J: Actual binding is of type
[org.slf4j.impl.Reload4jLoggerFactory]\n1 out of 1 operation(s)
failed.\n'{noformat}
Here is an instance of this failure
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
