[
https://issues.apache.org/jira/browse/KAFKA-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17746622#comment-17746622
]
Federico Valeri edited comment on KAFKA-15239 at 7/25/23 1:59 PM:
------------------------------------------------------------------
Hi [~showuon], thanks for raising this.
QuotaTest uses ProducerPerformanceService which is where the issue is located.
This is the test instance that is failing.
{code}
[INFO:2023-07-24 05:00:40,057]: RunnerClient: Loading test {'directory':
'/opt/kafka-dev/tests/kafkatest/tests/client', 'file_name': 'quota_test.py',
'cls_name': 'QuotaTest', 'method_name': 'test_quota', 'injected_args':
{'quota_type': 'client-id', 'old_client_throttling_behavior': True}}
...
Exception: No output from ProducerPerformance
{code}
If we look at ProducerPerformanceService stderr logs in ./results, we have the
reported exception.
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/kafka/common/utils/ThroughputThrottler
at
org.apache.kafka.tools.ProducerPerformance.start(ProducerPerformance.java:101)
at
org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:52)
Caused by: java.lang.ClassNotFoundException:
org.apache.kafka.common.utils.ThroughputThrottler
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 2 more
{code}
There are other system test instances that use ProducerPerformanceService and
are failing for the same reason. This is the full list.
{code}
TC_PATHS="tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota"
_DUCKTAPE_OPTIONS="--max-parallel=1 --deflake=1 --parameters
'{\"quota_type\":\"client-id\",\"old_client_throttling_behavior\":true}'" bash
tests/docker/run_tests.sh
TC_PATHS="tests/kafkatest/sanity_checks/test_performance_services.py::PerformanceServiceTest.test_version"
_DUCKTAPE_OPTIONS="--max-parallel=1 --deflake=1 --parameters
'{\"version\":\"0.8.2.2\",\"new_consumer\":false}'" bash
tests/docker/run_tests.sh
TC_PATHS="tests/kafkatest/sanity_checks/test_performance_services.py::PerformanceServiceTest.test_version"
_DUCKTAPE_OPTIONS="--max-parallel=1 --deflake=1 --parameters
'{\"version\":\"0.9.0.1\"}'" bash tests/docker/run_tests.sh
TC_PATHS="tests/kafkatest/sanity_checks/test_performance_services.py::PerformanceServiceTest.test_version"
_DUCKTAPE_OPTIONS="--max-parallel=1 --deflake=1 --parameters
'{\"version\":\"0.9.0.1\",\"new_consumer\":false}'" bash
tests/docker/run_tests.sh
TC_PATHS="tests/kafkatest/sanity_checks/test_performance_services.py::PerformanceServiceTest.test_version"
_DUCKTAPE_OPTIONS="--max-parallel=1 --deflake=1 --parameters
'{\"version\":\"1.1.1\",\"new_consumer\":false}'" bash tests/docker/run_tests.sh
{code}
The following commit broke all of them. QuotaTest was working with 3.5.0
revision and no more after cherry picking this commit. This is only on trunk,
so there is no need to mention anywhere.
https://github.com/apache/kafka/commit/125dbb92867c11739afc54c5e546f551f70d7113
Working on a fix.
was (Author: fvaleri):
Hi [~showuon], thanks for raising this.
QuotaTest uses ProducerPerformanceService which is where the issue is located.
This is the test instance that is failing.
{code}
[INFO:2023-07-24 05:00:40,057]: RunnerClient: Loading test {'directory':
'/opt/kafka-dev/tests/kafkatest/tests/client', 'file_name': 'quota_test.py',
'cls_name': 'QuotaTest', 'method_name': 'test_quota', 'injected_args':
{'quota_type': 'client-id', 'old_client_throttling_behavior': True}}
...
Exception: No output from ProducerPerformance
{code}
If we look at ProducerPerformanceService stderr logs in ./results, we have the
reported exception.
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/kafka/common/utils/ThroughputThrottler
at
org.apache.kafka.tools.ProducerPerformance.start(ProducerPerformance.java:101)
at
org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:52)
Caused by: java.lang.ClassNotFoundException:
org.apache.kafka.common.utils.ThroughputThrottler
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 2 more
{code}
There are other system test instances that use ProducerPerformanceService and
are failing for the same reason. This is the full list.
{code}
TC_PATHS="tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota"
_DUCKTAPE_OPTIONS="--max-parallel=1 --deflake=1 --parameters
'{\"quota_type\":\"client-id\",\"old_client_throttling_behavior\":true}'" bash
tests/docker/run_tests.sh
TC_PATHS="tests/kafkatest/sanity_checks/test_performance_services.py::PerformanceServiceTest.test_version"
_DUCKTAPE_OPTIONS="--max-parallel=1 --deflake=0 --parameters
'{\"version\":\"0.8.2.2\",\"new_consumer\":false}'" bash
tests/docker/run_tests.sh
TC_PATHS="tests/kafkatest/sanity_checks/test_performance_services.py::PerformanceServiceTest.test_version"
_DUCKTAPE_OPTIONS="--max-parallel=1 --deflake=0 --parameters
'{\"version\":\"0.9.0.1\"}'" bash tests/docker/run_tests.sh
TC_PATHS="tests/kafkatest/sanity_checks/test_performance_services.py::PerformanceServiceTest.test_version"
_DUCKTAPE_OPTIONS="--max-parallel=1 --deflake=0 --parameters
'{\"version\":\"0.9.0.1\",\"new_consumer\":false}'" bash
tests/docker/run_tests.sh
TC_PATHS="tests/kafkatest/sanity_checks/test_performance_services.py::PerformanceServiceTest.test_version"
_DUCKTAPE_OPTIONS="--max-parallel=1 --deflake=0 --parameters
'{\"version\":\"1.1.1\",\"new_consumer\":false}'" bash tests/docker/run_tests.sh
{code}
The following commit broke all of them. QuotaTest was working with 3.5.0
revision and no more after cherry picking this commit. This is only on trunk,
so there is no need to mention anywhere.
https://github.com/apache/kafka/commit/125dbb92867c11739afc54c5e546f551f70d7113
Working on a fix.
> producerPerformance system test for old client failed after v3.5.0
> ------------------------------------------------------------------
>
> Key: KAFKA-15239
> URL: https://issues.apache.org/jira/browse/KAFKA-15239
> Project: Kafka
> Issue Type: Test
> Components: system tests
> Affects Versions: 3.6.0
> Reporter: Luke Chen
> Priority: Major
>
> While running producer performance tool in system test for old client (ex:
> quota_test), we will try to run with the dev-branch's jar file, to make sure
> it is backward compatible, as described
> [here|https://github.com/apache/kafka/blob/trunk/tests/kafkatest/services/performance/producer_performance.py#L86-L88],.
> {code:java}
> # In order to ensure more consistent configuration between versions, always
> use the ProducerPerformance tool from the development branch {code}
>
> But in KAFKA-14525, we're moving tools from core module to a separate tool
> module, we're actually breaking the backward compatibility. We should fix the
> system test. Also maybe we should also mention anywhere about this backward
> compatibility issue?
> Note:
> This is the command run in system test. Suppose it's testing old client 3.4.0
> (file put under `~/Downloads/kafka_2.13-3.4.0` in my env), and running under
> the latest trunk env.
> {code:java}
> > for file in ./tools/build/libs/kafka-tools*.jar; do
> > CLASSPATH=$CLASSPATH:$file; done; for file in
> > ./tools/build/dependant-libs*/*.jar; do CLASSPATH=$CLASSPATH:$file; done;
> > export CLASSPATH; export
> > KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:config/tools-log4j.properties";
> > KAFKA_OPTS= KAFKA_HEAP_OPTS="-XX:+HeapDumpOnOutOfMemoryError"
> > ~/Downloads/kafka_2.13-3.4.0/bin/kafka-run-class.sh
> > org.apache.kafka.tools.ProducerPerformance --topic test_topic --num-records
> > 50000 --record-size 3000 --throughput -1 --producer-props
> > bootstrap.servers=localhost:9092 client.id=overridden_id
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/kafka/common/utils/ThroughputThrottler
> at
> org.apache.kafka.tools.ProducerPerformance.start(ProducerPerformance.java:101)
> at
> org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:52)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.kafka.common.utils.ThroughputThrottler
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
> ... 2 more
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
