Guozhang Wang created KAFKA-1589:
------------------------------------
Summary: Strengthen System Tests
Key: KAFKA-1589
URL: https://issues.apache.org/jira/browse/KAFKA-1589
Project: Kafka
Issue Type: Bug
Reporter: Guozhang Wang
Fix For: 0.9.0
Although the system test code is also part of the open source repository, not
too much attention is paid to this module today. The incurred results is that
we keep breaking the system tests with either changes on the admin tools, or
library upgrades that change the APIs like Zookeeper. And when the system tests
breaks / hangs / etc, it is also hard to debug the issue. We need to treat the
system test suite just as part of the open source code.
Based on my personal experience trouble shooting system tests, I would propose
doing at least the follow enhancement around system tests.
1. Add unit tests for all system util test tools, for example:
kafka_system_test_utils.get_controller_attributes
kafka_system_test_utils.get_leader_for
2. Add exception handling logic in the python test framework to clean-up the
testbed upon failures, so that the subsequent test cases will not be affected.
3. Remove timing based mechanism such as "sleep(5000) to wait for metadata to
be propagated" as much as possible to avoid transient failures.
After those enhancements, we should probably also pick a very small subset (say
one from each suite) of the system test cases into the patch reviewing process
along with the unit tests.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
