Ewen Cheslack-Postava created KAFKA-4594:
--------------------------------------------
Summary: Annotate integration tests and provide gradle build
targets to run subsets of tests
Key: KAFKA-4594
URL: https://issues.apache.org/jira/browse/KAFKA-4594
Project: Kafka
Issue Type: Bug
Components: unit tests
Reporter: Ewen Cheslack-Postava
Priority: Minor
It's possible to annotate unit tests and control which ones run based on these
annotations. Our tests have grown to take quite a long time because we've added
a lot of relatively expensive integration tests. Because these tests are all
interleaved with the true unit tests and block things like checkstyle from
running, we've ended up with pretty bad turn around time on feedback on simple
PR issues.
We could make this a lot nicer, and encourage users to actually run some tests
before submitting PRs with problems, by making it easier to get lightweight
feedback based only on unit tests. We'd want to annotate integration tests
(which can probably mostly be accomplished by annotating just a few shared
{{*TestHarness}} classes and then setting up the build tasks and dependencies
to run things in fast-fail, more user-friendly order. In particular, it'd be
nice to:
a) Have gradle tasks that can run just the unit tests + checkstyle. This
provides devs a *much* faster way to get at least basic feedback about whether
there are problems with their PR. PR Jenkins builds will also give much faster
feedback (and avoid clogging up the Jenkins slaves with lots of unnecessary
work that will ultimately just fail on a unit test anyway).
b) Have the order of tests be 1) build everything and run their unit tests, 2)
run checkstyle against all components, then 3) run integration tests. This
could help address one of the frustrating things about checkstyle tests today,
which are put after all tests to avoid annoying failures during development,
but also results in a lot of folks missing simple checkstyle issues before
submitting PRs.
The two obvious options I see for this are:
1. In JUnit4 (which we currently use), use Categories. These seem to
unfortunately still be under an experimental package. We pin to a specific
version of JUnit, so this may not be a huge issue. But it does mean any future
migration would be more painful as the annotations & integration with the test
runner would need to change in the future.
2. Wait for JUnit5 with its built-in Tag support. This would be tied to making
JDK8 the minimum supported Java version. Given current plans that would mean we
couldn't do this until at least 0.11.0.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
