I'm busily wiring up the tests found in maven-surefire-plugin/src/it into
the new IT module I set up earlier. One of these, "test7", tests a
project that has a JUnit test and a TestNG test in the same project.
The Surefire documentation says that if you put both TestNG and JUnit on
your classpath, TestNG is used to run all of the tests:
http://maven.apache.org/plugins/maven-surefire-plugin/usage.html
"[...] (including if you are using TestNG to execute your JUnit tests,
which occurs by default if both are present in Surefire)."
Currently test7 is failing, because the JUnit test isn't getting run; only
the TestNG test is running.
I think that makes sense: Surefire is just presenting both classes to
TestNG as classes to run, but we don't explictly set junit=true in TestNG,
so the JUnit test appears to have no tests in it (it has no @Test
annotations).
Unfortunately, if you DO set junit=true in TestNG, you only get the JUnit
test to run! The only way to get TestNG to run both JUnit and non-JUnit
tests is to separate them out in advance, passing in a list of JUnit tests
marked with junit=true, and a separate list of TestNG tests.
I think that whoever wrote that bit of documentation assumed (incorrectly)
that TestNG had already written code that could automatically determine
the difference between TestNG tests and JUnit tests and "do the right
thing" with each. (There are definitely some unclear cases; what about a
JUnit test with some TestNG annotations that would contradict the default
JUnit semantics? This isn't even terribly unlikely, especially if you're
trying to gradually convert your tests over to TestNG.)
As I see it, here are the options:
1) Don't support mixing JUnit tests with TestNG tests in a single Surefire
execution. Clever users can, if they absolutely must, try to configure
two executions of Surefire, one of which has junit=true, the other of
which doesn't. The advantage of this is that this is what the code
already does.
2) Try to write the code to separate out the tests ourselves in Surefire.
(This is high magic... I don't think I want to do this.)
Frankly, I don't like either of the options, but if I had to pick, I'd
pick #1.
More generally, I think we're stuck in a difficult spot because of the
design assumptions of Surefire. (I'll write up a separate post on what I
think about that.)
-Dan
