Benjamin Bentmann wrote:
But what I really want is to avoid bothering with suites at all. Sure,
TestNG has a powerful suite concept, nice, maybe we may want to use it
someday in the future but for now, we have just the simple use-case with
non-related test classes, i.e. we are happy by grouping at the class
level.
As I said, I don't think you're using the right tool for the job here.
TestNG is meant to support all that fancy stuff, and behaves a little
differently for that reason. You can get the exact behavior you want from
JUnit.
We have no suite(s), only a directory with some unrelated test classes.
Now Surefire goes ahead and creates a suite, automatically, without my
control, assuming the entire directory is a single suite. This
assumption does not fit our needs.
That's fair, but I claim that this assumption is a good one for people who
are using TestNG, most of whom pick TestNG over JUnit because they need to
do that weird stuff.
Again, if you want class-level logging, I encourage you to take up the
matter with the TestNG team. TestNG isn't meant to work this way today.
dfabulich wrote:
That option sounds really confusing. What would you call it? How would
you document it? You'd have to know a lot about the guts of Surefire and
TestNG for an option like that to make much sense.
Could think of something like "suiteMode=class|package|all":
This option controls how Surefire will assemble your test classes into a
suite. "class" means to run each class individually in its own suite.
"package" groups all classes from the same package into a separate suite.
"all" will aggregate all found test classes in a single suite. Each created
suite will then be passed to the testing framework for execution and will be
run independently of the other suites. Note that test classes that have
dependencies on each other, must be executed in the same suite. Therefore,
the values "class" and "package" should be used with care because Surefire
itself cannot detect such inter-class dependencies.
Doesn't this option smell bad to you? It does to me!
Instead of making this option, wouldn't it be nice if it just logged
correctly? If TestNG notified us when to log, we could log anything we
wanted.
Surefire is meant to be a wrapper around the underlying test framework;
it's not meant to provide features that the frameworks lack. I think this
is one of those cases where trying to provide the feature in Surefire
would be way harder, and would also be fighting the framework.
Say one starts of with JUnit, where Surefire executes classes
individually. Then one goes and converts his tests to use TestNG (they
have a tool for this). Conceptually, you have the same test design, but
now Surefire insists on running all classes in a single job/black-box. I
am sorry, but I still find this odd since I am lacking the required
knowledge about the guts of Surefire and TestNG ;-)
For the record, I agree that the difference between TestNG and JUnit is
strange. I claim that we only find it odd BECAUSE we have painfully
learned about these guts... most people don't notice or care about the
difference.
-Dan
