jamthief opened a new issue, #3348:
URL: https://github.com/apache/maven-surefire/issues/3348
### Affected version
3.5.5 (regression vs. 3.5.4)
### Bug description
When several concrete subclasses of a shared abstract test class run in the
same JVM fork, Surefire 3.5.5 emits one `TEST-<Subclass>.xml` per subclass as
expected, but the contents and the `testsuite/@name` are wrong:
1. Every per-subclass file's root `<testsuite>` carries
`name="<AbstractParentFQN>"` instead of the concrete subclass FQN.
2. Each per-subclass file contains the union of testcases from *that*
subclass and every subclass that ran before it in the same fork — not just its
own testcases. The last subclass to run produces a file byte-identical in
content to the aggregate.
3. An additional `TEST-<AbstractParentFQN>.xml` aggregate is also written,
containing the full union.
Net effect: the traditionally-disjoint `TEST-*.xml` set now overlaps
heavily. For N subclasses each contributing k tests, naïvely summing
`tests="…"` across `TEST-*.xml` gives `k·N·(N+1)/2` instead of `k·N` — an
`(N+1)/2 ×` overcount.
### Minimal scenario (hypothetical)
```java
// com/example/shared/SharedTestSuite.java
abstract class SharedTestSuite {
@Test void t001() { /* ... */ }
// ... 100 @Test methods total ...
@Test void t100() { /* ... */ }
}
class RunOnA extends SharedTestSuite { /* fixture A */ }
class RunOnB extends SharedTestSuite { /* fixture B */ }
class RunOnC extends SharedTestSuite { /* fixture C */ }
```
Run with defaults (`forkCount=1`, `reuseForks=true`), JUnit Jupiter.
### Observed on 3.5.5
```
TEST-com.example.shared.RunOnA.xml tests="100" # A only
TEST-com.example.shared.RunOnB.xml tests="200" # A ∪ B
TEST-com.example.shared.RunOnC.xml tests="300" # A ∪ B ∪ C
TEST-com.example.shared.SharedTestSuite.xml tests="300" # identical to
RunOnC.xml
```
All four files have `<testsuite name="com.example.shared.SharedTestSuite"
...>` — the abstract parent, not the concrete subclass.
### Expected (and the 3.5.4 behaviour)
```
TEST-com.example.shared.RunOnA.xml tests="100" # A only
TEST-com.example.shared.RunOnB.xml tests="100" # B only
TEST-com.example.shared.RunOnC.xml tests="100" # C only
```
Each file's `<testsuite name>` is the concrete subclass. No aggregate file
is emitted (or, if one is, it is clearly labelled and the per-subclass files
remain disjoint).
### Impact
- `tests=` sums across per-class XMLs overcount, which silently corrupts
every downstream report that follows the documented convention — Jenkins' JUnit
plugin, GitLab test reports, `surefire-report:report`, Allure, homegrown CI
aggregators.
- Failures are misattributed to the abstract parent rather than the concrete
subclass, making it harder to localise broken fixtures.
- The per-subclass and aggregate files now contain overlapping data,
breaking a long-standing contract that `TEST-*.xml` files partition the test
set.
### Worked on
3.5.4 and earlier: per-subclass files were disjoint and named after the
concrete subclass.
### Workaround
Read only `TEST-<AbstractParentFQN>.xml` (the aggregate) when it exists;
ignore the per-subclass files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
