Joris-Lambrechts-CCVLab opened a new issue, #3264:
URL: https://github.com/apache/maven-surefire/issues/3264
### Affected version
3.5.4
### Bug description
Hello,
We had some issues with tests not causing builds to fail. After debugging
the plugin, I found that the reported test is a combination of the test suite
and test method name. The test class name is not used. As a result, if we have
multiple test classes with test methods that have identical names, the reported
result of one test affects the other.
I've attached a little sample that reproduces the behaviour. I expect the
build to fail, but it does not.
```
[INFO] Scanning for projects...
[INFO]
[INFO] -------------< org.apache.maven:maven-surefire-plugin-bug
>-------------
[INFO] Building Archetype - maven-surefire-plugin-bug 1.0-SNAPSHOT
[INFO] from pom.xml
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- resources:3.3.1:resources (default-resources) @
maven-surefire-plugin-bug ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] Copying 4 resources from src/main/resources to target/classes
[INFO]
[INFO] --- compiler:3.13.0:compile (default-compile) @
maven-surefire-plugin-bug ---
[INFO] No sources to compile
[INFO]
[INFO] --- resources:3.3.1:testResources (default-testResources) @
maven-surefire-plugin-bug ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
/Users/jorislambrechts/Desktop/maven-surefire-plugin-bug/src/test/resources
[INFO]
[INFO] --- compiler:3.13.0:testCompile (default-testCompile) @
maven-surefire-plugin-bug ---
[INFO] Recompiling the module because of changed source code.
[WARNING] File encoding has not been set, using platform encoding UTF-8,
i.e. build is platform dependent!
[INFO] Compiling 3 source files with javac [debug target 1.8] to
target/test-classes
[WARNING] bootstrap class path is not set in conjunction with -source 8
not setting the bootstrap class path may lead to class files that cannot
run on JDK 8
--release 8 is recommended instead of -source 8 -target 1.8 because it
sets the bootstrap class path automatically
[WARNING] source value 8 is obsolete and will be removed in a future release
[WARNING] target value 8 is obsolete and will be removed in a future release
[WARNING] To suppress warnings about obsolete options, use -Xlint:-options.
[INFO]
[INFO] --- surefire:3.5.4:test (default-test) @ maven-surefire-plugin-bug ---
[INFO] Using auto detected provider
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[INFO] Running org.apache.maven.surefire.SuccessTest
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
0.016 s <<< FAILURE! -- in org.apache.maven.surefire.SuccessTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.011 s <<<
FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018
s -- in org.apache.maven.surefire.FailTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.028
s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
0.002 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<<
FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004
s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
0.002 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<<
FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003
s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
0.003 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<<
FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005
s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
0.001 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<<
FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003
s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
0.004 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.003 s <<<
FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009
s -- in org.apache.maven.surefire.RunSuite
[INFO]
[INFO] Results:
[INFO]
[WARNING] Flakes:
[WARNING] org.apache.maven.surefire.RunSuite.test
[INFO] Run 1: PASS
[ERROR] Run 2: RunSuite.test
[ERROR] Run 3: RunSuite.test
[ERROR] Run 4: RunSuite.test
[ERROR] Run 5: RunSuite.test
[ERROR] Run 6: RunSuite.test
[ERROR] Run 7: RunSuite.test
[INFO]
[INFO]
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO]
[INFO]
[INFO] --- surefire:3.5.4:test (integration-suite) @
maven-surefire-plugin-bug ---
[INFO] Skipping execution of surefire because it has already been run for
this configuration
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 1.512 s
[INFO] Finished at: 2026-02-16T16:36:47+01:00
[INFO]
------------------------------------------------------------------------
```
I did not create a PR because I'm not sure whether this is as intended or if
I'm configuring the plugin incorrectly.
Thanks,
Joris
[maven-surefire-plugin-bug.zip](https://github.com/user-attachments/files/25344862/maven-surefire-plugin-bug.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
