aghoussaini opened a new issue, #3428: URL: https://github.com/apache/maven-surefire/issues/3428
### Affected version 3.5.3 through 3.5.6, and current `master` ### Bug description ### Affected version 3.5.3 through 3.5.6, and current `master` ### Bug description Cucumber scenarios are reported as `Tests run: 0` when the JUnit Platform on the test classpath is older than 1.8, even though the scenarios execute and print normally. The XML report has `tests="0"` and no `<testcase>` elements. This is a follow-up to #834 / #840 / #853 / #857 / #3166. Those were fixed by #828 in 3.5.4, but the fix is silently version-gated: it depends on `TestIdentifier.getParentIdObject()`, which only exists from JUnit Platform 1.8. On anything older it no-ops and the original 3.5.3 behaviour comes back. #### Reproduction `pom.xml`: ```xml <project xmlns="http://maven.apache.org/POM/4.0.0";> <modelVersion>4.0.0</modelVersion> <groupId>demo</groupId><artifactId>repro</artifactId><version>1.0</version> <properties> <maven.compiler.source>17</maven.compiler.source> <maven.compiler.target>17</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <sf.version>3.5.6</sf.version> <junit5.version>5.7.2</junit5.version> <platform.version>1.7.2</platform.version> </properties> <dependencies> <dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.13.2</version><scope>test</scope></dependency> <dependency><groupId>io.cucumber</groupId><artifactId>cucumber-java</artifactId><version>7.18.1</version><scope>test</scope></dependency> <dependency><groupId>io.cucumber</groupId><artifactId>cucumber-junit</artifactId><version>7.18.1</version><scope>test</scope></dependency> <dependency><groupId>org.junit.vintage</groupId><artifactId>junit-vintage-engine</artifactId><version>${junit5.version}</version><scope>test</scope></dependency> <dependency><groupId>org.junit.platform</groupId><artifactId>junit-platform-launcher</artifactId><version>${platform.version}</version><scope>test</scope></dependency> </dependencies> <build><plugins> <plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>${sf.version}</version></plugin> <plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-failsafe-plugin</artifactId><version>${sf.version}</version> <executions><execution><goals><goal>integration-test</goal><goal>verify</goal></goals></execution></executions></plugin> </plugins></build> </project> ``` `src/test/java/com/repro/CucumberIT.java`: ```java package com.repro; import io.cucumber.junit.Cucumber; import io.cucumber.junit.CucumberOptions; import org.junit.runner.RunWith; @RunWith(Cucumber.class) @CucumberOptions(features = "classpath:com/repro", glue = "com.repro", plugin = {"pretty"}) public class CucumberIT {} ``` `src/test/java/com/repro/StepDefinitions.java`: ```java package com.repro; import io.cucumber.java.en.Given; public class StepDefinitions { @Given("a step that always passes") public void a_step_that_always_passes() {} } ``` `src/test/resources/com/repro/test.feature`: ```gherkin Feature: dummy Scenario: Run a dummy cucumber test Given a step that always passes ``` Then `mvn verify`. Expected: `Tests run: 1`. Actual: `Tests run: 0`, and `target/failsafe-reports/TEST-com.repro.CucumberIT.xml` contains `tests="0"` with no `<testcase>`. #### Version matrix All rows on Surefire/Failsafe 3.5.6, same project, only the JUnit versions changing: | junit-vintage-engine | junit-platform | Tests run | | --- | --- | --- | | 5.7.2 | 1.7.2 | **0** | | 5.8.2 | 1.8.2 | 1 | | 5.9.3 | 1.9.3 | 1 | | 5.10.3 | 1.10.3 | 1 | Mixed versions fail too — engine 1.7.2 with `junit-platform-launcher` declared at 1.10.3, and engine 1.10.3 with launcher at 1.7.2, both report 0. `JUnitPlatformProviderInfo.alignProviderVersions` re-resolves the launcher to match the detected `junit-platform-commons`, so bumping the launcher alone doesn't change what runs: ``` [DEBUG] Resolving artifact org.junit.platform:junit-platform-launcher:1.7.2 ``` The same project on **3.5.2** with platform 1.7.2 reports `Tests run: 1`, so this is a regression rather than a new baseline requirement. #### Root cause 3.5.3 (#815, SUREFIRE-1643) made `TestSetRunListener` bucket stats per `ReportEntry.getSourceName()`: ```java private TestSetStats getTestSetStats(ReportEntry report) { if (statPerSourceName) { return detailsPerSource.computeIfAbsent( report.getSourceName(), s -> new TestSetStats(trimStackTrace, isPlainFormat)); } return currentTestSetStats; } ``` Cucumber's scenario descriptors carry no `ClassSource`, so their source name is derived from the feature's display name, while `testSetCompleted` arrives under the runner class name. Two different keys, and the flush reads an empty bucket. #828 fixed that by resolving the class-level name from the top-level parent. But the walk gives up when the 1.8 API is absent (`RunListenerAdapter`): ```java private TestIdentifier findTopParent(TestIdentifier testIdentifier) { if (!hasParentId(testIdentifier)) { return testIdentifier; } ... } private boolean hasParentId(TestIdentifier testIdentifier) { Method getParentIdObjectMethod = ReflectionUtils.tryGetMethod(testIdentifier.getClass(), "getParentIdObject"); if (getParentIdObjectMethod == null) { return false; } ... } ``` On a pre-1.8 launcher, `hasParentId` returns `false`, `findTopParent` hands back the scenario descriptor itself, that has no `ClassSource`, `classLevelName` stays empty, and `toClassMethodName` falls through to the feature display name. Same mismatch as 3.5.3. #### Suggested fix `TestPlan.getParent(TestIdentifier)` has been in the API since 1.0 and does the same job as the reflective `getParentIdObject` path. Having `findTopParent` fall back to it instead of returning the input would make the #828 fix work on every supported platform version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
