[
https://issues.apache.org/jira/browse/SUREFIRE-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Javier A. Ortiz updated SUREFIRE-2180:
--------------------------------------
Description:
I'm having a weird issue I can't figure out. It's driving me insane.
I have a Maven project for which I'm running the same command both locally and
on Jenkins. The environment is the same as far as I can tell:
- Maven 3.9.2 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)
- JDK 11 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)
- TestNG 7.8.0
Command:
{code:java}
mvn -Ddependency.surefire.verbose=10 -Pcoverage verify -B -U -Dgroup=PLATFORM
{code}
Relevant POM section:
{code:java}
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.10</version>
</plugin>
...
</plugins>
</pluginManagement>
...
</build>
...
<profiles>
<profile>
<id>coverage</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>com/XXX/framework/dataproviders/**/*</exclude>
<exclude>com/XXX/v2/basetest/**/*</exclude>
<exclude>**/*Exception*</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<!-- attached to Maven test phase -->
<execution>
<id>report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
<!-- Add this checking -->
<execution>
<id>jacoco-check</id>
<goals>
<goal>check</goal>
</goals>
<configuration>
<rules>
<rule>
<element>BUNDLE</element>
<limits>
<limit>
<counter>INSTRUCTION</counter>
<value>COVEREDRATIO</value>
<minimum>60%</minimum>
</limit>
<limit>
<counter>CLASS</counter>
<value>MISSEDCOUNT</value>
<maximum>0</maximum>
</limit>
</limits>
</rule>
<rule>
<element>PACKAGE</element>
<limits>
<limit>
<counter>LINE</counter>
<value>COVEREDRATIO</value>
<minimum>60%</minimum>
</limit>
</limits>
</rule>
</rules>
<skip>${skipTests}</skip>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
{code}
One of the tests that have the issue (only tests with DataProviders) show this
pattern in the logs. Enabled verbose to level 10 in hopes that something would
show up.
Locally:
{code:java}
[2023-06-06 16:55:46.795] [INFO] [TestClass] Creating TestClass for [ClassImpl
class=com.XXX.framework.utils.TimeUtilsTest]
[2023-06-06 16:55:46.796] [INFO] Method public java.lang.Object[][]
com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but
also a return value: ignoring it. Use <suite allow-return-values="true"> to fix
this
[2023-06-06 16:55:46.797] [INFO] [TestClass] Adding method
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest
{code}
...
{code:java}
[2023-06-06 16:55:47.316] [INFO] ===== Test class
com.XXX.framework.utils.TimeUtilsTest
[2023-06-06 16:55:47.316] [INFO] @Test
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:com.XXX.framework.utils.TimeUtilsTest@283eb984]
[2023-06-06 16:55:47.316] [INFO] ======
{code}
...
{code:java}
[TestNG] INVOKING: "Surefire test" -
com.XXX.framework.utils.TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)(value(s):
NANOSECONDS, 60, "60ns", TestNameParameter(customName=60 NANOSECONDS))
{code}
...
Test executes and results, etc.
On Jenkins:
{code:java}
[2023-06-06 21:02:29.270] [INFO] [TestClass] Creating TestClass for [ClassImpl
class=com.XXX.framework.utils.TimeUtilsTest]
[2023-06-06 21:02:29.271] [INFO] Method public java.lang.Object[][]
com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but
also a return value: ignoring it. Use <suite allow-return-values="true"> to fix
this
[2023-06-06 21:02:29.271] [INFO] [TestClass] Adding method
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest
{code}
...
{code:java}
[2023-06-06 21:02:29.529] [INFO] ===== Test class
com.XXX.framework.utils.TimeUtilsTest
[2023-06-06 21:02:29.529] [INFO] @Test
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:com.XXX.framework.utils.TimeUtilsTest@52f9a620]
[2023-06-06 21:02:29.529] [INFO] ======
{code}
I see no other references about the test class in the rest of the trace.
Basically, all tests with a DataProvider don't run.
Maybe someone has a suggestion on what else to look for.
I've tried the following:
- Making sure the proper Test annotation is used (TestNG vs. JUnit
- Remove test groups or any other listener that could be interfering.
- Downgrade the TestNG version.
- Pray.
Here's the investigation on the TestNG side:
[https://github.com/testng-team/testng/issues/2924]
Question in StackOverflow:
https://stackoverflow.com/questions/76418882/why-is-testng-not-running-tests-with-dataprovider
was:
I'm having a weird issue I can't figure out. It's driving me insane.
I have a Maven project for which I'm running the same command both locally and
on Jenkins. The environment is the same as far as I can tell:
- Maven 3.9.2 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)
- JDK 11 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)
- TestNG 7.8.0
Command:
{code}
mvn -Ddependency.surefire.verbose=10 -Pcoverage verify -B -U -Dgroup=PLATFORM
{code}
Relevant POM section:
{code}
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.10</version>
</plugin>
...
</plugins>
</pluginManagement>
...
</build>
...
<profiles>
<profile>
<id>coverage</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>com/XXX/framework/dataproviders/**/*</exclude>
<exclude>com/XXX/v2/basetest/**/*</exclude>
<exclude>**/*Exception*</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<!-- attached to Maven test phase -->
<execution>
<id>report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
<!-- Add this checking -->
<execution>
<id>jacoco-check</id>
<goals>
<goal>check</goal>
</goals>
<configuration>
<rules>
<rule>
<element>BUNDLE</element>
<limits>
<limit>
<counter>INSTRUCTION</counter>
<value>COVEREDRATIO</value>
<minimum>60%</minimum>
</limit>
<limit>
<counter>CLASS</counter>
<value>MISSEDCOUNT</value>
<maximum>0</maximum>
</limit>
</limits>
</rule>
<rule>
<element>PACKAGE</element>
<limits>
<limit>
<counter>LINE</counter>
<value>COVEREDRATIO</value>
<minimum>60%</minimum>
</limit>
</limits>
</rule>
</rules>
<skip>${skipTests}</skip>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
{code}
One of the tests that have the issue (only tests with DataProviders) show this
pattern in the logs. Enabled verbose to level 10 in hopes that something would
show up.
Locally:
{code}
[2023-06-06 16:55:46.795] [INFO] [TestClass] Creating TestClass for [ClassImpl
class=com.XXX.framework.utils.TimeUtilsTest]
[2023-06-06 16:55:46.796] [INFO] Method public java.lang.Object[][]
com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but
also a return value: ignoring it. Use <suite allow-return-values="true"> to fix
this
[2023-06-06 16:55:46.797] [INFO] [TestClass] Adding method
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest
{code}
...
{code}
[2023-06-06 16:55:47.316] [INFO] ===== Test class
com.XXX.framework.utils.TimeUtilsTest
[2023-06-06 16:55:47.316] [INFO] @Test
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:com.XXX.framework.utils.TimeUtilsTest@283eb984]
[2023-06-06 16:55:47.316] [INFO] ======
{code}
...
{code}
[TestNG] INVOKING: "Surefire test" -
com.XXX.framework.utils.TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)(value(s):
NANOSECONDS, 60, "60ns", TestNameParameter(customName=60 NANOSECONDS))
{code}
...
Test executes and results, etc.
On Jenkins:
{code}
[2023-06-06 21:02:29.270] [INFO] [TestClass] Creating TestClass for [ClassImpl
class=com.XXX.framework.utils.TimeUtilsTest]
[2023-06-06 21:02:29.271] [INFO] Method public java.lang.Object[][]
com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but
also a return value: ignoring it. Use <suite allow-return-values="true"> to fix
this
[2023-06-06 21:02:29.271] [INFO] [TestClass] Adding method
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest
{code}
...
{code}
[2023-06-06 21:02:29.529] [INFO] ===== Test class
com.XXX.framework.utils.TimeUtilsTest
[2023-06-06 21:02:29.529] [INFO] @Test
TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
instance:com.XXX.framework.utils.TimeUtilsTest@52f9a620]
[2023-06-06 21:02:29.529] [INFO] ======
{code}
I see no other references about the test class in the rest of the trace.
Basically, all tests with a DataProvider don't run.
Maybe someone has a suggestion on what else to look for.
I've tried the following:
- Making sure the proper Test annotation is used (TestNG vs. JUnit
- Remove test groups or any other listener that could be interfering.
- Downgrade the TestNG version.
- Pray.
Here's the investigation on the TestNG side:
https://github.com/testng-team/testng/issues/2924
> Surefire not invoking tests with TestNG DataProvider
> ----------------------------------------------------
>
> Key: SUREFIRE-2180
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2180
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
> Affects Versions: 3.1.2
> Reporter: Javier A. Ortiz
> Priority: Major
>
> I'm having a weird issue I can't figure out. It's driving me insane.
> I have a Maven project for which I'm running the same command both locally
> and on Jenkins. The environment is the same as far as I can tell:
> - Maven 3.9.2 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)
> - JDK 11 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)
> - TestNG 7.8.0
> Command:
> {code:java}
> mvn -Ddependency.surefire.verbose=10 -Pcoverage verify -B -U -Dgroup=PLATFORM
> {code}
> Relevant POM section:
> {code:java}
> <build>
> <pluginManagement>
> <plugins>
> <plugin>
> <groupId>org.jacoco</groupId>
> <artifactId>jacoco-maven-plugin</artifactId>
> <version>0.8.10</version>
> </plugin>
> ...
> </plugins>
> </pluginManagement>
> ...
> </build>
> ...
> <profiles>
> <profile>
> <id>coverage</id>
> <build>
> <plugins>
> <plugin>
> <groupId>org.jacoco</groupId>
> <artifactId>jacoco-maven-plugin</artifactId>
> <configuration>
> <excludes>
> <exclude>com/XXX/framework/dataproviders/**/*</exclude>
> <exclude>com/XXX/v2/basetest/**/*</exclude>
> <exclude>**/*Exception*</exclude>
> </excludes>
> </configuration>
> <executions>
> <execution>
> <goals>
> <goal>prepare-agent</goal>
> </goals>
> </execution>
> <!-- attached to Maven test phase -->
> <execution>
> <id>report</id>
> <phase>test</phase>
> <goals>
> <goal>report</goal>
> </goals>
> </execution>
> <!-- Add this checking -->
> <execution>
> <id>jacoco-check</id>
> <goals>
> <goal>check</goal>
> </goals>
> <configuration>
> <rules>
> <rule>
> <element>BUNDLE</element>
> <limits>
> <limit>
> <counter>INSTRUCTION</counter>
> <value>COVEREDRATIO</value>
> <minimum>60%</minimum>
> </limit>
> <limit>
> <counter>CLASS</counter>
> <value>MISSEDCOUNT</value>
> <maximum>0</maximum>
> </limit>
> </limits>
> </rule>
> <rule>
> <element>PACKAGE</element>
> <limits>
> <limit>
> <counter>LINE</counter>
> <value>COVEREDRATIO</value>
> <minimum>60%</minimum>
> </limit>
> </limits>
> </rule>
> </rules>
> <skip>${skipTests}</skip>
> </configuration>
> </execution>
> </executions>
> </plugin>
> </plugins>
> </build>
> </profile>
> {code}
> One of the tests that have the issue (only tests with DataProviders) show
> this pattern in the logs. Enabled verbose to level 10 in hopes that something
> would show up.
> Locally:
> {code:java}
> [2023-06-06 16:55:46.795] [INFO] [TestClass] Creating TestClass for
> [ClassImpl class=com.XXX.framework.utils.TimeUtilsTest]
> [2023-06-06 16:55:46.796] [INFO] Method public java.lang.Object[][]
> com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation
> but also a return value: ignoring it. Use <suite allow-return-values="true">
> to fix this
> [2023-06-06 16:55:46.797] [INFO] [TestClass] Adding method
> TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
> instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest
> {code}
> ...
> {code:java}
> [2023-06-06 16:55:47.316] [INFO] ===== Test class
> com.XXX.framework.utils.TimeUtilsTest
> [2023-06-06 16:55:47.316] [INFO] @Test
> TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
> instance:com.XXX.framework.utils.TimeUtilsTest@283eb984]
> [2023-06-06 16:55:47.316] [INFO] ======
> {code}
> ...
> {code:java}
> [TestNG] INVOKING: "Surefire test" -
> com.XXX.framework.utils.TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)(value(s):
> NANOSECONDS, 60, "60ns", TestNameParameter(customName=60 NANOSECONDS))
> {code}
> ...
> Test executes and results, etc.
> On Jenkins:
> {code:java}
> [2023-06-06 21:02:29.270] [INFO] [TestClass] Creating TestClass for
> [ClassImpl class=com.XXX.framework.utils.TimeUtilsTest]
> [2023-06-06 21:02:29.271] [INFO] Method public java.lang.Object[][]
> com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation
> but also a return value: ignoring it. Use <suite allow-return-values="true">
> to fix this
> [2023-06-06 21:02:29.271] [INFO] [TestClass] Adding method
> TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
> instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest
> {code}
> ...
> {code:java}
> [2023-06-06 21:02:29.529] [INFO] ===== Test class
> com.XXX.framework.utils.TimeUtilsTest
> [2023-06-06 21:02:29.529] [INFO] @Test
> TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0,
> instance:com.XXX.framework.utils.TimeUtilsTest@52f9a620]
> [2023-06-06 21:02:29.529] [INFO] ======
> {code}
> I see no other references about the test class in the rest of the trace.
> Basically, all tests with a DataProvider don't run.
> Maybe someone has a suggestion on what else to look for.
> I've tried the following:
> - Making sure the proper Test annotation is used (TestNG vs. JUnit
> - Remove test groups or any other listener that could be interfering.
> - Downgrade the TestNG version.
> - Pray.
> Here's the investigation on the TestNG side:
> [https://github.com/testng-team/testng/issues/2924]
> Question in StackOverflow:
> https://stackoverflow.com/questions/76418882/why-is-testng-not-running-tests-with-dataprovider
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
