Cam Morris created SUREFIRE-1228:
------------------------------------
Summary: rerunFailingTestsCount + TestNG + @DataProvider = failed
tests pass
Key: SUREFIRE-1228
URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
Project: Maven Surefire
Issue Type: Bug
Components: TestNG support
Affects Versions: 2.19.1
Reporter: Cam Morris
Surefire confuses tests run with a data-provider with rerun tests so only one
of the data-provided tests need to pass.
Steps to reproduce:
# Create a testNG test with a data provider that passes with some of the data
and fails with others
{code:title=TestNG example|language=java}
@Test(dataProvider = "succeed")
public void fail(boolean succeed) {
if (!succeed) {
Assert.fail("nope");
}
}
@DataProvider(name="succeed")
public Object[][] arrayIndex() {
return new Object[][]{ {true}, {false}};
}
{code}
# Configure surefire to rerun failed tests
{code:title=pom.xml example|language=xml}
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
<modelVersion>4.0.0</modelVersion>
<groupId>cam.test</groupId>
<artifactId>testng-retry</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.9.9</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<rerunFailingTestsCount>1</rerunFailingTestsCount>
</configuration>
</plugin>
</plugins>
</build>
</project>
{code}
- *Expected results*: "tests run: 2, Failures: 1..." The tests should not
pass.
- *Actual results*: The different runs using the provider data are confused for
retry attempts. Just one of the data needs to succeed for the tests to be
assumed a success.
{code}mvn test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestRandomFail
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
FAILURE! - in TestRandomFail
fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
java.lang.AssertionError: nope
at TestRandomFail.fail(TestRandomFail.java:14)
Results :
Flaked tests:
TestRandomFail.fail(TestRandomFail)
Run 1: PASS
Run 2: TestRandomFail.fail:14 nope
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.251 s
[INFO] Finished at: 2016-02-10T07:39:21-07:00
[INFO] Final Memory: 9M/244M
[INFO] ------------------------------------------------------------------------
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
