[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141876#comment-15141876
]
Cam Morris commented on SUREFIRE-1228:
--------------------------------------
It seems to be rerunning the tests despite the fact that testNG is obviously in
use. And the retry passes the tests when it should be failing them. Without
the rerunFailingTestsCount set, the tests fail as expected.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> -------------------------------------------------------------------
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
> Affects Versions: 2.19.1
> Reporter: Cam Morris
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <project xmlns="http://maven.apache.org/POM/4.0.0";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
> <modelVersion>4.0.0</modelVersion>
> <groupId>cam.test</groupId>
> <artifactId>testng-retry</artifactId>
> <version>1.0-SNAPSHOT</version>
> <dependencies>
> <dependency>
> <groupId>org.testng</groupId>
> <artifactId>testng</artifactId>
> <version>6.9.10</version>
> <scope>test</scope>
> </dependency>
> </dependencies>
> <build>
> <plugins>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.19.1</version>
> <configuration>
> <rerunFailingTestsCount>1</rerunFailingTestsCount>
> </configuration>
> </plugin>
> </plugins>
> </build>
> </project>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
> ------------------------------------------------------------------------
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
