[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17127532#comment-17127532 ] Cam Morris commented on SUREFIRE-1228: -- I just discovered that testng has the ability to retry tests and has for a long time. It's implemented by pointing a specific test to a retry "analyzer" really a retry policy. I suppose this could be implemented in surefire by pointing each test to a default retry analyzer. It might conflict with tests that already have a retry analyzer. Here is the reference: [https://testng.org/doc/documentation-main.html#rerunning] > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris >Assignee: Tibor Digana >Priority: Major > Attachments: TestRandomFail.java, pom.xml > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.10 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, > Flakes: 1" The rerun confuses test runs with different data for retry > attempts. Just one of the data provider tests needs to succeed for the test > to be deemed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111205#comment-17111205 ] Cam Morris commented on SUREFIRE-1228: -- [~tibordigana] nice! Yes, I get that as well. That should help. Looking around the project I totally love your test harness. Cool that you can run against any version and the tests are easy to follow. Nice work! > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris >Assignee: Tibor Digana >Priority: Major > Attachments: TestRandomFail.java, pom.xml > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.10 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, > Flakes: 1" The rerun confuses test runs with different data for retry > attempts. Just one of the data provider tests needs to succeed for the test > to be deemed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110720#comment-17110720 ] Cam Morris commented on SUREFIRE-1228: -- I'm pulling down the code to attempt to implement this in the testng provider. A few pointers of where to start would be nice. > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris >Assignee: Tibor Digana >Priority: Major > Attachments: TestRandomFail.java, pom.xml > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.10 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, > Flakes: 1" The rerun confuses test runs with different data for retry > attempts. Just one of the data provider tests needs to succeed for the test > to be deemed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110718#comment-17110718 ] Cam Morris commented on SUREFIRE-1228: -- Revisiting this 4 years later... I think this should be reopened. It is a bug. If the testng provider doesn't support rerun, then it shouldn't say everything passed. It gives a false positive when a bug in the code should have been found. If rerun isn't supported than the build should fail with an error telling me it isn't supported. > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris >Assignee: Tibor Digana >Priority: Major > Attachments: TestRandomFail.java, pom.xml > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.10 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, > Flakes: 1" The rerun confuses test runs with different data for retry > attempts. Just one of the data provider tests needs to succeed for the test > to be deemed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141940#comment-15141940 ] Cam Morris commented on SUREFIRE-1228: -- Interesting! Without the data provider as a parameter to the @Test annotation it only runs one. rerunFailingTestsCount doesn't have any effect like you said. > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris > Attachments: TestRandomFail.java, pom.xml > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.10 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, > Flakes: 1" The rerun confuses test runs with different data for retry > attempts. Just one of the data provider tests needs to succeed for the test > to be deemed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141878#comment-15141878 ] Cam Morris commented on SUREFIRE-1228: -- Is this a feature of a not-yet released version of surefire, because the tests do rerun. > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris > Attachments: TestRandomFail.java, pom.xml > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.10 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, > Flakes: 1" The rerun confuses test runs with different data for retry > attempts. Just one of the data provider tests needs to succeed for the test > to be deemed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141876#comment-15141876 ] Cam Morris commented on SUREFIRE-1228: -- It seems to be rerunning the tests despite the fact that testNG is obviously in use. And the retry passes the tests when it should be failing them. Without the rerunFailingTestsCount set, the tests fail as expected. > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris > Attachments: TestRandomFail.java, pom.xml > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.10 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, > Flakes: 1" The rerun confuses test runs with different data for retry > attempts. Just one of the data provider tests needs to succeed for the test > to be deemed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cam Morris updated SUREFIRE-1228: - Attachment: pom.xml > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris > Attachments: TestRandomFail.java, pom.xml > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.10 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, > Flakes: 1" The rerun confuses test runs with different data for retry > attempts. Just one of the data provider tests needs to succeed for the test > to be deemed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cam Morris updated SUREFIRE-1228: - Attachment: (was: testng-retry.iml) > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris > Attachments: TestRandomFail.java > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.10 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, > Flakes: 1" The rerun confuses test runs with different data for retry > attempts. Just one of the data provider tests needs to succeed for the test > to be deemed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cam Morris updated SUREFIRE-1228: - Description: Surefire confuses tests run with a data-provider with rerun tests so only one of the data-provided tests need to pass. Steps to reproduce: # Create a testNG test with a data provider that passes with some of the data and fails with others {code:title=TestNG example|language=java} @Test(dataProvider = "succeed") public void fail(boolean succeed) { if (!succeed) { Assert.fail("nope"); } } @DataProvider(name="succeed") public Object[][] arrayIndex() { return new Object[][]{ {true}, {false}}; } {code} # Configure surefire to rerun failed tests {code:title=pom.xml example|language=xml} http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 cam.test testng-retry 1.0-SNAPSHOT org.testng testng 6.9.10 test org.apache.maven.plugins maven-surefire-plugin 2.19.1 1 {code} - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not pass. - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1" The rerun confuses test runs with different data for retry attempts. Just one of the data provider tests needs to succeed for the test to be deemed a success. {code}mvn test ... --- T E S T S --- Running TestRandomFail Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< FAILURE! - in TestRandomFail fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! java.lang.AssertionError: nope at TestRandomFail.fail(TestRandomFail.java:14) Results : Flaked tests: TestRandomFail.fail(TestRandomFail) Run 1: PASS Run 2: TestRandomFail.fail:14 nope Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 2.251 s [INFO] Finished at: 2016-02-10T07:39:21-07:00 [INFO] Final Memory: 9M/244M [INFO] {code} was: Surefire confuses tests run with a data-provider with rerun tests so only one of the data-provided tests need to pass. Steps to reproduce: # Create a testNG test with a data provider that passes with some of the data and fails with others {code:title=TestNG example|language=java} @Test(dataProvider = "succeed") public void fail(boolean succeed) { if (!succeed) { Assert.fail("nope"); } } @DataProvider(name="succeed") public Object[][] arrayIndex() { return new Object[][]{ {true}, {false}}; } {code} # Configure surefire to rerun failed tests {code:title=pom.xml example|language=xml} http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 cam.test testng-retry 1.0-SNAPSHOT org.testng testng 6.9.9 test org.apache.maven.plugins maven-surefire-plugin 2.19.1 1 {code} - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not pass. - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1" The rerun confuses test runs with different data for retry attempts. Just one of the data provider tests needs to succeed for the test to be deemed a success. {code}mvn test ... --- T E S T S --- Running TestRandomFail Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< FAILURE! - in TestRandomFail fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! java.lang.AssertionError: nope at TestRandomFail.fail(TestRandomFail.java:14) Results : Flaked tests: TestRandomFail.fail(TestRandomFail) Run 1: PASS Run 2: TestRandomFail.fail:14 nope Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 [INFO] [INFO] BUILD SUCCESS [INFO]
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140994#comment-15140994 ] Cam Morris commented on SUREFIRE-1228: -- Note that the attached pom has version 6.9.9 of testNG. I just tried the latest release, 6.9.10, with the same results. > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris > Attachments: TestRandomFail.java, testng-retry.iml > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.9 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, > Flakes: 1" The rerun confuses test runs with different data for retry > attempts. Just one of the data provider tests needs to succeed for the test > to be deemed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cam Morris updated SUREFIRE-1228: - Description: Surefire confuses tests run with a data-provider with rerun tests so only one of the data-provided tests need to pass. Steps to reproduce: # Create a testNG test with a data provider that passes with some of the data and fails with others {code:title=TestNG example|language=java} @Test(dataProvider = "succeed") public void fail(boolean succeed) { if (!succeed) { Assert.fail("nope"); } } @DataProvider(name="succeed") public Object[][] arrayIndex() { return new Object[][]{ {true}, {false}}; } {code} # Configure surefire to rerun failed tests {code:title=pom.xml example|language=xml} http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 cam.test testng-retry 1.0-SNAPSHOT org.testng testng 6.9.9 test org.apache.maven.plugins maven-surefire-plugin 2.19.1 1 {code} - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not pass. - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1" The rerun confuses test runs with different data for retry attempts. Just one of the data provider tests needs to succeed for the test to be deemed a success. {code}mvn test ... --- T E S T S --- Running TestRandomFail Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< FAILURE! - in TestRandomFail fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! java.lang.AssertionError: nope at TestRandomFail.fail(TestRandomFail.java:14) Results : Flaked tests: TestRandomFail.fail(TestRandomFail) Run 1: PASS Run 2: TestRandomFail.fail:14 nope Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 2.251 s [INFO] Finished at: 2016-02-10T07:39:21-07:00 [INFO] Final Memory: 9M/244M [INFO] {code} was: Surefire confuses tests run with a data-provider with rerun tests so only one of the data-provided tests need to pass. Steps to reproduce: # Create a testNG test with a data provider that passes with some of the data and fails with others {code:title=TestNG example|language=java} @Test(dataProvider = "succeed") public void fail(boolean succeed) { if (!succeed) { Assert.fail("nope"); } } @DataProvider(name="succeed") public Object[][] arrayIndex() { return new Object[][]{ {true}, {false}}; } {code} # Configure surefire to rerun failed tests {code:title=pom.xml example|language=xml} http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 cam.test testng-retry 1.0-SNAPSHOT org.testng testng 6.9.9 test org.apache.maven.plugins maven-surefire-plugin 2.19.1 1 {code} - *Expected results*: "tests run: 2, Failures: 1..." The tests should not pass. - *Actual results*: The different runs using the provider data are confused for retry attempts. Just one of the data needs to succeed for the tests to be assumed a success. {code}mvn test ... --- T E S T S --- Running TestRandomFail Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< FAILURE! - in TestRandomFail fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! java.lang.AssertionError: nope at TestRandomFail.fail(TestRandomFail.java:14) Results : Flaked tests: TestRandomFail.fail(TestRandomFail) Run 1: PASS Run 2: TestRandomFail.fail:14 nope Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 [INFO] [INFO] BUILD SUCCESS [INFO] --
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cam Morris updated SUREFIRE-1228: - Attachment: testng-retry.iml Attaching a pom.xml file used to reproduce the issue. It is configured to rerun the any failed tests once. > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris > Attachments: TestRandomFail.java, testng-retry.iml > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.9 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: The different runs using the provider data are confused > for retry attempts. Just one of the data needs to succeed for the tests to > be assumed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140963#comment-15140963 ] Cam Morris edited comment on SUREFIRE-1228 at 2/10/16 3:17 PM: --- Attaching a TestNG file to reproduce the issue. It should always fail. It is placed in src/test/java directory was (Author: cmorris_partnet): This file is placed in src/test/java directory > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris > Attachments: TestRandomFail.java > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.9 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: The different runs using the provider data are confused > for retry attempts. Just one of the data needs to succeed for the tests to > be assumed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[ https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cam Morris updated SUREFIRE-1228: - Attachment: TestRandomFail.java This file is placed in src/test/java directory > rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass > --- > > Key: SUREFIRE-1228 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 > Project: Maven Surefire > Issue Type: Bug > Components: TestNG support >Affects Versions: 2.19.1 >Reporter: Cam Morris > Attachments: TestRandomFail.java > > > Surefire confuses tests run with a data-provider with rerun tests so only one > of the data-provided tests need to pass. > Steps to reproduce: > # Create a testNG test with a data provider that passes with some of the data > and fails with others > {code:title=TestNG example|language=java} > @Test(dataProvider = "succeed") > public void fail(boolean succeed) { > if (!succeed) { > Assert.fail("nope"); > } > } > @DataProvider(name="succeed") > public Object[][] arrayIndex() { > return new Object[][]{ {true}, {false}}; > } > {code} > # Configure surefire to rerun failed tests > {code:title=pom.xml example|language=xml} > > http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > 4.0.0 > cam.test > testng-retry > 1.0-SNAPSHOT > > > org.testng > testng > 6.9.9 > test > > > > > > org.apache.maven.plugins > maven-surefire-plugin > 2.19.1 > > 1 > > > > > > {code} > - *Expected results*: "tests run: 2, Failures: 1..." The tests should not > pass. > - *Actual results*: The different runs using the provider data are confused > for retry attempts. Just one of the data needs to succeed for the tests to > be assumed a success. > {code}mvn test > ... > --- > T E S T S > --- > Running TestRandomFail > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< > FAILURE! - in TestRandomFail > fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! > java.lang.AssertionError: nope > at TestRandomFail.fail(TestRandomFail.java:14) > Results : > Flaked tests: > TestRandomFail.fail(TestRandomFail) > Run 1: PASS > Run 2: TestRandomFail.fail:14 nope > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2.251 s > [INFO] Finished at: 2016-02-10T07:39:21-07:00 > [INFO] Final Memory: 9M/244M > [INFO] > > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
Cam Morris created SUREFIRE-1228: Summary: rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass Key: SUREFIRE-1228 URL: https://issues.apache.org/jira/browse/SUREFIRE-1228 Project: Maven Surefire Issue Type: Bug Components: TestNG support Affects Versions: 2.19.1 Reporter: Cam Morris Surefire confuses tests run with a data-provider with rerun tests so only one of the data-provided tests need to pass. Steps to reproduce: # Create a testNG test with a data provider that passes with some of the data and fails with others {code:title=TestNG example|language=java} @Test(dataProvider = "succeed") public void fail(boolean succeed) { if (!succeed) { Assert.fail("nope"); } } @DataProvider(name="succeed") public Object[][] arrayIndex() { return new Object[][]{ {true}, {false}}; } {code} # Configure surefire to rerun failed tests {code:title=pom.xml example|language=xml} http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 cam.test testng-retry 1.0-SNAPSHOT org.testng testng 6.9.9 test org.apache.maven.plugins maven-surefire-plugin 2.19.1 1 {code} - *Expected results*: "tests run: 2, Failures: 1..." The tests should not pass. - *Actual results*: The different runs using the provider data are confused for retry attempts. Just one of the data needs to succeed for the tests to be assumed a success. {code}mvn test ... --- T E S T S --- Running TestRandomFail Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<< FAILURE! - in TestRandomFail fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE! java.lang.AssertionError: nope at TestRandomFail.fail(TestRandomFail.java:14) Results : Flaked tests: TestRandomFail.fail(TestRandomFail) Run 1: PASS Run 2: TestRandomFail.fail:14 nope Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 2.251 s [INFO] Finished at: 2016-02-10T07:39:21-07:00 [INFO] Final Memory: 9M/244M [INFO] {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)