[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127792#comment-17127792
]
Tibor Digana commented on SUREFIRE-1228:
[~cmorris_partnet] possibly yes in the future. Currently the users want us to
fork the reruns in separate JVMs. So many people have many other requirements.
But yes, your requirement is visible too.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127532#comment-17127532
]
Cam Morris commented on SUREFIRE-1228:
--
I just discovered that testng has the ability to retry tests and has for a
long time. It's implemented by pointing a specific test to a retry
"analyzer" really a retry policy. I suppose this could be implemented in
surefire by pointing each test to a default retry analyzer. It might conflict
with tests that already have a retry analyzer. Here is the reference:
[https://testng.org/doc/documentation-main.html#rerunning]
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111227#comment-17111227
]
Tibor Digana commented on SUREFIRE-1228:
[~cmorris_partnet]
If you have the ambition to extend our integration tests with this use case and
we don't have it already, feel free to open a new pull request on GitHub and I
will review your commit.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111205#comment-17111205
]
Cam Morris commented on SUREFIRE-1228:
--
[~tibordigana] nice! Yes, I get that as well. That should help. Looking
around the project I totally love your test harness. Cool that you can run
against any version and the tests are easy to follow. Nice work!
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110749#comment-17110749
]
Tibor Digana commented on SUREFIRE-1228:
[~cmorris_partnet]
Regardless of the "rerun", check it out with the project version
3.0.0-SNAPSHOT. We implemented another fix which identifies distinct runs of
each test. There should be total=2 and failure=1.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110720#comment-17110720
]
Cam Morris commented on SUREFIRE-1228:
--
I'm pulling down the code to attempt to implement this in the testng provider.
A few pointers of where to start would be nice.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110718#comment-17110718
]
Cam Morris commented on SUREFIRE-1228:
--
Revisiting this 4 years later... I think this should be reopened. It is a
bug. If the testng provider doesn't support rerun, then it shouldn't say
everything passed. It gives a false positive when a bug in the code should
have been found. If rerun isn't supported than the build should fail with an
error telling me it isn't supported.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505863#comment-15505863
]
Tibor Digana commented on SUREFIRE-1228:
[~juherr]
The rerun feature does not exist in TestNG provider.
If somebody has such ambition, she/he can implement it in TestNG provider since
of certain TestNG version.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505780#comment-15505780
]
Julien Herr commented on SUREFIRE-1228:
---
I don't know how rerunFailingTestsCount is implemented in the testng provider
(maybe it is not) but testng provides a similar feature out of the box named
RetryAnaylzer.
A workaround could be to replace rerunFailingTestsCount usage by it,
rerunFailingTestsCount could use it.
It is possible to rerun only a part of data provider tests but the feature was
broken for a while and I fixed it few days/weeks ago (should be available in
the latest version)
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504970#comment-15504970
]
Tibor Digana commented on SUREFIRE-1228:
[~cmorris_partnet]
[~eguy]
Please discuss this issue with TestNG team in GitHub regarding
{{@DataProvider}}.
I think {{rerunFailingTestsCount}} was misleading.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207909#comment-15207909
]
Tibor Digana commented on SUREFIRE-1228:
Simply do not use rerun with TestNG. It was not designed for TestNG, but faster
you can just set it to false in your POM.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204104#comment-15204104
]
Iakovenko Oksana commented on SUREFIRE-1228:
I have problems with rerunFailingTestsCount + TestNG related to this isssue:
For reproduce it use the same pom.xml as in this issue, but Test is without
@DataProvider :
@Test
public void fail() {
Assert.fail("nope");
}
}
And when I run 'mvn test ' it seems like parametr rerunFailingTestsCount does
not work:
---
T E S T S
---
Running oiak.rerunFailed.TestNgEGExample
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.732 sec <<<
FAILURE! - in oiak.rerunFailed.TestNgEGExample
fail(oiak.rerunFailed.TestNgEGExample) Time elapsed: 0.04 sec <<< FAILURE!
java.lang.AssertionError: nope
at oiak.rerunFailed.TestNgEGExample.fail(TestNgEGExample.java:10)
Results :
Failed tests:
TestNgEGExample.fail:10 nope
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD FAILURE
...
I expect this Test is rerun.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140994#comment-15140994
]
Cam Morris commented on SUREFIRE-1228:
--
Note that the attached pom has version 6.9.9 of testNG. I just tried the
latest release, 6.9.10, with the same results.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, testng-retry.iml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.9
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141855#comment-15141855
]
Tibor Digana commented on SUREFIRE-1228:
[~cmorris_partnet]
rerunFailingTestsCount is applicable only to {{surefire-junit4}} or
{{surefire-junit47}} provider.
As soon as you specify TestNG dependency in POM then {{surefire-testng}}
provider is activated and the rerun is not applicable.
Do you need more help?
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, testng-retry.iml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141878#comment-15141878
]
Cam Morris commented on SUREFIRE-1228:
--
Is this a feature of a not-yet released version of surefire, because the tests
do rerun.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141876#comment-15141876
]
Cam Morris commented on SUREFIRE-1228:
--
It seems to be rerunning the tests despite the fact that testNG is obviously in
use. And the retry passes the tests when it should be failing them. Without
the rerunFailingTestsCount set, the tests fail as expected.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141940#comment-15141940
]
Cam Morris commented on SUREFIRE-1228:
--
Interesting! Without the data provider as a parameter to the @Test annotation
it only runs one. rerunFailingTestsCount doesn't have any effect like you said.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141926#comment-15141926
]
Tibor Digana commented on SUREFIRE-1228:
I guess it is just wrong report, see the doc
https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#rerunFailingTestsCount
Can you make a try with a pure test class and random assert and one method?
No data provider and only pure @Test annotation of TestNG. This should never
use the rerun.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
