[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127532#comment-17127532
]
Cam Morris commented on SUREFIRE-1228:
--
I just discovered that testng has the ability to retry tests and has for a
long time. It's implemented by pointing a specific test to a retry
"analyzer" really a retry policy. I suppose this could be implemented in
surefire by pointing each test to a default retry analyzer. It might conflict
with tests that already have a retry analyzer. Here is the reference:
[https://testng.org/doc/documentation-main.html#rerunning]
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111205#comment-17111205
]
Cam Morris commented on SUREFIRE-1228:
--
[~tibordigana] nice! Yes, I get that as well. That should help. Looking
around the project I totally love your test harness. Cool that you can run
against any version and the tests are easy to follow. Nice work!
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110720#comment-17110720
]
Cam Morris commented on SUREFIRE-1228:
--
I'm pulling down the code to attempt to implement this in the testng provider.
A few pointers of where to start would be nice.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110718#comment-17110718
]
Cam Morris commented on SUREFIRE-1228:
--
Revisiting this 4 years later... I think this should be reopened. It is a
bug. If the testng provider doesn't support rerun, then it shouldn't say
everything passed. It gives a false positive when a bug in the code should
have been found. If rerun isn't supported than the build should fail with an
error telling me it isn't supported.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
>Assignee: Tibor Digana
>Priority: Major
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
Cam Morris created SUREFIRE-1228:
Summary: rerunFailingTestsCount + TestNG + @DataProvider = failed
tests pass
Key: SUREFIRE-1228
URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
Project: Maven Surefire
Issue Type: Bug
Components: TestNG support
Affects Versions: 2.19.1
Reporter: Cam Morris
Surefire confuses tests run with a data-provider with rerun tests so only one
of the data-provided tests need to pass.
Steps to reproduce:
# Create a testNG test with a data provider that passes with some of the data
and fails with others
{code:title=TestNG example|language=java}
@Test(dataProvider = "succeed")
public void fail(boolean succeed) {
if (!succeed) {
Assert.fail("nope");
}
}
@DataProvider(name="succeed")
public Object[][] arrayIndex() {
return new Object[][]{ {true}, {false}};
}
{code}
# Configure surefire to rerun failed tests
{code:title=pom.xml example|language=xml}
http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0
cam.test
testng-retry
1.0-SNAPSHOT
org.testng
testng
6.9.9
test
org.apache.maven.plugins
maven-surefire-plugin
2.19.1
1
{code}
- *Expected results*: "tests run: 2, Failures: 1..." The tests should not
pass.
- *Actual results*: The different runs using the provider data are confused for
retry attempts. Just one of the data needs to succeed for the tests to be
assumed a success.
{code}mvn test
...
---
T E S T S
---
Running TestRandomFail
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
FAILURE! - in TestRandomFail
fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
java.lang.AssertionError: nope
at TestRandomFail.fail(TestRandomFail.java:14)
Results :
Flaked tests:
TestRandomFail.fail(TestRandomFail)
Run 1: PASS
Run 2: TestRandomFail.fail:14 nope
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 2.251 s
[INFO] Finished at: 2016-02-10T07:39:21-07:00
[INFO] Final Memory: 9M/244M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140994#comment-15140994
]
Cam Morris commented on SUREFIRE-1228:
--
Note that the attached pom has version 6.9.9 of testNG. I just tried the
latest release, 6.9.10, with the same results.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, testng-retry.iml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.9
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cam Morris updated SUREFIRE-1228:
-
Attachment: testng-retry.iml
Attaching a pom.xml file used to reproduce the issue. It is configured to
rerun the any failed tests once.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, testng-retry.iml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.9
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: The different runs using the provider data are confused
> for retry attempts. Just one of the data needs to succeed for the tests to
> be assumed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cam Morris updated SUREFIRE-1228:
-
Description:
Surefire confuses tests run with a data-provider with rerun tests so only one
of the data-provided tests need to pass.
Steps to reproduce:
# Create a testNG test with a data provider that passes with some of the data
and fails with others
{code:title=TestNG example|language=java}
@Test(dataProvider = "succeed")
public void fail(boolean succeed) {
if (!succeed) {
Assert.fail("nope");
}
}
@DataProvider(name="succeed")
public Object[][] arrayIndex() {
return new Object[][]{ {true}, {false}};
}
{code}
# Configure surefire to rerun failed tests
{code:title=pom.xml example|language=xml}
http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0
cam.test
testng-retry
1.0-SNAPSHOT
org.testng
testng
6.9.10
test
org.apache.maven.plugins
maven-surefire-plugin
2.19.1
1
{code}
- *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
pass.
- *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes:
1" The rerun confuses test runs with different data for retry attempts. Just
one of the data provider tests needs to succeed for the test to be deemed a
success.
{code}mvn test
...
---
T E S T S
---
Running TestRandomFail
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
FAILURE! - in TestRandomFail
fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
java.lang.AssertionError: nope
at TestRandomFail.fail(TestRandomFail.java:14)
Results :
Flaked tests:
TestRandomFail.fail(TestRandomFail)
Run 1: PASS
Run 2: TestRandomFail.fail:14 nope
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 2.251 s
[INFO] Finished at: 2016-02-10T07:39:21-07:00
[INFO] Final Memory: 9M/244M
[INFO]
{code}
was:
Surefire confuses tests run with a data-provider with rerun tests so only one
of the data-provided tests need to pass.
Steps to reproduce:
# Create a testNG test with a data provider that passes with some of the data
and fails with others
{code:title=TestNG example|language=java}
@Test(dataProvider = "succeed")
public void fail(boolean succeed) {
if (!succeed) {
Assert.fail("nope");
}
}
@DataProvider(name="succeed")
public Object[][] arrayIndex() {
return new Object[][]{ {true}, {false}};
}
{code}
# Configure surefire to rerun failed tests
{code:title=pom.xml example|language=xml}
http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0
cam.test
testng-retry
1.0-SNAPSHOT
org.testng
testng
6.9.9
test
org.apache.maven.plugins
maven-surefire-plugin
2.19.1
1
{code}
- *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
pass.
- *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes:
1" The rerun confuses test runs with different data for retry attempts. Just
one of the data provider tests needs to succeed for the test to be deemed a
success.
{code}mvn test
...
---
T E S T S
---
Running TestRandomFail
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
FAILURE! - in TestRandomFail
fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
java.lang.AssertionError: nope
at TestRandomFail.fail(TestRandomFail.java:14)
Results :
Flaked tests:
TestRandomFail.fail(TestRandomFail)
Run 1: PASS
Run 2: TestRandomFail.fail:14 nope
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[jira] [Comment Edited] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140963#comment-15140963
]
Cam Morris edited comment on SUREFIRE-1228 at 2/10/16 3:17 PM:
---
Attaching a TestNG file to reproduce the issue. It should always fail. It is
placed in src/test/java directory
was (Author: cmorris_partnet):
This file is placed in src/test/java directory
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.9
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: The different runs using the provider data are confused
> for retry attempts. Just one of the data needs to succeed for the tests to
> be assumed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cam Morris updated SUREFIRE-1228:
-
Description:
Surefire confuses tests run with a data-provider with rerun tests so only one
of the data-provided tests need to pass.
Steps to reproduce:
# Create a testNG test with a data provider that passes with some of the data
and fails with others
{code:title=TestNG example|language=java}
@Test(dataProvider = "succeed")
public void fail(boolean succeed) {
if (!succeed) {
Assert.fail("nope");
}
}
@DataProvider(name="succeed")
public Object[][] arrayIndex() {
return new Object[][]{ {true}, {false}};
}
{code}
# Configure surefire to rerun failed tests
{code:title=pom.xml example|language=xml}
http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0
cam.test
testng-retry
1.0-SNAPSHOT
org.testng
testng
6.9.9
test
org.apache.maven.plugins
maven-surefire-plugin
2.19.1
1
{code}
- *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
pass.
- *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes:
1" The rerun confuses test runs with different data for retry attempts. Just
one of the data provider tests needs to succeed for the test to be deemed a
success.
{code}mvn test
...
---
T E S T S
---
Running TestRandomFail
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
FAILURE! - in TestRandomFail
fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
java.lang.AssertionError: nope
at TestRandomFail.fail(TestRandomFail.java:14)
Results :
Flaked tests:
TestRandomFail.fail(TestRandomFail)
Run 1: PASS
Run 2: TestRandomFail.fail:14 nope
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 2.251 s
[INFO] Finished at: 2016-02-10T07:39:21-07:00
[INFO] Final Memory: 9M/244M
[INFO]
{code}
was:
Surefire confuses tests run with a data-provider with rerun tests so only one
of the data-provided tests need to pass.
Steps to reproduce:
# Create a testNG test with a data provider that passes with some of the data
and fails with others
{code:title=TestNG example|language=java}
@Test(dataProvider = "succeed")
public void fail(boolean succeed) {
if (!succeed) {
Assert.fail("nope");
}
}
@DataProvider(name="succeed")
public Object[][] arrayIndex() {
return new Object[][]{ {true}, {false}};
}
{code}
# Configure surefire to rerun failed tests
{code:title=pom.xml example|language=xml}
http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0
cam.test
testng-retry
1.0-SNAPSHOT
org.testng
testng
6.9.9
test
org.apache.maven.plugins
maven-surefire-plugin
2.19.1
1
{code}
- *Expected results*: "tests run: 2, Failures: 1..." The tests should not
pass.
- *Actual results*: The different runs using the provider data are confused for
retry attempts. Just one of the data needs to succeed for the tests to be
assumed a success.
{code}mvn test
...
---
T E S T S
---
Running TestRandomFail
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
FAILURE! - in TestRandomFail
fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
java.lang.AssertionError: nope
at TestRandomFail.fail(TestRandomFail.java:14)
Results :
Flaked tests:
TestRandomFail.fail(TestRandomFail)
Run 1: PASS
Run 2: TestRandomFail.fail:14 nope
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cam Morris updated SUREFIRE-1228:
-
Attachment: TestRandomFail.java
This file is placed in src/test/java directory
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.9
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: The different runs using the provider data are confused
> for retry attempts. Just one of the data needs to succeed for the tests to
> be assumed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cam Morris updated SUREFIRE-1228:
-
Attachment: pom.xml
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141878#comment-15141878
]
Cam Morris commented on SUREFIRE-1228:
--
Is this a feature of a not-yet released version of surefire, because the tests
do rerun.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141876#comment-15141876
]
Cam Morris commented on SUREFIRE-1228:
--
It seems to be rerunning the tests despite the fact that testNG is obviously in
use. And the retry passes the tests when it should be failing them. Without
the rerunFailingTestsCount set, the tests fail as expected.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cam Morris updated SUREFIRE-1228:
-
Attachment: (was: testng-retry.iml)
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SUREFIRE-1228) rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141940#comment-15141940
]
Cam Morris commented on SUREFIRE-1228:
--
Interesting! Without the data provider as a parameter to the @Test annotation
it only runs one. rerunFailingTestsCount doesn't have any effect like you said.
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> ---
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
>Affects Versions: 2.19.1
>Reporter: Cam Morris
> Attachments: TestRandomFail.java, pom.xml
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> cam.test
> testng-retry
> 1.0-SNAPSHOT
>
>
> org.testng
> testng
> 6.9.10
> test
>
>
>
>
>
> org.apache.maven.plugins
> maven-surefire-plugin
> 2.19.1
>
> 1
>
>
>
>
>
> {code}
> - *Expected results*: "Tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: "Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
> Flakes: 1" The rerun confuses test runs with different data for retry
> attempts. Just one of the data provider tests needs to succeed for the test
> to be deemed a success.
> {code}mvn test
> ...
> ---
> T E S T S
> ---
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
