[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051337#comment-17051337
]
Anu commented on SUREFIRE-1751:
---
I was out sick for two weeks. Will try this again today and keep you posted.
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: TEST-com.test.Flakes.xml, Test.zip,
> image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038582#comment-17038582
]
Anu commented on SUREFIRE-1751:
---
Thanks [~tibordigana]. I tried creating a test method in the class like below
and ran the test with [^TEST-com.test.Flakes.xml] . The test passed (showed
correct number of tests, failures, flakes etc)
{code:java}
@Test
public void successfulSurefireTestReportWithFlakes() throws Exception {
TestSuiteXmlParser parser = new TestSuiteXmlParser( consoleLogger );
File surefireReport = new File(
"src/test/resources/junit-pathWithÜmlaut/TEST-com.test.Flakes.xml" );
assumeTrue( surefireReport.isFile() );
Collection suites = parser.parse(
surefireReport.getCanonicalPath() );
assertNotNull( suites );
assertEquals( 1, suites.size() );
ReportTestSuite suite = suites.iterator().next();
assertThat( suite.getNumberOfTests(), is( 1 ) );
assertEquals( 1, suite.getNumberOfTests() );
assertEquals( 1, suite.getNumberOfFlakes() );
assertEquals( 0, suite.getNumberOfFailures() );
assertEquals( 0, suite.getNumberOfErrors() );
assertEquals( 0, suite.getNumberOfSkipped() );
assertThat( suite.getFullClassName(), is( "com.test.Flakes" ) );
assertThat( suite.getPackageName(), is( "com.test" ) );
assertThat( suite.getName(), is( "Flakes" ) );
ReportTestCase test = suite.getTestCases().iterator().next();
assertTrue( test.isSuccessful() );
assertNull( test.getFailureDetail() );
assertNull( test.getFailureErrorLine() );
assertNull( test.getFailureType() );
assertThat( test.getFullClassName(), is( "com.test.Flakes" ) );
assertThat( test.getClassName(), is( "Flakes" ) );
assertThat( test.getName(), is( "run" ) );
assertThat( test.getFullName(), is( "com.test.Flakes.run" ) );
} {code}
I am new to the github world, so I am not sure how to commit this test to the
repository.
can you please help on how to proceed?
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: TEST-com.test.Flakes.xml, Test.zip,
> image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu updated SUREFIRE-1751:
--
Attachment: TEST-com.test.Flakes.xml
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: TEST-com.test.Flakes.xml, Test.zip,
> image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037195#comment-17037195
]
Anu commented on SUREFIRE-1751:
---
I forgot to take out surefire-report-parse from the pom before sending it you.
And sorry. I misunderstood.This is my first time creating a ticket with third
party. I added the code to github like below. Is that what you were looking for?
[https://github.com/anujose-c/AnuTest]
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: Test.zip, image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036573#comment-17036573
]
Anu commented on SUREFIRE-1751:
---
Thanks [~tibordigana] for the response. I have attached Test.zip with the
ticket. Can you check if that helps?
The command I used to run the tests was
{code:java}
mvn surefire-report:report -Dsurefire.rerunFailingTestsCount=2
-Dtest=Flakes,Pass{code}
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: Test.zip, image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu updated SUREFIRE-1751:
--
Attachment: Test.zip
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: Test.zip, image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (SUREFIRE-1751) Surefire report shows flaky tests as failures
Anu created SUREFIRE-1751:
-
Summary: Surefire report shows flaky tests as failures
Key: SUREFIRE-1751
URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
Project: Maven Surefire
Issue Type: Bug
Affects Versions: 3.0.0-M4
Reporter: Anu
Attachments: image-2020-02-13-11-41-03-906.png
We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
testing. Now we are trying to upgrade junit to junit5 and saw that the
r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
tests are now being retried. But the issue is flaky tests show up in the report
as Failures.
{color}
{color:#6a8759}ie. the log shows
{color}
[WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
But the actual surefire report shows like below
!image-2020-02-13-11-41-03-906.png!
This is blocking our junit5 upgrade task. Can you please provide a fix for this
at the earliest? Please let me know if you need anything else from me to help
with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
