[jira] Updated: (SUREFIRE-611) surefire exit code should explicitly distinguish between crashed and failed unit test(s)
[ http://jira.codehaus.org/browse/SUREFIRE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristian Rosenvold updated SUREFIRE-611: Component/s: Maven Surefire Plugin > surefire exit code should explicitly distinguish between crashed and failed > unit test(s) > > > Key: SUREFIRE-611 > URL: http://jira.codehaus.org/browse/SUREFIRE-611 > Project: Maven Surefire > Issue Type: Improvement > Components: Maven Surefire Plugin >Reporter: Tycho Lamerigts > Attachments: tc-junit-crash.tar.gz > > > - If a unit test crashes, surefire skips all remaining tests, prints a > message "[ERROR] There are test failures. ..." to the console, and does NOT > print a test summary report. > - If one or more unit tests fail, maven completes remaining tests, prints a > message "[ERROR] There are test failures. ..." to the console, and DOES > print a test summary report. > The only distinction between crash and fail is the presence (or absence) of a > summary report. > This is a problem for some tools (like TeamCity), because they typically > don't pick up on the fact that many testcases were not run at all in case of > a crashed unit test. Instead in case of a crashed test they simply report, > for example, 100 tests succeeded, 0 failed. They do not report "50 tests > skipped due to a crash after test #100", and it is quite hard for those tools > to recognize this situation. If surefire would report a crash more > explicitly, other tools can act accordingly. > I have attached a small maven project to reproduce the problem. By commenting > and uncommenting the marked line in file ATest.java, you can reproduce both > fail and crash scenarios. In the crash scenario, Maven will correctly report > a failed build (unless you use the -Dmaven.test.failure.ignore flag, in which > case it even won't do that), but it fails to indicate there was a crash which > caused tests (in this case 3) to be skipped. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (SUREFIRE-611) surefire exit code should explicitly distinguish between crashed and failed unit test(s)
[ http://jira.codehaus.org/browse/SUREFIRE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tycho Lamerigts updated SUREFIRE-611: - Attachment: tc-junit-crash.tar.gz > surefire exit code should explicitly distinguish between crashed and failed > unit test(s) > > > Key: SUREFIRE-611 > URL: http://jira.codehaus.org/browse/SUREFIRE-611 > Project: Maven Surefire > Issue Type: Improvement >Reporter: Tycho Lamerigts > Attachments: tc-junit-crash.tar.gz > > > - If a unit test crashes, surefire skips all remaining tests, prints a > message "[ERROR] There are test failures. ..." to the console, and does NOT > print a test summary report. > - If one or more unit tests fail, maven completes remaining tests, prints a > message "[ERROR] There are test failures. ..." to the console, and DOES > print a test summary report. > The only distinction between crash and fail is the presence (or absence) of a > summary report. > This is a problem for some tools (like TeamCity), because they typically > don't pick up on the fact that many testcases were not run at all in case of > a crashed unit test. Instead in case of a crashed test they simply report, > for example, 100 tests succeeded, 0 failed. They do not report "50 tests > skipped due to a crash after test #100", and it is quite hard for those tools > to recognize this situation. If surefire would report a crash more > explicitly, other tools can act accordingly. > I have attached a small maven project to reproduce the problem. By commenting > and uncommenting the marked line in file ATest.java, you can reproduce both > fail and crash scenarios. In the crash scenario, Maven will correctly report > a failed build (unless you use the -Dmaven.test.failure.ignore flag, in which > case it even won't do that), but it fails to indicate there was a crash which > caused tests (in this case 3) to be skipped. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
