[jira] [Commented] (SUREFIRE-1433) rerunFailingTestsCount does not help with crashes
[
https://issues.apache.org/jira/browse/SUREFIRE-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16215494#comment-16215494
]
Jesse Glick commented on SUREFIRE-1433:
---
Obviously the above is not a real test. I was merely demonstrating that
Surefire is not robust against unforeseeable fatal problems in the test JVM,
making it ill suited to long-running test suites. JUnit itself treats any
{{Throwable}} thrown out of a test method as a “failure” which
{{rerunFailingTestsCount}} will process; yet lower-level errors in a forked
test VM are not recoverable.
> rerunFailingTestsCount does not help with crashes
> -
>
> Key: SUREFIRE-1433
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1433
> Project: Maven Surefire
> Issue Type: Bug
> Components: process forking
>Affects Versions: 2.20.1
> Environment: Linux, Java 8u131, Maven 3.5.0
>Reporter: Jesse Glick
> Attachments: demo.zip
>
>
> I have a CI job running lots of very slow tests, and am using
> {{-Dsurefire.rerunFailingTestsCount=2 -Dmaven.test.failure.ignore
> -Dsurefire.timeout=7200 -DreuseForks=false -DforkCount=2}} to run Maven. It
> will typically run for about three hours, collecting a lot of passing tests,
> a few flaking tests with subsequent passes, and a few true failures. But then
> it will abruptly halt with the infamous
> {code:none}
> Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on
> project ...: ExecutionException The forked VM terminated without properly
> saying goodbye. VM crash or System.exit called?
> {code}
> The timing is unpredictable but the build seems to usually fail this way;
> sometimes I get lucky and it runs through all the tests, taking a number of
> hours. Diagnosing the crash is not likely to be straightforward, and frankly
> I do not care much—I just want Surefire to try again. Yet the
> {{rerunFailingTestsCount}} flag does not accomplish this—it only retries
> normal failures.
> You can see this from running the attached project. While you might expect it
> to go through 100 runs of the test, some failing politely and some crashing,
> instead you see something like
> {code:none}
> ...
> --- maven-surefire-plugin:2.20.1:test (default-test) @ demo ---
> ---
> T E S T S
> ---
> Running demo.CrashingTest
> failing
> failing
> failing
> crashing
> Results:
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>
> BUILD FAILURE
>
> Total time: 2.456 s
> Finished at: 2017-10-07T15:11:04-04:00
> Final Memory: 11M/222M
>
> Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test (default-test) on
> project demo: There are test failures.
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SUREFIRE-1433) rerunFailingTestsCount does not help with crashes
[
https://issues.apache.org/jira/browse/SUREFIRE-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196269#comment-16196269
]
Tibor Digana commented on SUREFIRE-1433:
You think this is normal in a test method?
{{Runtime.getRuntime().halt(99);}}
and you expect that Maven process will know if forked vm executed {{kill -9}}?
You think that parent jvm process should get any notification from child
process killer and that this is somehow covered by Java Specification and that
Surefire should take care?
> rerunFailingTestsCount does not help with crashes
> -
>
> Key: SUREFIRE-1433
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1433
> Project: Maven Surefire
> Issue Type: Bug
> Components: process forking
>Affects Versions: 2.20.1
> Environment: Linux, Java 8u131, Maven 3.5.0
>Reporter: Jesse Glick
> Attachments: demo.zip
>
>
> I have a CI job running lots of very slow tests, and am using
> {{-Dsurefire.rerunFailingTestsCount=2 -Dmaven.test.failure.ignore
> -Dsurefire.timeout=7200 -DreuseForks=false -DforkCount=2}} to run Maven. It
> will typically run for about three hours, collecting a lot of passing tests,
> a few flaking tests with subsequent passes, and a few true failures. But then
> it will abruptly halt with the infamous
> {code:none}
> Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on
> project ...: ExecutionException The forked VM terminated without properly
> saying goodbye. VM crash or System.exit called?
> {code}
> The timing is unpredictable but the build seems to usually fail this way;
> sometimes I get lucky and it runs through all the tests, taking a number of
> hours. Diagnosing the crash is not likely to be straightforward, and frankly
> I do not care much—I just want Surefire to try again. Yet the
> {{rerunFailingTestsCount}} flag does not accomplish this—it only retries
> normal failures.
> You can see this from running the attached project. While you might expect it
> to go through 100 runs of the test, some failing politely and some crashing,
> instead you see something like
> {code:none}
> ...
> --- maven-surefire-plugin:2.20.1:test (default-test) @ demo ---
> ---
> T E S T S
> ---
> Running demo.CrashingTest
> failing
> failing
> failing
> crashing
> Results:
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>
> BUILD FAILURE
>
> Total time: 2.456 s
> Finished at: 2017-10-07T15:11:04-04:00
> Final Memory: 11M/222M
>
> Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test (default-test) on
> project demo: There are test failures.
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
