On Mon, 6 Dec 2021 11:12:22 GMT, Aleksey Shipilev <sh...@openjdk.org> wrote:
> This adds the test repeat feature in the build system. This is convenient to > follow-up on intermittently failing tests: the `REPEAT_COUNT > 0` would run > the test multiple times, until we run out of repeats or the tests fail. > > With this sample test: > > > /* > * @test > * @run driver IntermittentTest > */ > public class IntermittentTest { > public static void main(String... args) { > if (Math.random() < 0.05) { > throw new IllegalStateException("Oops"); > } > } > } > > > ...a lucky run looks like this: > > > $ CONF=linux-x86_64-server-fastdebug make run-test > TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=5" > Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug' > Running tests using JTREG control variable 'REPEAT_COUNT=5' > Test selection 'sanity/IntermittentTest.java', will run: > * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java > > Running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java' > > Repeating Jtreg run: 1 out of 5 > Passed: sanity/IntermittentTest.java > Test results: passed: 1 > Report written to ... > Results written to ... > > Repeating Jtreg run: 2 out of 5 > Passed: sanity/IntermittentTest.java > Test results: passed: 1 > Report written to ... > Results written to ... > > Repeating Jtreg run: 3 out of 5 > Passed: sanity/IntermittentTest.java > Test results: passed: 1 > Report written to ... > Results written to ... > > Repeating Jtreg run: 4 out of 5 > Passed: sanity/IntermittentTest.java > Test results: passed: 1 > Report written to ... > Results written to ... > > Repeating Jtreg run: 5 out of 5 > Passed: sanity/IntermittentTest.java > Test results: passed: 1 > Report written to ... > Results written to ... > Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java' > Test report is stored in ... > > ============================== > Test summary > ============================== > TEST TOTAL PASS FAIL ERROR > > jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java > 1 1 0 0 > > ============================== > TEST SUCCESS > > > ...and unlucky run looks like this: > > > $ CONF=linux-x86_64-server-fastdebug make run-test > TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=50" > Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug' > Running tests using JTREG control variable 'REPEAT_COUNT=50' > Test selection 'sanity/IntermittentTest.java', will run: > * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java > ... > Repeating Jtreg run: 9 out of 50 > Passed: sanity/IntermittentTest.java > Test results: passed: 1 > Report written to ... > Results written to ... > > Repeating Jtreg run: 10 out of 50 > -------------------------------------------------- > TEST: sanity/IntermittentTest.java > TEST JDK: ... > > ... > > STDERR: > java.lang.IllegalStateException: Oops > at IntermittentTest.main(IntermittentTest.java:8) > at > java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) > at java.base/java.lang.reflect.Method.invoke(Method.java:577) > at > com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312) > at java.base/java.lang.Thread.run(Thread.java:833) > > JavaTest Message: Test threw exception: java.lang.IllegalStateException > JavaTest Message: shutting down test > > > TEST RESULT: Failed. Execution failed: `main' threw exception: > java.lang.IllegalStateException: Oops > -------------------------------------------------- > Test results: failed: 1 > Report written to ... > Results written to ... > Error: Some tests failed or other problems occurred. > > Failures detected, no more repeats. > Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java' > Test report is stored in ... > > ============================== > Test summary > ============================== > TEST TOTAL PASS FAIL ERROR > > jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java >>> 1 0 1 0 << > ============================== > TEST FAILURE Looks good. Thank you for fixing this! ------------- Marked as reviewed by ihse (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/6720