Here's my first concrete slow machine timed out test ...
jdk/test/java/lang/concurrent/forkjoin/Integrate.java
I had been looking at tests that had a declared "timeout=xxx",
but today I just started running the java/util/concurrent
tests at a variety of clock speeds using ejdk1.7.0 and
found a test that passes when running at 600MHz and
timed out at 300Mhz. The test passes at 300 MHz if I
include "-timeout:2" on the jtreg command line.
At 600Mhz the test runs for 84 seconds (under the default
120 second timeout). At 300Mhz the test runs for 168
seconds.
Since this test does not do an internal wait or delay operation
passing in a timeout factor would not help in this case.
In general it seems that tests that declare a timeout less than 120
seconds are indicating that an early termination for the test is acceptable.
Tests declaring a longer than 120 second timeout recognize that additional
processing time may be required.
I'll try a longer overnight run at 300MHz to see if I can catch some
other tests that are close to the 120 second threshold.
...
On 11/15/11 08:33 PM, David Holmes wrote:
Hi Gary,
On 16/11/2011 6:14 AM, Gary Adams wrote:
I've been scanning a number of the slow machine test
bugs that are reported and wanted to check to see if
anyone has looked into time dependencies in the regression
tests previously. From what I've been able to learn so far
individual bugs can use the "timeout" parameter to indicate to
the test harness an expected time to run.
The test harness has command line arguments where it can
filter out tests that take too long (timelimit) or can apply a
multiplier to
to the timeout when conditions are known to slow down the process
(timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp
I see that there are some wrappers that can be applied around running
a particular test to allow processing before main(). Could this mechanism
be exploited so the harness command line options could be made known
to the time dependent tests as command line arguments or as system
properties?
My thought is the current timeout granularity is too large and only applies
to the full test execution. If a test knew that a timeoutFactor was to
be applied,
it could internally adjust the time dependent delays appropriately. e.g.
not every sleep(), await(), join() with timeouts would need the
timeoutFactor applied.
I don't quite get what you mean about the timeouts applied to sleeps, awaits
etc. Depending on the test some of these are delays (eg sleep is usually used
this way) because it may not be feasible (or even possible) to coordinate the
threads directly; while others (await, wait etc) are actual timeouts - if they
expire it is an error because something appears to have gone wrong somewhere
(of course this can be wrong because the timeout was too short for a given
situation).
As many of these tests have evolved along with the testing infrastructure it
isn't always very clear who has responsibility for programming defensive
timeouts. And many tests are designed to be run stand-alone or under a test
harness, where exceptions due to timeouts are preferable to hangs.
Further, while we can add a scale factor for known retarding factors - like
Xcomp - there's no general way to assess the target machine capability (#
cores, speed) and load, as it may impact a given test. And historically there
has been a lack of resources to investigate and solve these issues.
Cheers,
David
Before any test could be updated the information would need to be available
from the test context.
Any feedback/pointers appreciated!
See
timeoutFactorArg
jtreg/src/share/classes/com/sun/javatest/regtest/Main.java
runOtherJVM()
jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java
maxTimeoutValue
jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
