Re: Passing time factor to tests run under jtreg
On 17/11/2011 20:28, Gary Adams wrote: : In general it seems that tests that declare a timeout less than 120 seconds are indicating that an early termination for the test is acceptable. Tests declaring a longer than 120 second timeout recognize that additional processing time may be required. I can't think of any need for tests to specify a timeout less then the default. When fixing a deadlock or some such bug then you will typically run the test with a JDK build that doesn't have the fix and a JDK build with the fix. When testing with the former then it's nice to have the test timeout quickly which is why some tests do have a short timeout. But since the bugs are long fixed then these tests should not hang or deadlock and so the default timeout should be fine. Clearly slower/stress tests that have the potential to excess the default timeout need to override the default timeout. In any case, for 300Mhz then I would definitely run with -timeoutFactor:2 or greater. A useful file to look at is JTreport/text/timeStats.txt as it shows the distribution of the test execution times. -Alan
Re: Passing time factor to tests run under jtreg
Gary, On 18/11/2011 6:28 AM, Gary Adams wrote: Here's my first concrete slow machine timed out test ... jdk/test/java/lang/concurrent/forkjoin/Integrate.java I had been looking at tests that had a declared timeout=xxx, but today I just started running the java/util/concurrent tests at a variety of clock speeds using ejdk1.7.0 and found a test that passes when running at 600MHz and timed out at 300Mhz. The test passes at 300 MHz if I include -timeout:2 on the jtreg command line. I think I have been misunderstanding the point you've been trying to make here. I'm not sure there is a simple relationship here with the use of internal delays/timeouts in a test. delays (wait long enough til XXX should have happened) would seem to need to be scaled under the same considerations as used for -timeout. Internal timeouts (give up after XXX time units because something seems to have gone wrong) on the other hand are typically much coarser/larger and so already accommodate a range of -timeout values implicitly. The scaling factor need to come from the environment launching the test, but the tests need to be modified to use it. At 600Mhz the test runs for 84 seconds (under the default 120 second timeout). At 300Mhz the test runs for 168 seconds. Since this test does not do an internal wait or delay operation passing in a timeout factor would not help in this case. In general it seems that tests that declare a timeout less than 120 seconds are indicating that an early termination for the test is acceptable. I agree with Alan that it doesn't make sense to specify timeouts less than the default. Tests declaring a longer than 120 second timeout recognize that additional processing time may be required. Most likely the test failed somewhere sometime and bumping the timeout fixed it. wash-rinse-repeat Cheers, David I'll try a longer overnight run at 300MHz to see if I can catch some other tests that are close to the 120 second threshold. ... On 11/15/11 08:33 PM, David Holmes wrote: Hi Gary, On 16/11/2011 6:14 AM, Gary Adams wrote: I've been scanning a number of the slow machine test bugs that are reported and wanted to check to see if anyone has looked into time dependencies in the regression tests previously. From what I've been able to learn so far individual bugs can use the timeout parameter to indicate to the test harness an expected time to run. The test harness has command line arguments where it can filter out tests that take too long (timelimit) or can apply a multiplier to to the timeout when conditions are known to slow down the process (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp I see that there are some wrappers that can be applied around running a particular test to allow processing before main(). Could this mechanism be exploited so the harness command line options could be made known to the time dependent tests as command line arguments or as system properties? My thought is the current timeout granularity is too large and only applies to the full test execution. If a test knew that a timeoutFactor was to be applied, it could internally adjust the time dependent delays appropriately. e.g. not every sleep(), await(), join() with timeouts would need the timeoutFactor applied. I don't quite get what you mean about the timeouts applied to sleeps, awaits etc. Depending on the test some of these are delays (eg sleep is usually used this way) because it may not be feasible (or even possible) to coordinate the threads directly; while others (await, wait etc) are actual timeouts - if they expire it is an error because something appears to have gone wrong somewhere (of course this can be wrong because the timeout was too short for a given situation). As many of these tests have evolved along with the testing infrastructure it isn't always very clear who has responsibility for programming defensive timeouts. And many tests are designed to be run stand-alone or under a test harness, where exceptions due to timeouts are preferable to hangs. Further, while we can add a scale factor for known retarding factors - like Xcomp - there's no general way to assess the target machine capability (# cores, speed) and load, as it may impact a given test. And historically there has been a lack of resources to investigate and solve these issues. Cheers, David Before any test could be updated the information would need to be available from the test context. Any feedback/pointers appreciated! See timeoutFactorArg jtreg/src/share/classes/com/sun/javatest/regtest/Main.java runOtherJVM() jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java maxTimeoutValue jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
Re: Passing time factor to tests run under jtreg
On 11/15/11 8:33 PM, David Holmes wrote: Hi Gary, On 16/11/2011 6:14 AM, Gary Adams wrote: I've been scanning a number of the slow machine test bugs that are reported and wanted to check to see if anyone has looked into time dependencies in the regression tests previously. From what I've been able to learn so far individual bugs can use the timeout parameter to indicate to the test harness an expected time to run. The test harness has command line arguments where it can filter out tests that take too long (timelimit) or can apply a multiplier to to the timeout when conditions are known to slow down the process (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp I see that there are some wrappers that can be applied around running a particular test to allow processing before main(). Could this mechanism be exploited so the harness command line options could be made known to the time dependent tests as command line arguments or as system properties? My thought is the current timeout granularity is too large and only applies to the full test execution. If a test knew that a timeoutFactor was to be applied, it could internally adjust the time dependent delays appropriately. e.g. not every sleep(), await(), join() with timeouts would need the timeoutFactor applied. I don't quite get what you mean about the timeouts applied to sleeps, awaits etc. Depending on the test some of these are delays (eg sleep is usually used this way) because it may not be feasible (or even possible) to coordinate the threads directly; while others (await, wait etc) are actual timeouts - if they expire it is an error because something appears to have gone wrong somewhere (of course this can be wrong because the timeout was too short for a given situation). The timeout being too short for a slow machine is the specific condition I'd like to address. As many of these tests have evolved along with the testing infrastructure it isn't always very clear who has responsibility for programming defensive timeouts. And many tests are designed to be run stand-alone or under a test harness, where exceptions due to timeouts are preferable to hangs. Some of the bug reports state an intermittent failure. Could be failing due to busy machine, slow machine or aggressive command line options. If a test has an advertized timeout, then it needs to be run stand-alone. If aggressive command line options are used the timefactor multiplier can compensate for the whole test timeout. I believe we can identify slow machines and apply the same timefactor multiplier. Further, while we can add a scale factor for known retarding factors - like Xcomp - there's no general way to assess the target machine capability (# cores, speed) and load, as it may impact a given test. And historically there has been a lack of resources to investigate and solve these issues. I'm not sure why test machine characteristics are not known at test time. Obviously some agent setup has been done to identify chip architecture and operating system. The cpu speed is just another attribute of the test machine. I understand the lack of resources to address this area in the past, but resources are being applied to ensure jdk8 is appropriate for embedded deployments. This seems like another low hanging fruit. e.g. not technically difficult, not particularly controversial Cheers, David Before any test could be updated the information would need to be available from the test context. Any feedback/pointers appreciated! See timeoutFactorArg jtreg/src/share/classes/com/sun/javatest/regtest/Main.java runOtherJVM() jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java maxTimeoutValue jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
Re: Passing time factor to tests run under jtreg
Gary - this might be something to bring up on the jtreg-use list. Ideally the tests wouldn't have any hardcoded timeouts but sometimes there isn't any other choice. -Alan On 15/11/2011 20:14, Gary Adams wrote: I've been scanning a number of the slow machine test bugs that are reported and wanted to check to see if anyone has looked into time dependencies in the regression tests previously. From what I've been able to learn so far individual bugs can use the timeout parameter to indicate to the test harness an expected time to run. The test harness has command line arguments where it can filter out tests that take too long (timelimit) or can apply a multiplier to to the timeout when conditions are known to slow down the process (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp I see that there are some wrappers that can be applied around running a particular test to allow processing before main(). Could this mechanism be exploited so the harness command line options could be made known to the time dependent tests as command line arguments or as system properties? My thought is the current timeout granularity is too large and only applies to the full test execution. If a test knew that a timeoutFactor was to be applied, it could internally adjust the time dependent delays appropriately. e.g. not every sleep(), await(), join() with timeouts would need the timeoutFactor applied. Before any test could be updated the information would need to be available from the test context. Any feedback/pointers appreciated! See timeoutFactorArg jtreg/src/share/classes/com/sun/javatest/regtest/Main.java runOtherJVM() jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java maxTimeoutValue jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
Re: Passing time factor to tests run under jtreg
Hi Gary, On 16/11/2011 6:14 AM, Gary Adams wrote: I've been scanning a number of the slow machine test bugs that are reported and wanted to check to see if anyone has looked into time dependencies in the regression tests previously. From what I've been able to learn so far individual bugs can use the timeout parameter to indicate to the test harness an expected time to run. The test harness has command line arguments where it can filter out tests that take too long (timelimit) or can apply a multiplier to to the timeout when conditions are known to slow down the process (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp I see that there are some wrappers that can be applied around running a particular test to allow processing before main(). Could this mechanism be exploited so the harness command line options could be made known to the time dependent tests as command line arguments or as system properties? My thought is the current timeout granularity is too large and only applies to the full test execution. If a test knew that a timeoutFactor was to be applied, it could internally adjust the time dependent delays appropriately. e.g. not every sleep(), await(), join() with timeouts would need the timeoutFactor applied. I don't quite get what you mean about the timeouts applied to sleeps, awaits etc. Depending on the test some of these are delays (eg sleep is usually used this way) because it may not be feasible (or even possible) to coordinate the threads directly; while others (await, wait etc) are actual timeouts - if they expire it is an error because something appears to have gone wrong somewhere (of course this can be wrong because the timeout was too short for a given situation). As many of these tests have evolved along with the testing infrastructure it isn't always very clear who has responsibility for programming defensive timeouts. And many tests are designed to be run stand-alone or under a test harness, where exceptions due to timeouts are preferable to hangs. Further, while we can add a scale factor for known retarding factors - like Xcomp - there's no general way to assess the target machine capability (# cores, speed) and load, as it may impact a given test. And historically there has been a lack of resources to investigate and solve these issues. Cheers, David Before any test could be updated the information would need to be available from the test context. Any feedback/pointers appreciated! See timeoutFactorArg jtreg/src/share/classes/com/sun/javatest/regtest/Main.java runOtherJVM() jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java maxTimeoutValue jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
