On Fri, 27 Oct 2023 06:55:43 GMT, Alan Bateman <al...@openjdk.org> wrote:
> The jtreg generated timeStats.txt is useful to see the distribution. For the > dev guide, I think it's good to say that tests that run in tier1 must run > quickly but I don't think we can suggest 10s as a limit. It might be okay to > say that 90% of tests in tier1 are expected to run in <10s and to think > carefully when adding tests that take more than XX seconds, I don't know what > XX, maybe 30, maybe 60. Right now there are a small number (in percentage > terms) of tests running in tier1 with execution times more than a minute but > some of these are with debug builds so they will be slower. So I think for > the dev guide then some guidance is good. I ran the various tier 1 tests on my local system (Linux VirtualBox hosed on windows), which is not especially fast, and here are the breakdowns: hotspot tier 1: 2,386 tests, ~90% 10 seconds or less Mean 4.87s langtools tier 1: 4,429 tests, 95+% 10 seconds or less Mean 2.96s jdk tier 1: 2,404 tests, 91+% 10 seconds or less Mean 4.46s If we expand the cutoff to 15 seconds, on this configuration the percentage of tests within that bound go up to ~96%, 97%, ~95%. On a faster config, I wouldn't be surprised if these fell within the 10 second bound instead. So in terms of a sentence or two of guidance, I think "aim for 10 seconds or less almost all of the time for a tier 1 test" is reasonable in this context. Each of the three sets of tests did have few outliers taking several minutes; those should probably be looked to to be refactored or split up. ------------- PR Comment: https://git.openjdk.org/jdk/pull/16384#issuecomment-1783920559
