Re: [PATCH] (Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long))
Hi, I have noticed that in gcc-13, test05 (in the 94749.cc testcase) is still enabled for simulators, and I have noticed that because of test05, the 27_io/basic_istream/ignore/char/94749.cc execution test is not terminating on our simulator for armv8.1-m.main+mve, even after 3 hours. The execution test was passing before this commit : https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=e30211cb0b3a2b88959e9bc40626a17461de52de Could you please provide some hints or ideas as to what might be causing this regression? I imagine that the issue could be with the simulator or with code-gen. However, could this also highlight a different issue in test05? Is this testing a commonly used feature or area of the compiler? And would it be worth re-including it for simulators? Kind regards, Vasee From: Libstdc++ on behalf of Jonathan Wakely via Libstdc++ Sent: 10 June 2023 08:12 To: Hans-Peter Nilsson Cc: Jonathan Wakely; libstdc++; gcc-patches Subject: Re: [PATCH] (Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long)) On Sat, 10 Jun 2023, 06:18 Hans-Peter Nilsson via Libstdc++, < libstd...@gcc.gnu.org> wrote: > Thank you for your consideration. (Or is that phrase only used > negatively?) > > > From: Jonathan Wakely > > Date: Fri, 9 Jun 2023 21:40:15 +0100 > > > test01, test02, test03 and test04 should run almost instantly. On my > system > > they take about 5 microseconds each. So I don't think splitting those up > > will help. > > Right. > > > I thought it would help to avoid re-allocating the buffer and zeroing it > > again. If we reuse the same buffer, then we just have to loop until we > > overflow the 32-bit counter. That would make the whole test run much > > faster, which would reduce the total time for a testsuite run. Splitting > > the file up into smaller files would not decrease the total time, only > > decrease the time for that single test so it doesn't time out. > > > > I've attached a patch that does that. I makes very little difference for > > me, probably because allocating zero-filled pages isn't actually > expensive > > on linux. Maybe it will make a differene for your simulator though? > > Nope, just some five seconds down (from about 10min 21s). > Bah, worth a try :) > > You could also try reducing the size of the buffer: > > +#ifdef SIMULATOR_TEST > > + static const streamsize bufsz = 16 << limits::digits10; > > +#else > > static const streamsize bufsz = 2048 << limits::digits10; > > +#endif > > Was that supposed to be with or without the patch? Anyway; > both: 606s. Only smaller bufsz: 614s. (All numbers subject > to usual system jitter.) > > > test06 is the really slow part, that takes 10+ seconds for me. But that > > entire function should already be skipped for simulators. > > Yep, we may have been here before... I certainly get a > deja-vu feeling here, but visiting old email conversations > of ours, it seems I easily conflate several similar ones. > I see that here, test06 was always #ifndef SIMULATOR_TEST. > > > We can probably skip test05 for simulators too, none of the code it tests > > is platform-specific, so as long as it's being tested on x86 we don't > > really need to test it on cris-elf too. > > Thanks. Let's do that, then. The similar s/wchar_t/char/ > test clocks in at "only" 3m30s, but I suggest treating it > the same, if nothing else than for symmetry. > > Ok as below? > OK for trunk, and all branches you care about. > -- >8 -- > Subject: [PATCH] testsuite: Cut down 27_io/basic_istream/.../94749.cc for > simulators > > The test wchar_t/94749.cc can take about 10 minutes on some > simulator/host combinations with char/94749.cc at a third of > that time. The cause is test05 which is quite heavy and > includes wrapping a 32-bit counter. Run it only for native > setups. > > * testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc (main) > [! SIMULATOR_TEST]: Also exclude running test05. > * testsuite/27_io/basic_istream/ignore/char/94749.cc: Ditto. > --- > libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc | 2 +- > .../testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git > a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc > b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc > index 6416863983b7..9160995c05ec 100644 > --- a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc > +++ b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc > @@ -221,8 +221,8 @@ main() >
Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long)
On Jun 12, 2023, at 1:35 AM, Bernhard Reutner-Fischer wrote: > > On Sat, 10 Jun 2023 11:29:36 -0700 > Mike Stump wrote: > >> On Jun 9, 2023, at 2:47 PM, Bernhard Reutner-Fischer >> wrote: > >>>But well. Either way, what >>> should we do about remote env, if there is one? If the target >>> supports it, send it and skip otherwise? > >> So, to focus a >> conversation, which target, which host, canadian? Which part of the >> environment? What part is missing you want to fix? Want to unify >> between targets... and so on. >> > > The most recent target where this came up again was GCN i think. > See the last block in > https://inbox.sourceware.org/gcc-patches/20230508195217.4897009f@nbbrfq/ > and Thomas' reference therein to Tobias' > https://inbox.sourceware.org/gcc-patches/018bcdeb-b3bb-1859-cd0b-a8a92e26d...@codesourcery.com/ > > thoughts? I kinds like the remote_setenv approach, but the low level details when one gets in there and wires up all they need to make it work are important. If someone wants to tilt at the problem, I'm inclined to let them find the incantation they like and approve it. I'd rather approve a patch that takes us a step closer to perfection then go another 10 years. For quoting. I like quite using ' if there is no ' in the string. For each ' in the string, replace it with '\'', inelegant, but easier to reason about and don't have to worry about as many meta characters if you try and go the "" route. This assumes something like sh is taking in commands. This isn't always the case. One can canadian into a windows box or some other weird thing so the mechanism used has to be next to the thing that knows what it is talking to. Just because someone contributes code for talking to sh, doesn't mean they have to bother with windows or other weird stuff. The next person to come along can tilt at that.
Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long)
On Sat, 10 Jun 2023 11:29:36 -0700 Mike Stump wrote: > On Jun 9, 2023, at 2:47 PM, Bernhard Reutner-Fischer > wrote: > > But well. Either way, what > > should we do about remote env, if there is one? If the target > > supports it, send it and skip otherwise? > So, to focus a > conversation, which target, which host, canadian? Which part of the > environment? What part is missing you want to fix? Want to unify > between targets... and so on. > The most recent target where this came up again was GCN i think. See the last block in https://inbox.sourceware.org/gcc-patches/20230508195217.4897009f@nbbrfq/ and Thomas' reference therein to Tobias' https://inbox.sourceware.org/gcc-patches/018bcdeb-b3bb-1859-cd0b-a8a92e26d...@codesourcery.com/ thoughts? thanks,
Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long)
On Jun 9, 2023, at 2:47 PM, Bernhard Reutner-Fischer wrote: > > On 9 June 2023 19:18:45 CEST, Mike Stump via Gcc-patches > wrote: >> simulation ports. Maybe a 20-100x speedup? If you want to go this way I'd >> say do it in python at the bottom as it would be nice to switch over to >> python in the next 5-20 years and away from tcl. > > Yes, i guess we have all pondered this for way long enough, but it is a lot > of work still. > > The nice side effect would be that we see such hogs easier and earlier, at > least more comfortable. But well. Either way, what should we do about remote > env, if there is one? If the target supports it, send it and skip otherwise? Testing is a large barrel of monkeys with a ton of small points, each of which is critical in some one. It is hard to talk about generalities when those details are very specific. So, to focus a conversation, which target, which host, canadian? Which part of the environment? What part is missing you want to fix? Want to unify between targets... and so on.
Re: [PATCH] (Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long))
On Sat, 10 Jun 2023, 06:18 Hans-Peter Nilsson via Libstdc++, < libstd...@gcc.gnu.org> wrote: > Thank you for your consideration. (Or is that phrase only used > negatively?) > > > From: Jonathan Wakely > > Date: Fri, 9 Jun 2023 21:40:15 +0100 > > > test01, test02, test03 and test04 should run almost instantly. On my > system > > they take about 5 microseconds each. So I don't think splitting those up > > will help. > > Right. > > > I thought it would help to avoid re-allocating the buffer and zeroing it > > again. If we reuse the same buffer, then we just have to loop until we > > overflow the 32-bit counter. That would make the whole test run much > > faster, which would reduce the total time for a testsuite run. Splitting > > the file up into smaller files would not decrease the total time, only > > decrease the time for that single test so it doesn't time out. > > > > I've attached a patch that does that. I makes very little difference for > > me, probably because allocating zero-filled pages isn't actually > expensive > > on linux. Maybe it will make a differene for your simulator though? > > Nope, just some five seconds down (from about 10min 21s). > Bah, worth a try :) > > You could also try reducing the size of the buffer: > > +#ifdef SIMULATOR_TEST > > + static const streamsize bufsz = 16 << limits::digits10; > > +#else > > static const streamsize bufsz = 2048 << limits::digits10; > > +#endif > > Was that supposed to be with or without the patch? Anyway; > both: 606s. Only smaller bufsz: 614s. (All numbers subject > to usual system jitter.) > > > test06 is the really slow part, that takes 10+ seconds for me. But that > > entire function should already be skipped for simulators. > > Yep, we may have been here before... I certainly get a > deja-vu feeling here, but visiting old email conversations > of ours, it seems I easily conflate several similar ones. > I see that here, test06 was always #ifndef SIMULATOR_TEST. > > > We can probably skip test05 for simulators too, none of the code it tests > > is platform-specific, so as long as it's being tested on x86 we don't > > really need to test it on cris-elf too. > > Thanks. Let's do that, then. The similar s/wchar_t/char/ > test clocks in at "only" 3m30s, but I suggest treating it > the same, if nothing else than for symmetry. > > Ok as below? > OK for trunk, and all branches you care about. > -- >8 -- > Subject: [PATCH] testsuite: Cut down 27_io/basic_istream/.../94749.cc for > simulators > > The test wchar_t/94749.cc can take about 10 minutes on some > simulator/host combinations with char/94749.cc at a third of > that time. The cause is test05 which is quite heavy and > includes wrapping a 32-bit counter. Run it only for native > setups. > > * testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc (main) > [! SIMULATOR_TEST]: Also exclude running test05. > * testsuite/27_io/basic_istream/ignore/char/94749.cc: Ditto. > --- > libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc | 2 +- > .../testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git > a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc > b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc > index 6416863983b7..9160995c05ec 100644 > --- a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc > +++ b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc > @@ -221,8 +221,8 @@ main() >test02(); >test03(); >test04(); > - test05(); > #ifndef SIMULATOR_TEST > + test05(); >test06(); > #endif > } > diff --git > a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc > b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc > index 65e0a326c109..a5b9eb71a389 100644 > --- a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc > +++ b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc > @@ -221,8 +221,8 @@ main() >test02(); >test03(); >test04(); > - test05(); > #ifndef SIMULATOR_TEST > + test05(); >test06(); > #endif > } > -- > 2.30.2 > >
[PATCH] (Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long))
Thank you for your consideration. (Or is that phrase only used negatively?) > From: Jonathan Wakely > Date: Fri, 9 Jun 2023 21:40:15 +0100 > test01, test02, test03 and test04 should run almost instantly. On my system > they take about 5 microseconds each. So I don't think splitting those up > will help. Right. > I thought it would help to avoid re-allocating the buffer and zeroing it > again. If we reuse the same buffer, then we just have to loop until we > overflow the 32-bit counter. That would make the whole test run much > faster, which would reduce the total time for a testsuite run. Splitting > the file up into smaller files would not decrease the total time, only > decrease the time for that single test so it doesn't time out. > > I've attached a patch that does that. I makes very little difference for > me, probably because allocating zero-filled pages isn't actually expensive > on linux. Maybe it will make a differene for your simulator though? Nope, just some five seconds down (from about 10min 21s). > You could also try reducing the size of the buffer: > +#ifdef SIMULATOR_TEST > + static const streamsize bufsz = 16 << limits::digits10; > +#else > static const streamsize bufsz = 2048 << limits::digits10; > +#endif Was that supposed to be with or without the patch? Anyway; both: 606s. Only smaller bufsz: 614s. (All numbers subject to usual system jitter.) > test06 is the really slow part, that takes 10+ seconds for me. But that > entire function should already be skipped for simulators. Yep, we may have been here before... I certainly get a deja-vu feeling here, but visiting old email conversations of ours, it seems I easily conflate several similar ones. I see that here, test06 was always #ifndef SIMULATOR_TEST. > We can probably skip test05 for simulators too, none of the code it tests > is platform-specific, so as long as it's being tested on x86 we don't > really need to test it on cris-elf too. Thanks. Let's do that, then. The similar s/wchar_t/char/ test clocks in at "only" 3m30s, but I suggest treating it the same, if nothing else than for symmetry. Ok as below? -- >8 -- Subject: [PATCH] testsuite: Cut down 27_io/basic_istream/.../94749.cc for simulators The test wchar_t/94749.cc can take about 10 minutes on some simulator/host combinations with char/94749.cc at a third of that time. The cause is test05 which is quite heavy and includes wrapping a 32-bit counter. Run it only for native setups. * testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc (main) [! SIMULATOR_TEST]: Also exclude running test05. * testsuite/27_io/basic_istream/ignore/char/94749.cc: Ditto. --- libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc | 2 +- .../testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc index 6416863983b7..9160995c05ec 100644 --- a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc +++ b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/char/94749.cc @@ -221,8 +221,8 @@ main() test02(); test03(); test04(); - test05(); #ifndef SIMULATOR_TEST + test05(); test06(); #endif } diff --git a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc index 65e0a326c109..a5b9eb71a389 100644 --- a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc +++ b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc @@ -221,8 +221,8 @@ main() test02(); test03(); test04(); - test05(); #ifndef SIMULATOR_TEST + test05(); test06(); #endif } -- 2.30.2
Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long)
On 9 June 2023 19:18:45 CEST, Mike Stump via Gcc-patches wrote: > simulation ports. Maybe a 20-100x speedup? If you want to go this way I'd > say do it in python at the bottom as it would be nice to switch over to > python in the next 5-20 years and away from tcl. Yes, i guess we have all pondered this for way long enough, but it is a lot of work still. The nice side effect would be that we see such hogs easier and earlier, at least more comfortable. But well. Either way, what should we do about remote env, if there is one? If the target supports it, send it and skip otherwise? thanks,
Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long)
On Fri, 9 Jun 2023 at 17:20, Hans-Peter Nilsson wrote: > Hi! > > The test 27_io/basic_istream/ignore/wchar_t/94749.cc takes > about 10 minutes to run for cris-elf in the "gdb simulator" > here on my arguably way-past-retirement machine (and it > looks like it gained a minute with LRA). I've seen it > timing out every now and then on busy days with load > > `nproc`. Usually it happens some time after I've forgot > about why. :) > > It has had some performance surgery before (pruning for > simulators, doubling timeout for ilp32). I'd probably just > try cutting along the function boundaries and keep those > parts separate that have >1 min execution time. > test01, test02, test03 and test04 should run almost instantly. On my system they take about 5 microseconds each. So I don't think splitting those up will help. test05 extracts INT_MAX characters from a stream, which is a LOT of work. It doesn't actually read those from a file, the "stream" is a custom streambuf that contains a buffer of millions of wchar_t and "reading" from the stream just increments a counter into that buffer. But we do have to allocate memory for that buffer and then zero-init that buffer. That's a lot of cycles. Then once we've done that, we need to keep looping until we overflow a 32-bit counter (we don't increment by 1 every loop, so it overflows pretty quickly). Then we do it again and again and again! Each time takes about half a second for me. I thought it would help to avoid re-allocating the buffer and zeroing it again. If we reuse the same buffer, then we just have to loop until we overflow the 32-bit counter. That would make the whole test run much faster, which would reduce the total time for a testsuite run. Splitting the file up into smaller files would not decrease the total time, only decrease the time for that single test so it doesn't time out. I've attached a patch that does that. I makes very little difference for me, probably because allocating zero-filled pages isn't actually expensive on linux. Maybe it will make a differene for your simulator though? You could also try reducing the size of the buffer: +#ifdef SIMULATOR_TEST + static const streamsize bufsz = 16 << limits::digits10; +#else static const streamsize bufsz = 2048 << limits::digits10; +#endif test06 is the really slow part, that takes 10+ seconds for me. But that entire function should already be skipped for simulators. We can probably skip test05 for simulators too, none of the code it tests is platform-specific, so as long as it's being tested on x86 we don't really need to test it on cris-elf too. diff --git a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc index 65e0a326c10..040e94aa4d6 100644 --- a/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc +++ b/libstdc++-v3/testsuite/27_io/basic_istream/ignore/wchar_t/94749.cc @@ -89,7 +89,7 @@ struct buff : std::basic_streambuf typedef std::streamsizestreamsize; typedef std::numeric_limits limits; - buff() : count(0), buf() { } + buff() : count(0), nonzero_chars(), buf() { } int_type underflow() { @@ -112,12 +112,23 @@ struct buff : std::basic_streambuf buf[headroom+1] = L'3'; this->setg(buf, buf, buf + headroom + 2); count = limits::max(); + nonzero_chars = headroom - 1; } return buf[0]; } + void reset() + { +buf[nonzero_chars] = char_type(); +buf[nonzero_chars+1] = char_type(); +buf[nonzero_chars+2] = char_type(); +nonzero_chars = 0; +count = 0; + } + streamsize count; + streamsize nonzero_chars; static const streamsize bufsz = 2048 << limits::digits10; char_type buf[bufsz + 2]; @@ -132,7 +143,8 @@ test05() typedef std::char_traits T; - std::basic_istream in(new buff); + buff* pbuf = new buff; + std::basic_istream in(pbuf); in.ignore(std::numeric_limits::max(), L'1'); VERIFY(in.good()); @@ -141,7 +153,9 @@ test05() VERIFY(in.get() == L'3'); VERIFY(in.get() == T::eof()); - delete in.rdbuf(new buff); + pbuf->reset(); + in.clear(); + VERIFY(in.gcount() == 0); in.ignore(std::numeric_limits::max(), L'2'); VERIFY(in.good()); @@ -150,7 +164,9 @@ test05() VERIFY(in.get() == L'3'); VERIFY(in.get() == T::eof()); - delete in.rdbuf(new buff); + pbuf->reset(); + in.clear(); + VERIFY(in.gcount() == 0); in.ignore(std::numeric_limits::max(), L'3'); VERIFY(in.good()); @@ -158,7 +174,9 @@ test05() VERIFY(in.gcount() == std::numeric_limits::max()); VERIFY(in.get() == T::eof()); - delete in.rdbuf(new buff); + pbuf->reset(); + in.clear(); + VERIFY(in.gcount() == 0); in.ignore(std::numeric_limits::max(), L'4'); VERIFY(in.eof()); @@ -166,7 +184,8 @@ test05() VERIFY(in.gcount() == std::numeric_limits::max()); VERIFY(in.get() == T::eof()); - delete in.rdbuf(0); + in.rdbuf(0); + delete pbuf; } void @@ -177,7
Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long)
> From: Mike Stump > Date: Fri, 9 Jun 2023 10:18:45 -0700 > On Jun 9, 2023, at 9:20 AM, Hans-Peter Nilsson via Gcc-patches > wrote: > > > > The test 27_io/basic_istream/ignore/wchar_t/94749.cc takes > > about 10 minutes to run for cris-elf in the "gdb simulator" > > I'd let the libstdc++ people comment on specific things. > I'll comment on general things. We could let line count > (or word count or character count) scale the timeout in > part, we could record times in a db and put an expected > run time into test cases or in an along side db. We could > have factors for slow systems, slow simulators. A 5 GHz > x86_64 will likely be faster that a 40 year old pdp11. We > can have these scale factors trigger off OS, cpu > statically, and/or we can do a quick bogomips calculation > and let that scale it and record that scaling factor in > the build tree. Wild plans, but with some points. Beware that uniform testing IMO weighs in much heavier than uniform test-time. Like, arm-eabi, rv32-elf and cris-elf, having common main factors, should test the same code and the same number of iterations, preferably regardless of simulator quality. (FWIW, I consider the cris-elf gdb simulator is *fast* - before 2021 or when built --disable-sim-hardware.) > A wealth of possibilities. And complexity! > Solutions that require maintenance or test case > modification are annoying. Yeah, but that maintenance annoyance unfortunately includes initial setup. You propose quite a major shift there. It sounds good, but sorry, but I must settle for just editing the test-case some way. brgds, H-P
Re: Splitting up 27_io/basic_istream/ignore/wchar_t/94749.cc (takes too long)
On Jun 9, 2023, at 9:20 AM, Hans-Peter Nilsson via Gcc-patches wrote: > > The test 27_io/basic_istream/ignore/wchar_t/94749.cc takes > about 10 minutes to run for cris-elf in the "gdb simulator" I'd let the libstdc++ people comment on specific things. I'll comment on general things. We could let line count (or word count or character count) scale the timeout in part, we could record times in a db and put an expected run time into test cases or in an along side db. We could have factors for slow systems, slow simulators. A 5 GHz x86_64 will likely be faster that a 40 year old pdp11. We can have these scale factors trigger off OS, cpu statically, and/or we can do a quick bogomips calculation and let that scale it and record that scaling factor in the build tree. A wealth of possibilities. Solutions that require maintenance or test case modification are annoying. Solutions that need port work are annoying. I'd be tempted to say bogomips into the build (test) tree. There are two parts, time to compile test cases and time to run them. I'd be fine with a half solution that only does what you need. The other part can be done by someone that has a need. I'd invite comments by others on other solutions or commentary on downsides. For example, having a 208 thread count machine that takes 2-3 minutes to run the full testsuite is nice. A problem arises when 4-10 test cases suddenly start timing out. You then go to around 10-53 minutes to test, which is annoying. Anything that boosts the timeouts can hinder early port bring up, which, we'd like to avoid. I mention it without much a solution other than a db approach in the test tree that records each test case and can identify test cases that timeout and trim the timeout for them to something nicer like base + 50% once they timeout with a larger allotment of time. We could entertain wild thoughts. For example, make a run cache that caches run results given an object. See an object in the future, just look it up in a hash cache for the object and return those results instead of running it. This can give you a large speedup in testing and would simultaneously advantage all slow simulation ports. Maybe a 20-100x speedup? If you want to go this way I'd say do it in python at the bottom as it would be nice to switch over to python in the next 5-20 years and away from tcl. A object cache in python, should be fairly small wether it is used for remembering run times from previous runs and setting a timeout based upon it, or as a does it run and pass or run and fail cache. The caches are likely only part of the problem, one still needs to have a timeout when no cache entry it present. They can speed testing for the day to day grind of people that run 1-200 times a week.