OK, I realize the idea I was originally thinking of didn't solve the problem of not wanting to rerun all the non-ruby tests on every different ruby protocol build.
For those following at home (prob. everyone other than Brad and me), the issue is that the test infrastructure is designed to run every test that applies to a given binary, but with N different ruby protocol builds, you end up running all of the ruby-independent tests N times even though it should be sufficient to run them once. Brad's solution only runs the ruby-independent tests on the build that uses the "default" ruby protocol. However, that default protocol is hard-coded in the test script, and when the actual default protocol gets changed but the test script is not updated, then you end up not running any ruby-independent tests (which is what has been happening lately). Ideally I'd like to stay with the model of running every test on every binary it applies to; while the current solution optimizes some unnecessary tests out of the full regression, it also eliminates some potentially useful tests for people who might not be using the default ruby protocol. One option that I'm leaning toward is to add a scons flag to disable compiling in the "classic" M5 cache model. Then we can disable it in all but one of the ruby builds, and not run any tests that require the classic M5 memory hierarchy for those builds. That would still leave us running the purely functional tests (that really don't require a cache hierarchy at all) on all of the different ruby protocol builds, but I don't think any of those are very long, so that shouldn't be so bad to put up with. Steve On Sun, Jan 31, 2010 at 10:53 PM, Steve Reinhardt <[email protected]> wrote: > I'm not sure it's the stickiness... my interpretation is that Derek > changed the default in the src/mem/protocol/SConsopts file which made > it inconsistent with what you assumed it was in tests/SConscript. > Anyway, I think the fix I suggested that completely removes any > dependence on the default value in the test system is more robust and > what we should do in the long run. > > Steve > > On Sun, Jan 31, 2010 at 10:33 PM, Beckmann, Brad <[email protected]> > wrote: >> Yep, the problem is the PROTOCOL variable is sticky and therefore isn't >> being set to the default. >> >> I just pushed a fix to add the default PROTOCOL variable to all build_opt >> files. That should fix the regression tester, but it would be nice to >> always set the PROTOCOL variable to the default unless explicitly specified >> at the command line. >> >> Brad >> >> >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] On Behalf >> Of Steve Reinhardt >> Sent: Sunday, January 31, 2010 4:40 PM >> To: M5 Developer List >> Subject: Re: [m5-dev] Cron <m5t...@zizzer> >> /z/m5/regression/do-regression--scratch all >> >> OK, I believe it's a problem Brad was already worried about... see this: >> >> diff -r a162a2b0b1f8 -r 5bd33f7c26ea tests/SConscript >> --- a/tests/SConscript Fri Jan 29 20:29:34 2010 -0800 >> +++ b/tests/SConscript Fri Jan 29 20:29:40 2010 -0800 >> @@ -264,11 +264,23 @@ >> else: >> configs += ['simple-atomic', 'simple-timing', 'o3-timing', 'memtest', >> 'simple-atomic-mp', 'simple-timing-mp', 'o3-timing-mp', >> - 'inorder-timing'] >> + 'inorder-timing', 'rubytest'] >> >> if env['RUBY']: >> - # Hack for Ruby >> - configs += [c + '-ruby' for c in configs] >> + # With Ruby, A protocol must be specified in the environment >> + assert(env['PROTOCOL']) >> + >> + # >> + # Is there a way to determine what is Protocol EnumVariable >> + # default and eliminate the need to hard code the default protocol >> below? >> + # >> + # If the binary includes the default ruby protocol, run both ruby and >> + # non-ruby versions of the tests. Otherwise just run the ruby versions. >> + # >> + if env['PROTOCOL'] == 'MI_example': >> + configs += [c + "-ruby" for c in configs] >> + else: >> + configs = [c + "-ruby-" + env['PROTOCOL'] for c in configs] >> >> cwd = os.getcwd() >> os.chdir(str(Dir('.').srcdir)) >> >> and then compare with this change, which is from Derek's merge changeset: >> >> diff -r 289ac904233d -r 3d308cbd1657 src/mem/protocol/SConsopts >> --- a/src/mem/protocol/SConsopts Wed Nov 18 18:00:41 2009 -0800 >> +++ b/src/mem/protocol/SConsopts Tue Jan 19 15:48:12 2010 -0600 >> @@ -50,7 +50,7 @@ >> 'MOESI_hammer', >> ] >> >> -opt = EnumVariable('PROTOCOL', 'Coherence Protocol for Ruby', 'MI_example', >> +opt = EnumVariable('PROTOCOL', 'Coherence Protocol for Ruby', >> 'MOESI_CMP_directory', >> all_protocols) >> >> >> ... so that mismatch is causing all the builds that use the default >> Ruby protocol to not find any applicable configurations since >> tests/SConscript is looking for a different default protocol. >> >> I think the best way to fix this is to forget trying to have the test >> configs track the 'default' Ruby protocol, but instead to always have >> the protocol explicit in the config name. This will require moving >> the existing *-ruby reference outputs, but it's more stable in the >> long run. Brad and I can probably discuss this tomorrow and make it >> happen. >> >> Steve >> >> 2010/1/31 Steve Reinhardt <[email protected]>: >>> I'm looking at it now... I'm not sure exactly what the error is, but it >>> looks like it's running *only* the ruby tests now, and acting like the >>> non-ruby tests don't exist. >>> >>> Steve >>> >>> On Sun, Jan 31, 2010 at 11:24 AM, Beckmann, Brad <[email protected]> >>> wrote: >>>> I agree, I don't think it is working correctly. I don't have access to >>>> the Michigan server and I can't reproduce the error. Could someone send >>>> out the do-regression script? >>>> >>>> Thanks, >>>> >>>> Brad >> _______________________________________________ >> m5-dev mailing list >> [email protected] >> http://m5sim.org/mailman/listinfo/m5-dev >> >> >> _______________________________________________ >> m5-dev mailing list >> [email protected] >> http://m5sim.org/mailman/listinfo/m5-dev >> > _______________________________________________ m5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/m5-dev
