Thanks Gabe. I had completely forgotten about the fact we can freely distribute some of those tests. You're suggestion on creating a second, shorter regression tester that focuses on testing different mechanisms sounds like a great idea. Hopefully we can get that done sometime.
In the meantime, let's just make a note to update the wiki in the near future on the current procedure for running the regression tester, pointing people to the binaries that we can't distribute ourselves. Brad > -----Original Message----- > From: [email protected] [mailto:[email protected]] On > Behalf Of Gabriel Michael Black > Sent: Monday, January 17, 2011 4:23 PM > To: [email protected] > Subject: Re: [m5-dev] EIO Regression Tests > > I think there are two important aspects of this issue. > > 1. Using regression tests we can't distribute freely has some > important limitations. It would be nice to replace them with ones we > can. > > 2. The majority of the regression tests we have now are really > benchmarks which provide basic coverage by working/not working and not > changing behavior unexpectedly. That's an important element to have > since it's a practical reality check and probably hits things we > wouldn't think to test. They have significant limitations, though, > since they take a long time to run and tend to exercise the same > simulator functionality over and over. For instance, gcc may generate > code that always has the same type of backward branch for a for loop. > Using gzip as a test will verify that that branch works, but possibly > not the slightly different variant that may, for instance, use a large > branch displacement. Even when writing code in x86 assembly it can be > impossible to predict which of the possibly many redundant instruction > encodings the assembler might pick. > > So, in everyone's infinite free time, I think we should replace our > benchmark based regressions with a smaller set of freely distributable > regressions/inputs, and augment them with shorter, targeted tests that > exercise particular mechanisms, circumstances, instructions, etc. > Instead of replacing our existing benchmarks which are useful as > actual benchmarks and are good to keep working, we could build up this > second set of tests in parallel. > > Gabe > > Quoting "Beckmann, Brad" <[email protected]>: > > > Hi Nilay, > > > > I understand your confusion. This is an example of where the wiki > > needs to be updated. I believe the wiki only mentions the > > encumbered tar ball and doesn't mention the encumbered hg repo on > > repo.m5sim.org. As far as the anagram test program goes, I remember > > Lisa and I encountered the same issue a while back and to resolve it > > I believe Lisa copied that test along with several other regression > > tester programs from Michigan to AMD. > > > > I can provide you those regression tester programs, but at a higher > > level, I think this is a good time to ask the question on how we > > want to provide external users all the files necessary to run the > > regression tester? As Nilay points out, the encumbered repo has > > some, but not all of the necessary files. I believe, one also needs > > another set of regression tester programs which include both the > > anagram files, as well as the SPECCPU files for the long regression > > tester runs. > > > > Thoughts? > > > > Brad > > > > > >> -----Original Message----- > >> From: [email protected] [mailto:[email protected]] On > >> Behalf Of Nilay Vaish > >> Sent: Monday, January 17, 2011 1:55 PM > >> To: M5 Developer List > >> Subject: Re: [m5-dev] EIO Regression Tests > >> > >> I figured that out, but there is no anagram directory in tests/test- > >> progs. > >> I, therefore, receive the following error: > >> > >> gzip: tests/test-progs/anagram/bin/alpha/eio/anagram-vshort.eio.gz: > No > >> such file or directory > >> > >> -- > >> Nilay > >> > >> On Mon, 17 Jan 2011, Steve Reinhardt wrote: > >> > >> > The one where the EIO code lives. That's it's name, at > >> > http://repo.m5sim.org. > >> > > >> > On Mon, Jan 17, 2011 at 12:59 PM, Nilay Vaish <[email protected]> > >> wrote: > >> > > >> >> What do you mean by the encumbered repository? > >> >> > >> >> > >> >> On Mon, 17 Jan 2011, Steve Reinhardt wrote: > >> >> > >> >> Yes, it should be a concern... it should work. Did you do a > pull > >> on the > >> >>> encumbered repository? There were some changes there needed to > >> maintain > >> >>> compatibility with the latest m5 dev repo. > >> >>> > >> >>> Otherwise you'll need to provide more detail about how things > >> failed. > >> >>> > >> >>> Steve > >> >>> > >> >>> On Mon, Jan 17, 2011 at 10:21 AM, Nilay Vaish > <[email protected]> > >> wrote: > >> >>> > >> >>> I just ran the regression tests for the patch (deals with SLICC > >> and cache > >> >>>> coherence protocols) that I need to commit. The EIO tests fail. > >> Should > >> >>>> this > >> >>>> be a concern? > >> >>>> > >> >>>> -- > >> >>>> Nilay > >> >>>> _______________________________________________ > >> >>>> m5-dev mailing list > >> >>>> [email protected] > >> >>>> http://m5sim.org/mailman/listinfo/m5-dev > >> >>>> > >> >>>> > >> >>> _______________________________________________ > >> >> m5-dev mailing list > >> >> [email protected] > >> >> http://m5sim.org/mailman/listinfo/m5-dev > >> >> > >> > > >> _______________________________________________ > >> m5-dev mailing list > >> [email protected] > >> http://m5sim.org/mailman/listinfo/m5-dev > > > > > > _______________________________________________ > > m5-dev mailing list > > [email protected] > > http://m5sim.org/mailman/listinfo/m5-dev > > > > > _______________________________________________ > m5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/m5-dev _______________________________________________ m5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/m5-dev
