So what's the point in having it without reference outputs? Sure you could run it, but as a regression it wouldn't be very valuable since you wouldn't know if anything changed or not. I guess you could run it to see if it completed, but I'm not sure how much value is in that (or whether anyone would ever do it).
I think some of the existing long regressions are already too long, and I don't really want to see us add regressions that are any longer. The goal of regressions is to make sure you didn't break anything, so you want to maximize the variety of things you test. It's not clear that (once you get above some threshold) running a test for longer is going to exercise new code interactions that might actually expose bugs... as Ali said, it's not clear that running 20 hours of bzip2 is a more stressful test than 10 or 5 or even 2 hours of running bzip2 (unless you're looking for a slow memory leak, but there are better ways of catching that). Steve On Fri, Feb 4, 2011 at 11:58 PM, Gabe Black <[email protected]> wrote: > That's actually what we're talking about. All the script bits are already > there, I'd just be adding reference outputs. > > Gabe > > > On 02/04/11 23:49, nathan binkert wrote: > > Can't he commit the regression without any reference stats? I thought it > wouldn't run if there were no refs. Perhaps the script is so simple as to > be not worth it? > > Nate > > On Fri, Feb 4, 2011 at 9:17 AM, Ali Saidi <[email protected]> wrote: > >> My opinion is that I've never had bzip find a bug that some other spec >> test hasn't. I think 20 hours is way too long for a regression test. We >> should either shrink the input size or just not have it. >> >> >> >> Ali >> >> >> >> On Thu, 03 Feb 2011 15:35:29 -0800, Gabe Black wrote: >> >> So how do people feel about this? bzip2 in x86 on O3 takes more than 20 >> hours to run. Should that be part of the regressions or not? I have it ready >> to go if that's acceptable, but to me it seems too long. >> >> Gabe >> >> On 02/01/11 17:30, Gabe Black wrote: >> >> Normally I wouldn't send stats updates out for review. I'm adding new >> regressions here, though, and I think people may be interested in the run >> time, especially for bzip2. I think it was about 20+ hours on opt on a big >> machine from HP. This may be trimmed down if/when performance bottlenecks >> get fixed since O3 will have to do less work, but cutting it in half is >> still a long, long time for a regression. I notice that the number of ticks >> for the benchmarks that are in common with Alpha is about 4 times higher, >> suggesting a performance bug of some sort (I doubt x86 gcc is 4X worse). >> >> Gabe >> >> On 02/01/11 17:23, Gabe Black wrote: >> >> This is an automatically generated e-mail. To reply, visit: >> http://reviews.m5sim.org/r/463/ >> Review request for Default, Ali Saidi, Gabe Black, Steve Reinhardt, and >> Nathan Binkert. >> By Gabe Black. >> Description >> >> X86: Add o3 regressions in SE mode. >> >> Diffs >> >> - tests/long/00.gzip/ref/x86/linux/o3-timing/config.ini (PRE-CREATION) >> - tests/long/00.gzip/ref/x86/linux/o3-timing/simerr (PRE-CREATION) >> - tests/long/00.gzip/ref/x86/linux/o3-timing/simout (PRE-CREATION) >> - tests/long/00.gzip/ref/x86/linux/o3-timing/stats.txt (PRE-CREATION) >> - tests/long/10.mcf/ref/x86/linux/o3-timing/config.ini (PRE-CREATION) >> - tests/long/10.mcf/ref/x86/linux/o3-timing/mcf.out (PRE-CREATION) >> - tests/long/10.mcf/ref/x86/linux/o3-timing/simerr (PRE-CREATION) >> - tests/long/10.mcf/ref/x86/linux/o3-timing/simout (PRE-CREATION) >> - tests/long/10.mcf/ref/x86/linux/o3-timing/stats.txt (PRE-CREATION) >> - tests/long/20.parser/ref/x86/linux/o3-timing/config.ini >> (PRE-CREATION) >> - tests/long/20.parser/ref/x86/linux/o3-timing/simerr (PRE-CREATION) >> - tests/long/20.parser/ref/x86/linux/o3-timing/simout (PRE-CREATION) >> - tests/long/20.parser/ref/x86/linux/o3-timing/stats.txt >> (PRE-CREATION) >> - tests/long/60.bzip2/ref/x86/linux/o3-timing/config.ini >> (PRE-CREATION) >> - tests/long/60.bzip2/ref/x86/linux/o3-timing/simerr (PRE-CREATION) >> - tests/long/60.bzip2/ref/x86/linux/o3-timing/simout (PRE-CREATION) >> - tests/long/60.bzip2/ref/x86/linux/o3-timing/stats.txt (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/config.ini >> (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/simerr (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/simout (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/smred.out (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/smred.pin (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/smred.pl1 (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/smred.pl2 (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/smred.sav (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/smred.sv2 (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/smred.twf (PRE-CREATION) >> - tests/long/70.twolf/ref/x86/linux/o3-timing/stats.txt (PRE-CREATION) >> - tests/quick/00.hello/ref/x86/linux/o3-timing/config.ini >> (PRE-CREATION) >> - tests/quick/00.hello/ref/x86/linux/o3-timing/simerr (PRE-CREATION) >> - tests/quick/00.hello/ref/x86/linux/o3-timing/simout (PRE-CREATION) >> - tests/quick/00.hello/ref/x86/linux/o3-timing/stats.txt >> (PRE-CREATION) >> >> View Diff <http://reviews.m5sim.org/r/463/diff/> >> _______________________________________________ m5-dev mailing list >> [email protected] http://m5sim.org/mailman/listinfo/m5-dev >> >> >> _______________________________________________ m5-dev mailing list >> [email protected] http://m5sim.org/mailman/listinfo/m5-dev >> >> >> >> >> _______________________________________________ >> m5-dev mailing list >> [email protected] >> http://m5sim.org/mailman/listinfo/m5-dev >> >> > > _______________________________________________ > m5-dev mailing list > [email protected]http://m5sim.org/mailman/listinfo/m5-dev > > > > _______________________________________________ > m5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/m5-dev > >
_______________________________________________ m5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/m5-dev
