On Thu, Dec 04, 2003 at 11:30:47PM +0000, Fergal Daly wrote: > I'm not sure about that 25%. Say the pmip calibrator doesn't fit in the CPU > cache on any machine, then if the tested algorithm fits in the CPU cache on > one machine but not on another then there will be a huge difference in the > number of "perl seconds" they require. > > Worse still, if the pmip calibrator fits in the cache on my fancy new machine > I'll probably get lots of false negatives because everything seems taking > "perl ages" to run. Then you have the multi user machine where the test > passes when the machine is quiet but fails when's there lots of cache > contention.
I disbelieve such low level considerations will manifest themselves as large differences during a single testing process. Remember, perl runs tons of machine code for just one line of Perl. This isn't C. Also, remember this is CPU time, not wallclock. > You also have a (somewhat rarer) problem with people changing CPUs and not > recalibrating their pmips. The recalibration would happen at runtime, not install time. Yes, this will eat a few seconds, but it would be more accurate. C'est le testing. > If Test::Harness had a protocol for warnings rather than just pass and fail > then this would be more useful, print STDERR or diag(). Test::Harness doesn't need to know anything about warnings. If it can't fail its not really a test. -- Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern/ It's Highball time!
