Jim Cromie wrote: > What's notable in its absence is any *real* use of perl-dist's tests. > I dug into the code, and found that this works. > > $> HARNESS_TIMER=1 make test > ext/threads/t/end.............................................ok 60 ms > ext/threads/t/err.............................................ok 1090 ms > ext/threads/t/exit............................................ok 365 ms > ext/threads/t/free2...........................................ok 2367 ms > ext/threads/t/free............................................ok 12383 ms > > so clearly: > - somebody thought of this already ;-) > - does Test::* support this feature ? > - t/harness doesnt, but HARNESS_OPTIONS could allow 't' > - it wouldnt be too hard to collect this data. > - could by default for -Dusedevel
Here's the problems with this approach: A) Unless I'm mistaken, its doing wallclock measurement. B) Its including time for the harness' own parsing. C) Its including time for Test::More / test.pl. D) The tests change. A is a known problem and we know how to fix it, record CPU and SYS time. B and C mean a change to Test::Harness or Test::More could throw the whole thing off. Since they change all the time you can't compare the results from Perl version A with Perl version B which are any significant distance apart. Maybe Perl's performance changed, maybe Test::More's did. There will be too many false positives to get useful results. B can probably be solved by just measuring the CPU and system time of the child process. C is tricker and I have no good suggestions. D is not really possible to solve in the general case. For any given version of Perl you've got different test code. Before a benchmark could be compared you'd have to see if the test file changed between versions. They will. You could try freezing the test suite and using that as a benchmark suite but eventually stuff will break. There is information to be gotten out of just saving test performance snapshots, but it is not trivial to interpret. Interpreted wrong it would lead to bad decisions. -- Robrt: People can't win Schwern: No, but they can riot after the game.
