On Wed, Nov 24, 2010 at 04:52:38PM +0200, Antti Kantee wrote:
> Thanks, I'll use your list as a starting point. One question though:
>
> On Wed Nov 24 2010 at 00:16:37 +0000, Andrew Doran wrote:
> > - build.sh on a static, unchanging source tree.
>
> >From the SSP discussion I have a recollection that build.sh can be
> very jittery, up to the order of 1% per build. I've never confirmed it
> myself, though. Did you notice anything like that?
There are other issues associated with build.sh as a benchmark.
* What are you trying to test? If you're trying to test the
efficiency of cache algorithms or the I/O subsystem (including
disk sort), for example, you need to test pairs of runs with
a cold boot of *ALL INVOLVED HARDWARE* (this includes disk
arrays etc) between each.
* If SSDs, hybrid disks, or other potentially self-reorganizing
media are involved, forget it, you just basically lose.
* If you're trying to test everything *but* the cache and I/O
subsystem, then you need to use a "warm up" procedure you can
have reasonable confidence works, for example always measuring
the Nth of N consecutive builds.
* It can be hard to construct a system configuration where NetBSD
kernel performance is actually the bottleneck and some other
hardware limitation is not. Or where there's only a single
bottleneck.
It's a good macrobenchmark but be careful. And report results in a
comprehensive way.
Thor
