On Sun, Dec 23, 2012 at 11:11 PM, Jeff Janes <jeff.ja...@gmail.com> wrote: > You could say that benchmarks should run long enough to average out such > changes
I would say any benchmark needs to be run long enough to reach a steady state before the measurements are taken. The usual practice is to run a series groups and observe the aggregate measurements for each group. For instance run 10 runs with each run including of 1000 repetitions of the transaction. Then you can observe at which point the averages for individual groups begin to behave consistently. If the first three are outliers but the remaining 7 are stable then discard the first three and take the average (or often median) of the remaining 7. If you include the early runs which are affected by non-steady-state conditions such as cache effects or file fragmentation then it can take a very long time for those effects to be erased by averaging with later results. Worse, it's very difficult to tell whether you've waited long enough. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers