On Mon, Aug 03, 2009 at 10:03:47AM -0500, Kevin Grittner wrote: > That's about 0.52% slower with the patch. Because there was over 10% > variation in the numbers with the patch, I tried leaving out the four > highest outliers on both, in case it was the result of some other > activity on the system (even though this machine should have been > pretty quiet over the weekend) and the difference fell to 0.09%.
What do people do when testing this? I think I'd look to something like Student's t-test to check for statistical significance. My working would go something like: I assume the variance is the same because it's being tested on the same machine. samples = 20 stddev = 144.26 avg1 = 4783.13 avg2 = 4758.46 t = 0.54 ((avg1 - avg2) / (stddev * sqrt(2/samples))) We then have to choose how certain we want to be that they're actually different, 90% is a reasonably easy level to hit (i.e. one part in ten, with 95% being more commonly quoted). For 20 samples we have 19 degrees of freedom--giving us a cut-off[1] of 1.328. 0.54 is obviously well below this allowing us to say that there's no "statistical significance" between the two samples at a 90% level. -- Sam http://samason.me.uk/ [1] http://en.wikipedia.org/wiki/Student's_t-distribution#Table_of_selected_values -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
