Thanks Peter :) I will make these computations tomorrow at the lab and see what the confidence interval looks like.
Steven. Le 2017-06-14 20:03, PBKResearch a écrit : > Steven > > The question you need to consider is how much the results vary from one > repetition to another. If they are very variable, then the average will still > be variable. In the case of your 100 repetitions, for instance, you can work > out a confidence interval for the mean quite easily. Find the mean and > standard deviation of the 100 repetitions; the standard error of the mean is > the sample standard deviation divided by the square root of the number of > values averaged (i.e. 10 in this case); the approximate 95% confidence > interval is the mean +/- twice the standard error of the mean. (This makes a > lot of simplifying assumptions, but should be sufficient for your purposes.) > > If, as I suspect, the width of the confidence interval is quite large in > relation to the mean, this means that you cannot consistently measure the > speed of operations like this. You could try greatly increasing the number of > repetitions, but the square root law is working against you. 10,000 > repetitions would give you an interval about 10% of the width of 100 > repetitions, 1,000,000 repetitions would reduce it to 1%. > > Hope this is helpful > > Peter Kenny > > FROM: Pharo-users [mailto:pharo-users-boun...@lists.pharo.org] ON BEHALF OF > Steven Costiou > SENT: 14 June 2017 18:14 > TO: Pharo users users <pharo-users@lists.pharo.org> > SUBJECT: [Pharo-users] About how to benchmark the lookup speed... > > Hi, > > i'm having trouble doing benchmarks in Pharo, at least i don't understand the > results (maybe i'm missing something). I want to compare anonymous subclasses > lookup speed with standard pharo classes lookup speed. > > I have a method m and a method m2 that execute the following code: ^ 100 > printString. I have implemented these 2 methods in one class then in one > anonymous subclass. I thought that measuring the execution speed of the same > code in these two different cases would give me different times if the lookup > speed was not the same. > > So I did the following: > > |o results repetitions | > > o := theClass new. > > results := OrderedCollection new. > > repetitions := 100. > > repetitions timesRepeat:[ results add: [1000000 timesRepeat:[o m]] > timeToRun]. > > I do the same for m2, and then i compute an average of all the values > measured in results. > > What i don't understand is that, for example, for an average on 100 > measurements, m is 1% slower and m2 is 2% faster in the Pharo class case than > with anonymous subclasses. But for 1 000 measurements, m is 11% faster but m2 > is 3% slower. Results continue to vary as i change the number of > measurements, but they do not increase with it (seems not to be linear). > > I don't have enough knowledge about how to benchmark code, or what will make > a difference in Pharo. For now the only explanations i have is that maybe the > results are too slow to be significant and then they can vary, or i have done > something wrong in how i measure it. > > How would you measure the lookup speed of a method ? > > Steven.
