Thanks John. His argument definitely makes sense (that algorithms that cause more garbage collection won't get penalized by median, unless, of course, they cause gc() to occur more than 50% of the time).
Most benchmarks of Julia code that I've done (or seen) have made some attempt to take gc() differences out of the equation, usually by explicitly calling gc() before the timing begins. For most algorithms, that would mean that the same number of gc() calls should occur for each repetition, in which case, I would think that any measure of central tendency (including mean and median) would be useful. Is there a problem with this reasoning? Cheers, Kevin On Mon, Jun 2, 2014 at 1:04 PM, John Myles White <johnmyleswh...@gmail.com> wrote: > For some reasons why one might want not to use the median, see > http://radfordneal.wordpress.com/2014/02/02/inaccurate-results-from-microbenchmark/ > > -- John > > On Jun 2, 2014, at 11:06 AM, Kevin Squire <kevin.squ...@gmail.com> wrote: > > median is probably also useful. I like it a little better in cases where > the code being tested triggers gc() more than half the time. > > On Monday, June 2, 2014, Steven G. Johnson <stevenj....@gmail.com> wrote: > >> >> >> On Monday, June 2, 2014 1:01:25 AM UTC-4, Jameson wrote: >>> >>> Therefore, for benchmarks, you should execute your code in a loop enough >>> times that the measurement error (of the hardware and OS) is not too >>> significant. >>> >> >> You can also often benchmark multiple times and take the minimum (not the >> mean!) time for reasonable results with fairly small time intervals. >> > >
