Alec Flett wrote: > 1) it seems like stddev is really a function of the testing methodology > not the actual measurements, and totally inappropriate for this chart. > How about we just assume that tests over a certain stddev is simply bad > data? We shouldn't all have to learn what stddev is good and what is bad > to interpret the chart. Just have a threshold and if we're over the > threshold, blank out that test result because the actual measurements > are totally bogus.
It doesn't work like that (or maybe I misunderstood). If the current measured value is +/- std dev from last measured value, we don't have a very high confidence in there being any real change (and additionally, if the current value is +/- from our target, we don't know if we really hit our target). Yet I'd actually like to see what the current measured value is. If the current measured value differs more, then it is likely that we got a real change which we are really interested in. Mostly you could ignore std dev in the table, but I am interested in seeing how that develops (it tells something about our testing quality). Something I just realized: it would make it a little bit clearer if the top headers said something like this: Windows (r 7503 vs r 7501) > 2) instead of "0.6" vs. "time" how about "target" vs. "actual" or > "target" vs. "current"? Jeffrey just said "wait, is 0.6 the target? or > is it a typo and it should be 0.5? I have trouble seeing how this is confusing (but you could convince me I guess). "0.6 target" is, well, our target performance numbers for 0.6 release. Once we hit those numbers we are good. > 3) if we're putting "s" in the seconds column, why can't we put "%" in > the percent column? Again, I look and see "13" and don't know what that > means. I do have % there. Or did I miss something? > 4) Why can't we put "+" in front of positive delta values? delta by > definition is +/-, so I find leaving out the "+" to be very confusing. I do have +/-, or did I miss a sign somewhere? -- Heikki Toivonen
signature.asc
Description: OpenPGP digital signature
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Open Source Applications Foundation "Dev" mailing list http://lists.osafoundation.org/mailman/listinfo/dev
