Heikki Toivonen wrote:
It doesn't work like that (or maybe I misunderstood). If the current
measured value is +/- std dev from last measured value, we don't have a
very high confidence in there being any real change (and additionally,
if the current value is +/- from our target, we don't know if we really
hit our target). Yet I'd actually like to see what the current measured
value is.
Maybe I have a naive understanding of standard deviation then. My
understanding is that if you repeatedly run the same test, and the
standard deviation is "high" (by some human-chosen measure) then the
variability in the measurements is too high to consider the test
reliable. What samples are you using for standard deviation? I don't
see how a high standard deviation results in something we're interested
in.
Something I just realized: it would make it a little bit clearer if the
top headers said something like this:
Windows (r 7503 vs r 7501)
absolutely.
2) instead of "0.6" vs. "time" how about "target" vs. "actual" or
"target" vs. "current"? Jeffrey just said "wait, is 0.6 the target? or
is it a typo and it should be 0.5?
I have trouble seeing how this is confusing (but you could convince me I
guess). "0.6 target" is, well, our target performance numbers for 0.6
release. Once we hit those numbers we are good.
What can I say, I find it confusing. I don't think there's much more
than that.. ?
3) if we're putting "s" in the seconds column, why can't we put "%" in
the percent column? Again, I look and see "13" and don't know what that
means.
I do have % there. Or did I miss something?
nope, pasted from tinderbox right now:
| Test |
0.6 |
time |
Δ % |
Δ time |
std.dev |
0.6 |
time |
Δ % |
Δ time |
std.dev |
0.6 |
time |
Δ % |
Δ time |
std.dev |
| Switching
Views |
1s |
1.14s |
-92 |
-0.55s |
0.0000s |
1s |
1.41s |
-82 |
-0.63s |
0.0000s |
0s |
0.58s |
-75 |
-0.25s |
0.0000s |
Note that first row, it has a % sign in the header, but not the actual
rows.. wheras the actual rows do have an "s"
4) Why can't we put "+" in front of positive delta values? delta by
definition is +/-, so I find leaving out the "+" to be very confusing.
I do have +/-, or did I miss a sign somewhere?
again, not in the table I'm seeing on tinderbox right now.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
|
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
