I think the point made my Bryan and Katie (and I have to agree with them) is
that there's no way to see at a glance if we reached acceptable performance
or not. The color code currently used is misleading (e.g. why is the Linux
perf of importing a 3000 events calendar green? it's far from acceptable, if
it's just to see that it is better than 0.5, it's easy to see at a glance...)
Proposal: use a color code that's relevant to the 0.6 targets (as described
in http://wiki.osafoundation.org/bin/view/Projects/PerformanceProject).
. red: perf is currently double the targetted acceptable value (200%)
. orange: perf is between 150% and 200% the targetted acceptable value
. yellow: perf is between 100% and 150% the targetted acceptable value
. green: perf is at or better than the targetted acceptable value
If the numbers do include comparisons with previous runs it would be great to
refrain from comparing apples and oranges. When a change occurs in a test or
in the test environment, a disclaimer should be presented alongside the
temporarily 'bogus' comparison numbers. Some hypothetical and not so
hypothetical examples:
- the linux test machine's hard drive crashed and a newer, faster, hard
drive was put in place
- the windows machine was re-installed for the 7th time after it got
infected with the 897#$ virus
- the mac machine's operating system was upgraded to Mac OS X 10.4.89
- the 3000 event calendar import test now includes committing the 3943 items
generated (that one's real)
Anyway, you get the idea...
Andi..
Cheers,
- Philippe
Heikki Toivonen wrote:
Katie Capps Parlante wrote:
If feasible, it would be useful to have an added column that tracks how
the current time compares to the target time for 0.6 release (at least
in the short term).
I considered this, and stearns requested this as well, but I don't see
that adding much value. The reason is that our targets are so simple and
uniform: 1 second for all cases except startup and importing 3000
events. You can see at a glance how 1.2 seconds compares to 1 second...
------------------------------------------------------------------------
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
