I know this is horribly late, but I just worked my way around to it!
Sorry for that.
Here's the recap of the Fedora 16 Graphics Test Week. The overall
numbers of tests done and bugs filed are down significantly, because I
was very busy and did not do a great job of arranging and promoting the
events this cycle. I hope we'll be able to do a better job for the
Fedora 17 graphics test week.
Here's the numbers on tests conducted and bugs filed. I counted each
non-example line in the main results matrix as a 'test conducted'. I
didn't count the runs in the 'bonus' 3D matrix, in order to provide a
more fair comparison with previous releases. This might affect the
bugs-to-tests ratio somewhat.
f11 nouveau: 104 tests, 42 bugs - ratio 0.40
f12 nouveau: 53 tests, 34 bugs - ratio 0.64
f13 nouveau: 78 tests, 26 bugs - ratio 0.33
f14 nouveau: 39 tests, 8 bugs - ratio 0.21
f15 nouveau: 83 tests, 55 bugs - ratio 0.66
f16 nouveau: 16 tests, 8 bugs - ratio 0.50
f11 radeon: 55 tests, 46 bugs - ratio 0.84
f12 radeon: 61 tests, 81 bugs - ratio 1.33
f13 radeon: 48 tests, 33 bugs - ratio 0.69
f14 radeon: 32 tests, 18 bugs - ratio 0.56
f15 radeon: 66 tests, 38 bugs - ratio 0.58
f16 radeon: 12 tests, 8 bugs - ratio 0.67
f11 intel: 23 tests, 21 bugs - ratio 0.91
f12 intel: 29 tests, 31 bugs - ratio 1.07
f13 intel: 38 tests, 38 bugs - ratio 1.00
f14 intel: 33 tests, 28 bugs - ratio 0.84
f15 intel: 37 tests, 25 bugs - ratio 0.68
f16 intel: 10 tests, 7 bugs - ratio 0.70
It's hard to draw any conclusions from such small numbers, but really,
none of them are far out of line with previous events.
Here's the chart for how reported bugs are handled, adding the results
from the F15 events (this chart gets updated on a one-cycle delay,
because it's concerned with how the bugs filed are handled after a
reasonable amount of time for the developers to look at them). A full
explanation of what this tracks and how it's calculated in the F13 recap
at
https://lists.fedoraproject.org/pipermail/test/2010-April/090271.html .
Some resolutions showed up this time that didn't before. I counted
'WORKSFORME' in with 'closeddupe' - i.e. as part of the set of bugs that
should just be completed discarded from consideration. I handled
'UPSTREAM' bugs on a case-by-case basis, going in and look at whether
they'd actually been fixed upstream or what, and putting them in the
most appropriate group. Also, if you try to reproduce these numbers,
note that you'll want to remove the bugs 123456 and 234567 from the
lists - they're used in the 'example' entries in the results matrix. The
bug totals are slightly different from the above chart because I
copy/pasted the F15 numbers in the above chart from the last recap, but
the numbers in this chart are newly generated, and so a few bugs that
were filed and added to the Wiki pages *after* the last recap went out
have been added to the numbers in this chart.
f11 nouveau: 42 bugs, 4 open, 8 closeddupe, 24 closedfixed, 6 closedunfixed -
70.59%
f12 nouveau: 34 bugs, 11 open, 8 closeddupe, 14 closedfixed, 1 closedunfixed -
53.85%
f13 nouveau: 27 bugs, 17 open, 6 closeddupe, 3 closedfixed, 1 closedunfixed -
14.29%
f14 nouveau: 8 bugs, 5 open, 3 closeddupe - 0% (small sample)
f15 nouveau: 58 bugs, 27 open, 13 closeddupe, 13 closedfixed, 5 closedunfixed -
28.89%
f11 radeon: 46 bugs, 14 open, 10 closeddupe, 19 closedfixed, 3 closedunfixed -
52.78%
f12 radeon: 81 bugs, 19 open, 32 closeddupe, 28 closedfixed, 2 closedunfixed -
57.14%
f13 radeon: 36 bugs, 28 open, 3 closeddupe, 5 closedfixed, 0 closedunfixed -
15.15%
f14 radeon: 18 bugs, 13 open, 0 closeddupe, 3 closedfixed, 2 closedunfixed -
16.67%
f15 radeon: 38 bugs, 17 open, 10 closeddupe, 9 closedfixed, 2 closedunfixed -
32.14%
f11 intel: 21 bugs, 7 open, 1 closeddupe, 12 closedfixed, 1 closedunfixed - 60%
f12 intel: 31 bugs, 7 open, 12 closeddupe, 12 closedfixed, 0 closedunfixed -
63.16%
f13 intel: 42 bugs, 26 open, 4 closeddupe, 11 closedfixed, 1 closedunfixed -
28.95%
f14 intel: 28 bugs, 21 open, 4 closeddupe, 1 closedfixed, 2 closedunfixed -
4.17%
f15 intel: 27 bugs, 8 open, 7 closeddupe, 12 closedfixed, 0 closedunfixed - 60%
There's definitely good news here - as noted in the last recap, we had a
good crop of reports from the F15 event, so there's no reason we
shouldn't be able to get back up to this level with better organization
and promotion for F17. Also, the number of bugs actually getting *fixed*
- which is the ultimate goal of the event - was significantly higher
than F13 and F14, not quite back to F11/F12 levels, but a lot better. So
the trend of fewer bugs getting fixed seems to have been stopped during
F15 cycle. Intel showed a particularly marked improvement here. (It's
worth noting that some of the bugs filed and fixed were actually on
GNOME rather than the X drivers; I didn't compare the level of non-X
bugs in the results in other releases, so that is an untracked variable.
If anyone wants to look into that, please do!)
Many thanks as al
