On 4/16/12 12:40 AM, Rob Weir wrote:
On Sun, Apr 15, 2012 at 6:22 PM, Andrea Pescetti<pesce...@apache.org> wrote:
On 15/04/2012 Rob Weir wrote:
On Sun, Apr 15, 2012 at 11:49 AM, Andrea Pescetti wrote:
Historically, OpenOffice.org produced a numbered Release Candidate
(OpenOffice.org 3.3 had ten, RC1 to RC10) that was made available to the
community exactly for the purpose of looking for unknown bugs. QA
activities
were planned for the RC phase and, after a few days of availability, a RC
was approved as final or rejected (and the Release Manager could, and at
times did, include fixes for previously known bugs that had been
accumulating and that were significant; none of them was a blocker in
itself, but each of them would have caused problems to users, so their
combined effect was blocking the release).
That is what we've been doing with the dev snapsots builds. Surely
you've seen the QA work Lily has done with testing them?
Sure, and it was great work. But those tests were run on versions that are
now quite outdated. Example: the spell check test asks you to verify that no
spell check is available and that dictionaries have been removed accurately,
while we all know that things are quite different in current builds.
The traditional OpenOffice.org Quality Assurance process had two features
that I'd like to keep:
1) Full QA is run on what we release. We need to ensure that OpenOffice
works now, not that previous builds worked.
"Full" to me means a test matrix on all platforms, all functions, all
languages, and includes performance, interop and other tests. A full
test pass probably takes 4-6 weeks if we do it right. I don't think
we have the opportunity to do more than one "full" test pass per
release. But this should happen much earlier, right after a "feature
freeze" when all features targeted for a release have been checked in.
2) The community at large is involved in testing, in a specific short period
just before the release. If "Release Candidate" has a different meaning now,
call it the "Pre-Release Phase", that will end with a "Release Candidate" we
can vote on rather confidently. We can't rely on volunteers doing QA on a
regular basis, but we can very easily find volunteers willing to stress-test
a "nearly final" version in a well defined timeframe (1-2 weeks).
I would keep this as a basis for discussing how to structure QA activities
in future (after 3.4), if others agree.
This is only a fraction of the picture. We also need to consider what
changes are allowed, and how those changes are reviewed, as we
progress toward a Release Candidate. That is a critical part of how
we maintain quality. If committers are checking in whatever they feel
like, up to the day we have a RC, then we will have serious quality
problems.
When we changing code we "invalidate" prior testing. Not in an
absolute sense, for most code changes there is an test impact, and
some portion of the "full test" needs to be redone, to verify both the
fix, but also that we did not break anything else. Again, this
requires discipline from committers. "Last minute" feature work, or
even undisciplined bug fixing, can cause more problems than it fixes.
So I think we want the trunk to be fee CTR up to a certain point
(feature freeze?) and after that point only changes that are
accompanied by a BZ issue are allowed, but still CTR. Then at some
point, after the main test pass is complete, we start prioritizing
bugs as we create snapshot builds. We gradually raise the level
required for bugs to be accepted. P3, P2, P1. In the final weeks we
fix only marked release blockers. Once we have the RC build we
branch the code. At that point (and maybe even earlier) we switch to
RTC, e.g., no changes are allowed unless they are reviewed first.
Was anything like this done before with OOo?
Yes, more or less in a similar way and I think it is a good way to
achieve a high quality product.
And on the other hand we should try to provide as much as possible
automated testing. And volunteers should do manual testing not only RC
candidates. A continuous testing on dev snapshots would be very much
appreciated if possible.
Juergen
-Rob
Regards,
Andrea.
