Re: [RELEASE] Planning QA activities
On 16/04/2012 xia zhao wrote: 2012/4/16 Andrea Pescetti Sure, and it was great work. But those tests were run on versions that are now quite outdated. Example: the spell check test asks you to verify that no spell check is available and that dictionaries have been removed accurately, while we all know that things are quite different in current builds. How do you means tests were run on versions that are now quite outdated? http://wiki.services.openoffice.org/wiki/QA/TestCases/ has several testcases that would only apply to older builds (like: testing removal of dictionaries, but these are now bundled; testing removal of fonts, but these are now replaced; testing removal of HTTP features, but in the meantime those have been replaced too). 1) Full QA is run on what we release. We need to ensure that OpenOffice works now, not that previous builds worked. Agree if the Full here means test matrix on full platforms and full langauges. The OpenOffice you means here I understand is AOO. For AOO, the problem is we lost manual test cases I wouldn't despair that we get them back at some point: as I wrote a long time ago, it would be enough to receive the test cases, the infrastructure for running them can be replaced. Regards, Andrea.
Re: [RELEASE] Planning QA activities (was: Dictionary extensions)
On Sun, Apr 15, 2012 at 6:22 PM, Andrea Pescetti pesce...@apache.org wrote: On 15/04/2012 Rob Weir wrote: On Sun, Apr 15, 2012 at 11:49 AM, Andrea Pescetti wrote: Historically, OpenOffice.org produced a numbered Release Candidate (OpenOffice.org 3.3 had ten, RC1 to RC10) that was made available to the community exactly for the purpose of looking for unknown bugs. QA activities were planned for the RC phase and, after a few days of availability, a RC was approved as final or rejected (and the Release Manager could, and at times did, include fixes for previously known bugs that had been accumulating and that were significant; none of them was a blocker in itself, but each of them would have caused problems to users, so their combined effect was blocking the release). That is what we've been doing with the dev snapsots builds. Surely you've seen the QA work Lily has done with testing them? Sure, and it was great work. But those tests were run on versions that are now quite outdated. Example: the spell check test asks you to verify that no spell check is available and that dictionaries have been removed accurately, while we all know that things are quite different in current builds. The traditional OpenOffice.org Quality Assurance process had two features that I'd like to keep: 1) Full QA is run on what we release. We need to ensure that OpenOffice works now, not that previous builds worked. Full to me means a test matrix on all platforms, all functions, all languages, and includes performance, interop and other tests. A full test pass probably takes 4-6 weeks if we do it right. I don't think we have the opportunity to do more than one full test pass per release. But this should happen much earlier, right after a feature freeze when all features targeted for a release have been checked in. 2) The community at large is involved in testing, in a specific short period just before the release. If Release Candidate has a different meaning now, call it the Pre-Release Phase, that will end with a Release Candidate we can vote on rather confidently. We can't rely on volunteers doing QA on a regular basis, but we can very easily find volunteers willing to stress-test a nearly final version in a well defined timeframe (1-2 weeks). I would keep this as a basis for discussing how to structure QA activities in future (after 3.4), if others agree. This is only a fraction of the picture. We also need to consider what changes are allowed, and how those changes are reviewed, as we progress toward a Release Candidate. That is a critical part of how we maintain quality. If committers are checking in whatever they feel like, up to the day we have a RC, then we will have serious quality problems. When we changing code we invalidate prior testing. Not in an absolute sense, for most code changes there is an test impact, and some portion of the full test needs to be redone, to verify both the fix, but also that we did not break anything else. Again, this requires discipline from committers. Last minute feature work, or even undisciplined bug fixing, can cause more problems than it fixes. So I think we want the trunk to be fee CTR up to a certain point (feature freeze?) and after that point only changes that are accompanied by a BZ issue are allowed, but still CTR. Then at some point, after the main test pass is complete, we start prioritizing bugs as we create snapshot builds. We gradually raise the level required for bugs to be accepted. P3, P2, P1. In the final weeks we fix only marked release blockers. Once we have the RC build we branch the code. At that point (and maybe even earlier) we switch to RTC, e.g., no changes are allowed unless they are reviewed first. Was anything like this done before with OOo? -Rob Regards, Andrea.
Re: [RELEASE] Planning QA activities
On 4/16/12 12:40 AM, Rob Weir wrote: On Sun, Apr 15, 2012 at 6:22 PM, Andrea Pescettipesce...@apache.org wrote: On 15/04/2012 Rob Weir wrote: On Sun, Apr 15, 2012 at 11:49 AM, Andrea Pescetti wrote: Historically, OpenOffice.org produced a numbered Release Candidate (OpenOffice.org 3.3 had ten, RC1 to RC10) that was made available to the community exactly for the purpose of looking for unknown bugs. QA activities were planned for the RC phase and, after a few days of availability, a RC was approved as final or rejected (and the Release Manager could, and at times did, include fixes for previously known bugs that had been accumulating and that were significant; none of them was a blocker in itself, but each of them would have caused problems to users, so their combined effect was blocking the release). That is what we've been doing with the dev snapsots builds. Surely you've seen the QA work Lily has done with testing them? Sure, and it was great work. But those tests were run on versions that are now quite outdated. Example: the spell check test asks you to verify that no spell check is available and that dictionaries have been removed accurately, while we all know that things are quite different in current builds. The traditional OpenOffice.org Quality Assurance process had two features that I'd like to keep: 1) Full QA is run on what we release. We need to ensure that OpenOffice works now, not that previous builds worked. Full to me means a test matrix on all platforms, all functions, all languages, and includes performance, interop and other tests. A full test pass probably takes 4-6 weeks if we do it right. I don't think we have the opportunity to do more than one full test pass per release. But this should happen much earlier, right after a feature freeze when all features targeted for a release have been checked in. 2) The community at large is involved in testing, in a specific short period just before the release. If Release Candidate has a different meaning now, call it the Pre-Release Phase, that will end with a Release Candidate we can vote on rather confidently. We can't rely on volunteers doing QA on a regular basis, but we can very easily find volunteers willing to stress-test a nearly final version in a well defined timeframe (1-2 weeks). I would keep this as a basis for discussing how to structure QA activities in future (after 3.4), if others agree. This is only a fraction of the picture. We also need to consider what changes are allowed, and how those changes are reviewed, as we progress toward a Release Candidate. That is a critical part of how we maintain quality. If committers are checking in whatever they feel like, up to the day we have a RC, then we will have serious quality problems. When we changing code we invalidate prior testing. Not in an absolute sense, for most code changes there is an test impact, and some portion of the full test needs to be redone, to verify both the fix, but also that we did not break anything else. Again, this requires discipline from committers. Last minute feature work, or even undisciplined bug fixing, can cause more problems than it fixes. So I think we want the trunk to be fee CTR up to a certain point (feature freeze?) and after that point only changes that are accompanied by a BZ issue are allowed, but still CTR. Then at some point, after the main test pass is complete, we start prioritizing bugs as we create snapshot builds. We gradually raise the level required for bugs to be accepted. P3, P2, P1. In the final weeks we fix only marked release blockers. Once we have the RC build we branch the code. At that point (and maybe even earlier) we switch to RTC, e.g., no changes are allowed unless they are reviewed first. Was anything like this done before with OOo? Yes, more or less in a similar way and I think it is a good way to achieve a high quality product. And on the other hand we should try to provide as much as possible automated testing. And volunteers should do manual testing not only RC candidates. A continuous testing on dev snapshots would be very much appreciated if possible. Juergen -Rob Regards, Andrea.
