On 2007-10-09T16:26:11, Alan Robertson <[EMAIL PROTECTED]> wrote:
Hi Alan,
thanks for providing this release plan. I like the schedule, and I hope
that going forward, the release plan can be condensed down a bit to a
few more clear rules ;-)
I'd offer the following feedback:
First, who will make up the test team?
> Key aspects of this procedure
>
> * Good communication
> * Testing will be done using CTS, and a test plan of manual tests, and
> ad hoc testing by the community
The test plan of manual tests probably should be replaced by the "ad
hoc" tests; any test process which actually follows a plan should be in
CTS or some other regression tests. Or at least, preferably so; the test
group should collect the most effective (ie: break something) manual
test cases so we can incorporate them going forward.
> Proposed Release Testing Procedure
>
> During the testing of a release, will follow a plan ??? which they will
> announce in advance. The test team is expected to announce any deviations
> from the plan to the community as they go along.
>
> I propose dividing the testing interval into two phases ??? initial and
> final testing intervals.
>
> Initial testing interval
>
> * Pull 'dev' into test
How long will the test intervals be, and is development supposed to
continue during that interval ...?
> o Developers who want bug fixes to be included in the next
> release will email the test team to pick them up for the next release. It
> is expected that fixes be given to the test team in the 'dev' tree.
This will happen on the linux-ha-dev list, right?
> Requests to have bug fixes included will be given explicitly by email, as
> implicit schemes based on descriptions, etc. are unreliable. The test team
> will in turn reply to such emails acknowledging that they will pick up the
> fix.
> o Each tester should run about one day's worth of testing each
> day and then pick up the next day's version, incorporate it and test it.
This seems to make being on the test team essentially a full-time job.
I'd be somewhat careful with that, and be rather happy if everyone
contributes as much testing as they can.
> o Package builds for 'test' are made available for major
> platforms (using Andrew's build process on OpenSUSE?)
Certainly an option; I build daily from /dev, I can easily offer a
repository which builds daily from test.
> * Repeat for a few weeks (2-4?). More for releases with major features,
> and fewer for releases which are contain only bug fix and minor features.
Ah, that answers part of my question above.
> Final Testing Interval
>
> * During final testing interval, bug fixes are restricted to only
> include regressions, and fixes that affect data integrity. The decision on
> including a fix will be made by a consensus between the test team, the
> developer and the community. In the unlikely even that they cannot come to
> a consensus, we'll have to figure out how to handle it if it comes up.
> * Each new update pushed to 'test' during final testing interval will
> be announced to the -dev mailing list. Of course, the usual RSS feeds for
> Hg will be available. However, since communication has been one of our
> failures in the past, we will communicate by both RSS and email ??? so that
> no one is missed.
> * Any non-trivial fix will cause the final testing interval to restart.
> During this final stage, developers must state if a bug fix is trivial or
> not based on their knowledge. If the test team disagrees with respect to
> their place in the testing procedure and their knowledge of the fix, they
> will dialog with the developers involved to reach a consensus.
How long will the final testing interval be? Another week? And I'm not
sure the separation is necessary; a two to three weeks stabilization
phase should be sufficient, w/o the distinction.
> Activities:
>
> * Developers and test team and volunteers test 'test' base using CTS
> * Test team and volunteers execute test plan against 'test' version
> * Test team and volunteers report bugs to community Bugzilla
> * Final testing interval shall include complete manual tests from test
> plan, and at least 5000 iterations of the CTS suite spread across at least
> two architectures, preferably more.
These are on-going activities anyway (and happen on dev as well), but it
is good to document them.
> Responsibilities
>
> This section outlines the responsibilities of each of the set of
> participating stakeholders in the Linux-HA project during the official test
> intervals.
>
> Developer Responsibilities
>
> * Communicate status, progress, and problems with other stakeholders
> using the linux-ha-dev email mailing list, and when appropriate, private
> emails, conference calls, and other phone calls.
> * provide timely bug fixes for bugs assigned to you to the 'dev'
> workspace
> * avoid committing major restructuring and similar high-impact things
> to 'dev' during the entire testing interval.
OK, this means the test interval shouldn't be too long.
> * Build RPMs using make rpm and run BasicSanityCheck before pushing any
> fixes to 'dev' during test interval
> * Run CTS tests during test intervals as specified
> * Run manual tests ??? from test plan and/or ad hoc
> * File bug reports for discovered bugs ??? making sure bugs get
> assigned appropriately
> * When creating bug fixes during testing period, email the -dev list
> asking for your fix(es) to be incorporated.
> * Verify that bugs you submitted are fixed after they've been
> incorporated into 'test' workspace.
I'd suggest that some of those points - those relating to the reporting
and verification of bugs - are moved over to the test team group, as the
division between test team, other testers, and developers is not going
to be as clear cut; however they act in several clear roles, so I'd not
list redundant activities for the roles. (If that sentence made any
sense!)
> Test team responsibilities
>
> * Communicate status, progress, and problems with other stakeholders
> using the linux-ha-dev email mailing list, and when appropriate, private
> emails. Status communication regarding high-priority bugs, missing fixes,
> fixed bugs, etc. is especially important during the test interval.
> * Publish overall schedule for the stages of the testing progress based
> on Alan's draft. Review, correct and update this schedule. After the first
> release, the test team will create future schedules.
> * Develop and publish a manual test plan on the wiki site for testing,
> participating in discussions with, and incorporating feedback from the
> community. This test plan must include limited testing for R1-style
> clusters, and also for upgrades as well as initial (clean) installs.
> * Manage the 'test' workspace
> * Incorporate bug fixes to the 'test' workspace according to the rules
> for the different phases of the test interval
> * Carry out test plan ??? both manual and automated testing. Make sure
> that someone is doing at least limited r1-style testing. If you don't do
> it, find someone who will do it for you. 500 iterations during end of
> initial test cycle, and 500 during final test cycle is sufficient.
The R1 testing is already a test plan item, and I don't think belongs
here. Of course the test team needs to make sure all major functionality
is covered by the test plan.
> * Check with other community members to ensure that we are covering
> these platforms at least minimally during testing:
> o x86 linux
> o x86_64 linux
> o ppc linux
> o s390x linux
> o x86 FreeBSD
> o x86 OpenBSD
> o x86 solaris
Same.
> * Create bug reports for bugs discovered while testing. Make sure bugs
> are assigned to an appropriate person.
I guess my point from above applies here: this is a generic tester
responsibility and should be listed with that role.
> * When release is completed, tag released version
> * After tagging release, push to the stable repository
> * Tell Alan to publish release and digitally sign tar ball, packages,
> etc.
>
> Community responsibilities
>
> Communicate status, progress, and problems with other stakeholders using
> the linux-ha-dev email mailing list, and when appropriate, private emails,
> conference calls, and other phone calls.
>
> * Evaluate and comment on test plan
> * Perform ad-hoc testing during entire test cycle
> * Perform BasicSanityCheck testing during test cycle
> * Offer opinions as appropriate on the severity of bugs and the
> relative importance of bug fixes in your environment and experience
> * Perform CTS testing (if possible) during the testing cycle
> * Make sure your OS and/or distribution works well with the new release
> * File good, detailed bug reports as appropriate. Dejan Muhamadagic has
> a tool which helps with this.
Generic tester responsibilities also include providing timely feedback
to developers inquiring wrt bugs filed. It can be a real pain otherwise.
Again, I'd suggest to split this along the roles of tester, developer,
release coordinator; people can act in several roles. This will make the
document shorter. (I'll not repeat that point below now to stop sounding
like a broken record ;-)
> SuSE/Novell folks
>
> In addition to your responsibilities as developers and/or community
> members, someone needs to make sure these things get done.
>
> * Communicate status, progress, and problems with other stakeholders
> using the linux-ha-dev email mailing list, and when appropriate, private
> emails, conference calls, and other phone calls.
> * During initial testing interval: produce nightly builds from the
> 'test' image
> * During final testing interval: produce builds corresponding to what
> the test team is testing (might be nightly builds)
Nitpick question, but in a global context: define "nightly" - will 00:00
GMT do?
> Open Issues
>
> * How will release notes and change log be created, and who will do it?
Every developer is responsible for providing a concise changelog entry
as part of their commit; the one-line summary should be meaningful to
_users_, testers, and admins (bugzilla number if it exists, and impact
on them); the rest of the message can be meaningful to developers.
> Testing outside Release Intervals
>
> We need to automate the following things against the 'test' tree daily to
> make the final test interval go smoothly.
Outside the release intervals, I'd suggest that people test dev.
Otherwise, test is going to be refreshed daily from dev, which sort of
doesn't make a lot of sense - it would be a mirror of dev. Instead, test
will be used (I guess) for preparing the "quick" releases which we need
to push because of the typos which have slipped in the last release ;-)
> *
>
> Nightly builds and nightly BasicSanityCheck and CTS runs with
> automatic emails to the mailing list on failure. This should include at
> least i586, x86_64, ppc, and s390x architectures.
> * Ideally we'd like to do something nightly for *BSD, Solaris, and OS/X
> ??? assuming we can get volunteers in the community to set up cron jobs for
> us
> * Scans by automated verification tools (Coverity (Stanford Checker)
> and/or BEAM)
Yes, all of this ought to be run against dev IMHO.
Regards,
Lars
--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
