On Tuesday, June 19, 2012 11:06:14 AM Michael Casadevall wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Mon, Jun 18, 2012 at 11:33 PM, Rick Spencer > > <rick.spen...@canonical.com> wrote: > > On Tue, Jun 19, 2012 at 12:15 AM, Kate Stewart > > > > <kate.stew...@canonical.com> wrote: > >> For Alpha1, we did 2 respin sets after the first set was built, > >> based on what the manual testing was finding and trying to get a > >> set of ARM desktop images. (Note: We did not have quantal arm > >> desktop images until the week of alpha 1, and then didn't have > >> them again with the dailies between 6/10-6/14). Having > >> milestones does > > force > > >> a focus on the full set of images. Daily images and the > >> automated testing are still mostly focusing on unit tests for the > >> x86 desktop and server images in virtualized hardware, and as > >> Martin says, the manual testing is still finding issues on the > >> real hardware that are causing respins. > > > > I believe there is widespread agreement on this thread that manual > > testing is good and necessary. I also think there is agreement that > > a faster cadence of complete manual testing than is accommodated by > > our current milestones would be desirable. I think it's fair to > > say that we can move ahead with increasing the frequency of manual > > testing with or without changes to our milestones. I will look to > > the Ubuntu Community team to begin with this, as they don't believe > > they are blocked by any other decisions to be made. > > > > I think the question on the table is, shall we drop most > > milestones altogether, or adopt a system such as Thierry suggests, > > where we use the most recent "good" daily as the milestone image? > > I have serious concerns with removing the milestones. As it > stands, several images, including the vast majority of the ARM images, > only get extensively tested at milestones due to the limited userbase > of the image (specifically, highbank and armadaxp as of right now is > limited to a handful of individuals in the world at the moment). > > Many critical issues with ARM (and to a lesser extent x86) > have only been found during milestone testing. Without a set of > defined and organized images for testing, more obscure parts of the > installer simply do not get tested; for instance, how many people are > going to test all possible server configurations or test the installer > with no network. > > These scenarios are not common for development, but can and do occur > regularly for many users who install Ubuntu for the first time. During > 12.04 development, during milestone testing, three bugs* relating to > both usecases were found to cause the installer to silently fail > midway through installation leaving the user with only a partially > configured system. > > Each milestone represents an opportunity for end-users and QA to test > our images in something more resembling a production environment, and > to test use-cases and recipes that may normally not see a lot of > coverage unless one is explicatively checking for edge cases. > > Milestones exist to give the Ubuntu developer community to step back, > and check to make sure nothing important has broken, and to gauge our > progress through a cycle. In addition, they provide a dedicated time > where as a community we step forth and check our images to ensure no > regressions have slipped by. > > If we remove the milestones, the only period of extensively and review > the images will be just before release. As such, any regressions that > are found would require a scrambled fix during a period that minimal > archive changes are desired and would both be costly in terms of > development effort, and risky as each final freeze upload always > carries the inherent chance of hosing something important. > > Unless the final intent is to ultimately abolish releases all > together and move to a rolling-release model, I don't believe we, > as a community, could successfully ship Ubuntu with its excellent > state of quality assurance without the cycles of alpha and beta images. > > * - Relevant bug reports: > https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/985737 > https://bugs.launchpad.net/livecd-rootfs/+bug/985258 > https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/985280
+1 I have to confess that when I threw out the idea of just abolishing the milestones way back in this thread I thought it was a sufficiently ridiculous idea that it would give people pause about dropping the freezes. People worried about velocity through a freeze can publish stuff in a PPA and ask them to test it during a freeze. I think this entire notion is going to add significant risk to the development cycle. Michael is right on target. Without a dedicated focus on human testing of various components things are going to be missed until the end game when broad user testing starts. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel