-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/21/2012 11:20 AM, Scott Kitterman wrote: > On Thursday, June 21, 2012 11:09:54 AM Michael Casadevall wrote: >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> >> On 06/21/2012 09:43 AM, Jono Bacon wrote: >>> On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall >>> >>> <mcasadev...@ubuntu.com> wrote: >>>>>>> If we had more resources we would love to provide help >>>>>>> for the flavors, and we are certainly happy to offer >>>>>>> any guidance and advice, with with our current >>>>>>> resources and staffing, Nick doesn't have the bandwidth >>>>>>> to handle more the than the Ubuntu ISOs and associated >>>>>>> testing. Saying that, I know Nick is in contact with >>>>>>> many of the flavors to ensure they get the support they >>>>>>> need to set up their own comprehensive testing plans. >>>> >>>> The first I heard of any of this was when this email was sent >>>> to u-devel. To be frank, changes of this magnitude should >>>> have been discussed in seasons at UDS-Q, and not in hallway >>>> conversations where many of the affected parties were not >>>> present. >>> >>> What changes of magnitude? We had already made it pretty clear >>> that my team is not focused on flavors (historically we have >>> never focused on flavors in our committed work items and always >>> on Ubuntu), and all I am outlining here is the cadence in how >>> Nick we be conducting his work. This is purely a way in which I >>> choose to manage my resources to serve Ubuntu best. >>> >>> Yes, these plans were not discussed at UDS, because they have >>> evolved since UDS, but Nick quite clearly laid out his testing >>> strategy at UDS (which is not to dissimilar from this...he had >>> scheduled testing plans for milestones and interim dailies). >> >> It has already been established by Rick that we can increase >> Canonical QA efforts without changing the milestone process. I'm >> embedding Rick's proposal here: >> >> On 06/17/2012 11:36 PM, Rick Spencer wrote: >>> (I changed the subject to better represent this branch of the >> >> conversation) >> >>> This discussion suggests that we don't need to release special >>> alpha and beta ISOs, but we do need: 1. A cadence of testing 2. >>> A trial run (or 2) of spinning ISOs 3. Development targets >>> >>> Therefore, I propose we: 1. Stop with the alphas and betas and >>> win back all of the development effort 2. *Increase* the >>> cadence of ISO testing to whatever we want or whatever the >>> community team can manage 3. Spin a trial ISO near what is not >>> beta time (maybe around current kernel freeze?) 4. Spin ISOs >>> for release candidates 5. Maintain the current Alpha and Beta >>> designations as development targets only (i.e. don't spin a >>> special image for them). >>> >>> Cheers, Rick >> >> The justification for this as I understand it is that the main >> Ubuntu image is moving to more consistent and constant QA >> practice, thus when combined with the active use of -proposed >> during freezes, and other tools would render the freezes and >> associates images moot. >> >> We've already established that we can increase QA frequency >> without changing the freeze/milestone process, and once the >> -proposed queue is fully functional, no development effort will >> be lost for those who do not wish to partake in testing; its >> simply a matter then of enabling -proposed on ones machine >> >>>> Many images such of Kubuntu have worked to have the various >>>> milestones and deadlines set in ways that allow them to >>>> incorperate the correct versions of their upstream packages >>>> (i.e. KDE 4.9 release dates are more or less timed that the >>>> RC and final will be available for Alpha 3 and Beta 1 >>>> respectively). >>>> >>>> Personal opinions aside, I object to large-scale changes in >>>> release planning after everyone already agreed to the >>>> current Quantal release schedule. >>> >>> The Kubuntu team are welcome to determine whatever milestones >>> they want - no one is suggesting anything needs to change for >>> the flavors in their testing cadence. I am purely stating how >>> Nick will be working: the only output of this work is assured >>> mandatory test case testing and this testing uncovering any >>> bugs. As I said earlier, this work is independent of whether we >>> remove milestones or not. >> >> The Kubuntu team wants to have standard alpha and beta images >> (i.e. the system that is in place and currently works for them). >> >> As a member of the release team, and someone who has seen the >> current system of freeze/milestone release catch image breaking >> bugs, I want the official desktop/server images to be >> cross-checked by the community by the same process that everyone >> else uses. >> >>>>> I don't think it is unreasonable for Canonical to focus >>>>> its resources on Ubuntu as opposed to the flavors. >>>> >>>> To what extent? >>> >>> To the extent that Canonical provides investment in Ubuntu, as >>> part of this investment is paying Nick's salary, and as Nick's >>> manager I choose how we resource his time and efforts. How we >>> resource Nick is not a community decision. >> >> I have no argument that Canonical directs Nick's work in whatever >> way we see fit. However, as someone paid by Canonical to ensure >> the high quality of our ARM images, I have a vested interest in >> knowing if I'm covered these efforts. >> >>>> As it stands, what is proposed will not only hinder but >>>> likely cost considerable QA coverage of the many images that >>>> are "community-supported". It is clear that non-Canonical >>>> backed images simply can not support a rolling-QA testing >>>> plan, and depend on the milestones system. >>>> >>>> By changing how a minority of how images are being tested, >>>> it will only serve to create confusion, and complications. As >>>> a release manager, am I now to track down each team, make >>>> sure from them that they've met whatever QA schedule they've >>>> set for themselves, and then release off that? >>>> >>>> As it stands, the vast majority of image testing comes at >>>> milestones, and often picks up people who are otherwise >>>> uninvolved in a flavors development. There has often been >>>> calls to ask for additional testers for X image in >>>> #ubuntu-testing should coverage be lacking, which have been >>>> always answered. >>> >>> As I have said a few times now...If a flavor can't maintain >>> the same level of testing, the flavor can just do the testing >>> it can cope with. This might mean just doing the testing with >>> the current cadence of milestones: just because I am ramping up >>> our manual testing efforts with Nick does not mean anyone else >>> has to change their own testing cadence. >> >> As you stated later, ARM is not directly apart of these efforts >> as of this moment. As such, the ARM images must remain on the >> current freeze/release process. >> >>> Flavors are in charge of their own destiny, and this includes >>> testing cadence. >> >> I believe the practical upshot of this is given that >> community-flavors do not wish to abandon the system that is >> simply easer to remove Ubuntu ISOs off the tracker (or otherwise >> marked as non-essential/low priority) at milestone time and allow >> everyone else to carry on as is. >> >> Any community image that wishes to follow in Canonical's stead >> is welcome to bring that up with the release manager, and after >> making sure a proper process of QA reporting is ensured will then >> be excused from the standard ISO tracking (and will not take part >> of any image milestones except final). >> >> To make sure that we don't release a sub-par product, all >> flavors should undergo full testing at the end of the cycle, with >> the understanding that should they not meet release standards, >> they can and will be dropped. >> >>>>> To be clear, the testing cadence is up to whatever the >>>>> flavor wants it to be; I am not suggesting we impose this >>>>> two-week cadence on anyone other than me imposing it on >>>>> Nick. :-) >>>> >>>> Then why change the existing system? >>>> >>>> Nothing about the existing system will prevent the Canonical >>>> QA efforts from implementing this. >>> >>> You are conflating two different things: testing cadence and >>> milestones. To be clear: the cadence of Nick's testing has >>> nothing to do with milestones and whether we choose to have >>> them or not have them. My only point here is that I am >>> disconnecting Nick's cadence from milestones so that if we do >>> choose to not have them in the future, nothing changes. >> >> No, I'm not. The former is being used to justify the removal of >> the later, and the current system of alpha/betas. >> >>>>> Naturally our flavors generally don't pay people to >>>>> coordinate this kind of testing (maybe Blue Systems could >>>>> invest here for Kubuntu?), but there is no requirement for >>>>> the flavors to test every two weeks. If you folks want to >>>>> test once a month or once every six weeks, then go ahead >>>>> and do that. Importantly though, as with all flavors, you >>>>> will need to coordinate this work yourselves within your >>>>> community. >>>> >>>> How does ARM factor into this? >>>> >>>> I have already cited the failure by Canonical QA to test the >>>> armhf+omap4 image for precise. Were it not for manual >>>> intervention, and the normal Ubuntu QA testing by both myself >>>> and other members of the community, I'm certain that we would >>>> have shipped an image that was not only substandard, but >>>> completely broken. >>>> >>>> It should be cited for completeness that this was caught at >>>> *Beta 2*, and only because there were voiced concerns on >>>> image quality that I sat and tested it extensively. >>>> >>>> Where were the QA efforts by Canonical QA for all the >>>> previous milestones? As it stands, I'm only aware of one >>>> person that was responsible for all ARM images that were >>>> supported during precise. >>>> >>>> Furthermore, speaking as the ARM Server Tech Lead in >>>> Canonical, the PES team has to manually QA our own images >>>> due to the fact that in general, we are the only teams with >>>> access to the hardware. We do NOT have the bandwidth required >>>> for this additional testing initiative. >>> >>> From my teams perspective, we were never asked to test ARM >>> (and thus left it our of Nick's targeting), although this is >>> something that we are going to be helping with moving forward. >> >> Good to know. >> >>>>> I see the milestones as quite disconnected from the >>>>> testing: the milestones are a useful means of consolidating >>>>> efforts around *something*, such as targeting work items >>>>> and deadlines; it is just that we happen to use these >>>>> milestones as a means to release. I personally don't see >>>>> the point of the alpha releases...our users don't use them, >>>>> and our developers are running the daily development branch >>>>> anyway, so to me it makes sense to get rid of the >>>>> milestones at some point. >>>> >>>> I use milestones extremely regularly. Both to gauge our goals >>>> by comparing what on the image is implemented to what is left >>>> and a common place to gauge any bug fixes between milestones >>>> to make sure we've fixed realistic issues between them. In >>>> addition, it provides a time where a considerable part of the >>>> developer community comes together to provide sanity checks >>>> on images. >>> >>> As has been said before, no-one is suggesting milestones for >>> targeting work are removed.... >> >> The confusion on the term milestone was already cleared up. Just >> for the record, when I say milestone, I consider it milestone >> images (mostly because that's when we confirm during QA testing >> that everything slated for that milestone went in, and wasn't >> accidently mismarked as completed or such). >> >>>> In addition, as a rule (and excluding precise), during >>>> development cycles, I generally only dist-upgrade my laptop >>>> during freezes from milestone to milestone until after >>>> Debian Import Freeze due to archive breakage and other >>>> issues. >>> >>> ...and here lies the point: you should be able to upgrade >>> *anytime*; not just at certain points. This is the gist behind >>> Rick's points about assuring stability in our development >>> release. >> >> True, this reason for the milestones existing right now will be >> resolved. That being said, it is only one part of the larger >> picture, and many that do run the development releases upgrade on >> a much more frequent timeframe. >> >>>> Precise was considerably less broken during development due >>>> to the simple fact we were syncing from Debian testing vs. >>>> unstable, which ensured a minimum of package breaking due to >>>> Debian's strict policies on how packages enter the testing >>>> archive. >>> >>> That is one of the reasons why Precise was less broken, but >>> there was also automated testing, acceptance criteria, >>> increased manual testing, and a stronger focus on QA. >> >> I'm awaiting an explanation on how a Canonical supported image >> then was nearly released in a broken state at Beta 2. >> >> One of the biggest hassles of running a development release is >> packages going into an uninstallable state or otherwise skewing. >> Syncing from testing solved this issue since our upstream for >> the period was always in a consistent state and thus we did not >> have issues such as transitions to worry about. >> >> Once our equivalent of these tools are in place, then yes, we >> will have a development release that can be upgraded both safely >> and at will. >> >>>>> My goal in this cycle is to ensure we have a regular >>>>> testing cadence for Ubuntu and not based on milestones; if >>>>> the Kubuntu team want to have your own internal milestones >>>>> for targeting work and testing, I see no reason why you >>>>> can't do that on your own schedule. >>>> >>>> I can see a very good one. We are forsaking consistency >>>> across all flavors for Canonical's interests. This even >>>> affects the Ubuntu Desktop and Server images on >>>> "community-supported" architectures as there is no one who >>>> will be doing this work for armel or powerpc. >>> >>> This is not about "Canonicals' interests" this is about >>> building efficiency in how we build Ubuntu; this efficiency >>> benefits not just Canonical staff, but our wider community of >>> participants who can assure a more stable development release. >>> >>> Nothing is being taken away here: flavors are still free to >>> coordinate their testing as they see fit. >>> >>> Jono >> >> Scott has made it clear that if the current system of alpha/betas >> is abolished, they will not be able to manage to maintain their >> current level of quality. Kubuntu (and by extension, the other >> flavors) depend on the current system as is. >> >> As ARM and PowerPC is not actively part of these new QA efforts, >> it is clear that we are also dependent on the current system to >> keep quality. >> >> It should be noted though that as long as one architecture in an >> image can't proscribe to the new testing method, the packageset >> relating to that image in the release pocket must be frozen as we >> have no way of freezing on a per-architecture basis. > > +1. > > If Ubuntu wants to skip the Alpha/Beta milestones because Canonical > thinks it's other QA efforts are sufficient, I think they should > feel free to do it, but the existing structure needs to stay in > place for the rest of us. > > Scott K >
I think the basic process should now be something like this: 1. Images can opt-out of the existing QA mechanisms once they understand the full risks involved of dong so. 2. Images that have out-opted are free to ignore freezes for their respective package sets (feature freeze and such still apply of course). As a thought, while the packageset in release must be frozen if one of the architectures used by that image do not opt-out, we can simply enabled proposed on the dailies to allow unbroken development efforts. 3. For pre-release freezes, all images will be re-added to the QA tracker, and undergo normal QA processes 4. As per normal, any image that fails will be excluded from the release (or 12.10 in this case). Michael -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJP42e+AAoJEHM+GkLSJHY5pkIQAJFYmxbA4rd7NYI6YIi7M5DR 9Moj+N4Mnp6LIwPdZa6lqXNqqgTAkrMZ0fp4TNrl1RaWEFz9/hExa2Lmq/5T8SHe 1naz/1Un3mxQhI1i6xoLdKqVrjy3JCCa5u6el/S769t6J1qgwsOeOQMBQnpkOzzR LpTw48CiS0tH5HWCJQC535+bquU3md77ttdkhsdz/zoYm+MgW7/q+LjbiFvglvEb gE8jxnJSvdR4U79bjw+otrE8wHMMvKezGJ5KIYxI3n5KuHWkOioOq/pouXdkUiXh zpIw7GT5n/opQNH0Z7vm1Iagj255Wbkiv2/+AM31wc4m+cD5WHZZmY+fyP0PTCKT i02wK3s0PI3fAOrLaRrYT/VY9vxRoW+/WSuNAkYjPsPywVEj0dqQOfa0UPZ1EQXM xCI2q6RlJky5i1EFoyEurl1TIcm+fdHJY+mqfEAnBZ1HAZwo8I4qguZ6NmwziWz3 HZTRcccSZ5uL3WoyYZ8PXzAAimO7hHjnQRHE9b5hXJjd4L98RE2sFtj4repCct25 MExgzhhUBY3J8q8vZCGADY0Z39F+/0es4gyzzop4qVOTKRFWHOcqwfkO551I8oK5 8xRE8M7sOuTsCg6ncfD0uAOFplIcnKGjHmhfCDZqP2LKJuujPYLeAzZY1hlg4cpQ Co/Fo9suN3KKsES+WGXl =7ubS -----END PGP SIGNATURE----- -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel