Re: Releasing Alphas and Betas without freezing
On Wed, Jun 20, 2012 at 9:24 PM, Scott Kitterman ubu...@kitterman.com wrote: On Wednesday, June 20, 2012 09:13:17 PM Jono Bacon wrote: On Wed, Jun 20, 2012 at 8:52 PM, Scott Kitterman ubu...@kitterman.com wrote: For our current Ubuntu ISOs. Flavors currently are coordinating their own testing efforts. They could either latch into the two week cadence, or use their own cadence if desired. I find it somewhat unfortunate that the community testing efforts exclude the community sponsored flavors in the Ubuntu project. I would have hoped that the community team was not just about Canonical's products. This shouldn't be a particularly big surprise; Canonical supports our flavors with infrastructure, but we primarily focus our engineering and community team staff members on Ubuntu. If we had more resources we would love to provide help for the flavors, and we are certainly happy to offer any guidance and advice, with with our current resources and staffing, Nick doesn't have the bandwidth to handle more the than the Ubuntu ISOs and associated testing. Saying that, I know Nick is in contact with many of the flavors to ensure they get the support they need to set up their own comprehensive testing plans. Perhaps I misunderstood, but I thought that you were saying this was about the community team he had organized to support ISO testing. Nothing to do with Canonical resources. I think that such a team should not be focused on just Canonical products. As a Kubuntu developer trying to get Kubuntu images tested for milestones, I've often gotten a lot of help from Canonical people in Ubuntu QA. I appreciate that. That's not what I'm talking about. Well to be clear, Nick is one member of the Ubuntu community who is focused on testing. Other people are welcome to coordinate testing campaigns and get others interested and excited about testing, but Nick's focus is explicitly on the Ubuntu ISOs. Of course, if community members want to volunteer to help test the flavors then that is great. I don't think it is unreasonable for Canonical to focus its resources on Ubuntu as opposed to the flavors. None of the non-Canonical flavors have the resources to pick up the pace of ISO testing. If Canonical is insistent on reorganizing the development process in a way that works better for them (dropping milestones and more frequent testing), you're going to leave us behind. Fundamentally the development and testing model has to be something that the entire project can support and I don't think it's unreasonable to expect that the community team person assigned to work on QA would be trying to build a team of community members to accomplish that. To be clear, the testing cadence is up to whatever the flavor wants it to be; I am not suggesting we impose this two-week cadence on anyone other than me imposing it on Nick. :-) My mail earlier in this thread was merely making it clear that from my teams perspective, we are assigning resources on a two-week cadence. Now, admittedly, a big reason why we can do this is because we pay Nick to help coordinate this work. Naturally our flavors generally don't pay people to coordinate this kind of testing (maybe Blue Systems could invest here for Kubuntu?), but there is no requirement for the flavors to test every two weeks. If you folks want to test once a month or once every six weeks, then go ahead and do that. Importantly though, as with all flavors, you will need to coordinate this work yourselves within your community. I see the milestones as quite disconnected from the testing: the milestones are a useful means of consolidating efforts around *something*, such as targeting work items and deadlines; it is just that we happen to use these milestones as a means to release. I personally don't see the point of the alpha releases...our users don't use them, and our developers are running the daily development branch anyway, so to me it makes sense to get rid of the milestones at some point. My goal in this cycle is to ensure we have a regular testing cadence for Ubuntu and not based on milestones; if the Kubuntu team want to have your own internal milestones for targeting work and testing, I see no reason why you can't do that on your own schedule. Jono -- Jono Bacon Ubuntu Community Manager www.ubuntu.com / www.jonobacon.org www.identi.ca/jonobacon www.twitter.com/jonobacon -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
Thought I would chime in with my thoughts on freezes, snapshot testing, and QA in general. Freezes: When I was testing the Arm images, there were times when I would go for weeks without an image. And if I remember correctly, there was a period during Natty where there were no images between milestones due to pool skew. And there were image specific changes being made at the time (jasper-initramfs, Ubiquity/oem-config, etc). When those changes can't be tested without an image, failures become critical at milestone, causing a lot of needless stress. Manual QA: Having been in a QA role for most of my career in computers (+25 years), I can site areas where automated QA can and will fail. Sure, repetitive tests can be automated, especially server stacks and background libraries. But how do you automate testing a GUI for usability (Evolution STILL has option windows that flow past the bottom of my netbook screen). Plus there are tests that find the oddball combination that most developers don't think users will select (enabling autologin encrypted home directory during install was found this way). When developing automation scripts for tests, they really should be designed to test on all platforms, not just simulated/virtualized environments. Otherwise others have to constantly reinvent the wheel to run the same test on systems that don't support that test environment. Snapshot retention: I set up my own internal mirror so that I could keep daily images between milestones. This way if I missed something during daily testing and we needed to determine how far back it went, I had a complete image history to go back to (Mono/Banshee in Oneiric). This also helped to determine if one package or a combination of package updates caused issues. As to testing 15 arm images during milestones, my secret was to take a daily image for one platform (panda) and focus on it over a period of 2-3 days, testing everything. Most bugs in applications there were the same on all arm platforms. That left hardware specific testing on other platforms to be done only when testing kernel/xorg updates, or (in the case of Mono not supporting SMP on arm) reproducing a bug found on the main test environment. I would also try to reproduce the bug on x86/amd64 to determine if the bug was arch specific. Often times, an application bug found on the panda would also be found on other architectures. While I no longer have the time to test Ubuntu images, I am saddened by the failures I do see on a daily basis on the LTS release (anyone try to run a remote desktop lately using rdesktop?). These issues should have been found prior to release. -- -- Tobin Davis The cutting edge is getting rather dull. -- Andy Purshottam -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Thu, 2012-06-21 at 10:11 +0100, Didier Roche wrote: This cycle, the next step for Unity and all related components is that each release is potentially the last one that is uploaded to ubuntu until the finale release. So, each version is a possible release candidate for Quantal. I do not want anymore to see half-backed unity features coming in. This is only possible now thanks to the huge quality increase we had last cycle and that we finally reached the feature level that we can expect from a UI. So, all extras should be precise, polished, reliable before getting into ubuntu. +1 :) So, removing the milestone freeze is completely aligned with that vision. Challenge is that we don't have a good schedule of when the Unity drops are going to happen and what features are emerging when. The other applications and products that work on top of Unity need to interface with it need to get synchronized in some way, so that effective system testing can occur. Having predictable times when we plan to release an image is a forcing function, in that they at least keep us focused on making sure we have system testing going on and that the community flavors, that are part of the Ubuntu project, can system test their emerging bits on the evolving infrastructure with some degree of confidence. Additional system testing and snapshotting of images at more than just the currently scheduled points would, of course, be very welcome and help with improving the quality, especially early in the 6 month cycle. ;) Question 3: shall we increase the rate of manual testing? This question also arose in the thread. I think there is widespread consensus that we should do this, and it is not actually related to the other questions. Community Team, is it feasible to increase the rate of full manual testing runs to every 2 weeks or similar? It was a hard job to keep regular contributors (reporting high quality results) tight redoing serious testing every 2 weeks for unity releases, but I'm completely confident Nick can do this job. :) +1 :) Big challenge for him and the QA team will be when the 12.04.1 testing has to happen in parallel with the quantal development testing. Question 4: shall we keep snapshots of the development release so that we can bisect more easily and find when bugs were introduced? This question also arose, and also is not tied to the other questions. QA Team, is it feasible to keep a set of snap shots somewhere for this purpose? That would really be awesome, especially if the reporting QA tools get better and we can run an older iso under a vm in a minute to just test something quickly :) Am trying to think about what makes sense to keep around, and across which products, and for how long, but if we can secure the disk space for this, agree it would be useful to the developers and testers. Preliminary thoughts are: * Try to keep all image that we characterize (ie. run system tests on, get boot speed measurements, etc.) available for a *useful* period. However since we'll never have infinite disk space, ;) and they are less useful over time, possibly some sort of age out strategy like: * keep at least the last month's worth of these characterized images available (ideally it would be nice to have a set of weekly ones ;) ) * keep at least a monthly snapshots from each of the 6 month development period around for historical reference. * keep these for all images that will be in the release manifest. Seem reasonable? better ideas? Kate -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Wednesday, June 20, 2012 11:14:05 PM Jono Bacon wrote: My goal in this cycle is to ensure we have a regular testing cadence for Ubuntu and not based on milestones; if the Kubuntu team want to have your own internal milestones for targeting work and testing, I see no reason why you can't do that on your own schedule. No. We can't. It requires Canonical resources to do a release. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Thu, Jun 21, 2012 at 6:02 PM, Michael Casadevall mcasadev...@ubuntu.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/20/2012 11:14 PM, Jono Bacon wrote: On Wed, Jun 20, 2012 at 9:24 PM, Scott Kitterman ubu...@kitterman.com wrote: On Wednesday, June 20, 2012 09:13:17 PM Jono Bacon wrote: On Wed, Jun 20, 2012 at 8:52 PM, Scott Kitterman ubu...@kitterman.com wrote: For our current Ubuntu ISOs. Flavors currently are coordinating their own testing efforts. They could either latch into the two week cadence, or use their own cadence if desired. All flavors as it stands right now are actually coordinated to a common testing and QA cycle based around milestones, as set forth by the release manager and the manifest. At each milestone, all images, regardless of backing are tested. If they fail they're removed from the manifest, and are excluded from the release. I find it somewhat unfortunate that the community testing efforts exclude the community sponsored flavors in the Ubuntu project. I would have hoped that the community team was not just about Canonical's products. This shouldn't be a particularly big surprise; Canonical supports our flavors with infrastructure, but we primarily focus our engineering and community team staff members on Ubuntu. No one is objecting to additional QA efforts dedicated to Canonical images. That being said, I've yet to see a stated reason on why this additional QA drive requires changing the milestone process. As Scott clearly points out what is being proposed will change the QA/milestone/release process will cause a fair bit of large grief for all flavors. Nothing is being proposed by Jono other than saying they will strive to increase the testing cadence for Ubuntu, which as you state is not related to alphas and betas. Cheers, Rick -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall mcasadev...@ubuntu.com wrote: If we had more resources we would love to provide help for the flavors, and we are certainly happy to offer any guidance and advice, with with our current resources and staffing, Nick doesn't have the bandwidth to handle more the than the Ubuntu ISOs and associated testing. Saying that, I know Nick is in contact with many of the flavors to ensure they get the support they need to set up their own comprehensive testing plans. The first I heard of any of this was when this email was sent to u-devel. To be frank, changes of this magnitude should have been discussed in seasons at UDS-Q, and not in hallway conversations where many of the affected parties were not present. What changes of magnitude? We had already made it pretty clear that my team is not focused on flavors (historically we have never focused on flavors in our committed work items and always on Ubuntu), and all I am outlining here is the cadence in how Nick we be conducting his work. This is purely a way in which I choose to manage my resources to serve Ubuntu best. Yes, these plans were not discussed at UDS, because they have evolved since UDS, but Nick quite clearly laid out his testing strategy at UDS (which is not to dissimilar from this...he had scheduled testing plans for milestones and interim dailies). Many images such of Kubuntu have worked to have the various milestones and deadlines set in ways that allow them to incorperate the correct versions of their upstream packages (i.e. KDE 4.9 release dates are more or less timed that the RC and final will be available for Alpha 3 and Beta 1 respectively). Personal opinions aside, I object to large-scale changes in release planning after everyone already agreed to the current Quantal release schedule. The Kubuntu team are welcome to determine whatever milestones they want - no one is suggesting anything needs to change for the flavors in their testing cadence. I am purely stating how Nick will be working: the only output of this work is assured mandatory test case testing and this testing uncovering any bugs. As I said earlier, this work is independent of whether we remove milestones or not. I don't think it is unreasonable for Canonical to focus its resources on Ubuntu as opposed to the flavors. To what extent? To the extent that Canonical provides investment in Ubuntu, as part of this investment is paying Nick's salary, and as Nick's manager I choose how we resource his time and efforts. How we resource Nick is not a community decision. As it stands, what is proposed will not only hinder but likely cost considerable QA coverage of the many images that are community-supported. It is clear that non-Canonical backed images simply can not support a rolling-QA testing plan, and depend on the milestones system. By changing how a minority of how images are being tested, it will only serve to create confusion, and complications. As a release manager, am I now to track down each team, make sure from them that they've met whatever QA schedule they've set for themselves, and then release off that? As it stands, the vast majority of image testing comes at milestones, and often picks up people who are otherwise uninvolved in a flavors development. There has often been calls to ask for additional testers for X image in #ubuntu-testing should coverage be lacking, which have been always answered. As I have said a few times now...If a flavor can't maintain the same level of testing, the flavor can just do the testing it can cope with. This might mean just doing the testing with the current cadence of milestones: just because I am ramping up our manual testing efforts with Nick does not mean anyone else has to change their own testing cadence. Flavors are in charge of their own destiny, and this includes testing cadence. To be clear, the testing cadence is up to whatever the flavor wants it to be; I am not suggesting we impose this two-week cadence on anyone other than me imposing it on Nick. :-) Then why change the existing system? Nothing about the existing system will prevent the Canonical QA efforts from implementing this. You are conflating two different things: testing cadence and milestones. To be clear: the cadence of Nick's testing has nothing to do with milestones and whether we choose to have them or not have them. My only point here is that I am disconnecting Nick's cadence from milestones so that if we do choose to not have them in the future, nothing changes. Naturally our flavors generally don't pay people to coordinate this kind of testing (maybe Blue Systems could invest here for Kubuntu?), but there is no requirement for the flavors to test every two weeks. If you folks want to test once a month or once every six weeks, then go ahead and do that. Importantly though, as with all flavors, you will need to coordinate this work yourselves
Re: Releasing Alphas and Betas without freezing
On Thu, Jun 21, 2012 at 6:43 PM, Jono Bacon j...@ubuntu.com The Kubuntu team are welcome to determine whatever milestones they want - no one is suggesting anything needs to change for the flavors in their testing cadence. I am purely stating how Nick will be working: the only output of this work is assured mandatory test case testing and this testing uncovering any bugs. As I said earlier, this work is independent of whether we remove milestones or not. I think the terminology here is the source of some confusion. I think when you (Jono) say milestones people hear specifically the already defined alpha and beta Milestones. I don't think you mean that. I think you are using the term milestones differently, to just mean chosen dates for testing. I think it's fair to say that flavors can choose a higher frequency for a testing cadence than just the alpha and beta dates, Cheers, Rick -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Thu, Jun 21, 2012 at 9:52 AM, Rick Spencer rick.spen...@canonical.com wrote: On Thu, Jun 21, 2012 at 6:43 PM, Jono Bacon j...@ubuntu.com The Kubuntu team are welcome to determine whatever milestones they want - no one is suggesting anything needs to change for the flavors in their testing cadence. I am purely stating how Nick will be working: the only output of this work is assured mandatory test case testing and this testing uncovering any bugs. As I said earlier, this work is independent of whether we remove milestones or not. I think the terminology here is the source of some confusion. I think when you (Jono) say milestones people hear specifically the already defined alpha and beta Milestones. I don't think you mean that. I think you are using the term milestones differently, to just mean chosen dates for testing. I think it's fair to say that flavors can choose a higher frequency for a testing cadence than just the alpha and beta dates, Agreed: to be clear, when I say milestones I am referring to a point in time when something happens, and in the context of this discussion testing I mean the point in time when a testing run kicks off. Apologies for the confusion. Jono -- Jono Bacon Ubuntu Community Manager www.ubuntu.com / www.jonobacon.org www.identi.ca/jonobacon www.twitter.com/jonobacon -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Thursday, June 21, 2012 06:52:58 PM Rick Spencer wrote: On Thu, Jun 21, 2012 at 6:43 PM, Jono Bacon j...@ubuntu.com The Kubuntu team are welcome to determine whatever milestones they want - no one is suggesting anything needs to change for the flavors in their testing cadence. I am purely stating how Nick will be working: the only output of this work is assured mandatory test case testing and this testing uncovering any bugs. As I said earlier, this work is independent of whether we remove milestones or not. I think the terminology here is the source of some confusion. I think when you (Jono) say milestones people hear specifically the already defined alpha and beta Milestones. I don't think you mean that. I think you are using the term milestones differently, to just mean chosen dates for testing. I think it's fair to say that flavors can choose a higher frequency for a testing cadence than just the alpha and beta dates, Yes, but that's not what's being discussed. What's being discusses is more testing at shorter intervals in order to abolish the alphas and the betas. Once they are gone, then we don't have the milestones to organize around anymore. It takes Canonical employees to execute a release milestone. Doing an Alpha/Beta without Canonical support isn't currently something that can be done. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/21/2012 09:43 AM, Jono Bacon wrote: On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall mcasadev...@ubuntu.com wrote: If we had more resources we would love to provide help for the flavors, and we are certainly happy to offer any guidance and advice, with with our current resources and staffing, Nick doesn't have the bandwidth to handle more the than the Ubuntu ISOs and associated testing. Saying that, I know Nick is in contact with many of the flavors to ensure they get the support they need to set up their own comprehensive testing plans. The first I heard of any of this was when this email was sent to u-devel. To be frank, changes of this magnitude should have been discussed in seasons at UDS-Q, and not in hallway conversations where many of the affected parties were not present. What changes of magnitude? We had already made it pretty clear that my team is not focused on flavors (historically we have never focused on flavors in our committed work items and always on Ubuntu), and all I am outlining here is the cadence in how Nick we be conducting his work. This is purely a way in which I choose to manage my resources to serve Ubuntu best. Yes, these plans were not discussed at UDS, because they have evolved since UDS, but Nick quite clearly laid out his testing strategy at UDS (which is not to dissimilar from this...he had scheduled testing plans for milestones and interim dailies). It has already been established by Rick that we can increase Canonical QA efforts without changing the milestone process. I'm embedding Rick's proposal here: On 06/17/2012 11:36 PM, Rick Spencer wrote: (I changed the subject to better represent this branch of the conversation) This discussion suggests that we don't need to release special alpha and beta ISOs, but we do need: 1. A cadence of testing 2. A trial run (or 2) of spinning ISOs 3. Development targets Therefore, I propose we: 1. Stop with the alphas and betas and win back all of the development effort 2. *Increase* the cadence of ISO testing to whatever we want or whatever the community team can manage 3. Spin a trial ISO near what is not beta time (maybe around current kernel freeze?) 4. Spin ISOs for release candidates 5. Maintain the current Alpha and Beta designations as development targets only (i.e. don't spin a special image for them). Cheers, Rick The justification for this as I understand it is that the main Ubuntu image is moving to more consistent and constant QA practice, thus when combined with the active use of -proposed during freezes, and other tools would render the freezes and associates images moot. We've already established that we can increase QA frequency without changing the freeze/milestone process, and once the -proposed queue is fully functional, no development effort will be lost for those who do not wish to partake in testing; its simply a matter then of enabling -proposed on ones machine Many images such of Kubuntu have worked to have the various milestones and deadlines set in ways that allow them to incorperate the correct versions of their upstream packages (i.e. KDE 4.9 release dates are more or less timed that the RC and final will be available for Alpha 3 and Beta 1 respectively). Personal opinions aside, I object to large-scale changes in release planning after everyone already agreed to the current Quantal release schedule. The Kubuntu team are welcome to determine whatever milestones they want - no one is suggesting anything needs to change for the flavors in their testing cadence. I am purely stating how Nick will be working: the only output of this work is assured mandatory test case testing and this testing uncovering any bugs. As I said earlier, this work is independent of whether we remove milestones or not. The Kubuntu team wants to have standard alpha and beta images (i.e. the system that is in place and currently works for them). As a member of the release team, and someone who has seen the current system of freeze/milestone release catch image breaking bugs, I want the official desktop/server images to be cross-checked by the community by the same process that everyone else uses. I don't think it is unreasonable for Canonical to focus its resources on Ubuntu as opposed to the flavors. To what extent? To the extent that Canonical provides investment in Ubuntu, as part of this investment is paying Nick's salary, and as Nick's manager I choose how we resource his time and efforts. How we resource Nick is not a community decision. I have no argument that Canonical directs Nick's work in whatever way we see fit. However, as someone paid by Canonical to ensure the high quality of our ARM images, I have a vested interest in knowing if I'm covered these efforts. As it stands, what is proposed will not only hinder but likely cost considerable QA
Re: Releasing Alphas and Betas without freezing
On Wednesday, June 20, 2012 11:14:05 PM Jono Bacon wrote: Well to be clear, Nick is one member of the Ubuntu community who is focused on testing. Other people are welcome to coordinate testing campaigns and get others interested and excited about testing, but Nick's focus is explicitly on the Ubuntu ISOs. Of course, if community members want to volunteer to help test the flavors then that is great. I don't think it is unreasonable for Canonical to focus its resources on Ubuntu as opposed to the flavors. I'm crystal clear that the Canonical community team's QA effort is focused on trying to get the broader community to do QA on Canonical products. I think that's quite unfortunate. Rather than just trying to get free labor for Canonical, I would have hoped you wanted to make QA better for the entire Ubuntu project. This is in marked contrast to Daniel Holbach's efforts (which I've been watching and appreciating, but not had much time to get involved with) to bring new blood into the Ubuntu development process. He's pursing the kind of holistic approach I'd hope to see from your entire team. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Thursday, June 21, 2012 11:09:54 AM Michael Casadevall wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/21/2012 09:43 AM, Jono Bacon wrote: On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall mcasadev...@ubuntu.com wrote: If we had more resources we would love to provide help for the flavors, and we are certainly happy to offer any guidance and advice, with with our current resources and staffing, Nick doesn't have the bandwidth to handle more the than the Ubuntu ISOs and associated testing. Saying that, I know Nick is in contact with many of the flavors to ensure they get the support they need to set up their own comprehensive testing plans. The first I heard of any of this was when this email was sent to u-devel. To be frank, changes of this magnitude should have been discussed in seasons at UDS-Q, and not in hallway conversations where many of the affected parties were not present. What changes of magnitude? We had already made it pretty clear that my team is not focused on flavors (historically we have never focused on flavors in our committed work items and always on Ubuntu), and all I am outlining here is the cadence in how Nick we be conducting his work. This is purely a way in which I choose to manage my resources to serve Ubuntu best. Yes, these plans were not discussed at UDS, because they have evolved since UDS, but Nick quite clearly laid out his testing strategy at UDS (which is not to dissimilar from this...he had scheduled testing plans for milestones and interim dailies). It has already been established by Rick that we can increase Canonical QA efforts without changing the milestone process. I'm embedding Rick's proposal here: On 06/17/2012 11:36 PM, Rick Spencer wrote: (I changed the subject to better represent this branch of the conversation) This discussion suggests that we don't need to release special alpha and beta ISOs, but we do need: 1. A cadence of testing 2. A trial run (or 2) of spinning ISOs 3. Development targets Therefore, I propose we: 1. Stop with the alphas and betas and win back all of the development effort 2. *Increase* the cadence of ISO testing to whatever we want or whatever the community team can manage 3. Spin a trial ISO near what is not beta time (maybe around current kernel freeze?) 4. Spin ISOs for release candidates 5. Maintain the current Alpha and Beta designations as development targets only (i.e. don't spin a special image for them). Cheers, Rick The justification for this as I understand it is that the main Ubuntu image is moving to more consistent and constant QA practice, thus when combined with the active use of -proposed during freezes, and other tools would render the freezes and associates images moot. We've already established that we can increase QA frequency without changing the freeze/milestone process, and once the -proposed queue is fully functional, no development effort will be lost for those who do not wish to partake in testing; its simply a matter then of enabling -proposed on ones machine Many images such of Kubuntu have worked to have the various milestones and deadlines set in ways that allow them to incorperate the correct versions of their upstream packages (i.e. KDE 4.9 release dates are more or less timed that the RC and final will be available for Alpha 3 and Beta 1 respectively). Personal opinions aside, I object to large-scale changes in release planning after everyone already agreed to the current Quantal release schedule. The Kubuntu team are welcome to determine whatever milestones they want - no one is suggesting anything needs to change for the flavors in their testing cadence. I am purely stating how Nick will be working: the only output of this work is assured mandatory test case testing and this testing uncovering any bugs. As I said earlier, this work is independent of whether we remove milestones or not. The Kubuntu team wants to have standard alpha and beta images (i.e. the system that is in place and currently works for them). As a member of the release team, and someone who has seen the current system of freeze/milestone release catch image breaking bugs, I want the official desktop/server images to be cross-checked by the community by the same process that everyone else uses. I don't think it is unreasonable for Canonical to focus its resources on Ubuntu as opposed to the flavors. To what extent? To the extent that Canonical provides investment in Ubuntu, as part of this investment is paying Nick's salary, and as Nick's manager I choose how we resource his time and efforts. How we resource Nick is not a community decision. I have no argument that Canonical directs Nick's work in whatever way we see fit. However, as someone paid by Canonical to ensure the high quality of our ARM images, I have a vested
Re: Releasing Alphas and Betas without freezing
On Thu, Jun 21, 2012 at 11:11 AM, Scott Kitterman ubu...@kitterman.com wrote: I don't think it is unreasonable for Canonical to focus its resources on Ubuntu as opposed to the flavors. I'm crystal clear that the Canonical community team's QA effort is focused on trying to get the broader community to do QA on Canonical products. I think that's quite unfortunate. Rather than just trying to get free labor for Canonical, I would have hoped you wanted to make QA better for the entire Ubuntu project. This is in marked contrast to Daniel Holbach's efforts (which I've been watching and appreciating, but not had much time to get involved with) to bring new blood into the Ubuntu development process. He's pursing the kind of holistic approach I'd hope to see from your entire team. We *are* working to make QA better for the entire Ubuntu project, but the point is that our focus is on *Ubuntu* and our specific efforts don't extend to coordinating flavor testing. This doesn't mean we are ignoring our flavors, or are not happy to offer advice or guidance, but my team (Daniel included) is not focusing their efforts on helping specific flavors achieve their goals. I myself am surprised that you find this surprising: while many of our efforts and programmes can bring value to the flavors (e.g. general developer growth, working with upstreams, translations work etc), we have rarely if ever assigned staff time to delivering on flavor work items. This is purely and simple about resourcing. Canonical is a company, and it needs to invest its resources carefully: sure, we would love to support all the flavors with more staff time (not just Kubuntu), but we simply don't have the resources to do so. Importantly, though, we are not stopping flavors from doing this work themselves...we are still providing the infrastructure and help and guidance we can offer in doing this work. Jono -- Jono Bacon Ubuntu Community Manager www.ubuntu.com / www.jonobacon.org www.identi.ca/jonobacon www.twitter.com/jonobacon -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
So we've clearly heard the opinion of Kubuntu...are there any other derivatives who wish to contribute to this discussion. I for one, would be interested in knowing/hearing how these suggested changes impact them. -Robbie -- Robbie Williamson rob...@ubuntu.com robbiew[irc.freenode.net] Don't make me angry...you wouldn't like me when I'm angry. -Bruce Banner -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Thursday, June 21, 2012 01:34:18 PM Robbie Williamson wrote: So we've clearly heard the opinion of Kubuntu...are there any other derivatives who wish to contribute to this discussion. I for one, would be interested in knowing/hearing how these suggested changes impact them. FWIW, while I am a Kubuntu developer, I'm also on the Ubuntu release team and active in a very broad range of activities across the distribution. My opinions aren't just from a Kubuntu perspective. I can tell you that every non-Canonical flavor struggles every milestone to get image testing done. Do more, more often just isn't going to happen in any of these flavors. If Ubuntu Desktop/Server want to take a leap of faith that this'll all turn out great, I don't expect I'm going to change anyone's mind, just take the rest of us with you. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 It has come to my attention that my emails in this thread have sounded somewhat accusatory towards both members of Canonical and Ubuntu. This was not my intent, and I apologize the way I may have came across, and hope that any party who was offended may find it in their hearts to forgive me. As such, I shall recluse myself from this discussion for now. Michael -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJP43IGAAoJEHM+GkLSJHY5nhEP/12k4Z2Zz6WqLchWze5/ydCQ FcMTCgFth1kgRKauZ6Z9Zzmc+UDVJwsKpeeaQSe003Va4t1f0Km+ExLDroxfoaAS oT+/3ydKl7lK3gY/kSXBq8YBGqfRE/xF4ERpHMNlmnSdiypoXVm8tmKorPbW26eW o+U7Zgd5J7VXmHam3C0WQ7sHCtAJ8+71okNjOYExx2vp/TjwDUklYw2GvsfXP4JC JN4jPt6qkrDThmWETUosfgXXu3HZIObEKAg0ZC9gL0P5wfNJcioT3NfJrvvMlMvU gGNaJHFydyocjXl+oeFCprIt+mDcOyklDpSwPSfLk507N5d4oUyO3+8igPS0+ecc KQE2BzYSXig/SBaGIlrUB3Lom6UfURxqI8jv3he9EvEawZp8dZnqmhxriDSwGTnv s3+di+GGch675B3YuVwR/olF3eZ9U9gvliYlaMn9U1sE2Ov6tCs/i683pCsCYsT6 y4l4PbVz8dSnzCz6HOiV78qQQ0YLu0rZRg6I0BG+q8yq04JxmE0acd82aYnMC5rn wwlEpyWWx4bWCbPTLFVmqN2NJP5m4EJ876uM26z7zu3KmWv34lHgBvmXfBdavHow Wf5WCFYJ+/JMznGH3ecp3JUHBPd4zl5PPE8mC/599yZ7XdOkwwTomaoQnDG7+7X5 0pjF7bbl6sSxfBsVWx+3 =pOR8 -END PGP SIGNATURE- -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Thu, Jun 21, 2012 at 12:20 PM, Andrew Starr-Bochicchio a.star...@gmail.com wrote: On Thu, Jun 21, 2012 at 3:12 PM, Michael Casadevall mcasadev...@ubuntu.com wrote: It has come to my attention that my emails in this thread have sounded somewhat accusatory towards both members of Canonical and Ubuntu. This was not my intent, and I apologize the way I may have came across, and hope that any party who was offended may find it in their hearts to forgive me. As such, I shall recluse myself from this discussion for now. I find that to be an extremely saddening outcome. I for one found your contributions to this discussion both constructive and clarifying. Speaking personally, I didn't find your comments offensive, it is important for us to have this discussion; my only critique would be that you were more pointed in some parts than you needed to be. Jono -- Jono Bacon Ubuntu Community Manager www.ubuntu.com / www.jonobacon.org www.identi.ca/jonobacon www.twitter.com/jonobacon -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Thursday, June 21, 2012 03:27:01 PM Nicholas Skaggs wrote: If I may speak for myself here, my goal is to encourage ubuntu as a project to be a leader in open source in the realm of quality. It's what I care about and I hope to be a part of making happen. My work at Canonical aligns with that in a very harmonious way. In no way do I wish to close out or marginalize flavors or other QA teams -- I trust my work has shown this to be quite the opposite. A brief example for illustration; in the past I have personally helped test some flavors images, and during the 'adopt an iso' campaign last cycle, while I was campaigning to help ensure a quality iso for ubuntu, I helped instruct people who wanted to test flavors images. This grows the ubuntu community as a whole and is a positive thing for the project. We have things in common and I encourage collaboration across flavors and teams (ubuntu included!). In this example, the important distinction to make is that while I helped test or encouraged others to test a flavors image, I won't claim responsibility for assuring quality on that image or ensuring it gets released. That of course is up to the flavors teams, and I would not usurp management or responsibility away from those teams.My primary focus is upon ubuntu and ensuring a healthy testing community which is able to help assure quality for each ubuntu release. This helps everyone who uses ubuntu in direct and indirect ways, including flavors. A healthy ubuntu QA community makes for a healthier ubuntu community. I agree and certainly appreciate your efforts to sustain and build QA effort across the Ubuntu project. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On 06/21/2012 02:14 PM, Scott Kitterman wrote: On Thursday, June 21, 2012 12:12:06 PM Michael Casadevall wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 It has come to my attention that my emails in this thread have sounded somewhat accusatory towards both members of Canonical and Ubuntu. This was not my intent, and I apologize the way I may have came across, and hope that any party who was offended may find it in their hearts to forgive me. As such, I shall recluse myself from this discussion for now. Michael I find that surprising and very unfortunate. I wouldn't have expected any members of the Ubuntu community to be offended by a good honest discussion of ideas. FTR, I'm *not* implyng Michael of doing this, but there is a difference between a discussion of ideas and inflamatory and incorrect accusations/assumptions. This thread has clearly gone of the rails a bit from it's original intent, thus stirring up emotions and I suspect poking wounds made from other unrelated changes in Ubuntu. I've valued Michael's input on this thread, and would certainly hope if he has any more constructive input to contribute...to do so. -- Robbie Williamson rob...@ubuntu.com robbiew[irc.freenode.net] Don't make me angry...you wouldn't like me when I'm angry. -Bruce Banner -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On 06/21/2012 03:34 PM, Robbie Williamson wrote: On 06/21/2012 02:00 PM, Stéphane Graber wrote: On 06/21/2012 02:34 PM, Robbie Williamson wrote: So we've clearly heard the opinion of Kubuntu...are there any other derivatives who wish to contribute to this discussion. I for one, would be interested in knowing/hearing how these suggested changes impact them. -Robbie Starting with just a bit of nitpicking but it's something we agreed at UDS that we'd try to get fixed in everyone's mind :) We're flavours not derivatives, flavours are fully integrated in the Ubuntu project and have been recognized as such by the various governance boards. I usually consider derivatives as referring to our downstreams like mint where they're indeed a derived product. Fair enough, but then this wiki page probably needs changing to reflect this: https://wiki.ubuntu.com/DerivativeTeam/Derivatives I also thought that page was wrong initially but it's actually kind of right. You'll notice that the flavours are listed separately under a flavours heading on that page, and what's listed afterwards are proper derivatives as in, out of the archive, custom spin based on Ubuntu. I guess we could argue that the flavours probably shouldn't be listed on that page at all to make it clearer. Kate also reminded me on IRC that she has an action item to at least remove mentions of derivatives across all the official Ubuntu websites. The wiki will always be trickier as people (such as that DerivativeTeam I never heard about) can write whatever they want... Now, speaking for Edubuntu, we don't feel like we could increase our testing frequency as we'll be increasing the number of platforms that we'll be supporting this cycle, don't have a lot of testers and generally don't feel the need for it. In the past we were only supporting a desktop installation on i386 and amd64. This cycle we're extending the desktop support to i386, amd64 and armhf. On top of that, we'll be introducing Edubuntu Server this cycle, that'll still be installed from our single media but will add a good dozen of extra services to test. The upstreams Edubuntu is working with are perfectly aware of our milestones and freeze periods and make sure their releases land on time so we have to ask for very little freeze exceptions or last minute upload (I don't think we asked for much more than 2 FFe last cycle). Changing the way we work after we agreed on the release schedule for this release would confuse our contributors and upstreams with no clear benefit for us. There are plenty of really good changes to the archive that are planned for this cycle as part of the archive reorg and increasing the use of -proposed, still with my Edubuntu release team hat on I don't think piling up changes is a good idea. I'd rather we do what we agreed on at UDS, try to encourage additional daily testing (because that never hurts, doesn't cost any development time and is beneficial) and discuss the next steps at the next UDS when we have concrete feedback on how these changes went. Thanks for the feedback Stephane. I think you've make some valid and reasonable points that we should consider. -- Stéphane Graber Ubuntu developer http://www.ubuntu.com signature.asc Description: OpenPGP digital signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/21/2012 02:45 PM, Stéphane Graber wrote: Starting with just a bit of nitpicking but it's something we agreed at UDS that we'd try to get fixed in everyone's mind :) We're flavours not derivatives, flavours are fully integrated in the Ubuntu project and have been recognized as such by the various governance boards. I usually consider derivatives as referring to our downstreams like mint where they're indeed a derived product. Fair enough, but then this wiki page probably needs changing to reflect this: https://wiki.ubuntu.com/DerivativeTeam/Derivatives I also thought that page was wrong initially but it's actually kind of right. You'll notice that the flavours are listed separately under a flavours heading on that page, and what's listed afterwards are proper derivatives as in, out of the archive, custom spin based on Ubuntu. I guess we could argue that the flavours probably shouldn't be listed on that page at all to make it clearer. Kate also reminded me on IRC that she has an action item to at least remove mentions of derivatives across all the official Ubuntu websites. The wiki will always be trickier as people (such as that DerivativeTeam I never heard about) can write whatever they want... Fair enough...and true. Thanks, - -Robbie - -- Robbie Williamson rob...@ubuntu.com robbiew[irc.freenode.net] Don't make me angry...you wouldn't like me when I'm angry. -Bruce Banner -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/jevUACgkQf9QjnyVYsWhYUQCfexoIOt41dZH/FRJSNPQA7qdf l7kAn3jdfhdh18Q/8Mw1hnU9FyT6QdY6 =Yxlx -END PGP SIGNATURE- -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
Hi Stéphane On 21/06/2012 15:45, Stéphane Graber wrote: You'll notice that the flavours are listed separately under a flavours heading on that page, and what's listed afterwards are proper derivatives as in, out of the archive, custom spin based on Ubuntu. It did say that flavours are derivatives, so I edited the text so that it makes more sense: https://wiki.ubuntu.com/DerivativeTeam/Derivatives?action=diffrev2=196rev1=195 I guess we could argue that the flavours probably shouldn't be listed on that page at all to make it clearer. Yep, and possibly a better explanation of the differences between flavours and derivatives (and make it clear that the flavours aren't derivatives) Kate also reminded me on IRC that she has an action item to at least remove mentions of derivatives across all the official Ubuntu websites. The wiki will always be trickier as people (such as that DerivativeTeam I never heard about) can write whatever they want... Yep, I haven't come across it that much, but when I do I'll just update it. -Jonathan -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On 06/21/2012 02:46 PM, Scott Kitterman wrote: On Thursday, June 21, 2012 11:25:09 AM Jono Bacon wrote: On Thu, Jun 21, 2012 at 11:11 AM, Scott Kitterman ubu...@kitterman.com wrote: I don't think it is unreasonable for Canonical to focus its resources on Ubuntu as opposed to the flavors. I'm crystal clear that the Canonical community team's QA effort is focused on trying to get the broader community to do QA on Canonical products. I think that's quite unfortunate. Rather than just trying to get free labor for Canonical, I would have hoped you wanted to make QA better for the entire Ubuntu project. This is in marked contrast to Daniel Holbach's efforts (which I've been watching and appreciating, but not had much time to get involved with) to bring new blood into the Ubuntu development process. He's pursing the kind of holistic approach I'd hope to see from your entire team. We *are* working to make QA better for the entire Ubuntu project, but the point is that our focus is on *Ubuntu* and our specific efforts don't extend to coordinating flavor testing. This doesn't mean we are ignoring our flavors, or are not happy to offer advice or guidance, but my team (Daniel included) is not focusing their efforts on helping specific flavors achieve their goals. I myself am surprised that you find this surprising: while many of our efforts and programmes can bring value to the flavors (e.g. general developer growth, working with upstreams, translations work etc), we have rarely if ever assigned staff time to delivering on flavor work items. This is purely and simple about resourcing. Canonical is a company, and it needs to invest its resources carefully: sure, we would love to support all the flavors with more staff time (not just Kubuntu), but we simply don't have the resources to do so. Importantly, though, we are not stopping flavors from doing this work themselves...we are still providing the infrastructure and help and guidance we can offer in doing this work. We're talking about two different Ubuntus. You're talking about Ubuntu the product defined by a set of images/metapackages/etc drawn from a subset of the Ubuntu Linux distribution's archive. I'm talking about Ubuntu as a project which is bigger than either of those. Scott K If I may speak for myself here, my goal is to encourage ubuntu as a project to be a leader in open source in the realm of quality. It's what I care about and I hope to be a part of making happen. My work at Canonical aligns with that in a very harmonious way. In no way do I wish to close out or marginalize flavors or other QA teams -- I trust my work has shown this to be quite the opposite. A brief example for illustration; in the past I have personally helped test some flavors images, and during the 'adopt an iso' campaign last cycle, while I was campaigning to help ensure a quality iso for ubuntu, I helped instruct people who wanted to test flavors images. This grows the ubuntu community as a whole and is a positive thing for the project. We have things in common and I encourage collaboration across flavors and teams (ubuntu included!). In this example, the important distinction to make is that while I helped test or encouraged others to test a flavors image, I won't claim responsibility for assuring quality on that image or ensuring it gets released. That of course is up to the flavors teams, and I would not usurp management or responsibility away from those teams.My primary focus is upon ubuntu and ensuring a healthy testing community which is able to help assure quality for each ubuntu release. This helps everyone who uses ubuntu in direct and indirect ways, including flavors. A healthy ubuntu QA community makes for a healthier ubuntu community. Thanks, Nicholas -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Wed, Jun 20, 2012 at 3:07 AM, Rick Spencer rick.spen...@canonical.com wrote: I think this was a very productive discussion. We considered a lot of possibilities from a lot of angles. All told, I think there are four points under discussion. I'd like to tease them out so we can move forward. Question 1: shall we stop freezing the archive at milestones? I believe there is not 100% consensus on this point, but enough support to try it for Alpha 2, a la Theirry's suggestion. QA Team/Foundations Team, do we/will we have the tools in place for Alpha 2? Question 2: shall we stop having milestones altogether? This question arose in thread. I don't believe there is consensus for doing this suddenly in 12.10. Question 3: shall we increase the rate of manual testing? This question also arose in the thread. I think there is widespread consensus that we should do this, and it is not actually related to the other questions. Community Team, is it feasible to increase the rate of full manual testing runs to every 2 weeks or similar? Question 4: shall we keep snapshots of the development release so that we can bisect more easily and find when bugs were introduced? This question also arose, and also is not tied to the other questions. QA Team, is it feasible to keep a set of snap shots somewhere for this purpose? Yes. I spoke to Canonical IS yesterday and we have the space to keep 1 or 2 of the testing snapshots around. There will be some work on the cdimage tooling to allow them to stay around and I'll coordinate with the proper people to make that happen. Thanks ~pete -- Pete Graner - Release Engineering QA Team Manager - pgra...@canonical.com Canonical Ltd. - http://www.canonical.com/ -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/19/2012 11:20 PM, Rick Spencer wrote: On Tue, Jun 19, 2012 at 8:06 PM, Michael Casadevall mcasadev...@ubuntu.com wrote: -BEGIN PGP SIGNED MESSAGE- . Many critical issues with ARM (and to a lesser extent x86) have only been found during milestone testing. Without a set of defined and organized images for testing, more obscure parts of the installer simply do not get tested; for instance, how many people are going to test all possible server configurations or test the installer with no network. But, again, why is the set of defined organized testing for milestones? That seems much too infrequent. Dependence on milestones is creating a lack of quality in this area, not improving it. We should not be allowing days to go by without a usable image. And, again, this is completely orthogonal to whether we freeze the archive for milestones, or have milestones at all. In other words, milestones seem a poor means to accomplish your goals here. We should organize a more rigorous and frequent cadence of testing for ARM images. I for one greatly welcome the return of return of more manual testing. Due to the massive number of images for ARM due the unique quirks of the architecture, and the limited manpower behind said efforts, the coverage per image was unfortunately lacking. Because of the amount of work, certain images were not as rigorously tested as they should have and as a result both Beta 2 for and the release candidate for the omap4 images were both woefully under-tested, and were only brought back up to a releasable quality at the zero-hour due to the herculean efforts of far too many parties to list. This new manual testing initiative should prevent us from ever having such a crisis ever again. That being said, due to the sheer number of ARM images due to the unique one-image-per-board that ARM requires, the lack of manpower has been a constant issue, and our previous manual testing efforts were woefully undermanned (granted, at the time, we had over 10-15 actively supported images). Although we do not have multiple image types as with x86 (live/alternate), we still have 6 images that are officially supported as of this moment. * Ubuntu Desktop for armhf+omap4 * Ubuntu Server for armhf+omap4 * Ubuntu Core for armhf * Netboot for armhf+omap4 * Netboot for armhf+armadaxp * Netboot for armhf+highbank (this obviously does not take into account any additional subarchitectures or flavors that may be added during quantal development). Michael -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJP4g+/AAoJEHM+GkLSJHY5NmcP/i185phR+AkeQRG6tqB0hz6E ofT4cWeFtkZO6pdZQa1NOYLrKloy3yVPug/H4xiSdgq59PQpAVqixZTuFxD/cZOR SwcobYIY6aJZAoq+7UDd/6tlgNTR0ymCPu2inrNpp1ieW50BA7Y3YZbZsrn7Bwjj 3ueDClQvLJAH7FxJKJPfl6mN9xGiPKOyDltAI1qV+KeEpuQQdMkkMO9ABESroq0H VX4n64a/9RpWaA5BhSa1TlXU2ajOEltk+YFzHRQiQX3q/3pfzDQQ2JTl1du04LTG AzCb4WbP6LZfFrt4l4378YaCrOfjTiqvjBU2V4GrwllRmM931Pg0t3tW405mNYdw +pR5JVJiNR9FQYvltyv9QJSREN+k820mK7P26QUGii6XN5rwxmP8EelmU54DlMiV JfFr7rO1g2Pf91iVEwqysFBHhtddN4sAjCE9kkwUwuKFAPo/qZ9/mgK/wBcQADxE yYCZywTiPUnK5ijvVHfzuK5qeHMoVSbFf7CKX+b4a8NliuAIyG189g58cyrKQSC7 h03KuCOe72KVDIfLpiVvYRaJ8R0f6Y6lEBpkasmAXVytWQ79k1fSnXhHXmb1sFvZ WgEoT0z6fvejB23W7CO3qVD2ZyT6Z3JAvKnk7tyLJrwJX829rG95qH9SHIGwt5sO uTfV2BKY0xBbbIdK3vbX =0EJ6 -END PGP SIGNATURE- -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
Jono Bacon j...@ubuntu.com wrote: On Wed, Jun 20, 2012 at 12:07 AM, Rick Spencer rick.spen...@canonical.com wrote: Question 3: shall we increase the rate of manual testing? This question also arose in the thread. I think there is widespread consensus that we should do this, and it is not actually related to the other questions. Community Team, is it feasible to increase the rate of full manual testing runs to every 2 weeks or similar? I talked with Nick Skaggs this week and we are happy to commit to manual testing every two weeks, starting a week on Thursday. Originally I required that Nick *assured* testing of all mandatory tests for each milestone, and I am asking him to provide the same assurances every two weeks. For all flavors or just the Canonical ones? Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
Le 20/06/2012 21:52, Michael Casadevall a écrit : While its not as obvious on x86/amd64, both ARM and powerpc have slower buildds and as such have missed daily images before due to the archive entering an uninstallable state. What you describe here an issue doesn't apply in a world where proposed is used as a staging area, things would not be copied to the release pocket before being built on all archs... (Btw why did you feel like crossposting on a second list was a good idea, it just makes the discussion harder to follow, and you can assume that release team members read the devel list to follow what's happening in Ubuntu) -- Sebastien Bacher -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Wednesday, June 20, 2012 02:41:00 PM Jono Bacon wrote: On Wed, Jun 20, 2012 at 11:39 AM, Scott Kitterman ubu...@kitterman.com wrote: I talked with Nick Skaggs this week and we are happy to commit to manual testing every two weeks, starting a week on Thursday. Originally I required that Nick *assured* testing of all mandatory tests for each milestone, and I am asking him to provide the same assurances every two weeks. For all flavors or just the Canonical ones? For our current Ubuntu ISOs. Flavors currently are coordinating their own testing efforts. They could either latch into the two week cadence, or use their own cadence if desired. I find it somewhat unfortunate that the community testing efforts exclude the community sponsored flavors in the Ubuntu project. I would have hoped that the community team was not just about Canonical's products. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Wed, Jun 20, 2012 at 8:52 PM, Scott Kitterman ubu...@kitterman.com wrote: For our current Ubuntu ISOs. Flavors currently are coordinating their own testing efforts. They could either latch into the two week cadence, or use their own cadence if desired. I find it somewhat unfortunate that the community testing efforts exclude the community sponsored flavors in the Ubuntu project. I would have hoped that the community team was not just about Canonical's products. This shouldn't be a particularly big surprise; Canonical supports our flavors with infrastructure, but we primarily focus our engineering and community team staff members on Ubuntu. If we had more resources we would love to provide help for the flavors, and we are certainly happy to offer any guidance and advice, with with our current resources and staffing, Nick doesn't have the bandwidth to handle more the than the Ubuntu ISOs and associated testing. Saying that, I know Nick is in contact with many of the flavors to ensure they get the support they need to set up their own comprehensive testing plans. Jono -- Jono Bacon Ubuntu Community Manager www.ubuntu.com / www.jonobacon.org www.identi.ca/jonobacon www.twitter.com/jonobacon -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Tuesday, June 19, 2012 11:06:14 AM Michael Casadevall wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, Jun 18, 2012 at 11:33 PM, Rick Spencer rick.spen...@canonical.com wrote: On Tue, Jun 19, 2012 at 12:15 AM, Kate Stewart kate.stew...@canonical.com wrote: For Alpha1, we did 2 respin sets after the first set was built, based on what the manual testing was finding and trying to get a set of ARM desktop images. (Note: We did not have quantal arm desktop images until the week of alpha 1, and then didn't have them again with the dailies between 6/10-6/14). Having milestones does force a focus on the full set of images. Daily images and the automated testing are still mostly focusing on unit tests for the x86 desktop and server images in virtualized hardware, and as Martin says, the manual testing is still finding issues on the real hardware that are causing respins. I believe there is widespread agreement on this thread that manual testing is good and necessary. I also think there is agreement that a faster cadence of complete manual testing than is accommodated by our current milestones would be desirable. I think it's fair to say that we can move ahead with increasing the frequency of manual testing with or without changes to our milestones. I will look to the Ubuntu Community team to begin with this, as they don't believe they are blocked by any other decisions to be made. I think the question on the table is, shall we drop most milestones altogether, or adopt a system such as Thierry suggests, where we use the most recent good daily as the milestone image? I have serious concerns with removing the milestones. As it stands, several images, including the vast majority of the ARM images, only get extensively tested at milestones due to the limited userbase of the image (specifically, highbank and armadaxp as of right now is limited to a handful of individuals in the world at the moment). Many critical issues with ARM (and to a lesser extent x86) have only been found during milestone testing. Without a set of defined and organized images for testing, more obscure parts of the installer simply do not get tested; for instance, how many people are going to test all possible server configurations or test the installer with no network. These scenarios are not common for development, but can and do occur regularly for many users who install Ubuntu for the first time. During 12.04 development, during milestone testing, three bugs* relating to both usecases were found to cause the installer to silently fail midway through installation leaving the user with only a partially configured system. Each milestone represents an opportunity for end-users and QA to test our images in something more resembling a production environment, and to test use-cases and recipes that may normally not see a lot of coverage unless one is explicatively checking for edge cases. Milestones exist to give the Ubuntu developer community to step back, and check to make sure nothing important has broken, and to gauge our progress through a cycle. In addition, they provide a dedicated time where as a community we step forth and check our images to ensure no regressions have slipped by. If we remove the milestones, the only period of extensively and review the images will be just before release. As such, any regressions that are found would require a scrambled fix during a period that minimal archive changes are desired and would both be costly in terms of development effort, and risky as each final freeze upload always carries the inherent chance of hosing something important. Unless the final intent is to ultimately abolish releases all together and move to a rolling-release model, I don't believe we, as a community, could successfully ship Ubuntu with its excellent state of quality assurance without the cycles of alpha and beta images. * - Relevant bug reports: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/985737 https://bugs.launchpad.net/livecd-rootfs/+bug/985258 https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/985280 +1 I have to confess that when I threw out the idea of just abolishing the milestones way back in this thread I thought it was a sufficiently ridiculous idea that it would give people pause about dropping the freezes. People worried about velocity through a freeze can publish stuff in a PPA and ask them to test it during a freeze. I think this entire notion is going to add significant risk to the development cycle. Michael is right on target. Without a dedicated focus on human testing of various components things are going to be missed until the end game when broad user testing starts. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
Scott Kitterman [2012-06-19 22:16 -0400]: +1 +1 as well, thanks Michael and Scott. Also, saying we'll drop milestones and introduce a regular schedule for testing some particular dailies to fix bugs not caught by automatic tests means nothing more than a simple renaming -- because that's exactly what milestones are today. So I also think that proved nicely what their purpose is and that we cannot do without them. We can certainly discuss having more or fewer of them, if the current distance between them is too high. Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Drop Alphas and Betas (was Re: Releasing Alphas and Betas without freezing)
(I changed the subject to better represent this branch of the conversation) This discussion suggests that we don't need to release special alpha and beta ISOs, but we do need: 1. A cadence of testing 2. A trial run (or 2) of spinning ISOs 3. Development targets Therefore, I propose we: 1. Stop with the alphas and betas and win back all of the development effort 2. *Increase* the cadence of ISO testing to whatever we want or whatever the community team can manage 3. Spin a trial ISO near what is not beta time (maybe around current kernel freeze?) 4. Spin ISOs for release candidates 5. Maintain the current Alpha and Beta designations as development targets only (i.e. don't spin a special image for them). Cheers, Rick On Sun, Jun 17, 2012 at 11:25 PM, Robert Ancell robert.anc...@canonical.com wrote: On 16/06/12 02:12, Rick Spencer wrote: In short, freezing the archive before an alpha or beta should not actually be contributing to either ensuring the installability of Ubuntu images or ensuring the quality of these images. This implies, therefore, that all the work around freezing, and all the productivity lost during a freeze, actually subtracts from the quality of Ubuntu by reducing our overall velocity for both features and bug fixes, since every day the image is good quality, and Alpha or Beta should be just that day's image tagged appropriately. In particular I find the alpha freeze kills our velocity and I wonder how more useful than a daily build the alpha release is (given it's so early in the cycle anyway). I'd support dropping the alpha and pointing at the dailies. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
Robbie Williamson wrote: Yeah, you are right there, so if we get working dailies every day do we still need alphas at all? Ideally we would have the automatic testing flagging isos green when they have no issue (with the goal to always have them good) and we would recommend people to just pick the current green iso. Can we just drop the image rolling part of milestones? We still probably want fixed checkpoints in the cycle to review the features, etc but they don't especially need to be linked with a special image... With our dailies, I've found that the milestones are most useful for planning bug fix landings and feature deliverables. I'd be +1 on dropping alphas all together. [...] I'd certainly agree that the main value of milestones is as target dates and to create an internal cadence within a cycle. That said, I still think there is value in singling out a given build once in a while to serve (1) as a reference point (this bug wasn't present at alpha2), (2) to encourage user testing (please test beta1), and (3) as an event that builds excitement towards the upcoming release (let me introduce the last milestone before 12.10 final release). The trick is to make the release process of the milestone as light as possible, so that it does not generate a significant extra amount of work, nor should it block development. In that respect, getting rid of soft freezes sounds like a very good idea. Something like taking a daily from the Tuesday and tagging it on Thursday as the milestone, unless a critical issue gets discovered in the meantime that justifies to use a later daily as the milestone. Regards, -- Thierry Carrez (ttx) -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Drop Alphas and Betas (was Re: Releasing Alphas and Betas without freezing)
On 18 June 2012 07:36, Rick Spencer rick.spen...@canonical.com wrote: Therefore, I propose we: 1. Stop with the alphas and betas and win back all of the development effort 2. *Increase* the cadence of ISO testing to whatever we want or whatever the community team can manage 3. Spin a trial ISO near what is not beta time (maybe around current kernel freeze?) 4. Spin ISOs for release candidates 5. Maintain the current Alpha and Beta designations as development targets only (i.e. don't spin a special image for them). From the point of view of a community member: The term Alpha can be quite frightening to some people, given that it's traditionally been associated with unstable software in the early stages of development. The few times I've installed Alpha version of Ubuntu it was no more than a couple of days before I had to revert back to the previous stable release because it was so unusable. In the 12.04 cycle, that was no longer the case and the Alpha was quite usable, so I think that since we've broken with the instability during the release cycle, perhaps we should break with the Alpha designation as well. If we want to get more people involved in testing early on, then I think this would be the way to go. I also think adding the Release Candidate (RC) designation towards the end of the cycle would encourage more people who are quite wary of installing pre-release software on their computer to get involved with last minute testing since RC indicates that it's pretty much done and all that's left is to iron out some minor glitches. I think releasing more Betas would also go towards getting more people involved in testing, as most people would give a new Beta a test once it's released, but forget about it soon after. Releasing a Beta each month, based on the last successful test ISO build, would give more milestones for people to get excited about, and bring back people who had drifted away after some earlier testing. Therefore, I propose: 1) Beta 1-4 at the end of months 1-4 2) Release Candidate release at the end of month 5 3) Full release at the end of month 6 4) All the while with new test ISOs being spun up every 2 or 3 weeks (or greater if that's too fast) -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Mon, 18 Jun 2012, Thierry Carrez wrote: With our dailies, I've found that the milestones are most useful for planning bug fix landings and feature deliverables. I'd be +1 on dropping alphas all together. [...] I'd certainly agree that the main value of milestones is as target dates and to create an internal cadence within a cycle. That said, I still think there is value in singling out a given build once in a while to serve (1) as a reference point (this bug wasn't present at alpha2), I will +1 that. I do not think it justifies alpha in itself, but it does justify keeping *some* historic builds around for a longer period of time. In cloud-images, anything we mark as milestone (which includes alphas) are kept indefinitely. The cost of a build is currently storage of ~2.1G. The value is that people can easily reproduce that this bug wasn't present in alpha2, as compared to I don't think this bug was present in alpha2 (which is almost entirely useless). I realize, that, because we do not keep historical content in the archive, alpha2 really only means packages included in the image (or ISO) in alpha2. That said, I do think it is useful to keep around some generally-functional builds to reference and bisect from. (And I surely wouldn't complain if someone said we should also then implement something like snapshot.debian.org). -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Mon, Jun 18, 2012 at 11:49:46AM +0200, Rick Spencer wrote: On Mon, Jun 18, 2012 at 7:02 AM, Martin Pitt martin.p...@ubuntu.com wrote: Sebastien Bacher [2012-06-15 17:26 +0200]: Can we just drop the image rolling part of milestones? Our automated tests are still wy to incomplete for this step. In manual testing we have found quite a number of real deal-breaker bugs which the automatic tests didn't pick up. We also need to test the current images on a wider range of real iron; which is something our automated QA could do one day, but doesn't right now. So regular manual testing rounds are still required, and the points when we do them might just as well be called milestones. But if the focus is testing, we should optimize the schedule around testing. For example, I think Ubuntu would benefit from more frequent rounds of such in depth testing than the current alpha/beta milestones provide. (I think every 2 weeks would be a good cadence). This could be very beneficial if it were more aggressively organized. We did something similar with proprietary driver testing one release a few years back. We had people join a team, and then had them install isos and run through a checklist once a week. I found it to be quite valuable, but you had to be very organized for it to be useful. So this wasn't just a install the image and file bugs exercise, but an deliberate look for serious regressions. By having each person provide a continuous series of data points we could spot anomalies much more easily. If someone is installing things exactly the same way, on the same hardware, every week, and all of a sudden one week it fails, that helps you narrow things down a lot. Or equally important is seeing that a fix you roll out does indeed restore functionality across multiple testers. The key was to be very specific in the data collection, else you can generate a lot of noise quickly. Make a printable survey form they can fill in as they go through the checklist procedure, and a system info dumping tool that captures all the logs when they're done that might be needed for bug reports. The QA team has a tool for capturing all this data and showing it in a tabulated form so you can spot patterns and changes over time. The most important thing is that the data actually get used. This testing can take a fair bit of time, but if the testers know their efforts are helping to make things tangibly better they can really get passionate about doing it. Bryce -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Drop Alphas and Betas (was Re: Releasing Alphas and Betas without freezing)
On Monday, June 18, 2012 01:30:34 PM Chris Wilson wrote: ... I also think adding the Release Candidate (RC) designation towards the end of the cycle would encourage more people who are quite wary of installing pre-release software on their computer to get involved with last minute testing since RC indicates that it's pretty much done and all that's left is to iron out some minor glitches. ... We used to call what's now Beta 1 a Release Candidate for similar reasons, but renamed it because it's not really a release candidate. Generally Release Candidate means Thing that will get released if no new significant issues turn up. Our current usage matches that and we should stick with it. I don't like the idea of turning Release Candidate into a marketing term in order to encourage more testing. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Drop Alphas and Betas (was Re: Releasing Alphas and Betas without freezing)
Ditën e Mon, 18/06/2012 më 15.30 -0400, Scott Kitterman ka shkruar: On Monday, June 18, 2012 01:30:34 PM Chris Wilson wrote: ... I also think adding the Release Candidate (RC) designation towards the end of the cycle would encourage more people who are quite wary of installing pre-release software on their computer to get involved with last minute testing since RC indicates that it's pretty much done and all that's left is to iron out some minor glitches. ... We used to call what's now Beta 1 a Release Candidate for similar reasons, but renamed it because it's not really a release candidate. Generally Release Candidate means Thing that will get released if no new significant issues turn up. Our current usage matches that and we should stick with it. I don't like the idea of turning Release Candidate into a marketing term in order to encourage more testing. And I think the idea here is that every single daily image fits into that category of what we're calling a Release Candidate. If we maintain high quality throughout the cycle, then at any point after the higher level freezes (feature, UI, string, etc…) we could theoretically point to any image and say this is the release candidate. If we can't do that, then we should at least be able to isolate where and why we can't (particular packages not meeting the quality standards, introducing problems late in cycle, etc…), and work on preventing that from happening in the future. signature.asc Description: This is a digitally signed message part -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Drop Alphas and Betas (was Re: Releasing Alphas and Betas without freezing)
On Monday, June 18, 2012 03:42:49 PM Rodney Dawes wrote: Ditën e Mon, 18/06/2012 më 15.30 -0400, Scott Kitterman ka shkruar: On Monday, June 18, 2012 01:30:34 PM Chris Wilson wrote: ... I also think adding the Release Candidate (RC) designation towards the end of the cycle would encourage more people who are quite wary of installing pre-release software on their computer to get involved with last minute testing since RC indicates that it's pretty much done and all that's left is to iron out some minor glitches. ... We used to call what's now Beta 1 a Release Candidate for similar reasons, but renamed it because it's not really a release candidate. Generally Release Candidate means Thing that will get released if no new significant issues turn up. Our current usage matches that and we should stick with it. I don't like the idea of turning Release Candidate into a marketing term in order to encourage more testing. And I think the idea here is that every single daily image fits into that category of what we're calling a Release Candidate. If we maintain high quality throughout the cycle, then at any point after the higher level freezes (feature, UI, string, etc…) we could theoretically point to any image and say this is the release candidate. If we can't do that, then we should at least be able to isolate where and why we can't (particular packages not meeting the quality standards, introducing problems late in cycle, etc…), and work on preventing that from happening in the future. Up until the last translation import is finished, we know everything is not a release candidate. After that, I agree. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Drop Alphas and Betas (was Re: Releasing Alphas and Betas without freezing)
On 06/18/2012 03:46 PM, Scott Kitterman wrote: On Monday, June 18, 2012 03:42:49 PM Rodney Dawes wrote: Ditën e Mon, 18/06/2012 më 15.30 -0400, Scott Kitterman ka shkruar: On Monday, June 18, 2012 01:30:34 PM Chris Wilson wrote: ... I also think adding the Release Candidate (RC) designation towards the end of the cycle would encourage more people who are quite wary of installing pre-release software on their computer to get involved with last minute testing since RC indicates that it's pretty much done and all that's left is to iron out some minor glitches. ... We used to call what's now Beta 1 a Release Candidate for similar reasons, but renamed it because it's not really a release candidate. Generally Release Candidate means Thing that will get released if no new significant issues turn up. Our current usage matches that and we should stick with it. I don't like the idea of turning Release Candidate into a marketing term in order to encourage more testing. And I think the idea here is that every single daily image fits into that category of what we're calling a Release Candidate. If we maintain high quality throughout the cycle, then at any point after the higher level freezes (feature, UI, string, etc…) we could theoretically point to any image and say this is the release candidate. If we can't do that, then we should at least be able to isolate where and why we can't (particular packages not meeting the quality standards, introducing problems late in cycle, etc…), and work on preventing that from happening in the future. Up until the last translation import is finished, we know everything is not a release candidate. After that, I agree. Scott K We typically need the last batch of outside-of-langpacks translation updates, a new batch of langpacks, a new base-files (for lsb_release), a new ubiquity (dropping any remaining alpha/beta warning) and matching debian-cd change (to actually mark the images as finals). Currently we consider that any of the above (including the debian-cd flag change) require re-testing of the images. -- Stéphane Graber Ubuntu developer http://www.ubuntu.com signature.asc Description: OpenPGP digital signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Mon, 2012-06-18 at 11:49 +0200, Rick Spencer wrote: On Mon, Jun 18, 2012 at 7:02 AM, Martin Pitt martin.p...@ubuntu.com wrote: Sebastien Bacher [2012-06-15 17:26 +0200]: Can we just drop the image rolling part of milestones? We still probably want fixed checkpoints in the cycle to review the features, etc but they don't especially need to be linked with a special image... Our automated tests are still wy to incomplete for this step. In manual testing we have found quite a number of real deal-breaker bugs which the automatic tests didn't pick up. We also need to test the current images on a wider range of real iron; which is something our automated QA could do one day, but doesn't right now. So regular manual testing rounds are still required, and the points when we do them might just as well be called milestones. But if the focus is testing, we should optimize the schedule around testing. For example, I think Ubuntu would benefit from more frequent rounds of such in depth testing than the current alpha/beta milestones provide. (I think every 2 weeks would be a good cadence). https://wiki.ubuntu.com/QuantalQuetzal/ReleaseInterlock Between the 12.04.1 and Quantal Milestones, the QA Testing and QA Community testing have a pretty full load already. (see columns) What was decided to try with Quantal was to do a more intense round of manual testing on the dailies, the week before the milestone, so that the bugs found could be fixed by development, and still give the developers a good window of focused transitions and feature development time. This possibly could be adjusted to a round of testing 2 weeks prior, but would have to be juggled in with the testing team's other commitments? We're releasing Beta 1 on 9/6, Beta 2 on 9/27 and Final on 10/18 - each 3 weeks apart, so not as much room there. Not sure how many of the other Ubuntu flavors (Kubuntu, Xubuntu, etc.) that come out with the alpha milestones would want to participate in a more frequent testing schedule though. They already skip some of the milestones, based on which of their packages are landing and resources are available to do the manual testing, but do have an implied dependency on Ubuntu alpha/betas being available. For Alpha1, we did 2 respin sets after the first set was built, based on what the manual testing was finding and trying to get a set of ARM desktop images. (Note: We did not have quantal arm desktop images until the week of alpha 1, and then didn't have them again with the dailies between 6/10-6/14). Having milestones does force a focus on the full set of images. Daily images and the automated testing are still mostly focusing on unit tests for the x86 desktop and server images in virtualized hardware, and as Martin says, the manual testing is still finding issues on the real hardware that are causing respins. Kate -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On 16/06/12 02:12, Rick Spencer wrote: In short, freezing the archive before an alpha or beta should not actually be contributing to either ensuring the installability of Ubuntu images or ensuring the quality of these images. This implies, therefore, that all the work around freezing, and all the productivity lost during a freeze, actually subtracts from the quality of Ubuntu by reducing our overall velocity for both features and bug fixes, since every day the image is good quality, and Alpha or Beta should be just that day's image tagged appropriately. In particular I find the alpha freeze kills our velocity and I wonder how more useful than a daily build the alpha release is (given it's so early in the cycle anyway). I'd support dropping the alpha and pointing at the dailies. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Sat, Jun 16, 2012 at 2:12 AM, Rick Spencer rick.spen...@canonical.com wrote: Hello all, At UDS I had some hallway discussions about why we freeze for Alphas and Betas, and the fact that I think it is time to drop this practice and rather focus on making Ubuntu good quality each day. Sadly, there was no session on this, thus this email to this list for discussion. +1 in general. One thing that occurs to me, is that I don't know what the Alpha and Betas are *for* for us I mean: in a traditional software product alpha releases would be used to guage customer reaction to new features and changes, betas are used to assess real-world defect rates (and once they drop low enough, general release can happen). We have landed substantial changes post-alpha-milestone in the past, and we probably will continue to do so (e.g. Gnome releases, Unity etc): so I'm not sure, *other* than defect rates, what Alpha does for us. I'm a huge fan of keeping trunk stable and release-quality always, which makes the beta process still useful, but one that doesn't need fixed beta releases, just get folk tracking trunk and reporting back. -Rob -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
Scott Kitterman [2012-06-15 10:46 -0400]: In Debian terms I'm seeing release-proposed as like unstable and release as testing. Is there a mechanism for direct uploads to testing (e.g. t-p-u)? I don't think we want that. Our system will move it over as soon as it's built, verified to not cause uninstallability, and pass the automatic tests, but it will not wait any extra time (unlike Debian's testing migration). When preparing a milestone release, it is rather more important to ensure to not let in a package that only built on half of the architectures and causes uninstallability. Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
Sebastien Bacher [2012-06-15 17:26 +0200]: Can we just drop the image rolling part of milestones? We still probably want fixed checkpoints in the cycle to review the features, etc but they don't especially need to be linked with a special image... Our automated tests are still wy to incomplete for this step. In manual testing we have found quite a number of real deal-breaker bugs which the automatic tests didn't pick up. We also need to test the current images on a wider range of real iron; which is something our automated QA could do one day, but doesn't right now. So regular manual testing rounds are still required, and the points when we do them might just as well be called milestones. Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Monday, June 18, 2012 06:59:17 AM Martin Pitt wrote: Scott Kitterman [2012-06-15 10:46 -0400]: In Debian terms I'm seeing release-proposed as like unstable and release as testing. Is there a mechanism for direct uploads to testing (e.g. t-p-u)? I don't think we want that. Our system will move it over as soon as it's built, verified to not cause uninstallability, and pass the automatic tests, but it will not wait any extra time (unlike Debian's testing migration). When preparing a milestone release, it is rather more important to ensure to not let in a package that only built on half of the architectures and causes uninstallability. This thought was in the context of dealing with uploads that were too invasive to hit a milestone, but we needed a smaller fix to get into the milestone? It seems wasteful to remove from -proposed and reupload, but I guess that would work. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Releasing Alphas and Betas without freezing
Hello all, At UDS I had some hallway discussions about why we freeze for Alphas and Betas, and the fact that I think it is time to drop this practice and rather focus on making Ubuntu good quality each day. Sadly, there was no session on this, thus this email to this list for discussion. I think it is time drop our Freeze practices for the alphas and betas. Here is my reasoning: 1. We are developing tools that allow us to efficiently use -proposed in a way that will ensure we will not have partially built or incompatible components in the release pocket ... ever. Including days we release Alphas and Betas: These blueprints tools to ensure that Ubuntu is not uninstallable or have other problems due to partially built components and such: https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-intermediary https://blueprints.launchpad.net/ubuntu/+spec/other-q-freeze-use-of-proposed I have been assured that the tools necessary to automate the work of moving components correctly from -proposed to the release will be ready before Alpha 2. 2. We are investing heavily in the daily quality of Ubuntu. For example ... We run the same automated tests on an alpha as we run on a daily: https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/ We tend to archive issues each day: http://people.canonical.com/~ubuntu-archive/testing/quantal_probs.html We ran all the manual ISO tests *before* we released Alpha 1, and we have the capability of doing this at will: http://iso.qa.ubuntu.com/qatracker/milestones/221/builds In short, freezing the archive before an alpha or beta should not actually be contributing to either ensuring the installability of Ubuntu images or ensuring the quality of these images. This implies, therefore, that all the work around freezing, and all the productivity lost during a freeze, actually subtracts from the quality of Ubuntu by reducing our overall velocity for both features and bug fixes, since every day the image is good quality, and Alpha or Beta should be just that day's image tagged appropriately. AIUI, A1 was delivered in such a manner, though without the tooling to ensure that moving from -proposed to the release pocket was efficient and automated. Cheers, Rick -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On 06/15/2012 10:12 AM, Rick Spencer wrote: Hello all, At UDS I had some hallway discussions about why we freeze for Alphas and Betas, and the fact that I think it is time to drop this practice and rather focus on making Ubuntu good quality each day. Sadly, there was no session on this, thus this email to this list for discussion. I think it is time drop our Freeze practices for the alphas and betas. Here is my reasoning: 1. We are developing tools that allow us to efficiently use -proposed in a way that will ensure we will not have partially built or incompatible components in the release pocket ... ever. Including days we release Alphas and Betas: These blueprints tools to ensure that Ubuntu is not uninstallable or have other problems due to partially built components and such: https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-intermediary https://blueprints.launchpad.net/ubuntu/+spec/other-q-freeze-use-of-proposed I have been assured that the tools necessary to automate the work of moving components correctly from -proposed to the release will be ready before Alpha 2. 2. We are investing heavily in the daily quality of Ubuntu. For example ... We run the same automated tests on an alpha as we run on a daily: https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/ We tend to archive issues each day: http://people.canonical.com/~ubuntu-archive/testing/quantal_probs.html We ran all the manual ISO tests *before* we released Alpha 1, and we have the capability of doing this at will: http://iso.qa.ubuntu.com/qatracker/milestones/221/builds In short, freezing the archive before an alpha or beta should not actually be contributing to either ensuring the installability of Ubuntu images or ensuring the quality of these images. This implies, therefore, that all the work around freezing, and all the productivity lost during a freeze, actually subtracts from the quality of Ubuntu by reducing our overall velocity for both features and bug fixes, since every day the image is good quality, and Alpha or Beta should be just that day's image tagged appropriately. AIUI, A1 was delivered in such a manner, though without the tooling to ensure that moving from -proposed to the release pocket was efficient and automated. Cheers, Rick Hi Rick, We certainly want to allow people to upload stuff to -proposed during a milestone week, but I don't agree that we should automatically copy from -proposed to the release pocket during a milestone week. We usually try to release all our images with the same versions of the packages, considering it takes us hours to rebuild everything, having seeded packages land during that time, will lead to images having different version of packages. As for what happened with Alpha 1, we simply asked everyone to upload their packages to -proposed and then cherry picked the packages we actually needed for the release from -proposed and copied them into the release pocket before rebuilding the images (we did that 3 times). As I understand it, the plan going forward is to have the release pocket be an alias of -proposed on upload, so that everything always lands into -proposed. After something lands in -proposed, is properly built and passes whatever other criteria we'll have, the package will be automatically copied to the release pocket. That last part (copy to the release pocket) would be what we'd block during a milestone week for any package that's seeded. These would be copied on a case by case basis by the release team and the images rebuilt afterwards. That'd essentially allow any non-seeded package to still flow to the release pocket and be available for everyone. All the others will be available for people running with -proposed enabled or will be available when we manually copy them to the release pocket or right after we release the milestone and we copy everything left in -proposed to the release pocket. -- Stéphane Graber Ubuntu developer http://www.ubuntu.com signature.asc Description: OpenPGP digital signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
Cheers, Rick Hi Rick, We certainly want to allow people to upload stuff to -proposed during a milestone week, but I don't agree that we should automatically copy from -proposed to the release pocket during a milestone week. We usually try to release all our images with the same versions of the packages, considering it takes us hours to rebuild everything, having seeded packages land during that time, will lead to images having different version of packages. As for what happened with Alpha 1, we simply asked everyone to upload their packages to -proposed and then cherry picked the packages we actually needed for the release from -proposed and copied them into the release pocket before rebuilding the images (we did that 3 times). If you have to go in and cherry pick packages to copy over, would it not make more sense to simply automatically copy everything over? Everything will be properly built before it is copied over anyway, right? As I understand it, the plan going forward is to have the release pocket be an alias of -proposed on upload, so that everything always lands into -proposed. After something lands in -proposed, is properly built and passes whatever other criteria we'll have, the package will be automatically copied to the release pocket. That last part (copy to the release pocket) would be what we'd block during a milestone week for any package that's seeded. These would be copied on a case by case basis by the release team and the images rebuilt afterwards. This sounds exactly like a freeze. You upload a package and it doesn't land in the distro for a week. That's a serious drag on velocity, and I don't see that it buys us anything but lost productivity and busy work as I described in my initial mail. The essential point is, Ubuntu should be good every day. There should be nothing special about an alpha or beta, it's simply the daily image on some chosen day. Making them special doesn't buy us anything, but has costs. That'd essentially allow any non-seeded package to still flow to the release pocket and be available for everyone. All the others will be available for people running with -proposed enabled or will be available when we manually copy them to the release pocket or right after we release the milestone and we copy everything left in -proposed to the release pocket. I thought it was specifically an anti-goal for people to run proposed during the development release. It would just expose developers to all the problems that we have without having proposed at all, and furthermore means that some developers are using proposed, while others aren't, splitting attention and sewing confusion. Cheers, Rick -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Friday, June 15, 2012 10:28:55 AM Stéphane Graber wrote: On 06/15/2012 10:12 AM, Rick Spencer wrote: Hello all, At UDS I had some hallway discussions about why we freeze for Alphas and Betas, and the fact that I think it is time to drop this practice and rather focus on making Ubuntu good quality each day. Sadly, there was no session on this, thus this email to this list for discussion. I think it is time drop our Freeze practices for the alphas and betas. Here is my reasoning: 1. We are developing tools that allow us to efficiently use -proposed in a way that will ensure we will not have partially built or incompatible components in the release pocket ... ever. Including days we release Alphas and Betas: These blueprints tools to ensure that Ubuntu is not uninstallable or have other problems due to partially built components and such: https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-interme diary https://blueprints.launchpad.net/ubuntu/+spec/other-q-freeze-use-of-propo sed I have been assured that the tools necessary to automate the work of moving components correctly from -proposed to the release will be ready before Alpha 2. 2. We are investing heavily in the daily quality of Ubuntu. For example ... We run the same automated tests on an alpha as we run on a daily: https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/ We tend to archive issues each day: http://people.canonical.com/~ubuntu-archive/testing/quantal_probs.html We ran all the manual ISO tests *before* we released Alpha 1, and we have the capability of doing this at will: http://iso.qa.ubuntu.com/qatracker/milestones/221/builds In short, freezing the archive before an alpha or beta should not actually be contributing to either ensuring the installability of Ubuntu images or ensuring the quality of these images. This implies, therefore, that all the work around freezing, and all the productivity lost during a freeze, actually subtracts from the quality of Ubuntu by reducing our overall velocity for both features and bug fixes, since every day the image is good quality, and Alpha or Beta should be just that day's image tagged appropriately. AIUI, A1 was delivered in such a manner, though without the tooling to ensure that moving from -proposed to the release pocket was efficient and automated. Cheers, Rick Hi Rick, We certainly want to allow people to upload stuff to -proposed during a milestone week, but I don't agree that we should automatically copy from -proposed to the release pocket during a milestone week. We usually try to release all our images with the same versions of the packages, considering it takes us hours to rebuild everything, having seeded packages land during that time, will lead to images having different version of packages. As for what happened with Alpha 1, we simply asked everyone to upload their packages to -proposed and then cherry picked the packages we actually needed for the release from -proposed and copied them into the release pocket before rebuilding the images (we did that 3 times). As I understand it, the plan going forward is to have the release pocket be an alias of -proposed on upload, so that everything always lands into -proposed. After something lands in -proposed, is properly built and passes whatever other criteria we'll have, the package will be automatically copied to the release pocket. That last part (copy to the release pocket) would be what we'd block during a milestone week for any package that's seeded. These would be copied on a case by case basis by the release team and the images rebuilt afterwards. That'd essentially allow any non-seeded package to still flow to the release pocket and be available for everyone. All the others will be available for people running with -proposed enabled or will be available when we manually copy them to the release pocket or right after we release the milestone and we copy everything left in -proposed to the release pocket. This looks complicated. Maybe it will be easier in practice. It also looks like a big shift of work from developers to the release team. Currently it's up to developers to decide what needs uploading to get to the milestone. During a soft freeze there's no action from the release team except coordination. With this mechanism, developers can upload whatever and it's up to the release team to figure out. I can see this being particularly a problem where a small fix is needed for the milestone, but the developer's work would naturally flow to a larger update. With no freeze and it being up to the release team, as Rick says, developer flow would continue and the release team would get stuck without good choices (I'm not sure if in this context there's a mechanism to do direct uploads to the release pocket?). In Debian terms
Re: Releasing Alphas and Betas without freezing
On Fri, Jun 15, 2012 at 4:48 PM, Scott Kitterman ubu...@kitterman.com wrote: On Friday, June 15, 2012 04:41:51 PM Rick Spencer wrote: ... The essential point is, Ubuntu should be good every day. There should be nothing special about an alpha or beta, it's simply the daily image on some chosen day. Making them special doesn't buy us anything, but has costs. ... Then cancel the whole milestone process, grab a daily and call it done. Kinda what I'm saying, yeah. I think we probably still need milestones for targeting bug fixes and features, but we could change that to be keyed off of months. I think there are also widespread external expectations that there are alphas and betas in a software project so I'm not quite ready to advocate for no alpha and betas at all. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On Friday, June 15, 2012 04:57:57 PM Rick Spencer wrote: On Fri, Jun 15, 2012 at 4:48 PM, Scott Kitterman ubu...@kitterman.com wrote: On Friday, June 15, 2012 04:41:51 PM Rick Spencer wrote: ... The essential point is, Ubuntu should be good every day. There should be nothing special about an alpha or beta, it's simply the daily image on some chosen day. Making them special doesn't buy us anything, but has costs. ... Then cancel the whole milestone process, grab a daily and call it done. Kinda what I'm saying, yeah. I think we probably still need milestones for targeting bug fixes and features, but we could change that to be keyed off of months. I think there are also widespread external expectations that there are alphas and betas in a software project so I'm not quite ready to advocate for no alpha and betas at all. I don't think you get to have it both ways. Either we stabilize an image and put a stamp on it and we need some kind of freeze or we don't. Trying to let developers continue to do their work while ignoring the milestone just pushes the problem of getting things fixed for the milestone on the release team. Scott K -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
Le 15/06/2012 17:05, Scott Kitterman a écrit : I don't think you get to have it both ways. Either we stabilize an image and put a stamp on it and we need some kind of freeze or we don't. Trying to let developers continue to do their work while ignoring the milestone just pushes the problem of getting things fixed for the milestone on the release team. Hey, Yeah, you are right there, so if we get working dailies every day do we still need alphas at all? Ideally we would have the automatic testing flagging isos green when they have no issue (with the goal to always have them good) and we would recommend people to just pick the current green iso. Can we just drop the image rolling part of milestones? We still probably want fixed checkpoints in the cycle to review the features, etc but they don't especially need to be linked with a special image... Cheers, Sebastien Bacher -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Releasing Alphas and Betas without freezing
On 06/15/2012 10:26 AM, Sebastien Bacher wrote: Le 15/06/2012 17:05, Scott Kitterman a écrit : I don't think you get to have it both ways. Either we stabilize an image and put a stamp on it and we need some kind of freeze or we don't. Trying to let developers continue to do their work while ignoring the milestone just pushes the problem of getting things fixed for the milestone on the release team. Hey, Yeah, you are right there, so if we get working dailies every day do we still need alphas at all? Ideally we would have the automatic testing flagging isos green when they have no issue (with the goal to always have them good) and we would recommend people to just pick the current green iso. Can we just drop the image rolling part of milestones? We still probably want fixed checkpoints in the cycle to review the features, etc but they don't especially need to be linked with a special image... With our dailies, I've found that the milestones are most useful for planning bug fix landings and feature deliverables. I'd be +1 on dropping alphas all together. If we need some specific test feedback on a given image, we can always issue calls for testing like we've done in the past. However, if we drop alphas, I think we might want to keep a single beta and consider an earlier RC in place of the Beta 2. These releases are typically aimed at getting the less developer savy/bug tolerant users to test and provide feedback, so I could see perhaps a more strenuous QA process put in place for them, i.e. for system integrated/stress testing, versus the typical Unit/Functional automated testing we have reporting to jenkins. It would also be good for the release team to have a couple practice runs before the real deal ;). Just my $0.02 -Robbie -- Robbie Williamson rob...@ubuntu.com robbiew[irc.freenode.net] Don't make me angry...you wouldn't like me when I'm angry. -Bruce Banner -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
