Re: Archive management plans for phone RTM
On Wednesday, June 04, 2014 22:07:24 Adam Conrad wrote: > On Thu, Jun 05, 2014 at 12:00:23AM -0400, Scott Kitterman wrote: > > Will phone RTM dates be better aligned with the Ubuntu development > > schedule in the future? > > I believe this is meant to be a one-off, as the base phone product is > still heavily in development. In the future, a new device should be > easy to enable on, say, 15.04+PPA or whatever, with the PPA bits being > rolled into 15.10, rather than having to do a heavy distro fork, as > the base 15.04 should have everything a phone needs, modulo minor HW > enablement tweaks. > > In other words, this shouldn't look any different from how Canonical > OEM currently handles enablement on new Dell laptops, or whatever, by > targetting the latest LTS, and then using a thin overlay for bits they > need to temporarily fork to make it go. Hopefully less opaque, since whatever OEM services is doing isn't AFAIK public. Scott K -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Archive management plans for phone RTM
On Thu, Jun 05, 2014 at 12:00:23AM -0400, Scott Kitterman wrote: > > Will phone RTM dates be better aligned with the Ubuntu development schedule > in > the future? I believe this is meant to be a one-off, as the base phone product is still heavily in development. In the future, a new device should be easy to enable on, say, 15.04+PPA or whatever, with the PPA bits being rolled into 15.10, rather than having to do a heavy distro fork, as the base 15.04 should have everything a phone needs, modulo minor HW enablement tweaks. In other words, this shouldn't look any different from how Canonical OEM currently handles enablement on new Dell laptops, or whatever, by targetting the latest LTS, and then using a thin overlay for bits they need to temporarily fork to make it go. ... Adam -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Archive management plans for phone RTM
On Thursday, June 05, 2014 01:56:26 Colin Watson wrote: > On Wed, Jun 04, 2014 at 04:54:03PM -0400, Scott Kitterman wrote: > > How will uploads to this new derived distribution get back into > > Ubuntu? > > I think I said something about this in my original mail, but I want to > set a rule that nothing gets into the derived distribution without being > in Ubuntu first (even if not completely landed, they should at least be > in the process, e.g. in -proposed), so it's purely a stabilisation > branch without new work of its own. I had considerable support from > relevant managers last week for this being a general rule, although the > people doing the direct customer engagement note that there may be some > urgent cases where it won't work out. As far as I can tell we should be > able to make sure that those are exceptions rather than the rule. > > When the branch closes, I intend to check through it to ensure that > everything has landed in Ubuntu. If the archive permissions all match > up, then it should be OK to copy source+binaries back on the grounds > that the same set of people were allowed to upload and review it; > otherwise, a few manually-checked source uploads shouldn't be a big > deal. > > Canonical is, as far as I can make out, strongly committed to making > sure that it's possible to continue rolling production phones forward to > newer versions without having to maintain lots of long-lived > per-customer branches, which implies making sure that everything in > stable branches also lands in mainline. That makes me confident that > this won't be forgotten even if I get hit by a bus or something. > > > Will uploads to this new derived distribution be limited to Ubuntu > > developers? If not, how is this being controlled? > > I would very much like the permissions on the derived distribution to be > set to exactly match those on Ubuntu, and the permissions on the derived > series to exactly match those on utopic, so that there's no uncertainty > about whether it's safe to copy packages back and forward. It's not > meant to be a laxer version of the primary archive, but if anything err > in the opposite direction. > > Most landings will be taking place using the CI engine, so the same > issues apply as in the recent conversation on ubuntu-devel. I expect > that by that point we will be using the ground-up rewrite of the engine > that's currently in preparation, and hopefully the task to ensure that > all uploads have some kind of review from somebody with LP > Archive.checkUpload() permission on the package will be completed by > then, at which point there should be no awkward grey areas. That said, > if that task isn't done, or if the CI work slips so that we're still > using the current "CI Train" engine by then, it's no worse than the > situation today. > > We'll presumably also need some version of the IRC notifications that > #ubuntu-release currently gets on queue and image events. I have no > very strong opinion on whether those should be on #ubuntu-release to > keep everything consolidated (especially if the permissions are set to > match so that the same people have access to handle them), or whether > that's too much noise that many of the denizens there don't care about. Personally, I'm fine with it being in #ubuntu-release. I think we're already much too fragmented and people are too comfortable in their own corner of the project. As you know from previous mails, I'm very troubled by the current CI train situation with respect to permissions. I believe it is completely inconsistent with our governance processes and represents, at best, an unknown from a security perspective. No worse than the situation today is no comfort for me (and yes, I know it's the same either way, but I think this is a critical infrastructure issue - the only reason this isn't a Tech Board item right now is it's already on the way to being resolved). Will phone RTM dates be better aligned with the Ubuntu development schedule in the future? Scott K -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: [Ubuntu-phone] Archive management plans for phone RTM
On Wed, Jun 04, 2014 at 06:27:36PM -0700, Alex Chiang wrote: > On Wed, Jun 4, 2014 at 1:20 PM, Colin Watson wrote: > > Can we use this strategy for other Ubuntu flavours or derivatives? > > > > In theory, maybe. It's very much easier to use this strategy for > > phone images, though, because the standard distribution mechanism is > > via system-image channels so we don't need to worry about updating > > sources.list or arranging for mirroring of the derived archive, etc. > > There's also a potentially serious social cost to having lots of > > archive branches, and I wouldn't want to do so except for short > > durations under tight management; I don't want to see the > > Ubuntu-and-flavours development community fracturing unnecessarily. > > Will this mechanism be able to support an arbitrary number of > commercial downstream flavors? My assumption here is that a commercial > downstream will branch off RTM itself, forming RTM'. Possibly, but it's rather heavyweight and I don't know how well it would scale to lots of them, so I'd prefer not to do too many like that at the moment. Just layering a PPA on top should do for most such cases and makes it easier to keep the downstreams up to date, which I gather we probably want to do anyway. With suitable use of apt pinning during image building we could even provide a simple mechanism for downstreams to elect to hold back a given package, without them having to micromanage everything. > I assume the "probably per-device" channel mechanism could be used here? Right, but you do need an archive or combination of archives to build an image from. > Will there be some mechanism to recreate the archive at the RTM or > RTM' branch point? I am thinking of a factory hotfix nightmare where > access to the exact source / package versions will be critical in > debugging the issue. I think we could use most of the same script that we'll need to use to do the initial branching to recover the RTM branchpoint and republish it somewhere else; we just need to keep the manifest or datestamp or whatever. Similarly the same kind of thing would work on a slightly different axis for what you're calling the RTM' branchpoint, if suitably parameterised. Once it's written I'd been planning to put the script in lp:ubuntu-archive-tools for convenient access. (I suppose it would also be possible to create artificial branchpoint series to act as tags, but I think that's overkill; that could end up being a significantly larger set of binaries to keep live on the publishing system when we really just need to be able to republish them as a contingency plan.) We should think about when to mark the rtm-14.09 (or whatever) series as "Current Stable Release" in LP, since that would provide a simple GA kind of tag without much effort, and everything after it could go into rtm-14.09-updates. > We can make the nightmare worse by imagining that it inexplicably > occurs 6 months after initial shipment. For Ubuntu this is fine because we don't garbage-collect pre-release binaries from development series until well after the point anyone is going to care about them. For RTM I'd need to check what the GC policy would be. -- Colin Watson [cjwat...@ubuntu.com] -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: [Ubuntu-phone] Archive management plans for phone RTM
On Wed, Jun 4, 2014 at 1:20 PM, Colin Watson wrote: > > The current plan is therefore: > > * Rick has asked me to take ownership of deciding when to branch for >RTM; note that this decision can be taken retroactively by copying >older versions of packages, so we can say "the image from three days >ago looked good but things have been broken since then, so let's >branch from that". I intend to take this decision based on how >stable images are and how much tension there is between phone work >and development of other parts of Ubuntu; Another factor that should feed into this decision should be the schedule/status of our currently ongoing commercial engagements with our phone OEMs, as they will be another constituency for whom minimized tension/disruption will be critical around that time frame. > * We will then have an "ubuntu-phone" (name TBD) distribution in >Launchpad with a series named something like "rtm-14.09", and will >land changes for RTM there, allowing other parts of Ubuntu to move on >without risk of breaking phone images. We'll have a -proposed pocket >as usual and proposed-migration will operate on it. I'd like >permissions on this distribution to be identical to those in Ubuntu >where possible, but we can discuss details there. There'll be >system-image channels for RTM, probably per-device. The "probably per-device" clause is interesting. Noting it here to feed into questions below. > * We don't want to have to maintain RTM series for a long time. The >plan is to move devices to the next stable branch along as soon as >possible. To facilitate this, the usual rule should be that changes >should only go into this distribution after they're already on their >way into mainline Ubuntu (as for SRUs), to make sure that we don't >end up in a situation where developers land changes on the stable >branch but don't quite get round to landing them in mainline. Agreed that this policy is critical to the long term success of Ubuntu phone images, as reducing/eliminating rootfs fragmentation is one of the key differentiators of our platform vs. Android. > = FAQ = > > Can we use this strategy for other Ubuntu flavours or derivatives? > > In theory, maybe. It's very much easier to use this strategy for > phone images, though, because the standard distribution mechanism is > via system-image channels so we don't need to worry about updating > sources.list or arranging for mirroring of the derived archive, etc. > There's also a potentially serious social cost to having lots of > archive branches, and I wouldn't want to do so except for short > durations under tight management; I don't want to see the > Ubuntu-and-flavours development community fracturing unnecessarily. Will this mechanism be able to support an arbitrary number of commercial downstream flavors? My assumption here is that a commercial downstream will branch off RTM itself, forming RTM'. I assume the "probably per-device" channel mechanism could be used here? A simplifying assumption is that the commercial downstreams will definitely want to follow the policy of "mainline first" and "have devices-in-the-field update from next stable branch ASAP", although be aware that the value of "P" may be on the order of "multiple months". Will there be some mechanism to recreate the archive at the RTM or RTM' branch point? I am thinking of a factory hotfix nightmare where access to the exact source / package versions will be critical in debugging the issue. We can make the nightmare worse by imagining that it inexplicably occurs 6 months after initial shipment. Thanks. -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: [Ubuntu-phone] Archive management plans for phone RTM
On Thu, Jun 05, 2014 at 12:10:25AM +0100, Dimitri John Ledkov wrote: > On 4 June 2014 21:20, Colin Watson wrote: > > Any other questions? > > Will PPAs be enabled on the derivative distribution, such that if I > wish/need I can build packages in the PPA against the RTM suite. Yes, there's no particular reason not to, and they'll have to be in order for the CI engine to work. -- Colin Watson [cjwat...@ubuntu.com] -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Archive management plans for phone RTM
On Wed, Jun 04, 2014 at 04:54:03PM -0400, Scott Kitterman wrote: > How will uploads to this new derived distribution get back into > Ubuntu? I think I said something about this in my original mail, but I want to set a rule that nothing gets into the derived distribution without being in Ubuntu first (even if not completely landed, they should at least be in the process, e.g. in -proposed), so it's purely a stabilisation branch without new work of its own. I had considerable support from relevant managers last week for this being a general rule, although the people doing the direct customer engagement note that there may be some urgent cases where it won't work out. As far as I can tell we should be able to make sure that those are exceptions rather than the rule. When the branch closes, I intend to check through it to ensure that everything has landed in Ubuntu. If the archive permissions all match up, then it should be OK to copy source+binaries back on the grounds that the same set of people were allowed to upload and review it; otherwise, a few manually-checked source uploads shouldn't be a big deal. Canonical is, as far as I can make out, strongly committed to making sure that it's possible to continue rolling production phones forward to newer versions without having to maintain lots of long-lived per-customer branches, which implies making sure that everything in stable branches also lands in mainline. That makes me confident that this won't be forgotten even if I get hit by a bus or something. > Will uploads to this new derived distribution be limited to Ubuntu > developers? If not, how is this being controlled? I would very much like the permissions on the derived distribution to be set to exactly match those on Ubuntu, and the permissions on the derived series to exactly match those on utopic, so that there's no uncertainty about whether it's safe to copy packages back and forward. It's not meant to be a laxer version of the primary archive, but if anything err in the opposite direction. Most landings will be taking place using the CI engine, so the same issues apply as in the recent conversation on ubuntu-devel. I expect that by that point we will be using the ground-up rewrite of the engine that's currently in preparation, and hopefully the task to ensure that all uploads have some kind of review from somebody with LP Archive.checkUpload() permission on the package will be completed by then, at which point there should be no awkward grey areas. That said, if that task isn't done, or if the CI work slips so that we're still using the current "CI Train" engine by then, it's no worse than the situation today. We'll presumably also need some version of the IRC notifications that #ubuntu-release currently gets on queue and image events. I have no very strong opinion on whether those should be on #ubuntu-release to keep everything consolidated (especially if the permissions are set to match so that the same people have access to handle them), or whether that's too much noise that many of the denizens there don't care about. -- Colin Watson [cjwat...@ubuntu.com] -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Re: Archive management plans for phone RTM
On Wednesday, June 04, 2014 21:20:50 Colin Watson wrote: > The Ubuntu phone images are due to hit the release-to-manufacturing > (RTM) point later this cycle. With the pace of the phone work, it > doesn't look practical to deal with this by SRUing all the required > changes into trusty - we're talking about substantial feature > development which probably wouldn't be manageable in trusty even with a > rather liberal SRU policy for phone-specific components. > > I've therefore been thinking about how we might manage to do a stable > phone image somewhere in the middle of a development release cycle; > while the extensive work we've done on rolling quality helps a lot, it > probably won't quite be enough on its own, and we don't want to have the > phone work interfere too much with other Ubuntu developers, so we need > some reasonable way to create a stable branch for RTM, preferably > without just hiding everything off in some non-Launchpad repository that > would require lots of work in our CI tools to handle. I spent some time > in Malta this week discussing this with William Grant of Launchpad and > various phone managers and developers, and I think we now have a > solution that meets the requirements, is technically sound, and might > potentially be useful elsewhere in the future. > > = PPA strategy (shelved) = > > The first idea I had was to copy everything relevant to the phone image > (probably a full build-dependency closure) into a PPA; create a second > PPA which could act as an equivalent of -proposed; adjust Launchpad and > launchpad-buildd, including the new live filesystem building code, to > allow this PPA to be entirely self-contained, so that builds don't refer > to the primary archive; and run whatever syncing and merging tools we > need to propagate changes from the primary archive to the RTM PPAs as > required. > > This very nearly works. There are some publisher performance problems > with putting a very large number of packages in a single PPA, but > William would have been happy to sort those out. However, a more > serious problem is that chroots are shared between utopic in the primary > archive and utopic PPAs, and there's currently no way to disconnect > those, so over time it would become more difficult to build updated > packages for this PPA. While we might be able to fix this, and doing so > would be a workable plan B, it points to PPAs being the wrong model. > > = Derived distribution strategy (candidate) = > > A feature called "derived distributions" was added to Launchpad a few > years ago, originally for use by Linaro: it allows branching an entire > distribution or part of it into a new distribution, with some tools to > help keep them in sync where appropriate. It has never actually been > used properly in production; it was tried once, ran into some small but > fatal problems, but staffing issues at the time meant that these were > never fixed. William has given it a fresh try on a non-production > Launchpad instance and it doesn't seem to be too far off working > reasonably, so we think that we can salvage this work. > > The usual Launchpad copying tools will still work much as before, and > we'll be able to have PPAs based on the new derived distribution. > > > The current plan is therefore: > > * CI and Foundations team members will spend the next two months making >sure that all the tools we need are in place. This includes fixing >up derived distributions, preparing an initialisation script, making >sure that the CI engine (train or airline, as appropriate) can handle >building for a derived distribution, and adding support for >ubuntu-touch image builds from derived distributions. > > * Monitor progress of phone images closely in the early part of August. >We should have a beta image around this point. > > * Rick has asked me to take ownership of deciding when to branch for >RTM; note that this decision can be taken retroactively by copying >older versions of packages, so we can say "the image from three days >ago looked good but things have been broken since then, so let's >branch from that". I intend to take this decision based on how >stable images are and how much tension there is between phone work >and development of other parts of Ubuntu; all other things being >equal I would prefer to keep the branch shorter, but we will need at >least a couple of weeks' clearance before RTM in order that we aren't >debugging our infrastructure right up to the wire. Since I have >pre-booked holiday in mid-August, William Grant will be my backup for >this. > > * As with the shelved PPA plan, we'll only copy packages that are >needed to build the phone image. This makes it easier to track >things and keeps our use of resources down, which has benefits such >as faster publishing. > > * We will then have an "ubuntu-phone" (name TBD) distribution in >Launchpad with a
Re: Archive management plans for phone RTM
I got the ubuntu-phone list's address wrong in my original post. I've bounced the original message to the correct address, but please make sure to replace ubuntu-ph...@lists.ubuntu.com with ubuntu-ph...@lists.launchpad.net when following up. Sorry about that. -- Colin Watson [cjwat...@ubuntu.com] -- Ubuntu-release mailing list Ubuntu-release@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
Archive management plans for phone RTM
The Ubuntu phone images are due to hit the release-to-manufacturing (RTM) point later this cycle. With the pace of the phone work, it doesn't look practical to deal with this by SRUing all the required changes into trusty - we're talking about substantial feature development which probably wouldn't be manageable in trusty even with a rather liberal SRU policy for phone-specific components. I've therefore been thinking about how we might manage to do a stable phone image somewhere in the middle of a development release cycle; while the extensive work we've done on rolling quality helps a lot, it probably won't quite be enough on its own, and we don't want to have the phone work interfere too much with other Ubuntu developers, so we need some reasonable way to create a stable branch for RTM, preferably without just hiding everything off in some non-Launchpad repository that would require lots of work in our CI tools to handle. I spent some time in Malta this week discussing this with William Grant of Launchpad and various phone managers and developers, and I think we now have a solution that meets the requirements, is technically sound, and might potentially be useful elsewhere in the future. = PPA strategy (shelved) = The first idea I had was to copy everything relevant to the phone image (probably a full build-dependency closure) into a PPA; create a second PPA which could act as an equivalent of -proposed; adjust Launchpad and launchpad-buildd, including the new live filesystem building code, to allow this PPA to be entirely self-contained, so that builds don't refer to the primary archive; and run whatever syncing and merging tools we need to propagate changes from the primary archive to the RTM PPAs as required. This very nearly works. There are some publisher performance problems with putting a very large number of packages in a single PPA, but William would have been happy to sort those out. However, a more serious problem is that chroots are shared between utopic in the primary archive and utopic PPAs, and there's currently no way to disconnect those, so over time it would become more difficult to build updated packages for this PPA. While we might be able to fix this, and doing so would be a workable plan B, it points to PPAs being the wrong model. = Derived distribution strategy (candidate) = A feature called "derived distributions" was added to Launchpad a few years ago, originally for use by Linaro: it allows branching an entire distribution or part of it into a new distribution, with some tools to help keep them in sync where appropriate. It has never actually been used properly in production; it was tried once, ran into some small but fatal problems, but staffing issues at the time meant that these were never fixed. William has given it a fresh try on a non-production Launchpad instance and it doesn't seem to be too far off working reasonably, so we think that we can salvage this work. The usual Launchpad copying tools will still work much as before, and we'll be able to have PPAs based on the new derived distribution. The current plan is therefore: * CI and Foundations team members will spend the next two months making sure that all the tools we need are in place. This includes fixing up derived distributions, preparing an initialisation script, making sure that the CI engine (train or airline, as appropriate) can handle building for a derived distribution, and adding support for ubuntu-touch image builds from derived distributions. * Monitor progress of phone images closely in the early part of August. We should have a beta image around this point. * Rick has asked me to take ownership of deciding when to branch for RTM; note that this decision can be taken retroactively by copying older versions of packages, so we can say "the image from three days ago looked good but things have been broken since then, so let's branch from that". I intend to take this decision based on how stable images are and how much tension there is between phone work and development of other parts of Ubuntu; all other things being equal I would prefer to keep the branch shorter, but we will need at least a couple of weeks' clearance before RTM in order that we aren't debugging our infrastructure right up to the wire. Since I have pre-booked holiday in mid-August, William Grant will be my backup for this. * As with the shelved PPA plan, we'll only copy packages that are needed to build the phone image. This makes it easier to track things and keeps our use of resources down, which has benefits such as faster publishing. * We will then have an "ubuntu-phone" (name TBD) distribution in Launchpad with a series named something like "rtm-14.09", and will land changes for RTM there, allowing other parts of Ubuntu to move on without risk of breaking phone images. We'll have a -proposed pocket as usual and proposed-migration w
