Fwd: Re: Fwd: Re: Upgrading pristine-xz on jubany
FYI. Stephane is highly experienced with LXC and does a lot of work with it. On 06/15/2012 04:43 AM, Dmitrijs Ledkovs wrote: Dear Stephane, Can you comment about running quantal LXC container on Lucid host? It would help package importer a lot. Regards, Dmitrijs Hi Dmitrijs, I wouldn't recommend running LXC on 10.04, the kernel lacks some required features and the userspace is really quite behind. Not to mention that these are not secured by apparmor. Instead I'd strongly recommend going with an Ubuntu 12.04 host and running the quantal container on top of that. This way you get the supported LXC stack, with apparmor and a working quantal template. Stéphane Original Message Subject: Re: Upgrading pristine-xz on jubany Date: Fri, 15 Jun 2012 10:32:59 +0200 From: Vincent Ladeuil vila+...@canonical.com To: Barry Warsaw ba...@ubuntu.com CC: ubuntu-distributed-devel@lists.ubuntu.com Barry Warsaw ba...@ubuntu.com writes: On Jun 14, 2012, at 05:21 PM, Vincent Ladeuil wrote: - I'm already running successful tests inside a quantal lxc container :) It has become for many of us not just a nice-to-have but a must-have for Ubuntu development. That's my understanding as well. Here are my last achievements for the week: - I got in touch with pristine-tar maintainers resulting in a trivial bugfix included in 1.25. This is a small step in getting *known* as a primary consumer but it also demonstrates that we can get fixes upstream quickly (1.25 has already been uploaded to sid and quantal). - I got in touch with xz maintainers and a fix is on its way there (many thanks to Lasse Collin for its invaluable help here). This will require an additional fix to pristine-xz which I will submit as soon as I can test the xz fix). With these fixes in place, on quantal, it should remain only 10 pristine-tar import failures out of the current 338 on jubany. Said failures include crazy stuff like tarballs containing files with chmod bits... I haven't looked more precisely how to fix that (and I'm not sure it's worth digging for now). And don't forget that when a package fail to import one release, all the subsequent ones are blocked as well. When we fixed the bzip2 issue last January, ~70 packages were blocked accounting for ~800 releases (don't quote on these numbers, it's just a vague remembering but the scales should be ok). I also have a pending patch for bzr-builddeb that makes it easier to test against pristine-tar failures (will probably submit an mp for that today). Roughly, both builddeb and pristine-tar use temp files so when the import fails, the context is lost. The fix is to save enough of the temp files to be able to re-run pristine-tar alone without re-trying an import (the test cycle is then reduced to seconds instead of hours). With these 3 fixes, we'll be in a far better position to be more reactive to pristine-tar failures in the future (running quantal will then mean that getting fixes will be as simple as stopping the importer, running apt-get upgrade and restart the importer). It also means that testing can occur on quantal without the need to install a bunch of pre-requisites in sync with what is deployed on jubany (which can quickly get totally out of control). I'm still investigating running a quantal lxc container right now on jubany (any feedback about lxc on *lucid* welcome especially known issues that has been fixed in precise). Once I validate this we can look at deploying a quantal lxc container on jubany. Vincent -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Upgrading pristine-xz on jubany
On Fri, 15 Jun 2012 10:32:59 +0200, Vincent Ladeuil vila+...@canonical.com wrote: Barry Warsaw ba...@ubuntu.com writes: On Jun 14, 2012, at 05:21 PM, Vincent Ladeuil wrote: - I'm already running successful tests inside a quantal lxc container :) It has become for many of us not just a nice-to-have but a must-have for Ubuntu development. That's my understanding as well. Here are my last achievements for the week: - I got in touch with pristine-tar maintainers resulting in a trivial bugfix included in 1.25. This is a small step in getting *known* as a primary consumer but it also demonstrates that we can get fixes upstream quickly (1.25 has already been uploaded to sid and quantal). - I got in touch with xz maintainers and a fix is on its way there (many thanks to Lasse Collin for its invaluable help here). This will require an additional fix to pristine-xz which I will submit as soon as I can test the xz fix). With these fixes in place, on quantal, it should remain only 10 pristine-tar import failures out of the current 338 on jubany. That's great work Vincent, I look forward to it being deployed, however that happens :-) Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Upgrading pristine-xz on jubany
Barry Warsaw ba...@python.org writes: On Jun 11, 2012, at 01:54 PM, Vincent Ladeuil wrote: It's unclear to me that we are *known* as a primary consumer. It's clear that we need fixes faster. Being able to report better bugs should helps us getting there and also make us known as a primary consumer. And it provides better service to Ubuntu developers if we can turn around an upgrade of that sort of thing without communication across a Dev/Ops boundary. I agree with this goal too. For the record, I have a trivial fix for pristine-xz that address 2 failures and a wip that could well fix the ~140 left on quantal. Foundations team had a brief discussion about this today, and we'd love to get this fixed. For example, we were looking at packagekit, which is out-of-date because of this problem. If I'm reading the thread correctly, the choices are roughly between upgrading jubany to precise and backporting pristine-tar and xz-utils (and their dependencies) to precise, or in some way getting the importer running on quantal. Yup, inside an lxc container. We're in favor of whichever approach can be accomplished quickest and gives us the highest probability of long-term importer improvement and success :). Reaching the long-term improvement (from which success is derived ;) means reducing the time between a new pristine-tar (xz, whatever) upstream version usable by the importer. Backporting any package so it becomes available on jubany requires (as of today): 1 - backporting to lucid, 2 - providing the lucid package in a ppa, 3 - ask losas/gsas to test it on jubany (requires restarting the importer and waiting for it to process one or several imports successfully), 4 - uninstalling the package from jubany, 5 - make it available to the -cat ppas (used on all data center machines and as such requires to be low risk and without fallouts for other uses), 6 - installing on jubany (and restart the importer). Using a quantal lxc container will allow: 1 - Benefit from quantal packages (more recent versions available at no cost), 2 - Use the same environment for testing and production, 3 - Remove the constraint to be accepted into the -cat archives. This is, IMHO, the main benefit as it reconciles -cat maintainers and udd maintainers constraints. If it helps, one of our guys would be willing to help backport some packages to precise, but I'll let him speak for himself. :) Thanks for the offer ! I'm a bit reluctant to backport pristine-tar and friends because: - I already did it twice in the past (1.17 and 1.20), - pristine-tar just released a newer version (1.25) (including a fix for http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=677241). Had I backported 1.24 that would have been a net loss. - I'm already running successful tests inside a quantal lxc container :) Vincent -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Upgrading pristine-xz on jubany
On 13/06/12 16:48, Barry Warsaw wrote: On Jun 11, 2012, at 01:54 PM, Vincent Ladeuil wrote: It's unclear to me that we are *known* as a primary consumer. It's clear that we need fixes faster. Being able to report better bugs should helps us getting there and also make us known as a primary consumer. And it provides better service to Ubuntu developers if we can turn around an upgrade of that sort of thing without communication across a Dev/Ops boundary. I agree with this goal too. For the record, I have a trivial fix for pristine-xz that address 2 failures and a wip that could well fix the ~140 left on quantal. Foundations team had a brief discussion about this today, and we'd love to get this fixed. For example, we were looking at packagekit, which is out-of-date because of this problem. If I'm reading the thread correctly, the choices are roughly between upgrading jubany to precise and backporting pristine-tar and xz-utils (and their dependencies) to precise, or in some way getting the importer running on quantal. We're in favor of whichever approach can be accomplished quickest and gives us the highest probability of long-term importer improvement and success :). If it helps, one of our guys would be willing to help backport some packages to precise, but I'll let him speak for himself. :) I defiantly would help back porting and validating backports if that's the route chosen. I heavily rely on package-import. It's no longer 'a demo' but really the only way I develop for ubuntu or debian (to have a nice debdiff). Regards, Dmitrijs. -- Regards, Dmitrijs. -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Upgrading pristine-xz on jubany
On Jun 13, 2012, at 05:25 PM, Dmitrijs Ledkovs wrote: I heavily rely on package-import. It's no longer 'a demo' but really the only way I develop for ubuntu or debian (to have a nice debdiff). Here, here. -Barry -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Upgrading pristine-xz on jubany
On 07/06/12 09:15, Vincent Ladeuil wrote: Max Bowsher _...@maxb.eu writes: An LXC just for this feels like an overengineered solution to me. Maintaining a single lxc container config file is less work than backporting a growing number of packages. It has the benefit of isolating the importer into its own container allowing us to rely on packages specifically targeted to our needs when needed without requiring inclusion in -cat archives and brings us closer to the platform used by packagers *avoiding* many failures. ... I.e. using an lxc container allows us to migrate to quantal *now* whereas jubany will get precise... at an unknown date. It also means that when pristine-tar is fixed upstream we get it more quickly deployed with minimal effort. I'd rather rely on the ubuntu community to keep the packages we need up-to-date than dealing with a growing list of packages that need to be backported by a smaller team. Right, I agree we should use stuff straight from Ubuntu when we can, but I don't agree that it's necessarily wrong to customize our deployment practices to get fixes faster. In particular, I've been pinged on IRC about whether we can fix a couple of these packages soon, and it seems to me that I could do just that - and do it this weekend - by installing new xz and pristine-tar versions just for the importer. So I'd quite like to do that. backporting pristine-tar is not enough, xz-utils is also needed, with its own dependencies. It can't be *that* onerous, can it? :-) Moreover, we want to hand over the deployment to Canonical ops so adding more manually installed dependencies doesn't seem to go in the right direction. OK, this is going to be controversial, but I don't think we should be planning to hand the deployment over to Canonical Ops any time soon. The needs, and bugs, of the importer still seem to be evolving at a pace where retaining full control in the hands of those with domain specific knowledge seems to be a far far better option than enforcing a Dev vs. Ops divide. Don't get me wrong, I *did* push to get to the point where we could use branches for bzr and bzr-builddeb (let's ignore distro-info...) but I'd prefer to draw the line there, these are packages we are upstream for and it makes sense to be able to quickly deploy fixes if only to test them before official releases are cut. On the other hand, we're (one of?) the primary consumers of pristine-tar, so it's not really much of a stretch. And it provides better service to Ubuntu developers if we can turn around an upgrade of that sort of thing without communication across a Dev/Ops boundary. Max. signature.asc Description: OpenPGP digital signature -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Upgrading pristine-xz on jubany
Max Bowsher _...@maxb.eu writes: On 06/06/12 13:43, John Arbash Meinel wrote: We're trying to work with webops to get Jubany at least upgraded from Lucid to Precise, Great, what's the estimated timeline for that? Unknown. and then follow it up with something beyond that. We're considering trying to get an LXC running Q, or maybe just backporting Q's pristine-tar for precise. An LXC just for this feels like an overengineered solution to me. Maintaining a single lxc container config file is less work than backporting a growing number of packages. It has the benefit of isolating the importer into its own container allowing us to rely on packages specifically targeted to our needs when needed without requiring inclusion in -cat archives and brings us closer to the platform used by packagers *avoiding* many failures. To put numbers in perspective, the pristine-tar import failures are currently (well, were a week ago, new ones appeared on a weekly basis in the last year): - 327 on jubany (lucid + some backports) - 275 on precise - 147 failures on quantal One way to look at it is to say that 180 failures were caused by the constraint to run lucid on jubany. backporting pristine-tar will also left 147 pending failures waiting for a fix that itself will require another backport or two. I.e. using an lxc container allows us to migrate to quantal *now* whereas jubany will get precise... at an unknown date. It also means that when pristine-tar is fixed upstream we get it more quickly deployed with minimal effort. I'd rather rely on the ubuntu community to keep the packages we need up-to-date than dealing with a growing list of packages that need to be backported by a smaller team. Now, if we can't use quantal, a precise lxc container where we can use a dedicated ppa will still reduce the backport work while still freeing us to respect Canonical ops constraints on them (backporting to lucid-cat means the backport should work for *all* machines in the DCs). Even backporting Q's pristine-tar to precise puts Canonical Ops on the critical path, whereas we could avoid that by doing what we've done for bzr itself and install a suitably recent version just for the pkg_import user. backporting pristine-tar is not enough, xz-utils is also needed, with its own dependencies. Moreover, we want to hand over the deployment to Canonical ops so adding more manually installed dependencies doesn't seem to go in the right direction. Don't get me wrong, I *did* push to get to the point where we could use branches for bzr and bzr-builddeb (let's ignore distro-info...) but I'd prefer to draw the line there, these are packages we are upstream for and it makes sense to be able to quickly deploy fixes if only to test them before official releases are cut. Vincent -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Upgrading pristine-xz on jubany
There are currently more than 300 packages failing to import due to pristine-tar failures, mainly in the pristine-xz step. I tried one, automake1.11, locally on precise, and pristine-xz succeeded. Consequently, I would like to upgrade pristine-xz on jubany. We could do this as a backported package... or we could just skip the need to route this through the Canonical Sysadmins, and install an updated version within /srv/package-import.canonical.com/new/ . I propose to go ahead with the latter, let me know your thoughts. Max. signature.asc Description: OpenPGP digital signature -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Upgrading pristine-xz on jubany
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 6/6/2012 2:29 PM, Max Bowsher wrote: There are currently more than 300 packages failing to import due to pristine-tar failures, mainly in the pristine-xz step. I tried one, automake1.11, locally on precise, and pristine-xz succeeded. Consequently, I would like to upgrade pristine-xz on jubany. We could do this as a backported package... or we could just skip the need to route this through the Canonical Sysadmins, and install an updated version within /srv/package-import.canonical.com/new/ . I propose to go ahead with the latter, let me know your thoughts. Max. Hi Max- Vincent has been looking into this. It looks like upgrading to the pristine-tar in precise brings us from 300 failures to about 250, upgrading to Q's pristine-tar actually brings us to about 150 failures. We're trying to work with webops to get Jubany at least upgraded from Lucid to Precise, and then follow it up with something beyond that. We're considering trying to get an LXC running Q, or maybe just backporting Q's pristine-tar for precise. John =:- -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/PUFgACgkQJdeBCYSNAAMzzACgqM0J7aRMen0kUa0lxbqC3jTw uOkAoLULAISoxeNWzHasyu622qf5wypH =KsRG -END PGP SIGNATURE- -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Upgrading pristine-xz on jubany
On 06/06/12 13:43, John Arbash Meinel wrote: We're trying to work with webops to get Jubany at least upgraded from Lucid to Precise, Great, what's the estimated timeline for that? and then follow it up with something beyond that. We're considering trying to get an LXC running Q, or maybe just backporting Q's pristine-tar for precise. An LXC just for this feels like an overengineered solution to me. Even backporting Q's pristine-tar to precise puts Canonical Ops on the critical path, whereas we could avoid that by doing what we've done for bzr itself and install a suitably recent version just for the pkg_import user. Max. signature.asc Description: OpenPGP digital signature -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
