Re: The archive now supports xz compression
Roger Leigh dixit: Possibly a stupid question here but: Given that we are now autosigning builds, why can't the slower arches use gzip, and then after upload they could be recompressed with xz (and resigned) on a faster arch? xz -2 is fast enough on m68k (IIRC, compresses not worse than bzip2 and is not slower than gzip). I think you could justify even something like xz --lzma=preset=3,dict=8M for speed freaks. However, see my mail in pine.bsm.4.64l.1112192227580@herc.mirbsd.org for a justification for using xz -2 for most packages and xz -2e for packages with really large data (that is not already precompressed, such as gzip’d manpages) on not-fast architectures (with an allowance for -6[e] for debug, but those wouldn’t be on the CDs anyway). bye, //mirabilos -- Solange man keine schmutzigen Tricks macht, und ich meine *wirklich* schmutzige Tricks, wie bei einer doppelt verketteten Liste beide Pointer XORen und in nur einem Word speichern, funktioniert Boehm ganz hervorragend. -- Andreas Bogk über boehm-gc in d.a.s.r -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/pine.bsm.4.64l.1205221205140.23...@herc.mirbsd.org
Re: Spending Debian money for porter boxes [Was: The archive now supports xz compression]
On Sun, 21 Aug 2011, Bastien ROUCARIES wrote: I concur and I'll be happy to approve such usage of Debian money. FWIW, what is needed to make this kind of things happen is not really money. What is missing is rather a bit of coordination of people that: 1) keep track of what hardware we need, Could this point tracked using a special package on the bts (or a tag) ? It will also help for finding printing device with hard to reproduce bug. If whoever is willing to track this wants to make use of the BTS for it, I have no objections, and will create such a pseudopackage following their directions. Don Armstrong -- I don't care how poor and inefficient a little country is; they like to run their own business. I know men that would make my wife a better husband than I am; but, darn it, I'm not going to give her to 'em. -- The Best of Will Rogers http://www.donarmstrong.com http://rzlab.ucr.edu -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110824015047.gp5...@rzlab.ucr.edu
Re: Spending Debian money for porter boxes [Was: The archive now supports xz compression]
Hi, On Sat Aug 20, 2011 at 20:45:18 +0200, Mike Hommey wrote: On Sun, Aug 14, 2011 at 09:19:02PM +0200, Lucas Nussbaum wrote: On 11/08/11 at 19:52 +, Philipp Kern wrote: On 2011-08-11, Adam Borowski kilob...@angband.pl wrote: Think of both user systems and the Debian buildds which will waste more time - an especially bad problem on slower architectures. The gain is especially meaningful for slower architectures, as they tend to have less disk space and slower network links (arm tends to be used in phones). No extra memory is needed -- decompression is not done in parallel with memory-hungry stages of dpkg's work. The decompression, merely 2.5 times slower than with gzip, is a tiny fraction of what dpkg takes. It takes a lot longer to compress on slower architectures (i.e. on the buildds), though. You could've built a whole package in that time. (Resorting to your style of argument.) Wouldn't it be better to get more buildds for those archs, then? That would be a totally appropriate use of Debian money... _Personaly_ i think, Debian has more than enough machines. Adding more machines just to have packages build faster is not a proper solution. Please keep in mind you need someone to admin those machines. Also please keep in mind that most hardware for architectures like arm, mips, mipsel we got are not 'end user hardware' but development boards and thus have other specs than the end user hardware. The kernel team in the past and in the present denied to build yet another kernel flavour for those machines, and this way DSA needs to take care of having up to date kernels for those machines. This is a no-option. IF you want to have more hardware, find boards that are/will be supported by the kernel team and ship them to existing hosting places. Yet another hosting in someones basement is not an option for a project machines. Just my 2¢, Martin -- Martin Zobel-Helas zo...@debian.org | Debian System Administrator Debian GNU/Linux Developer | Debian Listmaster GPG key http://go.debian.net/B11B627B | GPG Fingerprint: 6B18 5642 8E41 EC89 3D5D BDBB 53B1 AC6D B11B 627B -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110822071855.gb13...@ftbfs.de
Re: Spending Debian money for porter boxes [Was: The archive now supports xz compression]
Hi! Am 21.08.2011 18:59, schrieb Stefano Zacchiroli: [ more powerful hardware needed ] What is missing is rather a bit of coordination of people that: [..] Any taker? Well, according to [1] and [2], we have some hardware donations coordinators, and at [3] we have a list of needed hardware. I'm not involved in that area, but what you describe sounds actually pretty much like what they do / could do. So it would be interesting to know, why that approach currently doesn't work, and how that can be improved. 1: http://www.debian.org/intro/organization 2: http://www.debian.org/donations#equipment_donations 3: http://www.debian.org/misc/hardware_wanted Best regards, Alexander -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4e521876.9090...@schmehl.info
Re: Spending Debian money for porter boxes [Was: The archive now supports xz compression]
On Mon, Aug 22, 2011 at 10:51:02AM +0200, Alexander Reichle-Schmehl wrote:
What is missing is rather a bit of coordination of people that:
[..]
Any taker?
Well, according to [1] and [2], we have some hardware donations
coordinators, and at [3] we have a list of needed hardware.
I'm not involved in that area, but what you describe sounds actually
pretty much like what they do / could do. So it would be interesting to
know, why that approach currently doesn't work, and how that can be
improved.
I think that I've explained it in the [..] you have snipped, actually
:). But let me briefly outline the difference among a donation contact
point (which is what we have now, even though the name is slightly
different) and an hardware coordination team.
The former act as a contact point for external vendors that, without us
reaching out to them, wants to ask Debian we have that, are you
interested?. According to some feedback I got from people who are
behind that alias, it seems to be pretty useless. The kind of mails it
gets is I've an hold 486 dx with mouse and keyboard, do you need it?.
The hardware *coordination* team do actual coordination and tracking of
what we need — which tends to be rather specific hardware. Additionally,
the team should reach out to potential donors proactively, and fall back
to prepare buy orders via interaction with the DPL.
Again, I'm a bit outside of my territory here and I lack first-hand
experience. People from -admin and DSA can be more precise and helpful
than me.
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Quando anche i santi ti voltano le spalle, | . |. I've fans everywhere
ti resta John Fante -- V. Capossela ...| ..: |.. -- C. Adams
signature.asc
Description: Digital signature
Re: Spending Debian money for porter boxes [Was: The archive now supports xz compression]
On Sat, Aug 20, 2011 at 08:45:18PM +0200, Mike Hommey wrote:
Wouldn't it be better to get more buildds for those archs, then?
That would be a totally appropriate use of Debian money...
Speaking of which. It would also totally be an appropriate use of Debian
money to get new porter boxes that fit the buildds. Most of the non x86
porter boxes are pathetically slow, which is even sadder when you know
the buildd boxes for the same architectures are an order of magnitude
faster (and I'm almost not exagerating).
I concur and I'll be happy to approve such usage of Debian money. FWIW,
what is needed to make this kind of things happen is not really money.
What is missing is rather a bit of coordination of people that: 1) keep
track of what hardware we need, 2) take the time to do a first check if
we have vendors interested in donating that hardware, 3) fallback to get
a quote of the needed hardware, 4) get back to the DPL saying we need
that and it costs XXX, can we have it?.
That sort of hw coordination is what we need at present. Without it,
just saying $foo would be a good use of Debian money won't fly.
And while we are at it, DSA is well aware of the need of such a role and
I've on my TODO list to call for help for people interested on working
together with DSA on the above topics. I was planning to mention it in
the forthcoming bits from the DPL, but I guess that repeating it here
won't hurt...
Any taker?
Cheers.
PS Cc:-ing -admin that can for sure explain better than me what the hw
coordination role should be about
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Quando anche i santi ti voltano le spalle, | . |. I've fans everywhere
ti resta John Fante -- V. Capossela ...| ..: |.. -- C. Adams
signature.asc
Description: Digital signature
Re: Spending Debian money for porter boxes [Was: The archive now supports xz compression]
On Sun, Aug 21, 2011 at 6:59 PM, Stefano Zacchiroli lea...@debian.org wrote:
On Sat, Aug 20, 2011 at 08:45:18PM +0200, Mike Hommey wrote:
Wouldn't it be better to get more buildds for those archs, then?
That would be a totally appropriate use of Debian money...
Speaking of which. It would also totally be an appropriate use of Debian
money to get new porter boxes that fit the buildds. Most of the non x86
porter boxes are pathetically slow, which is even sadder when you know
the buildd boxes for the same architectures are an order of magnitude
faster (and I'm almost not exagerating).
I concur and I'll be happy to approve such usage of Debian money. FWIW,
what is needed to make this kind of things happen is not really money.
What is missing is rather a bit of coordination of people that: 1) keep
track of what hardware we need,
Could this point tracked using a special package on the bts (or a tag)
? It will also help for finding printing device with hard to reproduce
bug.
Bug that depend of this need to by blocked by hardware bug.
2) take the time to do a first check if
we have vendors interested in donating that hardware,
Using (abusing) tag upstream on the bts ?
3) fallback to get
a quote of the needed hardware,
using tag fixed-upstream ?
4) get back to the DPL saying we need
that and it costs XXX, can we have it?
Using tag confirmed ?
That sort of hw coordination is what we need at present. Without it,
just saying $foo would be a good use of Debian money won't fly.
And while we are at it, DSA is well aware of the need of such a role and
I've on my TODO list to call for help for people interested on working
together with DSA on the above topics. I was planning to mention it in
the forthcoming bits from the DPL, but I guess that repeating it here
won't hurt...
Any taker?
Cheers.
PS Cc:-ing -admin that can for sure explain better than me what the hw
coordination role should be about
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Quando anche i santi ti voltano le spalle, | . |. I've fans everywhere
ti resta John Fante -- V. Capossela ...| ..: |.. -- C. Adams
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/CAE2SPAZT=zvo5hoqmvdoo5srfj34bgfzpkuhcvrvhqmofg+...@mail.gmail.com
Spending Debian money for porter boxes [Was: The archive now supports xz compression]
On Sun, Aug 14, 2011 at 09:19:02PM +0200, Lucas Nussbaum wrote: On 11/08/11 at 19:52 +, Philipp Kern wrote: On 2011-08-11, Adam Borowski kilob...@angband.pl wrote: Think of both user systems and the Debian buildds which will waste more time - an especially bad problem on slower architectures. The gain is especially meaningful for slower architectures, as they tend to have less disk space and slower network links (arm tends to be used in phones). No extra memory is needed -- decompression is not done in parallel with memory-hungry stages of dpkg's work. The decompression, merely 2.5 times slower than with gzip, is a tiny fraction of what dpkg takes. It takes a lot longer to compress on slower architectures (i.e. on the buildds), though. You could've built a whole package in that time. (Resorting to your style of argument.) Wouldn't it be better to get more buildds for those archs, then? That would be a totally appropriate use of Debian money... Speaking of which. It would also totally be an appropriate use of Debian money to get new porter boxes that fit the buildds. Most of the non x86 porter boxes are pathetically slow, which is even sadder when you know the buildd boxes for the same architectures are an order of magnitude faster (and I'm almost not exagerating). Mike -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110820184518.ga8...@glandium.org
Re: The archive now supports xz compression
Le Mon, Aug 15, 2011 at 01:48:50AM +0200, Adam Borowski a écrit : * A year ago, I repacked CD1, .xz took 66% space needed by .gz. This time, on the whole archive, gains are somewhat smaller: 72%. I guess that CD1 is code-heavy while packages of lower priorities tend to have more data. Also, many files in /usr/share/doc are gzipped as per §12.3; that can prevent to get the full benefit of xz compression. In some of my packages containing mostly such files, the benefit of switching to xz is almost null. I wonder if it still makes sense to compress these files by default: - Most systems have enough space to keep them uncompressed, - others systems just do not install these files, - some filesystems are compressed on the fly, - the binary packages themselves are compressed. Perhaps we could consider allowing xz compression or no compression at all for the files in /usr/share/doc, especially when they are all contained in a dedicated package that is not dispensable. Have a nice day, -- Charles Plessy Debian Med packaging team, http://www.debian.org/devel/debian-med Tsurumi, Kanagawa, Japan -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110815081655.gb2...@merveille.plessy.net
Re: The archive now supports xz compression
Hi, 2011/8/15 Eduard Bloch e...@gmx.de: #include hallo.h * Roger Leigh [Sun, Aug 14 2011, 11:01:11PM]: On Sun, Aug 14, 2011 at 09:19:02PM +0200, Lucas Nussbaum wrote: On 11/08/11 at 19:52 +, Philipp Kern wrote: Wouldn't it be better to get more buildds for those archs, then? That would be a totally appropriate use of Debian money... Possibly a stupid question here but: Given that we are now autosigning builds, why can't the slower arches use gzip, and then after upload they could be recompressed with xz (and resigned) on a faster arch? This would allow xz compression on all arches, but not require slow arches to actually do the xz compression themselves. Yeah, but if we got that far then we could easily create a remote compressing service for such systems. Can be easily achieved by configuring xz as service in inetd.conf and hacking dpkg-dev to run something like socket -q localRocketFastSystem instead of xz on the build machine. Maybe we don't have to hack dpkg-dev. Maybe could replace xz with netcat on slow build machines. Cheers, Balint -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/cak0odpwvrqkcvzfvys0sspnxqsiuu-kzrz5kjztoutoydaw...@mail.gmail.com
Re: The archive now supports xz compression
On Mon, Aug 15, 2011 at 06:38:28AM +0200, Tollef Fog Heen wrote: ]] Adam Borowski | Does someone have an estimate how many core-hours would an archive rebuild | on such a machine take? Folks on IRC quoted numbers like 340, 240 on a | very fast box, more like 1500 -- too divergent for my liking. The | first number, 340, would mean switching to xz exclusively would increase | average build time by ~5%. About 14 on a 2x6-core Intel Xeon X5650 (2.67GHz) (with HT), lots of memory, fast disks. Ie, 250-300ish core-hours, sounds good. -- 1KB // Yo momma uses IPv4! -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110815091655.ga28...@angband.pl
Re: The archive now supports xz compression
Hi there! On Mon, 15 Aug 2011 10:16:55 +0200, Charles Plessy wrote: Also, many files in /usr/share/doc are gzipped as per §12.3; [...] - Most systems have enough space to keep them uncompressed, Which alone is not a good reason to not compress them. Perhaps we could consider allowing xz compression or no compression at all for the files in /usr/share/doc, especially when they are all contained in a dedicated package that is not dispensable. What about zless Co.? Are they available for xz as well? Thx, bye, Gismo / Luca pgpRNIe0i86Gz.pgp Description: PGP signature
Re: The archive now supports xz compression
On Mon, Aug 15, 2011 at 11:25:42AM +0200, Luca Capello wrote: What about zless Co.? Are they available for xz as well? xz-utils contains xzless, xzcat etc. -- WBR, wRAR signature.asc Description: Digital signature
Re: The archive now supports xz compression
Hi, I'm happy to hear xz support. Some font packages can get huge profit with this (e.g. fonts-vlgothic: 4924KB - 2132KB (half! :) On Thu, 11 Aug 2011 19:52:46 + (UTC) Philipp Kern tr...@philkern.de wrote: It takes a lot longer to compress on slower architectures (i.e. on the buildds), though. If there's a problem with slowness and CPU usage for some architecture, how about enabling xz compression with specifying architecture such as only all, i386 and amd64? i386 and amd64 have enough CPU power, and most of the users use those arch, so it's efficient. all has arch-independent data, compression would be done in i386 or amd64 on maintainers' desk, so no harm. If maintainer should decide to use xz compression or not, we should modify many debian/rules file to enable it. It's such a pain. - xz compression enable by default (no pain for package maintainers) - for limited arch - all, i386, amd64 (efficient) (or add kfreebsd-i386, kfreebsd-amd64, hurd-i386) -- Regards, Hideki Yamane henrich @ debian.or.jp/org http://wiki.debian.org/HidekiYamane -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110815212256.a52100b5a48eb482504e2...@debian.or.jp
Re: The archive now supports xz compression
On 11/08/11 at 19:52 +, Philipp Kern wrote: On 2011-08-11, Adam Borowski kilob...@angband.pl wrote: Think of both user systems and the Debian buildds which will waste more time - an especially bad problem on slower architectures. The gain is especially meaningful for slower architectures, as they tend to have less disk space and slower network links (arm tends to be used in phones). No extra memory is needed -- decompression is not done in parallel with memory-hungry stages of dpkg's work. The decompression, merely 2.5 times slower than with gzip, is a tiny fraction of what dpkg takes. It takes a lot longer to compress on slower architectures (i.e. on the buildds), though. You could've built a whole package in that time. (Resorting to your style of argument.) Wouldn't it be better to get more buildds for those archs, then? That would be a totally appropriate use of Debian money... Of course, it might require finding more buildd maintainers. But I must admit that I have no idea what buildd admins spend time on, and how it's possible to help them. For example, if we tried to have more identical buildds, instead of just one of each model, would it reduce the workload of buildd admins significantly? Lucas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110814191902.ga3...@xanadu.blop.info
Re: The archive now supports xz compression
On Sun, Aug 14, 2011 at 09:19:02PM +0200, Lucas Nussbaum wrote: On 11/08/11 at 19:52 +, Philipp Kern wrote: On 2011-08-11, Adam Borowski kilob...@angband.pl wrote: Think of both user systems and the Debian buildds which will waste more time - an especially bad problem on slower architectures. The gain is especially meaningful for slower architectures, as they tend to have less disk space and slower network links (arm tends to be used in phones). No extra memory is needed -- decompression is not done in parallel with memory-hungry stages of dpkg's work. The decompression, merely 2.5 times slower than with gzip, is a tiny fraction of what dpkg takes. It takes a lot longer to compress on slower architectures (i.e. on the buildds), though. You could've built a whole package in that time. (Resorting to your style of argument.) Wouldn't it be better to get more buildds for those archs, then? That would be a totally appropriate use of Debian money... Of course, it might require finding more buildd maintainers. But I must admit that I have no idea what buildd admins spend time on, and how it's possible to help them. For example, if we tried to have more identical buildds, instead of just one of each model, would it reduce the workload of buildd admins significantly? Replying more generally than the problem at hand, given that you did the same. :) The problem on those arches is finding stable buildds. We want boards that support more RAM that they normally do (e.g. on MIPS) or some with SATA disks and good throughput (e.g. on ARM). So you get to use development boards, which you can only get hosted by the company doing them (e.g. for ARM), or which are not stable unless heavily patched (because they normally use a custom kernel by the vendor, e.g. for MIPS). And then there are the boards which were just buggy CPU-wise (older Loongson ones). Andi Barth bought some, but it's unhelpful if you're able to crash them as a normal user due to a pipelining bug. The situation is getting better for those machines we have. DSA would like to run Debian kernels on them, but it seems to be hard. Hence you get to run some kernels with specific patchsets to be as near to a released kernel as possible, with some extensions to let the hardware run stable (or boot at all). Yes, there is a point where you don't want to add more buildds instead of getting faster ones. There's a bunch of locking in the wanna-build database, which used to be much worse, but which is still present in some way. There's the overhead of managing the machines for DSA. But the addition of four(?) additional builders hosted at ARM, which were identical, was great. Especially if they are all set up alike, you can just put up a clusterssh session to handle them. It's just that we now lack a bit of redundancy because all of them are hosted at the same location… Kind regards Philipp Kern signature.asc Description: Digital signature
Re: The archive now supports xz compression
On Sun, Aug 14, 2011 at 09:19:02PM +0200, Lucas Nussbaum wrote: Of course, it might require finding more buildd maintainers. But I must admit that I have no idea what buildd admins spend time on, and how it's possible to help them. A life in the day of a buildd maintainer would not be a bad continuation to the IRC QA sessions we've already had. -- Freedom-based blog/wiki/web hosting: http://www.branchable.com/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110814215211.ga26...@havelock.liw.fi
Re: The archive now supports xz compression
On Sun, Aug 14, 2011 at 09:19:02PM +0200, Lucas Nussbaum wrote: On 11/08/11 at 19:52 +, Philipp Kern wrote: On 2011-08-11, Adam Borowski kilob...@angband.pl wrote: Think of both user systems and the Debian buildds which will waste more time - an especially bad problem on slower architectures. The gain is especially meaningful for slower architectures, as they tend to have less disk space and slower network links (arm tends to be used in phones). No extra memory is needed -- decompression is not done in parallel with memory-hungry stages of dpkg's work. The decompression, merely 2.5 times slower than with gzip, is a tiny fraction of what dpkg takes. It takes a lot longer to compress on slower architectures (i.e. on the buildds), though. You could've built a whole package in that time. (Resorting to your style of argument.) Wouldn't it be better to get more buildds for those archs, then? That would be a totally appropriate use of Debian money... Possibly a stupid question here but: Given that we are now autosigning builds, why can't the slower arches use gzip, and then after upload they could be recompressed with xz (and resigned) on a faster arch? This would allow xz compression on all arches, but not require slow arches to actually do the xz compression themselves. Regards, Roger -- .''`. Roger Leigh : :' : Debian GNU/Linux http://people.debian.org/~rleigh/ `. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/ `-GPG Public Key: 0x25BFB848 Please GPG sign your mail. signature.asc Description: Digital signature
Re: The archive now supports xz compression
On Sun, Aug 14, 2011 at 09:19:02PM +0200, Lucas Nussbaum wrote: On 11/08/11 at 19:52 +, Philipp Kern wrote: On 2011-08-11, Adam Borowski kilob...@angband.pl wrote: Think of both user systems and the Debian buildds which will waste more time - an especially bad problem on slower architectures. The gain is especially meaningful for slower architectures, as they tend to have less disk space and slower network links (arm tends to be used in phones). No extra memory is needed -- decompression is not done in parallel with memory-hungry stages of dpkg's work. The decompression, merely 2.5 times slower than with gzip, is a tiny fraction of what dpkg takes. It takes a lot longer to compress on slower architectures (i.e. on the buildds), though. You could've built a whole package in that time. (Resorting to your style of argument.) Wouldn't it be better to get more buildds for those archs, then? That would be a totally appropriate use of Debian money... While having more buildds is always nice, it doesn't seem to me that they would be necessary just to switch to xz. I gathered some data: * A repack of the whole archive (amd64+all main, ~40GB) took close to three hours on a 6xPhenomII 2.8GHz box (ar p|gzip/bzip2 -d|xz). Does someone have an estimate how many core-hours would an archive rebuild on such a machine take? Folks on IRC quoted numbers like 340, 240 on a very fast box, more like 1500 -- too divergent for my liking. The first number, 340, would mean switching to xz exclusively would increase average build time by ~5%. * armel Cortex-A8 600MHz does xz consistently 12.1 times slower than one core of the above box (on a big compressible and a big uncompressible file), that's ~2.6 times slower per-MHz. Glancing at build logs of a few randomly chosen packages, I see armel builds taking respectively 16.9, 13.1, 18.8 and 15.1 times longer. Not sure what are the actual speeds of buildds, but it looks like armel would be affected by less than the above 5%. * A year ago, I repacked CD1, .xz took 66% space needed by .gz. This time, on the whole archive, gains are somewhat smaller: 72%. I guess that CD1 is code-heavy while packages of lower priorities tend to have more data. Raw data: http://angband.pl/tmp/rexz/gzip.gz and http://angband.pl/tmp/rexz/bzip2.gz (these numbers are data.tar.* alone) An empty package is bigger (180%: 36 vs 20 bytes). Packages with sizes 1000 bytes: 85%. Packages with sizes 1 bytes: 76%. * Compression time seems to be linear for all sizes that can be measured without tricks. * It is possible to repack .deb files after they are built. * Busybox (and thus d-i) can be compiled with xz support. Size-wise it's a clear gain (dpkg.deb saves 883008 bytes, apt.deb 751967 ...). This is the only place where the memory cost could possibly matter, though. * Other distributions that could run debootstrap have all since switched to xz[1], so it's mandatory there already. A possible concern would be deboostrapping from an outdated install of those. There seems to be a lot of confusion like do we have any guideline for the sort of space savings which justify using xz?. The d-d-a post in particular seems to suggest only big packages should be switched; my data suggests that switching many small packages is not significantly different from switching a single big one. Thus, I'd say it'd be simpler to just switch everything. [1]. Among major ones, it seems only Gentoo ships non-xz images, but xz is included in their system set (our essential). -- 1KB // Yo momma uses IPv4! signature.asc Description: Digital signature
Re: The archive now supports xz compression
On Sun, Aug 14, 2011 at 11:01:11PM +0100, Roger Leigh wrote: Possibly a stupid question here but: Given that we are now autosigning builds, why can't the slower arches use gzip, and then after upload they could be recompressed with xz (and resigned) on a faster arch? This would allow xz compression on all arches, but not require slow arches to actually do the xz compression themselves. Not a bad idea. I don't believe it is necessary -- in the estimate I've written in this thread the cost is below 5% of average build time, but if you consider that 5% to be unacceptable, repacking would work. -- 1KB // Yo momma uses IPv4! signature.asc Description: Digital signature
Re: The archive now supports xz compression
Raphael Hertzog wrote: Nope, sorry. I was referring to things like GNOME shipping only .tar.xz. I mean they would not take such a decision if getting an xz decompressor was a pain on many systems. There is a large distance between systems on which users are likely to build gnome from scratch (which involves installing many dependencies anyway), and systems on which users may want to run debootstrap. The latter can include embedded systems that don't have a compiler and are not of a common architecture, which makes xz difficult to install. It would be best if debootstrap's dependencies remained limited to things that are commonly enabled in busybox. -- see shy jo signature.asc Description: Digital signature
Re: The archive now supports xz compression
#include hallo.h * Roger Leigh [Sun, Aug 14 2011, 11:01:11PM]: On Sun, Aug 14, 2011 at 09:19:02PM +0200, Lucas Nussbaum wrote: On 11/08/11 at 19:52 +, Philipp Kern wrote: Wouldn't it be better to get more buildds for those archs, then? That would be a totally appropriate use of Debian money... Possibly a stupid question here but: Given that we are now autosigning builds, why can't the slower arches use gzip, and then after upload they could be recompressed with xz (and resigned) on a faster arch? This would allow xz compression on all arches, but not require slow arches to actually do the xz compression themselves. Yeah, but if we got that far then we could easily create a remote compressing service for such systems. Can be easily achieved by configuring xz as service in inetd.conf and hacking dpkg-dev to run something like socket -q localRocketFastSystem instead of xz on the build machine. Regards, Eduard. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110815000109.ga11...@rotes76.wohnheim.uni-kl.de
Re: The archive now supports xz compression
]] Adam Borowski | Does someone have an estimate how many core-hours would an archive rebuild | on such a machine take? Folks on IRC quoted numbers like 340, 240 on a | very fast box, more like 1500 -- too divergent for my liking. The | first number, 340, would mean switching to xz exclusively would increase | average build time by ~5%. About 14 on a 2x6-core Intel Xeon X5650 (2.67GHz) (with HT), lots of memory, fast disks. -- Tollef Fog Heen UNIX is user friendly, it's just picky about who its friends are -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87d3g7mgqj@qurzaw.varnish-software.com
Re: The archive now supports xz compression
On Thu, 11 Aug 2011, Colin Watson wrote: Since hardcoding gzip for base packages seems to be a bit brittle, we need to work towards allowing xz usage in debian-installer and accept it as a dependency for deboostrap on non-Debian systems (I don't think it's a big issue, xz is portable and is more and more widely used). Can you quantify that? I don't have hard numbers for the non-Debian systems where people report running debootstrap; perhaps you do ... Nope, sorry. I was referring to things like GNOME shipping only .tar.xz. I mean they would not take such a decision if getting an xz decompressor was a pain on many systems. ftp.gnu.org also ships .tar.xz nowadays. I guess that a first step is to have an xz udeb... No. busybox already has an unxz applet, so if we choose to do this then we should enable that applet in busybox-udeb, not add a separate udeb. Ok, then maybe a first step is to verify if deboostrap + busybox's unxz work correctly together to setup a debian chroot with all base packages recompiled with xz. Cheers, -- Raphaël Hertzog ◈ Debian Developer Follow my Debian News ▶ http://RaphaelHertzog.com (English) ▶ http://RaphaelHertzog.fr (Français) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110812125032.ge23...@rivendell.home.ouaza.com
Re: The archive now supports xz compression
On Fri, Aug 12, 2011 at 02:50:32PM +0200, Raphael Hertzog wrote: On Thu, 11 Aug 2011, Colin Watson wrote: Can you quantify that? I don't have hard numbers for the non-Debian systems where people report running debootstrap; perhaps you do ... Nope, sorry. I was referring to things like GNOME shipping only .tar.xz. I mean they would not take such a decision if getting an xz decompressor was a pain on many systems. I would love to have such faith. :-) -- Colin Watson [cjwat...@debian.org] -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110812131622.ga5...@riva.dynamic.greenend.org.uk
Re: The archive now supports xz compression
On Thu, Aug 11, 2011 at 05:12:36PM +0200, Ansgar Burchardt wrote: Hi, The archive software now accepts packages using xz for compression in addition to gzip and bzip2 for both source and binary packages. Hurray! please only use xz (or bzip2 for that matter) if your package really profits from its usage (for example, it provides a significant space saving). While those methods may compress better they often use more CPU time to do so and a very small decrease in package size is hardly worth the extra effort placed on slower systems. This is very bad advice. Do you remember my joke package goodbye less than two weeks ago? If you compared the optimized[1] debhelper-less dpkg-less code to standard slow debhelper, you lose 2.6 seconds per package[2]. In that time you can compress 8MB (and decompress that in 0.09s). Thus, if you care about amounts of CPU time that small, you can as well go through the whole archive replacing debhelper with goodbye, for all packages smaller than that 8MB. The cost is roughly linear, too -- so compressing a 20KB package costs about nothing. And what do we gain in return? A massive decrease of the archive size -- both disk space and bandwidth. Regular packages compress twice as much with xz as with gzip, with uncompressible data in the mix the average was 2/3 for amd64 CD1. Thus, I'd strongly recommend just compressing everything with xz, on all architectures. Preferably, as a default in dpkg-dev. Think of both user systems and the Debian buildds which will waste more time - an especially bad problem on slower architectures. The gain is especially meaningful for slower architectures, as they tend to have less disk space and slower network links (arm tends to be used in phones). No extra memory is needed -- decompression is not done in parallel with memory-hungry stages of dpkg's work. The decompression, merely 2.5 times slower than with gzip, is a tiny fraction of what dpkg takes. Please remember that packages in the base system[1] (and dependencies) *must* currently use gzip as otherwise debootstrap will be unable to install a system. 1: Meaning everything with Priority: required. I'm not as strongly opinionated here, but I guess decreasing the size of d-i images would be a huge win as well. Meow! [1]. Three calls of system() could be avoided for little effort and two more for substantial effort -- and all of them could be turned into execve(), but otherwise, it's pretty much as fast as you can get. With package build time below a single disk seek, further optimizations are pointless. The point of goodbye was abuse of policy and buzzword compliancy, not speed. [2]. Debhelper costs are surprisingly constant, even between a copy a single file package and full-blown autoconf+C ones. -- 1KB // Yo momma uses IPv4! -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110811183010.ga27...@angband.pl
Re: The archive now supports xz compression
On 2011-08-11, Adam Borowski kilob...@angband.pl wrote: Think of both user systems and the Debian buildds which will waste more time - an especially bad problem on slower architectures. The gain is especially meaningful for slower architectures, as they tend to have less disk space and slower network links (arm tends to be used in phones). No extra memory is needed -- decompression is not done in parallel with memory-hungry stages of dpkg's work. The decompression, merely 2.5 times slower than with gzip, is a tiny fraction of what dpkg takes. It takes a lot longer to compress on slower architectures (i.e. on the buildds), though. You could've built a whole package in that time. (Resorting to your style of argument.) Please remember that packages in the base system[1] (and dependencies) *must* currently use gzip as otherwise debootstrap will be unable to install a system. 1: Meaning everything with Priority: required. I'm not as strongly opinionated here, but I guess decreasing the size of d-i images would be a huge win as well. You cannot debootstrap it anymore. I can hardly call that a win. (So we'd need to wait a few releases for that.) [1]. Three calls of system() could be avoided for little effort and two more for substantial effort -- and all of them could be turned into execve(), but otherwise, it's pretty much as fast as you can get. With package build time below a single disk seek, further optimizations are pointless. The point of goodbye was abuse of policy and buzzword compliancy, not speed. It compiles the same script multiple times during build. Kind regards Philipp Kern -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/slrnj48coe.a7f.tr...@kelgar.0x539.de
Re: The archive now supports xz compression
Hi, On Thu, 11 Aug 2011, Adam Borowski wrote: Thus, I'd strongly recommend just compressing everything with xz, on all architectures. Preferably, as a default in dpkg-dev. I am very much in favor of this as well but after having discussed this at debconf with Colin Watson and Joey Hess, I'm not going to change this immediately. As Ansgar mentionned, it creates a new requirement: for debootstrap to work xz must be present and it's currently not present in debian-installer. So changing the default in dpkg-dev would either require to hardcode gzip for all base packages or accept xz as a new dependency for debootstrap. The set of base package is not clearly defined, it's roughly all required packages and their dependencies but this can't be really automated and can theoretically change at the whims of ftpmasters since the priorities are under the control of ftpmasters. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634248 Since hardcoding gzip for base packages seems to be a bit brittle, we need to work towards allowing xz usage in debian-installer and accept it as a dependency for deboostrap on non-Debian systems (I don't think it's a big issue, xz is portable and is more and more widely used). I guess that a first step is to have an xz udeb... Would you like to drive a release goal to have all packages of DVD1 use XZ compressions ? 1/ Ensure XZ support is integrated in debian-installer (and deboostrap fixed to add it as a dependency). 2/ Wait until we update dpkg to use XZ by default. 3/ Ensure all packages useful in DVD1 are updated or bin-nmued before the freeze. Cheers, -- Raphaël Hertzog ◈ Debian Developer Follow my Debian News ▶ http://RaphaelHertzog.com (English) ▶ http://RaphaelHertzog.fr (Français) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110811191955.gh6...@rivendell.home.ouaza.com
Re: The archive now supports xz compression
On Thu, Aug 11, 2011 at 09:19:55PM +0200, Raphael Hertzog wrote: As Ansgar mentionned, it creates a new requirement: for debootstrap to work xz must be present and it's currently not present in debian-installer. The main thing I consider to be difficult is that putting xz-compressed packages in the base system requires people on foreign systems to have an xz decompressor available. To a large extent this is outside our control. Since hardcoding gzip for base packages seems to be a bit brittle, we need to work towards allowing xz usage in debian-installer and accept it as a dependency for deboostrap on non-Debian systems (I don't think it's a big issue, xz is portable and is more and more widely used). Can you quantify that? I don't have hard numbers for the non-Debian systems where people report running debootstrap; perhaps you do ... I guess that a first step is to have an xz udeb... No. busybox already has an unxz applet, so if we choose to do this then we should enable that applet in busybox-udeb, not add a separate udeb. -- Colin Watson [cjwat...@debian.org] -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110811225511.ga20...@riva.dynamic.greenend.org.uk
