Re: Proposal: convert all udebs to xz compression
How is alteration from .gz to .xz going to do anything but create new upgrade or boot issues? what bug number is it? - Is lzh/ma xz built already built-in (to kernel or glibs)? What if kernel or libs changes or isn't yet installed? Use bin? Won't this mean all software and scripts that look into udebs will need to be re-written? Also that udebs will no longer be compatible with dpkg as they were? Just wondering what the "upgrade" is? --- Bob the Toucan — Copyright © 2005, 2006 Ville Koskinen ~ liblzma doesn't include any file I/O functions. A separate I/O library is planned, which would abstract handling of .gz Below is an incomplete and somewhat vague list of operating systems "most linux" "XZ Utils is free general-purpose data compression software with high compression ratio. XZ Utils also work on some not-so-POSIX systems. XZ Utils are the successor ~" * liblzma is a compression library with API similar to that of zlib. * xz is a command line tool with syntax similar to that of gzip. isn't that a 360 while not yet finished ? Guillem Jover wrote: On Sat, 2011-11-19 at 17:32:49 -0200, Otavio Salvador wrote: On Sat, Nov 19, 2011 at 17:21, Philipp Kern wrote: We could start with -z0 of course, which would be supported by dpkg-dev and hence a trivial patch of debhelper would do. But it doesn't gain that much. If you mean dpkg-deb, then as mentioned on the bug report -z0 will switch the compressor to none. The closer thing you can get right now is -z1. Dpkg people, can you comment on that? I've pinged people on IRC already about this bug earlier today. I'm not on IRC anymore. In any case I've been making up my mind over this and playing with some possible options and some draft code, I'll reply to the bug report when I have something more concrete. regards, guillem -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4ec9059c.6090...@cox.net
Re: Proposal: convert all udebs to xz compression
On Sun, Nov 20, 2011 at 08:50:20AM -0500, John D. Hendrickson and Sara Darnell wrote: > How is alteration from .gz to .xz going to do anything but create new upgrade > or boot issues? > what bug number is it? Could you please mind your own business instead of spamming people? udebs are the realm of the installer team and we're not interested in keeping any compatibility where it doesn't make sense. Thanks Philipp Kern -- .''`. Philipp KernDebian Developer : :' : http://philkern.de Stable Release Manager `. `' xmpp:p...@0x539.de Wanna-Build Admin `-finger pkern/k...@db.debian.org signature.asc Description: Digital signature
Re: Proposal: convert all udebs to xz compression
On Sat, 2011-11-19 at 17:32:49 -0200, Otavio Salvador wrote: > On Sat, Nov 19, 2011 at 17:21, Philipp Kern wrote: > > We could start with -z0 of course, which would be > > supported by dpkg-dev and hence a trivial patch of debhelper would do. > > But it doesn't gain that much. If you mean dpkg-deb, then as mentioned on the bug report -z0 will switch the compressor to none. The closer thing you can get right now is -z1. > Dpkg people, can you comment on that? I've pinged people on IRC > already about this bug earlier today. I'm not on IRC anymore. In any case I've been making up my mind over this and playing with some possible options and some draft code, I'll reply to the bug report when I have something more concrete. regards, guillem -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/2020011931.ga21...@gaara.hadrons.org
Re: Proposal: convert all udebs to xz compression
On Sat, Nov 19, 2011 at 17:21, Philipp Kern wrote: > Otavio, > > am Sat, Nov 19, 2011 at 04:55:12PM -0200 hast du folgendes geschrieben: >> On Wed, Nov 2, 2011 at 23:14, Otavio Salvador >> wrote: >> > On Sun, Oct 23, 2011 at 13:26, Philipp Kern wrote: >> > ... >> >> So my proposal is to switch the udeb compression default in dpkg to xz >> >> for wheezy, when the busybox and udpkg changes have landed. Then most >> >> udebs will get a translation upload anyway, if not they can be binNMUed >> >> to pick up the right compression. >> > ... >> > From my side you have a "go ahead" but I'd like to hear from Colin and >> > Joey if they can think about any con about doing it. >> What is the current status of it? > > blocked on inconclusive feedback of the dpkg devs on using the extreme variant > of xz (see #647915). We could start with -z0 of course, which would be > supported by dpkg-dev and hence a trivial patch of debhelper would do. > But it doesn't gain that much. Dpkg people, can you comment on that? I've pinged people on IRC already about this bug earlier today. > In other news I didn't yet upload the udpkg change because I waited for a > busybox upload to incorporate the already committed unxz inclusion. This has already been done. Please go ahead with udpkg then :-) -- Otavio Salvador O.S. Systems E-mail: ota...@ossystems.com.br http://www.ossystems.com.br Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/cap9odkr2m9qyvrapbsvsnzsl3zcz3wl0vw68_pscpjayqq8...@mail.gmail.com
Re: Proposal: convert all udebs to xz compression
Otavio, am Sat, Nov 19, 2011 at 04:55:12PM -0200 hast du folgendes geschrieben: > On Wed, Nov 2, 2011 at 23:14, Otavio Salvador wrote: > > On Sun, Oct 23, 2011 at 13:26, Philipp Kern wrote: > > ... > >> So my proposal is to switch the udeb compression default in dpkg to xz > >> for wheezy, when the busybox and udpkg changes have landed. Then most > >> udebs will get a translation upload anyway, if not they can be binNMUed > >> to pick up the right compression. > > ... > > From my side you have a "go ahead" but I'd like to hear from Colin and > > Joey if they can think about any con about doing it. > What is the current status of it? blocked on inconclusive feedback of the dpkg devs on using the extreme variant of xz (see #647915). We could start with -z0 of course, which would be supported by dpkg-dev and hence a trivial patch of debhelper would do. But it doesn't gain that much. In other news I didn't yet upload the udpkg change because I waited for a busybox upload to incorporate the already committed unxz inclusion. Kind regards, Philipp Kern -- .''`. Philipp KernDebian Developer : :' : http://philkern.de Stable Release Manager `. `' xmpp:p...@0x539.de Wanna-Build Admin `-finger pkern/k...@db.debian.org signature.asc Description: Digital signature
Re: Proposal: convert all udebs to xz compression
On Wed, Nov 2, 2011 at 23:14, Otavio Salvador wrote: > On Sun, Oct 23, 2011 at 13:26, Philipp Kern wrote: > ... >> So my proposal is to switch the udeb compression default in dpkg to xz >> for wheezy, when the busybox and udpkg changes have landed. Then most >> udebs will get a translation upload anyway, if not they can be binNMUed >> to pick up the right compression. > ... > From my side you have a "go ahead" but I'd like to hear from Colin and > Joey if they can think about any con about doing it. Philipp, What is the current status of it? -- Otavio Salvador O.S. Systems E-mail: ota...@ossystems.com.br http://www.ossystems.com.br Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/cap9odkp_1cxyzsyj86fqebqguwynu9jushzrjopx3wr1fh5...@mail.gmail.com
Re: Proposal: convert all udebs to xz compression
Philipp Kern wrote: > True. OTOH udebs are a very restricted environment. Should I look out for > additional dots? That might break compression formats with multiple > components. ;-) Why not loop until a known format or EOF is reached? -- see shy jo signature.asc Description: Digital signature
Re: Proposal: convert all udebs to xz compression
Hi,
On Sat, Nov 05, 2011 at 11:17:05AM -0400, Joey Hess wrote:
> Code review:
>
> + while (fgets(buf, sizeof(buf), infp)) {
> + if (strncmp(buf, "data.tar.", 9) == 0) {
> + compression_type = buf + 9;
> + break;
> + }
> + }
>
> There is no point in looping here, because if the ar -t output is somehow
> longer than sizeof(buf), the data.tar.* output could be split across buffers
> and so you've already lost. Anyway, with a buffer of 8192, that won't
> happen unless the deb format changes in a very unexpected way.
it's looping over the lines, as fgets stops at newlines.
> FWIW, this code would fail if the dpkg format got a new member named something
> like data.tar.gz.sign.
True. OTOH udebs are a very restricted environment. Should I look out for
additional dots? That might break compression formats with multiple
components. ;-)
> + puts(buf);
> leftover debugging?
Yep, thanks for spotting it.
Kind regards,
Philipp Kern
--
.''`. Philipp KernDebian Developer
: :' : http://philkern.de Stable Release Manager
`. `' xmpp:p...@0x539.de Wanna-Build Admin
`-finger pkern/k...@db.debian.org
signature.asc
Description: Digital signature
Re: Proposal: convert all udebs to xz compression
Otavio Salvador wrote:
> >From my side you have a "go ahead" but I'd like to hear from Colin and
> Joey if they can think about any con about doing it.
No objection to the concept.
Code review:
+ while (fgets(buf, sizeof(buf), infp)) {
+ if (strncmp(buf, "data.tar.", 9) == 0) {
+ compression_type = buf + 9;
+ break;
+ }
+ }
There is no point in looping here, because if the ar -t output is somehow
longer than sizeof(buf), the data.tar.* output could be split across buffers
and so you've already lost. Anyway, with a buffer of 8192, that won't
happen unless the deb format changes in a very unexpected way.
FWIW, this code would fail if the dpkg format got a new member named something
like data.tar.gz.sign.
+ puts(buf);
leftover debugging?
--
see shy jo
signature.asc
Description: Digital signature
Re: Proposal: convert all udebs to xz compression
On Sun, Oct 23, 2011 at 13:26, Philipp Kern wrote: ... > So my proposal is to switch the udeb compression default in dpkg to xz > for wheezy, when the busybox and udpkg changes have landed. Then most > udebs will get a translation upload anyway, if not they can be binNMUed > to pick up the right compression. ... >From your detailed analisys it does seem it is safe to move this way. The increase in busybox size is quite small and it seems to be the right time to do that change as we'll have enough time to have feedback about it. >From my side you have a "go ahead" but I'd like to hear from Colin and Joey if they can think about any con about doing it. -- Otavio Salvador O.S. Systems E-mail: ota...@ossystems.com.br http://www.ossystems.com.br Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/cap9odkpetpvmrrxwcqyycsh9glfjwtkfi3gc4yep4_oysdb...@mail.gmail.com
Proposal: convert all udebs to xz compression
Hi there,
last week I played around with xz compression and evaluated how d-i
could benefit from it. It turns out that we could go from a total udeb
size (on amd64) of 50 MB to 40 MB just by applying xz -0e compression
instead of the default gzip.
To achieve that you need UNXZ support in busybox, which gives you two
more pages of code (8k) and 700 bytes more in udpkg (both measured on
amd64). So we could neglect this change completely.
I went on and instrumented the unpack calls within d-i by calling GNU
time(1). With gunzip (I made it call the applet instead of relying on
seamless tar support, which isn't yet implemented for xz in busybox) I
get a maximum RSS of 2384 kbytes, assuming that time measures that
correctly. With xz -0 compression I get a maximum RSS of 2608 kbytes.
With the default xz compression of -6 I get a maximum RSS of 11040
kbytes on the largest udebs, as the dictionary is increased. But I only
save less than a megabyte on all udebs, comparing -6e (39484 kbytes) and
-0e (40439 kbytes). So there's absolutely no benefit in activating a
higher compression ratio. For those wondering about the -e bit: packing
udebs gets more costly through this, but its result is more efficient
(43697 kbytes for -0 vs. 40439 kbytes for -0e) while not imposing any
more load onto the unpacker. And we generally don't care about udebs
taking longer to compress given their tinyness and the load being spread
over many package builds anyway.
On my laptop the increased CPU load of unxz was barely noticeable. Some
more calls had a CPU time of > 0.00 (i.e. with unxz), but it doesn't
really matter. See [1], [2] and [3] for syslogs with instrumentation.
My patch to udpkg is attached. It's not the best C code, but udpkg is
full of tiny static buffers. I'd appreciate a review.
So my proposal is to switch the udeb compression default in dpkg to xz
for wheezy, when the busybox and udpkg changes have landed. Then most
udebs will get a translation upload anyway, if not they can be binNMUed
to pick up the right compression.
There doesn't seem to be a con at first glance, but if somebody sees
one: please speak up. At least it's untrue that xz will have any
measureable impact when used with -0e, also for embedded systems. If
you're doing a network install it might even be faster, because there
will be less fetching involved. net-retriever might even need less RAM.
Kind regards,
Philipp Kern
[1] http://people.debian.org/~pkern/syslog-gunzip-time.gz
[2] http://people.debian.org/~pkern/syslog-unxz-time.gz (xz -6)
[3] http://people.debian.org/~pkern/syslog-unxz-0-time.gz
--
.''`. Philipp KernDebian Developer
: :' : http://philkern.de Stable Release Manager
`. `' xmpp:p...@0x539.de Wanna-Build Admin
`-finger pkern/k...@db.debian.org
From 8c2870debf4c5bc76ff78b7e76a39da345ea5f7a Mon Sep 17 00:00:00 2001
From: Philipp Kern
Date: Mon, 17 Oct 2011 17:33:17 +0200
Subject: [PATCH] Implement xz support.
---
debian/changelog |6 +
udpkg.c | 57 ++---
2 files changed, 59 insertions(+), 4 deletions(-)
diff --git a/debian/changelog b/debian/changelog
index d68a836..a21ca32 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+udpkg (1.13) UNRELEASED; urgency=low
+
+ * Implement support for udebs compressed with xz.
+
+ -- Philipp Kern Mon, 17 Oct 2011 17:01:07 +0200
+
udpkg (1.12) unstable; urgency=low
* Redesign read_block interface, fixing crashes caused by memory leak fix
diff --git a/udpkg.c b/udpkg.c
index 4b35bc8..f684eb7 100644
--- a/udpkg.c
+++ b/udpkg.c
@@ -132,16 +132,63 @@ static int dpkg_dounpack(struct package_t *pkg)
"templates", "menutest", "isinstallable",
"config"
};
+ FILE *infp = NULL;
+ const char *compression_type = NULL;
+ const char *decompression_tool;
#ifdef DOREMOVE
char *p;
- FILE *infp = NULL, *outfp = NULL;
+ FILE *outfp = NULL;
#endif
DPRINTF("Unpacking %s\n", pkg->package);
cwd = getcwd(0, 0);
chdir("/");
- snprintf(buf, sizeof(buf), "ar -p %s data.tar.gz|tar -xzf -", pkg->file);
+
+ snprintf(buf, sizeof(buf), "ar -t %s", pkg->file);
+ if ((infp = popen(buf, "r")) == NULL)
+ {
+ FPRINTF(stderr, "Cannot retrieve archive members of %s: %s\n",
+ pkg->file, strerror(errno));
+ r = 1;
+ goto reset_cwd;
+ }
+
+ while (fgets(buf, sizeof(buf), infp)) {
+ if (strncmp(buf, "data.tar.", 9) == 0) {
+ compression_type = buf + 9;
+ break;
+ }
+ }
+ pclose(infp);
+
+ if (compression_type == NULL) {
+ FPRINTF(stderr, "No data member found in %s\n", pkg->file);
+ r = 1;
+ goto reset_cwd;
+ }
+
+ if (strcmp(compression_type, "gz\n") == 0)
+ {
+ compression_type = "gz";
+ decompression_tool = "gunzip";
+ }
+ else if (strcmp(compression_type, "xz\n") == 0)
+ {
+ compression_type = "xz";
+ decompression_tool = "unxz";
+ }
+ else
+ {
+ FPRINTF(stderr, "Invalid compression type for data member o
