Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
[ CCed Mohammed for the lzma package size reduction part. ]
Hi,
On Fri, 25 Jan 2008, Ian Jackson wrote:
> Raphael Hertzog writes ("Bug#456332: dpkg could use an elevated pre-depends
> or depends on lzma"):
> > The debian/control field is the only viable option IMO. It would be
> > somewhat similar to the Package-Type: header which has no real use except
> > influencing the behaviour of another tool during the build.
>
> Having slept on it I think this is the right answer.
> Compression-Type: gzip|bzip2|lzma|...
> converted into
> Pre-Depends: deb-decompressor-lzma
> by dpkg-gencontrol
The problem with that is when unpacking and repacking a .deb with a
different compression, dpkg-deb would have to munge the Pre-Depends.
Even if that should be safe with such virtual package, I don't feel
comfortable doing that.
The other problem is that all current packages using bzip2 or lzma
would have to gain such dependency (supposedly gzip ones as well?!),
there should not be many using bzip2 or lzma, though.
On Fri, 2008-01-25 at 18:54:10 +0000, Ian Jackson wrote:
> Raphael Hertzog writes ("Re: Bug#456332: dpkg could use an elevated
> pre-depends or depends on lzma"):
> > I'm still not convinced that this is the right approach, Pre-Depends are
> > supposed to not be used lightly. Does the package need to be configured,
> > isn't a simple dependency enough ?
Yes, I'm not convinced either that using a dependency is the way to go
(although yes, I agree that if we'd have to use one, that should be a
Pre-Depends).
> > And we're speaking of:
> > $ dpkg -s lzma | grep Installed
> > Installed-Size: 292
> > $ dpkg -s bzip2 | grep Installed
> > Installed-Size: 124
>
> Between them that's nearly half a megabyte. (That's not to mention
> the couple more we're bound to acquire if we make this actually work
> well.)
Probably lzma_alone (104 KiB) could be removed at some point from the
lzma package, and few docs moved to lzma-dev where they belong (20 KiB).
Also probably only that one package in base switched to lzma we'd recover
that space anyway.
> The downsides of a Pre-Depends is that it makes installation ordering
> much more tricky - but this Pre-Depends is in fact an accurate
> reflection of the situation. If the Pre-Depends is absent and
> installation operations are done that it would have prevented, those
> operations will fail.
Ian, I can see your point, about the correctness (depends on demand),
and not encoding the archive state in the metadata of all .deb (virtual
package). But on the other hand it's dpkg who is calling the lzma
binary, it's an implementation detail as Chris said. Also dpkg either
supports it or it does not, regardless of any Pre-Depends any package
might have. And we do something similar with Essential packages.
The only concern I can see with dpkg having the Pre-Depends is that
if we have a .deb with an unsupported compression, installation will
fail too late. But I don't think that's that important as this should
not happen on the archive anyway (those would get rejected).
Also just for reference, the main reason I didn't add the Pre-Depends to
dpkg in etch was because it was near a freeze (or already frozen, don't
remember now) and didn't want to destabilize d-i at that time. That was
probably a mistake, but I don't regret having played safe, there's
always time in the future to fix things.
regards,
guillem
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
Raphael Hertzog writes ("Re: Bug#456332: dpkg could use an elevated pre-depends
or depends on lzma"):
> I'm still not convinced that this is the right approach, Pre-Depends are
> supposed to not be used lightly. Does the package need to be configured,
> isn't a simple dependency enough ?
Yes, the decompressor needs to be configured because its dependencies
need to have been installed. And it has to be a predependency because
the _unpack_ requires the decompressor. Depends is not checked until
after unpacking.
> And we're speaking of:
> $ dpkg -s lzma | grep Installed
> Installed-Size: 292
> $ dpkg -s bzip2 | grep Installed
> Installed-Size: 124
Between them that's nearly half a megabyte. (That's not to mention
the couple more we're bound to acquire if we make this actually work
well.)
The downsides of a Pre-Depends is that it makes installation ordering
much more tricky - but this Pre-Depends is in fact an accurate
reflection of the situation. If the Pre-Depends is absent and
installation operations are done that it would have prevented, those
operations will fail.
Ian.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
On Fri, 25 Jan 2008, Ian Jackson wrote:
> Raphael Hertzog writes ("Bug#456332: dpkg could use an elevated pre-depends
> or depends on lzma"):
> > The debian/control field is the only viable option IMO. It would be
> > somewhat similar to the Package-Type: header which has no real use except
> > influencing the behaviour of another tool during the build.
>
> Having slept on it I think this is the right answer.
> Compression-Type: gzip|bzip2|lzma|...
> converted into
> Pre-Depends: deb-decompressor-lzma
> by dpkg-gencontrol
Ok.
> and into an appropriate -Z option by dpkg-buildpackage, then ?
Well, no. dpkg-buildpackage doesn't call "dpkg-deb --build". dpkg-gencontrol
must push the field in DEBIAN/control and "dpkb-deb --build" must read
that field there and use the corresponding compressor.
I'm still not convinced that this is the right approach, Pre-Depends are
supposed to not be used lightly. Does the package need to be configured,
isn't a simple dependency enough ?
And furthermore, while I can understand the need to not add the dependency
to dpkg itself, in practice most systems will have all the compressors
installed anyway.
And we're speaking of:
$ dpkg -s lzma | grep Installed
Installed-Size: 292
$ dpkg -s bzip2 | grep Installed
Installed-Size: 124
Cheers,
--
Raphaël Hertzog
Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
Ian Jackson writes ("Re: Bug#456332: dpkg could use an elevated pre-depends or
depends on lzma"):
> by dpkg-gencontrol and into an appropriate -Z option by
> dpkg-buildpackage, then ?
Oh, no, not dpkg-buildpackage. Hrm. It really has to be in dpkg-deb,
which ought to strip out the special header too.
Ian.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
Raphael Hertzog writes ("Bug#456332: dpkg could use an elevated pre-depends or
depends on lzma"):
> The debian/control field is the only viable option IMO. It would be
> somewhat similar to the Package-Type: header which has no real use except
> influencing the behaviour of another tool during the build.
Having slept on it I think this is the right answer.
Compression-Type: gzip|bzip2|lzma|...
converted into
Pre-Depends: deb-decompressor-lzma
by dpkg-gencontrol and into an appropriate -Z option by
dpkg-buildpackage, then ?
Ian.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
On Thu, 24 Jan 2008, Ian Jackson wrote: > How do we expect to choose which packages will use which compressors ? Right now, maintainers choose that by adding the right -Z option in the dh_builddeb call (which forwards the option to dpkg-deb -b). So they put it directly in the rules file. > Ideally in the end the maintainer would decide, so it ought to be in > the source package somewhere. But we want to override it too. > > I think for overriding it, an environment variable is fine. Perhaps > we could just tell maintainers to put >export DPKG_DEB_COMPRESSOR ?= ... > in their rules. Hrm, that doesn't work so well if different packages > want different compression schemes. A debian/control file field ? The debian/control field is the only viable option IMO. It would be somewhat similar to the Package-Type: header which has no real use except influencing the behaviour of another tool during the build. Cheers, -- Raphaël Hertzog Le best-seller français mis à jour pour Debian Etch : http://www.ouaza.com/livre/admin-debian/
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
Raphael Hertzog writes ("Re: Bug#456332: dpkg could use an elevated pre-depends
or depends on lzma"):
> On Thu, 24 Jan 2008, Ian Jackson wrote:
> > How about if we arrange to pass -Z to dpkg-gencontrol, and
> > have a table in dpkg-dev to map to
> > deb-decompressor- for values where it is needed ?
>
> The problematic part is "arrange to pass -Z to
> dpkg-gencontrol" ... the -Z of dpkg-buildpackage is for dpkg-source and
> not dpkg-deb -b. And even if we knew right from the start what compressor
> to use, the dpkg-gencontrol call is hidden in debian/rules and we have no
> way to pass anything (except maybe an environment variable which isn't
> really nice).
Yes, I obviously hadn't thought about this too deeply.
How do we expect to choose which packages will use which compressors ?
Ideally in the end the maintainer would decide, so it ought to be in
the source package somewhere. But we want to override it too.
I think for overriding it, an environment variable is fine. Perhaps
we could just tell maintainers to put
export DPKG_DEB_COMPRESSOR ?= ...
in their rules. Hrm, that doesn't work so well if different packages
want different compression schemes. A debian/control file field ?
Ian.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
On Thu, 24 Jan 2008, Ian Jackson wrote: > Yes, but if the decompressor packages are reorganised at all, or we > decide to change again how it is implemented in dpkg, all of those > packages will become broken. > > Given that this will be a pre-depends and will be in a large number of > packages, I think it is important that it be very stable. Ok, makes sense. > > I would only be in favor of that if we modify dpkg to auto-generate that > > dependency, unfortunately right now we only know how to compress when > > "dpkg-deb -b -Z" is called and this is after dpkg-gencontrol > > obviously. > > How about if we arrange to pass -Z to dpkg-gencontrol, and > have a table in dpkg-dev to map to > deb-decompressor- for values where it is needed ? The problematic part is "arrange to pass -Z to dpkg-gencontrol" ... the -Z of dpkg-buildpackage is for dpkg-source and not dpkg-deb -b. And even if we knew right from the start what compressor to use, the dpkg-gencontrol call is hidden in debian/rules and we have no way to pass anything (except maybe an environment variable which isn't really nice). Cheers, -- Raphaël Hertzog Le best-seller français mis à jour pour Debian Etch : http://www.ouaza.com/livre/admin-debian/
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
Raphael Hertzog writes ("Re: Bug#456332: dpkg could use an elevated pre-depends
or depends on lzma"):
> On Tue, 22 Jan 2008, Ian Jackson wrote:
> > I assume you mean why do it like that. The answer is that you don't
> > need the lzma support in cases where it's not needed. The
> > dependencies are more accurate.
>
> You don't need a virtual package for that. You just need to have each
> package depend on the right decompressor.
Yes, but if the decompressor packages are reorganised at all, or we
decide to change again how it is implemented in dpkg, all of those
packages will become broken.
Given that this will be a pre-depends and will be in a large number of
packages, I think it is important that it be very stable.
> I would only be in favor of that if we modify dpkg to auto-generate that
> dependency, unfortunately right now we only know how to compress when
> "dpkg-deb -b -Z" is called and this is after dpkg-gencontrol
> obviously.
How about if we arrange to pass -Z to dpkg-gencontrol, and
have a table in dpkg-dev to map to
deb-decompressor- for values where it is needed ?
Ian.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
On Tue, 22 Jan 2008, Ian Jackson wrote:
> Raphael Hertzog writes ("Bug#456332: dpkg could use an elevated pre-depends
> or depends on lzma"):
> > On Tue, 18 Dec 2007, Ian Jackson wrote:
> > > IMO the lzma binary package should Provide a new virtual package name,
> > > lzma-deb-support or some such. Packages could Pre-Depend on that.
> >
> > What does it bring?
>
> I assume you mean why do it like that. The answer is that you don't
> need the lzma support in cases where it's not needed. The
> dependencies are more accurate.
You don't need a virtual package for that. You just need to have each
package depend on the right decompressor.
I would only be in favor of that if we modify dpkg to auto-generate that
dependency, unfortunately right now we only know how to compress when
"dpkg-deb -b -Z" is called and this is after dpkg-gencontrol
obviously.
Cheers,
--
Raphaël Hertzog
Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
Raphael Hertzog writes ("Bug#456332: dpkg could use an elevated pre-depends or
depends on lzma"):
> On Tue, 18 Dec 2007, Ian Jackson wrote:
> > IMO the lzma binary package should Provide a new virtual package name,
> > lzma-deb-support or some such. Packages could Pre-Depend on that.
>
> What does it bring?
I assume you mean why do it like that. The answer is that you don't
need the lzma support in cases where it's not needed. The
dependencies are more accurate.
Also, see my previous comments: that dpkg should execute decompressors
as subprocesses (the way it used to before doogie changed it).
> Decompression also takes more memory than bzip2 but with 32M for
> the bigger packages, I think it's okay.
I have machines still in service with as little as 20Mby of RAM.
Obviously they don't have the biggest packages but such memory use
during decompression can be a problem.
Ian.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
On Tue, 18 Dec 2007, Ian Jackson wrote:
> Chris Cheney writes ("Bug#456332: dpkg could use an elevated pre-depends or
> depends on lzma"):
> > dpkg supports using lzma for compression of debs however it only
> > suggests the lzma binary package which is what is _currently_ used for
> > decompressing the debs. In the future it would be preferable if a
> > liblzma package existed that dpkg could use similarly to how it uses
> > libbz2. IMHO packages should not pre-depend on the lzma binary package
> > since it is an implementation level detail of how dpkg supports lzma
> > binaries currently. So dpkg needs to pre-depends/depends on the lzma
> > binary package so that packages can use lzma compression. However
> > packages using lzma compression will need a pre-depends on the correct
> > version of dpkg (the one with the depends/pre-depends on lzma).
>
> IMO the lzma binary package should Provide a new virtual package name,
> lzma-deb-support or some such. Packages could Pre-Depend on that.
What does it bring?
> (How valuable is lzma compression?)
The figures on https://wiki.ubuntu.com/dpkg-lzma are self-explanatory IMO.
It decompresses faster than bzip2, the compression ratio is way better.
The downside is that it uses quite a lot of memory and time to compress the
package.
Decompression also takes more memory than bzip2 but with 32M for
the bigger packages, I think it's okay.
Cheers,
--
Raphaël Hertzog
Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
Chris Cheney writes ("Bug#456332: dpkg could use an elevated pre-depends or
depends on lzma"):
> dpkg supports using lzma for compression of debs however it only
> suggests the lzma binary package which is what is _currently_ used for
> decompressing the debs. In the future it would be preferable if a
> liblzma package existed that dpkg could use similarly to how it uses
> libbz2. IMHO packages should not pre-depend on the lzma binary package
> since it is an implementation level detail of how dpkg supports lzma
> binaries currently. So dpkg needs to pre-depends/depends on the lzma
> binary package so that packages can use lzma compression. However
> packages using lzma compression will need a pre-depends on the correct
> version of dpkg (the one with the depends/pre-depends on lzma).
IMO the lzma binary package should Provide a new virtual package name,
lzma-deb-support or some such. Packages could Pre-Depend on that.
(How valuable is lzma compression?)
I disagree that dpkg ought to support lzma the same way it supports
bzip2. Rather I think it ought to support gzip and bzip2 the way it
supports lzma: by calling an external program, as I explained in my
mail <[EMAIL PROTECTED]>
to debian-devel on Tue, 16 Oct 2007 14:34:36 +0100.
Ian.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#456332: dpkg could use an elevated pre-depends or depends on lzma
Package: dpkg dpkg supports using lzma for compression of debs however it only suggests the lzma binary package which is what is _currently_ used for decompressing the debs. In the future it would be preferable if a liblzma package existed that dpkg could use similarly to how it uses libbz2. IMHO packages should not pre-depend on the lzma binary package since it is an implementation level detail of how dpkg supports lzma binaries currently. So dpkg needs to pre-depends/depends on the lzma binary package so that packages can use lzma compression. However packages using lzma compression will need a pre-depends on the correct version of dpkg (the one with the depends/pre-depends on lzma). FWIW - Ubuntu already does the above so that it can have lzma compressed debs. There is more information about what Ubuntu has done and is planning on doing and how much lzma helps at: https://wiki.ubuntu.com/dpkg-lzma There is also a patch in Ubuntu for apt-utils apt-ftparchive to support lzma compressed debs that I wrote that someone may want to pull into Debian. The above page has the patch along with including support for Packages files being compressed with lzma except for one pkgAcqIndex::Failed function that needs to be corrected. If the Packages file support isn't wanted then it will need to be split out of the diff. Thanks, Chris Cheney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
