Re: Please test gzip -9n - related to dpkg with multiarch support

[Goswin von Brederlow]

Russ Allbery  writes:

> Goswin von Brederlow  writes:
>
>> Changing the name in the package would break tools that rely on the name
>> (like packages.debian.org extracting the Changelog). Also ugly.
>
> We control the tools; we can change the tools.  Multiarch is a big deal.
> We weren't going to get through this without changing some tools.  (One
> should not read that as my support of this specific alternative, as I've
> not decided there yet, but in general I think it's fair game to change our
> tools to support multiarch.)

One should not read that as my rejection of this specific alternative, as
I've not decided there yet, but in general I think it's fair game to
change our tools to support multiarch.


Problem I have is with the timeframe of such a change. While we can
change packages.d.o quite quickly given somebody willing to write the
patch we can not quickly change the tools in stable. And I really really
really do not want to have to wait yet another release cycle.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87lintzq9k.fsf@frosties.localnet

Re: Please test gzip -9n - related to dpkg with multiarch support

[Russ Allbery]

Goswin von Brederlow  writes:

> Changing the name in the package would break tools that rely on the name
> (like packages.debian.org extracting the Changelog). Also ugly.

We control the tools; we can change the tools.  Multiarch is a big deal.
We weren't going to get through this without changing some tools.  (One
should not read that as my support of this specific alternative, as I've
not decided there yet, but in general I think it's fair game to change our
tools to support multiarch.)

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87k43vu389@windlord.stanford.edu

Re: Please test gzip -9n - related to dpkg with multiarch support

[Goswin von Brederlow]

Ian Jackson  writes:

> Goswin von Brederlow writes ("Re: Please test gzip -9n - related to dpkg with 
> multiarch support"):
>> Ian Jackson  writes:
>> > One thing which no-one yet seems to have suggested is to have
>> > multiarch:same packages put the changelog in a filename which is
>> > distinct for each architecture.  (It wouldn't have to be the triplet;
>> > the shorter Debian arch would do.)  Perhaps there are obvious reasons
>> > (which I have missed) why this is a terrible idea, but it seems to me
>> > that it's something we should consider.
>> 
>> Or dpkg could do that for you. At least for files in /usr/share/doc when
>> there is a collision.
>
> Urgh, I think this is a really ugly idea, compared to just having the
> packages contain the arch-specific filenames.  After all a
> multiarch:same package knows that it is, and can DTRT.
>
> Ian.

Changing the name in the package would break tools that rely on the
name (like packages.debian.org extracting the Changelog). Also ugly.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ty30p0re.fsf@frosties.localnet

Re: Please test gzip -9n - related to dpkg with multiarch support

[Ian Jackson]

Goswin von Brederlow writes ("Re: Please test gzip -9n - related to dpkg with 
multiarch support"):
> Ian Jackson  writes:
> > One thing which no-one yet seems to have suggested is to have
> > multiarch:same packages put the changelog in a filename which is
> > distinct for each architecture.  (It wouldn't have to be the triplet;
> > the shorter Debian arch would do.)  Perhaps there are obvious reasons
> > (which I have missed) why this is a terrible idea, but it seems to me
> > that it's something we should consider.
> 
> Or dpkg could do that for you. At least for files in /usr/share/doc when
> there is a collision.

Urgh, I think this is a really ugly idea, compared to just having the
packages contain the arch-specific filenames.  After all a
multiarch:same package knows that it is, and can DTRT.

Ian.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/20275.48825.356091.885...@chiark.greenend.org.uk

Re: Please test gzip -9n - related to dpkg with multiarch support

[Goswin von Brederlow]

Ian Jackson  writes:

> Russ Allbery writes ("Re: Please test gzip -9n - related to dpkg with 
> multiarch support"):
>> Another possible solution is to just give any package an implicit Replaces
>> (possibly constrained to /usr/share/doc) on any other package with the
>> same name and version and a different architecture.  This isn't as
>> defensive, in that it doesn't catch legitimate bugs where someone has made
>> a mistake and the packages contain different contents, but it also solves
>> the binNMU issue (well, "solves"; the changelog will randomly swap back
>> and forth between the packages, but I'm having a hard time being convinced
>> this is a huge problem).
>
> Well, it does mean that you might be lacking important information
> because the other changelog wouldn't be present on the system.
>
> One thing which no-one yet seems to have suggested is to have
> multiarch:same packages put the changelog in a filename which is
> distinct for each architecture.  (It wouldn't have to be the triplet;
> the shorter Debian arch would do.)  Perhaps there are obvious reasons
> (which I have missed) why this is a terrible idea, but it seems to me
> that it's something we should consider.
>
> Ian.

Or dpkg could do that for you. At least for files in /usr/share/doc when
there is a collision.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/871uq4rxzq.fsf@frosties.localnet

Re: Please test gzip -9n - related to dpkg with multiarch support

[Russ Allbery]

Steve Langasek  writes:
> On Wed, Feb 08, 2012 at 04:55:02PM -0800, Russ Allbery wrote:
>> Steve Langasek  writes:

>>> The unfounded assumption here is that you will always install a
>>> foreign-arch M-A: same package together with the native-arch version.
>>> If I install libaudio2:i386 because I want to play a game that's only
>>> available as a 32-bit binary and has this lib as a dependency, and
>>> nothing else on my system uses libaudio2, I still expect to get
>>> /usr/share/libaudio2/AuErrorDB installed.

>> How is that not a serious policy violation already?  AuErrorDB isn't
>> versioned with the SONAME, so libaudio2 and libaudio3 would not be
>> coinstallable.

> Because libaudio2 is in the directory name.

Oh, duh.  Sorry, I'm just blind.

> Also, it's not a policy violation for a library package to contain files
> that don't have sensibly versioned names; it's only a policy violation
> for the name to not change on soname bump.  So even if this were called
> /usr/share/AuErrorDB, it could be changed to
> /usr/share/libaudio3/AuErrorDB on soname change and still be compliant.

Good point.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ty30iz9a@windlord.stanford.edu

Re: Please test gzip -9n - related to dpkg with multiarch support

[Steve Langasek]

On Wed, Feb 08, 2012 at 04:55:02PM -0800, Russ Allbery wrote:
> Steve Langasek  writes:
> 
> > The unfounded assumption here is that you will always install a
> > foreign-arch M-A: same package together with the native-arch version.
> > If I install libaudio2:i386 because I want to play a game that's only
> > available as a 32-bit binary and has this lib as a dependency, and
> > nothing else on my system uses libaudio2, I still expect to get
> > /usr/share/libaudio2/AuErrorDB installed.

> How is that not a serious policy violation already?  AuErrorDB isn't
> versioned with the SONAME, so libaudio2 and libaudio3 would not be
> coinstallable.

Because libaudio2 is in the directory name.

Also, it's not a policy violation for a library package to contain files
that don't have sensibly versioned names; it's only a policy violation for
the name to not change on soname bump.  So even if this were called
/usr/share/AuErrorDB, it could be changed to /usr/share/libaudio3/AuErrorDB
on soname change and still be compliant.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature

Re: Please test gzip -9n - related to dpkg with multiarch support

[Russ Allbery]

Steve Langasek  writes:

> The unfounded assumption here is that you will always install a
> foreign-arch M-A: same package together with the native-arch version.
> If I install libaudio2:i386 because I want to play a game that's only
> available as a 32-bit binary and has this lib as a dependency, and
> nothing else on my system uses libaudio2, I still expect to get
> /usr/share/libaudio2/AuErrorDB installed.

How is that not a serious policy violation already?  AuErrorDB isn't
versioned with the SONAME, so libaudio2 and libaudio3 would not be
coinstallable.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874nv0n7vd@windlord.stanford.edu

Re: Please test gzip -9n - related to dpkg with multiarch support

[Steve Langasek]

On Wed, Feb 08, 2012 at 10:22:17AM +, Neil Williams wrote:

> After all, this is how cross/foreign architecture packages have
> *always* been handled in Debian via dpkg-cross. Nothing in /usr/share/
> matters for a cross package created by dpkg-cross (with the possible
> exception of /usr/share/pkg-config which was always anachronistic). Some
> template files are added but the package name includes the
> architecture, so these files are effectively in multiarch paths.

> There is nothing useful in /usr/share of a Multiarch: same package
> when installed as foreign architecture package. Emdebian & dpkg-cross
> have proved that by having nothing else until Multi-Arch. Anything you
> might need is in the native architecture package, so the best thing to
> do is widen the implicit exclusion to all of /usr/share in the incoming
> Multi-Arch: same package.

The unfounded assumption here is that you will always install a foreign-arch
M-A: same package together with the native-arch version.  If I install
libaudio2:i386 because I want to play a game that's only available as a
32-bit binary and has this lib as a dependency, and nothing else on my
system uses libaudio2, I still expect to get /usr/share/libaudio2/AuErrorDB
installed.

In general, anything that introduces assymetric handling between native and
foreign arch packages at the dpkg level is probably going to be a bad idea.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature

Re: Please test gzip -9n - related to dpkg with multiarch support

[Steve Langasek]

On Wed, Feb 08, 2012 at 11:56:06AM -0800, Russ Allbery wrote:
> Riku Voipio  writes:
> > On Wed, Feb 08, 2012 at 05:56:11PM +0100, Guillem Jover wrote:

> >> The same principle that applies to all dpkg output to avoid ambiguity
> >> would apply everywhere, whenever there's a “Multi-Arch: same” package
> >> name that needs to be unambiguous, it just always gets arch-qualified.
> >> The rest would stay as they are.

> > That is a major waste of space of having multiple copies of identical
> > files with different arch-qualified names. Is that really better
> > architecture to have multiple copies of identical files on user systems?

> Is it really, though?  The files we're talking about are not generally
> large.  I have a hard time seeing a case where the files would be large
> enough to cause any noticable issue and you wouldn't want to move them
> into a separate -common or -doc package anyway.

So I had a look at the Ubuntu archive, which already has a large collection
of packages converted to Multi-Arch: same, to provide some hard facts for
this discussion.

 - 1219 binary packages are marked Multi-Arch: same
 - 2197 files are shipped in /usr/share by these packages, outside of
   /usr/share/doc - which, by and large, are files that can actually be
   shared between architectures.
 - These files are distributed between 47 different subdirectories:
703 ./usr/share/man
604 ./usr/share/ada
187 ./usr/share/lintian
185 ./usr/share/locale
 93 ./usr/share/alsa
 70 ./usr/share/gtk-doc
 53 ./usr/share/bug
 36 ./usr/share/qt4
 35 ./usr/share/libtool
 22 ./usr/share/themes
 16 ./usr/share/lua
 16 ./usr/share/libphone-ui-shr
 15 ./usr/share/aclocal
 14 ./usr/share/icons
 11 ./usr/share/pam-configs
 11 ./usr/share/info
 10 ./usr/share/vala
  9 ./usr/share/gtk-engines
  8 ./usr/share/qalculate
  8 ./usr/share/OGRE
  7 ./usr/share/xml
  7 ./usr/share/libwacom
  7 ./usr/share/gir-1.0
  7 ./usr/share/dbconfig-common
  6 ./usr/share/mupen64plus
  6 ./usr/share/libgphoto2
  5 ./usr/share/pixmaps
  5 ./usr/share/openchange
  4 ./usr/share/mime-info
  4 ./usr/share/menu
  4 ./usr/share/libofx4
  4 ./usr/share/gstreamer-0.10
  3 ./usr/share/java
  3 ./usr/share/gconf
  3 ./usr/share/gcc-4.6
  3 ./usr/share/applications
  2 ./usr/share/guile
  2 ./usr/share/application-registry
  1 ./usr/share/tdsodbc
  1 ./usr/share/psqlodbc
  1 ./usr/share/pascal
  1 ./usr/share/libpam-ldap
  1 ./usr/share/libmyodbc
  1 ./usr/share/libaudio2
  1 ./usr/share/kde4
  1 ./usr/share/gst-plugins-base
  1 ./usr/share/avahi
 - For many of these files, it would be actively harmful to use
   architecture-qualified filenames.  Manpages included in -dev packages
   should not change names based on the architecture; having
   /usr/share/pam-config contain multiple files for the same profile, one
   for each architecture of the package that's installed, would not work
   correctly; etc.
 - If we needed to split the arch-indep contents out of the M-A: same
   package instead of reference counting in dpkg, that would be roughly 170
   new binary packages.  139 of them would contain 10 files or less
   (exclusive of /usr/share/doc).

I think there are pretty solid benefits to proceeding with a dpkg that
allows sharing files across M-A: same packages.  Even if we decided we
couldn't rely on gzip, there are still lots of other cases where this
matters.

And besides, consider that a M-A: same package shipping contents in a
non-architecture-qualified path that vary by architecture is *always* a bug
in that package, which will need to be fixed.  Requiring that M-A: same
packages don't use non-architecture-qualified paths even for files which
*don't* vary by architecture doesn't help much to ensure that we won't have
bugs.  It would be easier for lintian to spot errors in M-A: same packages
if we can say that any file that doesn't have an architecture-qualified path
is buggy, but at this point we already have Jakub's reports anyway, which we
could make a regular part of our archive consistency checks.  So I don't
believe that having dpkg be more strict about files that *could* be shared
will make the user experience any better; it just presents more occasions
for packages to be regarded as buggy and for dpkg to error out.

> foo-config binaries, as opposed to pkg-config files, are indeed going to
> continue to be a problem in model 2, but they're a problem anyway, no?

Yes, they definitely are.

> There's no guarantee that the amd64 and i386 version of a package will
> want the same flags, so we really need some way of having a
> multiarch-aware verson of the -config script.

Preferably by s/foo/pkg/.  pkgconfig gets this right, the standalone tools
all get it wrong, there's no good reason not to just replace them with
pkgconfig.

-- 
Steve Langasek   Give me a l

Re: Please test gzip -9n - related to dpkg with multiarch support

[Russ Allbery]

Guillem Jover  writes:
> On Wed, 2012-02-08 at 13:19:17 +, Ian Jackson wrote:

>> Well, it does mean that you might be lacking important information
>> because the other changelog wouldn't be present on the system.

> While the implicit Replaces seems the easy way out, it just seems even
> more fragile than the shared files approach.

Yes, this is definitely true.  I was mentioning it as an easy way out, but
it's aesthetically unappealing.

> And while the binNMU changelog issues might seem like a corner case,
> it's just a symptom of something that's not quite right.

Also true.  In fact, it's something that's been bothering me for a long
time with linked doc directories.  I'd like to prohibit them in more cases
so that we get the binNMU changelogs on disk.

> Instead of this, I'd rather see the shared files approach just dropped
> completely, and /usr/share/doc/ files for “Multi-Arch: same” packages be
> installed under /usr/share/doc/pkgname:arch/. This would solve all these
> problems in a clean way for the common case with just the two or three
> mandated files (changelog, changelog.Debian and copyright), if a package
> provides lots more files then they should be split anyway into either a
> libfooN-common libfoo-doc, or similar. And finally this would not be
> really confusing, given that one of the last interface changes was to
> make all dpkg output for all “Multi-Arch: same” packages be always
> arch-qualified.

The only thing I'm worried about here is that we lose something from the
UI perspective.  That's going to be a change historically from where we've
told users to look, and it's a little awkward.  But, thinking it over, the
set of packages that we're talking about is fairly limited.

-- 
Russ Allbery (r...@debian.org)   


--
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87liodrttk@windlord.stanford.edu

Re: Please test gzip -9n - related to dpkg with multiarch support

[Riku Voipio]

On Wed, Feb 08, 2012 at 05:56:11PM +0100, Guillem Jover wrote:
> On Wed, 2012-02-08 at 17:29:22 +0100, Mike Hommey wrote:
> > If you remove the shared files approach, how do you handle files like
> > lintian overrides, reportbug presubj and scripts, etc. ?
> 
> The same principle that applies to all dpkg output to avoid ambiguity
> would apply everywhere, whenever there's a “Multi-Arch: same” package
> name that needs to be unambiguous, it just always gets arch-qualified.
> The rest would stay as they are.

That is a major waste of space of having multiple copies of identical files
with different arch-qualified names. Is that really better architecture to
have multiple copies of identical files on user systems?

> So, at least for lintian and reportbug, given that these file/dir names
> are package name based they would just get arch-qualified when needed.

Another major frustration your no-shared-files proposal adds, is the need
to split the M-A: same libfoo-dev packages to libfoo-dev-common in order to
avoid overwriting /usr/include contents and /usr/bin/foo-config binaries. Our
packages are already heavily split slowing down Packages.gz downloads and
all other apt operations.

Riku


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120208175241.ga6...@afflict.kos.to

Re: Please test gzip -9n - related to dpkg with multiarch support

[Jonathan Nieder]

Ian Jackson wrote:

> Another relevant question is whether there are other files that are
> shared, and which don't want to move, besides ones in /usr/share/doc.

One example is header files in /usr/include, from -dev packages.  In
the simple examples I've seen, putting them in /usr/include/,
works fine.  It is always possible to split off a separate "Multiarch:
foreign" -dev-common package if needed in order to save space.

Another example is manpages, also in -dev packages.  That's more
fussy.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120208172627.GA6712@burratino

Re: Please test gzip -9n - related to dpkg with multiarch support

[Mike Hommey]

On Wed, Feb 08, 2012 at 05:13:20PM +0100, Guillem Jover wrote:
> On Wed, 2012-02-08 at 13:19:17 +, Ian Jackson wrote:
> > Well, it does mean that you might be lacking important information
> > because the other changelog wouldn't be present on the system.
> 
> While the implicit Replaces seems the easy way out, it just seems even
> more fragile than the shared files approach.
> 
> And while the binNMU changelog issues might seem like a corner case,
> it's just a symptom of something that's not quite right. And after
> this was brought up again I started considering that the shared file
> approach might have been flawed afterall, even if it might have seemed
> neat at the time (it's one of the reasons that part of the code has
> not been merged yet). The main reason it was enviaged was to handle the
> changelog and copyright files and to avoid needing to introduce an
> additional common package per source, for just those two/three files.
> 
> As a side remark, I think at least those two are actual package
> metadata and do belong in the .deb control member [0], and as such in
> the dpkg database. But that's for discussion on another time, because
> that would not fix the issue as upstream changelogs do conflict too,
> for example.
> 
>   
> 
> > One thing which no-one yet seems to have suggested is to have
> > multiarch:same packages put the changelog in a filename which is
> > distinct for each architecture.  (It wouldn't have to be the triplet;
> > the shorter Debian arch would do.)  Perhaps there are obvious reasons
> > (which I have missed) why this is a terrible idea, but it seems to me
> > that it's something we should consider.
> 
> Instead of this, I'd rather see the shared files approach just dropped
> completely, and /usr/share/doc/ files for “Multi-Arch: same” packages
> be installed under /usr/share/doc/pkgname:arch/. This would solve all
> these problems in a clean way for the common case with just the two or
> three mandated files (changelog, changelog.Debian and copyright), if
> a package provides lots more files then they should be split anyway
> into either a libfooN-common libfoo-doc, or similar. And finally this
> would not be really confusing, given that one of the last interface
> changes was to make all dpkg output for all “Multi-Arch: same”
> packages be always arch-qualified.

If you remove the shared files approach, how do you handle files like
lintian overrides, reportbug presubj and scripts, etc. ?

Mike


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120208162922.ga28...@glandium.org

Re: Please test gzip -9n - related to dpkg with multiarch support

[Ian Jackson]

Guillem Jover writes ("Re: Please test gzip -9n - related to dpkg with 
multiarch support"):
> On Wed, 2012-02-08 at 13:19:17 +, Ian Jackson wrote:
> > One thing which no-one yet seems to have suggested is to have
> > multiarch:same packages put the changelog in a filename which is
> > distinct for each architecture.  (It wouldn't have to be the triplet;
> > the shorter Debian arch would do.)  Perhaps there are obvious reasons
> > (which I have missed) why this is a terrible idea, but it seems to me
> > that it's something we should consider.
> 
> Instead of this, I'd rather see the shared files approach just dropped
> completely, and /usr/share/doc/ files for “Multi-Arch: same” packages
> be installed under /usr/share/doc/pkgname:arch/.

Right, that's kind of what I was suggesting although you've
generalised it.  It doesn't seem like an unreasonable idea to me.

Obviously it would mean that some (Debian-specific) software which
currently doesn't need to be multiarch-aware would need to be taught
about these new directory names.  But that seems like a reasonable
price to pay for solving the varying compressed shared files problem.

Another relevant question is whether there are other files that are
shared, and which don't want to move, besides ones in /usr/share/doc.
I haven't been following this in detail but if there are then we may
need to retain the possibility to have actually-identical shared
files.

Ian.


--
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/20274.43538.832430.612...@chiark.greenend.org.uk

Re: Please test gzip -9n - related to dpkg with multiarch support

[Guillem Jover]

On Wed, 2012-02-08 at 17:29:22 +0100, Mike Hommey wrote:
> If you remove the shared files approach, how do you handle files like
> lintian overrides, reportbug presubj and scripts, etc. ?

The same principle that applies to all dpkg output to avoid ambiguity
would apply everywhere, whenever there's a “Multi-Arch: same” package
name that needs to be unambiguous, it just always gets arch-qualified.
The rest would stay as they are.

So, at least for lintian and reportbug, given that these file/dir names
are package name based they would just get arch-qualified when needed.

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120208165611.ga25...@gaara.hadrons.org

Re: Please test gzip -9n - related to dpkg with multiarch support

[Guillem Jover]

On Wed, 2012-02-08 at 13:19:17 +, Ian Jackson wrote:
> Well, it does mean that you might be lacking important information
> because the other changelog wouldn't be present on the system.

While the implicit Replaces seems the easy way out, it just seems even
more fragile than the shared files approach.

And while the binNMU changelog issues might seem like a corner case,
it's just a symptom of something that's not quite right. And after
this was brought up again I started considering that the shared file
approach might have been flawed afterall, even if it might have seemed
neat at the time (it's one of the reasons that part of the code has
not been merged yet). The main reason it was enviaged was to handle the
changelog and copyright files and to avoid needing to introduce an
additional common package per source, for just those two/three files.

As a side remark, I think at least those two are actual package
metadata and do belong in the .deb control member [0], and as such in
the dpkg database. But that's for discussion on another time, because
that would not fix the issue as upstream changelogs do conflict too,
for example.

  

> One thing which no-one yet seems to have suggested is to have
> multiarch:same packages put the changelog in a filename which is
> distinct for each architecture.  (It wouldn't have to be the triplet;
> the shorter Debian arch would do.)  Perhaps there are obvious reasons
> (which I have missed) why this is a terrible idea, but it seems to me
> that it's something we should consider.

Instead of this, I'd rather see the shared files approach just dropped
completely, and /usr/share/doc/ files for “Multi-Arch: same” packages
be installed under /usr/share/doc/pkgname:arch/. This would solve all
these problems in a clean way for the common case with just the two or
three mandated files (changelog, changelog.Debian and copyright), if
a package provides lots more files then they should be split anyway
into either a libfooN-common libfoo-doc, or similar. And finally this
would not be really confusing, given that one of the last interface
changes was to make all dpkg output for all “Multi-Arch: same”
packages be always arch-qualified.

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120208161319.ga24...@gaara.hadrons.org

Re: Please test gzip -9n - related to dpkg with multiarch support

[Ian Jackson]

Russ Allbery writes ("Re: Please test gzip -9n - related to dpkg with multiarch 
support"):
> Another possible solution is to just give any package an implicit Replaces
> (possibly constrained to /usr/share/doc) on any other package with the
> same name and version and a different architecture.  This isn't as
> defensive, in that it doesn't catch legitimate bugs where someone has made
> a mistake and the packages contain different contents, but it also solves
> the binNMU issue (well, "solves"; the changelog will randomly swap back
> and forth between the packages, but I'm having a hard time being convinced
> this is a huge problem).

Well, it does mean that you might be lacking important information
because the other changelog wouldn't be present on the system.

One thing which no-one yet seems to have suggested is to have
multiarch:same packages put the changelog in a filename which is
distinct for each architecture.  (It wouldn't have to be the triplet;
the shorter Debian arch would do.)  Perhaps there are obvious reasons
(which I have missed) why this is a terrible idea, but it seems to me
that it's something we should consider.

Ian.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/20274.30293.295855.341...@chiark.greenend.org.uk

Re: Please test gzip -9n - related to dpkg with multiarch support

[Wouter Verhelst]

On Tue, Feb 07, 2012 at 10:04:04PM +, Neil Williams wrote:
> Maybe the way to solve this properly is to remove compression from the
> uniqueness check - compare the contents of the file in memory after
> decompression. Yes, it will take longer but it is only needed when the
> md5sum (which already exists) doesn't match.

Actually, I think the real way to fix this properly is to not compress
files in the package at all.

The contents.tar.gz is already a .tar.gz, which means it's compressed.
Doubly-compressing files hardly ever nets a benefit, so we're not
compressing files for the benefit of our mirrors.

The only reason why we compress files in /usr/share/doc is so that that
directory doesn't waste too much space. If that is the case, I think it
makes much more sense for files to be packaged inside .debs
uncompressed, and (optionally) for dpkg to compress them on the fly
should the system administrator request it. It would then make much more
sense for dpkg to consider the contents of the file, rather than the
on-disk representation, and not cause this kind of issues.

As an additional benefit, this will also allow those among us (like me)
who hate having to use 'gunzip -c /usr/share/doc/foo/bar.pdf.gz >
/tmp/bar.pdf; xpdf /tmp/bar.pdf' in order to be able to read some
documentation, to just request that files are not compressed.

-- 
The volume of a pizza of thickness a and radius z can be described by
the following formula:

pi zz a


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120208131025.ga27...@grep.be

Re: Please test gzip -9n - related to dpkg with multiarch support

[Ben Hutchings]

On Wed, 2012-02-08 at 07:57 +, Lars Wirzenius wrote:
> On Tue, Feb 07, 2012 at 10:49:23PM +, Ben Hutchings wrote:
> > But it's worse than this: even if dpkg decompresses before comparing,
> > debsums won't (and mustn't, for backward compatibility).  So it's
> > potentially necessary to fix up the md5sums file for a package
> > installed for multiple architectures, if it contains a file that was
> > compressed differently.
> 
> I'm uncomfortable with the idea of checking checksums only for
> uncompressed data. Compressed files have headers, and at least for
> some formats, it seems those headers can contain essentially 
> arbitrary data. This allows compressed files to be modified in
> rather significant ways, without debsums noticing, if debsums
> uncompresses before comparing.
> 
> Further, uncompressors have the potential for security problems.
> See https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2009-2624 for example.
> In other words: debsums needs to decompress to verify that no
> files have been tampered with, but doing so can invoke an attack.
> Such an attack may be unlikely, but it would seem to be a better design
> to not open up the possibility for it.

I wasn't suggesting debsums would do decompression.

Ben.

-- 
Ben Hutchings
The generation of random numbers is too important to be left to chance.
- Robert Coveyou


signature.asc
Description: This is a digitally signed message part

Re: Please test gzip -9n - related to dpkg with multiarch support

[Daniel Baumann]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/07/2012 11:04 PM, Neil Williams wrote:
> I'm not convinced that this is fixable at the gzip level, nor is it
> likely to be fixable by the trauma of changing from gzip to 
> something else.

while the original point of not considering compressors that are not
producing identical output across all archs in dpkgs multi-arch
implementation still stands, it might be worth noting (and at least
jftr) that lzip does not suffer from that problem in the first place.

- -- 
Address:Daniel Baumann, Donnerbuehlweg 3, CH-3012 Bern
Email:  daniel.baum...@progress-technologies.net
Internet:   http://people.progress-technologies.net/~daniel.baumann/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk8yWOYACgkQ+C5cwEsrK57kRACg4LIBEB+Yn6bMC9E+xybh/doX
FJAAn2Ufes5sZTMn4XHYQ2fr5ja/w6W6
=qbRU
-END PGP SIGNATURE-


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4f3258e6.9060...@progress-technologies.net

Re: Please test gzip -9n - related to dpkg with multiarch support

[Neil Williams]

On Wed, 8 Feb 2012 02:52:52 +0200
Riku Voipio  wrote:

> If it turns out not reasonable to expect the compression results to be
> identical, we should probably look into using dpkg --path-exclude= with
> /usr/share/{doc,man,info}/* when installing foreign-architecture packages.

That would be a suitable alternative to decompression checksums.

It sounds like an implicit Replaces: on the same package of any other
architecture for Multi-Arch: same packages limited
to /usr/share/{doc,man,info}/*.

> Very few Multi-Arch: same packages need to install identical compressed
> files outside these directories. In case it happens, the package needs to 
> use multiarch paths or split files to -common package. 

It's not that ugly - with a few looks at the list of problematic files.
e.g. libslang2-dev, in common with a number of other -dev packages,
includes a few example files in the -dev instead of using a -doc
package. The compression of those files causes conflicts in the -dev
package, at which point creating a -doc package doesn't seem that bad
an idea. Other options would be to not compress example files when
packaged inside -dev packages - after all, if the example files are
large enough for a lack of compression to matter, the examples should
be in a -doc package.

http://people.debian.org/~jwilk/multi-arch/same-md5sums.txt

> The ugliness of this
> solution is that the "specialness" of /usr/share/doc and others needs to
> embedded into the package system somewhere.

Packages can use multiarch paths for their own files, but there are
currently 80 occurrences of changelog.Debian.gz in the list of
problematic files. dpkg needs to handle that, packages have no option.

I'm wondering if /usr/share/{doc,man,info}/* is the right pattern.
Maybe it really is just /usr/share/*.

After all, this is how cross/foreign architecture packages have
*always* been handled in Debian via dpkg-cross. Nothing in /usr/share/
matters for a cross package created by dpkg-cross (with the possible
exception of /usr/share/pkg-config which was always anachronistic). Some
template files are added but the package name includes the
architecture, so these files are effectively in multiarch paths.

There is nothing useful in /usr/share of a Multiarch: same package
when installed as foreign architecture package. Emdebian & dpkg-cross
have proved that by having nothing else until Multi-Arch. Anything you
might need is in the native architecture package, so the best thing to
do is widen the implicit exclusion to all of /usr/share in the incoming
Multi-Arch: same package.

In the list, the only listings in the above file which are not
in /usr/share do look like bugs:

usr/bin/kvirc
usr/bin/croco-0.6-config
usr/bin/croco-0.6-config
usr/include/dspam/auto-config.h
usr/include/isl/stdint.h
usr/bin/magics-config
usr/include/OGRE/OgreBuildSettings.h
usr/include/pci/config.h
usr/lib/pkgconfig/popt.pc
usr/bin/ppl_pl
usr/bin/ppl-config
usr/include/ppl.hh
usr/include/ppl_c.h
usr/bin/ppl-config
usr/include/ppl.hh
usr/include/ppl_c.h
usr/lib/sasl2/berkeley_db.txt
usr/lib/libwrap.a
usr/include/XdmfConfig.h
usr/bin/whiptail

usr/lib/pkgconfig/popt.pc - needs to be a multiarch path
usr/bin/* is just wrong - bug reports invited.

usr/include/* means that the package concerned needs to use a multiarch
path for that include file(s).

That leaves:
usr/lib/sasl2/berkeley_db.txt
usr/lib/libwrap.a

.a files need multiarch paths, clearly.

So, apart from /usr/share which I can't see as important for
Multi-Arch: same packages, the list of remaining conflicts are bugs and
the gzip bug doesn't matter anymore.

-- 


Neil Williams
=
http://www.linux.codehelp.co.uk/



pgpDQTFHJxHrf.pgp
Description: PGP signature

Re: Please test gzip -9n - related to dpkg with multiarch support

[Lars Wirzenius]

On Tue, Feb 07, 2012 at 10:49:23PM +, Ben Hutchings wrote:
> But it's worse than this: even if dpkg decompresses before comparing,
> debsums won't (and mustn't, for backward compatibility).  So it's
> potentially necessary to fix up the md5sums file for a package
> installed for multiple architectures, if it contains a file that was
> compressed differently.

I'm uncomfortable with the idea of checking checksums only for
uncompressed data. Compressed files have headers, and at least for
some formats, it seems those headers can contain essentially 
arbitrary data. This allows compressed files to be modified in
rather significant ways, without debsums noticing, if debsums
uncompresses before comparing.

Further, uncompressors have the potential for security problems.
See https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2009-2624 for example.
In other words: debsums needs to decompress to verify that no
files have been tampered with, but doing so can invoke an attack.
Such an attack may be unlikely, but it would seem to be a better design
to not open up the possibility for it.

-- 
http://www.kickstarter.com/projects/docstory/mix-1-2-albanian


signature.asc
Description: Digital signature

Re: Please test gzip -9n - related to dpkg with multiarch support

[Henrique de Moraes Holschuh]

On Tue, 07 Feb 2012, Ben Hutchings wrote:
> But it's worse than this: even if dpkg decompresses before comparing,
> debsums won't (and mustn't, for backward compatibility).  So it's

Maybe you can switch to sha256 and add the new functionality while at
it?  Detect which mode (md5sum raw, sha256 uncompress) by the size of
the hash.  Old debsums won't work with the new files, but is that really
a problem?  That's what stable updates and backports are for...

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120207234212.gb13...@khazad-dum.debian.net

Re: Please test gzip -9n - related to dpkg with multiarch support

[Ben Hutchings]

On Tue, Feb 07, 2012 at 10:04:04PM +, Neil Williams wrote:
> On Tue, 07 Feb 2012 19:11:16 +0100
> Michael Biebl  wrote:
> 
> > On 07.02.2012 18:07, Joey Hess wrote:
> > > Neil Williams wrote:
> > >> I'd like to ask for some help with a bug which is tripping up my tests
> > >> with the multiarch-aware dpkg from experimental - #647522 -
> > >> non-deterministic behaviour of gzip -9n.
> > > 
> > > pristine-tar hat tricks[1] aside, none of gzip, bzip2, xz are required
> > > to always produce the same compressed file for a given input file, and I
> > > can tell you from experience that there is a wide amount of variation. If
> > > multiarch requires this, then its design is at worst broken, and at
> > > best, there will be a lot of coordination pain every time there is a
> > > new/different version of any of these that happens to compress slightly
> > > differently.
> 
> Exactly. I'm not convinced that this is fixable at the gzip level, nor
> is it likely to be fixable by the trauma of changing from gzip to
> something else. That would be pointless.
> 
> What matters, to me, is that package installations do not fail
> somewhere down the dependency chain in ways which are difficult to fix.
> Compression is used to save space, not to provide unique identification
> of file contents. As it is now clear that the compression is getting in
> the way of dealing with files which are (in terms of their actual
> *usable* content) identical, then the compression needs to be taken out
> of the comparison operation. Where the checksum matches that's all well
> and good (problems with md5sum collisions aside), where it does not
> match then dpkg cannot deem that the files conflict without creating a
> checksum based on the decompressed content of the two files.
[...]

But it's worse than this: even if dpkg decompresses before comparing,
debsums won't (and mustn't, for backward compatibility).  So it's
potentially necessary to fix up the md5sums file for a package
installed for multiple architectures, if it contains a file that was
compressed differently.

Ben.

-- 
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
  - Albert Camus


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120207224923.gc12...@decadent.org.uk

Re: Please test gzip -9n - related to dpkg with multiarch support

[Russ Allbery]

Neil Williams  writes:

> Maybe the way to solve this properly is to remove compression from the
> uniqueness check - compare the contents of the file in memory after
> decompression. Yes, it will take longer but it is only needed when the
> md5sum (which already exists) doesn't match.

Another possible solution is to just give any package an implicit Replaces
(possibly constrained to /usr/share/doc) on any other package with the
same name and version and a different architecture.  This isn't as
defensive, in that it doesn't catch legitimate bugs where someone has made
a mistake and the packages contain different contents, but it also solves
the binNMU issue (well, "solves"; the changelog will randomly swap back
and forth between the packages, but I'm having a hard time being convinced
this is a huge problem).

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87fwemz229@windlord.stanford.edu

Re: Please test gzip -9n - related to dpkg with multiarch support

[Neil Williams]

On Tue, 07 Feb 2012 19:11:16 +0100
Michael Biebl  wrote:

> On 07.02.2012 18:07, Joey Hess wrote:
> > Neil Williams wrote:
> >> I'd like to ask for some help with a bug which is tripping up my tests
> >> with the multiarch-aware dpkg from experimental - #647522 -
> >> non-deterministic behaviour of gzip -9n.
> > 
> > pristine-tar hat tricks[1] aside, none of gzip, bzip2, xz are required
> > to always produce the same compressed file for a given input file, and I
> > can tell you from experience that there is a wide amount of variation. If
> > multiarch requires this, then its design is at worst broken, and at
> > best, there will be a lot of coordination pain every time there is a
> > new/different version of any of these that happens to compress slightly
> > differently.

Exactly. I'm not convinced that this is fixable at the gzip level, nor
is it likely to be fixable by the trauma of changing from gzip to
something else. That would be pointless.

What matters, to me, is that package installations do not fail
somewhere down the dependency chain in ways which are difficult to fix.
Compression is used to save space, not to provide unique identification
of file contents. As it is now clear that the compression is getting in
the way of dealing with files which are (in terms of their actual
*usable* content) identical, then the compression needs to be taken out
of the comparison operation. Where the checksum matches that's all well
and good (problems with md5sum collisions aside), where it does not
match then dpkg cannot deem that the files conflict without creating a
checksum based on the decompressed content of the two files. A checksum
failure of a compressed file is clearly unreliable and will generate
dozens of unreproducible bugs.

MultiArch has many benefits but saving space is not why MultiArch
exists and systems which will use MultiArch in anger will not be likely
to be short of either RAM or swap space. Yes, the machines which are
*targeted* by the builds which occur as a result of having MultiArch
available for Emdebian will definitely be aimed at "low resource"
devices but those devices do NOT need to actually use MultiArch
themselves. In the parlance of --build, --host and autotools, MultiArch
is a build tool, not a host mechanism. If you've got the resources to
cross-build something, you have the resources to checksum the
decompressed content of some files.

As far as having MultiArch to install non-free i386 on amd64, it is
less of a problem simply because the number of packages installed as
MultiArch packages is likely to be a lot less. Even so, although the
likelihood drops, the effect of one of these collisions getting through
is the same.

> This seems to be a rather common problem as evindenced by e.g.
> 
> https://bugs.launchpad.net/ubuntu/+source/clutter-1.0/+bug/901522
> https://bugs.launchpad.net/ubuntu/+source/libtasn1-3/+bug/889303
> https://bugs.launchpad.net/ubuntu/oneiric/+source/pam/+bug/871083

See the number of .gz files in this list:
http://people.debian.org/~jwilk/multi-arch/same-md5sums.txt

> In Ubuntu they started to work-around that by excluding random files
> from being compressed. So far I refused to add those hacks to the Debian
> package as this needs to be addressed properly.

Maybe the way to solve this properly is to remove compression from the
uniqueness check - compare the contents of the file in memory after
decompression. Yes, it will take longer but it is only needed when the
md5sum (which already exists) doesn't match.

The core problem is that the times when the md5sum of the compressed
file won't match are unpredictable. No workaround is going to be
reliable because there is no apparent logic to the files which become
affected and any file which was affected at libfoo0_1.2.3 could well be
completely blameless in libfoo0_1.2.3+b1.

(binNMU's aren't the answer either because that could just as easily
transfer the bug from libfoo0 to libfoo-dev and so on.)

There appears to be plenty of evidence that checksums of compressed
files are only useful until the checksums fail to match, at which point
I think dpkg will just have to fall back to decompressing the contents
in RAM / swap and doing a fresh checksum on the contents of each
contentious compressed file. If the checksums of the contents match,
the compressed file on the filesystem wins.

Anything else and Debian loses all the reproducibility which is so
important to developers and users. When I need to make a cross-building
chroot from unstable (or write a tool for others to create such
chroots), it can't randomly fail today, work tomorrow and fail
with some other package the day after.

If others agree, I think that bug #647522, currently open against gzip,
could be reassigned to dpkg and retitled to not rely on checksums for
compressed files when determining MultiArch file collisions.

-- 


Neil Williams
=
http://www.linux.codehelp.co.uk/



pgptj0LtSC5J0.pgp
Description: PGP signature