Re: Summary: dpkg shared / reference counted files and version match
Jakub Wilk dixit: * binNMUs for the same version cannot be co-installed anyway as their changelogs differ. ↓ That could be “fixed” by using the same email address and a hardcoded date, or not including the binNMU entry at all, or moving that entry to a new field, etc. All of which seem like ugly hacks, or a possible loss of information. http://lists.debian.org/debian-devel/2012/02/msg00316.html Hrm. But where is the line between files that are the same and files that differ in, for example, a binNMU? Worse: think of using M-A as porting toolkit. Say, you’re working on arm64 and want to cross-compile using M-A (for example because the current¹ approach is deprecated – already, dpkg-cross needs a flag to convert libc6-dev as it’s M-A). But some package does not build unmodified, so the target architecture has it uploaded to d-ports.org “unreleased” with local changes. With Guillem’s refusal of file sharing, this would work as-is, but not otherwise because of differences between the package versions. (You could build, say, eglibc_2.13-27+arm64.1.dsc, locally for your host system and install it, but that’s just ouch.) Wasn’t M-A intended to be a porting tool (among others like a biarch/triarch replacement) just for this purpose? Please don't throw into the mud work of individual developers (including me) That’s what I thought at first when I read about this, too (except I’m not affected). But Guillem does seem to have a point. What do our “new architecture upbringers” (e.g. armhf people) say? (Maybe take powerpcspe, as it seems to require more changed packages.) bye, //mirabilos -- “Having a smoking section in a restaurant is like having a peeing section in a swimming pool.” -- Edward Burr -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/pine.bsm.4.64l.1203021620450.24...@herc.mirbsd.org
Re: Summary: dpkg shared / reference counted files and version match
On Sat, 2012-02-11 at 03:28:33 +0200, Riku Voipio wrote: * Because shared files packaging simpler for maintainers. Package profiliation and duplication of arch-independent data are merely effecty that happen when packagers workaround the lack of shared identical files. That way of thinking seems backwards to me, because those problems existed before any such solution was proposed, but oh well... On Fri, Feb 10, 2012 at 11:56:20PM +0100, Guillem Jover wrote: * To avoid massive package proliferation (due to the mandated copyright and changelog files), thus the work involved in a one time split and the size increase in Packages indices. * To avoid unneeded file duplication, thus wasted space (due to those mandated files, but also partially just as a consequence of not splitting files into new arch:all packages, per above). One solution for the binNMU changelogs and generated docs would be to use arch-qualified paths for those files. That is much more lightweight solution that arch-qualifying all files, even if identical. At no point I proposed arch-qualifying filenames for those files, that would make no sense, I expressely said *pathname* as in: /usr/share/doc/pkgname:arch/ For files in M-A:same packages under a pkgname based path, the pkgname should always be arch-qualified with the Debian architecture. Most of these could be handled automatically by debhelper and cdbs, this includes things like: /usr/share/doc/pkgname/ /usr/share/bug/pkgname /usr/share/lintian/overrides/pkgname /usr/share/mime-info/pkgname.* /usr/share/menu/pkgname ... I find the requring arch-qualified path for for arch-independent data ugly system architecture. But of course the beauty of architecture is in the eye of the beholder (and lets not forget that Unix with its worse-is-better philosophy[1] was never intended to be architectural masterpiece). Arch-qualifying pkgname:arch in paths is just a way to pin those paths to a specific package instance which they belong to, nothing more. And while I agree that arch-qualifying arch-indep data is conceptually “incorrect”, this is just making explicit a pre-existing reality, all those arch-indep files have been shipping on *arch:any* packages, which could be considered just as conceptually wrong. But this has allowed to split them in the past when it made sense and whenever the tradeoffs were worth it. For example the cases of -dev packages with a couple of header file or to avoid problematic unavailability of data for essential packages, etc. Nothing says the same cannot still be done with M-A:same packages. As for the worse-is-better reference, well I've also had that in mind but for the refcnt code, because while it's true that with refcnt the interface is supposedly simpler, it brings with it inconsitency, incorrectness and incompleteness, given the amount of corner-cases and similar... regards, guillem -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120214132653.ga22...@gaara.hadrons.org
Re: Summary: dpkg shared / reference counted files and version match
On Fri, 2012-02-10 at 17:35 -0800, Russ Allbery wrote: I think we have to do something saner with changelog files eventually regardless, but I'm curious: how did Ubuntu deal with the binNMU problem that Guillem identified? If you binNMU a library on amd64 but not on i386, as near as I can tell that's going to make the library not work for multiarch until the next sourceful upload, no? I think Ubuntu has binNMUs; haven't you run into this issue? Unless something's changed recently, Ubuntu doesn't have binNMUs in the same way Debian does. Instead, they have no change source uploads with rebuilds on all architectures. There have been discussions of introducing binNMUs - see https://bugs.launchpad.net/launchpad/+bug/245594 for example. It wasn't entirely clear to me from the surrounding discussions I found whether the remaining reasons that it hasn't been implemented are technical or simply because no-one's done the work. Regards, Adam -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1328953563.27786.8.ca...@jacala.jungle.funky-badger.org
Re: Summary: dpkg shared / reference counted files and version match
Hello,
On Sat, 11 Feb 2012, Jakub Wilk wrote:
* Deploying refcnt means that M-A:same packages must always be at
the same exact installed version, so that the file contents can
match.
↓
More difficult upgrade paths, as this ties the different arch
dependency trees around M-A:same barriers.
By allowing co-installation of two different versions of the same
package, you are opening a can of worms, regardless of whether
refcnt is implemented or not.
I'm tempted to agree but I can't come up with a good reason.
The most worrying problem would be a version skew between
2 instances of libfoo and its common libfoo-data.
It could be a source of subtle bugs if this leads to having
libfoo:i386 (1.0) + libfoo:amd64 (2.0) + libfoo-data:all (2.0)
But then the proper answer is for the maintainer to put
a tight dependency “Depends: libfoo-data (= ${source:Version})”.
Any other problem is nothing else than a usual dependency problem
that would likely also be triggered in a non-multiarch world. Or am
I missing something? Do you have examples of possible problems?
* binNMUs need to be performed in lockstep for *all*
architectures, because the installed versions need to match.
↓
Causing useless buildd usage and user downloads for arches not
affected. “Fixing” this by making dpkg treat binNMU versions
specially, besides being just another special case needed for
M-A:same packages,
What do you mean by “another”? Yes, MA:same packages needs special
treatment, because they _are_ special.
To some extent... in any case I agree that we should at some point do
something to allow bin-nmu of packages on some arches only and also to
allow co-installation of packages with versions differing only by their
binnmu number (this could be easily fixed by using version of the source
package intead of binary version).
I'm not convinced that Guillem's answer is the right one though. Your
suggestion below seams appealing too.
http://lists.debian.org/debian-devel/2012/02/msg00316.html
Quoting you:
Packages could simply split debian/changelog into two parts:
/u/s/d/$p/changelog(.Debian).gz - the architecture-independent part;
/u/s/d/$p/changelog.binNMU-$arch.gz - the binNMU part.
Cheers,
--
Raphaël Hertzog ◈ Debian Developer
Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120211101538.gm13...@rivendell.home.ouaza.com
Re: Summary: dpkg shared / reference counted files and version match
* Joey Hess jo...@debian.org, 2012-02-10, 22:35: /usr/share/doc/pkgname/ /usr/share/bug/pkgname /usr/share/lintian/overrides/pkgname /usr/share/mime-info/pkgname.* /usr/share/menu/pkgname ... (Joey, I'm guessing you might consider it too late to do some of these in debhelper for compat level 9, right?) I wouldn't worry about this, there's still time to make whatever changes might be needed in v10. It's not actually clear this would be a change that needs a compat level at all. Why is that not clear? -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120211104357.ga2...@jwilk.net
Re: Summary: dpkg shared / reference counted files and version match
On Sat, 11 Feb 2012, Raphael Hertzog wrote:
It could be a source of subtle bugs if this leads to having
libfoo:i386 (1.0) + libfoo:amd64 (2.0) + libfoo-data:all (2.0)
But then the proper answer is for the maintainer to put
a tight dependency “Depends: libfoo-data (= ${source:Version})”.
Err, that's heavily frowned because it breaks binNMUs if libfoo-data is
arch:all, so we must decide on what the best way to deal with it is, and
update best practice and policy accordingly.
Too bad we don't have lightweight sub-packages. That would just kill the
need for multiarch-same and avoid a lot of nasty issues. You'd shunt all
arch-dependent files to the arch-dep subpackage. Then we'd just have to
decide whether we'd allow binNMUs of subpackages, or do it the Ubuntu way
(which basically boils down to how painful it would be for the
smaller/slower autobuilders to switch from binNMUs to no-source-changes
rebuilds on all arches).
--
One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie. -- The Silicon Valley Tarot
Henrique Holschuh
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/2012024746.ga...@khazad-dum.debian.net
Re: Summary: dpkg shared / reference counted files and version match
* Raphael Hertzog hert...@debian.org, 2012-02-11, 11:15: * Deploying refcnt means that M-A:same packages must always be at the same exact installed version, so that the file contents can match. ↓ More difficult upgrade paths, as this ties the different arch dependency trees around M-A:same barriers. By allowing co-installation of two different versions of the same package, you are opening a can of worms, regardless of whether refcnt is implemented or not. I'm tempted to agree but I can't come up with a good reason. I did have some scary scenarios in mind when I wrote this, but it was apparently because: a) it was middle of the night; b) let's arch-qualify everything didn't fit well my mental model. So please disregard what I wrote in this point. Sorry for the noise. It shall be noted that if we allow co-installation of different versions, then we should disallow sharing files not only shipped directly in .deb, but also created by maintainer scripts. Some MA:same packages currently try to share data created by maintainer scripts, and probably (almost?) all of them do it wrong. See #647428 for an example. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/2012025057.ga8...@jwilk.net
Re: Summary: dpkg shared / reference counted files and version match
On Sat, 11 Feb 2012, Henrique de Moraes Holschuh wrote:
On Sat, 11 Feb 2012, Raphael Hertzog wrote:
It could be a source of subtle bugs if this leads to having
libfoo:i386 (1.0) + libfoo:amd64 (2.0) + libfoo-data:all (2.0)
But then the proper answer is for the maintainer to put
a tight dependency “Depends: libfoo-data (= ${source:Version})”.
Err, that's heavily frowned because it breaks binNMUs if libfoo-data is
arch:all, so we must decide on what the best way to deal with it is, and
update best practice and policy accordingly.
Yikes, never mind. ${source:version} exists exactly to avoid the above
problem. Sorry about this.
--
One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie. -- The Silicon Valley Tarot
Henrique Holschuh
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/2012025325.gb...@khazad-dum.debian.net
Re: Summary: dpkg shared / reference counted files and version match
On Sat, 11 Feb 2012 03:28:33 +0200 Riku Voipio riku.voi...@iki.fi wrote: Package proliferation and duplication of arch-independent data are merely effects that happen when packagers workaround the lack of shared identical files. +1 One solution for the binNMU changelogs and generated docs would be to use arch-qualified paths for those files. That is much more lightweight solution that arch-qualifying all files, even if identical. On Fri, Feb 10, 2012 at 11:56:20PM +0100, Guillem Jover wrote: For files in M-A:same packages under a pkgname based path, the pkgname should always be arch-qualified with the Debian architecture. Most of these could be handled automatically by debhelper and cdbs, this includes things like: /usr/share/doc/pkgname/ So this would be everything handled currently by dh_installdocs dh_installexamples ? /usr/share/bug/pkgname /usr/share/lintian/overrides/pkgname /usr/share/mime-info/pkgname.* /usr/share/menu/pkgname ... I find the requring arch-qualified path for for arch-independent data ugly system architecture. But of course the beauty of architecture is in the eye of the beholder (and lets not forget that Unix with its worse-is-better philosophy[1] was never intended to be architectural masterpiece). Personally if leaving out shared files makes you upload multiarch enabled dpkg to unstable before sagrada familia is complete, i can live it (cursing silently in my room converting packages to the new requirement...). I can take the trade-off of having something better-than-current soon over having the perfect version in distant future... If the changes needed in low-level toolchain-related -dev packages like libc*-dev, linux-libc-dev and libstdc++*-dev are not so onerous for the relevant maintainers that we can actually get a Multi-Arch cross-compiler installed, then (as I've said before), we can use separate build environments per architecture in order to get Multi-Arch into Wheezy, and I'll live with that. One chroot per cross-architecture is workable - as long as there is sufficient support to actually get the cross toolchain installed and operational alongside the native architecture toolchain. It's more important that we get something workable into Wheezy than to have the a broken version in Wheezy and the more elegant solution in Wheezy+1. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpor8FrUYj15.pgp Description: PGP signature
Re: Summary: dpkg shared / reference counted files and version match
* Jonathan Nieder jrnie...@gmail.com, 2012-02-10, 18:56: This is not so one-sided as you seem to be suggesting. Yes. I apologize for implying this. Jonathan who wishes he knew of a fifth approach without the downsides of the others proposed so far :) Let me try: 1. We allow sharing files between architectures. 2. If - package $pkg (of architecture $arch1) is already installed and, - package $pkg (of architecture $arch2) is going to be installed and, - $pkg:$arch1 and $pkg:$arch2 both ship a $file with _different_ hash then dpkg would rename $arch2's $file into $file.dpkg-$arch2, using a mechanism similar to diversions. If $pkg:$arch1 is later removed, dpkg would rename files with .dpkg-$archN suffix back to original names 3. We will teach debsums about these multiarch diversions. 4. We continue to treat hash mismatches as important (or maybe serious) bugs, even though they don't prevent package installation anymore. As a corollary, we still need a separate solution for the binNMU problem. (I stole the idea from Ron Lee, and all credit should go to him. But I didn't consult the details with Ron, so I this proposal doesn't make sense to you, I shall take all the blame.) -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120211123127.ga...@jwilk.net
Re: Summary: dpkg shared / reference counted files and version match
On Fri, 2012-02-10 at 17:16:29 -0800, Steve Langasek wrote: On Fri, Feb 10, 2012 at 06:56:00PM -0600, Jonathan Nieder wrote: As long as dependencies are accurate, I don't see how allowing co-installation of the same package for two different architectures at different versions is any more complicated than pinned to the same version. I think we're likely to see a lot of bugs introduced in the process of making the dependencies accurate if we go this route. So instead of refcounting the files, we move them to a new arch: all package. Ok; now, suppose some of these files aren't actually architecture-independent: they're data but they're generated differently on different architectures, and the library expects the data in its native architecture-dependent format. (See the parallel thread about endianness of data files for examples.) Doesn't this proposal eliminate one of our best defenses against this packaging error? Having them kicked back as mismatched files, either by dpkg or by the archive, seems better to me than letting them land on the user's system and break at runtime. While I agree this is a potential issue, it's not a new one at all or specific to multiarchified packages, it's actually inherent to every arch:all package generated from source packages producing arch:any binaries too, regardless of multiarch. Instead of treating M-A:same specially on this, I'd rather see a way to detect this for all current and existing cases instead, if deemed really necessary. regards, guillem -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120211131828.ga19...@gaara.hadrons.org
Re: Summary: dpkg shared / reference counted files and version match
On Sat, 2012-02-11 at 11:41:58 +0100, Stefano Zacchiroli wrote: That is a bug and ought to be fixed in its own right. Then, the discussion of how much we should rely on it or not is a different one, but it's good to separate the concerns. As mentioned on the summary, this is not specific to gzip (even if gzip could pickup development in the future), it is a general problem that extends to any other compressors and generated files too, for example docs (POD, html, etc), etc. guillem -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120211132523.gb19...@gaara.hadrons.org
Re: Summary: dpkg shared / reference counted files and version match
On Sat, 2012-02-11 at 00:46:53 +0100, Marco d'Itri wrote: It is not true that splitting the package is a one time action, every release which adds new files will require dealing with the split. Assuming a somewhat standard packaging, using debhelper and debian/tmp or debian/main-pkg, where all upstream files get intstalled to, and selected ones copied/moved over to their debian/pkg final destination. How is this any different from packaging any new upstream release, regardless of a split due to multiarch? regards, guillem -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120211133449.gc19...@gaara.hadrons.org
Re: Summary: dpkg shared / reference counted files and version match
Guillem Jover guil...@debian.org writes: On Sat, 2012-02-11 at 11:41:58 +0100, Stefano Zacchiroli wrote: That is a bug and ought to be fixed in its own right. Then, the discussion of how much we should rely on it or not is a different one, but it's good to separate the concerns. As mentioned on the summary, this is not specific to gzip (even if gzip could pickup development in the future), it is a general problem that extends to any other compressors and generated files too, for example docs (POD, html, etc), etc. Yeah, generated documentation potentially also varies in annoying ways. Man pages generated from POD, for example, will be different if the version of Pod::Man or Pod::Simple used to generate them is different, which could easily happen across a point release of Perl, and hence across a binNMU. I suspect Doxygen is even more fragile (although somewhat less likely to be included directly in a -dev package rather than a separate -docs package because it's larger). -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/8762fde9hb@windlord.stanford.edu
Re: Summary: dpkg shared / reference counted files and version match
On Sat, 11 Feb 2012 06:21:52 -0800 Russ Allbery r...@debian.org wrote: Guillem Jover guil...@debian.org writes: On Sat, 2012-02-11 at 11:41:58 +0100, Stefano Zacchiroli wrote: That is a bug and ought to be fixed in its own right. Then, the discussion of how much we should rely on it or not is a different one, but it's good to separate the concerns. As mentioned on the summary, this is not specific to gzip (even if gzip could pickup development in the future), it is a general problem that extends to any other compressors and generated files too, for example docs (POD, html, etc), etc. Yeah, generated documentation potentially also varies in annoying ways. Man pages generated from POD, for example, will be different if the version of Pod::Man or Pod::Simple used to generate them is different, which could easily happen across a point release of Perl, and hence across a binNMU. I suspect Doxygen is even more fragile (although somewhat less likely to be included directly in a -dev package rather than a separate -docs package because it's larger). gtk-doc-tools is also susceptible - it seems to change the id= contents on each build, much as doxygen does. Same version, same package, different HTML with multiple changes per file, usually one change per function described. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgphbetgfoiHQ.pgp Description: PGP signature
Re: Summary: dpkg shared / reference counted files and version match
m...@linux.it (Marco d'Itri) writes: As the maintainer of a few (popular) library packages I consider splitting these packages a complex and annoying workaround for deficiencies in tools. It is not true that splitting the package is a one time action, every release which adds new files will require dealing with the split. Why was the implicit Replaces scheme not considered? -- ciao, Marco Because replaces does not allow for removing packages. If you install A:amd64, then A:i386 and then remove A:i386 then files will be removed that are in A:amd64 because they were implicitly replaced. You would quickly end up with missing Changelog and copyright files, READMEs, manpages, include files, ... The only way to make this scheme work would be by implicit diversion. I.e. on file conflict dpkg would automatically divert all but one architecture to /path/file.dpkg-arch or something. On removal of /path/file it would undo one of the diversions of another arch so there again would be a /path/file. I would only allow this for files in /usr/share though (irespective of their content) and possibly identical files (meaning basically refcount them) in other locations. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87sjihco37.fsf@frosties.localnet
Re: Summary: dpkg shared / reference counted files and version match
Jonathan Nieder jrnie...@gmail.com writes: Hi, Jakub Wilk wrote: How about: * Because this the obvious and elegant way of doing things. It makes multiarchification easy for packagers, and invisible for uses, including those users who don't care about multi-arch (unless they rely on paths to the libraries, which they never should do). I suppose that is a matter of opinion. The need to make sure files that are not arch-qualified match at least functionally between architectures is subtle and not very easy. Tying package versions in different architectures, which makes dependency chains more rigid, is not invisible to users. 1) It is dead easy: - If they are in /usr/share then you already decided they are fit for all archs. - If they (potentially) aren't in /usr/share then you already decided they aren't fit for all archs. You might have to rething that decision but the decision is not completly something multiarch adds to the table. It already exists, just was never enforced like this. [scripts in /usr/bin and include files are an exception but the former must not be in library packages and the later is easy to decide] 2) Reference counting files or spliting packages doesn't change the decision: - If the file is fit for all archs then you can let the reference counting take care of it or you have to move the file into a -common package. - If the file isn't fit for all archs then you need to arch qualify it. As such it is irelevant to the problem at hand. However, in the special case of header files (which are text files that almost never vary between architectures), I agree that it would be nice to be able to preserve the simplicity of the current approach from the packager's perspective. Esspecially for the cases where you would have a -dev-common package with handfull of headers and a -dev package with only the .so link left. On file packages seem a bit excessive. Just saying. [...] By allowing co-installation of two different versions of the same package, you are opening a can of worms, regardless of whether refcnt is implemented or not. Could you elaborate on this? As long as dependencies are accurate, I don't see how allowing co-installation of the same package for two different architectures at different versions is any more complicated than pinned to the same version. And said dependencies are already completly neccessary for the mono-arch case other than the case of messing up the MA:same/foreign field. What multi-arch adds though is that packages have to be carefull about file formats. An MA:same package could require a different file format for a common file for different versions (and thus archs). Some MA:same package would have to Break with themself across all archs to ensure all archs upgrade to a new format together. Did you mean that? [...] * Once implemented, this âfeatureâ cannot be taken out, *ever*. This boils down to âI don't like itâ. If it's useful, why would one consider taking it out? After trying out the approach without refcounted files for a while, it would be painless for the project to say This is too much trouble and add refcounting. By contrast, it is very painful to move in the other direction. I think that is worth taking into account. [...] Proposed solution [...] This will require changes to the Policy, to which I (and hopefully other developers) will object. Last time I checked, multiarch is not in policy yet. Please don't throw into the mud work of individual developers (including me) who already converted their packages to multi-arch. I agree that the extra work of removing multi-arch: same for existing -dev packages that have been converted is a major downside. And on the other hand, the need throughout Debian infrastructure to support the very fragile refcount approach would be a downside to that approach. This is not so one-sided as you seem to be suggesting. Jonathan who wishes he knew of a fifth approach without the downsides of the others proposed so far :) All this fear about compression formats changing or being unable to decide when it is save to refcount a file are a bit stupid. Just because you have refcounting doesn't mean you can't split packages or arch qualify files. That will still be an option. The refcounting was never a solve all solution but a way to make the trivial cases not a total pain. - allow refcounting - use uncompressed files for Changelog, copyright, README, NEWS,... - fix the binNMU issue with Changelogs differing (make them metadata?) - split package that have/want compressed files - if you cant ensure / doubt files are identical split the package or arch qualify the files What it comes down to is this: Use refcounting for the trivial cases. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive:
Re: Summary: dpkg shared / reference counted files and version match
Stefano Zacchiroli z...@debian.org writes: On Fri, Feb 10, 2012 at 05:35:19PM -0800, Russ Allbery wrote: We need a guarantee that gzip will always produce the same output, which we already know isn't the case and which doesn't look sustainable going forward. My understanding of #647522 is different. There is a patch (by Cyril) and upstream (Paul Eggert) is looking into it. With interest, AFAICT. That is a bug and ought to be fixed in its own right. Then, the discussion of how much we should rely on it or not is a different one, but it's good to separate the concerns. Cheers. We also need to make sure that we will always have a gzip that can reproduce the current output. Otherwise pristine-tar will break down and that will be a big blow for many many version control repositories. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87k43tcmsp.fsf@frosties.localnet
Re: Summary: dpkg shared / reference counted files and version match
Raphael Hertzog hert...@debian.org writes:
Hello,
On Sat, 11 Feb 2012, Jakub Wilk wrote:
* Deploying refcnt means that M-A:same packages must always be at
the same exact installed version, so that the file contents can
match.
â
More difficult upgrade paths, as this ties the different arch
dependency trees around M-A:same barriers.
By allowing co-installation of two different versions of the same
package, you are opening a can of worms, regardless of whether
refcnt is implemented or not.
I'm tempted to agree but I can't come up with a good reason.
The most worrying problem would be a version skew between
2 instances of libfoo and its common libfoo-data.
It could be a source of subtle bugs if this leads to having
libfoo:i386 (1.0) + libfoo:amd64 (2.0) + libfoo-data:all (2.0)
In that case you can already have libfoo:i386 (1.0) + libfoo-data:all
(2.0) so the problem alreadyarises on a non-multiarch system.
But then the proper answer is for the maintainer to put
a tight dependency âDepends: libfoo-data (= ${source:Version})â.
Probably better would be libfoo-data: Breaks: libfoo ( 2.0~).
Any other problem is nothing else than a usual dependency problem
that would likely also be triggered in a non-multiarch world. Or am
I missing something? Do you have examples of possible problems?
Just that your example is already covered by non-multiarch.
MfG
Goswin
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87fwehcmkn.fsf@frosties.localnet
Re: Summary: dpkg shared / reference counted files and version match
On Fri, Feb 10, 2012 at 05:35:19PM -0800, Russ Allbery wrote: Steve Langasek vor...@debian.org writes: On Fri, Feb 10, 2012 at 06:56:00PM -0600, Jonathan Nieder wrote: I agree that the extra work of removing multi-arch: same for existing -dev packages that have been converted is a major downside. And on the other hand, the need throughout Debian infrastructure to support the very fragile refcount approach would be a downside to that approach. Which infrastructure do you have in mind? I don't see anything that's needed besides a dpkg implementation (which we have) and some tools to sanity-check coinstallability (which 1. we would need anyway, and 2. we also have a preliminary implementation of). We need a guarantee that gzip will always produce the same output, which we already know isn't the case and which doesn't look sustainable going forward. That's looking rather uncomfortable. The preliminary results there point to that being somewhat problematic. I guess we're looking at the same data, yet we seem to have reached opposite conclusions. - Riku reports that 33 out of 82k files have different compression when using current gzip vs. 10-year-old gzip. I'd be surprised if any of those binary packages hadn't been superseded long ago. It's not a guarantee, but I think the risks, and ultimate cost, of relying on gzip output to not change often and to just do sourceful rebuilds when it isn't are a lot smaller than if we go about manually splitting our packages further. - The cases where gzip output has been reported to not be reproducible seem to all boil down to a single issue with gzip being passed different arguments due to the unreproducible nature of *find*'s output. A patch has been made available already on the bug, and this patch seems to address the instances of the problem that we've hit so far in the Ubuntu archive. Now, it's worth following up with gzip upstream about our concerns, but even without that, I just don't see this being problematic. I think we have to do something saner with changelog files eventually regardless, but I'm curious: how did Ubuntu deal with the binNMU problem that Guillem identified? If you binNMU a library on amd64 but not on i386, as near as I can tell that's going to make the library not work for multiarch until the next sourceful upload, no? I think Ubuntu has binNMUs; haven't you run into this issue? Ubuntu deals with this problem by never having supported binNMUs. ;) This is a recognized flaw in Launchpad that folks want to address, but it does mean that this wasn't a blocker for ramping up multiarch there. But there have been discussions with the Debian buildd admins about how to solve this, with a reasonable consensus for splitting the binNMU changelogs into architecture-specific files. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120211185237.ga10...@virgil.dodds.net
Re: Summary: dpkg shared / reference counted files and version match
* Guillem Jover [2012-02-10 23:56 +0100]:
* binNMUs for the same version might not be co-installable because doc
generators, compressors, etc, might not always produce the same output.
... A possible fix, but only for the compressed files case might be to
ship them uncompresesd, but that counters the desire to reduce wasted
space.
An other fix is to reuse the compressed files from an existing binary
package (after verifying that the file's uncompressed content is the
same) as last resort solution. Such a tool could be used on buildds and
manually by maintainers if necessary, for example for binNMUs if we do
not solve the current gzip problem or if gzip's format will change.
* binNMUs for the same version cannot be co-installed anyway as their
changelogs differ.
Packages could ship /u/s/d/package/changelog:#ARCH#.{gz,Debian.gz}, and
in postinst create a symlink from the current changelog location to an
arch qualified one:
bn=/usr/share/doc/#PACKAGE#/changelog
ext=Debian.gz
[ -e $bn:#ARCH#.$ext ] || [ -L $bn:#ARCH#.$ext ] || ext=gz
[ -e $bn.$ext ] || [ -L $bn.$ext ] ||
ln -sn changelog:#ARCH#.$ext $bn.$ext 2/dev/null || :
The prerm script would either remove or change the symlink if it points
to the package being removed.
Notes/Remarks :
* Instead of symlinks, diversions could be used, possibly in
combination with hardlinks if the files do not differ.
* Line 2 and 3 could be replaced by an assignment, given that there are
two variants of both scripts and debhelper includes the appropriate
one.
* The according prerm currently has 373 bytes, and used to have way
more before I switched to a more compact code.
* If 'changelog' and architecture would be separated by a period and
not by a colon, it would be impossible to distinguish between, for
example, changelog.amd64.gz and changelog.old.gz without knowing
a complete list of all possible architectures.
* test -ef (as shell builtin) is not allowed in maintainer scripts.
Regards
Carsten
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120212013648.gb17...@furrball.stateful.de
Re: Summary: dpkg shared / reference counted files and version match
As the maintainer of a few (popular) library packages I consider splitting these packages a complex and annoying workaround for deficiencies in tools. It is not true that splitting the package is a one time action, every release which adds new files will require dealing with the split. Why was the implicit Replaces scheme not considered? -- ciao, Marco signature.asc Description: Digital signature
Re: Summary: dpkg shared / reference counted files and version match
* Guillem Jover guil...@debian.org, 2012-02-10, 23:56: [ Obviously this “summary” could be considered biased, but I do think the facts presented are accurate. ] Well, biased in an euphemism here... The two reasons for the shared / reference counted files (refcnt from now on) implementation in dpkg have been: * To avoid massive package proliferation (due to the mandated copyright and changelog files), thus the work involved in a one time split and the size increase in Packages indices. * To avoid unneeded file duplication, thus wasted space (due to those mandated files, but also partially just as a consequence of not splitting files into new arch:all packages, per above). Personally, I don't consider any of these reasons very important. How about: * Because this the obvious and elegant way of doing things. It makes multiarchification easy for packagers, and invisible for uses, including those users who don't care about multi-arch (unless they rely on paths to the libraries, which they never should do). * Deploying refcnt means that M-A:same packages must always be at the same exact installed version, so that the file contents can match. ↓ More difficult upgrade paths, as this ties the different arch dependency trees around M-A:same barriers. By allowing co-installation of two different versions of the same package, you are opening a can of worms, regardless of whether refcnt is implemented or not. * binNMUs need to be performed in lockstep for *all* architectures, because the installed versions need to match. ↓ Causing useless buildd usage and user downloads for arches not affected. “Fixing” this by making dpkg treat binNMU versions specially, besides being just another special case needed for M-A:same packages, What do you mean by “another”? Yes, MA:same packages needs special treatment, because they _are_ special. * binNMUs for the same version cannot be co-installed anyway as their changelogs differ. ↓ That could be “fixed” by using the same email address and a hardcoded date, or not including the binNMU entry at all, or moving that entry to a new field, etc. All of which seem like ugly hacks, or a possible loss of information. http://lists.debian.org/debian-devel/2012/02/msg00316.html * Once implemented, this “feature” cannot be taken out, *ever*. This boils down to “I don't like it”. If it's useful, why would one consider taking it out? Proposed solution - M-A:same packages cannot have any conflicting files with their foreign counterparts. Thus: For files in M-A:same packages under a pkgname based path, the pkgname should always be arch-qualified with the Debian architecture. Most of these could be handled automatically by debhelper and cdbs, this includes things like: /usr/share/doc/pkgname/ /usr/share/bug/pkgname /usr/share/lintian/overrides/pkgname /usr/share/mime-info/pkgname.* /usr/share/menu/pkgname ... This will require changes to the Policy, to which I (and hopefully other developers) will object. Please don't throw into the mud work of individual developers (including me) who already converted their packages to multi-arch. Thank you very much. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120211001446.gb2...@jwilk.net
Re: Summary: dpkg shared / reference counted files and version match
Hi, Jakub Wilk wrote: How about: * Because this the obvious and elegant way of doing things. It makes multiarchification easy for packagers, and invisible for uses, including those users who don't care about multi-arch (unless they rely on paths to the libraries, which they never should do). I suppose that is a matter of opinion. The need to make sure files that are not arch-qualified match at least functionally between architectures is subtle and not very easy. Tying package versions in different architectures, which makes dependency chains more rigid, is not invisible to users. However, in the special case of header files (which are text files that almost never vary between architectures), I agree that it would be nice to be able to preserve the simplicity of the current approach from the packager's perspective. [...] By allowing co-installation of two different versions of the same package, you are opening a can of worms, regardless of whether refcnt is implemented or not. Could you elaborate on this? As long as dependencies are accurate, I don't see how allowing co-installation of the same package for two different architectures at different versions is any more complicated than pinned to the same version. [...] * Once implemented, this “feature” cannot be taken out, *ever*. This boils down to “I don't like it”. If it's useful, why would one consider taking it out? After trying out the approach without refcounted files for a while, it would be painless for the project to say This is too much trouble and add refcounting. By contrast, it is very painful to move in the other direction. I think that is worth taking into account. [...] Proposed solution [...] This will require changes to the Policy, to which I (and hopefully other developers) will object. Last time I checked, multiarch is not in policy yet. Please don't throw into the mud work of individual developers (including me) who already converted their packages to multi-arch. I agree that the extra work of removing multi-arch: same for existing -dev packages that have been converted is a major downside. And on the other hand, the need throughout Debian infrastructure to support the very fragile refcount approach would be a downside to that approach. This is not so one-sided as you seem to be suggesting. Jonathan who wishes he knew of a fifth approach without the downsides of the others proposed so far :) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120211005559.GA32671@burratino
Re: Summary: dpkg shared / reference counted files and version match
On Fri, Feb 10, 2012 at 06:56:00PM -0600, Jonathan Nieder wrote: Could you elaborate on this? As long as dependencies are accurate, I don't see how allowing co-installation of the same package for two different architectures at different versions is any more complicated than pinned to the same version. I think we're likely to see a lot of bugs introduced in the process of making the dependencies accurate if we go this route. So instead of refcounting the files, we move them to a new arch: all package. Ok; now, suppose some of these files aren't actually architecture-independent: they're data but they're generated differently on different architectures, and the library expects the data in its native architecture-dependent format. (See the parallel thread about endianness of data files for examples.) Doesn't this proposal eliminate one of our best defenses against this packaging error? Having them kicked back as mismatched files, either by dpkg or by the archive, seems better to me than letting them land on the user's system and break at runtime. [...] Proposed solution [...] This will require changes to the Policy, to which I (and hopefully other developers) will object. Last time I checked, multiarch is not in policy yet. multiarch library paths are in policy (as an exception to the FHS). Please don't throw into the mud work of individual developers (including me) who already converted their packages to multi-arch. I agree that the extra work of removing multi-arch: same for existing -dev packages that have been converted is a major downside. And on the other hand, the need throughout Debian infrastructure to support the very fragile refcount approach would be a downside to that approach. Which infrastructure do you have in mind? I don't see anything that's needed besides a dpkg implementation (which we have) and some tools to sanity-check coinstallability (which 1. we would need anyway, and 2. we also have a preliminary implementation of). -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: Summary: dpkg shared / reference counted files and version match
On Fri, Feb 10, 2012 at 11:56:20PM +0100, Guillem Jover wrote: [ Obviously this “summary” could be considered biased, but I do think the facts presented are accurate. ] The two reasons for the shared / reference counted files (refcnt from now on) implementation in dpkg have been: Well the bias comes up quite quick when you conveniently omit the MAIN reason right here and rather discuss it at the end of document.. * Because shared files packaging simpler for maintainers. Package profiliation and duplication of arch-independent data are merely effecty that happen when packagers workaround the lack of shared identical files. * To avoid massive package proliferation (due to the mandated copyright and changelog files), thus the work involved in a one time split and the size increase in Packages indices. * To avoid unneeded file duplication, thus wasted space (due to those mandated files, but also partially just as a consequence of not splitting files into new arch:all packages, per above). This has the following implications: * Deploying refcnt means that M-A:same packages must always be at the same exact installed version, so that the file contents can match. ↓ More difficult upgrade paths, as this ties the different arch dependency trees around M-A:same barriers. * binNMUs need to be performed in lockstep for *all* architectures, because the installed versions need to match. ↓ Causing useless buildd usage and user downloads for arches not affected. “Fixing” this by making dpkg treat binNMU versions specially, besides being just another special case needed for M-A:same packages, would be wrong, as arch-indep content can actually change between builds, ex. generated documentation. * binNMUs for the same version might not be co-installable because doc generators, compressors, etc, might not always produce the same output. ↓ This is a pretty fragile thing to rely on. New architectures or local builds might give a hard time if generated output changed in the past. A possible fix, but only for the compressed files case might be to ship them uncompresesd, but that counters the desire to reduce wasted space. * binNMUs for the same version cannot be co-installed anyway as their changelogs differ. ↓ That could be “fixed” by using the same email address and a hardcoded date, or not including the binNMU entry at all, or moving that entry to a new field, etc. All of which seem like ugly hacks, or a possible loss of information. One solution for the binNMU changelogs and generated docs would be to use arch-qualified paths for those files. That is much more lightweight solution that arch-qualifying all files, even if identical. Given all the above, I'll be pulling off for now the file refcnt and version match logic from my pu/multiarch/master branch. If some compelling arguments are brought up, something I honestly don't really see happening, then they can be actually reintroduced at any point. Proposed solution - M-A:same packages cannot have any conflicting files with their foreign counterparts. Thus: For files in M-A:same packages under a pkgname based path, the pkgname should always be arch-qualified with the Debian architecture. Most of these could be handled automatically by debhelper and cdbs, this includes things like: /usr/share/doc/pkgname/ /usr/share/bug/pkgname /usr/share/lintian/overrides/pkgname /usr/share/mime-info/pkgname.* /usr/share/menu/pkgname ... I find the requring arch-qualified path for for arch-independent data ugly system architecture. But of course the beauty of architecture is in the eye of the beholder (and lets not forget that Unix with its worse-is-better philosophy[1] was never intended to be architectural masterpiece). Personally if leaving out shared files makes you upload multiarch enabled dpkg to unstable before sagrada familia is complete, i can live it (cursing silently in my room converting packages to the new requirement...). I can take the trade-off of having something better-than-current soon over having the perfect version in distant future... Riku -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120211012833.ga21...@afflict.kos.to
Re: Summary: dpkg shared / reference counted files and version match
Steve Langasek vor...@debian.org writes: On Fri, Feb 10, 2012 at 06:56:00PM -0600, Jonathan Nieder wrote: I agree that the extra work of removing multi-arch: same for existing -dev packages that have been converted is a major downside. And on the other hand, the need throughout Debian infrastructure to support the very fragile refcount approach would be a downside to that approach. Which infrastructure do you have in mind? I don't see anything that's needed besides a dpkg implementation (which we have) and some tools to sanity-check coinstallability (which 1. we would need anyway, and 2. we also have a preliminary implementation of). We need a guarantee that gzip will always produce the same output, which we already know isn't the case and which doesn't look sustainable going forward. That's looking rather uncomfortable. The preliminary results there point to that being somewhat problematic. I think we have to do something saner with changelog files eventually regardless, but I'm curious: how did Ubuntu deal with the binNMU problem that Guillem identified? If you binNMU a library on amd64 but not on i386, as near as I can tell that's going to make the library not work for multiarch until the next sourceful upload, no? I think Ubuntu has binNMUs; haven't you run into this issue? -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87zkcqrw2w@windlord.stanford.edu
Re: Summary: dpkg shared / reference counted files and version match
Guillem Jover wrote: Descriptions are only downloaded once nowadays. Not actually true; apt-get update downloads all package descriptions without even pdiffs nowadays, every time there's a change to any single description. Get:22 http://ftp.nl.debian.org unstable/main Translation-en [3,882 kB] /usr/share/doc/pkgname/ /usr/share/bug/pkgname /usr/share/lintian/overrides/pkgname /usr/share/mime-info/pkgname.* /usr/share/menu/pkgname ... (Joey, I'm guessing you might consider it too late to do some of these in debhelper for compat level 9, right?) I wouldn't worry about this, there's still time to make whatever changes might be needed in v10. It's not actually clear this would be a change that needs a compat level at all. More worrying is how to make apt-listchanges and everything else use these locations in time. -- see shy jo signature.asc Description: Digital signature
