Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
On Thu, 3 Feb 2011 15:39:49 + Wookey wrote: > dpkg-cross in fact only picks files out of packages by positive > identification as libraries or headers. It misses out generic 'other > stuff' in ((/usr)/lib, which generally works pretty well, but in > this tcl case it's not suffient for tcl-depending apps to cross-build. > > Bug http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599206 discusses > this issue. ... and http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=611736 is probably going to be the root bug (renamed) to handle the clarification of exactly what files we keep and how we rationalise the legacy regular expressions in dpkg-cross. Please add further specific cases to that bug. > dpkg-cross is intended to put files needed for cross-building into > -cross packages and it's currently missing this one out. Unfortunately > because it doesn't match any of the 'standard paterns for cross-useful > files' this can only be fixed by adding a fairly specific regexp, > which is worrying close to special-casing or deciding that in fact > just fishing out specific things from /usr/lib is too conservative and we > should take the view that everything in /usr/lib is potentially useful > in cross-building and should be copied into -cross packages. Generally, from a cursory view of what is left out, the situation we have is this: 0: dpkg-cross supports autotools cross-building through a series of very detailed, very complex and potentially fragile regular expressions but, importantly, ALSO uses a series of very detailed, very complex and potentially equally fragile regular expressions on the CONTENTS of those files, some of which Debian is quietly trying to drop from packages because of other problems. (e.g .la files) 1: dpkg-cross has explicit support for pkg-config which appears to work perfectly well, principally because pkg-config is inherently less complex than the entirety of autoconf|automake|libtool|dpkg-shlibdeps etc. 2: dpkg-cross has support for CMake - although only as a direct result of 0: and 1: 3: dpkg-cross has no idea how to help SCons or any number of other build systems out there, but then it mostly doesn't have to because many of those simply don't compile stuff, (they create Arch:all packages) and the ones that do compile stuff aren't used by sufficient numbers of cross-building people for sufficient complaints to be seen for dpkg-cross to have had the support created. 4: Nobody gave a damn about Tcl until sqlite added bindings. 5: Nobody adds support to dpkg-cross until someone complains loudly enough *and* comes up with sane patches. 6: dpkg-cross has huge amounts of legacy code which someone added at somepoint because it was important but nobody has actually had the guts to unilaterally remove because we can't tell if the original package has since been fixed. (s/fixed/broken in a different way/). > In multiarch world everything in (/usr)/lib is going to end up in > /usr/lib/ or /lib/, unless packages are > re-arranged to put them elsewhere, and we expect this to work > fine so perhaps dpkg-cross should start doing the same thing, and thus > discuver any problems this does potentially create. Would > that actually do any harm? What files which are currently missed out > of -cross packages would actually cause breakage if copied over into > /usr//lib? Let's not break dpkg-cross fundamentally for everyone though. We can choose to make a different dpkg-cross which is FAR simpler (because it blindly moves files without any kind of safeguard) but as this does not involve fixing the contents of certain files, numerous autotools packages will break. So, if people want a broken dpkg-cross for testing, let's have a dpkg-multi-cross which breaks their cross-building world (using a different Provides: and conflicting with the "standard" cross packages) and leave the existing world alone. That version of dpkg-multi-cross would be trivial to write - unpack the .deb, unconditionally move all files in ./usr/lib to ./usr/$triplet, handle /usr/include and remove everything else. Repack the .deb as -cross. Break world. > I just tried a modified dpkg-cross on a pile of packages and whilst > many come out the same, you do get quite a lot more files in some > packages and some packages that were previously null now have stuff in > them. e.g apache-modules. So there is quite a lot of bloat, but > probably no breakage. Retaining the changes to the contents of the files whilst simply adding lots more *stuff* to -cross packages is the least harmful option. However, because the contents haven't changed, it doesn't actually help us identify the issues that would arise with Multiarch much. -- Neil Williams = http://www.data-freedom.org/ http://www.linux.codehelp.co.uk/ http://e-mail.is-not-s.ms/ pgpWzNTlQbNBl.pgp Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinf
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
+++ Loïc Minier [2011-02-01 12:50 +0100]: > On Tue, Feb 01, 2011, Wookey wrote: > > But if something is looking for arch-independent stuff in /lib then in > > general that's wrong, and I'm not aware of any examples of > > correctly-packaged packages that need this. Any arch-independent files > > will be supplied by an arch all package that the build should depend > > on if needed. > > I might be getting your point wrong, but I certainly see a lot of files > in /lib itself which are arch-independent data used for early boot > (before /usr is available); PNG files and text files which would be > identical on all architectures. Sorry, I wasn't being very clear. By 'something is looking for arch-independent stuff in /lib' I really mean in /usr//lib, used during cross-building, (which will be put there by dpkg-cross-ed packages) (or in /usr/lib/ or /lib/ put there by dpkg on multiarch systems). Yes there are various things in /lib that are not arch-dependent. dpkg-cross does not put most(any?) of those in -cross packages. In fact this is so true that it doesn't copy /usr/lib/tcl8.5/tclConfig.sh over into the cross package either. I should have checked that. Bum. dpkg-cross in fact only picks files out of packages by positive identification as libraries or headers. It misses out generic 'other stuff' in ((/usr)/lib, which generally works pretty well, but in this tcl case it's not suffient for tcl-depending apps to cross-build. Bug http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599206 discusses this issue. dpkg-cross is intended to put files needed for cross-building into -cross packages and it's currently missing this one out. Unfortunately because it doesn't match any of the 'standard paterns for cross-useful files' this can only be fixed by adding a fairly specific regexp, which is worrying close to special-casing or deciding that in fact just fishing out specific things from /usr/lib is too conservative and we should take the view that everything in /usr/lib is potentially useful in cross-building and should be copied into -cross packages. In multiarch world everything in (/usr)/lib is going to end up in /usr/lib/ or /lib/, unless packages are re-arranged to put them elsewhere, and we expect this to work fine so perhaps dpkg-cross should start doing the same thing, and thus discuver any problems this does potentially create. Would that actually do any harm? What files which are currently missed out of -cross packages would actually cause breakage if copied over into /usr//lib? I just tried a modified dpkg-cross on a pile of packages and whilst many come out the same, you do get quite a lot more files in some packages and some packages that were previously null now have stuff in them. e.g apache-modules. So there is quite a lot of bloat, but probably no breakage. Internally we will use a dpkg-cross modified to add /usr/lib/*/tclConfig.sh to the list of things that are important for cross-building. This means we will notice any other 'awkward cases' due to missing files. An alternative view is that anything (such as sqlite3) depending on tclConfig.sh to build tcl extensions is broken and should be changed to use some other mechanism, and until then simply cannot be cross-built using the dpkg-cross mechanism. I am not familiar enough with tcl extensions to know if this is a reasonable stance or not, but given that it works just fine, and it's not hard to deal with, and (after the fix in debian bug#611650) it will carry on 'just working' in multiarch, I'm not convinced this is a reasonable stance. Which leaves us with deciding whether to just copy over tclConfig.sh or everying in /usr/lib/*/* in dpkg-cross? Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
On Tue, 1 Feb 2011 12:50:48 +0100 Loïc Minier wrote: > On Tue, Feb 01, 2011, Wookey wrote: > > But if something is looking for arch-independent stuff in /lib then in > > general that's wrong, and I'm not aware of any examples of > > correctly-packaged packages that need this. Any arch-independent files > > will be supplied by an arch all package that the build should depend > > on if needed. > > I might be getting your point wrong, but I certainly see a lot of files > in /lib itself which are arch-independent data used for early boot > (before /usr is available); PNG files and text files which would be > identical on all architectures. /lib vs /usr/lib is a different argument. I'm working on the basis that Wookey was meaning /usr/lib/ compared to /usr/share. The point about dpkg-cross is that it doesn't blindly take everything from /usr/lib and propagate that into /usr/$triplet/lib, it picks out stuff that it knows is useful to a cross package. Symlinks are included because there's no need to copy libfoo.so.0.0.1 as libfoo.so.0 etc. There's a complex list of regular expressions, allowing header files, static linking files, object files, .la files, anything in /usr/include/ and stuff in /usr/lib/pkg-config/ and stuff already under /usr//lib/. This list has aggregated over time. As multiarch is as far away as ever, I will discuss pruning that list significantly once Squeeze is released, leading to a dpkg-cross 3.0.0. The final list will only include stuff which dpkg-cross can reliably identify *and* which is absolutely essential to cross-builds. /usr/share won't be included, except for pkg-config files. -- Neil Williams = http://www.data-freedom.org/ http://www.linux.codehelp.co.uk/ http://e-mail.is-not-s.ms/ pgpBZmMGGvFoE.pgp Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 31/01/11 20:18, Steve Langasek wrote: > On Mon, Jan 31, 2011 at 05:16:15PM +, Wookey wrote: >> So, the questions is - which of these is the correct fix: >> 1) make dpkg-cross copy over symlinks even when they point into >> /usr/share >> 2) make tcl8.5 put tclConfig.sh in /usr/lib/tcl8.5 instead of >> /usr/share/tcltk/tcl8.5 >> 3) make sqlite (and similar packages) look in /usr/share/tcltk/tcl8.5 >> instead of /usr/lib/tcl8.5 when building > > 1+2: dpkg-cross should also copy over symlinks that point to /usr/share. The reason not to copy is that the target of the symlink does not exist in the -cross package and the one that exists in the native package in /usr/share contains all the wrong paths. A dangling symlink would be left if the -cross package is not installed. If the symlink is to be copied over not into /usr/share but into /usr/arm-linux-gnueabi/share then the native version of the file will still exist in /usr/share and none of the existing tools will look into /usr/arm-linux-gnueabi/share to find the correct version without a new wrapper. Nobody wins. dpkg-cross cannot reliably identify which symlinks should be preserved and how. dpkg-cross cannot copy across all symlinks which point to /usr/share because a lot of those are unwanted. The correct fix is 2. The only possible thing dpkg-cross could do is reverse the symlink and move the file from /usr/share into /usr/arm-linux-gnueabi/lib/ if a symlink to /usr/lib/ exists and not create the symlink in /usr/arm-linux-gnueabi/share at all - at which point the native version of the package still contains the symlink from /usr/share to /usr/lib/ with the native paths. Is that what is actually being requested for 1.? I don't see that it's reliable - it's masking the real bug with a horrible hack, again. How does this help when the package itself is buggy by not complying with the FHS and the cross-build looks in the wrong place? > When such symlinks exist, it's almost invariably because *something is > looking for the information in that location*. When is this not actually a bug in that package? The maintainer has agreed to fix the actual bug by implementing option 2 above, which we all seem to agree is the appropriate and optimal fix for the original issue. Under what circumstances would some other package fall into this problem *without* also being buggy in exactly the same way as tcl8.5? Yes, let's clarify policy but I don't think it's correct to expect dpkg-cross to handle packages which are simply buggy whilst risking breaking other packages which do things properly. dpkg-cross doesn't (and shouldn't) support package-specific exceptions in how -cross packages are created. > So as a general policy, they > should also be made available in a crossed package. The /usr/share/ location *must* change in the cross package so that it can be installed alongside the native, at which point any merit of retaining /usr/share is lost in the cross package because a wrapper script will still be needed to find the modified location, handle the dangling symlink or risk getting the wrong data (because the data in this case is clearly wrong for the cross build). Either the file is in the wrong place (as in this case) or the change of location is simply going to break without a package-specific wrapper script anyway because the locations of things in /usr/share should not (need to) change to support a cross-build. - -- Neil Williams = http://www.linux.codehelp.co.uk/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNR0GdAAoJEPFn5DyBQ7aC2lcP/1K8gLDnYffSZSb9lVarDh89 Rmvk1FLIHzJQ29kWYLE29ybpLCma7ms4T+PloVv7KLWMvS36jkLiCuM+a/sP8eU3 3uPkZ5rFTflXy8q2H4MhF/CQa94QExxthpDHgeQntz9j+5dKp+Nj3uNQaHjdh6XC aY+tyMulpwaR/B0gpQXtgAKffqvKtInLQDXnNd4YWEXbRgMndm9d37GyzYdVdOsz /udBf57SpQXuLX2M4d1cX1QmbUpwMqD3P7PeZQj7gcwX0loupm6MNFb7VQJM6gKp d6o/AYBBg2X3addJMDVQZdNsNYfFm0F6JfTlk0pgNyFUOGPFmyNFxOT3QjufloY9 ZH+xmvgC9UL6gSv5voQUIezn2MSLGblP648RwW8RgdHslNG60v6I22VcZSQDkcw4 BE2CUn/Y1w+q1A74MGSAPV7lB2b9QRnxjec1xfyo11as27mudlDKwFtrGI6mzDGk oZf/SYGkCQ1cOscq1aVJGkJocfA4rlwbM67zfFybz5W9j8UZSYOVX9F1oh/OpRPF UQNaWyiOF4lf/EUf3Ct7F6Dwz9TPKhxpRgwg3kfQqTlsgjLK/a1zIf4F1h1EjP4H f0sAhgJrYQkFUW8w1LDhK92EEDZdHW7De0cA/7tx4h/8ay+NdrcpEOMWTFDoGrcy QC4k9mp+Xwnaq78yHWaz =clO0 -END PGP SIGNATURE- ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
On Tue, Feb 01, 2011, Wookey wrote: > But if something is looking for arch-independent stuff in /lib then in > general that's wrong, and I'm not aware of any examples of > correctly-packaged packages that need this. Any arch-independent files > will be supplied by an arch all package that the build should depend > on if needed. I might be getting your point wrong, but I certainly see a lot of files in /lib itself which are arch-independent data used for early boot (before /usr is available); PNG files and text files which would be identical on all architectures. -- Loïc Minier ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
+++ Steve Langasek [2011-01-31 12:18 -0800]: > > So, the questions is - which of these is the correct fix: > > 1) make dpkg-cross copy over symlinks even when they point into > > /usr/share > > 2) make tcl8.5 put tclConfig.sh in /usr/lib/tcl8.5 instead of > > /usr/share/tcltk/tcl8.5 > > 3) make sqlite (and similar packages) look in /usr/share/tcltk/tcl8.5 > > instead of /usr/lib/tcl8.5 when building > > 1+2: dpkg-cross should also copy over symlinks that point to /usr/share. > When such symlinks exist, it's almost invariably because *something is > looking for the information in that location*. So as a general policy, they > should also be made available in a crossed package. But if something is looking for arch-independent stuff in /lib then in general that's wrong, and I'm not aware of any examples of correctly-packaged packages that need this. Any arch-independent files will be supplied by an arch all package that the build should depend on if needed. dpkg-cross has gone for 10 years without needing symlinks pointing into /usr/share, so I'm reluctant to add them without some evidence that it's actually correct. Its policy of only crossing arch-dependent files is the right one, I believe. (It does allow symlinks within /usr/src which presumably has/had a good reason.) Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
Hi Wookey, On Mon, Jan 31, 2011 at 05:16:15PM +, Wookey wrote: > Debian policy (8.2) says: > --- > It is recommended that supporting files and run-time support programs > that do not need to be invoked manually by users, but are nevertheless > required for the package to function, be placed (if they are binary) > in a subdirectory of /usr/lib, preferably under /usr/lib/package-name. > If the program or file is architecture independent, the recommendation > is for it to be placed in a subdirectory of /usr/share instead, > preferably under /usr/share/package-name. Following the package-name > naming convention ensures that the file names change when the shared > object version changes. > Files and support programs only useful when compiling software against > the library should be included in the development package for the > library > --- > A maintainer reading the above could reasonably decide that /usr/share > was the right place for this file, because the file itself (being just > shell) is arch-independent. The problem is that the contents are > arch-dependent. Policy is not at all clear on this distinction (It's > been making my head hurt all day). For multiarch, or existing > dpkg-cross cross-compiling, to work, arch-dependent needs to mean > either form _or_ content (see below for elaboration). I disagree that this would be a reasonable thing for the maintainer to do. The current policy language talks about architecture-dependence of the *file*, not of a file *format*, and there are obviously file formats that are architecture-independent but contain architecture-specific data that must therefore be part of an architecture: any package. So I think this is a clear policy violation in the existing tcl8.5-dev package; even if 8.2 doesn't prohibit the current behavior, the FHS surely does.[1] So I think your filing of bug #611650 was the correct course of action and the maintainer appears to agree. That said, I'm also happy to second patches to policy that clarify the wording and save maintainers from thinking it's ok to ship such files under /usr/share when it isn't. > So, the questions is - which of these is the correct fix: > 1) make dpkg-cross copy over symlinks even when they point into > /usr/share > 2) make tcl8.5 put tclConfig.sh in /usr/lib/tcl8.5 instead of > /usr/share/tcltk/tcl8.5 > 3) make sqlite (and similar packages) look in /usr/share/tcltk/tcl8.5 > instead of /usr/lib/tcl8.5 when building 1+2: dpkg-cross should also copy over symlinks that point to /usr/share. When such symlinks exist, it's almost invariably because *something is looking for the information in that location*. So as a general policy, they should also be made available in a crossed package. Cheers, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org [1] /usr/share : Architecture-independent data signature.asc Description: Digital signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
(please drop cc's other than debian-policy in replies if you want to work on that) Hi Wookey, Wookey wrote: > Debian policy (8.2) says: > --- > It is recommended that supporting files and run-time support programs > that do not need to be invoked manually by users, but are nevertheless > required for the package to function, be placed (if they are binary) > in a subdirectory of /usr/lib, preferably under /usr/lib/package-name. > If the program or file is architecture independent, the recommendation > is for it to be placed in a subdirectory of /usr/share instead, [...] > For multiarch, or existing > dpkg-cross cross-compiling, to work, arch-dependent needs to mean > either form _or_ content (see below for elaboration). I always had thought that it's content and that the "if they are binary" is only a red herring. I agree with you that the policy ought to be clarified. Thanks for bringing this up. Regards, Jonathan ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
For this specific tcl issue, there was some discussion in Debian #599206; I didn't comment back on the bug, but I liked Goswin's proposal -- Loïc Minier ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
