Bug#317082: Not just a dpkg bug
Frank Lichtenheld wrote: Yeah, that's indeed a problem. But that isn't solved by the current implementation either. When I think about it there is now way the -l option (if pointing to a directory that is not known to dpkg) changes anything about the build currently since the local shlibs information is considered before using the ldd paths. And the ldd paths are only used in conjunction with dpkg --search ... The only thing it does is supressing the error couldn't find path for X. You're right. BTW, dpkg-shlibdeps -l does not seem to be documented in its manual page or help text ATM. -- see shy jo signature.asc Description: Digital signature
Bug#317082: Not just a dpkg bug
On Wed, Jan 25, 2006 at 01:18:55PM -0500, Joey Hess wrote: BTW, dpkg-shlibdeps -l does not seem to be documented in its manual page or help text ATM. I was talking about dh_shlibdeps -l and the LD_LIBRARY_PATH handling in dpkg-shlibdeps. dpkg-shlibdeps has indeed no -l option. Sorry if that was confusing. Gruesse, -- Frank Lichtenheld [EMAIL PROTECTED] www: http://www.djpig.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
On Sun, Jan 22, 2006 at 10:14:16PM +0100, Frank Lichtenheld wrote: I've implemented this option. Patch and new script (since the patch is garbled with a little code clean-up I did while going through the script) are attached. Comments and/or testing welcome. Comment to myself: The current patch probably breaks dh_shlibdeps -l option because it doesn't honor LD_LIBRARY_PATH. Can someone tell me a package which really needs this option? The ones that I tested so far seem to build equally fine without it... This option wasneeded when package builds a binary that uses libraries built from the same source package. In this case, libraries may not be available outside of package build directory; so ldd called from dpkg-shlibdeps won't find them witout LD_LIBRARY_PATH. Don't yet know how this situation is handled in your patched dpkg-shlibdeps... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
On Tue, Jan 24, 2006 at 06:05:43PM +0300, Nikita V. Youshchenko wrote:
On Sun, Jan 22, 2006 at 10:14:16PM +0100, Frank Lichtenheld wrote:
Comment to myself: The current patch probably breaks dh_shlibdeps
-l option because it doesn't honor LD_LIBRARY_PATH. Can someone
tell me a package which really needs this option? The ones that
I tested so far seem to build equally fine without it...
This option wasneeded when package builds a binary that uses libraries
built from the same source package. In this case, libraries may not be
available outside of package build directory; so ldd called from
dpkg-shlibdeps won't find them witout LD_LIBRARY_PATH.
Don't yet know how this situation is handled in your patched
dpkg-shlibdeps...
Hmm, but dpkg-shlibdeps already tries to take care of this by using
all local shlibs files. Can someone point me to a package that really
misbuilds without -l ? This would really help me to understand the
real problem.
dpkg-shlibdeps calls ldd, which will just fail if LD_LIBRARY_PATH won't
point to directories with local libraries.
defined($c= open(P,-|)) || syserr(cannot fork for ldd);
if (!$c) { exec(ldd,--,$exec[$i]); syserr(cannot exec ldd); }
while (P) {
if (m,^\s+(\S+)\s+=\s+(\S+)\s+\(0x.+\)?$,) {
$so2path{$1} = $2;
}
}
close(P); $? subprocerr(ldd on \`$exec[$i]');
If I understand this code correctly, it will go to subprocerr() and die if
ldd returns non-zero.
Any package that uses local libs will be affected. E.g. zlib (if built in
environment where zlib is not yet installed).
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
On Tue, Jan 24, 2006 at 06:43:02PM +0300, Nikita V. Youshchenko wrote: dpkg-shlibdeps calls ldd, which will just fail if LD_LIBRARY_PATH won't point to directories with local libraries. That's not true. ldd will just happily print libfoo.so.1 = not found and exit with exit code 0. So this doesn't terminate dpkg-shlibdeps. Gruesse, -- Frank Lichtenheld [EMAIL PROTECTED] www: http://www.djpig.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
On Tue, Jan 24, 2006 at 06:43:02PM +0300, Nikita V. Youshchenko wrote: dpkg-shlibdeps calls ldd, which will just fail if LD_LIBRARY_PATH won't point to directories with local libraries. That's not true. ldd will just happily print libfoo.so.1 = not found and exit with exit code 0. So this doesn't terminate dpkg-shlibdeps. Hmm... New information for me. I was sure that ldd, as any other [good-behaving] unix utility, exits with non-zero status on errors. A bug in ldd? But if ldd does not dislike unresolved libraries, I see no other problems with dropping -l. Library files from non-standard paths won't be found by dpkg anyway, so can't be processed in way other than shlibs.local In this sence, new information that comes from ldd finding libraries in LD_LIBRARY_PATH dirs, is useless. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
Hello people. objdump isn't a solution either, while it sometimes can read the other shared library, it doesn't provide the linker search patch which is of critical importance to get this stuff right. Hmm... But what for is that search path? As far as I understand, dpkg-shlibdeps should just get NEEDED sonames, and match those against available shlibs files. And that is what code actually does. Path information is used only if something is not found in this way - that means, package that provides the library does not provide shlibs data, which is a bug anyway. This isn't quite true I think. The current dpkg-shlibdeps code works like this: 1) use ldd binary to find the paths to the linked libraries 2) use objdump -p binary to actually check which of this libraries are listed as NEEDED (Are there cases where ldd lists libraries that are not NEEDED?) 3) check the shlibs.local and shlibs.overides files for matches 4) for remaining libraries search the corresponding package with dpkg --search path Here the ldd information is used! 5) check the shlibs files of the packages identified Looks so. I was mistaken. From reading the source code of dpkg-cross' version of dpkg-shlibdeps it works like this: c1) like 2) c2) like 3) c3) determine the path to the library by just prepending $crossroot or $crossroot64 to it (both variables which can be set by the user) c4) like 4) c5) like 5) I don't see how the current way this works can deal with several library directories like /lib /usr/lib /usr/X11R6/lib. But maybe I'm missing something? In cross-compile environment created by dpkg-cross, all libraries are installed in the same directory. As for this directory being user-configurable, I believe it's a misfeature came from old versions of dpkg-cross, which should be eventually fixed. While it's there, depending on -arch-cross packages is unreliable, which is something I wish to avoid. So looks it is safe to remove any path processing from dpkg-slibdeps, and use only objdump. If something breaks, it should be fixed elsewhere (i.e. proper shlibs data added to packages). If we don't use the path information from ldd there are several ways to go: 1) use dpkg --search but only with the library name from objdump, not with the full path. Questions: - Are there cases where the library name from objdump isn't actually the filename of the library? - How do we decide wether a found file is really a usable library? (parse /etc/ld.so.conf?) 2) we could try to use the ldconfig cache to make to work of ldd for ourself. Questions: - Is this really an advantage? Or has the cache the same problems ldd has? I sometimes have to work with MontaVista toolchains. They to provide cross-ldd tool that just implements the same library-searching logic for non-native binaries as dynamic libker implements for native ones. I don't know if this thing is free etc, but it could be easilly implemented from scratch if we decide we need one. 3) we could parse /etc/ld.so.conf and search for the libraries ourself. this is essentially a more general version of the current way dpkg-cross works AFAICS 4) we could just parse _all_ shlib files, perhaps even generating a cache from them. Questions: - Seems a complex solution (if it uses caching). Is any of the simpler solutions feasible? - How slow is this? I believe this is *the* correct way. If should be *much* faster than calling dpkg -S - just because dpkg -S actually loads all *.list files, which is many times larger than *.shlibs files. As I side effect, this will allow to remove LD_LIBRARY_PATH thing from dh_shlibdeps, which gives some unwanted effects when cross-compiling packages (like finding cross version of libz.so.1 in LD_LIBRARY_PATH and refusing to load native binaries that link against libz.so.1). The only question is that is will *require* packages to provide valid shlibs.local files. We may make an experiment: try to build entire archive using dpkg-shlibdeps that is based on shlibs file checking only, and see how many packages get different dependences. One more question that came to mind right now: is it possible that more than one installed package provides shlibs infomation for the same soname? From the other hand, what is the wanted behaviour in this case? 1) use dpkg --search but only with the library name from objdump, not with the full path. Questions: - Are there cases where the library name from objdump isn't actually the filename of the library? - How do we decide wether a found file is really a usable library? (parse /etc/ld.so.conf?) Note that this method is used in the patch for #285857 which I'm currently evaluating for inclusion. So it seems the answer to question 1 is no (or else the patch in #285857 is useless)... That patch
Bug#317082: Not just a dpkg bug
On Fri, Jan 20, 2006 at 11:57:28AM +0300, Nikita V. Youshchenko wrote: 2) we could try to use the ldconfig cache to make to work of ldd for ourself. Questions: - Is this really an advantage? Or has the cache the same problems ldd has? Hmm. In theory, ldconfig shouldn't require the ability to execute 64-bit binaries in order to build a cache of their paths. The only thing I don't know is how 64-bit vs. 32-bit libs are cached? It would after all have to be possible to distinguish between them in order to use the cache for this. 3) we could parse /etc/ld.so.conf and search for the libraries ourself. this is essentially a more general version of the current way dpkg-cross works AFAICS I think the only question here is whether it's appropriate for dpkg to have its own internal representation of the library search path, since that means duplicating information between glibc and dpkg. IMHO, this is still the least-bad option. 4) we could just parse _all_ shlib files, perhaps even generating a cache from them. Questions: - Seems a complex solution (if it uses caching). Is any of the simpler solutions feasible? - How slow is this? I believe this is *the* correct way. If should be *much* faster than calling dpkg -S - just because dpkg -S actually loads all *.list files, which is many times larger than *.shlibs files. As I side effect, this will allow to remove LD_LIBRARY_PATH thing from dh_shlibdeps, which gives some unwanted effects when cross-compiling packages (like finding cross version of libz.so.1 in LD_LIBRARY_PATH and refusing to load native binaries that link against libz.so.1). The only question is that is will *require* packages to provide valid shlibs.local files. Hrm? It shouldn't; dpkg-shlibdeps is already capable of checking shlibs files within the build directory. One more question that came to mind right now: is it possible that more than one installed package provides shlibs infomation for the same soname? Yes, AFAIK *every* biarch package is going to provide shlibs for libraries that differ only by library path, not by soname. So that's not very useful for solving the problem at hand. From the other hand, what is the wanted behaviour in this case? If you aren't able to distinguish the libraries according to the path of the library they provide, the *only* sensible thing to do in such a case is to bail. As for that question, AFAIK library filename and soname are completely unrelated. So there may be a soname 'ABCDE' that is provided by libxxx.so. A binary linked against that library will get 'NEEDED ABCDE'. And dynamic linker and ldd will find it, as long as there is 'ABCDE' symlink or file in dynamic linker search path. Uh, consequently they are *not* unrelated: the soname specified in NEEDED *must* be a filename that exists on the system, be it symlink or otherwise, and per policy this file must be contained in the library package (as opposed to just being created by ldconfig on package install). -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. [EMAIL PROTECTED] http://www.debian.org/ signature.asc Description: Digital signature
Bug#317082: Not just a dpkg bug
On Thu, Jan 19, 2006 at 12:14:35AM +0100, Frank Lichtenheld wrote:
So looks it is safe to remove any path processing from dpkg-slibdeps, and
use only objdump. If something breaks, it should be fixed elsewhere (i.e.
proper shlibs data added to packages).
If we don't use the path information from ldd there are several ways to
go:
1) use dpkg --search but only with the library name from objdump, not
with the full path.
Questions: - Are there cases where the library name from objdump isn't
actually the filename of the library?
- How do we decide wether a found file is really a usable
library? (parse /etc/ld.so.conf?)
FWIW, the most robust way to handle this probably looks something like:
foreach $dir (@searchpath) {
$var = `dpkg --search $file`
do_stuff;
last if found;
}
That would seem to address the Hurd case (lib is available multiple times in
the search path due to symlinks, but only one of these files is known by
dpkg), while also eliminating collisions with similarly named files that are
not in the library search path at all (e.g., objects intended for use with
LD_PRELOAD or something).
The only requirement is that dpkg have an internal representation of the
library search path for the object type -- part of which comes from
/etc/ld.so.conf, part of which is hard-coded in ld.so. Oh... and then
there's RPATH...
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
[EMAIL PROTECTED] http://www.debian.org/
signature.asc
Description: Digital signature
Bug#317082: Not just a dpkg bug
On Fri, Jan 20, 2006 at 03:36:36AM -0800, Steve Langasek wrote: On Fri, Jan 20, 2006 at 11:57:28AM +0300, Nikita V. Youshchenko wrote: 2) we could try to use the ldconfig cache to make to work of ldd for ourself. Questions: - Is this really an advantage? Or has the cache the same problems ldd has? Hmm. In theory, ldconfig shouldn't require the ability to execute 64-bit binaries in order to build a cache of their paths. The only thing I don't know is how 64-bit vs. 32-bit libs are cached? It would after all have to be possible to distinguish between them in order to use the cache for this. It is possible: [EMAIL PROTECTED]:~$ /sbin/ldconfig -p | grep libbz2.so.1.0 libbz2.so.1.0 (libc6,x86-64) = /usr/lib64/libbz2.so.1.0 libbz2.so.1.0 (libc6) = /lib/libbz2.so.1.0 Might still be a mess to parse all those architecture specific names, though... Gruesse, -- Frank Lichtenheld [EMAIL PROTECTED] www: http://www.djpig.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
On Thu, Jan 19, 2006 at 12:14:35AM +0100, Frank Lichtenheld wrote: If we don't use the path information from ldd there are several ways to go: 1) use dpkg --search but only with the library name from objdump, not with the full path. Questions: - Are there cases where the library name from objdump isn't actually the filename of the library? - How do we decide wether a found file is really a usable library? (parse /etc/ld.so.conf?) Note that this method is used in the patch for #285857 which I'm currently evaluating for inclusion. So it seems the answer to question 1 is no (or else the patch in #285857 is useless) and the answer to question 2 is most likely it is listed in the shlibs file of the corresponding package which might have been the sense of Nikita's explanation of buggy packages that don't list their libs in their shlibs will need to be fixed that I just don't got. But now I definetly need some sleep before I can wind my head around more of that dpkg-shlibdeps code... Gruesse, -- Frank Lichtenheld [EMAIL PROTECTED] www: http://www.djpig.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
Frank Lichtenheld [EMAIL PROTECTED] writes: This isn't quite true I think. The current dpkg-shlibdeps code works like this: 1) use ldd binary to find the paths to the linked libraries 2) use objdump -p binary to actually check which of this libraries are listed as NEEDED (Are there cases where ldd lists libraries that are not NEEDED?) Yes. ldd will list all shared libraries pulled in by a binary, regardless of whether they're NEEDED by the binary itself or just NEEDED by one of the shared libraries it uses. For example: windlord:~ ldd /usr/bin/remctl linux-gate.so.1 = (0xe000) libgssapi_krb5.so.2 = /usr/lib/libgssapi_krb5.so.2 (0xb7f8e000) libc.so.6 = /lib/tls/libc.so.6 (0xb7e57000) libkrb5.so.3 = /usr/lib/libkrb5.so.3 (0xb7ddf000) libk5crypto.so.3 = /usr/lib/libk5crypto.so.3 (0xb7dbb000) libcom_err.so.2 = /lib/libcom_err.so.2 (0xb7db8000) libkrb5support.so.0 = /usr/lib/libkrb5support.so.0 (0xb7db3000) libresolv.so.2 = /lib/tls/libresolv.so.2 (0xb7da) /lib/ld-linux.so.2 (0xb7fb3000) windlord:~ objdump -p /usr/bin/remctl | grep NEEDED NEEDED libgssapi_krb5.so.2 NEEDED libc.so.6 windlord:~ objdump -p /usr/lib/libgssapi_krb5.so.2 | grep NEEDED NEEDED libkrb5.so.3 NEEDED libk5crypto.so.3 NEEDED libcom_err.so.2 NEEDED libkrb5support.so.0 NEEDED libresolv.so.2 NEEDED libc.so.6 -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
At Wed, 17 Aug 2005 22:05:42 +0200, Andreas Jochens wrote: I guess you will generally have many more issues than this one when you try to build 64-bit packages on a 32-bit buildd (e.g. compiling and running 64-bit programs from configure scripts, running 'make check' or 'make test' targets, using binaries which have been built by the package itself etc.) In the end it will be much easier to require a 64-bit machine to be used to build 32/64-bit biarch packages instead of trying to circumvent all these issues. Yes, that's one solution. However, instead, I would like to propose supporting 32bit and 64bit binaries as separated architectures. For example, think about ppc32 and ppc64. My proposal is to have both Debian/dists/sid/main/binary-powerpc and Debian/dists/sid/main/binary-ppc64. It's similar to the different architecture between i386 and amd64 - but ppc32 and ppc64 are not distinguished so much. If a package has (ex:) Features: biarch in debian/control (like Tollef's proposal), ppc64 buildd picks up to build this package. If it does not biarch capable package, it should not be built. It reduces much disk spaces on mirror servers, and the 90% of non-biarch-needed (binaries and libraries) packages do not need to consider about biarch problems. To install both 32/64 bit packages, we need to consider about the file duplication in /usr/share and /usr/bin - but fortunatelly the proposal of Scott's dpkg modification, FILTERS and CLASSES, probably fix this kind of problems. It's very simple way and we don't modify a lot of packages. If you guys like this idea, I'll write the proposal to debian-devel lists. Or is this idea already proposed? Regards, -- gotom -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
At Thu, 18 Aug 2005 10:24:14 +0200, Andreas Jochens wrote: There is already an inofficial buildd for the ppc64 architecture running for 'unstable'. The respective ppc64 package archive is located at deb http://debian-ppc64.alioth.debian.org/gcc4 unstable main and almost 95% of all source packages from 'unstable' have already been compiled for the ppc64 architecture. Indeed - if we put this thought forward, we'll reach to the whole native ppc64 support of all packages. BTW I think supporting all binary packages as 64bit native does not have much sense - most packages do not use such large memory space, and even they run slower than 32bit binaries. This is because I would like to introduce (ex:) Features flags. A small number of packages still need some minor patches to make this work, notably 'glibc' (see #301438 - the patch in that bug is outdated, but I could supply an updated patch for this). I didn't put your patch because the native ppc64 support was not adopted, from the discussion that was done in debian-powerpc lists. If your patch does not have any impact, please send your latest version to me. With this setup, the ppc64/powerpc situation would become very similar to the amd64/i386 situation and the same solution for multiarch/biarch support could be used for both cases. Yes. The current partial support for biarch imposes various modifications on package maintainers. Such method will not be spereaded - debian is the large voluntary organization, reducing cost of package maintainer's load is the primal principle. Regards, -- gotom -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
On 05-Aug-17 12:52, Steve Langasek wrote: No, that doesn't solve the problem. How are you supposed to invoke a 64-bit linker for a bi-arch build being done on a 32-bit buildd? I guess you will generally have many more issues than this one when you try to build 64-bit packages on a 32-bit buildd (e.g. compiling and running 64-bit programs from configure scripts, running 'make check' or 'make test' targets, using binaries which have been built by the package itself etc.) In the end it will be much easier to require a 64-bit machine to be used to build 32/64-bit biarch packages instead of trying to circumvent all these issues. Regards Andreas Jochens -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#317082: Not just a dpkg bug
At Wed, 17 Aug 2005 17:00:23 +0100, Scott James Remnant wrote: I don't think this is just a dpkg-dev bug, these bi-arch systems need to provide ldd or an equivalent that can read either form of shared library that it would support. objdump isn't a solution either, while it sometimes can read the other shared library, it doesn't provide the linker search patch which is of critical importance to get this stuff right. So I'm including libc6-dev (for want of a better package) in this bug, as we first need ldd on these bi-arch systems (or something similar) for dpkg-shlibdeps to use before we can fix that! ldd is very hard to handle 64bit binaries on 32bit systems without breaking glibc, kernel and various technical beautiful concepts. Please look at /usr/bin/ldd. It just passes LD_* variable to dynamic linker. When dynamic linker is invoked from kernel elf module, it parses LD_* environment variable and put the library path message to stdout. So ldd is just wrapper to take notice to dynamic linker. The actual dynamic linker path is specified in each binaries. For example, do strings /bin/ls |grep ld on both i386 and amd64. You'll find 64bit binaries designate 64bit dynamic loader that is placed in /lib64. And, dynamic loader itself is built with 64bit code - so we cannot invoke even dynamic loader. OK, now back to the discussion. The above means we cannot use ldd to search pathes. OTOH, Ryan suggested me to look at dpkg-shlibdeps in dpkg-cross package as I described. It's something adhoc fix, but I think we have no other way to support biarch nicely. Regards, -- gotom -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
