Bug#320698: Debian-specific binary deps patch
* Ralf Wildenhues wrote on Wed, Nov 30, 2005 at 11:07:47PM CET: Hmm. Sounds like quite a bit of work. And pretty dangerous, too. We'd need lots of good tests. One thing to always keep in mind (for upstream GNU Libtool) is that this change of operation must not change anything on other systems. Just to give an idea about what good test coverage would be: Let P be a program, arrows indicate direct linkage. Make dead sure we find and use the correct (un)installed libraries, for both the uninstalled and the installed program: - P -- libAor P -- libA -- libBor P -- libA -- libB `- libB - both uninstalled, A un-, B installed, both installed - A and B (to-be-)installed in same libdir, or different one - A and B (to-be-)installed in a default-searched dir (for either the link editor or the runtime linker) - there exist previously installed versions of one or both of the libraries - either link-time or runtime incompatible - in either the same libdir as the new ones will be, or other ones which are involved, or the default- searched dirs (for either the link editor or the runtime linker) [ I have no idea how to put the latter in a testsuite run by non-root ] - maybe combine with (low-priority): -static, -all-static, or configured --disable-static. If all these combinations work out fine, then there is still a whole class of issues yet to be solved: dlopening with library dependencies. So, now let dotted arrows indicate lt_dlopen()ed modules, M a module, D another library (libC is on some systems the C++ standard library). Something like this set of combination, or a bit less: - P .. M P -- libA .. M P .. M-- libB -- libD `- libB P .. M-- libB -- libD `- libB `- libD P -- libA .. M-- libB - libD P -- libA .. M-- libB `- libB P -- libA .. M-- libB - libD `- libD P -- libA .. M-- libB - libD `- libB `- libD P -- libA .. M-- libB `- libB P -- libA .. M-- libB `- libB `- libB P -- libA .. M-- libB - libD `- libB P -- libA .. M-- libB - libD `- libB `- libD P -- libA .. M-- libB - libD `- libB `- libB `- libD - repeat above list of combinations, expanded by M and libD, but do not try *all* possibilites (around 200M if I'm not mistaken) -- rather find a good representative subset (maybe a couple hundred). - It may even be useful to consider the case where installed modules depend on uninstalled libraries. I don't think this is particularly elegant, and it's clearly not portable, but disallowing this is likely to hurt some of the more elaborate packaging schemes (say, browser with lots of third-party plugins); rather, the lower-level libs (in this case libB) should be packaged separately from P and libA, so that M may be built before them but after libB. - I'm not certain we also need to take into account modules loading other modules, or multiple modules depending on different lower-level libraries. - DESTDIR installs or cross-compilation have not been taken into account *just* yet. One reason I'm writing this is that it may be useful to have this set of tests not only in the link_all_deplibs=no Debian libtool, but also in upstream GNU Libtool. I also think that an eventual merge upstream will be much easier, say, if the link_all_deplibs choice were not one done at configure time based on $host only, but maybe based on a configuration switch, or even a libtool --mode=link time switch (or an environment variable containing that switch). The best choice here is not yet clear to me, as issues like #297726 and #291641 should be fully understood and fixed first. This would likely require Debian to change their build machinery a bit, but I think it has ways to centralize this in a single or few places (and I think it can be done in a way that it won't need a flag day). Anybody with too much time on their hands is cordially invited to write tests for this, preferably for CVS HEAD with help of the Autotest test suite (stresstest.at is a good example of how to tackle this). (I may be abusing the Debian BTS as a TODO list; sorry about that.) Cheers, and thank you for reading this far, Ralf -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#320698: Debian-specific binary deps patch
Hi there,
Sorry for the long latency.
* Philip Martin wrote on Fri, Oct 07, 2005 at 01:38:48AM CEST:
Kurt Roeckx [EMAIL PROTECTED] writes:
After reading bug #291641, I'd have to agree this is a problem
with the linker.
It's not quite so clear to me. Here's the script I've been using to
duplicate the problem:
Hehe, nice script. Take it, do a
sed -e 's/libtool/$libtool/'
on it, and add
: ${libtool=libtool}
somewhere at the top.
(It's also very instructive to look at the output of objdump -p
by the way, in addition to `libtool --mode=execute ldd' which gives a
skewed view through LD_LIBRARY_PATH; so you might want to add that).
Now let's pretend we're not installing to $PWD/zz/lib, but
to /usr/lib. Just for a minute, ok?
Well, to simulate what libtool would do iff the installation path
is in both link editor's and the runtime linker's default search
path (and without actually messing with our installation), we need to
fake libtool knowledge:
sed libtool libtool-fake \
-e s#^\(sys_lib_search_path_spec=.*\)\\$#\1 $PWD/zz/lib\# \
-e s#^\(sys_lib_dlsearch_path_spec=.*\)\\$#\1 $PWD/zz/lib\#
Now try
libtool=./libtool-fake ./run
(given that your script is named 'run').
It will pass :)
In the following analysis, let 'libdir' be '$PWD/zz/lib'.
- two differences: libj won't be passed the -rpath $libdir,
hence won't have DT_RPATH $libdir. It will still have the DT_RPATH
entry for the uninstalled libi, correctly.
- linking of k (precise: zz/k/.libs/k) won't have *any* -rpath argument.
And it won't be passed '-li' on the command line.
This is precisely the situation in which the GNU ld mechanism
for walking DT_NEEDED + DT_RPATH combinations of indirect dependencies
(libraries pulled in by other libraries) works.
Note the --rpath to the install directory, that goes straight through
to the linker, and it appears that the command line option overrides
RPATH in the library.
Correct.
Is it a linker bug for the command line option to have priority?
I don't know.
It's documented that way.
Perhaps libtool should not be passing that --rpath during the link?
Well. In above scenario, it would fix the problem.
In general it's harder to avoid. Imagine we have a dependency on
another library, from another package, already installed. If it was
installed in a path which the linker won't find by itself, the -rpath
will be necessary for the final _installed_ executable, so that it can
find its libraries on its own.
What we could do now, to save the day, would be to disallow fast_install.
Relinking upon installation would have to be required then. This would
enable, on Linux, to remove the -rpath to $libdir from the uninstalled.
I believe.
Hmm. Sounds like quite a bit of work. And pretty dangerous, too. We'd
need lots of good tests. One thing to always keep in mind (for upstream
GNU Libtool) is that this change of operation must not change anything
on other systems. A good way to get many people annoyed at you.
Another way of solving the problem in libtool could be to do the grunt
work of traversing all indirect libraries and adding rpaths for them,
too. Ugly. Would likely need '.la' files for all of them so it could
be done portably. OTOH, it may be the only way we could get decent
indirect dependencies support on more than just GNU ld.
See, the way I see it, this whole automatic DT_NEEDED + DT_RPATH walking
by the link editor works best when all installed paths are found by
default, and thus the final program won't need any overrides, which then
make the linker turn off the weaker search methods.
Another thing not to underestimate is library authors that forget to
list all their dependent libraries they use directly. Finding a bug
related to this is bound to be horrific.
Perhaps libtool should be passing the build dir instead? (That might
mean that libtool has to relink the executable at install time to get
the correct final rpath.) Does that make it a libtool bug?
Hmm. I'd call it a ld limitation for which libtool does not yet have an
adequate workaround. But that may just be me. :)
Also, it may be helpful to note that there has been quite a bit of
discussion on the libtool list, on and off, about decent support for
indirect dependencies. Not many patches yet, though. :-/
Cheers,
Ralf
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#320698: Debian-specific binary deps patch
On Fri, Oct 07, 2005 at 12:38:48AM +0100, Philip Martin wrote: It's the second build that fails, when this libtool command: libtool --mode=linkgcc zz/k/k.lo zz/j/libj.la -o zz/k/k invokes this gcc command: gcc zz/k/.libs/k.o -o zz/k/.libs/k zz/j/.libs/libj.so -Wl,--rpath -Wl,/home/pm/sw/zz/lib Note the --rpath to the install directory, that goes straight through to the linker, and it appears that the command line option overrides RPATH in the library. Is it a linker bug for the command line option to have priority? I don't know. From the documentation: -rpath dir Add a directory to the runtime library search path. This is used when linking an ELF executable with shared objects. All -rpath arguments are concatenated and passed to the runtime linker, which uses them to locate shared objects at runtime. The -rpath option is also used when locating shared objects which are needed by shared objects explicitly included in the link; see the description of the -rpath-link option. And -rpath-link has this nice order: 1. Any directories specified by -rpath-link options. 2. Any directories specified by -rpath options. [...] [...] 5. For a native linker, the contents of the environment variable LD_LIBRARY_PATH. 6. For a native ELF linker, the directories in DT_RUNPATH or DT_RPATH of a shared library are searched for shared libraries needed by it. The DT_RPATH entries are ignored if DT_RUNPATH entries exist. 7. The default directories, normally /lib and /usr/lib. 8. For a native linker on an ELF system, if the file /etc/ld.so.conf exists, the list of directories found in that file. So it seems that the -rpath option to the install dir might be the problem. Or maybe an LD_LIBRARY_PATH? Perhaps libtool should not be passing that --rpath during the link? Perhaps libtool should be passing the build dir instead? (That might mean that libtool has to relink the executable at install time to get the correct final rpath.) Does that make it a libtool bug? You might be right about this, and I will take a look at it. Any idea why libtool passes the install path in the first place? How should it know where it's going to get installed at this point? PS: This might actually be the wrong bug in which we're discussing this. Kurt -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#320698: Debian-specific binary deps patch
Kurt Roeckx [EMAIL PROTECTED] writes: Any idea why libtool passes the install path in the first place? How should it know where it's going to get installed at this point? That's the way libtool works. When a shared library is linked the libtool command must include an rpath to the install directory, the directory path gets stored in the .la file, so when libtool links another library or application the path is available. PS: This might actually be the wrong bug in which we're discussing this. Or perhaps the bugs should be merged? -- Philip Martin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#320698: Debian-specific binary deps patch
On Thu, Oct 06, 2005 at 01:24:03AM +0100, Philip Martin wrote: It seems that Debian's libtool has a patch which will reduce dependencies in binaries it produces. One of the knock-on effects of this patch is bug 291641 where ld resolves symbols using installed libraries instead of libraries in the build directory. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=291641 Subversion 1.2 has a Build-Conflicts with Subversion 1.1 to avoid this problem. After reading bug #291641, I'd have to agree this is a problem with the linker. Has anybody took that to upstream binutils yet? Kurt -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#320698: Debian-specific binary deps patch
Kurt Roeckx [EMAIL PROTECTED] writes:
After reading bug #291641, I'd have to agree this is a problem
with the linker.
It's not quite so clear to me. Here's the script I've been using to
duplicate the problem:
#!/bin/bash
rm -rf zz
mkdir zz zz/i zz/j zz/k zz/lib zz/bin
R=-rpath $PWD/zz/lib
echo int this_is_i() { return 0; }zz/i/i.c
echo int this_is_j() { return this_is_i(); } zz/j/j.c
echo int main() { return this_is_j(); } zz/k/k.c
libtool --mode=compile gcc -c zz/i/i.c -o zz/i/i.lo
libtool --mode=compile gcc -c zz/j/j.c -o zz/j/j.lo
libtool --mode=compile gcc -c zz/k/k.c -o zz/k/k.lo
libtool --mode=linkgcc zz/i/i.lo -o zz/i/libi.la $R
libtool --mode=linkgcc zz/j/j.lo zz/i/libi.la -o zz/j/libj.la $R
libtool --mode=linkgcc zz/k/k.lo zz/j/libj.la -o zz/k/k
echo == first build complete
libtool --mode=install cp zz/i/libi.la $PWD/zz/lib /dev/null
libtool --mode=install cp zz/j/libj.la $PWD/zz/lib /dev/null
libtool --mode=install cp zz/k/k $PWD/zz/bin /dev/null
echo == install complete
echo ldd zz/k/k
libtool --mode=execute ldd zz/k/k
echo ldd zz/bin/k
libtool --mode=execute ldd zz/bin/k
echo == first build checked
echo int this_is_xi() { return 0; }zz/i/i.c
echo int this_is_j() { return this_is_xi(); } zz/j/j.c
libtool --mode=compile gcc -c zz/i/i.c -o zz/i/i.lo
libtool --mode=compile gcc -c zz/j/j.c -o zz/j/j.lo
libtool --mode=linkgcc zz/i/i.lo -o zz/i/libi.la $R
libtool --mode=linkgcc zz/j/j.lo zz/i/libi.la -o zz/j/libj.la $R
libtool --mode=linkgcc zz/k/k.lo zz/j/libj.la -o zz/k/k
echo == second build failed
rm -rf zz/lib/* zz/bin/*
echo == uninstall complete
libtool --mode=linkgcc zz/k/k.lo zz/j/libj.la -o zz/k/k $R
echo == third build complete
echo ldd zz/k/k
libtool --mode=execute ldd zz/k/k
echo == third build checked
It's the second build that fails, when this libtool command:
libtool --mode=linkgcc zz/k/k.lo zz/j/libj.la -o zz/k/k
invokes this gcc command:
gcc zz/k/.libs/k.o -o zz/k/.libs/k zz/j/.libs/libj.so -Wl,--rpath
-Wl,/home/pm/sw/zz/lib
Note the --rpath to the install directory, that goes straight through
to the linker, and it appears that the command line option overrides
RPATH in the library. Is it a linker bug for the command line option
to have priority? I don't know.
Perhaps libtool should not be passing that --rpath during the link?
Perhaps libtool should be passing the build dir instead? (That might
mean that libtool has to relink the executable at install time to get
the correct final rpath.) Does that make it a libtool bug?
--
Philip Martin
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#320698: Debian-specific binary deps patch
It seems that Debian's libtool has a patch which will reduce dependencies in binaries it produces. One of the knock-on effects of this patch is bug 291641 where ld resolves symbols using installed libraries instead of libraries in the build directory. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=291641 Subversion 1.2 has a Build-Conflicts with Subversion 1.1 to avoid this problem. -- Philip Martin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#320698: Debian-specific binary deps patch
Package: libtool
Version: 1.5.6-6
Severity: important
Hi,
It seems that Debian's libtool has a patch which will reduce
dependencies in binaries it produces.
It would be nice to bring this patch upstream, as the upstream
maintainers not running Debian are libtoolizing without this specific
change, and require the Debian maintainer to relibtoolize to reduce
dependencies, which is a lot of work.
Steve Langasek has brought this issue upstream at:
http://lists.gnu.org/archive/html/libtool/2005-07/threads.html#00174
Bye,
-- System Information:
Debian Release: testing/unstable
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: i386 (i686)
Shell: /bin/sh linked to /bin/bash
Kernel: Linux 2.6.12-1-686
Locale: [EMAIL PROTECTED], [EMAIL PROTECTED] (charmap=ISO-8859-15)
Versions of packages libtool depends on:
ii autotools-dev 20050422.1 Update infrastructure for config.{
ii cpp 4:4.0.0-2The GNU C preprocessor (cpp)
ii file4.12-1 Determines file type using magic
ii gcc [c-compiler]4:4.0.0-2The GNU C compiler
ii gcc-3.3 [c-compiler]1:3.3.6-7The GNU C compiler
ii gcc-3.4 [c-compiler]3.4.4-5 The GNU C compiler
ii gcc-4.0 [c-compiler]4.0.1-3 The GNU C compiler
ii libc6-dev [libc-dev]2.3.2.ds1-22 GNU C Library: Development Librari
Versions of packages libtool recommends:
ii libltdl3-dev 1.5.6-6A system independent dlopen wrappe
-- no debconf information
--
Loïc Minier [EMAIL PROTECTED]
Come, your destiny awaits!
