Re: transitive shared library dependencies and installation
Bob Friesenhahn writes: > On Sun, 5 Jan 2020, wf...@niif.hu wrote: > >> On the other hand, this overlinks the final binary: >> >> $ objdump -p .libs/translib | fgrep NEEDED >> NEEDED liba.so >> NEEDED libb.so >> NEEDED libc.so.6 >> >> libb.so is unneeded here (but is present in the installed program as >> well). Coincidentally, the most prominent search result >> https://wiki.mageia.org/en/Overlinking_issues_in_packaging mentions that >> "this is fixed using a patch from Debian" for libtool. >> >> What's your position on this? Is overlinking a problem or not? (It >> causes problems for distributions.) Should everybody use --as-needed >> globally to combat it? Something else entirely? > > This has been the most common complaint (in the GNU Linux world) I > have heard about libtool. There is a choice of working reliably and > portably (with possible over-linking) or relying on implicit library > dependencies (which are definitely not portable), or failing as you > have encountered. Is the presence of over-linking a portability aspect? I mean, couldn't libtool overlink only where it's unavoidable, and link precisely on systems which allow this (via -rpath-link or other means)? > A similar complaint is that libtool uses information in installed > ".la" files in order to link with all libraries that the > program/library is dependent on. This is the exact same issue as above, isn't it? > Due to this, GNU Linux distributions often patch out this capability, > and they don't distribute ".la" files. Yes, I think they are (meant to be) replaced by pkg-config, and distros have little interest in static linking or ltdl anyway (with a few exceptions, like ImageMagick). And ELF shared objects carry their dependency information internally. > Unfortunately, --as-needed may not be 100% reliable since it only > reliably detects direct dependence on library symbols, and not > "transitive" dependence. Maybe I misunderstand you, but that's its very point in my opinion. Anyway, the most trouble with it seems to originate from libtool reordering the linker flags when creating shared libraries, making --as-needed ineffective by moving it to the end of the command line. At least this used to be the case, see https://stackoverflow.com/questions/6852414/how-do-i-link-a-shared-library-with-as-needed-with-automake (This thread is too broad already, let's discuss that in a new one if you've got the bandwidth.) -- Thanks, Feri
Re: transitive shared library dependencies and installation
Roumen Petrov writes: > wf...@niif.hu wrote: > >>> So the right question is did reporter test with FSF version? >> >> I'm the reporter, and I didn't test any other version, as I wasn't >> even sure whether my example was correct and was supposed to work. > > You sample is correct . One minor nit is AM_PROG_AR - it is not > required for posted example. If you project requires use of this macro > you should request automake 1.11.2 as minimum. Hi Roumen, Thanks for the confirmation. Initially I didn't include AM_PROG_AR in the sample, but then autoreconf emitted much noise like /usr/share/automake-1.16/am/ltlibrary.am: warning: 'a/liba.la': linking libtool libraries using a non-POSIX /usr/share/automake-1.16/am/ltlibrary.am: archiver requires 'AM_PROG_AR' in 'configure.ac' Makefile.am:4: while processing Libtool library 'a/liba.la' (and the same for libb), and eventually I gave in. > Result on posted test case - pass: > $ ./translib ; echo $? > 12 Thanks for posting your results. Meanwhile I also tested with FSF libtool 2.4.6, and the final "make" indeed succeeds with it, unlike with Debian's libtool. The decisive difference is that with the FSF version the full build path to libb.so is explicitly added to the final link command, thus the linker needn't search for it by itself. On the other hand, this overlinks the final binary: $ objdump -p .libs/translib | fgrep NEEDED NEEDED liba.so NEEDED libb.so NEEDED libc.so.6 libb.so is unneeded here (but is present in the installed program as well). Coincidentally, the most prominent search result https://wiki.mageia.org/en/Overlinking_issues_in_packaging mentions that "this is fixed using a patch from Debian" for libtool. What's your position on this? Is overlinking a problem or not? (It causes problems for distributions.) Should everybody use --as-needed globally to combat it? Something else entirely? -- Thanks, Feri
Re: transitive shared library dependencies and installation
Bob Friesenhahn writes: > On Sat, 4 Jan 2020, wf...@niif.hu wrote: > >>> Above dependency explain all - lib_LTLIBRARIES is project (sample) >>> specific. Project should ensure order. >> >> Fair enogh, but my point (which I didn't really emphasize) is that >> installation could work in any order, if relinking didn't depend on >> already installed libraries. It's even necessary for mutually dependent >> libraries. > > That sounds like a great idea. However, there is a problem that > linking on some systems does depend on already installed libraries (or > will end up using them) and so the libraries need to be installed and > linked in a particular order. Do you happen to know which systems these are? Where are these constraints documented? Do the libtool maintainers (er, anybody) have access to such systems? I think these are prerequisites for effective libtool maintenance (and I'm probably just scratching the surface). -- Regards, Feri
Re: transitive shared library dependencies and installation
Roumen Petrov writes: > wf...@niif.hu wrote: > >> Roumen Petrov writes: >> >>> wf...@niif.hu wrote: >>> I'm experimenting with the attached skeleton project on a Debian buster system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9): > > Debian is key word. > Libtool is patched, not FSF. [...] > With and without new dtags result will be the same. Very well possible, it's just one difference I happened to know about. > It is long history It starts with 1* (1.5) libtool . Libtool 1.5 has > some issues with multiple dependent libraries (more then two). > From debian was proposed a patch related to library > dependencies. Unfortunately patch break existing regression test. From > debian never was proposed version that pass regression test. > > Libtool 2.0 fixes his issues related to multiple libraries. On the > same Debian did not stop to contribute patch that breaks libtool. > > As result when I decide to build something from source always to > updated sources to FSF version. > > So the right question is did reporter test with FSF version? I'm the reporter, and I didn't test any other version, as I wasn't even sure whether my example was correct and was supposed to work. Could you please provide some keywords to search for so that I can dig up the details of the above story? At the moment Debian carries 21 patches for libtool, if I could show that one of them breaks a valid use case, that would constitute a strong reason for dropping it. -- Thanks, Feri
Re: transitive shared library dependencies and installation
Roumen Petrov writes: > wf...@niif.hu wrote: > >> I'm experimenting with the attached skeleton project on a Debian buster >> system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9): > [SNIP] >> /usr/bin/ld: cannot find -lb >> collect2: error: ld returned 1 exit status >> libtool: error: error: relink 'a/liba.la' with the above command before >> installing it >> make[1]: *** [Makefile:446: install-libLTLIBRARIES] Error 1 >> make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib' >> make: *** [Makefile:798: install-am] Error 2 > > Libtool does not change "order" :(. Automake just uses project rules. > > > Let view Makefile.in > ... > install-exec-am: install-binPROGRAMS install-libLTLIBRARIES > ... > install-binPROGRAMS: install-libLTLIBRARIES > ... > install-libLTLIBRARIES: $(lib_LTLIBRARIES) > ... > Above dependency explain all - lib_LTLIBRARIES is project (sample) > specific. Project should ensure order. Fair enogh, but my point (which I didn't really emphasize) is that installation could work in any order, if relinking didn't depend on already installed libraries. It's even necessary for mutually dependent libraries. >> and use it from liba, linking the final binary fails: >> >> $ make >> [...] >> /bin/bash ./libtool --tag=CC --mode=link gcc -g -O2 -avoid-version -o >> translib translib.o a/liba.la >> libtool: link: gcc -g -O2 -o .libs/translib translib.o a/.libs/liba.so >> -Wl,-rpath -Wl,/tmp/translib/lib >> /usr/bin/ld: a/.libs/liba.so: undefined reference to `b2' >> >> As I understand it, the -rpath linker option on the above command makes >> a/.libs/liba.so use the already installed (old) version of libb, which >> lacks the b2 symbol. What's the solution here? Why isn't that -rpath >> option "delayed" until the relinking phase? > > Not reproducible with FSF release. Could you please share your link command? Does you system use the new dynamic tags (DT_RUNPATH instead of DT_RPATH)? -- Thanks, Feri
Re: transitive shared library dependencies and installation
Bob Friesenhahn writes: > On Thu, 2 Jan 2020, wf...@niif.hu wrote: > True, but man ld states in the -rpath-link option description that the directories specified by -rpath options are used with a very high priority for locating required shared libraries. >>> >>> This is interesting but I am not seeing any use of -rpath-link in >>> libtool. >> >> Neither do I, but I think it would be a possible way out of this >> situation. I wonder what the reason could be for libtool not using it. > > If Libtool were to depend on non-portable features, [...] then it > could not longer be described as a portability tool. In my understanding, Libtool is a portability shim, providing a regular interface for developers across systems with wildly varying features. It already uses non-portable features, just different ones on different architectures. I don't say it should use -rpath-link generally, I only suggested that it might be an efficient solution on systems supporting it. But I expect all systems supporting shared objects to allow using and installing them some way, irrespective of their interdependencies. Am I overly naive? > As far as I am aware, libtool is currently "between maintainers" and > needs fresh volunteers to become a libtool maintainer. Fresh volunteers with arcane knowledge... quite a tall order! :) Back to my problem: first I'd like to establish whether the behavior I see is a bug, and if it is, is it in Libtool or in my code or somewhere else. And then find a solution or at least a workaround. Opinions? -- Thanks, Feri
Re: transitive shared library dependencies and installation
Bob Friesenhahn writes: > On Thu, 2 Jan 2020, wf...@niif.hu wrote: >> >>> In this case libtool is a slave of Automake and Automake is >>> controlling the build and installation order. This means that it >>> should be expected behavior. Installation is serialized and thus >>> order matters. >> >> But in case of a dependency cycle there's no correct order if libtool >> insists on using the installation directory for the -L option for >> relinking. And the installation directory may contain an incompatible >> version of the library anyway, just like it happens below. > > Indeed, sometimes it is necessary to re-organize libraries and code in > order to be successful given serialized linking. I think libtool could support arbitrary shared library dependencies by using the -rpath-link option as necessary. and use it from liba, linking the final binary fails: $ make [...] /bin/bash ./libtool --tag=CC --mode=link gcc -g -O2 -avoid-version -o translib translib.o a/liba.la libtool: link: gcc -g -O2 -o .libs/translib translib.o a/.libs/liba.so -Wl,-rpath -Wl,/tmp/translib/lib /usr/bin/ld: a/.libs/liba.so: undefined reference to `b2' As I understand it, the -rpath linker option on the above command makes a/.libs/liba.so use the already installed (old) version of libb, which lacks the b2 symbol. What's the solution here? Why isn't that -rpath option "delayed" until the relinking phase? >>> >>> The -rpath linker option did not cause the library to be used. >> >> The above gcc linking command is successful if I omit the -rpath option. >> (The built-in RUNPATH of liba.so contains the build directory first. >> Just for the record: Debian's ld defaults to --enable-new-dtags.) > > I am not sure exactly what --enable-new-dtags means, but Ubuntu 18.04 > LTS's manual page says that this is not its default. This may > indicate changing behavior. I suspect it's just outdated documentation. What --enable-new-dtags does is adding DT_RUNPATH instead of DT_RPATH to ELF objects. See which one you find after building my example. >>> The '-r' in -rpath stands for 'run' and thus it is a path stored in a >>> built shared library or executable which tells it where to search for >>> other libraries when th program is executed. >> >> True, but man ld states in the -rpath-link option description that the >> directories specified by -rpath options are used with a very high >> priority for locating required shared libraries. > > This is interesting but I am not seeing any use of -rpath-link in > libtool. Neither do I, but I think it would be a possible way out of this situation. I wonder what the reason could be for libtool not using it. >>> I am not seeing 'libb' mentioned on the link line and that may be the >>> cause of the problem you are seeing. >> >> Sure enough, adding libb.la explicitly to the link command works around >> the issue, but since the binary does not use libb directly, it doesn't >> sound like a good solution to me, because it leads to overlinking. > > Libtool can only know what has been passed to it via the command line > and via the prior information it stored in the .la files (initially > found due to the command line). Yes, and liba indeed has the build directory libb.la in its dependency_libs, so the information is there and could be used with -rpath-link, unless I'm missing something important. > I have always found it to be good practice to include all dependency > libraries in an Automake 'LIBADD' statement so that all dependency > libraries are passed to libtool. But that causes overlinking, which is unpopular. And I'm not sure --as-needed could help here. > If the .la file passed to libtool mentions the dependencies, then > libtool is supposed to do the right thing. I think it does, see above. Libtool still fails to link the executable if an incompatible library is already installed (and you use a custom --prefix). > At this point I am wondering why I am the only person on this list who > has piped up with some sort of a response. :-( It's rather hard to get support for libtool, but I sorely need some, because this issue hurts a real software project. Since this is the first time I seriously deal with libtool, I certainly miss most of the picture, so I sincerely appreciate your "piping up". Let's hope others will join with time with their insights. -- Thanks, Feri
Re: transitive shared library dependencies and installation
Bob Friesenhahn writes: > On Wed, 1 Jan 2020, wf...@niif.hu wrote: > >> make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib' >> make: *** [Makefile:798: install-am] Error 2 >> >> No cyclic dependencies here, so this can be worked around by >> >> -lib_LTLIBRARIES = a/liba.la b/libb.la >> +lib_LTLIBRARIES = b/libb.la a/liba.la >> >> in this case; is this expected (and documented) behavior? > > In this case libtool is a slave of Automake and Automake is > controlling the build and installation order. This means that it > should be expected behavior. Installation is serialized and thus > order matters. But in case of a dependency cycle there's no correct order if libtool insists on using the installation directory for the -L option for relinking. And the installation directory may contain an incompatible version of the library anyway, just like it happens below. >> and use it from liba, linking the final binary fails: >> >> $ make >> [...] >> /bin/bash ./libtool --tag=CC --mode=link gcc -g -O2 -avoid-version -o >> translib translib.o a/liba.la >> libtool: link: gcc -g -O2 -o .libs/translib translib.o a/.libs/liba.so >> -Wl,-rpath -Wl,/tmp/translib/lib >> /usr/bin/ld: a/.libs/liba.so: undefined reference to `b2' >> >> As I understand it, the -rpath linker option on the above command makes >> a/.libs/liba.so use the already installed (old) version of libb, which >> lacks the b2 symbol. What's the solution here? Why isn't that -rpath >> option "delayed" until the relinking phase? > > The -rpath linker option did not cause the library to be used. The above gcc linking command is successful if I omit the -rpath option. (The built-in RUNPATH of liba.so contains the build directory first. Just for the record: Debian's ld defaults to --enable-new-dtags.) > The '-r' in -rpath stands for 'run' and thus it is a path stored in a > built shared library or executable which tells it where to search for > other libraries when th program is executed. True, but man ld states in the -rpath-link option description that the directories specified by -rpath options are used with a very high priority for locating required shared libraries. > Perhaps the controlling parameter is hardcode_into_libs. If your > libtool comes from an OS distribution rather than from a FSF/GNU > tarball, then it is possible that its behavior may have been modified. I don't know what to expect, here's what I can see: $ ./libtool --config | fgrep hardcode_into_libs hardcode_into_libs=yes > I am not seeing 'libb' mentioned on the link line and that may be the > cause of the problem you are seeing. Sure enough, adding libb.la explicitly to the link command works around the issue, but since the binary does not use libb directly, it doesn't sound like a good solution to me, because it leads to overlinking. -- Thanks, Feri
transitive shared library dependencies and installation
Hi,
I'm experimenting with the attached skeleton project on a Debian buster
system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9):
$ autoreconf -f -i
$ ./configure --prefix=/tmp/translib
$ make
$ ./translib; echo $?
6
$ make install
make[1]: Entering directory '/home/wferi/ha/pacemaker/translib'
/bin/mkdir -p '/tmp/translib/lib'
/bin/bash ./libtool --mode=install /usr/bin/install -c a/liba.la b/libb.la
'/tmp/translib/lib'
libtool: warning: relinking 'a/liba.la'
libtool: install: (cd /home/wferi/ha/pacemaker/translib; /bin/bash
"/home/wferi/ha/pacemaker/translib/libtool" --tag CC --mode=relink gcc -g -O2
-avoid-version -o a/liba.la -rpath /tmp/translib/lib a/a.lo b/libb.la )
libtool: relink: gcc -shared -fPIC -DPIC a/.libs/a.o -Wl,-rpath
-Wl,/tmp/translib/lib -L/tmp/translib/lib -lb -g -O2 -Wl,-soname -Wl,liba.so
-o a/.libs/liba.so
/usr/bin/ld: cannot find -lb
collect2: error: ld returned 1 exit status
libtool: error: error: relink 'a/liba.la' with the above command before
installing it
make[1]: *** [Makefile:446: install-libLTLIBRARIES] Error 1
make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib'
make: *** [Makefile:798: install-am] Error 2
No cyclic dependencies here, so this can be worked around by
-lib_LTLIBRARIES = a/liba.la b/libb.la
+lib_LTLIBRARIES = b/libb.la a/liba.la
in this case; is this expected (and documented) behavior?
Anyway: if, after a successful installation, I introduce a new symbol
(b2) in libb
diff --git a/a/a.c b/a/a.c
index ffc6553..8edd6f5 100644
--- a/a/a.c
+++ b/a/a.c
@@ -3,5 +3,5 @@
int a (int x)
{
-return b(x)+1;
+return b(x)+b2(x)+1;
}
diff --git a/b/b.c b/b/b.c
index 27bd35f..a723efc 100644
--- a/b/b.c
+++ b/b/b.c
@@ -4,3 +4,8 @@ int b (int x)
{
return x+2;
}
+
+int b2 (int x)
+{
+return x+3;
+}
diff --git a/b/b.h b/b/b.h
index 2290498..c0ea0fb 100644
--- a/b/b.h
+++ b/b/b.h
@@ -1 +1,2 @@
int b (int x);
+int b2 (int x);
and use it from liba, linking the final binary fails:
$ make
[...]
/bin/bash ./libtool --tag=CC --mode=link gcc -g -O2 -avoid-version -o
translib translib.o a/liba.la
libtool: link: gcc -g -O2 -o .libs/translib translib.o a/.libs/liba.so
-Wl,-rpath -Wl,/tmp/translib/lib
/usr/bin/ld: a/.libs/liba.so: undefined reference to `b2'
As I understand it, the -rpath linker option on the above command makes
a/.libs/liba.so use the already installed (old) version of libb, which
lacks the b2 symbol. What's the solution here? Why isn't that -rpath
option "delayed" until the relinking phase?
--
Thanks,
Feri
translib.tgz
Description: application/gtar-compressed
