Re: transitive shared library dependencies and installation

[Roumen Petrov]

Hi Russ, libtool members,


I work on open source projects in my spare time. And so I can not 
provide feedback promptly. Today I have time to re read mail thread and 
to clarify my strong position.


Russ Allbery wrote:

Roumen Petrov  writes:


Urgh, article with very limited niche. First is only for shared libraries,
next unresolved externals in shared library, and finally requires
"advanced" loader.

Overlinking isn't about unresolved externals in shared libraries.
With unresolved I think about something different but this area is 
mostly out of scope to this thread.



It's
about avoiding linking with shared libraries that *don't* resolve any
external symbols.
This is one point - limited to "shared". Issue is not applicable for 
"static".



   It doesn't require an advanced loader, just any
reasonably modern shared library system; Solaris 2.6 from many years ago
had the same issue.
I'm thinking about Android - with modern shared library system and poor 
loader.

Autotools should work for such hosts as well.


   It is specific to systems where the linker is willing
to encode dependencies to libraries (in NEEDED, for instance) that aren't
actually required, but that's the default behavior of, say, GNU binutils.
Yes binutils adds to "needed" all passed on command line libraries for 
ELF but not for COFF.



The standard problem is that liba depends on libb, and a binary depends on
liba (but not libb).
Yes if you mean that binary uses only functions from "liba" and no one 
of all those functions need function from "libb". Let skip use of "data" 
from a library.


If B [binary] uses Fa() [function from a] and Fa() uses Fb() [function 
from] the thinks are different.

If we perform static build we must list both libraries.
Another case is shared build. For COFF is used import library. Import 
library does not add "dependency". So we must list both libraries.


Also for COFF list may contain "extra libraries". Binutil linker will 
link(pick) only ready used - there is no overlinking!




Libtool's default behavior in the presence of *.la
files will link the binary with both liba and libb, even though it only
needs to be linked with liba.

Only in "shared" build for ELF.


This causes a lot of problems for the
dependency management in a distribution if allowed to happen archive-wide,
since it creates tons of unnecessary dependency edges that make dependency
resolution and archive management much harder.

And the main question is who is responsible to resolve this issues.


Avoiding *.la files solves most of this.

Using *la files is very helpful for static or for COFF shared builds!


--as-needed helps when the
upstream Makefiles are designed to support static linking (where all the
libraries have to be listed), or where upstream just didn't pay attention
to this or didn't consider overlinking to be a problem, but has been known
to have bugs where it leaves out dependencies that actually are required.
This is specific for ELF. It seems to me discussion for linker 
capability exceed scope of this mail list.



For the software I personally maintain, --enable-reduced-dependencies
changes the behavior of Autoconf macros to not add the full transitive set
of libraries and instead only link with the immediate dependencies.  But I
think I'm the only person who uses that convention.  For newer software
that uses pkg-config, pkg-config supports this inherently if the library
maintainers write correct pkg-config files (which a lot of them don't).

The article may have limited applicability in the grand picture of all
systems and all ways of linking libraries, but it applies to pretty much
every Linux distribution (and probably any other distribution of any kind
of UNIX that uses shared libraries and package dependencies).  So by
quantity of Libtool installs and invocations, it's significant.


Open source is not limited to Linux. Build should be work on many 
OS-es(complilers, linkers, loaders).


Article https://wiki.mageia.org/en/Overlinking_issues_in_packaging 
inform us about issue in preamble.
Unfortunately context is totally misleading. It proposes correction in 
some build helpers - this definitely breaks other platforms.



As exercise:
Write configure script that test for presence of OpenSSL function 
EVP_add_cipher. Use existing macro AC_LINK_IFELSE. Ensure that it will 
work for shared and static build. Ensure that test will work on linux, 
mingw (solaris, android and etc.) hosts.


Note configure script does not use linker wrapper scripts like libtool 
and I would like to suggest do not use macro PKG_CHECK_MODULES.



I guess that after successful implementation my opinion about article 
will be more clear.



Also article point to "underlinking". With other words proposed 
solutions for build helpers (pkg-config, libtool) will lead to 
underlinking on other platforms..



Off topic: For COFF host there is tool that from DLL creates import 
library. DLL lists other needed libraries and COFF does not 

Re: transitive shared library dependencies and installation

[Russ Allbery]

Roumen Petrov  writes:

> Urgh, article with very limited niche. First is only for shared libraries,
> next unresolved externals in shared library, and finally requires
> "advanced" loader.

Overlinking isn't about unresolved externals in shared libraries.  It's
about avoiding linking with shared libraries that *don't* resolve any
external symbols.  It doesn't require an advanced loader, just any
reasonably modern shared library system; Solaris 2.6 from many years ago
had the same issue.  It is specific to systems where the linker is willing
to encode dependencies to libraries (in NEEDED, for instance) that aren't
actually required, but that's the default behavior of, say, GNU binutils.

The standard problem is that liba depends on libb, and a binary depends on
liba (but not libb).  Libtool's default behavior in the presence of *.la
files will link the binary with both liba and libb, even though it only
needs to be linked with liba.  This causes a lot of problems for the
dependency management in a distribution if allowed to happen archive-wide,
since it creates tons of unnecessary dependency edges that make dependency
resolution and archive management much harder.

Avoiding *.la files solves most of this.  --as-needed helps when the
upstream Makefiles are designed to support static linking (where all the
libraries have to be listed), or where upstream just didn't pay attention
to this or didn't consider overlinking to be a problem, but has been known
to have bugs where it leaves out dependencies that actually are required.

For the software I personally maintain, --enable-reduced-dependencies
changes the behavior of Autoconf macros to not add the full transitive set
of libraries and instead only link with the immediate dependencies.  But I
think I'm the only person who uses that convention.  For newer software
that uses pkg-config, pkg-config supports this inherently if the library
maintainers write correct pkg-config files (which a lot of them don't).

The article may have limited applicability in the grand picture of all
systems and all ways of linking libraries, but it applies to pretty much
every Linux distribution (and probably any other distribution of any kind
of UNIX that uses shared libraries and package dependencies).  So by
quantity of Libtool installs and invocations, it's significant.

-- 
Russ Allbery (ea...@eyrie.org) 

Re: transitive shared library dependencies and installation

[Roumen Petrov]

Hi Feri,

wf...@niif.hu wrote:

[SNIP]
On the other hand, this overlinks the final binary:
$ objdump -p .libs/translib | fgrep NEEDED
   NEEDED   liba.so
   NEEDED   libb.so
   NEEDED   libc.so.6

libb.so is unneeded here (but is present in the installed program as
well).  Coincidentally, the most prominent search result
https://wiki.mageia.org/en/Overlinking_issues_in_packaging
Urgh, article with very limited niche. First is only for shared 
libraries, next unresolved externals in shared library, and finally 
requires "advanced" loader.


For instance on Microsoft Windows shared library (DLL) does not allow 
"unresolved symbols".
Remark: Initial example requires -no-undefined flags as library LDFLAGS 
(libtool requirement) for mingw* builds.
Even that fact that for executable build command includes all libraries 
linker may exclude some of them. I guess that this is because usually 
build uses "import libraries". Dunno what is behavior if build is 
directly with DLLs.


Another system is Android. System allows unresolved externals but loader 
is quite simple. To load dynamically shared library is must not contain 
unresolved externals.


Static build: partial or complete. There is no rule in general. Linker 
picks required symbols from archives and may require some dependent 
libraries. So in some cases environment variable PKG_CONFIG set to 
pkg-config --static is solutions.




  mentions that
"this is fixed using a patch from Debian" for libtool.


I wonder why is patched libtool.
Libtool is not only Debian. Is is supposed to work on all supported 
OS-se, distributions and etc.
Because of those "fixes" it is mandatory to bootstrap source tarbal 
generated on Debian with M4 macros from FSF sources.

Now I do this unconditionally on all source tarballs - it save my time.


What's your position on this?  Is overlinking a problem or not?  (It
causes problems for distributions.)
No Idea. I guess that this is more related to the rules describing 
package dependencies.
A package could be split into multiple sub-package. A project may 
produce one "big" library or several "small" libraries. In both cases 
upgrade of related package is maintenance hassle.


Topic is not for libtool . It is more general.



Should everybody use --as-needed
globally to combat it?  Something else entirely?


On some OS-es --as-needed is enabled by default. I'm not aware why this 
model was chosen instead "s/tatus quo"./


As needed is fine with limited shared library format like those used on 
Microsoft windows  (COFF?) - no unresolved, no weak symbols. In the past 
loader does not guarantee that load of library follow dependency rules.


Also as needed may break use of weak aliases (ELF format).

I'm not sure that libtool respect order of linker options like 
--no-as-needed or --as-needed.


As alternative could be used patchelf.

It seems to me question is how to remove some extra dependencies : 
automatically (--as-needed) or manually (using tools like patchelf).


May be topic is not for libtool list.

Regards,
Roumen Petrov

Re: transitive shared library dependencies and installation

[wferi]

Bob Friesenhahn  writes:

> On Sun, 5 Jan 2020, wf...@niif.hu wrote:
>
>> On the other hand, this overlinks the final binary:
>>
>> $ objdump -p .libs/translib | fgrep NEEDED
>>  NEEDED   liba.so
>>  NEEDED   libb.so
>>  NEEDED   libc.so.6
>>
>> libb.so is unneeded here (but is present in the installed program as
>> well).  Coincidentally, the most prominent search result
>> https://wiki.mageia.org/en/Overlinking_issues_in_packaging mentions that
>> "this is fixed using a patch from Debian" for libtool.
>>
>> What's your position on this?  Is overlinking a problem or not?  (It
>> causes problems for distributions.)  Should everybody use --as-needed
>> globally to combat it?  Something else entirely?
>
> This has been the most common complaint (in the GNU Linux world) I
> have heard about libtool.  There is a choice of working reliably and
> portably (with possible over-linking) or relying on implicit library
> dependencies (which are definitely not portable), or failing as you
> have encountered.

Is the presence of over-linking a portability aspect?  I mean, couldn't
libtool overlink only where it's unavoidable, and link precisely on
systems which allow this (via -rpath-link or other means)?

> A similar complaint is that libtool uses information in installed
> ".la" files in order to link with all libraries that the
> program/library is dependent on.

This is the exact same issue as above, isn't it?

> Due to this, GNU Linux distributions often patch out this capability,
> and they don't distribute ".la" files.

Yes, I think they are (meant to be) replaced by pkg-config, and distros
have little interest in static linking or ltdl anyway (with a few
exceptions, like ImageMagick).  And ELF shared objects carry their
dependency information internally.

> Unfortunately, --as-needed may not be 100% reliable since it only
> reliably detects direct dependence on library symbols, and not
> "transitive" dependence.

Maybe I misunderstand you, but that's its very point in my opinion.
Anyway, the most trouble with it seems to originate from libtool
reordering the linker flags when creating shared libraries, making
--as-needed ineffective by moving it to the end of the command line.
At least this used to be the case, see
https://stackoverflow.com/questions/6852414/how-do-i-link-a-shared-library-with-as-needed-with-automake
(This thread is too broad already, let's discuss that in a new one if
you've got the bandwidth.)
-- 
Thanks,
Feri

Re: transitive shared library dependencies and installation

[Bob Friesenhahn]

On Sun, 5 Jan 2020, wf...@niif.hu wrote:


On the other hand, this overlinks the final binary:

$ objdump -p .libs/translib | fgrep NEEDED
 NEEDED   liba.so
 NEEDED   libb.so
 NEEDED   libc.so.6

libb.so is unneeded here (but is present in the installed program as
well).  Coincidentally, the most prominent search result
https://wiki.mageia.org/en/Overlinking_issues_in_packaging mentions that
"this is fixed using a patch from Debian" for libtool.

What's your position on this?  Is overlinking a problem or not?  (It
causes problems for distributions.)  Should everybody use --as-needed
globally to combat it?  Something else entirely?


This has been the most common complaint (in the GNU Linux world) I 
have heard about libtool.  There is a choice of working reliably and 
portably (with possible over-linking) or relying on implicit library 
dependencies (which are definitely not portable), or failing as you 
have encountered.  A similar complaint is that libtool uses 
information in installed ".la" files in order to link with all 
libraries that the program/library is dependent on.  Due to this, GNU 
Linux distributions often patch out this capability, and they don't 
distribute ".la" files.


Unfortunately, --as-needed may not be 100% reliable since it only 
reliably detects direct dependence on library symbols, and not 
"transitive" dependence.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: transitive shared library dependencies and installation

[wferi]

Roumen Petrov  writes:

> wf...@niif.hu wrote:
>
>>> So the right question is did reporter test with FSF version?
>>
>> I'm the reporter, and I didn't test any other version, as I wasn't
>> even sure whether my example was correct and was supposed to work.
>
> You sample is correct .  One minor nit is AM_PROG_AR - it is not
> required for posted example. If you project requires use of this macro
> you should request automake 1.11.2 as minimum.

Hi Roumen,

Thanks for the confirmation.  Initially I didn't include AM_PROG_AR in
the sample, but then autoreconf emitted much noise like

/usr/share/automake-1.16/am/ltlibrary.am: warning: 'a/liba.la': linking libtool 
libraries using a non-POSIX
/usr/share/automake-1.16/am/ltlibrary.am: archiver requires 'AM_PROG_AR' in 
'configure.ac'
Makefile.am:4:   while processing Libtool library 'a/liba.la'

(and the same for libb), and eventually I gave in.

> Result on posted test case - pass:
> $ ./translib ; echo $?
> 12

Thanks for posting your results.  Meanwhile I also tested with FSF
libtool 2.4.6, and the final "make" indeed succeeds with it, unlike with
Debian's libtool.  The decisive difference is that with the FSF version
the full build path to libb.so is explicitly added to the final link
command, thus the linker needn't search for it by itself.

On the other hand, this overlinks the final binary:

$ objdump -p .libs/translib | fgrep NEEDED
  NEEDED   liba.so
  NEEDED   libb.so
  NEEDED   libc.so.6

libb.so is unneeded here (but is present in the installed program as
well).  Coincidentally, the most prominent search result
https://wiki.mageia.org/en/Overlinking_issues_in_packaging mentions that
"this is fixed using a patch from Debian" for libtool.

What's your position on this?  Is overlinking a problem or not?  (It
causes problems for distributions.)  Should everybody use --as-needed
globally to combat it?  Something else entirely?
-- 
Thanks,
Feri

Re: transitive shared library dependencies and installation

[Roumen Petrov]

Hi Feri,

wf...@niif.hu wrote:

[SNIP]

It is long history It starts with 1* (1.5) libtool . Libtool 1.5 has
some issues with multiple dependent libraries (more then two).
 From debian was proposed a patch related to library
dependencies. Unfortunately patch break existing regression test. From
debian never was proposed version that pass regression test.

Libtool 2.0 fixes his issues related to multiple libraries. On the
same Debian did not stop to contribute patch that breaks libtool.

As result when I decide to build something from source always to
updated sources to FSF version.

So the right question is did reporter test with FSF version?

I'm the reporter, and I didn't test any other version, as I wasn't even
sure whether my example was correct and was supposed to work.

You sample is correct .
One minor nit is AM_PROG_AR - it is not required for posted example. If 
you project requires use of this macro you should request automake 
1.11.2 as minimum.



Could you
please provide some keywords to search for so that I can dig up the
details of the above story?

Please check libtool achives.
To me situation on Debian is closed case.


  At the moment Debian carries 21 patches for
libtool, if I could show that one of them breaks a valid use case, that
would constitute a strong reason for dropping it.


In moment I do tests on some sources from development or stable branches 
on an another system (archaic :) ):


$ cat /etc/centos-release
CentOS release 6.10 (Final)

$ autoconf --version
autoconf (GNU Autoconf) 2.63

$ automake --version
automake (GNU automake) 1.11.1

$ libtool --version
ltmain.sh (GNU libtool) 2.2.6b

All is based on system packages except tested by me.


Result on posted test case - pass:
$ ./translib ; echo $?
12

$ readelf -a  a/.libs/liba.so  | grep RPATH
 0x000f (RPATH)  Library rpath: 
[/home/centos/.builds/test/autotools/translib/b/.libs:/tmp/translib/lib]


$ readelf -a .libs/lt-translib  | grep PATH
 0x000f (RPATH)  Library rpath: 
[/home/centos/.builds/test/autotools/translib/a/.libs:/home/centos/.builds/test/autotools/translib/b/.libs:/tmp/translib/lib]


$ readelf -a .libs/translib  | grep PATH
 0x000f (RPATH)  Library rpath: [/tmp/translib/lib]

$ readelf -a /tmp/translib/lib/liba.so | grep PATH
 0x000f (RPATH)  Library rpath: [/tmp/translib/lib]


Regards,
Roumen Petrov

Re: transitive shared library dependencies and installation

[Bob Friesenhahn]

On Sat, 4 Jan 2020, Russ Allbery wrote:


wf...@niif.hu writes:

Bob Friesenhahn  writes:



That sounds like a great idea.  However, there is a problem that
linking on some systems does depend on already installed libraries (or
will end up using them) and so the libraries need to be installed and
linked in a particular order.



Do you happen to know which systems these are?  Where are these
constraints documented?


It's been a very long time since I've built shared libraries for any
platform other than Linux, but my distant recollection is that AIX and
HP-UX encoded the full path to the dynamic library against which a binary
was linked.  Therefore, if you built an executable against uninstalled
libraries, the executable would have a reference to the build directory,
hence the explicit relinking step after installation to pick up the new
paths.  I seem to recall that we caused an outage for campus AIX systems
because of this; I'm less sure about HP-UX.

I don't remember whether this also applied to interdependencies between
shared libraries (or if those platforms even supported encoding
dependencies in shared libraries; a lot of platforms didn't).  I vaguely
recall that mutually-dependent shared libraries were actually impossible
on at least some UNIX platforms, and thus not portable.


The above matches my own memories.  I do not know the current level of 
usage for AIX or HP-UX but expect that they are still used for 
business purposes because at various times they were market leaders 
and offered the type of support that large businesses needed for 
critical systems.


Mutually-dependent shared libraries are to be avoided in any case.

Using Autotools, I do not recall any particular effort involved with 
supporting AIX, HP-UX, or even IRIX in GraphicsMagick builds.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: transitive shared library dependencies and installation

[Russ Allbery]

wf...@niif.hu writes:
> Bob Friesenhahn  writes:

>> That sounds like a great idea.  However, there is a problem that
>> linking on some systems does depend on already installed libraries (or
>> will end up using them) and so the libraries need to be installed and
>> linked in a particular order.

> Do you happen to know which systems these are?  Where are these
> constraints documented?

It's been a very long time since I've built shared libraries for any
platform other than Linux, but my distant recollection is that AIX and
HP-UX encoded the full path to the dynamic library against which a binary
was linked.  Therefore, if you built an executable against uninstalled
libraries, the executable would have a reference to the build directory,
hence the explicit relinking step after installation to pick up the new
paths.  I seem to recall that we caused an outage for campus AIX systems
because of this; I'm less sure about HP-UX.

I don't remember whether this also applied to interdependencies between
shared libraries (or if those platforms even supported encoding
dependencies in shared libraries; a lot of platforms didn't).  I vaguely
recall that mutually-dependent shared libraries were actually impossible
on at least some UNIX platforms, and thus not portable.

Note that AIX is very, very different than ELF-based UNIX systems (which
include Linux, Solaris, and I believe many of the BSDs) in how it handles
both shared libraries and system calls.  AIX shared libraries are quite
unusual.  I distantly recall that they're ar archives that contain a
special object file named shr5.o or something like that?

Both Tru64 and IRIX also had some odd linking behavior; I don't remember
if they would have had similar issues.

At the time, at Stanford we made the decision to build all binaries static
on all platforms other than Solaris and Linux because the shared library
behavior on other platforms was so odd and surprising that we weren't
confident we would get the details correct and avoid introducing weird
bugs.

It's of course an open question how much any given project cares about
supporting those UNIX variants these days.  Some of them are still sold
today, but the market share is quite small; others are varying degrees of
obsolete.  But that was the problem Libtool was originally written to
solve, and is why it does so many things that on Linux are weird and
unnecessary.

-- 
Russ Allbery (ea...@eyrie.org) 

Re: transitive shared library dependencies and installation

[wferi]

Bob Friesenhahn  writes:

> On Sat, 4 Jan 2020, wf...@niif.hu wrote:
>
>>> Above dependency explain all - lib_LTLIBRARIES is project (sample)
>>> specific. Project should ensure order.
>>
>> Fair enogh, but my point (which I didn't really emphasize) is that
>> installation could work in any order, if relinking didn't depend on
>> already installed libraries.  It's even necessary for mutually dependent
>> libraries.
>
> That sounds like a great idea.  However, there is a problem that
> linking on some systems does depend on already installed libraries (or
> will end up using them) and so the libraries need to be installed and
> linked in a particular order.

Do you happen to know which systems these are?  Where are these
constraints documented?  Do the libtool maintainers (er, anybody) have
access to such systems?  I think these are prerequisites for effective
libtool maintenance (and I'm probably just scratching the surface).
-- 
Regards,
Feri

Re: transitive shared library dependencies and installation

[wferi]

Roumen Petrov  writes:

> wf...@niif.hu wrote:
>
>> Roumen Petrov  writes:
>>
>>> wf...@niif.hu wrote:
>>>
 I'm experimenting with the attached skeleton project on a Debian buster
 system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9):
>
> Debian is key word.
> Libtool is patched, not FSF.

[...]

> With and without new dtags result will be the same.

Very well possible, it's just one difference I happened to know about.

> It is long history It starts with 1* (1.5) libtool . Libtool 1.5 has
> some issues with multiple dependent libraries (more then two).
> From debian was proposed a patch related to library
> dependencies. Unfortunately patch break existing regression test. From
> debian never was proposed version that pass regression test.
>
> Libtool 2.0 fixes his issues related to multiple libraries. On the
> same Debian did not stop to contribute patch that breaks libtool.
>
> As result when I decide to build something from source always to
> updated sources to FSF version.
>
> So the right question is did reporter test with FSF version?

I'm the reporter, and I didn't test any other version, as I wasn't even
sure whether my example was correct and was supposed to work.  Could you
please provide some keywords to search for so that I can dig up the
details of the above story?  At the moment Debian carries 21 patches for
libtool, if I could show that one of them breaks a valid use case, that
would constitute a strong reason for dropping it.
-- 
Thanks,
Feri

Re: transitive shared library dependencies and installation

[Bob Friesenhahn]

On Sat, 4 Jan 2020, wf...@niif.hu wrote:

...
Above dependency explain all - lib_LTLIBRARIES is project (sample)
specific. Project should ensure order.


Fair enogh, but my point (which I didn't really emphasize) is that
installation could work in any order, if relinking didn't depend on
already installed libraries.  It's even necessary for mutually dependent
libraries.


That sounds like a great idea.  However, there is a problem that 
linking on some systems does depend on already installed libraries (or 
will end up using them) and so the libraries need to be installed and 
linked in a particular order.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: transitive shared library dependencies and installation

[Roumen Petrov]

Hello,

wf...@niif.hu wrote:

Roumen Petrov  writes:


wf...@niif.hu wrote:


I'm experimenting with the attached skeleton project on a Debian buster

Debian is key word.


system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9):

Libtool is patched, not FSF.

[SNIP]

/usr/bin/ld: cannot find -lb
collect2: error: ld returned 1 exit status
libtool:   error: error: relink 'a/liba.la' with the above command before 
installing it
make[1]: *** [Makefile:446: install-libLTLIBRARIES] Error 1
make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib'
make: *** [Makefile:798: install-am] Error 2

[SNIP]

and use it from liba, linking the final binary fails:

$ make
[...]
/bin/bash ./libtool  --tag=CC   --mode=link gcc  -g -O2 -avoid-version  -o 
translib translib.o a/liba.la
libtool: link: gcc -g -O2 -o .libs/translib translib.o  a/.libs/liba.so 
-Wl,-rpath -Wl,/tmp/translib/lib
/usr/bin/ld: a/.libs/liba.so: undefined reference to `b2'

As I understand it, the -rpath linker option on the above command makes
a/.libs/liba.so use the already installed (old) version of libb, which
lacks the b2 symbol.  What's the solution here?  Why isn't that -rpath
option "delayed" until the relinking phase?

Not reproducible with FSF release.

Could you please share your link command?  Does you system use the new
dynamic tags (DT_RUNPATH instead of DT_RPATH)?
This does not make sense. With and without  new dtags result will be the 
same.


It is long history It starts with 1* (1.5) libtool . Libtool 1.5 has 
some issues with multiple dependent libraries (more then two).
From debian was proposed a patch related to library dependencies. 
Unfortunately patch break existing regression test. From debian never 
was proposed version that pass regression test.


Libtool 2.0 fixes his issues related to multiple libraries. On the same 
Debian did not stop to contribute patch that breaks libtool.


As result when I decide to build something from source always to updated 
sources to FSF version.


So the right question is did reporter test with FSF version?


Regards,
Roumen Petrov

Re: transitive shared library dependencies and installation

[wferi]

Roumen Petrov  writes:

> wf...@niif.hu wrote:
>
>> I'm experimenting with the attached skeleton project on a Debian buster
>> system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9):
> [SNIP]
>> /usr/bin/ld: cannot find -lb
>> collect2: error: ld returned 1 exit status
>> libtool:   error: error: relink 'a/liba.la' with the above command before 
>> installing it
>> make[1]: *** [Makefile:446: install-libLTLIBRARIES] Error 1
>> make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib'
>> make: *** [Makefile:798: install-am] Error 2
>
> Libtool does not change "order" :(. Automake just uses project rules.
>
>
> Let view Makefile.in
> ...
> install-exec-am: install-binPROGRAMS install-libLTLIBRARIES
> ...
> install-binPROGRAMS: install-libLTLIBRARIES
> ...
> install-libLTLIBRARIES: $(lib_LTLIBRARIES)
> ...
> Above dependency explain all - lib_LTLIBRARIES is project (sample)
> specific. Project should ensure order.

Fair enogh, but my point (which I didn't really emphasize) is that
installation could work in any order, if relinking didn't depend on
already installed libraries.  It's even necessary for mutually dependent
libraries.

>> and use it from liba, linking the final binary fails:
>>
>> $ make
>> [...]
>> /bin/bash ./libtool  --tag=CC   --mode=link gcc  -g -O2 -avoid-version  -o 
>> translib translib.o a/liba.la
>> libtool: link: gcc -g -O2 -o .libs/translib translib.o  a/.libs/liba.so 
>> -Wl,-rpath -Wl,/tmp/translib/lib
>> /usr/bin/ld: a/.libs/liba.so: undefined reference to `b2'
>>
>> As I understand it, the -rpath linker option on the above command makes
>> a/.libs/liba.so use the already installed (old) version of libb, which
>> lacks the b2 symbol.  What's the solution here?  Why isn't that -rpath
>> option "delayed" until the relinking phase?
>
> Not reproducible with FSF release.

Could you please share your link command?  Does you system use the new
dynamic tags (DT_RUNPATH instead of DT_RPATH)?
-- 
Thanks,
Feri

Re: transitive shared library dependencies and installation

[Roumen Petrov]

wf...@niif.hu wrote:

Hi,

I'm experimenting with the attached skeleton project on a Debian buster
system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9):

[SNIP]

/usr/bin/ld: cannot find -lb
collect2: error: ld returned 1 exit status
libtool:   error: error: relink 'a/liba.la' with the above command before 
installing it
make[1]: *** [Makefile:446: install-libLTLIBRARIES] Error 1
make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib'
make: *** [Makefile:798: install-am] Error 2


Libtool does not change "order" :(. Automake just uses project rules.


Let view Makefile.in
...
install-exec-am: install-binPROGRAMS install-libLTLIBRARIES
...
install-binPROGRAMS: install-libLTLIBRARIES
...
install-libLTLIBRARIES: $(lib_LTLIBRARIES)
...
Above dependency explain all - lib_LTLIBRARIES is project (sample) 
specific. Project should ensure order.




[SNIP]


and use it from liba, linking the final binary fails:

$ make
[...]
/bin/bash ./libtool  --tag=CC   --mode=link gcc  -g -O2 -avoid-version  -o 
translib translib.o a/liba.la
libtool: link: gcc -g -O2 -o .libs/translib translib.o  a/.libs/liba.so 
-Wl,-rpath -Wl,/tmp/translib/lib
/usr/bin/ld: a/.libs/liba.so: undefined reference to `b2'

As I understand it, the -rpath linker option on the above command makes
a/.libs/liba.so use the already installed (old) version of libb, which
lacks the b2 symbol.  What's the solution here?  Why isn't that -rpath
option "delayed" until the relinking phase?

Not reproducible with FSF release.

Regards,
Roumen Petrov

Re: transitive shared library dependencies and installation

[Bob Friesenhahn]

On Fri, 3 Jan 2020, Richard Purdie wrote:


Libtool must also work for static linking.  It seems to me that your
issue also impacts static linking.


I think the challenge that libtool has here is that many of these older
systems that libtool supports aren't so prevalent anymore.


This is true.  It is possible that support for some archaic compilers 
and systems should be removed since there is no way to verify that the 
support works anymore, and there might not be any more active users.


Regardless, libtool must continue to support static linking, even if 
deployment usually uses dynamic linking.  As a developer, I use static 
linkage for daily development yet recommend dynamic linkage for 
most deployments.



We have a number of libtool patches to sort cross compile issues which
really need discussion with upstream libtool. With the lack of
releases, our incentive to do that diminishes :(.


You can become part of upstream libtool and commit your fixes 
directly.



Finding new maintainers with the amount of knowledge of weird older
systems needed is going to be a struggle which only gets worse over
time :(


Most systems are not so weird.  Most problems with legacy targets have 
been due to shell and common utility differences/bugs (e.g. depending 
on 'bash' syntax or behavior of 'echo'.  These problems are due to the 
Autotools assumption that Autotools should be able to work with the 
shell and utilities that originally came with the system.


If it is impossible to find a representative target system to test on, 
then that is an indication that support should be deprecated.  Users 
of unpopular systems need to stand up for themselves and assist, even 
if only by providing remote access to test on the system.



I do worry about the future here as libtool is a key part of the
autotools fabric but its most likely to be wholesale replaced given how
things are trending.


The current trend is that all of Autotools at risk even though it 
still works pretty well.  Perhaps Automake is in best shape since it 
was released most recently.  Autoconf has continued to be developed, 
but has not been released.


Autoconf 2012
libtool 2014
Automake 2017

Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: transitive shared library dependencies and installation

[Richard Purdie]

On Thu, 2020-01-02 at 16:24 -0600, Bob Friesenhahn wrote:
> On Thu, 2 Jan 2020, wf...@niif.hu wrote:
> > > If Libtool were to depend on non-portable features, [...] then it
> > > could not longer be described as a portability tool.
> > 
> > In my understanding, Libtool is a portability shim, providing a
> > regular
> > interface for developers across systems with wildly varying
> > features.
> > It already uses non-portable features, just different ones on
> > different
> > architectures.  I don't say it should use -rpath-link generally, I
> > only
> > suggested that it might be an efficient solution on systems
> > supporting
> > it.  But I expect all systems supporting shared objects to allow
> > using
> > and installing them some way, irrespective of their
> > interdependencies.
> > Am I overly naive?
> 
> Certainly, libtool could use -rpath-link where it is supported but 
> libtool provides a common feature set and if it is only possible to 
> support a feature where -rpath-link is available, then offering the 
> feature would defeat the purpose of being a portability tool.
> 
> Sometimes it is better to force the using software to conform to the 
> limitations.
> 
> Libtool must also work for static linking.  It seems to me that your 
> issue also impacts static linking.

I think the challenge that libtool has here is that many of these older
systems that libtool supports aren't so prevalent anymore.

I work on the Yocto Project where we do a ton of cross compiling and
have to work with libtool but its mostly Linux with a small amount of
mingw/baremetal/darwin bits.

We have a number of libtool patches to sort cross compile issues which
really need discussion with upstream libtool. With the lack of
releases, our incentive to do that diminishes :(.

By reducing everything to the lowest common denominator, it means
libtool struggles to take advantage of any new linker technology too.

Finding new maintainers with the amount of knowledge of weird older
systems needed is going to be a struggle which only gets worse over
time :(

I do worry about the future here as libtool is a key part of the
autotools fabric but its most likely to be wholesale replaced given how
things are trending.

Cheers,

Richard

Re: transitive shared library dependencies and installation

[Bob Friesenhahn]

On Thu, 2 Jan 2020, wf...@niif.hu wrote:


If Libtool were to depend on non-portable features, [...] then it
could not longer be described as a portability tool.


In my understanding, Libtool is a portability shim, providing a regular
interface for developers across systems with wildly varying features.
It already uses non-portable features, just different ones on different
architectures.  I don't say it should use -rpath-link generally, I only
suggested that it might be an efficient solution on systems supporting
it.  But I expect all systems supporting shared objects to allow using
and installing them some way, irrespective of their interdependencies.
Am I overly naive?


Certainly, libtool could use -rpath-link where it is supported but 
libtool provides a common feature set and if it is only possible to 
support a feature where -rpath-link is available, then offering the 
feature would defeat the purpose of being a portability tool.


Sometimes it is better to force the using software to conform to the 
limitations.


Libtool must also work for static linking.  It seems to me that your 
issue also impacts static linking.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: transitive shared library dependencies and installation

[wferi]

Bob Friesenhahn  writes:

> On Thu, 2 Jan 2020, wf...@niif.hu wrote:
>
 True, but man ld states in the -rpath-link option description that the
 directories specified by -rpath options are used with a very high
 priority for locating required shared libraries.
>>>
>>> This is interesting but I am not seeing any use of -rpath-link in
>>> libtool.
>>
>> Neither do I, but I think it would be a possible way out of this
>> situation.  I wonder what the reason could be for libtool not using it.
>
> If Libtool were to depend on non-portable features, [...] then it
> could not longer be described as a portability tool.

In my understanding, Libtool is a portability shim, providing a regular
interface for developers across systems with wildly varying features.
It already uses non-portable features, just different ones on different
architectures.  I don't say it should use -rpath-link generally, I only
suggested that it might be an efficient solution on systems supporting
it.  But I expect all systems supporting shared objects to allow using
and installing them some way, irrespective of their interdependencies.
Am I overly naive?

> As far as I am aware, libtool is currently "between maintainers" and
> needs fresh volunteers to become a libtool maintainer.

Fresh volunteers with arcane knowledge... quite a tall order! :)

Back to my problem: first I'd like to establish whether the behavior I
see is a bug, and if it is, is it in Libtool or in my code or somewhere
else.  And then find a solution or at least a workaround.  Opinions?
-- 
Thanks,
Feri

Re: transitive shared library dependencies and installation

[Bob Friesenhahn]

On Thu, 2 Jan 2020, wf...@niif.hu wrote:

True, but man ld states in the -rpath-link option description that the
directories specified by -rpath options are used with a very high
priority for locating required shared libraries.


This is interesting but I am not seeing any use of -rpath-link in
libtool.


Neither do I, but I think it would be a possible way out of this
situation.  I wonder what the reason could be for libtool not using it.


Libtool is a portability tool.  If it were to depend on non-portable 
features, then software would be implemented using it which is 
non-portable and the software developers might not be aware of it. 
If Libtool were to require GNU Linux, or GNU ld, or GNU gcc. or GNU 
libc, or ELF, then it could not longer be described as a portability 
tool.


Indeed, if one makes some assumptions and is willing to lose some 
portability then it is not difficult to use underlying tools directly, 
without using libtool at all.


At one time it was assumed that GNU software would reach the largest 
number of potential users by being implemented in a portable fashion. 
An alternative would have been to require that GNU software can only 
be compiled on GNU systems (this would have been impossible to start 
with since such a system did not exist) using GNU tools.



It's rather hard to get support for libtool, but I sorely need some,
because this issue hurts a real software project.  Since this is the
first time I seriously deal with libtool, I certainly miss most of the
picture, so I sincerely appreciate your "piping up".  Let's hope others
will join with time with their insights.


As far as I am aware, libtool is currently "between maintainers" and 
needs fresh volunteers to become a libtool maintainer.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: transitive shared library dependencies and installation

[Bob Friesenhahn]

On Thu, 2 Jan 2020, wf...@niif.hu wrote:


In this case libtool is a slave of Automake and Automake is
controlling the build and installation order.  This means that it
should be expected behavior.  Installation is serialized and thus
order matters.


But in case of a dependency cycle there's no correct order if libtool
insists on using the installation directory for the -L option for
relinking.  And the installation directory may contain an incompatible
version of the library anyway, just like it happens below.


Indeed, sometimes it is necessary to re-organize libraries and code in 
order to be successful given serialized linking.



and use it from liba, linking the final binary fails:

$ make
[...]
/bin/bash ./libtool  --tag=CC   --mode=link gcc  -g -O2 -avoid-version  -o 
translib translib.o a/liba.la
libtool: link: gcc -g -O2 -o .libs/translib translib.o  a/.libs/liba.so 
-Wl,-rpath -Wl,/tmp/translib/lib
/usr/bin/ld: a/.libs/liba.so: undefined reference to `b2'

As I understand it, the -rpath linker option on the above command makes
a/.libs/liba.so use the already installed (old) version of libb, which
lacks the b2 symbol.  What's the solution here?  Why isn't that -rpath
option "delayed" until the relinking phase?


The -rpath linker option did not cause the library to be used.


The above gcc linking command is successful if I omit the -rpath option.
(The built-in RUNPATH of liba.so contains the build directory first.
Just for the record: Debian's ld defaults to --enable-new-dtags.)


I am not sure exactly what --enable-new-dtags means, but Ubuntu 18.04 
LTS's manual page says that this is not its default.  This may 
indicate changing behavior.



The '-r' in -rpath stands for 'run' and thus it is a path stored in a
built shared library or executable which tells it where to search for
other libraries when th program is executed.


True, but man ld states in the -rpath-link option description that the
directories specified by -rpath options are used with a very high
priority for locating required shared libraries.


This is interesting but I am not seeing any use of -rpath-link in 
libtool.



Perhaps the controlling parameter is hardcode_into_libs.  If your
libtool comes from an OS distribution rather than from a FSF/GNU
tarball, then it is possible that its behavior may have been modified.


I don't know what to expect, here's what I can see:

$ ./libtool --config | fgrep hardcode_into_libs
hardcode_into_libs=yes


I am not seeing 'libb' mentioned on the link line and that may be the
cause of the problem you are seeing.


Sure enough, adding libb.la explicitly to the link command works around
the issue, but since the binary does not use libb directly, it doesn't
sound like a good solution to me, because it leads to overlinking.


Libtool can only know what has been passed to it via the command line 
and via the prior information it stored in the .la files (initially 
found due to the command line).


I have always found it to be good practice to include all dependency 
libraries in an Automake 'LIBADD' statement so that all dependency 
libraries are passed to libtool.  If the .la file passed to libtool 
mentions the dependencies, then libtool is supposed to do the right 
thing.


At this point I am wondering why I am the only person on this list who 
has piped up with some sort of a response. :-(


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: transitive shared library dependencies and installation

[wferi]

Bob Friesenhahn  writes:

> On Wed, 1 Jan 2020, wf...@niif.hu wrote:
>
>> make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib'
>> make: *** [Makefile:798: install-am] Error 2
>>
>> No cyclic dependencies here, so this can be worked around by
>>
>> -lib_LTLIBRARIES = a/liba.la b/libb.la
>> +lib_LTLIBRARIES = b/libb.la a/liba.la
>>
>> in this case; is this expected (and documented) behavior?
>
> In this case libtool is a slave of Automake and Automake is
> controlling the build and installation order.  This means that it
> should be expected behavior.  Installation is serialized and thus
> order matters.

But in case of a dependency cycle there's no correct order if libtool
insists on using the installation directory for the -L option for
relinking.  And the installation directory may contain an incompatible
version of the library anyway, just like it happens below.

>> and use it from liba, linking the final binary fails:
>>
>> $ make
>> [...]
>> /bin/bash ./libtool  --tag=CC   --mode=link gcc  -g -O2 -avoid-version  -o 
>> translib translib.o a/liba.la
>> libtool: link: gcc -g -O2 -o .libs/translib translib.o  a/.libs/liba.so 
>> -Wl,-rpath -Wl,/tmp/translib/lib
>> /usr/bin/ld: a/.libs/liba.so: undefined reference to `b2'
>>
>> As I understand it, the -rpath linker option on the above command makes
>> a/.libs/liba.so use the already installed (old) version of libb, which
>> lacks the b2 symbol.  What's the solution here?  Why isn't that -rpath
>> option "delayed" until the relinking phase?
>
> The -rpath linker option did not cause the library to be used.

The above gcc linking command is successful if I omit the -rpath option.
(The built-in RUNPATH of liba.so contains the build directory first.
Just for the record: Debian's ld defaults to --enable-new-dtags.)

> The '-r' in -rpath stands for 'run' and thus it is a path stored in a
> built shared library or executable which tells it where to search for
> other libraries when th program is executed.

True, but man ld states in the -rpath-link option description that the
directories specified by -rpath options are used with a very high
priority for locating required shared libraries.

> Perhaps the controlling parameter is hardcode_into_libs.  If your
> libtool comes from an OS distribution rather than from a FSF/GNU
> tarball, then it is possible that its behavior may have been modified.

I don't know what to expect, here's what I can see:

$ ./libtool --config | fgrep hardcode_into_libs
hardcode_into_libs=yes

> I am not seeing 'libb' mentioned on the link line and that may be the
> cause of the problem you are seeing.

Sure enough, adding libb.la explicitly to the link command works around
the issue, but since the binary does not use libb directly, it doesn't
sound like a good solution to me, because it leads to overlinking.
-- 
Thanks,
Feri

Re: transitive shared library dependencies and installation

[Bob Friesenhahn]

On Wed, 1 Jan 2020, wf...@niif.hu wrote:

make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib'
make: *** [Makefile:798: install-am] Error 2

No cyclic dependencies here, so this can be worked around by

-lib_LTLIBRARIES = a/liba.la b/libb.la
+lib_LTLIBRARIES = b/libb.la a/liba.la

in this case; is this expected (and documented) behavior?


In this case libtool is a slave of Automake and Automake is 
controlling the build and installation order.  This means that it 
should be expected behavior.  Installation is serialized and thus 
order matters.



and use it from liba, linking the final binary fails:

$ make
[...]
/bin/bash ./libtool  --tag=CC   --mode=link gcc  -g -O2 -avoid-version  -o 
translib translib.o a/liba.la
libtool: link: gcc -g -O2 -o .libs/translib translib.o  a/.libs/liba.so 
-Wl,-rpath -Wl,/tmp/translib/lib
/usr/bin/ld: a/.libs/liba.so: undefined reference to `b2'

As I understand it, the -rpath linker option on the above command makes
a/.libs/liba.so use the already installed (old) version of libb, which
lacks the b2 symbol.  What's the solution here?  Why isn't that -rpath
option "delayed" until the relinking phase?


The -rpath linker option did not cause the library to be used.  The 
'-r' in -rpath stands for 'run' and thus it is a path stored in a 
built shared library or executable which tells it where to search for 
other libraries when th program is executed.


The libtool configuration may be dumped using

  ./libtool --config

Perhaps the controlling parameter is hardcode_into_libs.  If your 
libtool comes from an OS distribution rather than from a FSF/GNU 
tarball, then it is possible that its behavior may have been modified.


I am not seeing 'libb' mentioned on the link line and that may be the 
cause of the problem you are seeing.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt