Re: transitive shared library dependencies and installation

[Bob Friesenhahn]

On Sat, 4 Jan 2020, Russ Allbery wrote:


wf...@niif.hu writes:

Bob Friesenhahn  writes:



That sounds like a great idea.  However, there is a problem that
linking on some systems does depend on already installed libraries (or
will end up using them) and so the libraries need to be installed and
linked in a particular order.



Do you happen to know which systems these are?  Where are these
constraints documented?


It's been a very long time since I've built shared libraries for any
platform other than Linux, but my distant recollection is that AIX and
HP-UX encoded the full path to the dynamic library against which a binary
was linked.  Therefore, if you built an executable against uninstalled
libraries, the executable would have a reference to the build directory,
hence the explicit relinking step after installation to pick up the new
paths.  I seem to recall that we caused an outage for campus AIX systems
because of this; I'm less sure about HP-UX.

I don't remember whether this also applied to interdependencies between
shared libraries (or if those platforms even supported encoding
dependencies in shared libraries; a lot of platforms didn't).  I vaguely
recall that mutually-dependent shared libraries were actually impossible
on at least some UNIX platforms, and thus not portable.


The above matches my own memories.  I do not know the current level of 
usage for AIX or HP-UX but expect that they are still used for 
business purposes because at various times they were market leaders 
and offered the type of support that large businesses needed for 
critical systems.


Mutually-dependent shared libraries are to be avoided in any case.

Using Autotools, I do not recall any particular effort involved with 
supporting AIX, HP-UX, or even IRIX in GraphicsMagick builds.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: transitive shared library dependencies and installation

[Russ Allbery]

wf...@niif.hu writes:
> Bob Friesenhahn  writes:

>> That sounds like a great idea.  However, there is a problem that
>> linking on some systems does depend on already installed libraries (or
>> will end up using them) and so the libraries need to be installed and
>> linked in a particular order.

> Do you happen to know which systems these are?  Where are these
> constraints documented?

It's been a very long time since I've built shared libraries for any
platform other than Linux, but my distant recollection is that AIX and
HP-UX encoded the full path to the dynamic library against which a binary
was linked.  Therefore, if you built an executable against uninstalled
libraries, the executable would have a reference to the build directory,
hence the explicit relinking step after installation to pick up the new
paths.  I seem to recall that we caused an outage for campus AIX systems
because of this; I'm less sure about HP-UX.

I don't remember whether this also applied to interdependencies between
shared libraries (or if those platforms even supported encoding
dependencies in shared libraries; a lot of platforms didn't).  I vaguely
recall that mutually-dependent shared libraries were actually impossible
on at least some UNIX platforms, and thus not portable.

Note that AIX is very, very different than ELF-based UNIX systems (which
include Linux, Solaris, and I believe many of the BSDs) in how it handles
both shared libraries and system calls.  AIX shared libraries are quite
unusual.  I distantly recall that they're ar archives that contain a
special object file named shr5.o or something like that?

Both Tru64 and IRIX also had some odd linking behavior; I don't remember
if they would have had similar issues.

At the time, at Stanford we made the decision to build all binaries static
on all platforms other than Solaris and Linux because the shared library
behavior on other platforms was so odd and surprising that we weren't
confident we would get the details correct and avoid introducing weird
bugs.

It's of course an open question how much any given project cares about
supporting those UNIX variants these days.  Some of them are still sold
today, but the market share is quite small; others are varying degrees of
obsolete.  But that was the problem Libtool was originally written to
solve, and is why it does so many things that on Linux are weird and
unnecessary.

-- 
Russ Allbery (ea...@eyrie.org) 

Re: transitive shared library dependencies and installation

[wferi]

Bob Friesenhahn  writes:

> On Sat, 4 Jan 2020, wf...@niif.hu wrote:
>
>>> Above dependency explain all - lib_LTLIBRARIES is project (sample)
>>> specific. Project should ensure order.
>>
>> Fair enogh, but my point (which I didn't really emphasize) is that
>> installation could work in any order, if relinking didn't depend on
>> already installed libraries.  It's even necessary for mutually dependent
>> libraries.
>
> That sounds like a great idea.  However, there is a problem that
> linking on some systems does depend on already installed libraries (or
> will end up using them) and so the libraries need to be installed and
> linked in a particular order.

Do you happen to know which systems these are?  Where are these
constraints documented?  Do the libtool maintainers (er, anybody) have
access to such systems?  I think these are prerequisites for effective
libtool maintenance (and I'm probably just scratching the surface).
-- 
Regards,
Feri

Re: transitive shared library dependencies and installation

[wferi]

Roumen Petrov  writes:

> wf...@niif.hu wrote:
>
>> Roumen Petrov  writes:
>>
>>> wf...@niif.hu wrote:
>>>
 I'm experimenting with the attached skeleton project on a Debian buster
 system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9):
>
> Debian is key word.
> Libtool is patched, not FSF.

[...]

> With and without new dtags result will be the same.

Very well possible, it's just one difference I happened to know about.

> It is long history It starts with 1* (1.5) libtool . Libtool 1.5 has
> some issues with multiple dependent libraries (more then two).
> From debian was proposed a patch related to library
> dependencies. Unfortunately patch break existing regression test. From
> debian never was proposed version that pass regression test.
>
> Libtool 2.0 fixes his issues related to multiple libraries. On the
> same Debian did not stop to contribute patch that breaks libtool.
>
> As result when I decide to build something from source always to
> updated sources to FSF version.
>
> So the right question is did reporter test with FSF version?

I'm the reporter, and I didn't test any other version, as I wasn't even
sure whether my example was correct and was supposed to work.  Could you
please provide some keywords to search for so that I can dig up the
details of the above story?  At the moment Debian carries 21 patches for
libtool, if I could show that one of them breaks a valid use case, that
would constitute a strong reason for dropping it.
-- 
Thanks,
Feri

Re: transitive shared library dependencies and installation

[Bob Friesenhahn]

On Sat, 4 Jan 2020, wf...@niif.hu wrote:

...
Above dependency explain all - lib_LTLIBRARIES is project (sample)
specific. Project should ensure order.


Fair enogh, but my point (which I didn't really emphasize) is that
installation could work in any order, if relinking didn't depend on
already installed libraries.  It's even necessary for mutually dependent
libraries.


That sounds like a great idea.  However, there is a problem that 
linking on some systems does depend on already installed libraries (or 
will end up using them) and so the libraries need to be installed and 
linked in a particular order.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: transitive shared library dependencies and installation

[Roumen Petrov]

Hello,

wf...@niif.hu wrote:

Roumen Petrov  writes:


wf...@niif.hu wrote:


I'm experimenting with the attached skeleton project on a Debian buster

Debian is key word.


system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9):

Libtool is patched, not FSF.

[SNIP]

/usr/bin/ld: cannot find -lb
collect2: error: ld returned 1 exit status
libtool:   error: error: relink 'a/liba.la' with the above command before 
installing it
make[1]: *** [Makefile:446: install-libLTLIBRARIES] Error 1
make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib'
make: *** [Makefile:798: install-am] Error 2

[SNIP]

and use it from liba, linking the final binary fails:

$ make
[...]
/bin/bash ./libtool  --tag=CC   --mode=link gcc  -g -O2 -avoid-version  -o 
translib translib.o a/liba.la
libtool: link: gcc -g -O2 -o .libs/translib translib.o  a/.libs/liba.so 
-Wl,-rpath -Wl,/tmp/translib/lib
/usr/bin/ld: a/.libs/liba.so: undefined reference to `b2'

As I understand it, the -rpath linker option on the above command makes
a/.libs/liba.so use the already installed (old) version of libb, which
lacks the b2 symbol.  What's the solution here?  Why isn't that -rpath
option "delayed" until the relinking phase?

Not reproducible with FSF release.

Could you please share your link command?  Does you system use the new
dynamic tags (DT_RUNPATH instead of DT_RPATH)?
This does not make sense. With and without  new dtags result will be the 
same.


It is long history It starts with 1* (1.5) libtool . Libtool 1.5 has 
some issues with multiple dependent libraries (more then two).
From debian was proposed a patch related to library dependencies. 
Unfortunately patch break existing regression test. From debian never 
was proposed version that pass regression test.


Libtool 2.0 fixes his issues related to multiple libraries. On the same 
Debian did not stop to contribute patch that breaks libtool.


As result when I decide to build something from source always to updated 
sources to FSF version.


So the right question is did reporter test with FSF version?


Regards,
Roumen Petrov

Re: transitive shared library dependencies and installation

[wferi]

Roumen Petrov  writes:

> wf...@niif.hu wrote:
>
>> I'm experimenting with the attached skeleton project on a Debian buster
>> system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9):
> [SNIP]
>> /usr/bin/ld: cannot find -lb
>> collect2: error: ld returned 1 exit status
>> libtool:   error: error: relink 'a/liba.la' with the above command before 
>> installing it
>> make[1]: *** [Makefile:446: install-libLTLIBRARIES] Error 1
>> make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib'
>> make: *** [Makefile:798: install-am] Error 2
>
> Libtool does not change "order" :(. Automake just uses project rules.
>
>
> Let view Makefile.in
> ...
> install-exec-am: install-binPROGRAMS install-libLTLIBRARIES
> ...
> install-binPROGRAMS: install-libLTLIBRARIES
> ...
> install-libLTLIBRARIES: $(lib_LTLIBRARIES)
> ...
> Above dependency explain all - lib_LTLIBRARIES is project (sample)
> specific. Project should ensure order.

Fair enogh, but my point (which I didn't really emphasize) is that
installation could work in any order, if relinking didn't depend on
already installed libraries.  It's even necessary for mutually dependent
libraries.

>> and use it from liba, linking the final binary fails:
>>
>> $ make
>> [...]
>> /bin/bash ./libtool  --tag=CC   --mode=link gcc  -g -O2 -avoid-version  -o 
>> translib translib.o a/liba.la
>> libtool: link: gcc -g -O2 -o .libs/translib translib.o  a/.libs/liba.so 
>> -Wl,-rpath -Wl,/tmp/translib/lib
>> /usr/bin/ld: a/.libs/liba.so: undefined reference to `b2'
>>
>> As I understand it, the -rpath linker option on the above command makes
>> a/.libs/liba.so use the already installed (old) version of libb, which
>> lacks the b2 symbol.  What's the solution here?  Why isn't that -rpath
>> option "delayed" until the relinking phase?
>
> Not reproducible with FSF release.

Could you please share your link command?  Does you system use the new
dynamic tags (DT_RUNPATH instead of DT_RPATH)?
-- 
Thanks,
Feri

Re: transitive shared library dependencies and installation

[Roumen Petrov]

wf...@niif.hu wrote:

Hi,

I'm experimenting with the attached skeleton project on a Debian buster
system (autoconf 2.69-11, automake 1:1.16.1-4 and libtool 2.4.6-9):

[SNIP]

/usr/bin/ld: cannot find -lb
collect2: error: ld returned 1 exit status
libtool:   error: error: relink 'a/liba.la' with the above command before 
installing it
make[1]: *** [Makefile:446: install-libLTLIBRARIES] Error 1
make[1]: Leaving directory '/home/wferi/ha/pacemaker/translib'
make: *** [Makefile:798: install-am] Error 2


Libtool does not change "order" :(. Automake just uses project rules.


Let view Makefile.in
...
install-exec-am: install-binPROGRAMS install-libLTLIBRARIES
...
install-binPROGRAMS: install-libLTLIBRARIES
...
install-libLTLIBRARIES: $(lib_LTLIBRARIES)
...
Above dependency explain all - lib_LTLIBRARIES is project (sample) 
specific. Project should ensure order.




[SNIP]


and use it from liba, linking the final binary fails:

$ make
[...]
/bin/bash ./libtool  --tag=CC   --mode=link gcc  -g -O2 -avoid-version  -o 
translib translib.o a/liba.la
libtool: link: gcc -g -O2 -o .libs/translib translib.o  a/.libs/liba.so 
-Wl,-rpath -Wl,/tmp/translib/lib
/usr/bin/ld: a/.libs/liba.so: undefined reference to `b2'

As I understand it, the -rpath linker option on the above command makes
a/.libs/liba.so use the already installed (old) version of libb, which
lacks the b2 symbol.  What's the solution here?  Why isn't that -rpath
option "delayed" until the relinking phase?

Not reproducible with FSF release.

Regards,
Roumen Petrov