Re: [EXTERNAL] Re: Q: Forcing a -Wl,-rpath arg to static lib users
On Fri, Nov 18, 2022 at 12:28 PM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Fri, 18 Nov 2022, Oleg Smolsky wrote: > >> > >> The libtool provided as part of a Linux distribution often hacks > >> libtool so that it does not include full dependency information in the > >> library.la files. They do this in order to avoid "excessive linkage" > >> because they do not want the program/library to retain full linkage > >> details in case the OS changes the libraries. > >> > > > > Oh, that's a very interesting hint! thanks, Bob! I am using libtoon from > > the distro. > > > > What does it take to take libtool from upstream? Certainly I can fetch > its > > source... but how do I marry that with the `autoreconf` invocation that > > drives build system generation? > > A reasonable thing to do would be to download the autoconf, automake, > and libtool tarballs from https://ftp.gnu.org/gnu/. Build and install > each one using the same installation prefix. It is best if the > installation prefix does not interfere with your operating system (the > default of "/usr/local" normally works). > > OK, I've just installed the latest tagged versions of [autoconf, automake, libtool] into /opt/3p, regenerated my build system and compiled the minimal test case. The key variable that I would like to carry the -Wl,rpath flags is still blank for my static lib: # Linker flags that cannot go in dependency_libs. inherited_linker_flags='' So, do you think these versions are new enough? [autoconf] GitRevision = v2.71 Source = https://github.com/autotools-mirror/autoconf.git [automake] GitRevision = v1.16.5 Source = https://github.com/autotools-mirror/automake.git [libtool] GitRevision = v2.4.7 Source = https://github.com/autotools-mirror/libtool.git Thanks! Oleg.
Re: [EXTERNAL] Re: Q: Forcing a -Wl,-rpath arg to static lib users
On Thu, Nov 17, 2022 at 4:05 PM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Wed, 16 Nov 2022, Oleg Smolsky wrote: > > > > Leaving it here for posterity. Perhaps someone will do this with a bit > more > > finesse and turn it into a proper feature. > > Are you using libtool as originally distributed by the FSF or are you > using a libtool provided by a Linux system package? > > The libtool provided as part of a Linux distribution often hacks > libtool so that it does not include full dependency information in the > library.la files. They do this in order to avoid "excessive linkage" > because they do not want the program/library to retain full linkage > details in case the OS changes the libraries. > Oh, that's a very interesting hint! thanks, Bob! I am using libtoon from the distro. What does it take to take libtool from upstream? Certainly I can fetch its source... but how do I marry that with the `autoreconf` invocation that drives build system generation? Oleg.
Re: Q: Forcing a -Wl,-rpath arg to static lib users
On Mon, Nov 14, 2022 at 8:36 PM Oleg Smolsky wrote: > Hello folks! I've been using libtool (via autotools) for a while, but have > been unable to find a clean solution to the following problem: > > I have a static `liblocal.la` (built in my build system) that links to a > third-party shared lib. Obviously I can use my own lib to express that > dependency: > > liblocal_la_LIBADD = -lexternal -L/somewhere/lib > > The missing bit of functionality is -Wl,-rpath,/somewhere/lib that I need > to impose onto the leaf-level executables that link my local lib. > Unfortunately these attempts do not work: > 1. Cannot add that flag to _LIBADD as libtool complains: "...belong in > 'liblocal_la_LDFLAGS" > 2. Cannot add that flag to _LDFLAGS as it does not carry all the way to > the leaf-level executable > > I have found that I can manually jam the flag into the > `inherited_linker_flags` variable inside the .la file and that gives me the > desired result... but I don't know how to do that via Makefile.am. > ...and I found a hack that lets me jam these flags into the `inherited_linker_flags` variable. Here is a patch for the `libtool` script generated by libtool into $builddir: ``` --- libtool.base2022-11-14 21:18:04.867760568 -0800 +++ libtool.patched 2022-11-14 21:17:32.967784300 -0800 @@ -7826,6 +7826,7 @@ ;; -Wl,*) + func_append new_inherited_linker_flags " $arg" func_stripname '-Wl,' '' "$arg" args=$func_stripname_result arg= ``` Leaving it here for posterity. Perhaps someone will do this with a bit more finesse and turn it into a proper feature. Oleg.
Q: Forcing a -Wl,-rpath arg to static lib users
Hello folks! I've been using libtool (via autotools) for a while, but have been unable to find a clean solution to the following problem: I have a static `liblocal.la` (built in my build system) that links to a third-party shared lib. Obviously I can use my own lib to express that dependency: liblocal_la_LIBADD = -lexternal -L/somewhere/lib The missing bit of functionality is -Wl,-rpath,/somewhere/lib that I need to impose onto the leaf-level executables that link my local lib. Unfortunately these attempts do not work: 1. Cannot add that flag to _LIBADD as libtool complains: "...belong in 'liblocal_la_LDFLAGS" 2. Cannot add that flag to _LDFLAGS as it does not carry all the way to the leaf-level executable I have found that I can manually jam the flag into the `inherited_linker_flags` variable inside the .la file and that gives me the desired result... but I don't know how to do that via Makefile.am. I'd appreciate any and all hints/suggestions. Thanks! Oleg.
Re: [EXTERNAL] Re: LD_LIBRARY_PATH in wrapper scripts
On Sun, Aug 22, 2021 at 7:01 AM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Sat, 21 Aug 2021, Oleg Smolsky wrote: > >> > >> So, if libtool does not believe that /usr/lib/x86_64-linux-gnu is in > >> the default search path, it adds it. > >> > > Right, and my compiler is in /opt/gcc-11/ and so that addition to > > LD_LIBRARY_PATH is wrong. The system's compiler is older than what we use > > and the forced (older )version of libstdc++ breaks the executable. > > You should talk to the person who built and installed this compiler. > I recollect that it was you. :-) > > He, he, that's right! > Does > >gcc -print-search-dirs > > produce correct/useful output for your compiler? > Well, let's see: libraries: =/opt/gcc-11/lib/gcc/x86_64-pc-linux-gnu/11.2.0/:/opt/gcc-11/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../x86_64-pc-linux-gnu/lib/x86_64-pc-linux-gnu/11.2.0/:/opt/gcc-11/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../x86_64-pc-linux-gnu/lib/x86_64-linux-gnu/:/opt/gcc-11/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../x86_64-pc-linux-gnu/lib/../lib64/:/opt/gcc-11/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../x86_64-pc-linux-gnu/11.2.0/:/opt/gcc-11/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../x86_64-linux-gnu/:/opt/gcc-11/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../lib64/:/lib/x86_64-pc-linux-gnu/11.2.0/:/lib/x86_64-linux-gnu/:/lib/../lib64/:/usr/lib/x86_64-pc-linux-gnu/11.2.0/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib64/:/opt/gcc-11/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../x86_64-pc-linux-gnu/lib/:/opt/gcc-11/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../:/lib/:/usr/lib/ So, it looks like the '/usr/lib/x86_64-linux-gnu/' bit came straight from that spec... And yet, the previous GCC build that we used (gcc-10) has a nearly identical output and the associated application build does exhibit the linking issue with libstdc++. Hmm... so I am still struggling to understand the cause and effect here... > > If the system's 'ldconfig' is not aware of your compiler's library > installation, then the system will search the normal places for the > libraries unless you add an -rpath option to the build. > Right, each application is linked with -rpath settings to find that right libstdc++ that comes from /opt. > > Of course if ldconfig is aware of your compiler's library installation > then that causes problems if you are using multiple compilers. > > Right, I never modify the system's ldconfig settings as our toolchain and libs are separate. My Ubuntu 20 VM has the following (stock) settings: $ cat /etc/ld.so.conf.d/x86_64-linux-gnu.conf /usr/local/lib/x86_64-linux-gnu /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu Oleg.
Re: [EXTERNAL] Re: LD_LIBRARY_PATH in wrapper scripts
On Sat, Aug 21, 2021 at 4:25 PM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Sat, 21 Aug 2021, Oleg Smolsky wrote: > > > > > Hello there! We have an autotools-based build system setup with a custom > GCC where we take all build- and run-time dependencies (except for glibc) > > from /opt. Things have worked well on Ubuntu 16, yet I'm hitting a funky > issue when building on Ubuntu20 (libtool 2.4.6-14 according to dpkg). The > > issue comes down to one of the wrapper scripts that contains this gem: > > > > # Add our own library path to LD_LIBRARY_PATH > > > LD_LIBRARY_PATH="/usr/lib/x86_64-linux-gnu:/home/oleg/project/_obj/.libs:/opt/gcc-11/lib/../lib64:$LD_LIBRARY_PATH" > > > > Most of our libs are statically linked with exception of just one. So > tests/apps that use that shared lib end up with libtool wrappers... and they > > work correctly on Ubuntu16 (libtool 2.4.6-0.1 according to dpkg). It > seems that libtool version just does not stamp out this extra variable... > > > > The manual fix is to remove the "/usr/lib/x86_64-linux-gnu" bit from the > LD_LIBRARY_PATH above... and yet I have no idea where it came from or > > whether there is a way to influence its composition from a Makefile.am > file. > > I think that libtool tries to deduce the default run-time linker > search path (e.g. as used/provided to ldconfig, and also documented in > the 'ld.so' manual page) and it also tries to learn where the > compiler's own libraries are installed. > > So, if libtool does not believe that /usr/lib/x86_64-linux-gnu is in > the default search path, it adds it. > Right, and my compiler is in /opt/gcc-11/ and so that addition to LD_LIBRARY_PATH is wrong. The system's compiler is older than what we use and the forced (older )version of libstdc++ breaks the executable. So far the cleanest hack I've found is to patch the generated "libtool" script so that the generated test wrappers are correct. I do it that in the following crude way: AC_CONFIG_COMMANDS([libtool-test-wrapper-fix], [sed -Ei . libtool]) That's obviously both nasty and fragile. Is there a better way to influence the composition of LD_LIBRARY_FLAGS? It comes from this: $shlibpath_var=\"$temp_rpath\$$shlibpath_var\" Thanks! Oleg.
Re: LD_LIBRARY_PATH in wrapper scripts
On 2021-08-21 07:56, Oleg Smolsky wrote: Hello there! We have an autotools-based build system setup with a custom GCC where we take all build- and run-time dependencies (except for glibc) from /opt. Things have worked well on Ubuntu 16, yet I'm hitting a funky issue when building on Ubuntu20 (libtool 2.4.6-14 according to dpkg). The issue comes down to one of the wrapper scripts that contains this gem: # Add our own library path to LD_LIBRARY_PATH LD_LIBRARY_PATH="/usr/lib/x86_64-linux-gnu:/home/oleg/project/_obj/.libs:/opt/gcc-11/lib/../lib64:$LD_LIBRARY_PATH" Most of our libs are statically linked with exception of just one. So tests/apps that use that shared lib end up with libtool wrappers... and they work correctly on Ubuntu16 (libtool 2.4.6-0.1 according to dpkg). It seems that libtool version just does not stamp out this extra variable... So, I've just looked through libtool source and the problematic line is stamped out by the "libtool" script. So, here is a minimal hack that addresses the immediate issue: --- libtool 2021-08-21 09:16:55.317889207 -0700 +++ libtool.patched 2021-08-21 09:15:34.309736278 -0700 @@ -6083,7 +6083,9 @@ if test yes = "$shlibpath_overrides_runpath" && test -n "$shlibpath_var" && test -n "$temp_rpath"; then $ECHO "\ # Add our own library path to $shlibpath_var - $shlibpath_var=\"$temp_rpath\$$shlibpath_var\" + $shlibpath_var=\".libs\" # Some systems cannot cope with colon-terminated $shlibpath_var # The second colon is a workaround for a bug in BeOS R4 sed Is there a user-facing way to control this? Thanks! Oleg.
LD_LIBRARY_PATH in wrapper scripts
Hello there! We have an autotools-based build system setup with a custom GCC where we take all build- and run-time dependencies (except for glibc) from /opt. Things have worked well on Ubuntu 16, yet I'm hitting a funky issue when building on Ubuntu20 (libtool 2.4.6-14 according to dpkg). The issue comes down to one of the wrapper scripts that contains this gem: # Add our own library path to LD_LIBRARY_PATH LD_LIBRARY_PATH="/usr/lib/x86_64-linux-gnu:/home/oleg/project/_obj/.libs:/opt/gcc-11/lib/../lib64:$LD_LIBRARY_PATH" Most of our libs are statically linked with exception of just one. So tests/apps that use that shared lib end up with libtool wrappers... and they work correctly on Ubuntu16 (libtool 2.4.6-0.1 according to dpkg). It seems that libtool version just does not stamp out this extra variable... The manual fix is to remove the "/usr/lib/x86_64-linux-gnu" bit from the LD_LIBRARY_PATH above... and yet I have no idea where it came from or whether there is a way to influence its composition from a Makefile.am file. I'd appreciate any and all suggestions. Thanks in advance, Oleg.
Re: [EXTERNAL] Re: Unhelpful automatic 3rd-party library linkage
On Tue, Jun 29, 2021 at 3:30 PM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Tue, 29 Jun 2021, Oleg Smolsky wrote: > > > ...and I have figured out the source of the mystery linker flags: zmq > build > > leaves libzmq.la file which contains this: > > > > # Libraries that this one depends upon. > > dependency_libs=' -lrt -lpthread /opt/gcc-10/lib/../lib64/libstdc++.la' > > > > It looks like automake/libtool finds this file (BTW, when is it found?) > and > > transforms `-lzmq` into a whole bunch of things (with explicit .so names > > and dependencies)... > > Yes, that is part of the function of libtool. In fact (as you can > see), the libstdc++.la file was provided with the compiler. > > These are features that you may love or hate depending on what you are > doing. > Right, I use these features for tracking dependencies in our product build... yet it breaks things when a 3rd-party lib is shipped. My current solution is to strip the ".la" file from all automake/libtool-based 3rd-party libs that we build/ship. Is that resonable? > > [...snip...] > The compilation toolchain you are using is set up to not put its > libraries in the default system directories. As a result, the > libstc++.so.6 file needs to be found somehow. > > Indeed! And I do that with explicit -L and -Wl,--rpath options when I link the application. The 3rd-party libs use the same technique... but I don't rebuild 'em on every compiler upgrade (because I can get away with that). The arrangement only broke when the product build (which uses automake/libtool) found an ".la" file from a 3rd-party project and I moved between compiler versions... which to me looks like a leaky abstraction in this case. Oleg.
Re: [EXTERNAL] Re: Unhelpful automatic 3rd-party library linkage
On Tue, Jun 29, 2021 at 3:22 PM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Tue, 29 Jun 2021, Oleg Smolsky wrote: > > > > It looks like GCC9 references come from libzmq: > > > > $ ldd /opt/3p/lib/libzmq.so | grep libstd > >libstdc++.so.6 => /opt/gcc-9/lib/../lib64/libstdc++.so.6 > > (0x7f95f8d9f000) > > > > Obviously the 3rd-party library was built a while ago with GCC9. At the > > time it was linked to the compiler's runtime... but now the main > > application has moved to GCC11 and I'm linking to the runtime that is > > correct right now. > > > > It looks like automake/libtool try to be helpful and check the library's > > dependencies... but that gets in the way as the new libstdc++ is a strict > > superset of the old one. They maintain ABI compatibility and so scenarios > > like these are supported. > > Are you absolutely sure that the above is true? You specified c++17 > when compiling your application. Are the libstdc++ ABI's the same > across GCC versions and C++ language versions? > Well, I want to claim that I am absolutely sure :) My understanding is that there have been no ABI breaks in the GCC/libstdc++ ever (even noting the 5.x move to the Standard-compliant std::string). The general principle is to let people/distros upgrade gcc/libstdc++ in the OS and let the old apps continue running. > > Is there a way to suppress dependency tracking for the 3rd-party > libraries? > > I wish Libtool/automake was not trying to be smart and simply passed > > "-lzmq" directly to the linker. Yet instead, the actual .so file is > > discovered and then its libstdc++.so is linked. This is just wrong for > the > > scenario at hand. > > Assuming that the whole system does not have these directories in the > default search path (e.g. via ldconfig), it appears that this is a > recorded implicit dependency which is encoded in the library itself. > The only way to remove such an implicit dependency is to rebuild the > library (e.g. libzmq.so) with different options. > > If the persons who delivered the compilers to you expected that the > C++ library was truely reusable, then they would not have have put > everything under /opt/gcc-foo directories (also suggesting that these > directories are removable). Instead they would have put the C++ > run-time libraries in a standard system location. For example, under > Ubuntu Linux, I see that libstdc++.so.6 is at > /usr/lib/x86_64-linux-gnu/libstdc++.so.6 which is a common system > directory. > :) I am the person who maintains the compilers (installed into /opt/gcc-xx) and 3rd-party libs (installed into /opt/3p) at our shop. I don't care to update the system's compiler or libs as we don't use them at all. That is, our build system uses our compiler and only links to the 3rd-party dependencies from /opt. > As far as I am aware, there is no option to request that libtool > not perform the full linkage that it does. A common work-around is to > remove the ".la" files that libtool produces and installs. > > It is possible that GCC itself is pre-programmed (e.g. via the spec > file) to record this information when it links with the C++ standard > library. > Right, I figured this very point out just a couple hours ago - the extra flags/libs (along with the -lzmq transformation) come from the ".la" file. I've rebuilt the lib, purged the file and things look good now for my build. Could you shed some light on how this .la file is supposed to be used? I see that it tries to be helpful by capturing the dependencies... but it seems to destroy the standard `-lfoo` contract. IE it appears that it reduces the level of abstraction needlessly for artifacts that are distributed/stored. Is this ".la" thing meant only for build systems where the whole tree is built from scratch at the same time? Thanks! Oleg.
Re: Unhelpful automatic 3rd-party library linkage
...and I have figured out the source of the mystery linker flags: zmq build leaves libzmq.la file which contains this: # Libraries that this one depends upon. dependency_libs=' -lrt -lpthread /opt/gcc-10/lib/../lib64/libstdc++.la' It looks like automake/libtool finds this file (BTW, when is it found?) and transforms `-lzmq` into a whole bunch of things (with explicit .so names and dependencies)... On Tue, Jun 29, 2021 at 10:57 AM Oleg Smolsky wrote: > Hello there! I'm using automake+libtool for building C++ code and hitting > a peculiar issue with 3rd-party libraries. It comes up when a library is > built with one version of GCC and then the application is built with a > newer compler. > > So, given the following wrapper lib that pulls libzmq: > > noinst_LTLIBRARIES += libnsevent.la > nodist_libnsevent_la_SOURCES = libs/nsevent/wrapper.cpp > libnsevent_la_LIBADD = -lzmq > > Libtool results in the following g++ invocation when linking the > application: > > /opt/gcc-11/bin/g++ -std=c++17 -Wl,-rpath -Wl,/opt/gcc-11/lib64 > -Wl,-rpath -Wl,/opt/3p/lib -o ns_conn_test > libs/netsvc/test/ns_conn_test/src/cpp/NsConnTest.o -L/opt/3p/lib > ./.libs/libnsevent.a /opt/3p/lib/libzmq.so > /opt/gcc-9/lib/../lib64/libstdc++.so /opt/gcc-11/lib/../lib64/libstdc++.so > -lpthread -lrt -ldl -lm -lz -Wl,-rpath -Wl,/opt/3p/lib -Wl,-rpath > -Wl,/opt/gcc-9/lib/../lib64 -Wl,-rpath -Wl,/opt/gcc-11/lib/../lib64 > -Wl,-rpath -Wl,/opt/3p/lib -Wl,-rpath -Wl,/opt/gcc-9/lib/../lib64 > -Wl,-rpath -Wl,/opt/gcc-11/lib/../lib64 > > It looks like GCC9 references come from libzmq: > > $ ldd /opt/3p/lib/libzmq.so | grep libstd > libstdc++.so.6 => /opt/gcc-9/lib/../lib64/libstdc++.so.6 > (0x7f95f8d9f000) > > Obviously the 3rd-party library was built a while ago with GCC9. At the > time it was linked to the compiler's runtime... but now the main > application has moved to GCC11 and I'm linking to the runtime that is > correct right now. > > It looks like automake/libtool try to be helpful and check the library's > dependencies... but that gets in the way as the new libstdc++ is a strict > superset of the old one. They maintain ABI compatibility and so scenarios > like these are supported. > > Is there a way to suppress dependency tracking for the 3rd-party > libraries? I wish Libtool/automake was not trying to be smart and simply > passed "-lzmq" directly to the linker. Yet instead, the actual .so file is > discovered and then its libstdc++.so is linked. This is just wrong for the > scenario at hand. > > Thanks in advance, > Oleg. >
Unhelpful automatic 3rd-party library linkage
Hello there! I'm using automake+libtool for building C++ code and hitting a peculiar issue with 3rd-party libraries. It comes up when a library is built with one version of GCC and then the application is built with a newer compler. So, given the following wrapper lib that pulls libzmq: noinst_LTLIBRARIES += libnsevent.la nodist_libnsevent_la_SOURCES = libs/nsevent/wrapper.cpp libnsevent_la_LIBADD = -lzmq Libtool results in the following g++ invocation when linking the application: /opt/gcc-11/bin/g++ -std=c++17 -Wl,-rpath -Wl,/opt/gcc-11/lib64 -Wl,-rpath -Wl,/opt/3p/lib -o ns_conn_test libs/netsvc/test/ns_conn_test/src/cpp/NsConnTest.o -L/opt/3p/lib ./.libs/libnsevent.a /opt/3p/lib/libzmq.so /opt/gcc-9/lib/../lib64/libstdc++.so /opt/gcc-11/lib/../lib64/libstdc++.so -lpthread -lrt -ldl -lm -lz -Wl,-rpath -Wl,/opt/3p/lib -Wl,-rpath -Wl,/opt/gcc-9/lib/../lib64 -Wl,-rpath -Wl,/opt/gcc-11/lib/../lib64 -Wl,-rpath -Wl,/opt/3p/lib -Wl,-rpath -Wl,/opt/gcc-9/lib/../lib64 -Wl,-rpath -Wl,/opt/gcc-11/lib/../lib64 It looks like GCC9 references come from libzmq: $ ldd /opt/3p/lib/libzmq.so | grep libstd libstdc++.so.6 => /opt/gcc-9/lib/../lib64/libstdc++.so.6 (0x7f95f8d9f000) Obviously the 3rd-party library was built a while ago with GCC9. At the time it was linked to the compiler's runtime... but now the main application has moved to GCC11 and I'm linking to the runtime that is correct right now. It looks like automake/libtool try to be helpful and check the library's dependencies... but that gets in the way as the new libstdc++ is a strict superset of the old one. They maintain ABI compatibility and so scenarios like these are supported. Is there a way to suppress dependency tracking for the 3rd-party libraries? I wish Libtool/automake was not trying to be smart and simply passed "-lzmq" directly to the linker. Yet instead, the actual .so file is discovered and then its libstdc++.so is linked. This is just wrong for the scenario at hand. Thanks in advance, Oleg.
Re: Strange linker flags transformation
On Tue, Mar 16, 2021 at 3:29 PM Oleg Smolsky wrote: > /bin/ld: fatal error: /opt/3p/lib: pread failed: Is a directory > > Slight correction - this particular issue comes from a typo in a composite -Wl, argument. Not much to this mystery. Yet the trailer is puzzling...
Strange linker flags transformation
Hello, I'm experimenting with a different compiler setup and, hitting an issue with Libtool while tweaking linker flags. Here is what I am trying to invoke: /bin/bash ./libtool --tag=CC --mode=link /opt/gcc-10sr/bin/gcc -g -Wall -O2 -ljsoncpp -lmodpbase64 -lczmq -lzmq -lcrypto -lsqlite3 -Wl,--dynamic-linker,/opt/sysroot/lib64/ld-linux-x86-64.so.2 -Wl,-rpath,/opt/sysroot/lib64 -Wl,-rpath,/opt/gcc-10sr/lib64,/opt/3p/lib -L/opt/gcc-10sr/lib64 -L/opt/3p/lib -o xxx_test test/xxx_test/src/xxx_test.o libtask.la libutest.la -lpthread -lrt And here is what is actually invoked: libtool: link: /opt/gcc-10sr/bin/gcc -g -Wall -O2 -Wl,--dynamic-linker -Wl,/opt/sysroot/lib64/ld-linux-x86-64.so.2 -Wl,-rpath -Wl,/opt/sysroot/lib64 -Wl,-rpath -Wl,/opt/gcc-10sr/lib64 -Wl,/opt/3p/lib -o xxx_test test/xxx_test/src/xxx_test.o -ljsoncpp /opt/3p/lib/libczmq.so /opt/3p/lib/libzmq.so -lcrypto /opt/3p/lib/libsqlite3.so -L/opt/gcc-10sr/lib64 -L/opt/3p/lib ./.libs/libtask.a /opt/3p/lib/libmodpbase64.so /opt/gcc-10sr/lib/../lib64/libstdc++.so ./.libs/libutest.a -lpthread -lrt -Wl,-rpath -Wl,/opt/3p/lib -Wl,-rpath -Wl,/opt/gcc-10sr/lib/../lib64 -Wl,-rpath -Wl,/opt/3p/lib -Wl,-rpath -Wl,/opt/gcc-10sr/lib/../lib64 I am trying to understand where the trailer of that command is coming from (the stuff after -lrt). These linker flags are malformed. This specific example results in the following message: /bin/ld: fatal error: /opt/3p/lib: pread failed: Is a directory I am trying to understand how this transformation is made... but struggling. I get different errors depending on how the -Wl,xx bits are cut... I would appreciate any pointers. Thanks! Oleg.
Re: [EXTERNAL] Re: Q: library dependencies
On 2020-09-25 09:28, Bob Friesenhahn wrote: With convenience libraries, there may be a necessary build order but the object files are not 'linked' before going into the convenience libraries (as a proper library would be) so all linking is when the final library or program is linked using the objects from the convenience libraries. Hi Bob, how would I make a "proper" library? Would that change the composition logic for my DAG of dependencies? Oleg.
Re: [EXTERNAL] Re: Q: library dependencies
On Fri, Sep 25, 2020 at 6:58 AM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Thu, 24 Sep 2020, Oleg Smolsky wrote: > > Libtool convenience libraries are not "real" static libraries. > Instead they are object files stored in an archive file. Prior to use > (when linking using libtool) the objects are extracted and passed to > the linker directly (perhaps with renaming to assure there are no > over-writes during the extraction). Well, AFAIK any well-formed .a file (an archive) is a static lib. And these can be passed to the linker. Are you saying that libtool extracts the individual .o files instead passing -lfoo (for libfoo.a)? > While libtool convenience > libraries are certainly "convenient" they can dramatically slow down > the build. It takes time to do the operations associated with the ar > files, but most importantly it prohibits parallelism in the build. > Yes, this is certainly a concern. I've noticed stalls in my non-recursive build. I'm guessing that is due to repackaging the libs. > > > So, I am not sure what can be done here. Could you clarify the following > > please: > > - Does the aforementioned renaming bloat the executable? > > Bloating of the executable depends on the object files linked with it. > Linkers may vary as to how smart they are at ignoring supplied objects > where no symbols have been used. > > > - How is the linker able to resolve mutiply-defined symbols for the > > duplicated nodes in the DAG? > > The convenience library does not do anything regarding mutiply-defined > symbols (at the C language level) while linking. If there is a > conflict then the linker should normally fail. > I find the situation puzzling. My project has just over a hundred of these LT convenience libraries and there are several places where collisions result in renaming. Yet nowhere do I see multiply defined symbols. If libtool/automake scripts do not do dedup... I can only guess that the linker sees .o files inside multiple .a archives (or multiple .o files coming from distinct directories via the command line) and performs dedup. Is that getting closer to the truth? > > - Is there an alternative way to express inter-lib dependencies with > > automake/libtool? > > Using a non-recursive build and using Automake macros to express what > is currently expressed as convenence libraries (using original object > files in place rather than storing in 'ar' format and extracting over > and over) will tremendously decrease build times. Automake supports > an include syntax which allows distributing build information in the > project (e.g. putting it in subdirectories next to source files) yet > incorporating in single build. If everything is ruled by one > non-recursive 'make', and the dependencies are properly cascading, > then using Automake's non-recursive build capability will provide a > huge boost to build performance, and particulary on modern multicore > systems. > > My setup is a large, non-recursive thing with "include" statements. On the top of that I am trying to understand the issues and costs imposed by the convenience libraries. My primary ask is transitive dependencies between the libs as I have more than 100 of them (and several dozen apps and hundreds of tests). Thanks in advance! Oleg.
Re: [EXTERNAL] Re: Q: library dependencies
On Thu, Sep 24, 2020 at 12:17 PM Nick Bowler wrote: > On 2020-09-24, Oleg Smolsky wrote: > > Hello, I'm working on an autotools-based build system and finally > decided to > > look into these messages: > > > > CXXLDlibtop.la > > copying selected object files to avoid basename conflicts... > > > > It looks like libtool links (copies?) .o files when it finds a dupe in > the > > dependency DAG. Here is the situation I have w.r.t. the dependencies: > > > > libtop.la: liba.la, libx.la > > liba.la: libb.la libc.la libx.la > > > > All these libraries are static, noinst_LILIBRARIES and the dependencies > are > > expressed via _LIBADD syntax: > > > > libtop_la_LIBADD = liba.la libx.la > > > > I do this to get transitive dependencies where the top-level executable > just > > needs to link to link libtop.la. > > > > I think the issue is with libx.la as it is present in two different > branches > > of the dependency DAG. This kind of thing is common in large projects and > > seems strange that libtool is trying to be smart about it. > > > > Is there a way to tell automake/libtool to disable this copying and just > do > > straightforward linking? I wonder if the current behavior results in > > unnecessary bloat?.. Or perhaps I am saved by the linker that would > dedup? > > Unless I misunderstand something, what you describe is the documented > and expected behaviour. > > When using primaries such as noinst_LTLIBRARIES, Automake will generate > rules that create libtool convenience libraries. Such libraries are > basically intended to be a convenient shorthand for directly listing > the libtool objects that make up the convenience library whenever you > use libtool to link with it. > > So when you create another libtool convenience library, liba.la, linking > with libx.la, since libx.la is a shorthand for directly listing all > its objects this creates a new library that contains all the objects > that made up libx.la. > > Now when creating the libtop.la convenience library, you link against > _both_ convenience libraries liba.la and libx.la. This directly includes > all the objects of liba.la and all the objects of libx.la. But as the > objects of libx.la are duplicated in liba.la, this means some objects > are to be included twice into libtop.la. Libtool has to rename one of > the objects to make this work (hence the "copying ... to avoid basename > conflicts"). > Right, that's the behavior I am seeing. Thank you for confirming! > > I assume this is not the desired result. Probably simplest to just > arrange for libx.la to either be a dependency of libtop.la OR liba.la, > but not both. > > Well, both libtop.la and liba.la use the lower-level lib and I want to support all users (as I have many paths through the dependency DAG). I provided a reduced/simplified example as it is representative of my codebase - the dependencies have many collisions, sometimes with more steps in between (but no cycles). The thing here is that there are many libraries and I've expressed the dependencies with via _LIBADD. I would like to claim that these are both (syntactically) correct and (logically) sound. These dependencies serve a useful purpose: any user can link to any library and will then get the whole list of transitive dependencies. This is a boon when the DAG is large and complex. (IIRC CMake works it out correctly) So, I am not sure what can be done here. Could you clarify the following please: - Does the aforementioned renaming bloat the executable? - How is the linker able to resolve mutiply-defined symbols for the duplicated nodes in the DAG? - Is there an alternative way to express inter-lib dependencies with automake/libtool? Just to make it clear, I understand that my current dependency DAG can be turned into a tree for some cases. I could also eliminate the dependencies altogether and force it upon the programs to express the whole dependency set. The problem with that is dependency leakage - an addition to a lower-level lib negatively effects every single direct and indirect user. Really uncool in terms of maintenance. Thanks! Oleg.
Q: library dependencies
Hello, I'm working on an autotools-based build system and finally decided to look into these messages: CXXLD libtop.la copying selected object files to avoid basename conflicts... It looks like libtool links (copies?) .o files when it finds a dupe in the dependency DAG. Here is the situation I have w.r.t. the dependencies: libtop.la: liba.la, libx.la liba.la: libb.la libc.la libx.la All these libraries are static, noinst_LILIBRARIES and the dependencies are expressed via _LIBADD syntax: libtop_la_LIBADD = liba.la libx.la I do this to get transitive dependencies where the top-level executable just needs to link to link libtop.la. I think the issue is with libx.la as it is present in two different branches of the dependency DAG. This kind of thing is common in large projects and seems strange that libtool is trying to be smart about it. Is there a way to tell automake/libtool to disable this copying and just do straightforward linking? I wonder if the current behavior results in unnecessary bloat?.. Or perhaps I am saved by the linker that would dedup? Could someone clarify how this is supposed to work for large projects please? Thanks! Oleg.
