[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
https://sourceware.org/bugzilla/show_bug.cgi?id=15478 Fangrui Song changed: What|Removed |Added CC||i at maskray dot me --- Comment #15 from Fangrui Song --- This can properly be closed as an intended behavior. GNU ld since 2.22 has defaulted to --no-copy-dt-needed-entries. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #1 from Cary Coutant 2013-05-16 23:51:51 UTC --- > situation (see attached tar.bz2 to reproduce): > libmylib.so has unresolved symbols that are found in libmyplugin.so > myapp.c++ calls into libmylib.so > myapp.c++ is being compiled with -lmylib and -lmyplugin > > expected behaviour, and behaviour with gnu ld: > myapp is linked against mylib and myplugin > > observed behaviour: > myapp is only linked against mylib since it does not make direct calls into > myplugin > myapp is not executable (fails with message about myplugin symbols not being > resolved in mylib) > > workaround: > link with -no-as-needed > > Can you comment on this observed behaviour? thanks I think this is intended behavior for gold. It's expected that each library will have its own dependencies recorded so that we only record direct dependencies in the dynamic table. In your case, since libmylib.so has references to libmyplugin.so, there should be a DT_NEEDED entry in libmylib.so for libmyplugin.so. If you link libmylib.so with -L. -lmyplugin, it should work. -cary -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #2 from Dave Cunningham 2013-05-17 01:46:41 UTC --- > I think this is intended behavior for gold. It's expected that each > library will have its own dependencies recorded so that we only record > direct dependencies in the dynamic table. In your case, since > libmylib.so has references to libmyplugin.so, there should be a > DT_NEEDED entry in libmylib.so for libmyplugin.so. If you link > libmylib.so with -L. -lmyplugin, it should work. Yes that mode of operation does work but is unfortunately not an option for us as we would like to combine libmylib.so with a different API-compatible libmyplugin.so in each of a whole range of myapp executables. To resolve the symbol in the libmylib.so build would mean building a number of libmylib.so (one for each libmyplugin.so) and this would make our build take much more time, as well as inflating the size of our releases. It seems that it should be fairly easy for gold to broaden its 'as-needed' criteria by scanning for unresolved symbols in the .so files (as well as in the .o files) being linked. That would seem to make the behaviour compatible with the old behaviour of the old linker. If that's not possible then -no-as-needed should be the default I think. -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #3 from Cary Coutant 2013-05-17 04:50:56 UTC --- > Yes that mode of operation does work but is unfortunately not an option for us > as we would like to combine libmylib.so with a different API-compatible > libmyplugin.so in each of a whole range of myapp executables. To resolve the > symbol in the libmylib.so build would mean building a number of libmylib.so > (one for each libmyplugin.so) and this would make our build take much more > time, as well as inflating the size of our releases. > > It seems that it should be fairly easy for gold to broaden its 'as-needed' > criteria by scanning for unresolved symbols in the .so files (as well as in > the > .o files) being linked. That would seem to make the behaviour compatible with > the old behaviour of the old linker. I think this is a behavior of the Gnu linker that gold is better off not copying. The linker ends up having to replicate the path searching done at runtime by the dynamic loader, and that works only when the libraries are in their installed locations. Making that work at development time leads to all kinds of unexpected behavior. > If that's not possible then -no-as-needed should be the default I think. It is the default in gold, but GCC recently started passing --as-needed to the linker by default. -cary -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #4 from Dave Cunningham 2013-05-17 15:58:14 UTC --- > I think this is a behavior of the Gnu linker that gold is better off > not copying. The linker ends up having to replicate the path searching > done at runtime by the dynamic loader, and that works only when the > libraries are in their installed locations. Making that work at > development time leads to all kinds of unexpected behavior. I don't see any need for additional searching. The linker already has to search for the library through the paths specified by -L, and it needs to read the library to ensure that unresolved symbols in the .o files can be bound in the libraries. So all the information is already available. -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #5 from Cary Coutant 2013-05-17 18:48:34 UTC --- >> I think this is a behavior of the Gnu linker that gold is better off >> not copying. The linker ends up having to replicate the path searching >> done at runtime by the dynamic loader, and that works only when the >> libraries are in their installed locations. Making that work at >> development time leads to all kinds of unexpected behavior. > > I don't see any need for additional searching. The linker already has to > search for the library through the paths specified by -L, and it needs to read > the library to ensure that unresolved symbols in the .o files can be bound in > the libraries. So all the information is already available. In this specific case, yes, no searching is necessary, but in the general case, if we're going to try to resolve undefined symbols in shared libraries, we need to process the dependencies of those shared libraries, which means trying to find them via the embedded rpaths in those libraries. -cary -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #6 from Dave Cunningham 2013-05-17 19:45:56 UTC --- > In this specific case, yes, no searching is necessary, but in the > general case, if we're going to try to resolve undefined symbols in > shared libraries, we need to process the dependencies of those shared > libraries, which means trying to find them via the embedded rpaths in > those libraries. So it seems in the general case, if you have an undefined symbol in a library, you cannot be sure from where it is intended to be satisfied. If it is a symbol that is *not* provided by any other library on the linker commandline, then it's a reasonable assumption that it should be satisfied externally (i.e. via dynamic linking). So then it can safely be ignored. But if that symbol is present in a file on the linker commandline then that seems like a pretty strong hint that it is intended to be satisfied using that specific arrangement. The problem I have with this is that -as-needed is second-guessing the build system. If a library is not needed, the build system can choose not to add it to the linker commandline. So this really ought to be a transparent optimisation, which means it should use the most precise heuristics available and err on the side of caution where merited. In this case, erring on the side of caution means deciding that a library is indeed needed. This seems to be the policy of the old implementation. If it's not intended to be a transparent optimisation, then it is clearly only applicable in specific cases and should not therefore be the default behaviour of gcc. But gcc may be assuming that -as-needed behaves like the old ld and is therefore safe to turn on by default. -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #7 from Ian Lance Taylor 2013-05-18 18:10:26 UTC --- Rather than make this rather complex behaviour even more complex in order to satisfy your rather unusual use case, I suggest that you use the --allow-shlib-undefined option. -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #8 from Dave Cunningham 2013-05-19 23:26:42 UTC --- There doesn't seem to be any relationship between --allow-shlib-undefined and --as-needed. There is no undefined symbol in the link job, everything is satisfied by the libraries on the commandline and gold does not complain about any undefined symbol. The problem is that gold decides not to add DT_NEEDED for one of the libraries because it wrongly believes that it does not need to. This yields an error when the generated binary is executed and the symbols are resolved at runtime (unless one uses LD_PRELOAD to add the library that gold missed out). Turning off --as-needed seems to be a fine workaround (gcc adds it by default, which is half the problem). If you plan to only support a weaker version of --as-needed than is supported in the previous binutils, I think it would be appropriate to rename the commandline parameter to avoid compatability issues. At least change the documentation to be more specific about what 'needed' means in this version compared to previously. However, I don't think you have any compelling reason to ignore symbols in the .so files on the commandline, so the easiest option is to be backwards-compatible. Here is the documentation from older binutils which talks about --as-needed interpreting undefined symbols both .o and .so files: --as-needed --no-as-needed This option affects ELF DT_NEEDED tags for dynamic libraries mentioned on the command line after the --as-needed option. Normally the linker will add a DT_NEEDED tag for each dynamic library mentioned on the command line, regardless of whether the library is actually needed or not. --as-needed causes a DT_NEEDED tag to only be emitted for a library that satisfies an undefined symbol reference from a regular object file or, if the library is not found in the DT_NEEDED lists of other libraries linked up to that point, an undefined symbol reference from another dynamic library. --no-as-needed restores the default behaviour. The gold documentation is very ambiguous: --as-needed Only set DT_NEEDED for shared libraries if used -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #9 from Cary Coutant 2013-05-20 16:46:00 UTC --- > Here is the documentation from older binutils which talks about --as-needed > interpreting undefined symbols both .o and .so files: > >--as-needed >--no-as-needed >This option affects ELF DT_NEEDED tags for >dynamic libraries mentioned on the command line >after the --as-needed option. Normally the >linker will add a DT_NEEDED tag for each dynamic >library mentioned on the command line, >regardless of whether the library is actually >needed or not. --as-needed causes a DT_NEEDED >tag to only be emitted for a library that >satisfies an undefined symbol reference from a >regular object file or, if the library is not >found in the DT_NEEDED lists of other libraries >linked up to that point, an undefined symbol >reference from another dynamic library. >--no-as-needed restores the default behaviour. That is rather explicit, isn't it? For gold to match this behavior, I think all we need to do is remove the test for sym->in_reg() in the following code in Symbol_table::set_dynsym_indexes: // If the symbol is defined in a dynamic object and is // referenced strongly in a regular object, then mark the // dynamic object as needed. This is used to implement // --as-needed. if (sym->is_from_dynobj() && sym->in_reg() && !sym->is_undef_binding_weak()) sym->object()->set_is_needed(); Ian, what do you think? -cary -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478
--- Comment #10 from Cary Coutant 2013-05-20
17:41:58 UTC ---
> That is rather explicit, isn't it? For gold to match this behavior, I
> think all we need to do is remove the test for sym->in_reg() in the
> following code in Symbol_table::set_dynsym_indexes:
>
> // If the symbol is defined in a dynamic object and is
> // referenced strongly in a regular object, then mark the
> // dynamic object as needed. This is used to implement
> // --as-needed.
> if (sym->is_from_dynobj()
> && sym->in_reg()
> && !sym->is_undef_binding_weak())
>sym->object()->set_is_needed();
Close, but not quite. If the symbol isn't referenced from the main
program, we won't be adding a dynsym entry for it, so the test for
as-needed needs to be copied to the if clause just above:
diff --git a/gold/symtab.cc b/gold/symtab.cc
index 2e17529..595df56 100644
--- a/gold/symtab.cc
+++ b/gold/symtab.cc
@@ -2381,7 +2381,17 @@ Symbol_table::set_dynsym_indexes(unsigned int index,
// and without a version.
if (!sym->should_add_dynsym_entry(this))
- sym->set_dynsym_index(-1U);
+ {
+ sym->set_dynsym_index(-1U);
+ // If the symbol is defined in a dynamic object and is
+ // referenced strongly in a dynamic object, then mark the
+ // dynamic object as needed. This is used to implement
+ // --as-needed. This is for compatibility with the GNU
+ // linker.
+ if (sym->is_from_dynobj()
+ && !sym->is_undef_binding_weak())
+ sym->object()->set_is_needed();
+ }
else if (!sym->has_dynsym_index())
{
sym->set_dynsym_index(index);
I've added a test case for this, too. Ian, if you think this is
reasonable, I'll go ahead and commit it.
-cary
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are on the CC list for the bug.
___
bug-binutils mailing list
bug-binutils@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478
--- Comment #11 from Cary Coutant 2013-05-20
21:43:58 UTC ---
> diff --git a/gold/symtab.cc b/gold/symtab.cc
> index 2e17529..595df56 100644
> --- a/gold/symtab.cc
> +++ b/gold/symtab.cc
> @@ -2381,7 +2381,17 @@ Symbol_table::set_dynsym_indexes(unsigned int index,
>// and without a version.
>
>if (!sym->should_add_dynsym_entry(this))
> - sym->set_dynsym_index(-1U);
> + {
> + sym->set_dynsym_index(-1U);
> + // If the symbol is defined in a dynamic object and is
> + // referenced strongly in a dynamic object, then mark the
> + // dynamic object as needed. This is used to implement
> + // --as-needed. This is for compatibility with the GNU
> + // linker.
> + if (sym->is_from_dynobj()
> + && !sym->is_undef_binding_weak())
> + sym->object()->set_is_needed();
> + }
>else if (!sym->has_dynsym_index())
> {
> sym->set_dynsym_index(index);
>
> I've added a test case for this, too. Ian, if you think this is
> reasonable, I'll go ahead and commit it.
Still not quite there. This patch breaks --as-needed completely,
always marking every shared library as needed.
The more I try to make this work and think about the complications,
the more I think gold is already doing it right. Gold doesn't
currently track whether a symbol defined in a shared object is
referenced by a different shared object. Even if it did, we'd have to
keep track of all such references and do a transitive closure on each
reference to make sure that the reference comes from a "needed"
library before marking the new library as "needed". I'm now retreating
to my earlier position that this is not something we want gold to
support. We should be encouraging proper program structure where each
load module declares its direct dependencies, and only its direct
dependencies.
I guess that means we should document somewhere what the difference is
between Gnu ld and gold. I don't think changing the name of the option
is right, because it's a subtle difference that arguably affects only
programs whose dependencies are already improperly recorded. But since
Gnu ld does explicitly cover this particular case, we should probably
say something. We don't have a gold manual, nor do we have a document
that lists the differences between Gnu ld and gold -- we should have
at least one of those, but until we do, should we just edit the help
text? I'm not sure how to describe this difference in an economical
way that will fit in the help text.
-cary
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are on the CC list for the bug.
___
bug-binutils mailing list
bug-binutils@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #12 from Dave Cunningham 2013-05-21 00:27:30 UTC --- I guess you need to do a full mark/sweep. It's similar to garbage collection, except instead of heap objects you have .so files, the .o files are the stack, and the symbols are the fields of objects (or variables on the stack). So you'd have to start from each .o file, and scan it looking for symbols satisfied by .so files, then if you touch a .so file that hasn't been marked yet, you mark it and recursively being again with that .so file as a base. Anything that ends up not marked is not 'needed'. The algorithm sounds pretty simple but if you're not recording information about links between .so files then that would be an issue. Maybe you can communicate with the gcc people and get them to not use this flag by default? The documentation I pasted was from man pages, not sure if those are from upstream or added by Ubuntu. -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #13 from Ian Lance Taylor 2013-05-21 04:18:43 UTC --- As far as I can see plain GCC only uses --as-needed in the form --as-needed -lgcc_s --no-as-needed If your GCC is always passing --as-needed to ld I think that must be Debian- or Ubuntu-specific. I do see that /usr/bin/gcc always passes --as-needed on my Ubuntu Precise system, and that it does not do so on Fedora 16. -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
[Bug gold/15478] -no-as-needed required to avoid runtime symbol lookup error
http://sourceware.org/bugzilla/show_bug.cgi?id=15478 --- Comment #14 from Dave Cunningham 2013-05-21 15:31:17 UTC --- OK, so here are the options, in decreasing order of my personal preference :) 1) Do the graph traversal to make --as-needed compatible with GNU ld 2) Rename --as-needed to --as-needed-fast, keep -as-needed as a no-op for compatability 3) Document --as-needed as having different semantics to GNU ld, and advise Ubuntu/Debian to stop patching gcc to use it by default (at least when using gold). -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ bug-binutils mailing list bug-binutils@gnu.org https://lists.gnu.org/mailman/listinfo/bug-binutils
