[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 Jan Hubicka hubicka at gcc dot gnu.org changed: What|Removed |Added CC||jason at redhat dot com, ||mark at codesourcery dot ||com --- Comment #21 from Jan Hubicka hubicka at gcc dot gnu.org 2011-01-05 13:36:37 UTC --- I am re-building now. Martin's edge cgraph_opt streaming fix is needed and flag_shlib needs to be set in lto-options.c With this fixed oprofile shows that cc1plus spends a lot of time in lookup_filed_1. 40259 5.6000 cc1plus cc1plus lookup_field_1 20275 2.8203 cc1plus cc1plus longest_match 15843 2.2038 libc-2.11.1.so libc-2.11.1.so _int_malloc 12409 1.7261 libc-2.11.1.so libc-2.11.1.so memset 10680 1.4856 cc1plus cc1plus htab_find_slot_with_hash 10471 1.4565 libc-2.11.1.so libc-2.11.1.so vfprintf 9004 1.2525 cc1plus cc1plus deflate_slow 8580 1.1935 cc1plus cc1plus ggc_internal_alloc_stat 8300 1.1545 libc-2.11.1.so libc-2.11.1.so memcpy 8100 1.1267 cc1plus cc1plus ht_lookup_with_hash 8044 1.1189 libpython2.6.so.1.0 libpython2.6.so.1.0 /usr/lib64/libpython2.6.so.1.0 7840 1.0905 cc1plus cc1plus _cpp_lex_direct 6340 0.8819 cc1plus cc1plus pointer_set_insert I am adding c++ maintainers to CC as this seems like relatively low hanging fruit for noticeable compilation speedup? It tends to show in oprofile as 5-7% of compile time.
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 --- Comment #22 from Mark Mitchell mark at codesourcery dot com 2011-01-06 03:55:40 UTC --- On 1/5/2011 5:36 AM, hubicka at gcc dot gnu.org wrote: 40259 5.6000 cc1plus cc1plus lookup_field_1 I've looked at this, in the distant past. I don't think the routine itself is *very* low-hanging fruit; it's already using an inline log n algorithm to find a field in most cases, and I bet that's as good as a hash table since n is generally relatively small. But, maybe in most cases is wrong; there is a slow-path, and we should confirm that most of the time is in the fast-path code. We could also try a bit of memoization; I wouldn't be surprised if we often lookup x.y several times in a row. More often, when I've looked at this kind of thing, though, I've concluded that the problem was that we were calling the routine too often, rather than the routine itself was too slow. Quite possibly we could improve algorithms that are using lookup_field_1 so that they didn't do so as often, by building caches or otherwise. For that, we'd need to look at the callers of lookup_field_1. So, in summary, I'd recommend three things: * Split lookup_field_1 into its fast-path and slow-path code so that we can profile it and figure out which code is taking up most of the time. * Assuming it's fast-path code, look at the frequent callers and think about how to optimize them.
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 --- Comment #20 from H.J. Lu hjl.tools at gmail dot com 2010-12-17 22:25:56 UTC --- (In reply to comment #19) Filled in the GNU LD bug as http://sourceware.org/bugzilla/show_bug.cgi?id=12323 It should have been fixed on hjl/lto-mixed branch at http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 Jan Hubicka hubicka at gcc dot gnu.org changed: What|Removed |Added Depends on||46940 --- Comment #18 from Jan Hubicka hubicka at gcc dot gnu.org 2010-12-14 15:36:47 UTC --- Filled in the sefault as PR46940 It is really a sickness of mozilla sources definint _INT symbol, _moz symbol and function of same name and visibility and using both. In any case we should handle this gratefully too. Honza
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 --- Comment #19 from Jan Hubicka hubicka at gcc dot gnu.org 2010-12-15 00:44:25 UTC --- Filled in the GNU LD bug as http://sourceware.org/bugzilla/show_bug.cgi?id=12323
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 --- Comment #17 from Jan Hubicka hubicka at gcc dot gnu.org 2010-12-12 23:52:52 UTC --- Current mainline crashes: Program received signal SIGSEGV, Segmentation fault. lto_cgraph_replace_node (slot=value optimized out, data=value optimized out) at ../../gcc/lto-symtab.c:227 227 if (prevailing_node-same_body_alias) (gdb) bt #0 lto_cgraph_replace_node (slot=value optimized out, data=value optimized out) at ../../gcc/lto-symtab.c:227 #1 lto_symtab_merge_cgraph_nodes_1 (slot=value optimized out, data=value optimized out) at ../../gcc/lto-symtab.c:798 #2 0x00b0ae08 in htab_traverse_noresize (htab=value optimized out, callback=0x60eca0 lto_symtab_merge_cgraph_nodes_1, info=0x0) at ../../libiberty/hashtab.c:784 #3 0x004aabf9 in read_cgraph_and_symbols () at ../../gcc/lto/lto.c:2213 #4 lto_main () at ../../gcc/lto/lto.c:2438 #5 0x006cb658 in compile_file (argc=2627, argv=0x11a7460) at ../../gcc/toplev.c:579 #6 do_compile (argc=2627, argv=0x11a7460) at ../../gcc/toplev.c:1874 #7 toplev_main (argc=2627, argv=0x11a7460) at ../../gcc/toplev.c:1937 #8 0x76597bc6 in __libc_start_main () from /lib64/libc.so.6 #9 0x00493411 in _start () at ../sysdeps/x86_64/elf/start.S:113 I guess it is fallout of the merging patch. It is weird since previaling_node is NULL. _moz_cairo_surface_destroy/567259(-1) @0x7ffebef47c60 (asm: _moz_cairo_surface_destroy) visibilit: 2 binds_local called by: CreateSimilarSurface/567227 (0.21 per call) CreateSimilarSurface/567227 (0.14 per call) Init/567225 (0.39 per call) _ZN11gfxASurface7ReleaseEv.part.2/567209 (1.00 per call) calls: References: Refering this function: $5 = void I also generated profile. samples %image name app name symbol name 228038 25.3225 lto1 lto1 htab_find_slot_with_hash 82588 9.1710 lto1 lto1 iterative_hash_hashval_t 58000 6.4406 lto1 lto1 type_pair_eq 32557 3.6153 lto1 lto1 gimple_lookup_type_leader 31622 3.5115 lto1 lto1 gtc_visit 29149 3.2369 lto1 lto1 htab_expand 27463 3.0496 lto1 lto1 gimple_type_hash_1 24348 2.7037 lto1 lto1 gimple_types_compatible_p 24217 2.6892 lto1 lto1 inflate_fast 21984 2.4412 lto1 lto1 gimple_types_compatible_p_1 21796 2.4203 libc-2.11.1.so libc-2.11.1.so memset 21700 2.4097 libc-2.11.1.so libc-2.11.1.so _int_malloc 17894 1.9870 lto1 lto1 lookup_type_pair.isra.120.constprop.129 16087 1.7864 lto1 lto1 ggc_set_mark 15719 1.7455 lto1 lto1 gt_ggc_mx_lang_tree_node Our abuse of hashing is making us slow. It is not only type merging but all the hashing during streaming in.
Re: [Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
Yeh, precisely. The ironly file is a placeholder into which we put the symbols found in the lto symtab so that they can take part in the link and their resolutions be determined. We have no way of conveying any symbol type The error comes out after the lto1 invocation, so why the ironly section is still around? I would expect it to be discarded at that time and replaced by whatever compiler returns to you. On the other hand, discarding won't help if there was non-LTO module referencing TLS var also used by LTO module I guess.
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 --- Comment #13 from Jan Hubicka hubicka at ucw dot cz 2010-12-02 08:47:28 UTC --- Yeh, precisely. The ironly file is a placeholder into which we put the symbols found in the lto symtab so that they can take part in the link and their resolutions be determined. We have no way of conveying any symbol type The error comes out after the lto1 invocation, so why the ironly section is still around? I would expect it to be discarded at that time and replaced by whatever compiler returns to you. On the other hand, discarding won't help if there was non-LTO module referencing TLS var also used by LTO module I guess.
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 --- Comment #14 from Dave Korn davek at gcc dot gnu.org 2010-12-02 08:52:20 UTC --- (In reply to comment #13) Yeh, precisely. The ironly file is a placeholder into which we put the symbols found in the lto symtab so that they can take part in the link and their resolutions be determined. We have no way of conveying any symbol type The error comes out after the lto1 invocation, so why the ironly section is still around? I would expect it to be discarded at that time and replaced by whatever compiler returns to you. It's the symbol from the ironly section that remains, and it gets discarded and replaced by the the symbol from the real object file by the linker multiple_definition callback hook when _bfd_generic_link_add_one_symbol is called to add the symbol from the real object file into the link hash table. Unfortunately, the elf linker has some additional checking that it does before calling that routine which preemptively complains about the multiple definition before the linker hook has a chance to replace the original ironly symbol by the new one.
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 --- Comment #15 from Richard Guenther rguenth at gcc dot gnu.org 2010-12-02 09:41:58 UTC --- (In reply to comment #10) I am just trying to get Mozilla building with GNU ld instead of gold. First problem is that Mozilla links some of libraries as: /abuild/jh/trunk-install/bin/gcc -O3 -flto -flto-partition=none -fuse-linker-plugin -shared -Wl,-soname -Wl,libplds4.so -o libplds4.so ./plarena.o ./plhash.o ./plvrsion.o -L/abuild/jh/build-mozilla-new7/dist/lib -lnspr4 i.e. there is missing -fPIC that means that we compile into non-PIC code and GNU LD eventually complains about PC32 relocations into symbols that can be overwritten. Is this valid? If so, we need to work out -fPIC ourselves at LTO time It's valid I think and we try to work out fPIC ourselves in the funny LTO option handling code (but the options are not re-applied at ltrans stage I think, so it doesn't work at all with WHOPR). Richard. Honza
Re: [Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
It's valid I think and we try to work out fPIC ourselves in the funny LTO option handling code (but the options are not re-applied at ltrans stage I think, so it doesn't work at all with WHOPR). Hmm, the link command above is LTO, not WHOPR. I wonder why we don't work out -fPIC ourselves then... Honza
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 --- Comment #16 from Jan Hubicka hubicka at ucw dot cz 2010-12-02 15:34:48 UTC --- It's valid I think and we try to work out fPIC ourselves in the funny LTO option handling code (but the options are not re-applied at ltrans stage I think, so it doesn't work at all with WHOPR). Hmm, the link command above is LTO, not WHOPR. I wonder why we don't work out -fPIC ourselves then... Honza
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 --- Comment #10 from Jan Hubicka hubicka at gcc dot gnu.org 2010-12-01 23:58:30 UTC --- I am just trying to get Mozilla building with GNU ld instead of gold. First problem is that Mozilla links some of libraries as: /abuild/jh/trunk-install/bin/gcc -O3 -flto -flto-partition=none -fuse-linker-plugin -shared -Wl,-soname -Wl,libplds4.so -o libplds4.so ./plarena.o ./plhash.o ./plvrsion.o-L/abuild/jh/build-mozilla-new7/dist/lib -lnspr4 i.e. there is missing -fPIC that means that we compile into non-PIC code and GNU LD eventually complains about PC32 relocations into symbols that can be overwritten. Is this valid? If so, we need to work out -fPIC ourselves at LTO time Honza
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 Jan Hubicka hubicka at gcc dot gnu.org changed: What|Removed |Added CC||dave.korn.cygwin at gmail ||dot com --- Comment #11 from Jan Hubicka hubicka at gcc dot gnu.org 2010-12-02 00:36:44 UTC --- OK, working around the previous issues we fail with: /abuild/jh/trunk-install/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/../../../../x86_64-unknown-linux-gnu/bin/ld: gTLSIsMainThread: TLS reference in /tmp/cczRYvg1.ltrans0.ltrans.o mismatches non-TLS definition in nsThreadManager.o.ironly section .text Dave, is this a GNU LD bug? It seems to me that most likely that nsThreadManager.o.ironly section is the one got from lto plugin and we don't put TLS annotations there because we have no way to do so? Honza
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 Dave Korn davek at gcc dot gnu.org changed: What|Removed |Added CC||davek at gcc dot gnu.org --- Comment #12 from Dave Korn davek at gcc dot gnu.org 2010-12-02 01:03:43 UTC --- (In reply to comment #11) OK, working around the previous issues we fail with: /abuild/jh/trunk-install/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/../../../../x86_64-unknown-linux-gnu/bin/ld: gTLSIsMainThread: TLS reference in /tmp/cczRYvg1.ltrans0.ltrans.o mismatches non-TLS definition in nsThreadManager.o.ironly section .text Dave, is this a GNU LD bug? It seems to me that most likely that nsThreadManager.o.ironly section is the one got from lto plugin and we don't put TLS annotations there because we have no way to do so? Yeh, precisely. The ironly file is a placeholder into which we put the symbols found in the lto symtab so that they can take part in the link and their resolutions be determined. We have no way of conveying any symbol type info. We'll need to handle this in the multiple-def linker hook in LD's plugin code, by getting it to copy type info from the newly-added symbols to the ironly ones. Oh, hang on, that won't work. elflink.c calls _bfd_elf_merge_symbol /before/ _bfd_generic_link_add_one_symbol, which is where the multiple-def hook gets called back from. So it'll error on the mismatch before we get a chance to do anything about it. That's awkward. Need to scratch my head over that for a bit.
[Bug lto/45375] [meta-bug] Issues with building Mozilla with LTO
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375 Jan Hubicka hubicka at gcc dot gnu.org changed: What|Removed |Added Summary|[meta-bug] Mozilla does not |[meta-bug] Issues with |build with LTO |building Mozilla with LTO --- Comment #9 from Jan Hubicka hubicka at gcc dot gnu.org 2010-10-18 20:48:03 UTC --- Updated summary, Mozilla now builds with unpatched mainline (with checking disabled)
