On Fri, Apr 21, 2023 at 2:16 PM John DelSignore <jdelsign...@perforce.com> wrote:
> > On 4/21/23 16:36, David Blaikie wrote: > > On Fri, Apr 21, 2023 at 12:44 PM John DelSignore <jdelsign...@perforce.com> > wrote: > >> Well, it took a long time to compile 5 CUs that contained your test code, >> and things were looking promising, but the link failed: >> >> rocm2 42 04/21 15:14 /build/jdelsign/fatty % make >> g++ -g -c fatty4.cxx -o fatty4.o >> g++ -g -c fatty5.cxx -o fatty5.o >> g++ -g -o fatty fatty.o fatty2.o fatty3.o fatty4.o fatty5.o >> fatty5.o:(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 >> against `.debug_info' >> collect2: error: ld returned 1 exit status >> make: *** [Makefile:5: fatty] Error 1 >> rocm2 43 04/21 15:39 /build/jdelsign/fatty % >> >> I guess I'm now in favor of the proposal to get rid of .debug_aranges. :-) >> > I guess, backing up - what's your goal/what're you trying to do with DWARF > over 4GB? > > We have a TotalView user that has a gigantic executable (~9GB) and the > .debug_info section alone is 4.9GB. A few of the other sections are about > 1GB, but under the 32-bit limit. That's about all I know, because the user > is at a secure location and cannot share the executable or other details. > > TotalView had a built-in limit of 4GB for DWARF sections. Recently, I > increased TV's DWARF section sizes to 64-bits to handle this user's code, > but I'm sure there are other changes that are needed to properly handle a > .debug_info section that is that big. So, I'm looking for a test case to > feed to TotalView. > Makes sense > I changed "fatty" to compile with clang++ (instead of g++), and that > worked. I have the following section sizes: > > rocm2 55 04/21 16:40 /build/jdelsign/fatty % readelf -SW fatty|grep debug > [30] .debug_abbrev PROGBITS 0000000000000000 002948 000670 > 00 0 0 1 > [31] .debug_info PROGBITS 0000000000000000 002fb8 15e82990e > 00 0 0 1 > [32] .debug_str_offsets PROGBITS 0000000000000000 15e82c8c6 > c0db3e8 00 0 0 1 > [33] .debug_str PROGBITS 0000000000000000 16a907cae > 217ee2aa 01 MS 0 0 1 > [34] .debug_addr PROGBITS 0000000000000000 18c0f5f58 000050 > 00 0 0 1 > [35] .debug_line PROGBITS 0000000000000000 18c0f5fa8 0001dd > 00 0 0 1 > [36] .debug_line_str PROGBITS 0000000000000000 18c0f6185 00004c > 01 MS 0 0 1 > rocm2 56 04/21 16:41 /build/jdelsign/fatty % > > That looks like a decent test case (I haven't tried TotalView on it yet), > so thanks for the suggestion. > > > You do have to use DWARF64 for a .debug_info section over 4GB - for any > section-relative reference in that section, such as cross-CU references > (sec_offset), or aranges or debug_names I think, etc. Because with DWARF32 > the section references are 32bit, so can't exceed 4GB. > > I suspected that the application might contain 64-bit DWARF, so I asked > the user dump/grep the .debug_info section and oddly enough all of the CUs > are 32-bit DWARF. I have no idea if they are damaged. The CUs must not > reference any DIEs outside their own CU. > Yeah, I guess if you don't use sec_offset (usually only appears with LTO, at least that's the case with clang) and don't use aranges - there's no need to refer to the absolute offset of a thing in .debug_info, so you can exceed the 32 bit limit in total .debug_info size without needing DWARF64. > > (also, with a different example, you'd get string data over 4GB, which > you'd also need DWARF64 for, or in DWARFv5 (with some dispensations from > DWARFv6) you could use DWARF64 for .debug_str_offsets (assuming all string > references were strx forms) without using DWARF64 for everything else) > > Understood. Luckily, my test program's string table made it with some room > to spare. > > > In Split DWARF if each contribution is <4GB you only need a 64 bit > cu/tu_index and 64 bit str_offsets (& you could even do that selectively, > only using DWARF64 str_offsets for contributions that need 64 bit offsets), > since nothing else references across whole sections - which makes it much > more scalable/easier to solve. > > Yes, I asked if they'd be willing to consider Split DWARF, but I haven't > received an answer yet. I think it would save considerable link time and > disk space, but they may have reasons why Split DWARF would not work for > them. > > BTW, given that "gold" is deprecated, is there a linker than can build a > decent ".gdb_index" to use with the Split DWARF? Without a decent index, > it's hard to get same-day service out of the debugger with these giant > executables. > lld has support for .gdb_index from .debug_gnu_pubnames/pubtypes - that's what we use at Google. (& for Split DWARF, as far as I recall/understand, but don't have a concrete example these days - gdb /requires/ an index for correctness, it'll assume that a lookup failure is authoritative - it won't actually load in all the .dwo/dwp content in the case of a missing index - though, I might be wrong, or maybe it's changed, because, as I said, don't have a clear example that demonstrates this failure at the moment) > Also, another issue is that even if you have simple/small bits of DWARF32 > (in some precompiled library, etc), if your total program exceeds 64 bit, > you may not be able link the program because that DWARF32 might end up > being put at the end of the section, and so the debug_aranges, for > instance, needs to record the offset of the CU in 32 bits but can't. So > there's various discussions about linkers being able to sort the DWARF32 > contributions earlier/first before the DWARF64 contributions. Then you > could still link DWARF32 precompiled libraries into huge programs that > exceed the DWARF32 limits. I can go find some links to those > threads/discussions if you need them, I think some happened in the LLVM > open source community. > > Looks like clang dodged all of those problems somehow: > > rocm2 60 04/21 16:58 /build/jdelsign/fatty % llvm-dwarfdump fatty | grep > "Compile Unit" > 0x00000000: Compile Unit: length = 0x461a1e96, format = DWARF32, version = > 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0000, addr_size = 0x08 > (next unit at 0x461a1e9a) > 0x461a1e9a: Compile Unit: length = 0x461a1e99, format = DWARF32, version = > 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0148, addr_size = 0x08 > (next unit at 0x8c343d37) > 0x8c343d37: Compile Unit: length = 0x461a1e99, format = DWARF32, version = > 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0292, addr_size = 0x08 > (next unit at 0xd24e5bd4) > 0xd24e5bd4: Compile Unit: length = 0x461a1e99, format = DWARF32, version = > 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x03dc, addr_size = 0x08 > (next unit at 0x118687a71) > 0x118687a71: Compile Unit: length = 0x461a1e99, format = DWARF32, version > = 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0526, addr_size = 0x08 > (next unit at 0x15e82990e) > rocm2 61 04/21 17:13 /build/jdelsign/fatty % > > This should give the debuggers something to chew on for a while. > Cool cool :)
-- Dwarf-discuss mailing list Dwarf-discuss@lists.dwarfstd.org https://lists.dwarfstd.org/mailman/listinfo/dwarf-discuss