Re: DWZ 0.14 released
On 9 Mar 2021, Jakub Jelinek via Binutils spake thusly: > On Tue, Mar 09, 2021 at 11:38:07AM +, Hannes Domani via Dwz wrote: >> Am Dienstag, 9. März 2021, 10:10:47 MEZ hat Mark Wielaard >> Folgendes geschrieben: >> >> > Hi Allan, >> > >> > On Tue, Mar 09, 2021 at 09:06:54AM +0100, Allan Sandfeld Jensen wrote: >> > > Btw, question for gcc/binutils >> > > >> > > Any reason the work done by tools like dwz couldn't be done in the >> > > compiler or >> > > linker? Seems a bit odd to have a post-linker that optimizes the >> > > generated >> > > code, when optimizations should already be enabled. >> > >> > >> > dwz does two kinds of optimization. First it attempts to optimize the >> > DWARF debugging information for a given object (executable or shared >> > library). Secondly it tries to put shared pieces of a list of given >> > objects into a supplemental file that gets referenced from all the >> > given object files. >> > >> > Technically the first optimization could be done by the linker. But >> > the second optimization is really a post-linker step. >> >> Related question: If it were part of binutils, maybe it could be adapted to >> optimize DWARF debugging information of PE files as well. > > dwz intentionally uses libelf, it often deals with very large amounts of > debug info that only barely fit into the address space limitations on > certain arches or physical memory for good performance, and any kind of > abstraction penalty (e.g. bfd) would make it slower and more memory hungry. Well, it's not *impossible*. You could in theory do what CTF has done: move the dedup machinery into a library which is then called both from the linker (providing its input via BFD and explicitly not supporting multifile unification of DWARF) and from a separate dedup tool still called 'dwz' (providing its input via libelf, and flipping a switch allowing dedup across files and producing an additional output which the tool then writes to the unified file). Howver, link speed would likely be affected if dedup is on by default (it's still a concern of mine with libctf even though none of my testcases take more than a couple of seconds to dedup: I know how to shave a *lot* of time off that, I just haven't done it yet). It's quite possible that you'll save as much time by not having to write as much DWARF out as you lose deduplicating -- but the write time is usually hidden from the user anyway since writeback is usually buffered on most operating systems. So this is probably not as helpful as it might appear :( Also, it might not be acceptable to have dwz depend on a shared library provided by binutils, nor to have binutils depend (even optionally?) on a shared library provided by dwz...
Re: DWZ 0.14 released
On Tue, Mar 09, 2021 at 11:38:07AM +, Hannes Domani via Dwz wrote: > Am Dienstag, 9. März 2021, 10:10:47 MEZ hat Mark Wielaard > Folgendes geschrieben: > > > Hi Allan, > > > > On Tue, Mar 09, 2021 at 09:06:54AM +0100, Allan Sandfeld Jensen wrote: > > > Btw, question for gcc/binutils > > > > > > Any reason the work done by tools like dwz couldn't be done in the > > > compiler or > > > linker? Seems a bit odd to have a post-linker that optimizes the generated > > > code, when optimizations should already be enabled. > > > > > > dwz does two kinds of optimization. First it attempts to optimize the > > DWARF debugging information for a given object (executable or shared > > library). Secondly it tries to put shared pieces of a list of given > > objects into a supplemental file that gets referenced from all the > > given object files. > > > > Technically the first optimization could be done by the linker. But > > the second optimization is really a post-linker step. > > Related question: If it were part of binutils, maybe it could be adapted to > optimize DWARF debugging information of PE files as well. dwz intentionally uses libelf, it often deals with very large amounts of debug info that only barely fit into the address space limitations on certain arches or physical memory for good performance, and any kind of abstraction penalty (e.g. bfd) would make it slower and more memory hungry. So no, PE support is not possible. Jakub
Re: DWZ 0.14 released
Am Dienstag, 9. März 2021, 10:10:47 MEZ hat Mark Wielaard Folgendes geschrieben: > Hi Allan, > > On Tue, Mar 09, 2021 at 09:06:54AM +0100, Allan Sandfeld Jensen wrote: > > Btw, question for gcc/binutils > > > > Any reason the work done by tools like dwz couldn't be done in the compiler > > or > > linker? Seems a bit odd to have a post-linker that optimizes the generated > > code, when optimizations should already be enabled. > > > dwz does two kinds of optimization. First it attempts to optimize the > DWARF debugging information for a given object (executable or shared > library). Secondly it tries to put shared pieces of a list of given > objects into a supplemental file that gets referenced from all the > given object files. > > Technically the first optimization could be done by the linker. But > the second optimization is really a post-linker step. Related question: If it were part of binutils, maybe it could be adapted to optimize DWARF debugging information of PE files as well. Hannes
Re: DWZ 0.14 released
Hi Allan, On Tue, Mar 09, 2021 at 09:06:54AM +0100, Allan Sandfeld Jensen wrote: > Btw, question for gcc/binutils > > Any reason the work done by tools like dwz couldn't be done in the compiler > or > linker? Seems a bit odd to have a post-linker that optimizes the generated > code, when optimizations should already be enabled. dwz does two kinds of optimization. First it attempts to optimize the DWARF debugging information for a given object (executable or shared library). Secondly it tries to put shared pieces of a list of given objects into a supplemental file that gets referenced from all the given object files. Technically the first optimization could be done by the linker. But the second optimization is really a post-linker step. Cheers, Mark
Re: DWZ 0.14 released
Btw, question for gcc/binutils Any reason the work done by tools like dwz couldn't be done in the compiler or linker? Seems a bit odd to have a post-linker that optimizes the generated code, when optimizations should already be enabled. Best regards Allan On Montag, 8. März 2021 13:43:11 CET Tom de Vries wrote: > Hi, > > DWZ 0.14 has been released. > > You can download dwz from the sourceware FTP server here: > > https://sourceware.org/ftp/dwz/releases/ > ftp://sourceware.org/pub/dwz/releases/ > > The vital stats: > > Sizemd5sumName > 184KiB cf60e4a65d9cc38c7cdb366e9a29ca8e dwz-0.14.tar.gz > 144KiB 1f1225898bd40d63041d54454fcda5b6 dwz-0.14.tar.xz > > There is a web page for DWZ at: > > https://sourceware.org/dwz/ > > DWZ 0.14 includes the following changes and enhancements: > > * DWARF 5 support. The tool now handles most of DWARF version 5 > (at least everything emitted by GCC when using -gdwarf-5). > > Not yet supported are DW_UT_type units (DWARF 4 .debug_types > are supported), .debug_names (.gdb_index is supported) and some > forms and sections that are only emitted by GCC when > generating Split DWARF (DW_FORM_strx and .debug_str_offsets, > DW_FORM_addrx and .debug_addr, DW_FORM_rnglistx and > DW_FORM_loclistsx). https://sourceware.org/PR24726 > > * .debug_sup support. DWARF Supplementary Object Files > (DWARF 5, section 7.3.6) can now be generated when using > the --dwarf-5 option. To keep compatibility with existing DWARF > consumers this isn't the default yet. > > Without the --dwarf-5 option instead of a .debug_sup section dwz > will generate a .gnu_debugaltlink section and will use > DW_FORM_GNU_strp_alt and DW_FORM_GNU_reg_alt, instead of > DW_FORM_strp_sup and DW_FORM_ref_sup > > * An experimental optimization has been added that exploits the > One-Definition-Rule of C++. It's enabled using the --odr option, and > off by default. This optimization causes struct/union/class DIEs with > the same name to be considered equal. The optimization can be set to > a lower aggressiveness level using --odr-mode=basic, to possibly be > able to workaround problems without having to switch off the > optimization altogether. > > * The clean-up of temporary files in hardlink mode has been fixed. > > * The DIE limits --low-mem-die-limit / -l and > --max-die-limit / -L can now be disabled using respectively > -l none and -L none. Note that -l none disables the limit, whereas > -l 0 sets the limit to zero. > > * The usage message has been: > - updated to show that -r and -M are exclusive. > - updated to show at -v and -? cannot be combined with other options. > - extended to list all options in detail. > - restyled to wrap at 80 chars. > > * An option --no-import-optimize was added that switches off an > optimization that attempts to reduce the number of > DW_TAG_imported_unit DIEs. This can be used f.i. in case the > optimization takes too long. > > * A heuristic has been added that claims more memory earlier (without > increasing the peak memory usage) to improve compression time. > > * A heuristic has been added that estimates whether one of the two DIE > limits will be hit. If so, it will do an exact DIE count to verify > this. If the exact DIE count finds that the low-mem DIE limit is > indeed hit, processing is done in low-mem mode from the start, rather > than processing in regular mode first. If the exact DIE count finds > that the max DIE limit is indeed hit, processing is skipped > altogether. > > * Various other performance improvements. > > * A case where previously we would either hit the assertion > "dwz: dwz.c:9461: write_die: Assertion `refd != NULL' failed" (in > regular mode) or a segmentation fault (in low-mem mode), now is > handled by "dwz: Couldn't find DIE at DW_FORM_ref_addr offset 0x". > > * A case where a reference from a partial unit to a compile unit was > generated has been fixed. This could happen if a DIE was referenced > using a CU-relative DWARF operator. > > * A case has been fixed for low-mem mode where instead of issuing > "dwz: Couldn't find DIE referenced by DW_OP_GNU_implicit_pointer" dwz > would run into a segfault instead. > > * A multi-file case where we run into ".debug_line reference above end > of section" has been fixed. > > * The following assertion failures were fixed: > - dwz: dwz.c:9310: write_die: Assertion ` > value && refdcu->cu_kind != CU_ALT > ' failed. > - dwz: dwz.c:9920: recompute_abbrevs: Assertion ` > off == cu_size > ' failed. > > * The assert condition of this assertion has been fixed: > - write_types: Assertion `ref && ref->die_dup == NULL'.
DWZ 0.14 released
Hi, DWZ 0.14 has been released. You can download dwz from the sourceware FTP server here: https://sourceware.org/ftp/dwz/releases/ ftp://sourceware.org/pub/dwz/releases/ The vital stats: Sizemd5sumName 184KiB cf60e4a65d9cc38c7cdb366e9a29ca8e dwz-0.14.tar.gz 144KiB 1f1225898bd40d63041d54454fcda5b6 dwz-0.14.tar.xz There is a web page for DWZ at: https://sourceware.org/dwz/ DWZ 0.14 includes the following changes and enhancements: * DWARF 5 support. The tool now handles most of DWARF version 5 (at least everything emitted by GCC when using -gdwarf-5). Not yet supported are DW_UT_type units (DWARF 4 .debug_types are supported), .debug_names (.gdb_index is supported) and some forms and sections that are only emitted by GCC when generating Split DWARF (DW_FORM_strx and .debug_str_offsets, DW_FORM_addrx and .debug_addr, DW_FORM_rnglistx and DW_FORM_loclistsx). https://sourceware.org/PR24726 * .debug_sup support. DWARF Supplementary Object Files (DWARF 5, section 7.3.6) can now be generated when using the --dwarf-5 option. To keep compatibility with existing DWARF consumers this isn't the default yet. Without the --dwarf-5 option instead of a .debug_sup section dwz will generate a .gnu_debugaltlink section and will use DW_FORM_GNU_strp_alt and DW_FORM_GNU_reg_alt, instead of DW_FORM_strp_sup and DW_FORM_ref_sup * An experimental optimization has been added that exploits the One-Definition-Rule of C++. It's enabled using the --odr option, and off by default. This optimization causes struct/union/class DIEs with the same name to be considered equal. The optimization can be set to a lower aggressiveness level using --odr-mode=basic, to possibly be able to workaround problems without having to switch off the optimization altogether. * The clean-up of temporary files in hardlink mode has been fixed. * The DIE limits --low-mem-die-limit / -l and --max-die-limit / -L can now be disabled using respectively -l none and -L none. Note that -l none disables the limit, whereas -l 0 sets the limit to zero. * The usage message has been: - updated to show that -r and -M are exclusive. - updated to show at -v and -? cannot be combined with other options. - extended to list all options in detail. - restyled to wrap at 80 chars. * An option --no-import-optimize was added that switches off an optimization that attempts to reduce the number of DW_TAG_imported_unit DIEs. This can be used f.i. in case the optimization takes too long. * A heuristic has been added that claims more memory earlier (without increasing the peak memory usage) to improve compression time. * A heuristic has been added that estimates whether one of the two DIE limits will be hit. If so, it will do an exact DIE count to verify this. If the exact DIE count finds that the low-mem DIE limit is indeed hit, processing is done in low-mem mode from the start, rather than processing in regular mode first. If the exact DIE count finds that the max DIE limit is indeed hit, processing is skipped altogether. * Various other performance improvements. * A case where previously we would either hit the assertion "dwz: dwz.c:9461: write_die: Assertion `refd != NULL' failed" (in regular mode) or a segmentation fault (in low-mem mode), now is handled by "dwz: Couldn't find DIE at DW_FORM_ref_addr offset 0x". * A case where a reference from a partial unit to a compile unit was generated has been fixed. This could happen if a DIE was referenced using a CU-relative DWARF operator. * A case has been fixed for low-mem mode where instead of issuing "dwz: Couldn't find DIE referenced by DW_OP_GNU_implicit_pointer" dwz would run into a segfault instead. * A multi-file case where we run into ".debug_line reference above end of section" has been fixed. * The following assertion failures were fixed: - dwz: dwz.c:9310: write_die: Assertion ` value && refdcu->cu_kind != CU_ALT ' failed. - dwz: dwz.c:9920: recompute_abbrevs: Assertion ` off == cu_size ' failed. * The assert condition of this assertion has been fixed: - write_types: Assertion `ref && ref->die_dup == NULL'.