Hello,
On 05/24/2019 06:24 AM, Jakub Jelinek wrote:
On Tue, May 21, 2019 at 03:44:47PM -0700, Indu Bhagat wrote:
Yes and No :) And that is indeed one of the motivation of the project - to
allow CTF generation where it's most suited aka the toolchain.
There do exist utilties for generation of CTF from DWARF. For example, one of
them is the dwarf2ctf in the DTrace Linux distribution. dwarf2ctf works offline
to transform DWARF generated by the compiler into CTF.
So, if there is a conversion utility, why don't we just change gcc so that
if some new option is passed on the gcc driver link line, then that
post-processing utility will be invoked too?
Performing DWARF to CTF conversion at link-time is not recommended because for
any large project, it is time-consuming to generate, read in, and convert DWARF
to CTF in a post-processing step. These costs are prohibitive enough and affect
CTF adoption.
Data for some projects below.
A dependency of an external conversion utility for "post-processing" DWARF
offline poses several problems:
1. Deployment problems: the converter should be distributed and integrated in
the build system of the program. This, on occasions, can be intrusive. For
example, in terms of dependencies: the dwarf2ctf converter depends on
libdwarf from elfutils suite, glib2 (used for the GHashTable), zlib (used to
compress the CTF information) and libctf (which both reads and writes the
CTF data).
I don't see this as a problem.
2. Performance problems: the conversion from DWARF to CTF can take a long time,
especially in big programs such as the Linux kernel.
So optimize it? Linux kernel certainly doesn't have extra large debug
information, compared to other projects.
DWARF generation only for it to be transformed into something compact like CTF
and eventually to be stripped away is not an efficient workflow; Further, only
a subset of this is amenable to "optimizations", i.e. compile-time generation
of DWARF is unavoidable in this flow.
For example, for the Linux kernel, the debuginfo in the object files is ~9GiB
(The object files without debuginfo is ~1.5GiB). That's a 500% increase in the
space requirements. Generating the DWARF at compile time adds 20% to the compile
time. Next, reading in this DWARF using libdwarf from elftuils takes ~1 min 10
seconds. Next, a conversion utility like dwarf2ctf will then need to perform
de-duplication, on this rather voluminous DWARF, as you see, for the purpose on
hand.
Not just the kernel, for another relatively large internal application we
measured a 19% and 23% increase in compile time for generating DWARF for
a randomly chosen subsets of C and C++ components respectively. For the entire
application, it already takes an order of 3-4 hours to perform a parallel build.
The space requirements for building the entire application with -g also increase
to about 5.5x.
These numbers I state above are on sufficiently beefy platforms. On the
"wimpier" platforms or single-threaded builds for that matter, the increase in
time costs to generate DWARF and then use an external utility (at link-time
or offline) just aggravates the pain for developers, who may need to do even
multiple builds a day.
So, in summary, this increase in build times and space costs is noticeable
(even prohibitive in some cases) and does impact a subset of kernel and
application developers who need to use CTF. Above all, effectively it limits
CTF adoption.
3. Maintainability problems: the converter should be maintained in order to
reflect potential changes in the DWARF generated by the compiler.
If you integrate the support into GCC, then it will need to be maintained
there as well, I bet it will be more work than on the conversion utility.
Yes, work is needed on all components of the toolchain to integrate CTF
generation. We are working on the compiler, linker and debugger (GDB) currently.
We are committing to maintain the CTF generation in GCC, just like we are doing
for Binutils, and will be doing for GDB.
Jakub
