Hi all,
Here I propose, along with Greg Clayton, Processor Trace support for LLDB. I’m attaching a link to the
document that contains this proposal if that’s easier to read for you:
https://docs.google.com/document/d/1cOVTGp1sL_HBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI/edit#heading=h.t5mblb9ugv8f
<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1cOVTGp1sL-5FHBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI_edit-23heading-3Dh.t5mblb9ugv8f&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=erxV6KMIZvIQjyWYW8YpOiKz-WqJt4giKQA34YMHsRY&m=DuuwXHUQJpW4TcCay4hPsBund-eBI2uVaVimqEPsp5k&s=o6vqoYYbn-Tz_d34hoLJvWhEnnhracOO6yDsMzq8wR0&e=>.
Please make any comments in this mail list.
If you want to quickly know what Processor Trace can do, you can read this
https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace
<https://urldefense.proofpoint.com/v2/url?u=https-3A__easyperf.net_blog_2019_08_23_Intel-2DProcessor-2DTrace&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=erxV6KMIZvIQjyWYW8YpOiKz-WqJt4giKQA34YMHsRY&m=DuuwXHUQJpW4TcCay4hPsBund-eBI2uVaVimqEPsp5k&s=iaErHaf8byXlZb1YFUk0BpQ-duMhNouUUMyktLm3soQ&e=>.
Any comments are appreciated, especially the ones regarding the commands the
user will interact with.
Thanks,
Walter Erquinigo.
# RFC: Processor Trace Support in LLDB
# What is processor tracing?
Processor tracing works by capturing information about the execution of a
process so that the control flow of the program can be reconstructed later.
Implementations of this are Intel Processor Trace for X86, x86_64
([https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html))
and ARM CoreSight for some ARM devices
([https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace](https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace)).
As a clarifying example, with these technologies it’s possible to trace all the
threads of a process, and after the process has finished, reconstruct every
single instruction address each thread has executed. This could include some
additional information like timestamps, async CPU events, kernel instructions,
bus clock ratio changes, etc. On the other hand, memory and registers are not
traced as a way to limit the size of the trace.
# Intel Processor Trace as the first implementation
We’ll focus on Intel Processor Trace (Intel PT), but in a generic way so that
in the future similar technologies can be onboarded in LLDB.
Intel PT has the following features:
* Control flow tracing in a highly encoded format
* 3% to 5% slowdown when capturing
* No memory nor registers captured
* Kernel tracing support
* Timestamps of branches are produced, which can be used for profiling
* Adjustable size of trace buffer
* Supported on most Intel CPUs since 2015
* X86 and x86_64 only
* Official support only on Linux
* Basic support on Windows
* Decoding/analysis can be done on any operating system
A very nice introduction to Intel PT can be found
[https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html)
and
[https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace](https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace).
Totally recommended to fully grasp the impact of this project.
More technical details are in
[https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt](https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt).
Even more technical details are in the processor manual
[https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3c-part-3-manual.pdf](https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3c-part-3-manual.pdf)
# Basic Definitions
* Trace file: A trace file basically contains the information of the target
addresses of each branch or jump within the program execution in a highly
encoded format.
* Capturing: The act of tracing a process and producing a trace file.
* Decoding: Decoding outputs a sequential list of instructions given a trace
file and the images of a process. Decoding is generally an offline step as it’s
expensive.
* Trace buffer: In order to limit the size of the trace, an on-memory
circular buffer can be used, keeping the most recent branching information. The
trace file is a snapshot of this.
* Gap: Sporadically some branching information can be lost or be impossible
to decode, which creates a gap in the reconstructed control flow.
# New LLDB features
* Loading traces: We want to load traces potentially from other computers,
and have LLDB symbolicating it. A flow like the following should be possible \
```
$ trace load /path/to/trace
$ trace dump --instructions
pid: '1234', tid: '1981309'
a.out`main
[57] 0x400549 <+13>: movl %eax, -0x4(%rbp)
a.out`bar()
[56] 0x40053b <+46>: retq
[55] 0x40053a <+45>: leave
[54] 0x400537 <+42>: movl -0x4(%rbp), %eax
[53] 0x400535 <+40>: jle 0x400525 ; <+24> at
main.cpp:7
[52] 0x400531 <+36>: cmpl $0x3, -0x8(%rbp)
[51] 0x40052d <+32>: addl $0x1, -0x8(%rbp)
[50] 0x40052a <+29>: addl %eax, -0x4(%rbp)
a.out`foo()
[49] 0x400567 <+15>: retq
[48] 0x400566 <+14>: popq %rbp
[47] 0x400563 <+11>: movl -0x4(%rbp), %eax
[46] 0x40055c <+4>: movl $0x2a, -0x4(%rbp)
...
[1] 0x400559 <+1>: movq %rsp, %rbp
[0] 0x400558 <+0>: pushq %rbp
// Format:
```
` // [instruction index] <instruction disassembly> \
`Notice the resemblance to loading a core file, but in this case we can get the
control flow, printed in reverse order in this example.
* Decoding: LLDB can use libipt
([https://github.com/intel/libipt](https://github.com/intel/libipt)), which is
the low level Intel PT decoding library, to convert trace files into
instructions.
* Showing instructions: LLDB can output the list of instructions of the
control flow, as shown above
* Showing function calls: Similarly, LLDB can print a hierarchical view of
the function calls. A flow like this should be possible: \
```
$ trace load /path/to/trace
$ trace dump --function-calls
pid: '1234', tid: '1981309'
[50] a.out`bar() 0x40052a
[45] a.out`zaz() 0x400558
[40] a.out`baz() 0x400559
[30] a.out`foo() 0x400567
```
` [0] a.out`main 0x400000 \
\
`This functionality allows LLDB to reconstruct the call stack at any point and
potentially do reverse debugging.
* Capturing: LLDB can also do the Intel PT capturing of a live process, so
that at any stop the user can do reverse stepping or simply inspect the trace.
A possible flow is:
```
$ <stopped at main>
$ b main.cpp:50
$ trace start intel-pt // this initiates the tracing
$ continue
$ <stopped at main.cpp:50>
$ trace dump --instructions
pid: '1234', tid: '1981309'
a.out`main
[57] 0x400549 <+13>: movl %eax, -0x4(%rbp)
a.out`bar()
[56] 0x40053b <+46>: retq
[55] 0x40053a <+45>: leave
```
Displaying time information: If the trace contains timing information, we
could also display it along with each instruction, e.g.
```
a.out`bar()
[56: 1600284226]: 0x40053b <+46>: retq
...
[4: 1600284200]: 0x40053a <+45>: leave
// Format:
// [instruction index: unix timestamp] <instruction disassembly>
```
Furthermore, we could display the time spent in each function.
# Future LLDB features
* Reverse Stepping: With the hierarchical reconstruction of the function
calls, along with the individual instructions, LLDB can offer reverse stepping.
Operations like reverse-next, reverse-step-out, reverse-continue could work by
traversing the trace. We plan to work on this once the features presented above
are in place.
* Trace-based profiling
* SB API of the mentioned features
# Why is this useful?
* Bug root-causing:
* For example, a crash in a production Release build ends up being
analyzed with logs, a coredump, and a stack trace. Logs are not comprehensive,
and a stack trace only contains the final state of the program. Providing the
user with the control flow of the last milliseconds gives a tremendous amount
of information that is game-changing in root-causing issues. It could be said
that the user goes from a single stack trace to a list of stack traces.
* Reverse stepping enables more efficient debugging, as it reduces the
number of iterations to efficiently root-cause bugs. More often than not,
reproducing a bug takes a considerable amount of time, and the user needs to
reproduce it several times until the correct breakpoints are hit. This takes a
considerable amount of time. Giving the user the information of what has been
executed so far can help them figuring out where’s the location to place a
breakpoint, or to very easily figure out what went wrong.
* Low cost: unlike other similar technologies, Intel PT has an almost
negligible performance cost regardless of whether the build is optimized or
not, making it appealing to a wide range of scenarios.
* This infrastructure can be used for enabling other tools like
non-sample-based profilers with instruction-level accuracy, security analyzers
that check if certain memory regions are executed, and trace comparators, which
could find bugs by comparing similar traces.
# Goals of this document:
* Gather feedback on the basic Trace implementation, which would include the
following basic operations: loading, decoding, and dumping.
* Create awareness about this work.
* Get a green light on the current set of patches implementing this feature
starting with https://reviews.llvm.org/D85705.
# Non-Goals:
* Discuss how reverse-stepping will be implemented. This can be left for
another discussion. Once the Trace architecture is in place and robust,
reverse-stepping can then be discussed, as it’s a more controversial change
than this one.
* Explain thoroughly Intel PT.
# Existing Tool Support
* GDB has a basic implementation of the features above
([https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html](https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html))
and some ideas are taken from there.
* Perf is a standalone tool that can do capturing and decoding.
* The Linux kernel has full support for doing capturing at thread, logical
cpu or cgroup level.
* Intel developed a basic version of Intel PT support in LLDB as an external
plugin. [https://reviews.llvm.org/D33674](https://reviews.llvm.org/D33674),
[https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b](https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b).
# New Trace Commands
Based on this patch
[https://reviews.llvm.org/D85705](https://reviews.llvm.org/D85705), there would
be a common Trace class along with plug-in implementations.
## Trace loading
### $ trace load /path/to/trace/settings/file.json
As decoding a trace requires the images of the object files, the trace files
and some CPU information, it’s convenient to have a JSON file that describes an
entire trace session. The following JSON schema could be used.
```
{
"trace": {
… // plug-in specific information
},
"processes": [ // process information common to all trace plug-ins
{
"pid": integer,
"triple": string, // llvm-triple
"threads": [
{
"tid": integer,
"traceFile": string
}
],
"modules": [
{
"systemPath": string, // original path of the module at runtime
"file"?: string, // copy of the file if not available at "systemPath"
"loadAddress": string, // string address in hex or decimal form
"uuid"?: string,
}
]
}
]
}
// Notes:
// All paths are either absolute or relative to the settings file.
```
**Corefiles:**
We plan to extend this schema to support corefiles, but we would leave it out
of this discussion, as can be easily seen as an extension of this basic schema.
**Implementation details:**
To make our first implementation easier, we’ll ask for an individual trace file
per thread. This is the simpler collection mode for Intel PT.
The entire json file will be translated into a Trace object, which contains the
trace information of each thread and process in it.
Each process in the json file will be represented as a new Target. Similarly,
threads and modules for each target will be created following the json file.
This is very similar to what loading a minidump or coredump does.
Each Target will be associated with a Trace, and multiple targets can share the
same Trace. The contract is that Trace is assumed to end at the current PC of
each thread of the target.
### $ trace schema <plug-in>
This command prints the JSON schema of the trace settings file for the provided
plug-in. It would output something similar to this
```
{
"trace": {
"type": "intel-pt",
"pt_cpu": {
"vendor": "intel" | "unknown",
"family": integer,
"model": integer,
"stepping": integer
}
},
"processes": [
{
"pid": integer,
"triple": string, // llvm-triple
"threads": [
{
"tid": integer,
"traceFile": string
}
],
"modules": [
{
"systemPath": string, // original path of the module at runtime
"file"?: string, // copy of the file if not available at "systemPath"
"loadAddress": string, // string address in hex or decimal form
"uuid"?: string,
}
]
}
]
}
// Notes:
// All paths are either absolute or relative to the settings file.
```
### $ trace dump [--verbose] [-t tid1] [-t tid2] ...
Print the trace information corresponding to the provided thread ids of the
currently selected target, which would mainly include the same information as
the trace settings file. If no tid is provided, the currently selected thread
is used. This would be useful for debugging. The information would be like
Modules:
<module info like systemPath, file, load address, uuid, size>
Threads:
<thread info like location of trace file, number of instructions (if
already decoded), number of function calls (if already decoded)>
If <--verbose> is passed, the original settings.json file is printed as well.
## Decoder-based commands
The following commands require decoding the trace and are of the form. “trace dump
<action> [-t <tid>]”. If tids are not specified, then the current thread
or the current target will be used.
### $ trace dump --instructions [-t <tid>] [-c <count> = 10] [-o
<offset> = 0]
This command would print the last <count> instructions starting at the given
offset from the last instruction in the trace. The output would be similar to that of
the “disassembly” command and would include timing information if available.
```
$ trace dump --instructions -c 5
pid: '1234', tid: '1981309'
a.out`main
[57] 0x400549 <+13>: movl %eax, -0x4(%rbp)
a.out`bar()
[56] 0x40053b <+46>: retq
[55] 0x40053a <+45>: leave
[54] error -13. 'no memory mapped at this address'
a.out`foo()
[53] 0x400567 <+15>: retq
```
Repeating the command would continue printing where it was left off in the last
run.
**Implementation details:**
Each instruction output by the decoder is either an actual instruction or an
error. An error can be caused due to a collection error (e.g. internal CPU
buffer overflow error) or a decoding error (e.g. the image of an object file is
missing while decoding). These errors represent gaps in the trace and the user
should know about them, so we print them accordingly in this dump.
Each instruction (including errors) has an index in the decoded trace, and
serves as a checkpoint.
### $ trace dump --function-calls [-t <tid>] [-c <count> = 10] [-o
<offset> = 0] [--flat]
This command would print the hierarchical list of function calls. Similar to the
“--instructions” command, it would show the last <count> function calls with
the given offset from the last instructions. Timing information would be included if
available.
```
$ trace dump --function-calls
pid: '1234', tid: '1981309'
[50] a.out`bar() 0x40052a
[45] a.out`zaz() 0x400558
[40] a.out`baz() 0x400559
[30] a.out`foo() 0x400567
[0] a.out`main 0x400000
```
Repeating the command would continue printing where it was left off in the last
run.
If <--flat> is passed, then instead of a hierarchical view, a flat list
would be produced.
## Capturing command
### $ trace start <plugin_name> [-t <tid>] [--all] [-b
<buffer_size_in_KB>]
This command will start tracing the given thread of the currently selected
target, or all the threads of that target if “--all” is passed. If “--all” is
passed, any thread created after this command will also be traced automatically.
Besides, the optional -b parameter can define the size of each trace buffer to
be created. I haven’t yet decided a default one, but 1M might be acceptable, as
it traces around 1 million instructions on average according to Intel, and
that’s more than enough for a useful analysis.
For an initial implementation, the plugin_name parameter will be required (e.g.
intel-pt). Later a more automated mechanism for finding the right plugin can be
implemented.
**Implementation notes:**
There’s already a basic implementation in lldb as an external plugin. It’s in
[https://reviews.llvm.org/source/llvm-github/browse/master/lldb/tools/intel-features/intel-pt/](https://reviews.llvm.org/source/llvm-github/browse/master/lldb/tools/intel-features/intel-pt/)
created by
[https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b](https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b).
It hasn’t received much attention and has been mostly unmaintained since it
was created. It’s already capable of tracing a given thread and collecting the
trace buffer. We plan to reuse that logic, which is already working.
A Trace object will be created and will be associated with the current Target.
Any interaction with trace, like dumping instructions, will trigger a fetch of
the most recent trace buffer, unless it hasn’t changed.
When multiple threads are traced, each one will have its own trace buffer, as
sharing one buffer in multiple threads requires knowing when each context
switch happened so that the decoded trace can be split correctly among threads.
This is beyond the scope of the initial version of this project.
### $ trace save /path/to/file.json [--copy-images]
This creates a bundle trace with settings saved in the given json file for the
current process. By default, it doesn’t create any copy of the images loaded on
the process, unless the “--copy-images” parameter is specified. That parameter
is useful for analyzing the trace in a machine other than where it was captured.
# Remote Protocol Changes
No remote protocol changes are required, as
[https://reviews.llvm.org/D33674](https://reviews.llvm.org/D33674) and
[https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b](https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b)
already created them some years ago.
# Build Requirements
In order to build LLDB with this support, it has to be linked with a build of
libipt [https://github.com/intel/libipt](https://github.com/intel/libipt),
which is the decoder.
# Operating System Requirements for Collection/Tracing
Collection can only be done on linux if the file
/sys/bus/event_source/devices/intel_pt/type is defined. The logic gating this
feature is already checked in and defined in
[https://reviews.llvm.org/D33674](https://reviews.llvm.org/D33674).
# Testing
It’s fortunately straightforward to test this feature. It’s possible to capture
traces with perf or with the future “trace start” / ”trace save” commands and
create trace bundles with their corresponding settings .json file. Analyzing
those traces should give the same results on any machine, making testing
deterministic.
[https://reviews.llvm.org/D85705](https://reviews.llvm.org/D85705) and
descendents already implement some deterministic tests.
_______________________________________________
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
