DTrace

From TracingWiki

[hide]

[edit] Overview

DTrace is a dynamic instrumentation framework built for and currently integrated into the Solaris 10 and Open Solaris operating systems. Apple has already integrated DTrace in Mac OS X 10.5 and added Instruments and Shark as user interfaces for system and application viewing. The main components of the system are the DTrace framework and the instrumentation providers both residing in the kernel. The main idea behind separating the two sub-components resides in the fact that different providers can have different instrumentation techniques. Instrumentation providers are loadable kernel modules that define a set of probes they can activate on demand. These probes are advertised to consumers and can be identified by the following 4-tuple: <provider name, module, function, probe name>. When not enabled, DTrace has zero probe effect.

Users can specify in a script file the 4 elements to activate a specific probe, or any subset of the 4 elements to only activate the probes that match with the specified subset. When a probe fires, the corresponding provider transfers control to the DTrace framework which disables interrupts on the current CPU, and executes the corresponding actions previously specified in the script file. The DTrace framework handles the case where multiple consumers are assigned to the same probe. When done, interrupts are enabled, and control is transferred back to the provider.

The DTrace framework allocates a per-CPU kernel buffer for every consumer. Data is read out of the kernel by having two per-CPU buffers: an active and an inactive buffer. When the consumer issues a call to read a buffer on a particular CPU, the active and inactive buffers are switched after disabling the interrupts on the CPU. Then, interrupts are reenabled, and the inactive buffer is copied out to the consumer. When a consumer buffer lacks sufficient space for data recording, a drop count is incremented and data is lost for this consumer.

[edit] Statically Defined Tracing

The Statically defined tracing has a different mechanism. The code is statically instrumented at first through a C macro that expands to a call to a non-existent function having a well-defined prefix (__dtrace_probe_). When the kernel linker detects this, it automatically replaces this call with a no-operation, and saves the tuple <function name, address of the call site>. When the corresponding probe is enabled, the provider replaces dynamically the no-operation by a call to transfer control into the DTrace framework. When the probe is disabled, the added no-op instruction has a negligible effect. Other providers can be used to gather information regarding disk input and output, lock contention, CPU scheduling and process creation and termination. This mechanism induces an observable probing impact even when tracing isn't enabled.

[edit] D Language

The D language is derived from a large subset of C, and allows access to the kernel's native types and global variables. Users can specify inside the D script file the providers names, the probes names, and the actions to take whenever each probe is hit. This file can then be compiled by the D compiler implemented in the DTrace library and invoked by the DTrace command. Upon execution, the DTrace framework enables the probes that were specified in the file by making appropriate calls to the probe's providers.

[edit] User-Space Tracing

User space tracing is also possible using DTrace. The first step is to write a data provider in the D programming language and specify probes in it. Then in the user's C code, the macro DTRACE_PROBE can be called where tracing is desired, with the two essential parameters: the provider name and probe name. Other pertinent arguments can also be passed. The dtrace -G option can be used to link the provider and probe definitions with the application object file.

Examples of user space tracing with dtrace is available at http://www.postgresql.org/docs/8.2/interactive/dynamic-trace.html. http://blogs.sun.com/barts/entry/putting_user_defined_dtrace_probe

[edit] Kernel Tracing Impact

The impact of DTrace dynamic tracing is studied in this section. The chosen test was to compile the gcc-3.4.6 source code while Dtracing only one event: the function entry of the read system call. The test is run on a Pentium 4, 3GHz, with 2 GB of RAM. The results are shown in the table below.

	All syscall probes enabled	Read syscall probes enabled	No probing
Avg. exec. time	41m 55.4995	41m 33.522	41m 12.593
Avg. num. of ev.	36,372,603	5,571,071	-
Avg. cost per event (us)	1.18	3.7	-

Tracing only the read syscall takes 0.85% more time to finish; tracing all syscalls (232 different events) would take 1.73% more time to finish.

[edit] User-Space Tracing Impact

The impact of the statically defined user-space tracing with DTrace was measured on a Pentium 4 with 2GB of RAM running Solaris 10 (SunOS 5.10). The test was to convert an mp3 file (1.6 MB) into a wav file using mpg123 (version 1.6.4) while tracing every function entry and exit. The idea was to compile the program with the -finstrument-functions option of gcc which adds a call to the functions __cyg_profile_func_enter and __cyg_profile_func_exit respectively at every function entry and exit of the program. There, we add the DTRACE_PROBE() macros enabling us to trace every function entry and exit called at execution time. The results are shown below:

Running the program with no tracing: 0m1.081s

Running the program with no tracing; program compiled with the gcc option -finstrument-functions: 0m1.086s (0.46% slower than normal execution)

Running the program with no tracing; program compiled with the gcc option -finstrument-functions and DTrace macros: 0m2.668s (146.8% slower than normal execution)

Running the program while tracing; program compiled with the gcc option -finstrument-functions and DTrace macros:
- Generating text trace (103 MB): 0m9.161s (747.45% slower than normal execution; 3.53 usec per event)
- Outputting to /dev/null: 0m8.027s (642.55% slower than normal execution; 3.03 usec per event).

[edit] References

The paper that introduced DTrace. Contains an interesting case study that demonstrates how DTrace can be used to debug performance problems.

@InProceedings{ dtrace,
title = "Dynamic instrumentation of production systems",
author = "B.M. Cantrill and M.W. Shapiro and A.H. Leventhal",
address = "Boston, MA, USA",
journal = "Proceedings of the General Track 2004 USENIX Annual Technical Conference",
pages = "15--28",
year = "2004",
abstract = "This paper presents DTrace, a new facility for dynamic instrumentation of production systems. DTrace features the ability to dynamically instrument both user-level and kernel-level software in a unified and absolutely safe fashion. When not explicitly enabled, DTrace has zero probe effect - the system operates exactly as if DTrace were not present at all. DTrace allows for many tens of thousands of instrumentation points, with even the smallest of systems offering on the order of 30,000 such points in the kernel alone. We have developed a C-like high-level control language to describe the predicates and actions at a given point of instrumentation. The language features user-defined variables, including thread-local variables and associative arrays. To eliminate the need for most postprocessing, the facility features a scalable mechanism for aggregating data and a mechanism for speculative tracing. DTrace has been integrated into the Solaris operating system and h
as been used to find serious systemic performance problems on production systems - problems that could not be found using preexisting facilities",
keywords = "high level languages, instrumentation, operating system kernels, production engineering computing, production facilities",
url = ""http://www.usenix.org/event/usenix04/tech/general/full_papers/cantrill/cantrill_html/"
class="external free"
title="http://www.usenix.org/event/usenix04/tech/general/full_papers/cantrill/cantrill_html/"
rel="nofollow">http://www.usenix.org/event/usenix04/tech/general/full_papers/cantrill/cantrill_html/",
copyright = "Copyright 2005, IEE",
language = "English"
}

Another paper with an interesting case study.

@Article{ plainsight,
	title = "Hidden in plain sight",
	author = "Bryan Cantrill",
	publisher = "ACM",
	address = "New York, NY, USA",
	journal = "Queue",
	pages = "26--36",
	volume = "4",
	number = "1",
	year = "2006",
	issn = "1542-7730",
	url = ""http://portal.acm.org/citation.cfm?doid=1117389.1117401"
 class="external free"
 title="http://portal.acm.org/citation.cfm?doid=1117389.1117401"
 rel="nofollow">http://portal.acm.org/citation.cfm?doid=1117389.1117401",
	doi = "http://doi.acm.org/10.1145/1117389.1117401"
}

Technical details of the x86 implementation of DTrace.

@Misc{ dtrace-x86,
	title = "The DTrace backend on Solaris for x86/x64",
	author = "Frank Hofmann",
	note = "\url{http://opensolaris.org/os/project/czosug/events_archive/czosug2_dtrace_x86.pdf}",
	url = ""http://opensolaris.org/os/project/czosug/events_archive/czosug2_dtrace_x86.pdf"
 class="external free"
 title="http://opensolaris.org/os/project/czosug/events_archive/czosug2_dtrace_x86.pdf"
 rel="nofollow">http://opensolaris.org/os/project/czosug/events_archive/czosug2_dtrace_x86.pdf"
}

[linuxkernelnewbies] DTrace - TracingWiki