* Tom Zanussi ([EMAIL PROTECTED]) wrote: > This is the documentation for the Generic Trace Setup and Control > patchset, first submitted a couple of weeks ago. See > > http://marc.info/?l=linux-kernel&m=118214274912586&w=2 > > for a more detailed description. > > I've updated this patch to incorporate the suggestions made by Alexey > Dobriyan in that thread. >
It would be nice, since it claims to be "generic", to support things brought forward by other tracers like LTTng, such as : multiple channels (for low, medium and high event rate data, with user-selectable sizes), hybrid trace mode (combined normal trace channels and flight recorder channels). Please see comments below, > Signed-off-by: Tom Zanussi <[EMAIL PROTECTED]> > Signed-off-by: David Wilder <[EMAIL PROTECTED]> > --- > gtsc.txt | 247 > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 247 insertions(+) > > diff --git a/Documentation/gtsc.txt b/Documentation/gtsc.txt > new file mode 100644 > index 0000000..470d1fc > --- /dev/null > +++ b/Documentation/gtsc.txt > @@ -0,0 +1,247 @@ > +Generic Trace Setup and Control (GTSC) > +================================== > +In the kernel, GTSC provides a simple API for starting and managing > +data channels to user space. GTSC builds on the relay interface. For a > +complete description of the relay interface, please see: > +Documentation/filesystems/relay.txt. > + > +GTSC provides one layer in a complete tracing application. The idea of > +the GTSC is to provide a kernel API for the setup and control of tracing > +channels. User of GTSC must provide a data layer responsible for formatting > +and writing data into the trace channels. > + > +A layered approach to tracing > +============================= > +A complete kernel tracing application consists of a data provider and a data > +consumer. Both provider and consumer contain three layers; each layer works > +in tandem with the corresponding layer in the opposite side. The layers are > +represented in the following diagram. > + > +Provider Data layer > + Formats raw trace data and provides data-related service. > + For example, adding timestamps used by consumer to sort data. > + > +Provider Control layer > + Provided by GTSC. Creates trace channels and informs the data layer > + and consumer of the current state of the trace channels. > + > +Provider Buffering layer > + Provided by relay. This layer buffers data in the > + kernel for consumption by the consumer's buffer > + layer. > + > +Provider (in-kernel facility) > +----------------------------------------------------------------------------- > +Consumer (user application) > + > + > +Consumer Buffer layer > + Reads/consumes data from the provider's data buffers. > + > +Consumer Control layer > + Communicates to the provider's control layer to control the state > + of the trace channels. > + > +Consumer Data layer > + Sorts and formats data as provided by the provider's data layer. > + > +The provider is coded as a kernel facility. The consumer is coded as > +a user application. > + > + > +GTSC - Features > +============== > +The GTSC exploits services and features provided by relay. These features > are: > +- The creation and destruction of relay channels. > +- Buffer management. Overwrite or non-overwrite modes can be selected > + as well as global or per-CPU buffering. > + > +Overwrite mode can be called "flight recorder mode". Flight recorder > +mode is selected by setting the TRACE_FLIGHT_CHANNEL flag when > +creating trace channels. In flight mode when a tracing buffer is > +full, the oldest records in the buffer will be discarded to make room > +as new records arrive. In the default non-overwrite mode, new records > +may be written only if the buffer has room. In either case, to > +prevent data loss, a user space reader must keep the buffers > +drained. GTSC provides a means to detect the number of records that > +have been dropped due to a buffer-full condition (non-overwrite mode > +only). > + > +When per-CPU buffers are used, relay creates one debugfs file for each > +running CPU. The user-space consumer of the data is responsible for > +reading the per-CPU buffers and collating the records presumably using > +a time stamp or sequence number included in the trace records. The > +use of global buffers eliminates this extra work of sequencing > +records; however the provider's data layer must hold a lock when > +writing records. The lock prevents writers running on different CPUs > +from overwriting each other's data. However, buffering may be slower > +because write to the buffer are serialized. Global buffering is > +selected by setting the TRACE_GLOBAL_CHANNEL flag when creating trace > +channels. > + > +GTSC User Interface > +=================== > +When a GTSC channel is created and tracing has been started, the following > +directories and files are created in the root of the mounted debugfs. > + > +/debug (root of the debugfs) > + /<trace-root-dir> > + /<trace-name> > + trace0 ... traceN (Per-CPU trace data, one per CPU) > + state (Used to start and stop tracing) > + dropped (number of records dropped due > + to a full-buffer condition, only) > + for non-TRACE_FLIGHT_CHANNELs) > + rewind ('un-consume' channel data i.e. > + start next read at the beginning > + again (TRACE_FLIGHT_CHANNELS > only) > + nr_sub (Number of sub-buffers in channel) > + sub_size (Size of sub-buffers in channnel) > + > +Trace data is gathered from the trace[0...N] files using one of the > available > +interfaces provided by relay (see Documentation/filesystems/relay.txt). > +When using the READ(2) interface, as data is read it is marked as consumed by > +the relay subsystem. Therefore, subsequent reads will only return unconsumed > +data. > + > +GTSC Kernel API > +=============== > +An overview of the GTSC Kernel API is now given. More details of the API can > +be found in linux/gtsc.h. > + > +The steps a kernel data provider takes to utilize the GTSC are: > +1) Set up a gtsc channel - trace_setup() > +2) Start the GTSC channel - trace_start() > +3) Write one or more trace records into the channel (using the relay API). > +4) Optionally stop and start tracing as desired - trace_start()/trace_stop() > +5) Destroy the GTSC channel and underlying relay channel - trace_cleanup(). > + > +GTSC Example > +=========== > +This small sample module creates a GTSC channel. It places a kprobe on the > +function do_fork(). The kprobe handler will write a timestamp and > +current->pid to the GTSC channel. > + > +You can build the kernel module kprobes_gtsc.ko using the following Makefile: > +------------------------------------CUT------------------------------------- > +obj-m := kprobes_gtsc.o > +KDIR := /lib/modules/$(shell uname -r)/build > +PWD := $(shell pwd) > +default: > + $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules > +clean: > + rm -f *.mod.c *.ko *.o > +----------------------------------CUT-------------------------------------- > +/* kprobes_gtsc.c - An example of using GTSC in a kprobes module */ > +#include <linux/kernel.h> > +#include <linux/module.h> > +#include <linux/kprobes.h> > +#include <linux/gtsc.h> > + > +#define PROBE_POINT "do_fork" > + > +static struct kprobe kp; > +struct trace_info *kprobes_trace; > + > +#define GTSC_PRINTF_TMPBUF_SIZE (1024) > +static char gtsc_printf_tmpbuf[NR_CPUS][GTSC_PRINTF_TMPBUF_SIZE]; > + Still doing an extra copy hey? :) > +/* This lock is needed only when using global relay buffers */ > +static DEFINE_SPINLOCK(gtsc_printf_lock); > + > +void gtsc_printf(struct trace_info *trace, const char *format, ...) > +{ > + va_list args; > + void *buf; > + int len; > + char *record; > + > + if (!trace) > + return; > + if (!trace_running(trace)) > + return; > + > + /* get a timestamp */ > + buf = gtsc_printf_tmpbuf[smp_processor_id()]; > + len = sprintf(buf,"[%lld] ", trace_timestamp()); > + > + /* get the data */ > + va_start(args, format); > + len += vsnprintf(buf+len, GTSC_PRINTF_TMPBUF_SIZE, format, args); > + va_end(args); > + > + /* > + * Locking can be eliminated by specifying per-cpu buffers > + * when calling trace_setup(). > + */ > + spin_lock(>sc_printf_lock); > + I really hope nobody is willing to call gtsc_printf from an interrupt handler.. :) Easy dead embrace in perspective. > + /* Send everything to the relay buffer */ > + if ((record = relay_reserve(trace->rchan, len))) > + memcpy(record, buf, len); > + > + spin_unlock(>sc_printf_lock); > +} > + > + > +int handler_pre(struct kprobe *p, struct pt_regs *regs) > +{ > + gtsc_printf(kprobes_trace,"%d\n", current->pid); > + return 0; > +} > + > +int init_module(void) > +{ > + int ret; > + /* setup gtsc, use a global relay buffer */ > + kprobes_trace = trace_setup("gtsc_example",PROBE_POINT, > + 1024,8, > + TRACE_GLOBAL_CHANNEL | TRACE_FLIGHT_CHANNEL); > + if (!kprobes_trace) > + return -1; > + > + trace_start(kprobes_trace); > + > + /* setup the kprobe */ > + kp.pre_handler = handler_pre; > + kp.post_handler = NULL; > + kp.fault_handler = NULL; > + kp.symbol_name = PROBE_POINT; > + if ((ret=register_kprobe(&kp)) < 0) > + gtsc_printf(kprobes_trace,"register_kprobe failed %d\n", ret); > + return 0; > +} > + > +void cleanup_module(void) > +{ > + unregister_kprobe(&kp); > + /* close down the gtsc channel */ > + trace_stop(kprobes_trace); > + trace_cleanup(kprobes_trace); > +} > + > +MODULE_LICENSE("GPL"); > +-----------------------------CUT-------------------------------------------- > +How to run the example: > +$ mount -t debugfs /debug > +$ make > +$ insmod kprobes_gtsc.ko > + > +To view the data produced by the module: > +$ cat /debug/gtsc_example/do_fork/trace0 > + > +Remove the module. > +$ rmmod kprobes_gtsc > + > +The function trace_cleanup() is called when the module > +is removed. This will cause the GTSC channel to be destroyed and the > +corresponding files to disappear from the debug file system. > + > +Credits > +======= > +GTSC adapted from blktrace authored by Jens Axboe ([EMAIL PROTECTED]). > + > +Major contributions to GTSC were made by: > +Tom Zanussi <[EMAIL PROTECTED]> > +Martin Hunt <[EMAIL PROTECTED]> > +David Wilder <[EMAIL PROTECTED]> > > > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/