This is a proposal to synchronize the start of sample collection throughout all places where samples are collected in RTLA, using a lockless method of tracking the count of active tracefs instances set to the timerlat tracer.
There are three different places where timerlat samples are collected in RTLA: two tracefs instances (auto-analysis and record), and a BPF program attached to the osnoise:timerlat_sample tracepoint. The BPF program collects all samples and generates the main statistics from them in either top or hist mode, while the auto-analysis and record instances are used as ring buffers with the intention of capturing the tail of both timerlat samples and additional tracepoints. In instances where BPF support is not available, the BPF program is replaced by a third tracefs instance, called the "main instance". One problem with this approach is each of these sample collectors is turned on [1] separately, in the present state of RTLA, in the following order: - record instance - auto-analysis instance - BPF program / main instance [1] By "turned on", I mean toggling tracing on (for tracefs instances) or attachment (for the BPF program). Note that timerlat starts measurement on tracer registration, even if tracing is off by default. This might have been originally unintentional - considering that explicitly turning off tracing stops the measurement threads - but is now relied upon in RTLA. This leads to some samples being seen only by tracers that are enabled earlier, leading to confusing results. For example, auto-analysis might show spikes that are not seen in the histogram, and trace output (record) might show samples that are not seen by either the histogram or auto-analysis. To enable RTLA to analyze samples consistently, the first patch adds two fields to the osnoise:timerlat_sample tracepoint: instances_registered and instances_on. During the recording of a timerlat sample, timerlat counts how many instances are registered and how many are on, and attaches the information to the osnoise:timerlat_sample trace event, which is moved to occur after the samples are recorded. The second patch makes RTLA count how many tracefs instances are to be enabled in total, passes this number to the BPF program, and makes it drop any samples that arrive before the instances are turned on. This ensures all samples recorded in the main statistics (e.g. histogram) are seen by all active tracefs instances, that is, auto-analysis and trace output (record). The third patch then moves the attachment of the BPF program before the enabling of the tracefs instances, provided that the kernel supports instance counting introduced in the first patch. The second patch enforces that all samples recorded by the program are seen by the last enabled tracefs instance. Synchronization between different tracefs instances, that is, auto-analysis and trace output, and the main instance in non-BPF mode, is possible by looking at the osnoise:timerlat_sample record that comes after each sample, but is not yet implemented in the current version of the patchset. Tomas Glozar (3): tracing/osnoise: Record timerlat instance counts rtla/timerlat_bpf: Filter samples unseen by tracer rtla/timerlat: Attach BPF program before tracers include/trace/events/osnoise.h | 14 ++++++--- kernel/trace/trace_osnoise.c | 10 +++++-- tools/tracing/rtla/src/timerlat.bpf.c | 9 ++++++ tools/tracing/rtla/src/timerlat.c | 43 +++++++++++++++++++++++++-- tools/tracing/rtla/src/timerlat.h | 2 ++ tools/tracing/rtla/src/timerlat_bpf.c | 8 +++++ 6 files changed, 78 insertions(+), 8 deletions(-) -- 2.52.0
