Hi everyone,
We're excited to announce the availability of the first release
candidate of LTTng 2.15, codenamed “Péché Mortel”!
Only a few months after LTTng 2.14, this release delivers an impressive
number of features considering the short time frame, in particular:
• CTF 2 (Common Trace Format) as the default trace format.
• Reduced memory footprint for user space tracing.
• Improved resiliency of shared LTTng-UST buffers.
• Redesigned lttng-list(1) command with memory usage reporting.
• Extended automation capabilities for the lttng-add-trigger(1) command.
Read on for the details.
WHAT'S NEW IN LTTng 2.15?
━━━━━━━━━━━━━━━━━━━━━━━━━
New features and changes in LTTng 2.15:
• Choose the trace format for each recording session with the new
`--trace-format` option of the lttng-create(1) command:
$ lttng create my-session --trace-format=ctf-1.8
Select between CTF 1.8 [1] and CTF 2 [2].
CTF 2 now is the default trace format for all newly created recording
sessions. Babeltrace 2 [3] supports CTF 2 since version 2.1 (January
2025). To keep the default set to CTF 1.8 for your application, use
the new `--default-trace-format` option of lttng-sessiond(8).
The LTTng project adopts CTF 2 because it has many benefits over
CTF 1.8:
Easier decoding:
CTF 1.8 requires parsing TSDL metadata, a CTF-specific language,
which increases implementation time and limits tool integration.
CTF 2 replaces this with JSON Text Sequences, readable by standard
JSON parsers, significantly reducing new decoding code.
Cleaner trace streaming:
CTF 1.8 allows partial metadata items, forcing parsers to detect
item boundaries manually, increasing complexity and bugs.
CTF 2 separates metadata fragments with an RS byte, enabling
simple chunk identification without decoding.
Better trace readability (not in LTTng yet):
CTF 1.8 lacks semantic annotations, so readers rely on external
domain knowledge to interpret event record data.
CTF 2 introduces user-defined attributes to describe semantics,
enabling automated analysis (recognizing IP addresses,
for example).
Broader type support (not in LTTng yet):
CTF 2 adds native field classes that can improve readability,
performance, and trace size, unlike CTF 1.8's more limited types.
Examples include a BLOB type for binary data, richer string
encodings, bit maps, optionals, and LEB128 integers.
This per-recording session configuration replaces the mechanism
introduced in LTTng 2.14 where setting the
`LTTNG_EXPERIMENTAL_FORCE_CTF_2` environment variable to `1` at
`lttng-sessiond` startup forced _all_ recording sessions to produce
CTF 2 traces.
• A new channel memory reclaim operation, exclusively available for user
space channels, can free memory used by eligible sub-buffers by
deallocating their backing memory when no longer needed.
The two ways to reclaim memory are:
Automatic reclaim:
Configure a per-channel, automatic reclaim policy which
periodically tries to reclaim memory for sub-buffers older than a
given age, or as soon as they're consumed, with the new
`--auto-reclaim-memory` option of the
lttng-enable-channel(1) command.
This new option is only available in discard mode (without the
`--overwrite` option).
Immediate reclaim:
The new lttng-reclaim-memory(1) command reclaims memory
immediately for eligible sub-buffers of one or more channels.
Only reclaim the sub-buffers older than a given age with the
`--older-than` option:
$ lttng reclaim-memory --older-than=2m
Both memory reclaim modes are possible because channel ring buffers
are now backed by sparse files [4] when the file system supports them.
• Choose the buffer _preallocation_ policy of a user space channel with
the new `--buffer-preallocation` option of the
lttng-enable-channel(1) command:
$ lttng enable-channel my-channel --userspace \
--buffer-preallocation=on-demand
LTTng can either preallocate all ring buffer memory up front (pre-2.15
behaviour; still the default) or allocate as needed. Choose
preallocation to ensure predictable memory usage and avoid run-time
allocation latency; choose on demand to minimize the initial footprint
for low traffic, for short-lived recording sessions, or in conjunction
with memory reclaims.
This feature is also enabled by sparse file-backed ring buffers.
• Before this release, an application terminated while writing to a user
space ring-buffer could leave a sub-buffer stuck in a
partially-written state, making it unusable.
A new health check mechanism, the _watchdog timer_, now periodically
monitors user space ring-buffers and makes stalled sub-buffers
consumable again to keep trace recording running smoothly.
This mechanism is enabled by default and can be controlled with the
new `--watchdog-timer` option of the lttng-enable-channel(1) command.
As of LTTng 2.15, it only applies to user space channels with a
per-user buffer ownership model.
• We completely revamped the human-readable output of the
lttng-list(1) command.
The command now has a clean tree structure. Colors help improve the
readability of objects and properties when the connected terminal
supports it: the `LTTNG_TERM_COLOR` and `NO_COLOR` [5] environment
variables provide further terminal color control.
✔ Channel `channel0` ❬42 event rules❭
🞂 Auto. memory reclaim policy: None
🞂 Loss mode: Discard newest event record
🞂 Preallocation policy: On demand
🞂 Ring buffer configuration: 4 sub-buffers of 512.00 KiB per CPU,
per Unix user
🞂 Timer periods:
🞂 Monitor timer: 1.00 s
🞂 Read timer: Inactive
🞂 Switch timer: Inactive
🞂 Watchdog timer: 2.00 s
🞂 Statistics:
🞂 Discarded event records: 23,182
By default, the command:
‣ Adds empty lines between blocks of related information. Remove those
empty lines with the new `compact` setting of the new
`--style` option.
‣ Truncates lines, adding an ellipsis, to fit the current terminal
width. Avoid truncation with the new `--no-truncate` option.
If you still prefer or need the previous (“legacy”) version, set the
`LTTNG_LIST_LEGACY` environment variable to `1` before you run
lttng-list(1). Please note, however, that the legacy output will never
show anything related to features introduced after LTTng 2.14.
• The lttng-list(1) command now shows the total memory usage of
each channel:
🞂 Memory usage: ❲●●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 19.07 MiB / 40.12 MiB
The new `--mem-usage` option controls the display mode of memory
usage: `compact` shows the memory usage for each Unix user or process
while `full` shows the memory usage for each CPU (if available):
🞂 Memory usage: ❲●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 13.30 MiB / 40.12 MiB
┆ For UID 1000 (64-bit): ❲●●●●●┄┄┄┄┄┄┄┄┄┄┄❳ 13.30 MiB / 40.12 MiB
┆ CPU 0: ❲●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 540.00 KiB / 2.51 MiB
┆ CPU 1: ❲●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 360.00 KiB / 2.51 MiB
┆ CPU 2: ❲●●●●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 1.30 MiB / 2.51 MiB
┆ CPU 3: ❲●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 256.00 KiB / 2.51 MiB
┆ CPU 4: ❲●●●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 1.20 MiB / 2.51 MiB
┆ CPU 5: ❲●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 1.07 MiB / 2.51 MiB
┆ CPU 6: ❲●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 768.00 KiB / 2.51 MiB
┆ CPU 7: ❲●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 524.00 KiB / 2.51 MiB
┆ CPU 8: ❲●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳ 792.00 KiB / 2.51 MiB
• The lttng-status(1) command now accepts the same options
as lttng-list(1):
$ lttng status --mem-usage=compact --channel=my-channel
lttng-status(1) forwards its options as is to lttng-list(1) for the
current recording session.
• The lttng(1) commands won't emit multi-byte UTF-8 sequences when the
`LTTNG_NO_UTF_8` environment variable is set to `1`, making a best
effort to find alternatives.
• New trigger conditions are now available for the
lttng-add-trigger(1) command:
`channel-buffer-usage-ge` and `channel-buffer-usage-le`:
Satisfied when the buffer usage of a given channel becomes
greater/less than or equal to some threshold (size in bytes or
ratio of the total channel buffer size):
$ lttng add-trigger --condition=channel-buffer-usage-ge \
--session=my-session --channel=my-channel \
--domain=kernel --threshold-ratio=0.75 \
--action=notify
`session-consumed-size-ge`:
Satisfied when the total consumed size of the tracing data of all
the channels of a recording session becomes greater than or equal
to a threshold:
$ lttng add-trigger --condition=session-consumed-size-ge \
--session=my-session --threshold-size=100M \
--action=snapshot-session other-session
`session-rotation-starts`:
`session-rotation-finishes`:
Satisfied when the rotation operation of a recording session
starts or finishes:
$ lttng add-trigger --condition=session-rotation-finishes \
--session=my-session --action=notify
• The `liblttng-ctl` C API is updated to support the new features above:
‣ Set the trace format of a recording session descriptor with
lttng_session_descriptor_set_trace_format() before creating the
recording session.
‣ Immediately reclaim user space channel memory for eligible
sub-buffers with lttng_reclaim_channel_memory().
Get the results with lttng_reclaim_handle_get_reclaimed_subbuffer_count()
and lttng_reclaim_handle_get_pending_subbuffer_count().
Optionally wait for pending sub-buffers to be reclaimed with
lttng_reclaim_handle_wait_for_completion().
When done, destroy the memory reclaim operation handle
with lttng_reclaim_handle_destroy().
‣ Set and get the automatic memory reclamation policy of a user space
channel with lttng_channel_set_automatic_memory_reclamation_policy()
and lttng_channel_get_automatic_memory_reclamation_policy().
‣ Set and get the buffer preallocation policy of a user space channel
with lttng_channel_set_preallocation_policy()
and lttng_channel_get_preallocation_policy().
‣ Set and get the watchdog timer period of a channel with
lttng_channel_set_watchdog_timer_interval()
and lttng_channel_get_watchdog_timer_interval().
‣ Enumerate and query the CPU ID and memory usage of the data streams
of a channel with the data stream info API:
⁃ lttng_channel_get_data_stream_info_sets()
⁃ lttng_data_stream_info_sets_get_count()
⁃ lttng_data_stream_info_sets_get_at_index()
⁃ lttng_data_stream_info_set_get_count()
⁃ lttng_data_stream_info_set_get_uid()
⁃ lttng_data_stream_info_set_get_pid()
⁃ lttng_data_stream_info_set_get_app_bitness()
⁃ lttng_data_stream_info_set_get_at_index()
⁃ lttng_data_stream_info_get_cpu_id()
⁃ lttng_data_stream_info_get_memory_usage()
⁃ lttng_data_stream_info_sets_destroy()
• The session daemon and relay daemon now support systemd's sd_notify()
protocol, enabling proper synchronization with dependent systemd units
by signaling when the daemons are ready to receive commands and when
they're stopping.
• The internal v2.15 communication protocol between LTTng-tools and
LTTng-UST is incompatible with the LTTng 2.14 one: you cannot use
LTTng-tools 2.15 with LTTng-UST 2.14 and vice versa, for example.
• The lttng(1) Zsh completion function now fully completes the
`add-trigger` command.
From now on, the Debian/Ubuntu packages of LTTng-tools will contain
the Zsh completion functions.
VERSION NAME
━━━━━━━━━━━━
This release is named after Péché Mortel, the iconic Imperial Stout from
Montréal's Dieu du Ciel! and a long-time team favorite.
Deep black and unapologetically intense, Péché Mortel pours with a
dense, mocha-tinted head and an aroma that immediately announces freshly
roasted coffee. Decadent and dangerously drinkable, it is the ideal
companion for long, cold winter evenings.
IMPORTANT LINKS
━━━━━━━━━━━━━━━
LTTng tarball:
• <https://lttng.org/files/lttng-tools/lttng-tools-2.15.0-rc1.tar.bz2>
• <https://lttng.org/files/lttng-ust/lttng-ust-2.15.0-rc1.tar.bz2>
• <https://lttng.org/files/lttng-modules/lttng-modules-2.15.0-rc1.tar.bz2>
LTTng website:
<https://lttng.org/>
LTTng 2.15 documentation:
<https://lttng.org/docs/v2.15>
Mailing list for support and development:
<https://lists.lttng.org/>
IRC channel:
`#lttng` on `irc.oftc.net`
Bug tracker:
<https://bugs.lttng.org/projects/lttng/>
GitHub organization:
<https://github.com/lttng>
Continuous integration:
• <https://ci.lttng.org/view/LTTng-tools/>
• <https://ci.lttng.org/view/LTTng-ust/>
• <https://ci.lttng.org/view/LTTng-modules/>
Code review:
• <https://review.lttng.org/q/project:lttng-tools>
• <https://review.lttng.org/q/project:lttng-ust>
• <https://review.lttng.org/q/project:lttng-modules>
REFERENCES
━━━━━━━━━━
[1]: https://diamon.org/ctf/v1.8.3/ (CTF v1.8.3 specification)
[2]: https://diamon.org/ctf/ (CTF 2 specification)
[3]: https://babeltrace.org/
[4]: https://en.wikipedia.org/wiki/Sparse_file
[5]: https://no-color.org/
--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
