[lttng-dev] [PATCH lttng-tools v2 1/5] Fix: probes should be compared strictly by events metadata
Currently, events are compared using names and signatures. Events
with different payloads but identical name and signatures could
lead to corrupted trace because the Session Daemon would consider them
identical and give them the same event ID.
Events should be compared using the name, loglevel, fields and
model_emf_uri to ensure that their respective metadata is the same.
Signed-off-by: Francis Deslauriers
---
src/bin/lttng-sessiond/Makefile.am | 3 +-
src/bin/lttng-sessiond/ust-field-utils.c | 289 +++
src/bin/lttng-sessiond/ust-field-utils.h | 29
src/bin/lttng-sessiond/ust-registry.c| 42 -
tests/unit/Makefile.am | 1 +
5 files changed, 354 insertions(+), 10 deletions(-)
create mode 100644 src/bin/lttng-sessiond/ust-field-utils.c
create mode 100644 src/bin/lttng-sessiond/ust-field-utils.h
diff --git a/src/bin/lttng-sessiond/Makefile.am
b/src/bin/lttng-sessiond/Makefile.am
index 413fe75..6fc1809 100644
--- a/src/bin/lttng-sessiond/Makefile.am
+++ b/src/bin/lttng-sessiond/Makefile.am
@@ -40,7 +40,8 @@ lttng_sessiond_SOURCES = utils.c utils.h \
if HAVE_LIBLTTNG_UST_CTL
lttng_sessiond_SOURCES += trace-ust.c ust-registry.c ust-app.c \
ust-consumer.c ust-consumer.h ust-thread.c \
- ust-metadata.c ust-clock.h agent-thread.c agent-thread.h
+ ust-metadata.c ust-clock.h agent-thread.c
agent-thread.h \
+ ust-field-utils.h ust-field-utils.c
endif
# Add main.c at the end for compile order
diff --git a/src/bin/lttng-sessiond/ust-field-utils.c
b/src/bin/lttng-sessiond/ust-field-utils.c
new file mode 100644
index 000..8388052
--- /dev/null
+++ b/src/bin/lttng-sessiond/ust-field-utils.c
@@ -0,0 +1,289 @@
+/*
+ * Copyright (C) 2018 - Francis Deslauriers
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License, version 2 only, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 51
+ * Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+ */
+
+#include
+#include
+
+#include "ust-field-utils.h"
+
+/*
+ * The ustctl_field is made of a combination of C basic types
+ * ustctl_basic_type and _ustctl_basic_type.
+ *
+ * ustctl_basic_type contains an enumeration describing the abstract type.
+ * _ustctl_basic_type does _NOT_ contain an enumeration describing the
+ * abstract type.
+ *
+ * A layer is needed to use the same code for both structures.
+ * When dealing with _ustctl_basic_type, we need to use the abstract type of
+ * the ustctl_type struct.
+ */
+
+/*
+ * Compare two ustctl_integer_type fields.
+ * Returns 1 if both are identical.
+ */
+static bool match_ustctl_field_integer(struct ustctl_integer_type *first,
+ struct ustctl_integer_type *second)
+{
+ if (first->size != second->size) {
+ goto no_match;
+ }
+ if (first->alignment != second->alignment) {
+ goto no_match;
+ }
+ if (first->signedness != second->signedness) {
+ goto no_match;
+ }
+ if (first->encoding != second->encoding) {
+ goto no_match;
+ }
+ if (first->base != second->base) {
+ goto no_match;
+ }
+ if (first->reverse_byte_order != second->reverse_byte_order) {
+ goto no_match;
+ }
+
+ return true;
+
+no_match:
+ return false;
+}
+
+/*
+ * Compare two _ustctl_basic_type fields known to be of type integer.
+ * Returns 1 if both are identical.
+ */
+static bool match_ustctl_field_integer_from_raw_basic_type(
+ union _ustctl_basic_type *first, union
_ustctl_basic_type *second)
+{
+ return match_ustctl_field_integer(&first->integer, &second->integer);
+}
+
+/*
+ * Compare two _ustctl_basic_type fields known to be of type enum.
+ * Returns 1 if both are identical.
+ */
+static bool match_ustctl_field_enum_from_raw_basic_type(
+ union _ustctl_basic_type *first, union
_ustctl_basic_type *second)
+{
+ /*
+* Compare enumeration ID. Enumeration ID is provided to the
application by
+* the session daemon before event registration.
+*/
+ if (first->enumeration.id != second->enumeration.id) {
+ goto no_match;
+ }
+
+ /*
+* Sanity check of the name and container type. Those were already
checked
+* during enum registratio
[lttng-dev] [PATCH lttng-tools v2 0/5] Support probes with the same name but different event payload
This patch set allows for multiple probes with the same name to be
hooked on the same callsite. Right now, the Session Daemon only
considers the name and signature of the events to determine if two
events are identical. This could lead to trace corruptions when two
probes would have the same name and signature but really different
binary trace layouts.
We now compare probes by doing a deep compare of every field. If two
events are _exactly_ the same and have the same metadata then the same
event ID will be used, if they are different a new event ID is created.
When used with its corresponding UST patch set[1], it allows for dynamic
library upgrade scenarios during tracing. The user can now dlopen a new
version of a provider library and dlclose an old version without
restarting the process.
This patch set also includes regression tests for both the deep
comparaison and the newly added dlclose capability.
[1]: https://github.com/frdeso/lttng-ust/tree/dlclose-support
Francis Deslauriers (5):
Fix: probes should be compared strictly by events metadata
Fix: calling ht_{hash, match}_enum with wrong argument
Tests: allow the use of regular expressions to match events
Tests: add function to validate the number of an event name in
metadata
Tests: add duplicated providers tests
configure.ac| 1 +
src/bin/lttng-sessiond/Makefile.am | 3 +-
src/bin/lttng-sessiond/ust-field-utils.c| 289
src/bin/lttng-sessiond/ust-field-utils.h| 29 +++
src/bin/lttng-sessiond/ust-registry.c | 46 +++-
tests/fast_regression | 1 +
tests/regression/ust/multi-lib/Makefile.am | 114 ++
tests/regression/ust/multi-lib/README | 21 ++
tests/regression/ust/multi-lib/callsites.c | 34 +++
tests/regression/ust/multi-lib/callsites.h | 21 ++
tests/regression/ust/multi-lib/multi-lib-test.c | 251
tests/regression/ust/multi-lib/probes.c | 18 ++
tests/regression/ust/multi-lib/probes.h | 195
tests/regression/ust/multi-lib/test_multi_lib | 262 +
tests/unit/Makefile.am | 1 +
tests/utils/utils.sh| 27 ++-
16 files changed, 1299 insertions(+), 14 deletions(-)
create mode 100644 src/bin/lttng-sessiond/ust-field-utils.c
create mode 100644 src/bin/lttng-sessiond/ust-field-utils.h
create mode 100644 tests/regression/ust/multi-lib/Makefile.am
create mode 100644 tests/regression/ust/multi-lib/README
create mode 100644 tests/regression/ust/multi-lib/callsites.c
create mode 100644 tests/regression/ust/multi-lib/callsites.h
create mode 100644 tests/regression/ust/multi-lib/multi-lib-test.c
create mode 100644 tests/regression/ust/multi-lib/probes.c
create mode 100644 tests/regression/ust/multi-lib/probes.h
create mode 100755 tests/regression/ust/multi-lib/test_multi_lib
--
2.7.4
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] [PATCH lttng-tools v2 3/5] Tests: allow the use of regular expressions to match events
Signed-off-by: Francis Deslauriers
---
tests/utils/utils.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/utils/utils.sh b/tests/utils/utils.sh
index e8dfcda..9bf1fcc 100644
--- a/tests/utils/utils.sh
+++ b/tests/utils/utils.sh
@@ -1457,7 +1457,7 @@ function validate_trace_exp()
which $BABELTRACE_BIN >/dev/null
skip $? -ne 0 "Babeltrace binary not found. Skipping trace validation"
- traced=$($BABELTRACE_BIN $trace_path 2>/dev/null | grep ${event_exp} |
wc -l)
+ traced=$($BABELTRACE_BIN $trace_path 2>/dev/null | grep
--extended-regexp ${event_exp} | wc -l)
if [ "$traced" -ne 0 ]; then
pass "Validate trace for expression '${event_exp}', $traced
events"
else
@@ -1476,7 +1476,7 @@ function validate_trace_only_exp()
which $BABELTRACE_BIN >/dev/null
skip $? -ne 0 "Babeltrace binary not found. Skipping trace matches"
- local count=$($BABELTRACE_BIN $trace_path | grep ${event_exp} | wc -l)
+ local count=$($BABELTRACE_BIN $trace_path | grep --extended-regexp
${event_exp} | wc -l)
local total=$($BABELTRACE_BIN $trace_path | wc -l)
if [ "$count" -ne 0 ] && [ "$total" -eq "$count" ]; then
--
2.7.4
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] [PATCH lttng-tools v2 5/5] Tests: add duplicated providers tests
Signed-off-by: Francis Deslauriers --- configure.ac| 1 + tests/fast_regression | 1 + tests/regression/ust/multi-lib/Makefile.am | 114 +++ tests/regression/ust/multi-lib/README | 21 ++ tests/regression/ust/multi-lib/callsites.c | 34 +++ tests/regression/ust/multi-lib/callsites.h | 21 ++ tests/regression/ust/multi-lib/multi-lib-test.c | 251 +++ tests/regression/ust/multi-lib/probes.c | 18 ++ tests/regression/ust/multi-lib/probes.h | 195 ++ tests/regression/ust/multi-lib/test_multi_lib | 262 10 files changed, 918 insertions(+) create mode 100644 tests/regression/ust/multi-lib/Makefile.am create mode 100644 tests/regression/ust/multi-lib/README create mode 100644 tests/regression/ust/multi-lib/callsites.c create mode 100644 tests/regression/ust/multi-lib/callsites.h create mode 100644 tests/regression/ust/multi-lib/multi-lib-test.c create mode 100644 tests/regression/ust/multi-lib/probes.c create mode 100644 tests/regression/ust/multi-lib/probes.h create mode 100755 tests/regression/ust/multi-lib/test_multi_lib diff --git a/configure.ac b/configure.ac index b6ea39c..e22872c 100644 --- a/configure.ac +++ b/configure.ac @@ -1090,6 +1090,7 @@ AC_CONFIG_FILES([ tests/regression/ust/buffers-pid/Makefile tests/regression/ust/periodical-metadata-flush/Makefile tests/regression/ust/multi-session/Makefile + tests/regression/ust/multi-lib/Makefile tests/regression/ust/overlap/Makefile tests/regression/ust/overlap/demo/Makefile tests/regression/ust/linking/Makefile diff --git a/tests/fast_regression b/tests/fast_regression index bbce068..f76b53d 100644 --- a/tests/fast_regression +++ b/tests/fast_regression @@ -21,6 +21,7 @@ regression/tools/regen-statedump/test_ust regression/ust/before-after/test_before_after regression/ust/buffers-pid/test_buffers_pid regression/ust/multi-session/test_multi_session +regression/ust/multi-lib/test_multi_lib regression/ust/nprocesses/test_nprocesses regression/ust/overlap/test_overlap regression/ust/java-jul/test_java_jul diff --git a/tests/regression/ust/multi-lib/Makefile.am b/tests/regression/ust/multi-lib/Makefile.am new file mode 100644 index 000..f78ce7f --- /dev/null +++ b/tests/regression/ust/multi-lib/Makefile.am @@ -0,0 +1,114 @@ +noinst_SCRIPTS = test_multi_lib +noinst_PROGRAMS = exec-with-callsites exec-without-callsites + +exec_with_callsites_SOURCES = multi-lib-test.c callsites.c +exec_with_callsites_LDFLAGS = -ldl -lpopt +exec_with_callsites_CFLAGS = -DHAS_CALLSITES=1 + +exec_without_callsites_SOURCES = multi-lib-test.c +exec_without_callsites_LDFLAGS = -ldl -lpopt -llttng-ust +exec_without_callsites_LDADD = probes.o +exec_without_callsites_CFLAGS = -DHAS_CALLSITES=0 + +PROBES_SRC=probes.c probes.h +PROBES_LDF=-shared -module -llttng-ust -avoid-version -rpath $(abs_builddir)/.libs/ +PROBES_CF=-c -I$(srcdir)/ + +probes.o: probes.c probes.h + $(CC) $(PROBES_CF) -o $@ $< + +noinst_LTLIBRARIES = libprobes_a.la libprobes_a_prime.la \ + libprobes_b.la libprobes_c.la libprobes_c_prime.la \ + libprobes_d.la libprobes_e.la libprobes_f.la \ + libprobes_g.la libprobes_h.la libprobes_i.la \ + libprobes_j.la libprobes_k.la libprobes_l.la \ + libprobes_m.la libprobes_n.la libprobes_o.la \ + libprobes_p.la + +noinst_LTLIBRARIES += libcallsites_1.la libcallsites_2.la + +CALLSITES_SRC=callsites.c callsites.h +CALLSITES_LDF=-shared -module -llttng-ust -avoid-version -rpath $(abs_builddir)/.libs/ +CALLSITES_CF=-c -I. + +libprobes_a_la_SOURCES = $(PROBES_SRC) +libprobes_a_la_LDFLAGS = $(PROBES_LDF) +libprobes_a_la_CFLAGS = $(PROBES_CF) -DACTIVATE_PROBES_A + +libprobes_a_prime_la_SOURCES = $(PROBES_SRC) +libprobes_a_prime_la_LDFLAGS = $(PROBES_LDF) +libprobes_a_prime_la_CFLAGS = $(PROBES_CF) -DACTIVATE_PROBES_A + +libprobes_b_la_SOURCES = $(PROBES_SRC) +libprobes_b_la_LDFLAGS = $(PROBES_LDF) +libprobes_b_la_CFLAGS = $(PROBES_CF) -DACTIVATE_PROBES_B + +libprobes_c_la_SOURCES = $(PROBES_SRC) +libprobes_c_la_LDFLAGS = $(PROBES_LDF) +libprobes_c_la_CFLAGS = $(PROBES_CF) -DACTIVATE_PROBES_C + +libprobes_c_prime_la_SOURCES = $(PROBES_SRC) +libprobes_c_prime_la_LDFLAGS = $(PROBES_LDF) +libprobes_c_prime_la_CFLAGS = $(PROBES_CF) -DACTIVATE_PROBES_C + +libprobes_d_la_SOURCES = $(PROBES_SRC) +libprobes_d_la_LDFLAGS = $(PROBES_LDF) +libprobes_d_la_CFLAGS = $(PROBES_CF) -DACTIVATE_PROBES_D + +libprobes_e_la_SOURCES = $(PROBES_SRC) +libprobes_e_la_LDFLAGS = $(PROBES_LDF) +libprobes_e_la_CFLAGS = $(PROBES_CF) -DACTIVATE_PROBES_E + +libprobes_f_la_SOURCES = $(PROBES_SRC) +libprobes_f_la_LDFLAGS = $(PROBES_LDF) +libprobes_f_la_CFLAGS = $(PROBES_CF) -DACTIVATE_PROBES_F + +libprobes_g_la_SOURCES = $(PROBES_
[lttng-dev] [PATCH lttng-ust] Fix: destructors do not run on probe provider dlclose
Calling dlclose on a probe provider library does not unregister the
probes from the callsites as the destructors are not executed.
The __tracepoints__disable_destructors weak symbol was exposed by probe
providers, liblttng-ust.so and liblttng-ust-tracepoint.so libraries. If
a probe provider was loaded first into the address space, its definition
would be binded to the symbol. All the subsequent libraries using the
symbol would use the existing definition of the symbol. Thus creating a
a situation where liblttng-ust.so or liblttng-ust-tracepoint.so would
have a dependency on the probe provider library.
This was preventing the dynamic loader from unloading the library as it
was still in use by other libraries. Because of this, the execution of
its destructors and the unregistration of the probes was postponed.
To overcome this issue, we no longer expose this symbol in the
tracepoint.h file to remove the explicit dependency of the probe
provider on the symbol. We instead use the existing dlopen handle on
liblttng-ust-tracepoint.so to get an handle on a getter and setter
functions for this value.
Signed-off-by: Francis Deslauriers
---
include/lttng/tracepoint.h | 31 ++-
liblttng-ust/tracepoint.c | 26 ++
2 files changed, 40 insertions(+), 17 deletions(-)
diff --git a/include/lttng/tracepoint.h b/include/lttng/tracepoint.h
index 39f2c4d..95f5de9 100644
--- a/include/lttng/tracepoint.h
+++ b/include/lttng/tracepoint.h
@@ -221,26 +221,13 @@ struct lttng_ust_tracepoint_dlopen {
void (*rcu_read_lock_sym_bp)(void);
void (*rcu_read_unlock_sym_bp)(void);
void *(*rcu_dereference_sym_bp)(void *p);
+ void (*tracepoint_set_destructors_disabled)(int is_disabled);
+ int (*tracepoint_get_destructors_disabled)(void);
};
extern struct lttng_ust_tracepoint_dlopen tracepoint_dlopen;
extern struct lttng_ust_tracepoint_dlopen *tracepoint_dlopen_ptr;
-/* Disable tracepoint destructors. */
-int __tracepoints__disable_destructors __attribute__((weak));
-
-/*
- * Programs that have threads that survive after they exit, and
- * therefore call library destructors, should disable the tracepoint
- * destructors by calling tracepoint_disable_destructors(). This will
- * leak the tracepoint instrumentation library shared object, leaving
- * its teardown to the operating system process teardown.
- */
-static inline void tracepoint_disable_destructors(void)
-{
- __tracepoints__disable_destructors = 1;
-}
-
/*
* These weak symbols, the constructor, and destructor take care of
* registering only _one_ instance of the tracepoints per shared-ojbect
@@ -335,7 +322,8 @@ __tracepoints__destroy(void)
return;
if (!tracepoint_dlopen_ptr)
tracepoint_dlopen_ptr = &tracepoint_dlopen;
- if (!__tracepoints__disable_destructors
+ if (tracepoint_dlopen_ptr->tracepoint_get_destructors_disabled
+ &&
!tracepoint_dlopen_ptr->tracepoint_get_destructors_disabled()
&& tracepoint_dlopen_ptr->liblttngust_handle
&& !__tracepoint_ptrs_registered) {
ret = dlclose(tracepoint_dlopen_ptr->liblttngust_handle);
@@ -423,6 +411,14 @@ __tracepoints__ptrs_init(void)
URCU_FORCE_CAST(int (*)(struct lttng_ust_tracepoint * const *),
dlsym(tracepoint_dlopen_ptr->liblttngust_handle,
"tracepoint_unregister_lib"));
+ tracepoint_dlopen_ptr->tracepoint_set_destructors_disabled =
+ URCU_FORCE_CAST(void (*)(int),
+ dlsym(tracepoint_dlopen_ptr->liblttngust_handle,
+ "tracepoint_set_destructors_disabled"));
+ tracepoint_dlopen_ptr->tracepoint_get_destructors_disabled =
+ URCU_FORCE_CAST(int (*)(void),
+ dlsym(tracepoint_dlopen_ptr->liblttngust_handle,
+ "tracepoint_get_destructors_disabled"));
__tracepoint__init_urcu_sym();
if (tracepoint_dlopen_ptr->tracepoint_register_lib) {
tracepoint_dlopen_ptr->tracepoint_register_lib(__start___tracepoints_ptrs,
@@ -444,7 +440,8 @@ __tracepoints__ptrs_destroy(void)
tracepoint_dlopen_ptr = &tracepoint_dlopen;
if (tracepoint_dlopen_ptr->tracepoint_unregister_lib)
tracepoint_dlopen_ptr->tracepoint_unregister_lib(__start___tracepoints_ptrs);
- if (!__tracepoints__disable_destructors
+ if (tracepoint_dlopen_ptr->tracepoint_get_destructors_disabled
+ &&
!tracepoint_dlopen_ptr->tracepoint_get_destructors_disabled()
&& tracepoint_dlopen_ptr->liblttngust_handle
&&
[lttng-dev] [PATCH lttng-ust v2] Fix: destructors do not run on probe provider dlclose
Calling dlclose on the probe provider library that first loaded
__tracepoints__disable_destructors in the symbol table does not
unregister the probes from the callsites as the destructors are not
executed.
The __tracepoints__disable_destructors weak symbol is exposed by probe
providers, liblttng-ust.so and liblttng-ust-tracepoint.so libraries. If
a probe provider is loaded first into the address space, its definition
is binded to the symbol. All the subsequent loaded libraries using the
symbol will use the existing definition of the symbol, thus creating a
situation where liblttng-ust.so or liblttng-ust-tracepoint.so depend on
the probe provider library.
This prevents the dynamic loader from unloading the library as it is
still in use by other libraries. Because of this, the execution of its
destructors and the unregistration of the probes is postponed.
To overcome this issue, we no longer expose this symbol in the
tracepoint.h file to remove the explicit dependency of the probe
provider on the symbol. We instead use the existing dlopen handle on
liblttng-ust-tracepoint.so to get handles on getter and setter functions
for this value.
Signed-off-by: Francis Deslauriers
---
include/lttng/tracepoint.h | 82 +++---
liblttng-ust/tracepoint.c | 35
2 files changed, 98 insertions(+), 19 deletions(-)
diff --git a/include/lttng/tracepoint.h b/include/lttng/tracepoint.h
index 39f2c4d..d285347 100644
--- a/include/lttng/tracepoint.h
+++ b/include/lttng/tracepoint.h
@@ -226,21 +226,6 @@ struct lttng_ust_tracepoint_dlopen {
extern struct lttng_ust_tracepoint_dlopen tracepoint_dlopen;
extern struct lttng_ust_tracepoint_dlopen *tracepoint_dlopen_ptr;
-/* Disable tracepoint destructors. */
-int __tracepoints__disable_destructors __attribute__((weak));
-
-/*
- * Programs that have threads that survive after they exit, and
- * therefore call library destructors, should disable the tracepoint
- * destructors by calling tracepoint_disable_destructors(). This will
- * leak the tracepoint instrumentation library shared object, leaving
- * its teardown to the operating system process teardown.
- */
-static inline void tracepoint_disable_destructors(void)
-{
- __tracepoints__disable_destructors = 1;
-}
-
/*
* These weak symbols, the constructor, and destructor take care of
* registering only _one_ instance of the tracepoints per shared-ojbect
@@ -265,6 +250,47 @@ struct lttng_ust_tracepoint_dlopen tracepoint_dlopen
struct lttng_ust_tracepoint_dlopen *tracepoint_dlopen_ptr
__attribute__((weak, visibility("hidden")));
+/*
+ * Tracepoint dynamic linkage handling (callbacks). Hidden visibility: shared
+ * across objects in a module/main executable. The callbacks are used to
+ * control and check if the destructors should be executed.
+ */
+struct lttng_ust_tracepoint_destructors_syms {
+ void (*tracepoint_enable_destructors)(void);
+ void (*tracepoint_disable_destructors)(void);
+ int (*tracepoint_get_destructors_state)(void);
+};
+
+extern struct lttng_ust_tracepoint_destructors_syms
tracepoint_destructors_syms;
+extern struct lttng_ust_tracepoint_destructors_syms
*tracepoint_destructors_syms_ptr;
+
+struct lttng_ust_tracepoint_destructors_syms tracepoint_destructors_syms
+ __attribute__((weak, visibility("hidden")));
+struct lttng_ust_tracepoint_destructors_syms *tracepoint_destructors_syms_ptr
+ __attribute__((weak, visibility("hidden")));
+
+static inline void tracepoint_disable_destructors(void)
+{
+ if (!tracepoint_dlopen_ptr)
+ tracepoint_dlopen_ptr = &tracepoint_dlopen;
+ if (!tracepoint_destructors_syms_ptr)
+ tracepoint_destructors_syms_ptr = &tracepoint_destructors_syms;
+ if (tracepoint_dlopen_ptr->liblttngust_handle
+ &&
tracepoint_destructors_syms_ptr->tracepoint_disable_destructors)
+
tracepoint_destructors_syms_ptr->tracepoint_disable_destructors();
+}
+
+static inline void tracepoint_enable_destructors(void)
+{
+ if (!tracepoint_dlopen_ptr)
+ tracepoint_dlopen_ptr = &tracepoint_dlopen;
+ if (!tracepoint_destructors_syms_ptr)
+ tracepoint_destructors_syms_ptr = &tracepoint_destructors_syms;
+ if (tracepoint_dlopen_ptr->liblttngust_handle
+ &&
tracepoint_destructors_syms_ptr->tracepoint_enable_destructors)
+
tracepoint_destructors_syms_ptr->tracepoint_enable_destructors();
+}
+
#ifndef _LGPL_SOURCE
static inline void lttng_ust_notrace
__tracepoint__init_urcu_sym(void);
@@ -335,8 +361,11 @@ __tracepoints__destroy(void)
return;
if (!tracepoint_dlopen_ptr)
tracepoint_dlopen_ptr = &tracepoint_dlopen;
- if (!__tracepoints__disable_destructors
- &
[lttng-dev] [PATCH lttng-ust v3] Fix: destructors do not run on probe provider dlclose
Calling dlclose on the probe provider library that first loaded
__tracepoints__disable_destructors in the symbol table does not
unregister the probes from the callsites as the destructors are not
executed.
The __tracepoints__disable_destructors weak symbol is exposed by probe
providers, liblttng-ust.so and liblttng-ust-tracepoint.so libraries. If
a probe provider is loaded first into the address space, its definition
is bound to the symbol. All the subsequent loaded libraries using the
symbol will use the existing definition of the symbol, thus creating a
situation where liblttng-ust.so or liblttng-ust-tracepoint.so depend on
the probe provider library.
This prevents the dynamic loader from unloading the library as it is
still in use by other libraries. Because of this, the execution of its
destructors and the unregistration of the probes is postponed.
To overcome this issue, we no longer expose this symbol in the
tracepoint.h file to remove the explicit dependency of the probe
provider on the symbol. We instead use the existing dlopen handle on
liblttng-ust-tracepoint.so to get handles on getter and setter functions
for this value.
Signed-off-by: Francis Deslauriers
---
include/lttng/tracepoint.h | 82 +++---
liblttng-ust/tracepoint.c | 61 ++
2 files changed, 124 insertions(+), 19 deletions(-)
diff --git a/include/lttng/tracepoint.h b/include/lttng/tracepoint.h
index 39f2c4d..d285347 100644
--- a/include/lttng/tracepoint.h
+++ b/include/lttng/tracepoint.h
@@ -226,21 +226,6 @@ struct lttng_ust_tracepoint_dlopen {
extern struct lttng_ust_tracepoint_dlopen tracepoint_dlopen;
extern struct lttng_ust_tracepoint_dlopen *tracepoint_dlopen_ptr;
-/* Disable tracepoint destructors. */
-int __tracepoints__disable_destructors __attribute__((weak));
-
-/*
- * Programs that have threads that survive after they exit, and
- * therefore call library destructors, should disable the tracepoint
- * destructors by calling tracepoint_disable_destructors(). This will
- * leak the tracepoint instrumentation library shared object, leaving
- * its teardown to the operating system process teardown.
- */
-static inline void tracepoint_disable_destructors(void)
-{
- __tracepoints__disable_destructors = 1;
-}
-
/*
* These weak symbols, the constructor, and destructor take care of
* registering only _one_ instance of the tracepoints per shared-ojbect
@@ -265,6 +250,47 @@ struct lttng_ust_tracepoint_dlopen tracepoint_dlopen
struct lttng_ust_tracepoint_dlopen *tracepoint_dlopen_ptr
__attribute__((weak, visibility("hidden")));
+/*
+ * Tracepoint dynamic linkage handling (callbacks). Hidden visibility: shared
+ * across objects in a module/main executable. The callbacks are used to
+ * control and check if the destructors should be executed.
+ */
+struct lttng_ust_tracepoint_destructors_syms {
+ void (*tracepoint_enable_destructors)(void);
+ void (*tracepoint_disable_destructors)(void);
+ int (*tracepoint_get_destructors_state)(void);
+};
+
+extern struct lttng_ust_tracepoint_destructors_syms
tracepoint_destructors_syms;
+extern struct lttng_ust_tracepoint_destructors_syms
*tracepoint_destructors_syms_ptr;
+
+struct lttng_ust_tracepoint_destructors_syms tracepoint_destructors_syms
+ __attribute__((weak, visibility("hidden")));
+struct lttng_ust_tracepoint_destructors_syms *tracepoint_destructors_syms_ptr
+ __attribute__((weak, visibility("hidden")));
+
+static inline void tracepoint_disable_destructors(void)
+{
+ if (!tracepoint_dlopen_ptr)
+ tracepoint_dlopen_ptr = &tracepoint_dlopen;
+ if (!tracepoint_destructors_syms_ptr)
+ tracepoint_destructors_syms_ptr = &tracepoint_destructors_syms;
+ if (tracepoint_dlopen_ptr->liblttngust_handle
+ &&
tracepoint_destructors_syms_ptr->tracepoint_disable_destructors)
+
tracepoint_destructors_syms_ptr->tracepoint_disable_destructors();
+}
+
+static inline void tracepoint_enable_destructors(void)
+{
+ if (!tracepoint_dlopen_ptr)
+ tracepoint_dlopen_ptr = &tracepoint_dlopen;
+ if (!tracepoint_destructors_syms_ptr)
+ tracepoint_destructors_syms_ptr = &tracepoint_destructors_syms;
+ if (tracepoint_dlopen_ptr->liblttngust_handle
+ &&
tracepoint_destructors_syms_ptr->tracepoint_enable_destructors)
+
tracepoint_destructors_syms_ptr->tracepoint_enable_destructors();
+}
+
#ifndef _LGPL_SOURCE
static inline void lttng_ust_notrace
__tracepoint__init_urcu_sym(void);
@@ -335,8 +361,11 @@ __tracepoints__destroy(void)
return;
if (!tracepoint_dlopen_ptr)
tracepoint_dlopen_ptr = &tracepoint_dlopen;
- if (!__tracepoints__disable_destructors
- &
[lttng-dev] [PATCH lttng-ust v4] Fix: destructors do not run on probe provider dlclose
Calling dlclose on the probe provider library that first loaded
__tracepoints__disable_destructors in the symbol table does not
unregister the probes from the callsites as the destructors are not
executed.
The __tracepoints__disable_destructors weak symbol is exposed by probe
providers, liblttng-ust.so and liblttng-ust-tracepoint.so libraries. If
a probe provider is loaded first into the address space, its definition
is bound to the symbol. All the subsequent loaded libraries using the
symbol will use the existing definition of the symbol, thus creating a
situation where liblttng-ust.so or liblttng-ust-tracepoint.so depend on
the probe provider library.
This prevents the dynamic loader from unloading the library as it is
still in use by other libraries. Because of this, the execution of its
destructors and the unregistration of the probes is postponed.
To overcome this issue, we no longer expose this symbol in the
tracepoint.h file to remove the explicit dependency of the probe
provider on the symbol. We instead use the existing dlopen handle on
liblttng-ust-tracepoint.so to get handles on getter and setter functions
for this value.
Signed-off-by: Francis Deslauriers
---
include/lttng/tracepoint.h | 82 +++---
liblttng-ust/tracepoint.c | 56 +++
2 files changed, 119 insertions(+), 19 deletions(-)
diff --git a/include/lttng/tracepoint.h b/include/lttng/tracepoint.h
index 39f2c4d..d285347 100644
--- a/include/lttng/tracepoint.h
+++ b/include/lttng/tracepoint.h
@@ -226,21 +226,6 @@ struct lttng_ust_tracepoint_dlopen {
extern struct lttng_ust_tracepoint_dlopen tracepoint_dlopen;
extern struct lttng_ust_tracepoint_dlopen *tracepoint_dlopen_ptr;
-/* Disable tracepoint destructors. */
-int __tracepoints__disable_destructors __attribute__((weak));
-
-/*
- * Programs that have threads that survive after they exit, and
- * therefore call library destructors, should disable the tracepoint
- * destructors by calling tracepoint_disable_destructors(). This will
- * leak the tracepoint instrumentation library shared object, leaving
- * its teardown to the operating system process teardown.
- */
-static inline void tracepoint_disable_destructors(void)
-{
- __tracepoints__disable_destructors = 1;
-}
-
/*
* These weak symbols, the constructor, and destructor take care of
* registering only _one_ instance of the tracepoints per shared-ojbect
@@ -265,6 +250,47 @@ struct lttng_ust_tracepoint_dlopen tracepoint_dlopen
struct lttng_ust_tracepoint_dlopen *tracepoint_dlopen_ptr
__attribute__((weak, visibility("hidden")));
+/*
+ * Tracepoint dynamic linkage handling (callbacks). Hidden visibility: shared
+ * across objects in a module/main executable. The callbacks are used to
+ * control and check if the destructors should be executed.
+ */
+struct lttng_ust_tracepoint_destructors_syms {
+ void (*tracepoint_enable_destructors)(void);
+ void (*tracepoint_disable_destructors)(void);
+ int (*tracepoint_get_destructors_state)(void);
+};
+
+extern struct lttng_ust_tracepoint_destructors_syms
tracepoint_destructors_syms;
+extern struct lttng_ust_tracepoint_destructors_syms
*tracepoint_destructors_syms_ptr;
+
+struct lttng_ust_tracepoint_destructors_syms tracepoint_destructors_syms
+ __attribute__((weak, visibility("hidden")));
+struct lttng_ust_tracepoint_destructors_syms *tracepoint_destructors_syms_ptr
+ __attribute__((weak, visibility("hidden")));
+
+static inline void tracepoint_disable_destructors(void)
+{
+ if (!tracepoint_dlopen_ptr)
+ tracepoint_dlopen_ptr = &tracepoint_dlopen;
+ if (!tracepoint_destructors_syms_ptr)
+ tracepoint_destructors_syms_ptr = &tracepoint_destructors_syms;
+ if (tracepoint_dlopen_ptr->liblttngust_handle
+ &&
tracepoint_destructors_syms_ptr->tracepoint_disable_destructors)
+
tracepoint_destructors_syms_ptr->tracepoint_disable_destructors();
+}
+
+static inline void tracepoint_enable_destructors(void)
+{
+ if (!tracepoint_dlopen_ptr)
+ tracepoint_dlopen_ptr = &tracepoint_dlopen;
+ if (!tracepoint_destructors_syms_ptr)
+ tracepoint_destructors_syms_ptr = &tracepoint_destructors_syms;
+ if (tracepoint_dlopen_ptr->liblttngust_handle
+ &&
tracepoint_destructors_syms_ptr->tracepoint_enable_destructors)
+
tracepoint_destructors_syms_ptr->tracepoint_enable_destructors();
+}
+
#ifndef _LGPL_SOURCE
static inline void lttng_ust_notrace
__tracepoint__init_urcu_sym(void);
@@ -335,8 +361,11 @@ __tracepoints__destroy(void)
return;
if (!tracepoint_dlopen_ptr)
tracepoint_dlopen_ptr = &tracepoint_dlopen;
- if (!__tracepoints__disable_destructors
- &
[lttng-dev] [CFP] Tracing Summit 2018 Call for Presentations, October 25th, 2018, Edinburgh, UK
Hi, This is a Call for Presentations for the Tracing Summit 2018 which will be held in Edinburgh, UK on October 25th, 2018, at the Edinburgh International Conference Centre, co-located with the Open Source Summit Europe & Embedded Linux Conference Europe. The Tracing Summit is single-day, single-track conference on the topic of tracing. The event focuses on the field of software and hardware tracing, gathering developers and end-users of tracing and trace analysis tools. The main goal of the Tracing Summit is to provide space for discussion between people of the various areas that benefit from tracing, namely parallel, distributed and/or real-time systems, as well as kernel development. We are welcoming 30 minutes presentations from both users and developers, on topics covering, but not limited to: * Investigation workflow of Real-Time, latency, and throughput issues, * Trace collection and extraction, * Trace filtering, * Trace aggregation, * Trace formats, * Tracing multi-core systems, * Trace abstraction, * Trace modeling, * Automated trace analysis (e.g. dependency analysis), * Tracing large clusters and distributed systems, * Hardware-level tracing (e.g. DSP, GPU, bare-metal), * Trace visualisation, * Interaction between debugging and tracing, * Tracing remote control, * Analysis of large trace datasets, * Cloud trace collection and analysis, * Integration between trace tools, * Live tracing & monitoring. Those can cover recently available technologies, ongoing work, and yet non-existing technologies (which are compellingly interesting to end-users). Please understand that this open forum is not the proper place to present sales or marketing pitches, nor technologies which are prevented from being freely used in open source. If you are interested in presenting, please submit a proposal to submiss...@tracingsummit.org before *September 1st, 2018, at 23:59 EST*. Please provide the following information: * Talk title, * Talk abstract, (900 characters maximum) * Description of the targeted audience (900 characters maximum), * Short biography (900 characters maximum). Attending the Tracing Summit is free of charge. More details on registration will be available shortly, be sure to checkout the Tracing Summit 2018 website for all the latest information on this topic: https://www.tracingsummit.org/wiki/TracingSummit2018 The Tracing Summit is organized by the Linux Foundation Diagnostic and Monitoring Workgroup (https://diamon.org). The Tracing Summit is sponsored by EfficiOS <https://www.efficios.com/>. Please let us know if your company is interested in sponsoring this event. Thank you, On behalf of the Diagnostic and Monitoring Workgroup, Francis Deslauriers & Mathieu Desnoyers ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] [CFP] Call For Presentations reminder and registration for Tracing Summit 2018 on Oct 25th in Edinburgh, UK
Hi all, This is a friendly reminder that the deadline for submitting talk proposals for Tracing Summit 2018[1] is coming up soon (Sept 1st). Submit you talk today using submiss...@tracingsummit.org and don't forget to include the following: - Talk title, - Talk abstract (900 characters maximum), - Description of the targeted audience (900 characters maximum), - Short biography (900 characters maximum). The Tracing Summit 2018 will be held in Edinburgh, UK on October 25th, 2018, at the Edinburgh International Conference Centre, co-located with the Open Source Summit & Embedded Linux Conference 2018. You can register right now either through the Open Source Summit Europe registration page[2] by adding the free Tracing Summit addon or by registering specifically for the Tracing Summit using this registration form[3]. Thank you, On behalf of the Diagnostic and Monitoring Workgroup, Francis Deslauriers [1] https://tracingsummit.org/wiki/TracingSummit2018 [2] https://www.regonline.com/OSSEU2018 [3] https://regonline.com/tracingsummit18 ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] [PATCH lttng-ust] Move symbol preventing unloading of probe providers
Issue
=
Calling dlclose on the probe provider library that first loaded
__tracepoints__disable_destructors in the symbol table does not
unregister the probes from the callsites as the destructors are not
executed.
The __tracepoints__disable_destructors weak symbol is exposed by probe
providers, liblttng-ust.so and liblttng-ust-tracepoint.so libraries. If
a probe provider is loaded first into the address space, its definition
is bound to the symbol. All the subsequent loaded libraries using the
symbol will use the existing definition of the symbol, thus creating a
situation where liblttng-ust.so or liblttng-ust-tracepoint.so depend on
the probe provider library.
This prevents the dynamic loader from unloading the library as it is
still in use by other libraries. Because of this, the execution of its
destructors and the unregistration of the probes is postponed. Since the
unregistration of the probes is postponed, event will be generate if the
callsite is executed even though the probes should not be loaded.
Solution
To overcome this issue, we no longer expose this symbol in the
tracepoint.h file to remove the explicit dependency of the probe
provider on the symbol. We instead use the existing dlopen handle on
liblttng-ust-tracepoint.so and use dlsym to get handles on functions
that disable and get the state of the destructors.
Version compatibility
=
- This change is backward compatible with UST applications and libraries
built on lttng-ust version before 2.11. Those applications will use
the __tracepoints__disable_destructors symbol that is now only exposed
by the liblttng-ust-tracepoint.so library. This symbol is alway
checked in 2.11 in case an old app is running.
- Applications built with this change will also work in older versions
of lttng-ust as there is a check to see if the new destructor state
checking method should be used, if it should not we fallback to a
compatibility method. To ensure compatibility in this case, we also
look up and keep up to date the __tracepoints__disable_destructors
value using the dlopen-dlsym combo.
Signed-off-by: Francis Deslauriers
---
include/lttng/tracepoint.h | 108 ++---
liblttng-ust/tracepoint.c | 40 +
2 files changed, 123 insertions(+), 25 deletions(-)
diff --git a/include/lttng/tracepoint.h b/include/lttng/tracepoint.h
index 39f2c4d..9b52bc6 100644
--- a/include/lttng/tracepoint.h
+++ b/include/lttng/tracepoint.h
@@ -226,21 +226,6 @@ struct lttng_ust_tracepoint_dlopen {
extern struct lttng_ust_tracepoint_dlopen tracepoint_dlopen;
extern struct lttng_ust_tracepoint_dlopen *tracepoint_dlopen_ptr;
-/* Disable tracepoint destructors. */
-int __tracepoints__disable_destructors __attribute__((weak));
-
-/*
- * Programs that have threads that survive after they exit, and
- * therefore call library destructors, should disable the tracepoint
- * destructors by calling tracepoint_disable_destructors(). This will
- * leak the tracepoint instrumentation library shared object, leaving
- * its teardown to the operating system process teardown.
- */
-static inline void tracepoint_disable_destructors(void)
-{
- __tracepoints__disable_destructors = 1;
-}
-
/*
* These weak symbols, the constructor, and destructor take care of
* registering only _one_ instance of the tracepoints per shared-ojbect
@@ -265,6 +250,39 @@ struct lttng_ust_tracepoint_dlopen tracepoint_dlopen
struct lttng_ust_tracepoint_dlopen *tracepoint_dlopen_ptr
__attribute__((weak, visibility("hidden")));
+/*
+ * Tracepoint dynamic linkage handling (callbacks). Hidden visibility: shared
+ * across objects in a module/main executable. The callbacks are used to
+ * control and check if the destructors should be executed.
+ */
+struct lttng_ust_tracepoint_destructors_syms {
+ int *old_tracepoint_disable_destructors;
+ void (*tracepoint_disable_destructors)(void);
+ int (*tracepoint_get_destructors_state)(void);
+};
+
+extern struct lttng_ust_tracepoint_destructors_syms
tracepoint_destructors_syms;
+extern struct lttng_ust_tracepoint_destructors_syms
*tracepoint_destructors_syms_ptr;
+
+struct lttng_ust_tracepoint_destructors_syms tracepoint_destructors_syms
+ __attribute__((weak, visibility("hidden")));
+struct lttng_ust_tracepoint_destructors_syms *tracepoint_destructors_syms_ptr
+ __attribute__((weak, visibility("hidden")));
+
+static inline void tracepoint_disable_destructors(void)
+{
+ if (!tracepoint_dlopen_ptr)
+ tracepoint_dlopen_ptr = &tracepoint_dlopen;
+ if (!tracepoint_destructors_syms_ptr)
+ tracepoint_destructors_syms_ptr = &tracepoint_destructors_syms;
+ if (tracepoint_dlopen_ptr->liblttngust_handle
+ &&
tracepoint_destructors_syms_ptr->tracepoint_disable_destructors) {
+
tracepoint_
[lttng-dev] Tracing Summit 2018 - CFP extended until Sept. 10th
Hi all, To take into account that a lot of people are on vacations over the month of August, we are announcing the extension of the CFP for the Tracing Summit 2018 until Monday September 10th. This gives 6 more days to get your talk proposals in. Thank you to all of you who submitted talks. We received great ones! More info: https://tracingsummit.org/wiki/TracingSummit2018 Thank you, -- On behalf of the Diagnostic and Monitoring Workgroup, Francis Deslauriers ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] Tracing Summit 2018 Schedule Released
Hi all, We're happy to announce the schedule for the Tracing Summit 2018 is now available. We're going to have an amazing conference with diverse topics such as trace analysis, hardware tracing, and distributed systems tracing. Check out the schedule and talk abstracts on the Tracing Summit wiki. [1] Tracing Summit 2018 will be held in Edinburgh, UK on October 25th, 2018, Don't forget to register for the summit, it's free! [2] See you in October! [1]: https://tracingsummit.org/wiki/TracingSummit2018#Schedule [2]: https://tracingsummit.org/wiki/TracingSummit2018#Registration -- Francis Deslauriers Software developer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] [BETA RELEASE] LTTng Scope 0.4
Hi all,
We are announcing the release of LTTng Scope 0.4.0.
This release has a few improvements over LTTng Scope 0.3:
* Open more than one trace thanks to the new "Trace Project Setup"
window. (e.g. Kernel + UST traces)
You can always reopen this window to add traces to or remove traces
from the current trace project by clicking
"File" -> "Project Setup..." or by right-clicking the project item in
the left pane and clicking "Project Setup".
* Event count chart: use vertical bars instead of an area chart. A given
bar indicates the number of event records of which the timestamps are
between the beginning and the end of the bar.
* Event count chart: use natural ticks, depending on the current zoom
level, on the horizontal (time) axis. For example, the difference
between individual ticks can be 100 ms, 500 ms, 1 s, 2 s, 15 s, 30
minutes, 1 day, etc.
* Event count chart: add axis units ("Count / Time") below the view's
title on the left.
* Add the version to the main window's title bar.
* Bug fix: refresh the views when you move vertical dividers or when you
resize the main window.
* Bug fix: event count chart: round tick labels correctly to avoid
noticing floating point number errors.
Of course, because LTTng Scope's development is still in the beta phase,
you could discover bugs when trying the tool. Feel free to report them
on GitHub: <https://github.com/lttng/lttng-scope/issues>.
Project's description and screenshot:
<https://lttng.org/beta/#lttng-scope>
GitHub page:
<https://github.com/lttng/lttng-scope>
Installer for Linux:
<https://lttng.org/files/lttng-scope/lttng-scope-0.4.0-linux-x64-installer.run>
Installer for Windows:
<https://lttng.org/files/lttng-scope/lttng-scope-0.4.0-windows-installer.exe>
Installer for macOS:
<https://lttng.org/files/lttng-scope/lttng-scope-0.4.0-osx-installer.dmg>
On the behalf of the EfficiOS team,
--
Francis Deslauriers
Software developer
EfficiOS inc.
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] Videos of the Tracing Summit 2018
Hi, The videos of the Tracing Summit 2018 are now online: https://www.youtube.com/playlist?list=PLuo4E47p5_7aBCdobf_XpEPcDFNPhpRNs You can also access them from the schedule here: https://tracingsummit.org/wiki/TracingSummit2018#Schedule -- On behalf of the Diagnostic and Monitoring Workgroup, Francis Deslauriers ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] "Hands-free" tracepoints using LD_PRELOAD
Hi Brian, If I understand correctly, you want developers to provide a whitelist of functions that they want to instrument in an arbitrary shared library, and then have a system that instruments those functions without having to recompile the library with LTTng-UST tracepoints. Can you give us an example of annotation that the shim developer would provide? Do you need to support the extraction of function arguments? If you don't, here are two straightforward and tested approaches to solve your problem: First, overriding target symbols using LD_PRELOAD (like you suggested above): 1. Create boilerplate script #1 that generates tracepoints definitions [1] based on function name, 2. Create boilerplate script #2 that generates a function calling the right tracepoint and the original target function [2], 3. For each target symbol "bar", using script #1, create a tracepoint definition "libfoo:bar_tp", and compile them into a tracepoints.o. object file, 4. For each target symbol "bar", using script #2, create a function "bar" that calls the bar_tp tracepoint (e.g. tracepoint(libfoo, bar_tp) ) and then calls the original bar function in libfoo.so (using dlsym(3)), and compile those callsites into a callsites.o object file, 5. Create a shared library libshim.so using tracepoints.o and callsites.o object files and link it to lttng-ust lib and the dynamic loader lib using :"-llttng-ust -ldl", Those steps results in the creation of a libshim.so file that contains the "bar" overriding function (containing the tracepoint callsite) that will be called instead of "bar" function in the libfoo.so library when LD_PRELOADing it (e.g. LD_PRELOAD=libshim.so ./fooapp). Here is the Makefile I used to test this [3] to give you a sense of the pipeline. Second, using the newly added --userspace-probe instrumentation: This option allows you to instrument any ELF symbol without recompiling or restarting the target process. For example, to instrument the "bar" function in the /tmp/libfoo.so file you could run to following command: > lttng enable-event -k --userspace-probe="elf:/tmp/libfoo.so:bar" $event_name You could write a script that takes the list of target functions and adds userspace-probes on each of them. This approach has the advantage that you can also instrument functions in the binary and not only functions in shared libraries. This instrumentation type uses the kernel tracer and thus requires using the lttng-modules. If you do need to save the function arguments, I believe it's possible using the first approach, by tweaking the two boilerplate scripts depending on what are the annotations you want to provide. Supporting argument extraction with userspace-probes is part of our long term goal and would be a really powerful feature but will require a significant engineering effort to accomplish. [1] : https://lttng.org/docs/v2.10/#doc-tpp-header [2]: https://lttng.org/docs/v2.10/#doc-probing-the-application-source-code [3]: http://paste.ubuntu.com/p/5fBSgRqXsB/ I hope this make sense. Please don't hesitate if you want more details. Thank you, Francis Le ven. 18 janv. 2019, à 16 h 04, Brian Rossa a écrit : > > Hello, > > I have a custom tracing solution that I'm interested in porting to LTTng. It > works as follows: > > Input is a whitelist of mangled names taken from a target library, libfoo.so. > For each name, generate a logging shim (example) for the corresponding call. > Compile all the logging shims into a shared library, libshim.so. > Trace any application using libfoo by running it as LD_PRELOAD=libshim.so > ./fooapp > > There are two nice things about this approach that I would like to preserve: > > The shim developer only needs to provide the whitelist and a bit of extra > annotation. From there, the shim can be be authored using a boilerplate > generator. > The app developer doesn't have to do anything other than pass the LD_PRELOAD > flag. > > The downside is that the only tracepoints are those corresponding to the > whitelist, but I'm fine with that. > > Can this kind of "hands-free" developer experience be supported by LTTng? > > Thanks! > ~br > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] "Hands-free" tracepoints using LD_PRELOAD
- The whitelist of mangled names >> >- Implementations of struct "wrappers" that provide custom ostream >> >operators >> >- A map between type names and wrapper names >> > >> > The machinery here seems fairly general-purpose, but I don't presume to be >> > an expert. My implementation is somewhat error-prone, and my main hope in >> > reaching out to the mailing list was that LTTng already had some of these >> > steps better-implemented. >> >> AFAIK, lttng does not have an equivalent. >> >> > Step #2 is particularly problematic due to >> > ambiguities in the mangling grammar, and will need support going forward to >> > generalize well. >> >> What is the status of this step in your project? >> >> What are the problems that make your implementation "error-pone"? >> >> Would you mind linking us to said project so we can have a look? >> >> I would be interested in seeing at first lttng tracepoint used as Francis >> demonstrated and see from there were this project can go. >> >> > >> > I would be happy to contribute some or all of my implementation if it's >> > something that the LTTng community would be interested in supporting and >> > extending. >> >> We are clearly open for discussion and helping you improve the project. I am >> not >> so sure on supporting and extending it. Others might have a different >> opinion. >> >> >> Cheers >> >> -- >> Jonathan Rajotte-Julien >> EfficiOS -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Tracepoints firing without being enabled in LTTng
Hi Mosleh, It's expected to not see any event when running lttng list -u if no lttng-ust instrumented applications are running. What are the exact lttng commands you are running to see those unexpected events? Thank you, Francis Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin a écrit : > > Hello, > > I am having a strange issue with my current application. For some context, I > have integrated LTTng into a Yocto build and put custom trace points in my > embedded system application. Once I load the built image onto my system and > start a session daemon, I try to list all available userspace events (using > lttng list -u), but no events show up. > > The strange occurrence is when I create a session (lltng create my-sess) and > not start tracing the trace events will display on the screen regardless > (when starting session in live mode with lttng view). I was wondering if > anyone has any insight on this issue I am having. > > Thanks for any assistance, > Mosleh > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Tracepoints firing without being enabled in LTTng
Adding back lttng-dev mailing list. So you don't even run a lttng-enable-event command? That's odd. Can you send us the output of the following commands: Is it possible that there are other lttng-sessiond daemons running on the machine? > ps aux | grep "lttng-sessiond" Verify if there are other active tracing sessions: > lttng list Check the details of the current session: > lttng status Thank you, Francis Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin a écrit : > > Hi Francis, > > Thank you for the quick response. > Just to clear things up, my application is running when I run that command. > > The commands I am running are: > > lttng-sessiond -d > > lttng create my-sess --live > After this command I start my application. If I then do the command: > > lttng view > I will see the tracepoints in the application in the terminal even though I > have not done: lttng start. > > I thought it was strange that I cannot find my application in the: lttng list > command under any domain. > Also the reason I start my application after creating a session is because > the system automatically starts the application using systemd, > in order to get tracing data at all I have to manually stop the application > and start it again after creating a session. > > Thanks again for any help, > Mosleh > > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers > wrote: >> >> Hi Mosleh, >> It's expected to not see any event when running lttng list -u if no >> lttng-ust instrumented applications are running. >> >> What are the exact lttng commands you are running to see those >> unexpected events? >> >> Thank you, >> Francis >> >> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin a >> écrit : >> > >> > Hello, >> > >> > I am having a strange issue with my current application. For some context, >> > I have integrated LTTng into a Yocto build and put custom trace points in >> > my embedded system application. Once I load the built image onto my system >> > and start a session daemon, I try to list all available userspace events >> > (using lttng list -u), but no events show up. >> > >> > The strange occurrence is when I create a session (lltng create my-sess) >> > and not start tracing the trace events will display on the screen >> > regardless (when starting session in live mode with lttng view). I was >> > wondering if anyone has any insight on this issue I am having. >> > >> > Thanks for any assistance, >> > Mosleh >> > ___ >> > lttng-dev mailing list >> > lttng-dev@lists.lttng.org >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> >> -- >> Francis Deslauriers >> Computer Engineer >> EfficiOS inc. -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Tracepoints firing without being enabled in LTTng
Can you start the lttng-sessiond in verbose mode using the following command: > lttng-sessiond -vvv and then do the lttng create, start of application and lttng view on another console. Then copy the output of the sessiond on https://pastebin.com/ and send us the link. Thank you, Francis Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin a écrit : > That's correct, I don't enable any events or do: lttng start. > Here is some snapshots of starting everything up, followed by starting my > application. > [image: image.png] > > Here are the outputs for the commands: > > [image: image.png] > > [image: image.png] > > [image: image.png] > > Note that even though the session is inactive, when a trace event occurs > in my application it is being recorded in the appropriate directory: > > [image: image.png] > > Also maybe important, the system has root only user. Would having the > application belonging to a different user and trying to run as root cause > these kinds of complications? I did notice when testing on a raspberry pi > that running an application as a root user would mean running sessions as > root user as well. > > Thanks, > Mosleh > > On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers < > francis.deslauri...@efficios.com> wrote: > >> Adding back lttng-dev mailing list. >> >> So you don't even run a lttng-enable-event command? >> That's odd. >> >> Can you send us the output of the following commands: >> Is it possible that there are other lttng-sessiond daemons running on >> the machine? >> > ps aux | grep "lttng-sessiond" >> Verify if there are other active tracing sessions: >> > lttng list >> Check the details of the current session: >> > lttng status >> >> Thank you, >> Francis >> >> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin a >> écrit : >> > >> > Hi Francis, >> > >> > Thank you for the quick response. >> > Just to clear things up, my application is running when I run that >> command. >> > >> > The commands I am running are: >> > > lttng-sessiond -d >> > > lttng create my-sess --live >> > After this command I start my application. If I then do the command: >> > > lttng view >> > I will see the tracepoints in the application in the terminal even >> though I have not done: lttng start. >> > >> > I thought it was strange that I cannot find my application in the: >> lttng list command under any domain. >> > Also the reason I start my application after creating a session is >> because the system automatically starts the application using systemd, >> > in order to get tracing data at all I have to manually stop the >> application and start it again after creating a session. >> > >> > Thanks again for any help, >> > Mosleh >> > >> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers < >> francis.deslauri...@efficios.com> wrote: >> >> >> >> Hi Mosleh, >> >> It's expected to not see any event when running lttng list -u if no >> >> lttng-ust instrumented applications are running. >> >> >> >> What are the exact lttng commands you are running to see those >> >> unexpected events? >> >> >> >> Thank you, >> >> Francis >> >> >> >> >> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin >> a écrit : >> >> > >> >> > Hello, >> >> > >> >> > I am having a strange issue with my current application. For some >> context, I have integrated LTTng into a Yocto build and put custom trace >> points in my embedded system application. Once I load the built image onto >> my system and start a session daemon, I try to list all available userspace >> events (using lttng list -u), but no events show up. >> >> > >> >> > The strange occurrence is when I create a session (lltng create >> my-sess) and not start tracing the trace events will display on the screen >> regardless (when starting session in live mode with lttng view). I was >> wondering if anyone has any insight on this issue I am having. >> >> > >> >> > Thanks for any assistance, >> >> > Mosleh >> >> > ___ >> >> > lttng-dev mailing list >> >> > lttng-dev@lists.lttng.org >> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> >> >> >> -- >> >> Francis Deslauriers >> >> Computer Engineer >> >> EfficiOS inc. >> >> >> >> -- >> Francis Deslauriers >> Computer Engineer >> EfficiOS inc. >> > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Tracepoints firing without being enabled in LTTng
Hi Mosleh, Just to be clear: When you run your app as a service (background), lttng list -u doesn't display any events AND you see events in the live session without having run enable-event and start commands. When you run your app normally, lttng list -u displays events AND you only see events after running enable-event and start commands. Is that correct? Please provide the output of the lttng-sessiond in verbose mode using the following command: > lttng-sessiond -vvv please provide the output of your application as a service with this environment variable: LTTNG_UST_DEBUG=1 and please provide us with the service file describing how your service is started. Put all of this on a pastebin.com Is your application mass-closing file descriptors when it's daemonizing? how about using fork() syscall without an exec() after it? If so, have a look at this part of the UST man page: https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons Thank you, Francis Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin a écrit : > > Hello, > > Sorry for the late reply. I was able to figure out the weird behavior from > the application. Turns out if I run my application as a service, lttng no > longer can see any of the UST tracepoints and causes the behavior I was > explaining. > When I execute the application normally (./app_name), the functionality is as > expected. Is this the expected behavior? Is there some way I can trace the > userspace application while running it as a service simultaneously? > > Thanks for any info, > Mosleh > > On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers > wrote: >> >> Can you start the lttng-sessiond in verbose mode using the following command: >> > lttng-sessiond -vvv >> and then do the lttng create, start of application and lttng view on another >> console. >> >> Then copy the output of the sessiond on https://pastebin.com/ and send us >> the link. >> >> Thank you, >> Francis >> >> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin a >> écrit : >>> >>> That's correct, I don't enable any events or do: lttng start. >>> Here is some snapshots of starting everything up, followed by starting my >>> application. >>> >>> >>> Here are the outputs for the commands: >>> >>> >>> >>> >>> >>> >>> >>> Note that even though the session is inactive, when a trace event occurs in >>> my application it is being recorded in the appropriate directory: >>> >>> >>> >>> Also maybe important, the system has root only user. Would having the >>> application belonging to a different user and trying to run as root cause >>> these kinds of complications? I did notice when testing on a raspberry pi >>> that running an application as a root user would mean running sessions as >>> root user as well. >>> >>> Thanks, >>> Mosleh >>> >>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers >>> wrote: >>>> >>>> Adding back lttng-dev mailing list. >>>> >>>> So you don't even run a lttng-enable-event command? >>>> That's odd. >>>> >>>> Can you send us the output of the following commands: >>>> Is it possible that there are other lttng-sessiond daemons running on >>>> the machine? >>>> > ps aux | grep "lttng-sessiond" >>>> Verify if there are other active tracing sessions: >>>> > lttng list >>>> Check the details of the current session: >>>> > lttng status >>>> >>>> Thank you, >>>> Francis >>>> >>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin a >>>> écrit : >>>> > >>>> > Hi Francis, >>>> > >>>> > Thank you for the quick response. >>>> > Just to clear things up, my application is running when I run that >>>> > command. >>>> > >>>> > The commands I am running are: >>>> > > lttng-sessiond -d >>>> > > lttng create my-sess --live >>>> > After this command I start my application. If I then do the command: >>>> > > lttng view >>>> > I will see the tracepoints in the application in the terminal even >>>> > though I have not done: lttng start. >>>> > >>>> > I thought it was strange that I cannot find my application in the: lttng >>>> > list comm
Re: [lttng-dev] Tracepoints firing without being enabled in LTTng
Le mer. 20 févr. 2019, à 09 h 10, Mosleh Uddin a écrit : > > Hello, > > Here is what I do to get the consistent behavior: > When the app is running as a service, lttng list -u does not display any > events, initially I do have to lttng enable-events and lttng start to see the > tracepoints in lttng view. However, after lttng start, I can disable all > events and I can still see the events coming through in lttng view. When I > run the app normally, lttng list -u displays the tracepoints I place in the > app, and all functionality of lttng works as expected. > > I apologize, I am in the process of trying to get you the outputs you are > asking for, however due to confidentiality with the app the outputs of > running the commands need to be reviewed by others. I will try to get this to > you as soon as I can. I understand. Thanks, Francis > > I can say the service is being kick started by systemd. Also I believe the > app is mass-closing file descriptors when it is daemonizing. I tried to > dynamically load the liblttng-ust-fd.so but the behavior did not change. > > Thanks, > Mosleh > > On Tue, Feb 19, 2019 at 5:16 PM Francis Deslauriers > wrote: >> >> Hi Mosleh, >> Just to be clear: >> When you run your app as a service (background), lttng list -u doesn't >> display any events AND you see events in the live session without >> having run enable-event and start commands. >> When you run your app normally, lttng list -u displays events AND you >> only see events after running enable-event and start commands. >> Is that correct? >> >> Please provide the output of the lttng-sessiond in verbose mode using >> the following command: >> > lttng-sessiond -vvv >> please provide the output of your application as a service with this >> environment variable: LTTNG_UST_DEBUG=1 >> and please provide us with the service file describing how your >> service is started. >> Put all of this on a pastebin.com >> >> Is your application mass-closing file descriptors when it's >> daemonizing? how about using fork() syscall without an exec() after >> it? If so, have a look at this part of the UST man page: >> https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons >> >> Thank you, >> Francis >> >> >> Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin a >> écrit : >> > >> > Hello, >> > >> > Sorry for the late reply. I was able to figure out the weird behavior from >> > the application. Turns out if I run my application as a service, lttng no >> > longer can see any of the UST tracepoints and causes the behavior I was >> > explaining. >> > When I execute the application normally (./app_name), the functionality is >> > as expected. Is this the expected behavior? Is there some way I can trace >> > the userspace application while running it as a service simultaneously? >> > >> > Thanks for any info, >> > Mosleh >> > >> > On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers >> > wrote: >> >> >> >> Can you start the lttng-sessiond in verbose mode using the following >> >> command: >> >> > lttng-sessiond -vvv >> >> and then do the lttng create, start of application and lttng view on >> >> another console. >> >> >> >> Then copy the output of the sessiond on https://pastebin.com/ and send us >> >> the link. >> >> >> >> Thank you, >> >> Francis >> >> >> >> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin a >> >> écrit : >> >>> >> >>> That's correct, I don't enable any events or do: lttng start. >> >>> Here is some snapshots of starting everything up, followed by starting >> >>> my application. >> >>> >> >>> >> >>> Here are the outputs for the commands: >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> Note that even though the session is inactive, when a trace event occurs >> >>> in my application it is being recorded in the appropriate directory: >> >>> >> >>> >> >>> >> >>> Also maybe important, the system has root only user. Would having the >> >>> application belonging to a different user and trying to run as root >> >>> cause these kinds of complicatio
[lttng-dev] Tracing Summit 2019: Announcement and Call for Proposals, August 20th, 2019, San Diego, CA
Hi all, This is a Call for Proposals for the Tracing Summit 2019 [0] which will be held in San Diego, California, USA at the Hilton San Diego Bayfront, co-located with the Open Source Summit & Embedded Linux Conference North America 2019 [1] on August 20th, 2019. You can subscribe to our mailing list ot get the latest info on the event: [2] The Tracing Summit is single-day, single-track conference on the topic of tracing. The event focuses on the field of software and hardware tracing, gathering developers and end-users of tracing and trace analysis tools. The main goal of the Tracing Summit is to provide space for discussion between people of the various areas that benefit from tracing, namely parallel, distributed and/or real-time systems, as well as kernel development. We are welcoming 30 minute presentations from both end users and developers, on topics covering, but not limited to: -Investigation workflow of Real-Time, latency, and throughput issues, -Trace collection and extraction, -Trace filtering, -Trace aggregation, -Trace formats, -Tracing multi-core systems, -Trace abstraction, -Trace modeling, -Automated trace analysis (e.g. dependency analysis), -Tracing large clusters and distributed systems, -Hardware-level tracing (e.g. DSP, GPU, bare-metal), -Trace visualisation, -Interaction between debugging and tracing, -Tracing remote control, -Analysis of large trace datasets, -Cloud trace collection and analysis, -Integration between trace tools, -Live tracing & monitoring, -Programmable tracing (e.g. eBPF). Those can cover recently available technologies, ongoing work, and yet non-existing technologies (which are compellingly interesting to end-users). Please understand that this open forum is not the proper place to present sales or marketing pitches, nor technologies which are prevented from being freely used in open source. * Submit you talk today using this form: https://forms.gle/L6mR2ZN3Nt7bUxY98 * The submission deadline is July 1st 2019 at 23:59 EST. This year tickets are $100 and seating is limited to 100 people. To register, you can add the Tracing Summit as an addon to your Open Source Summit ticket [3] or use this link [4] to register solely for the Tracing Summit. The Tracing Summit is sponsored by EfficiOS. We are actively looking for sponsors to support the Tracing Summit. Contact us at i...@tracingsummit.org. Please send any query about this conference to i...@tracingsummit.org. This event is organized by Francis Deslauriers and Mathieu Desnoyers on the behalf of the Linux Foundation Diagnostic and Monitoring Workgroup [5]. [0] : https://tracingsummit.org [1]: https://events.linuxfoundation.org/events/open-source-summit-north-america-2019 [2]: http://eepurl.com/goakfv [3]: https://www.cvent.com/events/open-source-summit-embedded-linux-conference-north-america-2019/registration-0ed1330c38bd4845be29b91c9d2444ce.aspx?fqp=true [4]: https://www.cvent.com/events/tracing-summit-2019/registration-63351d06945a46d890b8e5a200dbc0fc.aspx?fqp=true [5]: https://diamon.org -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] convert call_site to func/offset
Hi Changcheng, This is not currently supported by Babeltrace. It could probably be done the same way we do it with the "ip" and "vpid" contexts in the debug-info component class but it's not done yet. Thanks, Francis Le mer. 1 juill. 2020, à 11 h 58, Liu, Changcheng via lttng-dev < lttng-dev@lists.lttng.org> a écrit : > Hi all, > I checked __cyg_profile_func_enter implementation. > The caller address is recorded into call_site field. > > Does anyone know how to let babeltrace show the call_site into > "func/offset" format instead of the running virtual address? > > B.R. > Changcheng > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] lttng-modules: block: invalid/unknown value for rwbs field
Hi,
You see this "Unknown value 33" because there are no label with the
value 33 in the "block_rq_type" enumeration.
More specifically, you see the value "33" because it's the result of two
OR-ed bitflags (RWBS_FLAG_WRITE | RWBS_FLAG_SYNC).
The type of the "rwbs" field in the event was changed from integer to
enum in that commit [0]. That change was made to allow trace readers to
print OR-ed values more nicely have all they need to do it. In fact,
babeltrace2 implements such feature and you can try it with the
following command:
$ babeltrace2 -c sink.text.pretty -p print-enum-flags=true
And would print that field as such: ( "RWBS_FLAG_WRITE" |
"RWBS_FLAG_SYNC" : container = 33 )
I believe this feature is turn off in Babeltrace2 by default to be
backward compatible with Babeltrace 1.5.
It's important to note that before that change in the kernel tracer
Babeltrace would have simply printed "rwbs = 33" so you're not losing
any information here.
So, try it out with Babeltrace2, it should work as you expect.
Cheers,
Francis
[0]
https://github.com/lttng/lttng-modules/commit/23634515e7271c8c8594ad87a6685232d4eff297
On 3/9/21 8:10 AM, Awais Belal via lttng-dev wrote:
Hi,
I am using lttng-modules 2.12.5 with a 5.4.93 kernel on an aarch64
target. After creating a session I run
*lttng enable-event -k block**
to enable all the block_ kernel events but when I run babeltrace on
the generated traces I get a handful of
*[warning] Unknown value 33 in enum.
...
...
*
with different values in place of 33. This does not happen with any
other kernel event type
*BabelTrace Trace Viewer and Converter 1.5.8*
Also if I try to do a 'lttng view' on this session I can identify that
this has something to do with the rwbs field as I get
*[14:03:42.649719750] (+0.05750) imx8qxpmek block_bio_remap: {
cpu_id = 3 }, { dev = 187695200, sector = 2460080, nr_sector = 8, rwbs
= ( : container = 33 ), old_dev = 187695203, old_sector =
2105776 }
[14:03:42.649722125] (+0.02375) imx8qxpmek block_bio_queue: {
cpu_id = 3 }, { dev = 187695200, sector = 2460080, nr_sector = 8, rwbs
= ( : container = 33 ), tid = 428, comm = "jbd2/mmcblk1p3-" }
[14:03:42.649725000] (+0.02875) imx8qxpmek block_bio_backmerge: {
cpu_id = 3 }, { dev = 187695200, sector = 2460080, nr_sector = 8, rwbs
= ( : container = 33 ), tid = 428, comm = "jbd2/mmcblk1p3-" }
...
...*
I believe in the log points to a symptom? While for the
events/container id that do not generate this warning I have in the
same trace
*[14:03:44.697572375] (+2.036943500) imx8qxpmek block_bio_remap: {
cpu_id = 0 }, { dev = 187695200, sector = 9382032, nr_sector = 8, rwbs
= ( "RWBS_FLAG_WRITE" : container = 1 ), old_dev = 187695206,
old_sector = 278672 }
[14:03:44.697584375] (+0.12000) imx8qxpmek block_bio_queue: {
cpu_id = 0 }, { dev = 187695200, sector = 9382032, nr_sector = 8, rwbs
= ( "RWBS_FLAG_WRITE" : container = 1 ), tid = 1609, comm =
"kworker/u8:0" }
[14:03:44.697604250] (+0.19875) imx8qxpmek block_getrq: { cpu_id =
0 }, { dev = 187695200, sector = 9382032, nr_sector = 8, rwbs = (
"RWBS_FLAG_WRITE" : container = 1 ), tid = 1609, comm = "kworker/u8:0" }*
The interesting thing to note with the working events is that all of
them have "RWBS_FLAG_WRITE" and no other types from
https://github.com/lttng/lttng-modules/blob/stable-2.12/instrumentation/events/lttng-module/block.h#L39
which I would expect to see.
I am lost for hints as to what I need to look up... any help would be
highly appreciated.
--
BR,
Awais
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Assertion tgt_fc failed when trying to trace callstack-user
Hi Evan, On 8/1/21 5:11 AM, Evan Galea via lttng-dev wrote: Hi there Thanks for help on my previous issue. I was able to get callstack-user to work and see it in my traces. However, in order to get it to run I needed to enable-event --kernel all. Is it necessary to have all kernel events enabled to run callstack-user? Or is it just one specific event? When I try to run it without enabling all kernel events the command is accepted and it runs but I get the following error when trying to view my traces: (╯°□°)╯︵ ┻━┻ ctf-meta-update-value-storing-indexes.c:65: update_field_class_stored_value_index(): Assertion `tgt_fc` failed. Aborted (core dumped) What does this error mean? Should I just continue to enable all kernel events? It's a Babeltrace 2 error. Now, we need to find out if the root cause is the LTTng kernel tracer or Babeltrace2 itself. What is the "one specific event" you're enabling that seems to cause this issue? Do you have access to a Babeltrace 1.5 build? if so, could you try to read that same trace using it? Could you please share a small trace reproducing this issue? What version of LTTng-tools, LTTng-Modules, and Babeltrace2 are you using? Cheers, Francis My current command setup, not sure if relevant lttng create && lttng enable-channel mychannel --subbuf-size 1 lttng enable-event --kernel --all && lttng add-context -u -t vpid -t vtid -t procname && lttng add-context --kernel --type callstack-user && lttng add-context --kernel --type callstack-kernel && lttng enable-event --jul jello again thank you for your time Evan Galea ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Assertion tgt_fc failed when trying to trace callstack-user
Hi Evan, It turns out it's a Babeltrace 2 bug. Here the fix: https://review.lttng.org/c/babeltrace/+/6257. It should be merged into the master and stable-2.0 branches soon. Thank you for reporting the issue, Francis On 8/4/21 3:55 AM, Evan Galea wrote: Hi Francis, thanks for your reply. As it turns out, running babeltrace on the file worked and I'm able to read the traces. I am running babeltrace 1.5.8, lttng 2.12.4, lttng-tools 2.12.x stable, and lttng modules 5.8.0-63. I attached the trace of the file. On Mon, Aug 2, 2021 at 8:56 AM Francis Deslauriers <mailto:francis.deslauri...@efficios.com>> wrote: Hi Evan, On 8/1/21 5:11 AM, Evan Galea via lttng-dev wrote: Hi there Thanks for help on my previous issue. I was able to get callstack-user to work and see it in my traces. However, in order to get it to run I needed to enable-event --kernel all. Is it necessary to have all kernel events enabled to run callstack-user? Or is it just one specific event? When I try to run it without enabling all kernel events the command is accepted and it runs but I get the following error when trying to view my traces: (╯°□°)╯︵ ┻━┻ ctf-meta-update-value-storing-indexes.c:65: update_field_class_stored_value_index(): Assertion `tgt_fc` failed. Aborted (core dumped) What does this error mean? Should I just continue to enable all kernel events? It's a Babeltrace 2 error. Now, we need to find out if the root cause is the LTTng kernel tracer or Babeltrace2 itself. What is the "one specific event" you're enabling that seems to cause this issue? Do you have access to a Babeltrace 1.5 build? if so, could you try to read that same trace using it? Could you please share a small trace reproducing this issue? What version of LTTng-tools, LTTng-Modules, and Babeltrace2 are you using? Cheers, Francis My current command setup, not sure if relevant lttng create && lttng enable-channel mychannel --subbuf-size 1 lttng enable-event --kernel --all && lttng add-context -u -t vpid -t vtid -t procname && lttng add-context --kernel --type callstack-user && lttng add-context --kernel --type callstack-kernel && lttng enable-event --jul jello again thank you for your time Evan Galea ___ lttng-dev mailing list lttng-dev@lists.lttng.org <mailto:lttng-dev@lists.lttng.org> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev <https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev> ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
