----- Original Message ----- > From: "Mathieu Desnoyers" <[email protected]> > To: [email protected] > Cc: [email protected] > Sent: Tuesday, February 18, 2014 4:02:15 PM > Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump > > ----- Original Message ----- > > From: "Corey Minyard" <[email protected]> > > To: "Mathieu Desnoyers" <[email protected]> > > Cc: "David Goulet" <[email protected]>, [email protected] > > Sent: Tuesday, February 18, 2014 2:29:23 PM > > Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump > > > > I have attached some gdb macros for dumping LTT buffers from a kernel > > coredump. I haven't done a lot of testing, though. > > Very cool! I'll have a look when things get less busy here. I hope my > earlier reply helps clearing things out. > > > > > I do have one question. I've been getting the metadata by taking a > > snapshot right when the trace starts and saving that to stick into the > > trace output from the coredump. Is that the best way, or is there some > > other way I can extract it? There doesn't seem to be a metadata channel > > running in snapshot mode. > > Actually, it's better if you grab the metadata near when you grab the > coredump. Grabbing the metadata at the beginning of your tracing session > will not have the info about other modules that might be coming and going > while tracing is active (which might contain tracepoints). > > You could grab the metadata from the coredump actually, this would be the > best approach. Looking at lttng modules 2.4 rc, you will want to look at > > /lttng-events.h > > struct lttng_session { > ... > struct lttng_metadata_cache *metadata_cache; > ... > }; > > Which has the metadata string: > > struct lttng_metadata_cache { > char *data; /* Metadata cache */ > unsigned int cache_alloc; /* Metadata allocated size (bytes) */ > unsigned int metadata_written; /* Number of bytes written in > metadata cache */ > struct kref refcount; /* Metadata cache usage */ > struct list_head metadata_stream; /* Metadata stream list */ > }; > > You simply need to dump this metadata string into a "metadata" file, add a > "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only > CTF metadata", which can be read by babeltrace. > > If you happen to core dump while the metadata cache is being resized, you > might > encounter a corrupted metadata. It should not happen frequently, but it's > possible. > We could eventually change lttng_metadata_printf() so it makes sure to always > keep a readable metadata around (by e.g. using kcalloc rather than krealloc > and > dealing carefully with cache_alloc vs data fields updates).
Oops, I meant "kzalloc". I'm attaching a patch that should take care of making sure it is always in a valid state when a core dump occurs. You should be able to safely read "metadata_flushed" bytes of data at any time. Whatever has been flushed should be parseable by Babeltrace. Thoughts ? Thanks, Mathieu > > Thoughts ? > > Thanks, > > Mathieu > > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com > > _______________________________________________ > lttng-dev mailing list > [email protected] > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
commit 875fbbbe6167b3847cdf4de925722be9dcf685cc Author: Mathieu Desnoyers <[email protected]> Date: Tue Feb 18 16:35:05 2014 -0500 Ensure metadata can be extracted from core dumps at any time Signed-off-by: Mathieu Desnoyers <[email protected]> diff --git a/lttng-events.c b/lttng-events.c index 0512a3f..0e67520 100644 --- a/lttng-events.c +++ b/lttng-events.c @@ -603,7 +603,7 @@ int lttng_metadata_output_channel(struct lttng_metadata_stream *stream, if (stream->metadata_in != stream->metadata_out) return 0; - len = stream->metadata_cache->metadata_written - + len = stream->metadata_cache->metadata_flushed - stream->metadata_in; if (!len) return 0; @@ -631,17 +631,41 @@ end: return ret; } +static +void lttng_metadata_flush(struct lttng_session *session) +{ + struct lttng_metadata_cache *cache = session->metadata_cache; + struct lttng_metadata_stream *stream; + + if (cache->metadata_written == cache->metadata_flushed) + return; + + /* + * Append to metadata before incrementing "metadata flushed", + * ensuring that core dumps always read up to "metadata flushed" + * bytes of valid metadata. + */ + barrier(); + cache->metadata_flushed = cache->metadata_written; + + list_for_each_entry(stream, &cache->metadata_stream, list) + wake_up_interruptible(&stream->read_wait); +} + /* * Write the metadata to the metadata cache. * Must be called with sessions_mutex held. + * Grow the metadata cache with allocation, copy, populating the new + * structure, and then free the old structure, to ensure that there is + * always a consistent state to grab from kernel core dump. */ +static int lttng_metadata_printf(struct lttng_session *session, const char *fmt, ...) { char *str; size_t len; va_list ap; - struct lttng_metadata_stream *stream; WARN_ON_ONCE(!ACCESS_ONCE(session->active)); @@ -654,18 +678,33 @@ int lttng_metadata_printf(struct lttng_session *session, len = strlen(str); if (session->metadata_cache->metadata_written + len > session->metadata_cache->cache_alloc) { - char *tmp_cache_realloc; - unsigned int tmp_cache_alloc_size; - - tmp_cache_alloc_size = max_t(unsigned int, - session->metadata_cache->cache_alloc + len, - session->metadata_cache->cache_alloc << 1); - tmp_cache_realloc = krealloc(session->metadata_cache->data, - tmp_cache_alloc_size, GFP_KERNEL); - if (!tmp_cache_realloc) + char *new_cache, *old_cache; + unsigned int new_cache_alloc_size, old_cache_alloc_size; + + old_cache_alloc_size = session->metadata_cache->cache_alloc; + new_cache_alloc_size = max_t(unsigned int, + old_cache_alloc_size + len, + old_cache_alloc_size << 1); + new_cache = kzalloc(new_cache_alloc_size, GFP_KERNEL); + if (!new_cache) goto err; - session->metadata_cache->cache_alloc = tmp_cache_alloc_size; - session->metadata_cache->data = tmp_cache_realloc; + session->metadata_cache->cache_alloc = new_cache_alloc_size; + old_cache = session->metadata_cache->data; + memcpy(new_cache, old_cache, old_cache_alloc_size); + /* + * Copy the old cache into the new one before storing + * the pointer to new cache into "data", so core dumps + * read valid metadata. + */ + barrier(); + session->metadata_cache->data = new_cache; + /* + * Store "data" before freeing the old cache, thus + * ensuring kernel core dumps always read up to + * "metadata_flushed" bytes of valid metadata. + */ + barrier(); + kfree(old_cache); } memcpy(session->metadata_cache->data + session->metadata_cache->metadata_written, @@ -673,9 +712,6 @@ int lttng_metadata_printf(struct lttng_session *session, session->metadata_cache->metadata_written += len; kfree(str); - list_for_each_entry(stream, &session->metadata_cache->metadata_stream, list) - wake_up_interruptible(&stream->read_wait); - return 0; err: @@ -906,6 +942,7 @@ int _lttng_event_metadata_statedump(struct lttng_session *session, event->metadata_dumped = 1; end: + lttng_metadata_flush(session); return ret; } @@ -958,6 +995,7 @@ int _lttng_channel_metadata_statedump(struct lttng_session *session, chan->metadata_dumped = 1; end: + lttng_metadata_flush(session); return ret; } @@ -967,7 +1005,9 @@ end: static int _lttng_stream_packet_context_declare(struct lttng_session *session) { - return lttng_metadata_printf(session, + int ret; + + ret = lttng_metadata_printf(session, "struct packet_context {\n" " uint64_clock_monotonic_t timestamp_begin;\n" " uint64_clock_monotonic_t timestamp_end;\n" @@ -977,6 +1017,8 @@ int _lttng_stream_packet_context_declare(struct lttng_session *session) " uint32_t cpu_id;\n" "};\n\n" ); + lttng_metadata_flush(session); + return ret; } /* @@ -993,7 +1035,9 @@ int _lttng_stream_packet_context_declare(struct lttng_session *session) static int _lttng_event_header_declare(struct lttng_session *session) { - return lttng_metadata_printf(session, + int ret; + + ret = lttng_metadata_printf(session, "struct event_header_compact {\n" " enum : uint5_t { compact = 0 ... 30, extended = 31 } id;\n" " variant <id> {\n" @@ -1022,6 +1066,8 @@ int _lttng_event_header_declare(struct lttng_session *session) lttng_alignof(uint32_t) * CHAR_BIT, lttng_alignof(uint16_t) * CHAR_BIT ); + lttng_metadata_flush(session); + return ret; } /* @@ -1209,6 +1255,7 @@ skip_session: } session->metadata_dumped = 1; end: + lttng_metadata_flush(session); return ret; } diff --git a/lttng-events.h b/lttng-events.h index 6b39304..9d43d47 100644 --- a/lttng-events.h +++ b/lttng-events.h @@ -326,6 +326,7 @@ struct lttng_metadata_cache { char *data; /* Metadata cache */ unsigned int cache_alloc; /* Metadata allocated size (bytes) */ unsigned int metadata_written; /* Number of bytes written in metadata cache */ + unsigned int metadata_flushed; /* Number of bytes flushed in metadata cache */ struct kref refcount; /* Metadata cache usage */ struct list_head metadata_stream; /* Metadata stream list */ };
_______________________________________________ lttng-dev mailing list [email protected] http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
