[lttng-dev] Videos of the Tracing Summit 2018
Hi, The videos of the Tracing Summit 2018 are now online: https://www.youtube.com/playlist?list=PLuo4E47p5_7aBCdobf_XpEPcDFNPhpRNs You can also access them from the schedule here: https://tracingsummit.org/wiki/TracingSummit2018#Schedule -- On behalf of the Diagnostic and Monitoring Workgroup, Francis Deslauriers ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] "Hands-free" tracepoints using LD_PRELOAD
Hi Brian, If I understand correctly, you want developers to provide a whitelist of functions that they want to instrument in an arbitrary shared library, and then have a system that instruments those functions without having to recompile the library with LTTng-UST tracepoints. Can you give us an example of annotation that the shim developer would provide? Do you need to support the extraction of function arguments? If you don't, here are two straightforward and tested approaches to solve your problem: First, overriding target symbols using LD_PRELOAD (like you suggested above): 1. Create boilerplate script #1 that generates tracepoints definitions [1] based on function name, 2. Create boilerplate script #2 that generates a function calling the right tracepoint and the original target function [2], 3. For each target symbol "bar", using script #1, create a tracepoint definition "libfoo:bar_tp", and compile them into a tracepoints.o. object file, 4. For each target symbol "bar", using script #2, create a function "bar" that calls the bar_tp tracepoint (e.g. tracepoint(libfoo, bar_tp) ) and then calls the original bar function in libfoo.so (using dlsym(3)), and compile those callsites into a callsites.o object file, 5. Create a shared library libshim.so using tracepoints.o and callsites.o object files and link it to lttng-ust lib and the dynamic loader lib using :"-llttng-ust -ldl", Those steps results in the creation of a libshim.so file that contains the "bar" overriding function (containing the tracepoint callsite) that will be called instead of "bar" function in the libfoo.so library when LD_PRELOADing it (e.g. LD_PRELOAD=libshim.so ./fooapp). Here is the Makefile I used to test this [3] to give you a sense of the pipeline. Second, using the newly added --userspace-probe instrumentation: This option allows you to instrument any ELF symbol without recompiling or restarting the target process. For example, to instrument the "bar" function in the /tmp/libfoo.so file you could run to following command: > lttng enable-event -k --userspace-probe="elf:/tmp/libfoo.so:bar" $event_name You could write a script that takes the list of target functions and adds userspace-probes on each of them. This approach has the advantage that you can also instrument functions in the binary and not only functions in shared libraries. This instrumentation type uses the kernel tracer and thus requires using the lttng-modules. If you do need to save the function arguments, I believe it's possible using the first approach, by tweaking the two boilerplate scripts depending on what are the annotations you want to provide. Supporting argument extraction with userspace-probes is part of our long term goal and would be a really powerful feature but will require a significant engineering effort to accomplish. [1] : https://lttng.org/docs/v2.10/#doc-tpp-header [2]: https://lttng.org/docs/v2.10/#doc-probing-the-application-source-code [3]: http://paste.ubuntu.com/p/5fBSgRqXsB/ I hope this make sense. Please don't hesitate if you want more details. Thank you, Francis Le ven. 18 janv. 2019, à 16 h 04, Brian Rossa a écrit : > > Hello, > > I have a custom tracing solution that I'm interested in porting to LTTng. It > works as follows: > > Input is a whitelist of mangled names taken from a target library, libfoo.so. > For each name, generate a logging shim (example) for the corresponding call. > Compile all the logging shims into a shared library, libshim.so. > Trace any application using libfoo by running it as LD_PRELOAD=libshim.so > ./fooapp > > There are two nice things about this approach that I would like to preserve: > > The shim developer only needs to provide the whitelist and a bit of extra > annotation. From there, the shim can be be authored using a boilerplate > generator. > The app developer doesn't have to do anything other than pass the LD_PRELOAD > flag. > > The downside is that the only tracepoints are those corresponding to the > whitelist, but I'm fine with that. > > Can this kind of "hands-free" developer experience be supported by LTTng? > > Thanks! > ~br > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Tracepoints firing without being enabled in LTTng
Le mer. 20 févr. 2019, à 09 h 10, Mosleh Uddin a écrit : > > Hello, > > Here is what I do to get the consistent behavior: > When the app is running as a service, lttng list -u does not display any > events, initially I do have to lttng enable-events and lttng start to see the > tracepoints in lttng view. However, after lttng start, I can disable all > events and I can still see the events coming through in lttng view. When I > run the app normally, lttng list -u displays the tracepoints I place in the > app, and all functionality of lttng works as expected. > > I apologize, I am in the process of trying to get you the outputs you are > asking for, however due to confidentiality with the app the outputs of > running the commands need to be reviewed by others. I will try to get this to > you as soon as I can. I understand. Thanks, Francis > > I can say the service is being kick started by systemd. Also I believe the > app is mass-closing file descriptors when it is daemonizing. I tried to > dynamically load the liblttng-ust-fd.so but the behavior did not change. > > Thanks, > Mosleh > > On Tue, Feb 19, 2019 at 5:16 PM Francis Deslauriers > wrote: >> >> Hi Mosleh, >> Just to be clear: >> When you run your app as a service (background), lttng list -u doesn't >> display any events AND you see events in the live session without >> having run enable-event and start commands. >> When you run your app normally, lttng list -u displays events AND you >> only see events after running enable-event and start commands. >> Is that correct? >> >> Please provide the output of the lttng-sessiond in verbose mode using >> the following command: >> > lttng-sessiond -vvv >> please provide the output of your application as a service with this >> environment variable: LTTNG_UST_DEBUG=1 >> and please provide us with the service file describing how your >> service is started. >> Put all of this on a pastebin.com >> >> Is your application mass-closing file descriptors when it's >> daemonizing? how about using fork() syscall without an exec() after >> it? If so, have a look at this part of the UST man page: >> https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons >> >> Thank you, >> Francis >> >> >> Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin a >> écrit : >> > >> > Hello, >> > >> > Sorry for the late reply. I was able to figure out the weird behavior from >> > the application. Turns out if I run my application as a service, lttng no >> > longer can see any of the UST tracepoints and causes the behavior I was >> > explaining. >> > When I execute the application normally (./app_name), the functionality is >> > as expected. Is this the expected behavior? Is there some way I can trace >> > the userspace application while running it as a service simultaneously? >> > >> > Thanks for any info, >> > Mosleh >> > >> > On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers >> > wrote: >> >> >> >> Can you start the lttng-sessiond in verbose mode using the following >> >> command: >> >> > lttng-sessiond -vvv >> >> and then do the lttng create, start of application and lttng view on >> >> another console. >> >> >> >> Then copy the output of the sessiond on https://pastebin.com/ and send us >> >> the link. >> >> >> >> Thank you, >> >> Francis >> >> >> >> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin a >> >> écrit : >> >>> >> >>> That's correct, I don't enable any events or do: lttng start. >> >>> Here is some snapshots of starting everything up, followed by starting >> >>> my application. >> >>> >> >>> >> >>> Here are the outputs for the commands: >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> Note that even though the session is inactive, when a trace event occurs >> >>> in my application it is being recorded in the appropriate directory: >> >>> >> >>> >> >>> >> >>> Also maybe important, the system has root only user. Would having the >> >>> application belonging to a different user and trying to run as root >> >>> cause these kinds of complications? I did notice whe
Re: [lttng-dev] Tracepoints firing without being enabled in LTTng
Hi Mosleh, It's expected to not see any event when running lttng list -u if no lttng-ust instrumented applications are running. What are the exact lttng commands you are running to see those unexpected events? Thank you, Francis Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin a écrit : > > Hello, > > I am having a strange issue with my current application. For some context, I > have integrated LTTng into a Yocto build and put custom trace points in my > embedded system application. Once I load the built image onto my system and > start a session daemon, I try to list all available userspace events (using > lttng list -u), but no events show up. > > The strange occurrence is when I create a session (lltng create my-sess) and > not start tracing the trace events will display on the screen regardless > (when starting session in live mode with lttng view). I was wondering if > anyone has any insight on this issue I am having. > > Thanks for any assistance, > Mosleh > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Tracepoints firing without being enabled in LTTng
Adding back lttng-dev mailing list. So you don't even run a lttng-enable-event command? That's odd. Can you send us the output of the following commands: Is it possible that there are other lttng-sessiond daemons running on the machine? > ps aux | grep "lttng-sessiond" Verify if there are other active tracing sessions: > lttng list Check the details of the current session: > lttng status Thank you, Francis Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin a écrit : > > Hi Francis, > > Thank you for the quick response. > Just to clear things up, my application is running when I run that command. > > The commands I am running are: > > lttng-sessiond -d > > lttng create my-sess --live > After this command I start my application. If I then do the command: > > lttng view > I will see the tracepoints in the application in the terminal even though I > have not done: lttng start. > > I thought it was strange that I cannot find my application in the: lttng list > command under any domain. > Also the reason I start my application after creating a session is because > the system automatically starts the application using systemd, > in order to get tracing data at all I have to manually stop the application > and start it again after creating a session. > > Thanks again for any help, > Mosleh > > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers > wrote: >> >> Hi Mosleh, >> It's expected to not see any event when running lttng list -u if no >> lttng-ust instrumented applications are running. >> >> What are the exact lttng commands you are running to see those >> unexpected events? >> >> Thank you, >> Francis >> >> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin a >> écrit : >> > >> > Hello, >> > >> > I am having a strange issue with my current application. For some context, >> > I have integrated LTTng into a Yocto build and put custom trace points in >> > my embedded system application. Once I load the built image onto my system >> > and start a session daemon, I try to list all available userspace events >> > (using lttng list -u), but no events show up. >> > >> > The strange occurrence is when I create a session (lltng create my-sess) >> > and not start tracing the trace events will display on the screen >> > regardless (when starting session in live mode with lttng view). I was >> > wondering if anyone has any insight on this issue I am having. >> > >> > Thanks for any assistance, >> > Mosleh >> > ___ >> > lttng-dev mailing list >> > lttng-dev@lists.lttng.org >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> >> -- >> Francis Deslauriers >> Computer Engineer >> EfficiOS inc. -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Tracepoints firing without being enabled in LTTng
Can you start the lttng-sessiond in verbose mode using the following command: > lttng-sessiond -vvv and then do the lttng create, start of application and lttng view on another console. Then copy the output of the sessiond on https://pastebin.com/ and send us the link. Thank you, Francis Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin a écrit : > That's correct, I don't enable any events or do: lttng start. > Here is some snapshots of starting everything up, followed by starting my > application. > [image: image.png] > > Here are the outputs for the commands: > > [image: image.png] > > [image: image.png] > > [image: image.png] > > Note that even though the session is inactive, when a trace event occurs > in my application it is being recorded in the appropriate directory: > > [image: image.png] > > Also maybe important, the system has root only user. Would having the > application belonging to a different user and trying to run as root cause > these kinds of complications? I did notice when testing on a raspberry pi > that running an application as a root user would mean running sessions as > root user as well. > > Thanks, > Mosleh > > On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers < > francis.deslauri...@efficios.com> wrote: > >> Adding back lttng-dev mailing list. >> >> So you don't even run a lttng-enable-event command? >> That's odd. >> >> Can you send us the output of the following commands: >> Is it possible that there are other lttng-sessiond daemons running on >> the machine? >> > ps aux | grep "lttng-sessiond" >> Verify if there are other active tracing sessions: >> > lttng list >> Check the details of the current session: >> > lttng status >> >> Thank you, >> Francis >> >> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin a >> écrit : >> > >> > Hi Francis, >> > >> > Thank you for the quick response. >> > Just to clear things up, my application is running when I run that >> command. >> > >> > The commands I am running are: >> > > lttng-sessiond -d >> > > lttng create my-sess --live >> > After this command I start my application. If I then do the command: >> > > lttng view >> > I will see the tracepoints in the application in the terminal even >> though I have not done: lttng start. >> > >> > I thought it was strange that I cannot find my application in the: >> lttng list command under any domain. >> > Also the reason I start my application after creating a session is >> because the system automatically starts the application using systemd, >> > in order to get tracing data at all I have to manually stop the >> application and start it again after creating a session. >> > >> > Thanks again for any help, >> > Mosleh >> > >> > On Fri, Feb 15, 2019 at 10:32 AM Francis Deslauriers < >> francis.deslauri...@efficios.com> wrote: >> >> >> >> Hi Mosleh, >> >> It's expected to not see any event when running lttng list -u if no >> >> lttng-ust instrumented applications are running. >> >> >> >> What are the exact lttng commands you are running to see those >> >> unexpected events? >> >> >> >> Thank you, >> >> Francis >> >> >> >> >> >> Le ven. 15 févr. 2019, à 10 h 08, Mosleh Uddin >> a écrit : >> >> > >> >> > Hello, >> >> > >> >> > I am having a strange issue with my current application. For some >> context, I have integrated LTTng into a Yocto build and put custom trace >> points in my embedded system application. Once I load the built image onto >> my system and start a session daemon, I try to list all available userspace >> events (using lttng list -u), but no events show up. >> >> > >> >> > The strange occurrence is when I create a session (lltng create >> my-sess) and not start tracing the trace events will display on the screen >> regardless (when starting session in live mode with lttng view). I was >> wondering if anyone has any insight on this issue I am having. >> >> > >> >> > Thanks for any assistance, >> >> > Mosleh >> >> > ___ >> >> > lttng-dev mailing list >> >> > lttng-dev@lists.lttng.org >> >> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >> >> >> >> -- >> >> Francis Deslauriers >> >> Computer Engineer >> >> EfficiOS inc. >> >> >> >> -- >> Francis Deslauriers >> Computer Engineer >> EfficiOS inc. >> > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Tracepoints firing without being enabled in LTTng
Hi Mosleh, Just to be clear: When you run your app as a service (background), lttng list -u doesn't display any events AND you see events in the live session without having run enable-event and start commands. When you run your app normally, lttng list -u displays events AND you only see events after running enable-event and start commands. Is that correct? Please provide the output of the lttng-sessiond in verbose mode using the following command: > lttng-sessiond -vvv please provide the output of your application as a service with this environment variable: LTTNG_UST_DEBUG=1 and please provide us with the service file describing how your service is started. Put all of this on a pastebin.com Is your application mass-closing file descriptors when it's daemonizing? how about using fork() syscall without an exec() after it? If so, have a look at this part of the UST man page: https://lttng.org/man/3/lttng-ust/v2.10/#doc-_using_lttng_ust_with_daemons Thank you, Francis Le mar. 19 févr. 2019, à 16 h 37, Mosleh Uddin a écrit : > > Hello, > > Sorry for the late reply. I was able to figure out the weird behavior from > the application. Turns out if I run my application as a service, lttng no > longer can see any of the UST tracepoints and causes the behavior I was > explaining. > When I execute the application normally (./app_name), the functionality is as > expected. Is this the expected behavior? Is there some way I can trace the > userspace application while running it as a service simultaneously? > > Thanks for any info, > Mosleh > > On Fri, Feb 15, 2019 at 2:48 PM Francis Deslauriers > wrote: >> >> Can you start the lttng-sessiond in verbose mode using the following command: >> > lttng-sessiond -vvv >> and then do the lttng create, start of application and lttng view on another >> console. >> >> Then copy the output of the sessiond on https://pastebin.com/ and send us >> the link. >> >> Thank you, >> Francis >> >> Le ven. 15 févr. 2019, à 14 h 36, Mosleh Uddin a >> écrit : >>> >>> That's correct, I don't enable any events or do: lttng start. >>> Here is some snapshots of starting everything up, followed by starting my >>> application. >>> >>> >>> Here are the outputs for the commands: >>> >>> >>> >>> >>> >>> >>> >>> Note that even though the session is inactive, when a trace event occurs in >>> my application it is being recorded in the appropriate directory: >>> >>> >>> >>> Also maybe important, the system has root only user. Would having the >>> application belonging to a different user and trying to run as root cause >>> these kinds of complications? I did notice when testing on a raspberry pi >>> that running an application as a root user would mean running sessions as >>> root user as well. >>> >>> Thanks, >>> Mosleh >>> >>> On Fri, Feb 15, 2019 at 11:55 AM Francis Deslauriers >>> wrote: >>>> >>>> Adding back lttng-dev mailing list. >>>> >>>> So you don't even run a lttng-enable-event command? >>>> That's odd. >>>> >>>> Can you send us the output of the following commands: >>>> Is it possible that there are other lttng-sessiond daemons running on >>>> the machine? >>>> > ps aux | grep "lttng-sessiond" >>>> Verify if there are other active tracing sessions: >>>> > lttng list >>>> Check the details of the current session: >>>> > lttng status >>>> >>>> Thank you, >>>> Francis >>>> >>>> Le ven. 15 févr. 2019, à 11 h 29, Mosleh Uddin a >>>> écrit : >>>> > >>>> > Hi Francis, >>>> > >>>> > Thank you for the quick response. >>>> > Just to clear things up, my application is running when I run that >>>> > command. >>>> > >>>> > The commands I am running are: >>>> > > lttng-sessiond -d >>>> > > lttng create my-sess --live >>>> > After this command I start my application. If I then do the command: >>>> > > lttng view >>>> > I will see the tracepoints in the application in the terminal even >>>> > though I have not done: lttng start. >>>> > >>>> > I thought it was strange that I cannot find my application in the: lttng >>>> > list command under any domain. >>
Re: [lttng-dev] "Hands-free" tracepoints using LD_PRELOAD
es >> >- Implementations of struct "wrappers" that provide custom ostream >> >operators >> >- A map between type names and wrapper names >> > >> > The machinery here seems fairly general-purpose, but I don't presume to be >> > an expert. My implementation is somewhat error-prone, and my main hope in >> > reaching out to the mailing list was that LTTng already had some of these >> > steps better-implemented. >> >> AFAIK, lttng does not have an equivalent. >> >> > Step #2 is particularly problematic due to >> > ambiguities in the mangling grammar, and will need support going forward to >> > generalize well. >> >> What is the status of this step in your project? >> >> What are the problems that make your implementation "error-pone"? >> >> Would you mind linking us to said project so we can have a look? >> >> I would be interested in seeing at first lttng tracepoint used as Francis >> demonstrated and see from there were this project can go. >> >> > >> > I would be happy to contribute some or all of my implementation if it's >> > something that the LTTng community would be interested in supporting and >> > extending. >> >> We are clearly open for discussion and helping you improve the project. I am >> not >> so sure on supporting and extending it. Others might have a different >> opinion. >> >> >> Cheers >> >> -- >> Jonathan Rajotte-Julien >> EfficiOS -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] Tracing Summit 2019: Announcement and Call for Proposals, August 20th, 2019, San Diego, CA
Hi all, This is a Call for Proposals for the Tracing Summit 2019 [0] which will be held in San Diego, California, USA at the Hilton San Diego Bayfront, co-located with the Open Source Summit & Embedded Linux Conference North America 2019 [1] on August 20th, 2019. You can subscribe to our mailing list ot get the latest info on the event: [2] The Tracing Summit is single-day, single-track conference on the topic of tracing. The event focuses on the field of software and hardware tracing, gathering developers and end-users of tracing and trace analysis tools. The main goal of the Tracing Summit is to provide space for discussion between people of the various areas that benefit from tracing, namely parallel, distributed and/or real-time systems, as well as kernel development. We are welcoming 30 minute presentations from both end users and developers, on topics covering, but not limited to: -Investigation workflow of Real-Time, latency, and throughput issues, -Trace collection and extraction, -Trace filtering, -Trace aggregation, -Trace formats, -Tracing multi-core systems, -Trace abstraction, -Trace modeling, -Automated trace analysis (e.g. dependency analysis), -Tracing large clusters and distributed systems, -Hardware-level tracing (e.g. DSP, GPU, bare-metal), -Trace visualisation, -Interaction between debugging and tracing, -Tracing remote control, -Analysis of large trace datasets, -Cloud trace collection and analysis, -Integration between trace tools, -Live tracing & monitoring, -Programmable tracing (e.g. eBPF). Those can cover recently available technologies, ongoing work, and yet non-existing technologies (which are compellingly interesting to end-users). Please understand that this open forum is not the proper place to present sales or marketing pitches, nor technologies which are prevented from being freely used in open source. * Submit you talk today using this form: https://forms.gle/L6mR2ZN3Nt7bUxY98 * The submission deadline is July 1st 2019 at 23:59 EST. This year tickets are $100 and seating is limited to 100 people. To register, you can add the Tracing Summit as an addon to your Open Source Summit ticket [3] or use this link [4] to register solely for the Tracing Summit. The Tracing Summit is sponsored by EfficiOS. We are actively looking for sponsors to support the Tracing Summit. Contact us at i...@tracingsummit.org. Please send any query about this conference to i...@tracingsummit.org. This event is organized by Francis Deslauriers and Mathieu Desnoyers on the behalf of the Linux Foundation Diagnostic and Monitoring Workgroup [5]. [0] : https://tracingsummit.org [1]: https://events.linuxfoundation.org/events/open-source-summit-north-america-2019 [2]: http://eepurl.com/goakfv [3]: https://www.cvent.com/events/open-source-summit-embedded-linux-conference-north-america-2019/registration-0ed1330c38bd4845be29b91c9d2444ce.aspx?fqp=true [4]: https://www.cvent.com/events/tracing-summit-2019/registration-63351d06945a46d890b8e5a200dbc0fc.aspx?fqp=true [5]: https://diamon.org -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Tracing Summit 2019: Announcement and Call for Proposals, August 20th, 2019, San Diego, CA
Hi all, This is a friendly reminder that the deadline for submitting talk proposals for the Tracing Summit 2019 [1] is coming up soon (July 1st). Submit you talk today through this form: https://forms.gle/L6mR2ZN3Nt7bUxY98. The Tracing Summit 2019 will be held in San Diego, California on August 20th, 2019, at the Hilton San Diego Bayfront, co-located with the Open Source Summit & Embedded Linux Conference North America 2019 This year tickets are $100 and seating is limited to 100 people. To register, you can add the Tracing Summit as an addon to your Open Source Summit ticket or use this link to register solely for the Tracing Summit: https://www.cvent.com/events/tracing-summit-2019/registration-63351d06945a46d890b8e5a200dbc0fc.aspx. We are actively looking for sponsors to support the Tracing Summit. Contact us at i...@tracingsummit.org. More info: https://tracingsummit.org/wiki/TracingSummit2019 Thank you, On behalf of the Diagnostic and Monitoring Workgroup, Tracing Summit organizers Le mar. 16 avr. 2019, à 11 h 54, Francis Deslauriers a écrit : > > Hi all, > > This is a Call for Proposals for the Tracing Summit 2019 [0] which > will be held in San Diego, California, USA at the Hilton San Diego > Bayfront, co-located with the Open Source Summit & Embedded Linux > Conference North America 2019 [1] on August 20th, 2019. > > You can subscribe to our mailing list ot get the latest info on the event: [2] > > The Tracing Summit is single-day, single-track conference on the topic > of tracing. The event focuses on the field of software and hardware > tracing, gathering developers and end-users of tracing and trace > analysis tools. The main goal of the Tracing Summit is to provide > space for discussion between people of the various areas that benefit > from tracing, namely parallel, distributed and/or real-time systems, > as well as kernel development. > > We are welcoming 30 minute presentations from both end users and > developers, on topics covering, but not limited to: > -Investigation workflow of Real-Time, latency, and throughput issues, > -Trace collection and extraction, > -Trace filtering, > -Trace aggregation, > -Trace formats, > -Tracing multi-core systems, > -Trace abstraction, > -Trace modeling, > -Automated trace analysis (e.g. dependency analysis), > -Tracing large clusters and distributed systems, > -Hardware-level tracing (e.g. DSP, GPU, bare-metal), > -Trace visualisation, > -Interaction between debugging and tracing, > -Tracing remote control, > -Analysis of large trace datasets, > -Cloud trace collection and analysis, > -Integration between trace tools, > -Live tracing & monitoring, > -Programmable tracing (e.g. eBPF). > > Those can cover recently available technologies, ongoing work, and yet > non-existing technologies (which are compellingly interesting to > end-users). Please understand that this open forum is not the proper > place to present sales or marketing pitches, nor technologies which > are prevented from being freely used in open source. > > * Submit you talk today using this form: https://forms.gle/L6mR2ZN3Nt7bUxY98 * > > The submission deadline is July 1st 2019 at 23:59 EST. > > This year tickets are $100 and seating is limited to 100 people. To > register, you can add the Tracing Summit as an addon to your Open > Source Summit ticket [3] or use this link [4] to register solely for > the Tracing Summit. > > The Tracing Summit is sponsored by EfficiOS. We are actively looking > for sponsors to support the Tracing Summit. Contact us at > i...@tracingsummit.org. > > Please send any query about this conference to i...@tracingsummit.org. > > This event is organized by Francis Deslauriers and Mathieu Desnoyers > on the behalf of the Linux Foundation Diagnostic and Monitoring > Workgroup [5]. > > [0] : https://tracingsummit.org > [1]: > https://events.linuxfoundation.org/events/open-source-summit-north-america-2019 > [2]: http://eepurl.com/goakfv > [3]: > https://www.cvent.com/events/open-source-summit-embedded-linux-conference-north-america-2019/registration-0ed1330c38bd4845be29b91c9d2444ce.aspx?fqp=true > [4]: > https://www.cvent.com/events/tracing-summit-2019/registration-63351d06945a46d890b8e5a200dbc0fc.aspx?fqp=true > [5]: https://diamon.org > > -- > Francis Deslauriers > Computer Engineer > EfficiOS inc. -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] Tracing Summit 2019 - Schedule Release
Hi all, We're happy to announce the schedule for the Tracing Summit 2019 is now available. We are going to have an amazing conference with diverse topics such as trace analysis, GPU tracing, distributed systems tracing and more. Checkout the schedule and talk abstracts on the website.[1] Tracing Summit 2019 will be held in San Diego, California on August 20th, 2019, Don't forget to register for the summit either when buying your Open Source Summit ticket or using this link [2]! See you in August! The Tracing Summit Organizers [1] https://tracingsummit.org/wiki/TracingSummit2019 [2] https://www.cvent.com/events/tracing-summit-2019/registration-63351d06945a46d890b8e5a200dbc0fc.aspx?fqp=true -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] tracing page_faults with lttng?
Le sam. 2 nov. 2019, à 11 h 41, Milian Wolff via lttng-dev a écrit : > > On Samstag, 12. Oktober 2019 23:31:51 CET Milian Wolff via lttng-dev wrote: > > On Montag, 29. Mai 2017 10:44:50 CEST Milian Wolff wrote: > > > On Wednesday, May 24, 2017 6:26:38 PM CEST Francis Deslauriers wrote: > > > > Hi Milian, > > > > Are you running on an x86 processor? > > > > > > Yes, Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz. > > > > > > > The pagefault tracepoints are called : x86_exceptions_page_fault_user, > > > > x86_exceptions_page_fault_kernel > > > > Can you see those tracepoints when you run: lttng list --kernel > > > > > > Yes: > > > > > > $ lttng list --kernel | grep fault > > > > > > x86_exceptions_page_fault_user (loglevel: TRACE_EMERG (0)) (type: > > > tracepoint) > > > > > > x86_exceptions_page_fault_kernel (loglevel: TRACE_EMERG (0)) (type: > > > tracepoint) > > > > > > kvm_async_pf_doublefault (loglevel: TRACE_EMERG (0)) (type: > > > tracepoint) > > > > Hey there, > > > > a follow up to the above question: What about tracing page faults on ARM? > > Perf can do that, but I see no page_fault trace points in the output of > > `lttng list -k`? > > ping? Hi Milian, I believe that the page fault event you see with Perf is a perf event, not a trace event. What is the name of the event you see exactly? In order to trace the page faults on Arm with LTTng, we will need to add trace event tracepoints to the do_page_fault() functions in the arm (or arm64) architecture of the kernel. Francis > > -- > Milian Wolff | milian.wo...@kdab.com | Senior Software Engineer > KDAB (Deutschland) GmbH, a KDAB Group company > Tel: +49-30-521325470 > KDAB - The Qt, C++ and OpenGL > Experts___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] Tracing Summit 2019 - Videos
Hi all, The videos of the Tracing Summit 2019 talks are now available on Youtube here: https://www.youtube.com/playlist?list=PLuo4E47p5_7ZeRpUZEUyF2kWyA6KzTTkI. See you all next year! -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] convert call_site to func/offset
Hi Changcheng, This is not currently supported by Babeltrace. It could probably be done the same way we do it with the "ip" and "vpid" contexts in the debug-info component class but it's not done yet. Thanks, Francis Le mer. 1 juill. 2020, à 11 h 58, Liu, Changcheng via lttng-dev < lttng-dev@lists.lttng.org> a écrit : > Hi all, > I checked __cyg_profile_func_enter implementation. > The caller address is recorded into call_site field. > > Does anyone know how to let babeltrace show the call_site into > "func/offset" format instead of the running virtual address? > > B.R. > Changcheng > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -- Francis Deslauriers Computer Engineer EfficiOS inc. ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Assertion tgt_fc failed when trying to trace callstack-user
Hi Evan, On 8/1/21 5:11 AM, Evan Galea via lttng-dev wrote: Hi there Thanks for help on my previous issue. I was able to get callstack-user to work and see it in my traces. However, in order to get it to run I needed to enable-event --kernel all. Is it necessary to have all kernel events enabled to run callstack-user? Or is it just one specific event? When I try to run it without enabling all kernel events the command is accepted and it runs but I get the following error when trying to view my traces: (╯°□°)╯︵ ┻━┻ ctf-meta-update-value-storing-indexes.c:65: update_field_class_stored_value_index(): Assertion `tgt_fc` failed. Aborted (core dumped) What does this error mean? Should I just continue to enable all kernel events? It's a Babeltrace 2 error. Now, we need to find out if the root cause is the LTTng kernel tracer or Babeltrace2 itself. What is the "one specific event" you're enabling that seems to cause this issue? Do you have access to a Babeltrace 1.5 build? if so, could you try to read that same trace using it? Could you please share a small trace reproducing this issue? What version of LTTng-tools, LTTng-Modules, and Babeltrace2 are you using? Cheers, Francis My current command setup, not sure if relevant lttng create && lttng enable-channel mychannel --subbuf-size 1 lttng enable-event --kernel --all && lttng add-context -u -t vpid -t vtid -t procname && lttng add-context --kernel --type callstack-user && lttng add-context --kernel --type callstack-kernel && lttng enable-event --jul jello again thank you for your time Evan Galea ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] Assertion tgt_fc failed when trying to trace callstack-user
Hi Evan, It turns out it's a Babeltrace 2 bug. Here the fix: https://review.lttng.org/c/babeltrace/+/6257. It should be merged into the master and stable-2.0 branches soon. Thank you for reporting the issue, Francis On 8/4/21 3:55 AM, Evan Galea wrote: Hi Francis, thanks for your reply. As it turns out, running babeltrace on the file worked and I'm able to read the traces. I am running babeltrace 1.5.8, lttng 2.12.4, lttng-tools 2.12.x stable, and lttng modules 5.8.0-63. I attached the trace of the file. On Mon, Aug 2, 2021 at 8:56 AM Francis Deslauriers <mailto:francis.deslauri...@efficios.com>> wrote: Hi Evan, On 8/1/21 5:11 AM, Evan Galea via lttng-dev wrote: Hi there Thanks for help on my previous issue. I was able to get callstack-user to work and see it in my traces. However, in order to get it to run I needed to enable-event --kernel all. Is it necessary to have all kernel events enabled to run callstack-user? Or is it just one specific event? When I try to run it without enabling all kernel events the command is accepted and it runs but I get the following error when trying to view my traces: (╯°□°)╯︵ ┻━┻ ctf-meta-update-value-storing-indexes.c:65: update_field_class_stored_value_index(): Assertion `tgt_fc` failed. Aborted (core dumped) What does this error mean? Should I just continue to enable all kernel events? It's a Babeltrace 2 error. Now, we need to find out if the root cause is the LTTng kernel tracer or Babeltrace2 itself. What is the "one specific event" you're enabling that seems to cause this issue? Do you have access to a Babeltrace 1.5 build? if so, could you try to read that same trace using it? Could you please share a small trace reproducing this issue? What version of LTTng-tools, LTTng-Modules, and Babeltrace2 are you using? Cheers, Francis My current command setup, not sure if relevant lttng create && lttng enable-channel mychannel --subbuf-size 1 lttng enable-event --kernel --all && lttng add-context -u -t vpid -t vtid -t procname && lttng add-context --kernel --type callstack-user && lttng add-context --kernel --type callstack-kernel && lttng enable-event --jul jello again thank you for your time Evan Galea ___ lttng-dev mailing list lttng-dev@lists.lttng.org <mailto:lttng-dev@lists.lttng.org> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev <https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev> ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
