----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <shehaby...@gmail.com> wrote:
> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid " and made sure the > value was actually written by "cat /proc/sys/kernel/perf_event_paranoid" > It executed normally 2 times but then got the same error. Can you provide the output when reproducing the issue with the LTTNG_UST_DEBUG=1 environment variable set when starting your test program ? Thanks, Mathieu > Shehab Y. Elsayed, MSc. > PhD Student > The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering > University of Toronto > E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | > shehaby...@gmail.com ] > On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers < [ > mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] > > wrote: >> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed < [ >> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1 >>> I run the program as a normal user >>> The glibc version I get by running "ldd --version" is 2.17 >> Can you reproduce the issue after you do this as root ? >> echo "-1" > /proc/sys/kernel/perf_event_paranoid >> Based on this documentation of the Linux kernel: >> Documentation/sysctl/kernel.txt: >> perf_event_paranoid: >> Controls use of the performance events system by unprivileged >> users (without CAP_SYS_ADMIN). The default value is 2. >> -1: Allow use of (almost) all events by all users >> Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK >> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN >> Disallow raw tracepoint access by users without CAP_SYS_ADMIN >> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN >> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN >> Thanks, >> Mathieu >>> Shehab Y. Elsayed, MSc. >>> PhD Student >>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>> University of Toronto >>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | >>> shehaby...@gmail.com ] >>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [ >>> mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] > >>> wrote: >>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [ >>>> mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] > >>>> wrote: >>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [ >>>>> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>>>>> Yes, I tried with only one of those contexts and I still ran into the >>>>>> same >>>>>> problem. >>>>> What is the setting returned by >>>>> cat /proc/sys/kernel/perf_event_paranoid >>>>> on your system ? And do you run your test program as root or normal user ? >>>>> Please CC the mailing list on your reply. >>>> I will also need to know what glibc version you have on your system. >>>> Thanks, >>>> Mathieu >>>>> Thanks, >>>>> Mathieu >>>>>> Shehab Y. Elsayed, MSc. >>>>>> PhD Student >>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>>>>> University of Toronto >>>>>> E-mail: [ >>>>>> https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | >>>>>> shehaby...@gmail.com ] >>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [ >>>>>> mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com ] >>>>>> > >>>>>> wrote: >>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [ >>>>>>> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>>>>>>> Hi Mathieu, >>>>>>>> Thank you very much for your reply. >>>>>>>> I manually built lttng-ust from source (commit #: >>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a). >>>>>>>> This is how I set up the tracing session: >>>>>>>> 1- lttng create lu_ncb_8_native -o {path} >>>>>>>> 2- lttng enable-event --userspace >>>>>>>> lttng_ust_pthread:pthread_mutex_lock_req >>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq >>>>>>>> lttng enable-event --userspace >>>>>>>> lttng_ust_pthread:pthread_mutex_lock_trylock >>>>>>>> lttng enable-event --userspace >>>>>>>> lttng_ust_pthread:pthread_mutex_lock_unlock >>>>>>>> 3- lttng add-context -u -t procname >>>>>>>> lttng add-context -u -t vpid >>>>>>>> lttng add-context -u -t pthread_id >>>>>>>> lttng add-context -u -t vtid >>>>>>>> lttng add-context -u -t ip >>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles >>>>>>>> lttng add-context -u -t perf:thread:cycles >>>>>>>> lttng add-context -u -t perf:thread:instructions >>>>>>>> 4- lttng start >>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb >>>>>>>> -p8 -n8096 >>>>>>>> -b32 >>>>>>>> 6- lttng stop >>>>>>>> 7- lttng destroy >>>>>>> Can you reproduce if you remove the following contexts ? >>>>>>> lttng add-context -u -t perf:thread:cpu-cycles >>>>>>> lttng add-context -u -t perf:thread:cycles >>>>>>> lttng add-context -u -t perf:thread:instructions >>>>>>> And if you only keep a single of those contexts ? >>>>>>> Thanks, >>>>>>> Mathieu >>>>>>>> Shehab Y. Elsayed, MSc. >>>>>>>> PhD Student >>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>>>>>>> University of Toronto >>>>>>>> E-mail: [ >>>>>>>> https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | >>>>>>>> shehaby...@gmail.com ] >>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [ >>>>>>>> mailto:mathieu.desnoy...@efficios.com | mathieu.desnoy...@efficios.com >>>>>>>> ] > >>>>>>>> wrote: >>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [ >>>>>>>>> mailto:shehaby...@gmail.com | shehaby...@gmail.com ] > wrote: >>>>>>>>>> Hello All, >>>>>>>>>> I am trying to instrument a pthread application using the provided >>>>>>>>>> pthread >>>>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error >>>>>>>>>> ( fasttop >>>>>>>>>> )" error. >>>>>>>>> Please provide more information about the version of lttng-ust you >>>>>>>>> are using, >>>>>>>>> and how you setup >>>>>>>>> your tracing session. >>>>>>>>> Thanks, >>>>>>>>> Mathieu >>>>>>>>>> Here is a description of what I have tried and noticed: >>>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes >>>>>>>>>> works as >>>>>>>>>> expected. >>>>>>>>>> 2- From my experiments, the problem happens (more frequently at >>>>>>>>>> least) when >>>>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and >>>>>>>>>> instructions). >>>>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after >>>>>>>>>> setting LD _ >>>>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng >>>>>>>>>> documents. >>>>>>>>>> 4- Here is the backtrace printed after exiting: >>>>>>>>>> ======= Backtrace : ========= >>>>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited] >>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng >>>>>>>>>> _destroy_context+0x35)[0x7ffff7471575] >>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng >>>>>>>>>> _session_destroy+0x21c)[0x7ffff747363c] >>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906] >>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>>>>>> +0x9f)[0x7ffff746dccf] >>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>>>>>> +0x9f)[0x7ffff746dccf] >>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>>>>>> +0x9f)[0x7ffff746dccf] >>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi >>>>>>>>>> _exit+0x68)[0x7ffff746ead8] >>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3] >>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust >>>>>>>>>> _exit+0x67)[0x7ffff745ed57] >>>>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a] >>>>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49] >>>>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95] >>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51] >>>>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35] >>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44] >>>>>>>>>> 5- Also, this is a backtrace I obtained from gdb : >>>>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6 >>>>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6 >>>>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6 >>>>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6 >>>>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field ( >>>>>>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418 >>>>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context ( >>>>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278 >>>>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap ( >>>>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172 >>>>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900) >>>>>>>>>> at lttng -events.c:247 >>>>>>>>>> #8 0x00007ffff7676906 in lttng _release_session ( >>>>>>>>>> objd =<optimized out>) at lttng - ust - abi .c:601 >>>>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1, >>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216 >>>>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2, >>>>>>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216 >>>>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18, >>>>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216 >>>>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy () >>>>>>>>>> at lttng - ust - abi .c:235 >>>>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002 >>>>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1) >>>>>>>>>> at lttng - ust -comm.c:1807 >>>>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit () >>>>>>>>>> at lttng - ust -comm.c:1874 >>>>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini () >>>>>>>>>> from /lib64/ ld - linux -x86-64.so.2 >>>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers () >>>>>>>>>> from /lib64/ libc .so.6 >>>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6 >>>>>>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>, >>>>>>>>>> argv =<optimized out>) at lu .c:368 >>>>>>>>>> Any ideas, why this happens and how to fix it? >>>>>>>>>> Thanks, >>>>>>>>>> Shehab >>>>>>>>>> _______________________________________________ >>>>>>>>>> lttng-dev mailing list >>>>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ] >>>>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | >>>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] >>>>>>>>> -- >>>>>>>>> Mathieu Desnoyers >>>>>>>>> EfficiOS Inc. >>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>>>>>> -- >>>>>>> Mathieu Desnoyers >>>>>>> EfficiOS Inc. >>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>>>> -- >>>>> Mathieu Desnoyers >>>>> EfficiOS Inc. >>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>>> -- >>>> Mathieu Desnoyers >>>> EfficiOS Inc. >>>> [ http://www.efficios.com/ | http://www.efficios.com ] >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> [ http://www.efficios.com/ | http://www.efficios.com ] -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
_______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev