On Fri, Nov 01, 2013 at 08:18:59PM +0000, Mathieu Desnoyers wrote: > ----- Original Message ----- > > From: "Mathieu Desnoyers" <mathieu.desnoy...@efficios.com> > > To: "Vladimir Nikulichev" <n...@tbricks.com> > > Cc: lttng-dev@lists.lttng.org, "Paul E. McKenney" > > <paul...@linux.vnet.ibm.com> > > Sent: Friday, November 1, 2013 9:55:14 AM > > Subject: Re: [lttng-dev] bug in urcu > > > > ----- Original Message ----- > > > From: "Mathieu Desnoyers" <mathieu.desnoy...@efficios.com> > > > To: "Vladimir Nikulichev" <n...@tbricks.com> > > > Cc: "Paul E. McKenney" <paul...@linux.vnet.ibm.com> > > > Sent: Friday, November 1, 2013 9:42:16 AM > > > Subject: Re: bug in urcu > > > > > > ----- Original Message ----- > > > > From: "Vladimir Nikulichev" <n...@tbricks.com> > > > > To: "Mathieu Desnoyers" <mathieu.desnoy...@efficios.com> > > > > Cc: "Paul E. McKenney" <paul...@linux.vnet.ibm.com> > > > > Sent: Tuesday, October 22, 2013 1:32:11 PM > > > > Subject: Re: bug in urcu > > > > > > > > > > > > > > It looks like an issue with the thread-local storage (TLS) > > > > > compatibility > > > > > layer. > > > > > > > > > > Can you show me the output of ./configure on that machine ? I'm > > > > > especially > > > > > interested in the output of: > > > > > > > > > > Thread Local Storage (TLS): __thread. (example on my machine) > > > > > > > > It uses pthread_getspecific() by default: > > > > > > Good catch! > > > > > > looking at the output of > > > > > > nm tests/unit/.libs/test_urcu_multiflavor : > > > > > > U __tls_access_rcu_reader > > > > > > seems to be the issue. We're missing macro expansion in tls-compat.h. With > > > the patch attached: > > > > > > U __tls_access_rcu_reader_bp > > > U __tls_access_rcu_reader_mb > > > U __tls_access_rcu_reader_memb > > > U __tls_access_rcu_reader_sig > > > > > > which should fix your issue. Can you try it out and let me know if it > > > fixes > > > your problem ? > > > > Extra question (Paul ? Adding lttng-dev in CC): > > > > Please note that this affects an unusual configuration of userspace RCU > > (with TLS pthread key fallback), needed for some BSD that don't support > > compiler TLS. Strictly speaking, this should require bumping the URCU > > library soname version major number, because it breaks the ABI presented > > to applications on those unusual configurations. However, since this is > > only for unusual configurations, I wonder if we should bump the soname > > version major number or not ? If we do need to bump the soname, can we > > really do this in a stable version fix (0.7, 0.8), or do we need to push > > a 0.9 out and document the limitation for 0.7 and 0.8 ? > > I just found a way to keep ABI compatibility for 0.7 and 0.8, and abort() the > application if it's using the old ABI (with symbol clash) when there are > multiple instances of this symbol loaded. This involves tricks with weak > symbols, a constructor, a reference count, and a wrapper around the bogus > symbol, but it works !! Note that it only affects users of urcu that have > _LGPL_SOURCE defined.
Sounds good to me! ;-) I trust that you have also added copious comments... Thanx, Paul > > > > checking whether gcc accepts -g... (cached) yes > > > > checking for gcc option to accept ISO C89... (cached) none needed > > > > checking whether gcc understands -c and -o together... (cached) yes > > > > checking dependency style of gcc... (cached) gcc3 > > > > checking whether make sets $(MAKE)... (cached) yes > > > > checking how to print strings... printf > > > > checking for a sed that does not truncate output... /usr/bin/sed > > > > checking for grep that handles long lines and -e... /usr/bin/grep > > > > checking for egrep... /usr/bin/grep -E > > > > checking for fgrep... /usr/bin/grep -F > > > > checking for ld used by gcc... > > > > /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld > > > > checking if the linker > > > > (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) is GNU > > > > ld... > > > > no > > > > checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm > > > > checking the name lister (/usr/bin/nm) interface... BSD nm > > > > checking whether ln -s works... yes > > > > checking the maximum length of command line arguments... 196608 > > > > checking whether the shell understands some XSI constructs... yes > > > > checking whether the shell understands "+="... yes > > > > checking how to convert x86_64-apple-darwin12.5.0 file names to > > > > x86_64-apple-darwin12.5.0 format... func_convert_file_noop > > > > checking how to convert x86_64-apple-darwin12.5.0 file names to > > > > toolchain > > > > format... func_convert_file_noop > > > > checking for /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld > > > > option to reload object files... -r > > > > checking for objdump... no > > > > checking how to recognize dependent libraries... pass_all > > > > checking for dlltool... no > > > > checking how to associate runtime and link libraries... printf %s\n > > > > checking for ar... ar > > > > checking for archiver @FILE support... no > > > > checking for strip... strip > > > > checking for ranlib... ranlib > > > > checking command to parse /usr/bin/nm output from gcc object... ok > > > > checking for sysroot... no > > > > checking for mt... no > > > > checking if : is a manifest tool... no > > > > checking for dsymutil... dsymutil > > > > checking for nmedit... nmedit > > > > checking for lipo... lipo > > > > checking for otool... otool > > > > checking for otool64... no > > > > checking for -single_module linker flag... yes > > > > checking for -exported_symbols_list linker flag... yes > > > > checking for -force_load linker flag... yes > > > > checking how to run the C preprocessor... gcc -E > > > > checking for ANSI C header files... yes > > > > checking for sys/types.h... yes > > > > checking for sys/stat.h... yes > > > > checking for stdlib.h... yes > > > > checking for string.h... yes > > > > checking for memory.h... yes > > > > checking for strings.h... yes > > > > checking for inttypes.h... yes > > > > checking for stdint.h... yes > > > > checking for unistd.h... yes > > > > checking for dlfcn.h... yes > > > > checking for objdir... .libs > > > > checking if gcc supports -fno-rtti -fno-exceptions... no > > > > checking for gcc option to produce PIC... -fno-common -DPIC > > > > checking if gcc PIC flag -fno-common -DPIC works... yes > > > > checking if gcc static flag -static works... no > > > > checking if gcc supports -c -o file.o... yes > > > > checking if gcc supports -c -o file.o... (cached) yes > > > > checking whether the gcc linker > > > > (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) supports > > > > shared > > > > libraries... yes > > > > checking dynamic linker characteristics... darwin12.5.0 dyld > > > > checking how to hardcode library paths into programs... immediate > > > > checking whether stripping libraries is possible... yes > > > > checking if libtool supports shared libraries... yes > > > > checking whether to build shared libraries... yes > > > > checking whether to build static libraries... yes > > > > checking for inline... inline > > > > checking for pid_t... yes > > > > checking for size_t... yes > > > > checking for stdlib.h... (cached) yes > > > > checking for GNU libc compatible malloc... yes > > > > checking for stdlib.h... (cached) yes > > > > checking for unistd.h... (cached) yes > > > > checking for sys/param.h... yes > > > > checking for getpagesize... yes > > > > checking for working mmap... yes > > > > checking for bzero... yes > > > > checking for gettimeofday... yes > > > > checking for munmap... yes > > > > checking for sched_getcpu... no > > > > checking for strtoul... yes > > > > checking for sysconf... yes > > > > checking if architecture really supports the mfence instruction... yes > > > > checking for sys_futex()... no > > > > checking for cpu_set_t... no > > > > checking whether CPU_ZERO works... no > > > > checking whether CPU_SET works... no > > > > checking for sched_setaffinity... no > > > > checking that generated files are newer than configure... done > > > > configure: creating ./config.status > > > > config.status: creating Makefile > > > > config.status: creating doc/Makefile > > > > config.status: creating doc/examples/Makefile > > > > config.status: creating tests/Makefile > > > > config.status: creating tests/common/Makefile > > > > config.status: creating tests/unit/Makefile > > > > config.status: creating tests/benchmark/Makefile > > > > config.status: creating tests/regression/Makefile > > > > config.status: creating liburcu.pc > > > > config.status: creating liburcu-bp.pc > > > > config.status: creating liburcu-cds.pc > > > > config.status: creating liburcu-qsbr.pc > > > > config.status: creating liburcu-mb.pc > > > > config.status: creating liburcu-signal.pc > > > > config.status: creating config.h > > > > config.status: creating urcu/config.h > > > > config.status: linking urcu/arch/x86.h to urcu/arch.h > > > > config.status: linking urcu/uatomic/x86.h to urcu/uatomic.h > > > > config.status: executing depfiles commands > > > > config.status: executing libtool commands > > > > SMP support enabled. > > > > Thread Local Storage (TLS): pthread_getspecific(). > > > > > > > > > > > > > > > > It seems to be another object file/static data issue. When it works with > > > > rcu_reader, it first initializes it (pthread_key_create) from liburcu.2, > > > > but > > > > latter stacks come from liburcu-bp.2. > > > > I ran it in debugger with breakpoints at pthread_key_create(), > > > > pthread_getspecific(), pthread_setspecific(). Each hit I recorded key > > > > numbers, values, etc. First time a library assigns the key number 0x103, > > > > second time 0x104. > > > > > > > > (lldb) bt > > > > * thread #1: tid = 0x1c03, 0x00007fff8a5c0224 > > > > libsystem_c.dylib`pthread_key_create, stop reason = breakpoint 1.1 > > > > frame #0: 0x00007fff8a5c0224 libsystem_c.dylib`pthread_key_create > > > > frame #1: 0x0000000100004cea liburcu.2.dylib`__tls_access_rcu_reader > > > > + > > > > 58 > > > > at urcu.c:110 > > > > frame #2: 0x0000000100000c5e test_urcu_multiflavor`test_mf_bp > > > > [inlined] > > > > _rcu_read_lock_bp + 10 at urcu-bp.h:210 > > > > frame #3: 0x0000000100000c54 test_urcu_multiflavor`test_mf_bp + 4 at > > > > test_urcu_multiflavor-bp.c:34 > > > > frame #4: 0x0000000100000989 test_urcu_multiflavor`main() + 9 at > > > > test_urcu_multiflavor.c:44 > > > > frame #5: 0x00007fff8d8d67e1 libdyld.dylib`start + 1 > > > > > > > > (lldb) register read rdi rsi > > > > rdi = 0x0000000100008300 __tls_rcu_reader.5085 > > > > rsi = 0x00007fff8a5d2801 libsystem_c.dylib`free > > > > > > > > ... continue ... > > > > > > > > > > > > (lldb) bt > > > > * thread #1: tid = 0x1c03, 0x00007fff8a5a6128 > > > > libsystem_c.dylib`pthread_getspecific, stop reason = breakpoint 2.1 > > > > frame #0: 0x00007fff8a5a6128 libsystem_c.dylib`pthread_getspecific > > > > frame #1: 0x0000000100004d0c liburcu.2.dylib`__tls_access_rcu_reader > > > > + > > > > 92 > > > > at urcu.c:110 > > > > frame #2: 0x0000000100000c5e test_urcu_multiflavor`test_mf_bp > > > > [inlined] > > > > _rcu_read_lock_bp + 10 at urcu-bp.h:210 > > > > frame #3: 0x0000000100000c54 test_urcu_multiflavor`test_mf_bp + 4 at > > > > test_urcu_multiflavor-bp.c:34 > > > > frame #4: 0x0000000100000989 test_urcu_multiflavor`main() + 9 at > > > > test_urcu_multiflavor.c:44 > > > > frame #5: 0x00007fff8d8d67e1 libdyld.dylib`start + 1 > > > > (lldb) register read rdi > > > > rdi = 0x0000000000000103 > > > > > > > > ... continue ... > > > > > > > > > > > > (lldb) bt > > > > * thread #1: tid = 0x1c03, 0x00007fff8a5c03a0 > > > > libsystem_c.dylib`pthread_setspecific, stop reason = breakpoint 3.1 > > > > frame #0: 0x00007fff8a5c03a0 libsystem_c.dylib`pthread_setspecific > > > > frame #1: 0x0000000100004d32 liburcu.2.dylib`__tls_access_rcu_reader > > > > + > > > > 130 at urcu.c:110 > > > > frame #2: 0x0000000100000c5e test_urcu_multiflavor`test_mf_bp > > > > [inlined] > > > > _rcu_read_lock_bp + 10 at urcu-bp.h:210 > > > > frame #3: 0x0000000100000c54 test_urcu_multiflavor`test_mf_bp + 4 at > > > > test_urcu_multiflavor-bp.c:34 > > > > frame #4: 0x0000000100000989 test_urcu_multiflavor`main() + 9 at > > > > test_urcu_multiflavor.c:44 > > > > frame #5: 0x00007fff8d8d67e1 libdyld.dylib`start + 1 > > > > > > > > > > > > (lldb) register read rdi rsi > > > > rdi = 0x0000000000000103 > > > > rsi = 0x00000001001038d0 > > > > > > > > ... continue ... > > > > > > > > > > > > (lldb) bt > > > > * thread #1: tid = 0x1c03, 0x00007fff8a5c0224 > > > > libsystem_c.dylib`pthread_key_create, stop reason = breakpoint 1.1 > > > > frame #0: 0x00007fff8a5c0224 libsystem_c.dylib`pthread_key_create > > > > frame #1: 0x00000001000288ca > > > > liburcu-bp.2.dylib`__tls_access_rcu_reader > > > > + > > > > 58 at urcu-bp.c:117 > > > > frame #2: 0x0000000100029bc5 liburcu-bp.2.dylib`rcu_bp_register + 53 > > > > at > > > > urcu-bp.c:483 > > > > frame #3: 0x0000000100000c69 test_urcu_multiflavor`test_mf_bp > > > > [inlined] > > > > _rcu_read_lock_bp + 21 at urcu-bp.h:211 > > > > frame #4: 0x0000000100000c54 test_urcu_multiflavor`test_mf_bp + 4 at > > > > test_urcu_multiflavor-bp.c:34 > > > > frame #5: 0x0000000100000989 test_urcu_multiflavor`main() + 9 at > > > > test_urcu_multiflavor.c:44 > > > > frame #6: 0x00007fff8d8d67e1 libdyld.dylib`start + 1 > > > > > > > > > > > > static/urcu-bp.h: > > > > 206 static inline void _rcu_read_lock(void) > > > > 207 { > > > > 208 unsigned long tmp; > > > > 209 > > > > 210 if (caa_unlikely(!URCU_TLS(rcu_reader))) > > > > 211 rcu_bp_register(); /* If not yet registered. */ > > > > 212 cmm_barrier(); /* Ensure the compiler does not reorder us with > > > > mutex > > > > */ > > > > 213 tmp = URCU_TLS(rcu_reader)->ctr; > > > > 214 _rcu_read_lock_update(tmp); > > > > 215 } > > > > > > > > > > > > ... continue ... > > > > > > > > > > > > (lldb) bt > > > > * thread #1: tid = 0x1c03, 0x00007fff8a5a6131 > > > > libsystem_c.dylib`pthread_getspecific + 9, stop reason = instruction > > > > step > > > > over > > > > frame #0: 0x00007fff8a5a6131 libsystem_c.dylib`pthread_getspecific + > > > > 9 > > > > frame #1: 0x00000001000288ec > > > > liburcu-bp.2.dylib`__tls_access_rcu_reader > > > > + > > > > 92 at urcu-bp.c:117 > > > > frame #2: 0x0000000100029bc5 liburcu-bp.2.dylib`rcu_bp_register + 53 > > > > at > > > > urcu-bp.c:483 > > > > frame #3: 0x0000000100000c69 test_urcu_multiflavor`test_mf_bp > > > > [inlined] > > > > _rcu_read_lock_bp + 21 at urcu-bp.h:211 > > > > frame #4: 0x0000000100000c54 test_urcu_multiflavor`test_mf_bp + 4 at > > > > test_urcu_multiflavor-bp.c:34 > > > > frame #5: 0x0000000100000989 test_urcu_multiflavor`main() + 9 at > > > > test_urcu_multiflavor.c:44 > > > > frame #6: 0x00007fff8d8d67e1 libdyld.dylib`start + 1 > > > > > > > > (lldb) register read rdi > > > > rdi = 0x0000000000000104 > > > > > > > > ... continue ... > > > > > > > > > > > > (lldb) bt > > > > * thread #1: tid = 0x1c03, 0x00007fff8a5c03a0 > > > > libsystem_c.dylib`pthread_setspecific, stop reason = breakpoint 3.1 > > > > frame #0: 0x00007fff8a5c03a0 libsystem_c.dylib`pthread_setspecific > > > > frame #1: 0x0000000100028912 > > > > liburcu-bp.2.dylib`__tls_access_rcu_reader > > > > + > > > > 130 at urcu-bp.c:117 > > > > frame #2: 0x0000000100029bc5 liburcu-bp.2.dylib`rcu_bp_register + 53 > > > > at > > > > urcu-bp.c:483 > > > > frame #3: 0x0000000100000c69 test_urcu_multiflavor`test_mf_bp > > > > [inlined] > > > > _rcu_read_lock_bp + 21 at urcu-bp.h:211 > > > > frame #4: 0x0000000100000c54 test_urcu_multiflavor`test_mf_bp + 4 at > > > > test_urcu_multiflavor-bp.c:34 > > > > frame #5: 0x0000000100000989 test_urcu_multiflavor`main() + 9 at > > > > test_urcu_multiflavor.c:44 > > > > frame #6: 0x00007fff8d8d67e1 libdyld.dylib`start + 1 > > > > > > > > (lldb) register read rdi rsi > > > > rdi = 0x0000000000000104 > > > > rsi = 0x00000001001000e0 > > > > > > > > ... continue, skip add_thread() ... > > > > > > > > > > > > (lldb) bt > > > > * thread #1: tid = 0x1c03, 0x00007fff8a5a6128 > > > > libsystem_c.dylib`pthread_getspecific, stop reason = breakpoint 2.1 > > > > frame #0: 0x00007fff8a5a6128 libsystem_c.dylib`pthread_getspecific > > > > frame #1: 0x00000001000288ec > > > > liburcu-bp.2.dylib`__tls_access_rcu_reader > > > > + > > > > 92 at urcu-bp.c:117 > > > > frame #2: 0x0000000100029e46 liburcu-bp.2.dylib`rcu_bp_register > > > > [inlined] > > > > add_thread + 470 at urcu-bp.c:425 > > > > frame #3: 0x0000000100029c70 liburcu-bp.2.dylib`rcu_bp_register + > > > > 224 > > > > at > > > > urcu-bp.c:492 > > > > frame #4: 0x0000000100000c69 test_urcu_multiflavor`test_mf_bp > > > > [inlined] > > > > _rcu_read_lock_bp + 21 at urcu-bp.h:211 > > > > frame #5: 0x0000000100000c54 test_urcu_multiflavor`test_mf_bp + 4 at > > > > test_urcu_multiflavor-bp.c:34 > > > > frame #6: 0x0000000100000989 test_urcu_multiflavor`main() + 9 at > > > > test_urcu_multiflavor.c:44 > > > > frame #7: 0x00007fff8d8d67e1 libdyld.dylib`start + 1 > > > > > > > > (lldb) register read rdi > > > > rdi = 0x0000000000000104 > > > > > > > > (lldb) n > > > > Process 59961 stopped > > > > * thread #1: tid = 0x1c03, 0x00007fff8a5a6131 > > > > libsystem_c.dylib`pthread_getspecific + 9, stop reason = instruction > > > > step > > > > over > > > > frame #0: 0x00007fff8a5a6131 libsystem_c.dylib`pthread_getspecific + > > > > 9 > > > > libsystem_c.dylib`pthread_getspecific + 9: > > > > -> 0x7fff8a5a6131: ret > > > > > > > > (lldb) register read rax > > > > rax = 0x00000001001000e0 > > > > > > > > ... next ... > > > > -> 425 URCU_TLS(rcu_reader) = rcu_reader_reg; > > > > ... next ... > > > > ... continue ... > > > > > > > > > > > > (lldb) bt > > > > * thread #1: tid = 0x1c03, 0x00007fff8a5a6128 > > > > libsystem_c.dylib`pthread_getspecific, stop reason = breakpoint 2.1 > > > > frame #0: 0x00007fff8a5a6128 libsystem_c.dylib`pthread_getspecific > > > > frame #1: 0x0000000100004d0c liburcu.2.dylib`__tls_access_rcu_reader > > > > + > > > > 92 > > > > at urcu.c:110 > > > > frame #2: 0x0000000100000c6e test_urcu_multiflavor`test_mf_bp > > > > [inlined] > > > > _rcu_read_lock_bp + 26 at urcu-bp.h:213 > > > > frame #3: 0x0000000100000c54 test_urcu_multiflavor`test_mf_bp + 4 at > > > > test_urcu_multiflavor-bp.c:34 > > > > frame #4: 0x0000000100000989 test_urcu_multiflavor`main() + 9 at > > > > test_urcu_multiflavor.c:44 > > > > frame #5: 0x00007fff8d8d67e1 libdyld.dylib`start + 1 > > > > (lldb) register read rdi > > > > rdi = 0x0000000000000103 > > > > (lldb) c > > > > Process 59961 resuming > > > > Process 59961 stopped > > > > * thread #1: tid = 0x1c03, 0x0000000100000c71 > > > > test_urcu_multiflavor`test_mf_bp [inlined] _rcu_read_lock_bp + 29 at > > > > urcu-bp.h:213, stop reason = EXC_BAD_ACCESS (code=1, address=0x0) > > > > frame #0: 0x0000000100000c71 test_urcu_multiflavor`test_mf_bp > > > > [inlined] > > > > _rcu_read_lock_bp + 29 at urcu-bp.h:213 > > > > 210 if (caa_unlikely(!URCU_TLS(rcu_reader))) > > > > 211 rcu_bp_register(); /* If not yet registered. */ > > > > 212 cmm_barrier(); /* Ensure the compiler does not reorder us > > > > with > > > > mutex */ > > > > -> 213 tmp = URCU_TLS(rcu_reader)->ctr; > > > > 214 _rcu_read_lock_update(tmp); > > > > 215 } > > > > 216 > > > > > > > > CRASH > > > > > > -- > > > Mathieu Desnoyers > > > EfficiOS Inc. > > > http://www.efficios.com > > > > > > > -- > > Mathieu Desnoyers > > EfficiOS Inc. > > http://www.efficios.com > > > > _______________________________________________ > > lttng-dev mailing list > > lttng-dev@lists.lttng.org > > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com > _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev