[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[pawel_sikora at zoho dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #12 from Pawel Sikora  ---
(In reply to Maxim Ostapenko from comment #11)
> Created attachment 39882 [details]
> Untested fix
> 
> Untested fix (works for me with attached testcase).
> 

this fix works for me. thanks.

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[m.ostapenko at samsung dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #11 from Maxim Ostapenko  ---
Created attachment 39882
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=39882=edit
Untested fix

Untested fix (works for me with attached testcase).

To sum up:

1) dlopen grabs a __GI___pthread_mutex_lock in main thread.
2) main thread calls pthread_create, ASan intercepts it, calls real
pthread_create and waits for the second thread to be "fully initialized".
3) Newly created thread tries to access a thread local "disable_counter" in
LSan (to complete its "full initialization") and hangs in tls_get_addr_tail,
because it also tries to acquire __GI___pthread_mutex_lock.

The issue doesn't reproduce on older Glibc (e.g. 2.19 on my Ubuntu box) because
tls_get_addr_tail doesn't try to acquire __GI___pthread_mutex_lock there (yes,
2.19 and 2.23 have many differences in TLS implementation).

The deadlock doesn't occur with ASan static linkage because in this case
"disable_counter" resides in static tls, thus tls_get_addr_tail isn't called.

The simple fix would be just using __attribute__((tls_model("initial-exec")))
for  "disable_counter" in LSan. This should be fine since nobody would dlopen
{A, L}San runtime in any case.

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[m.ostapenko at samsung dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #10 from Maxim Ostapenko  ---
(In reply to Pawel Sikora from comment #9)
> (In reply to Maxim Ostapenko from comment #8)
> 
> > Hm, perhaps environment issue. What version of Glibc do you use?
> 
> glibc-2.23.1-10.fc24.x86_64

Reproduced with current trunk Glibc:

max@max:/tmp/bug$ make
rm -f m *.so
~/install/master/bin/g++ -g2 -Og -flto -fsanitize=address
-L/home/max/install/glibc/usr/lib -L/home/max/install/glibc/lib
-I/home/max/install/glibc/include  -Wl,-rpath=/home/max/install/glibc/lib
-Wl,--dynamic-linker=/home/max/install/glibc/lib/ld-2.24.90.so s.cpp -shared -o
s.so -fPIC
~/install/master/bin/g++ -g2 -Og -flto -fsanitize=address
-L/home/max/install/glibc/usr/lib -L/home/max/install/glibc/lib
-I/home/max/install/glibc/include  -Wl,-rpath=/home/max/install/glibc/lib
-Wl,--dynamic-linker=/home/max/install/glibc/lib/ld-2.24.90.so m.cpp -o m
max@max:/tmp/bug$ LD_LIBRARY_PATH=/home/max/install/master/lib64/ ./m 
initializing library...
^C


Seems to be ASan + recent Glibc (probably 2.23+) issue.

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[pawel_sikora at zoho dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #9 from Pawel Sikora  ---
(In reply to Maxim Ostapenko from comment #8)

> Hm, perhaps environment issue. What version of Glibc do you use?

glibc-2.23.1-10.fc24.x86_64

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[m.ostapenko at samsung dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #8 from Maxim Ostapenko  ---
(In reply to Pawel Sikora from comment #7)
> (In reply to Maxim Ostapenko from comment #6)
> > The attached testcase works for me with current trunk GCC:
> > 
> > max@max:/tmp/bug$ make
> > rm -f m *.so
> > ~/install/master/bin/g++ -fuse-ld=gold -g2 -Og -flto -fsanitize=address
> > s.cpp -shared -o s.so -fPIC
> > ~/install/master/bin/g++ -fuse-ld=gold -g2 -Og -flto -fsanitize=address
> > m.cpp -o m
> > max@max:/tmp/bug$ LD_LIBRARY_PATH=/home/max/install/master/lib64 ./m 
> > initializing library...
> > done.
> > thread started.
> > max@max:/tmp/bug$ echo $?
> > 0
> 
> 
> strange, i've tested gcc-trunk and it locks in the same way as 6.2.1.
> 
> 
> ~/src/gcc-install/usr/local/bin/g++ -v
> Using built-in specs.
> COLLECT_GCC=/home/pawels/src/gcc-install/usr/local/bin/g++
> COLLECT_LTO_WRAPPER=/home/pawels/src/gcc-install/usr/local/bin/../libexec/
> gcc/x86_64-pc-linux-gnu/7.0.0/lto-wrapper
> Target: x86_64-pc-linux-gnu
> Configured with: /home/pawels/src/gcc/configure --with-arch=x86-64
> --with-linker-hash-style=gnu --disable-multilib --disable-nls
> --disable-libssp --disable-libgomp --disable-libquadmath --disable-libitm
> --disable-libcilkrts --disable-libvtv --disable-liboffloadmic
> --disable-libmpx --enable-tls --enable-libstdcxx-allocator=new
> --enable-extern-template --enable-libstdcxx-time=rt
> --enable-libstdcxx-threads --disable-libstdcxx-dual-abi
> --enable-libstdcxx-filesystem-ts=no --enable-symvers=gnu-versioned-namespace
> --disable-libstdcxx-pch --enable-lto --enable-plugin --enable-c99
> --enable-long-long --enable-linux-futex --enable-threads=posix
> --enable-shared --with-pic --enable-gold --enable-__cxa_atexit
> --enable-gnu-unique-object --enable-initfini-array --enable-languages=c,c++
> --enable-checking=release --with-long-double-128 --disable-cld
> --disable-bootstrap
> Thread model: posix
> gcc version 7.0.0 20161025 (experimental) (GCC) 
> 
> ~/src/gcc-install/usr/local/bin/g++ -fuse-ld=gold -g2 -Og -fsanitize=address
> -Wl,-rpath,/home/pawels/src/gcc-install/usr/local/lib64 -flto s.cpp -shared
> -o s.so -fPIC -pthread
> ~/src/gcc-install/usr/local/bin/g++ -fuse-ld=gold -g2 -Og -fsanitize=address
> -Wl,-rpath,/home/pawels/src/gcc-install/usr/local/lib64 -flto m.cpp -o m -ldl
> 
> [pawels@pawels]~/src/bug% ./m
> initializing library...
> ^C

Hm, perhaps environment issue. What version of Glibc do you use?

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[pawel_sikora at zoho dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #7 from Pawel Sikora  ---
(In reply to Maxim Ostapenko from comment #6)
> The attached testcase works for me with current trunk GCC:
> 
> max@max:/tmp/bug$ make
> rm -f m *.so
> ~/install/master/bin/g++ -fuse-ld=gold -g2 -Og -flto -fsanitize=address
> s.cpp -shared -o s.so -fPIC
> ~/install/master/bin/g++ -fuse-ld=gold -g2 -Og -flto -fsanitize=address
> m.cpp -o m
> max@max:/tmp/bug$ LD_LIBRARY_PATH=/home/max/install/master/lib64 ./m 
> initializing library...
> done.
> thread started.
> max@max:/tmp/bug$ echo $?
> 0


strange, i've tested gcc-trunk and it locks in the same way as 6.2.1.


~/src/gcc-install/usr/local/bin/g++ -v
Using built-in specs.
COLLECT_GCC=/home/pawels/src/gcc-install/usr/local/bin/g++
COLLECT_LTO_WRAPPER=/home/pawels/src/gcc-install/usr/local/bin/../libexec/gcc/x86_64-pc-linux-gnu/7.0.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /home/pawels/src/gcc/configure --with-arch=x86-64
--with-linker-hash-style=gnu --disable-multilib --disable-nls --disable-libssp
--disable-libgomp --disable-libquadmath --disable-libitm --disable-libcilkrts
--disable-libvtv --disable-liboffloadmic --disable-libmpx --enable-tls
--enable-libstdcxx-allocator=new --enable-extern-template
--enable-libstdcxx-time=rt --enable-libstdcxx-threads
--disable-libstdcxx-dual-abi --enable-libstdcxx-filesystem-ts=no
--enable-symvers=gnu-versioned-namespace --disable-libstdcxx-pch --enable-lto
--enable-plugin --enable-c99 --enable-long-long --enable-linux-futex
--enable-threads=posix --enable-shared --with-pic --enable-gold
--enable-__cxa_atexit --enable-gnu-unique-object --enable-initfini-array
--enable-languages=c,c++ --enable-checking=release --with-long-double-128
--disable-cld --disable-bootstrap
Thread model: posix
gcc version 7.0.0 20161025 (experimental) (GCC) 

~/src/gcc-install/usr/local/bin/g++ -fuse-ld=gold -g2 -Og -fsanitize=address
-Wl,-rpath,/home/pawels/src/gcc-install/usr/local/lib64 -flto s.cpp -shared -o
s.so -fPIC -pthread
~/src/gcc-install/usr/local/bin/g++ -fuse-ld=gold -g2 -Og -fsanitize=address
-Wl,-rpath,/home/pawels/src/gcc-install/usr/local/lib64 -flto m.cpp -o m -ldl

[pawels@pawels]~/src/bug% ./m
initializing library...
^C

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[m.ostapenko at samsung dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

Maxim Ostapenko  changed:

   What|Removed |Added

 CC||m.ostapenko at samsung dot com

--- Comment #6 from Maxim Ostapenko  ---
The attached testcase works for me with current trunk GCC:

max@max:/tmp/bug$ make
rm -f m *.so
~/install/master/bin/g++ -fuse-ld=gold -g2 -Og -flto -fsanitize=address s.cpp
-shared -o s.so -fPIC
~/install/master/bin/g++ -fuse-ld=gold -g2 -Og -flto -fsanitize=address m.cpp
-o m
max@max:/tmp/bug$ LD_LIBRARY_PATH=/home/max/install/master/lib64 ./m 
initializing library...
done.
thread started.
max@max:/tmp/bug$ echo $?
0

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[kcc at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #5 from Kostya Serebryany  ---
(In reply to Pawel Sikora from comment #3)
> (In reply to Kostya Serebryany from comment #2)
> > Is -flto important here?
> > Does this happen with clang? (does not happen for me)
> > Does this happen if you use statically linked asan (-static-libasan)?
> 
> ad1). lto isn't important.
> ad2). with static libasan testcase works.
> ad3). i can't verify with clang on fedora 24 due to linking error:
> 
> clang++ -fuse-ld=gold -g2 -O -fsanitize=address s.cpp -shared -o s.so -fPIC
> clang++ -fuse-ld=gold -g2 -O -fsanitize=address m.cpp -o m
> /usr/bin/ld.gold: error: cannot open
> /usr/bin/../lib64/clang/3.8.0/lib/linux/libclang_rt.asan-x86_64.a: No such
> file or directory
> /usr/bin/ld.gold: error: cannot open
> /usr/bin/../lib64/clang/3.8.0/lib/linux/libclang_rt.asan_cxx-x86_64.a: No
> such file or directory

Apparently, the clang installation is broken. 
Besides 3.8 is too old and is irrelevant anyway. 
Your test case works for me with clang trunk (clang links asan runtime
statically) 

> m.cpp:5: error: undefined reference to '__asan_init'
> m.cpp:5: error: undefined reference to '__asan_version_mismatch_check_v6'
> m.cpp:5: error: undefined reference to '__asan_register_globals'
> m.cpp:5: error: undefined reference to '__asan_unregister_globals'

>> ./s.so: undefined symbol: __asan_option_detect_stack_use_after_return
This does not happen with clang.

So, this looks like a gcc-specific bug, probably related to dynamic linking of
asan-rt. 

Hopefully the GCC folks can help. Ejecting myself from here. 
If something like this reproduces in clang trunk, please summon me back.

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[pawel_sikora at zoho dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #4 from Pawel Sikora  ---
(In reply to Pawel Sikora from comment #3)
> (In reply to Kostya Serebryany from comment #2)
> > Is -flto important here?
> > Does this happen with clang? (does not happen for me)
> > Does this happen if you use statically linked asan (-static-libasan)?
> 

> ad2). with static libasan testcase works.

correction (there was a bug in my testing). so, static asan doesn't start.

g++ -fuse-ld=gold -g2 -Og -fsanitize=address -static-libasan s.cpp -shared -o
s.so -fPIC -pthread
g++ -fuse-ld=gold -g2 -Og -fsanitize=address -static-libasan m.cpp -o m -ldl
[pawels@pawels]~/src/bug% ./m
./s.so: undefined symbol: __asan_option_detect_stack_use_after_return

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[pawel_sikora at zoho dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #3 from Pawel Sikora  ---
(In reply to Kostya Serebryany from comment #2)
> Is -flto important here?
> Does this happen with clang? (does not happen for me)
> Does this happen if you use statically linked asan (-static-libasan)?

ad1). lto isn't important.
ad2). with static libasan testcase works.
ad3). i can't verify with clang on fedora 24 due to linking error:

clang++ -fuse-ld=gold -g2 -O -fsanitize=address s.cpp -shared -o s.so -fPIC
clang++ -fuse-ld=gold -g2 -O -fsanitize=address m.cpp -o m
/usr/bin/ld.gold: error: cannot open
/usr/bin/../lib64/clang/3.8.0/lib/linux/libclang_rt.asan-x86_64.a: No such file
or directory
/usr/bin/ld.gold: error: cannot open
/usr/bin/../lib64/clang/3.8.0/lib/linux/libclang_rt.asan_cxx-x86_64.a: No such
file or directory
m.cpp:5: error: undefined reference to '__asan_init'
m.cpp:5: error: undefined reference to '__asan_version_mismatch_check_v6'
m.cpp:5: error: undefined reference to '__asan_register_globals'
m.cpp:5: error: undefined reference to '__asan_unregister_globals'

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[kcc at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #2 from Kostya Serebryany  ---
Is -flto important here?
Does this happen with clang? (does not happen for me)
Does this happen if you use statically linked asan (-static-libasan)?

[Bug sanitizer/77982] deadlock in asan thread initialization/interception.

[pawel_sikora at zoho dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77982

--- Comment #1 from Pawel Sikora  ---
Created attachment 39842
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=39842=edit
reduced testcase.

% gdb ./m
GNU gdb (GDB) Fedora 7.11.1-86.fc24


(gdb) r
Starting program: /home/pawels/src/bug/m 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
initializing library...
[New Thread 0x734ff700 (LWP 26786)]
^C
Thread 1 "m" received signal SIGINT, Interrupt.
__sanitizer::internal_sched_yield () at
../../../../libsanitizer/sanitizer_common/sanitizer_linux.cc:304
304 }


(gdb) thread apply all bt

Thread 2 (Thread 0x734ff700 (LWP 26786)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x75baea76 in __GI___pthread_mutex_lock (mutex=0x77ffd908
<_rtld_local+2312>) at ../nptl/pthread_mutex_lock.c:115
#2  0x77deb6da in tls_get_addr_tail (ti=0x77163b30,
dtv=0x6120bed0, the_map=0x77ff7658) at dl-tls.c:765
#3  0x76e68f4b in __interceptor___tls_get_addr (arg=0x77163b30) at
../../../../libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4207
#4  0x76f21ec4 in __lsan::DisabledInThisThread () at
../../../../libsanitizer/lsan/lsan_common.cc:33
#5  0x76e62815 in __asan::Allocator::Allocate (this=0x77166d20
<__asan::instance>, size=, alignment=,
stack=0x734fe430, alloc_type=, 
can_fill=) at
../../../../libsanitizer/asan/asan_allocator.cc:449
#6  0x76efe1d1 in __interceptor_realloc (ptr=0x0, size=32) at
../../../../libsanitizer/asan/asan_malloc_linux.cc:83
#7  0x75bae035 in pthread_getattr_np (thread_id=,
attr=attr@entry=0x734fed90) at pthread_getattr_np.c:161
#8  0x76f179fd in __sanitizer::GetThreadStackTopAndBottom
(at_initialization=at_initialization@entry=false,
stack_top=stack_top@entry=0x734fee18,
stack_bottom=stack_bottom@entry=0x734fee20)
at ../../../../libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cc:112
#9  0x76f17f5e in __sanitizer::GetThreadStackAndTls (main=, stk_addr=stk_addr@entry=0x77f88020,
stk_size=stk_size@entry=0x77f88028, tls_addr=tls_addr@entry=0x77f88030, 
tls_size=tls_size@entry=0x734fee70) at
../../../../libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cc:388
#10 0x76f0b675 in __asan::AsanThread::SetThreadStackAndTls
(this=this@entry=0x77f88000) at
../../../../libsanitizer/asan/asan_thread.cc:197
#11 0x76f0b8a2 in __asan::AsanThread::Init
(this=this@entry=0x77f88000) at
../../../../libsanitizer/asan/asan_thread.cc:153
#12 0x76f0ba68 in __asan::AsanThread::ThreadStart (this=0x77f88000,
os_id=26786, signal_thread_is_registered=0x7fffc428) at
../../../../libsanitizer/asan/asan_thread.cc:166
#13 0x75bac5ca in start_thread (arg=0x734ff700) at
pthread_create.c:333
#14 0x762cff6d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x77fcf780 (LWP 26778)):
#0  __sanitizer::internal_sched_yield () at
../../../../libsanitizer/sanitizer_common/sanitizer_linux.cc:304
#1  0x76e685c5 in __interceptor_pthread_create (thread=0x7fffccb0,
attr=, start_routine=0x77fdbb60 , arg=0x0)
at ../../../../libsanitizer/asan/asan_interceptors.cc:256
#2  0x77fdbc02 in startPolling () at s.cpp:19
#3  0x77de8d8a in call_init (l=, argc=argc@entry=1,
argv=argv@entry=0x7fffdb88, env=env@entry=0x7fffdb98) at dl-init.c:72
#4  0x77de8e9b in call_init (env=0x7fffdb98, argv=0x7fffdb88,
argc=1, l=) at dl-init.c:30
#5  _dl_init (main_map=main_map@entry=0x61a1f280, argc=1,
argv=0x7fffdb88, env=0x7fffdb98) at dl-init.c:120
#6  0x77deda91 in dl_open_worker (a=a@entry=0x7fffcfb0) at
dl-open.c:564
#7  0x77de8c34 in _dl_catch_error
(objname=objname@entry=0x7fffcfa0,
errstring=errstring@entry=0x7fffcfa8,
mallocedp=mallocedp@entry=0x7fffcf9f, 
operate=operate@entry=0x77ded640 ,
args=args@entry=0x7fffcfb0) at dl-error.c:187
#8  0x77decfe9 in _dl_open (file=0x400900 "./s.so", mode=-2147483647,
caller_dlopen=0x76e7e454 <__interceptor_dlopen(char const*, int)+116>,
nsid=-2, argc=, argv=, 
env=0x7fffdb98) at dl-open.c:649
#9  0x75fc9f09 in dlopen_doit (a=a@entry=0x7fffd1e0) at dlopen.c:66
#10 0x77de8c34 in _dl_catch_error (objname=0x773848b0
, errstring=0x773848b8
, mallocedp=0x773848a8
, 
operate=0x75fc9eb0 , args=0x7fffd1e0) at
dl-error.c:187
#11 0x75fca591 in _dlerror_run (operate=operate@entry=0x75fc9eb0
, args=args@entry=0x7fffd1e0) at dlerror.c:163
#12 0x75fc9fa2 in __dlopen (file=, mode=)
at dlopen.c:87
#13 0x76e7e454 in __interceptor_dlopen (filename=0x400900 "./s.so",
flag=1) at