Re: [Mono-dev] Shutdown hangs
Switching to mono 4.0.5 seems a little better, but the problem (or a similar problem) is still there, and I get segfaults and other problems with 4.0. Does anyone have any suggestions for brute-force ways to fix this problem? E.g., can I modify ves_icall_System_Environment_Exit or similar to just call exit()? I haven't looked at the code but from the call stacks it seems like this happens in an attempt to make an orderly shutdown of threads. While that would be nice, it's much more important to me for the application to actually exit. If I have to, I'll make the process kill itself with SIGKILL, but I'd rather not since then I lose the actual exit code. chris -Original Message- From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Chris Swiedler Sent: Monday, June 13, 2016 9:55 AM To: mono-devel-list <mono-devel-list@lists.ximian.com> Subject: [Mono-dev] Shutdown hangs I'm getting semi-reliable shutdown hangs in a mono command-line app using mono 4.2.3 on Centos 7 64-bit. The app does different things at different times, and the shutdowns only seem to happen when it connects to our SQL server database (using the Mono database bindings, not ODBC). We've also seen an occasional segfault, also seemingly only when doing database work, though those may have gone away since upgrading to 4.2.3. Does anyone have any thoughts? Pstack output below. Thanks, chris Thread 5 (Thread 0x7f4c777ff700 (LWP 11658)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x005f786c in thread_func (thread_data=0x0) at sgen-thread-pool.c:118 #2 0x7f4c77ffedf5 in start_thread (arg=0x7f4c777ff700) at pthread_create.c:308 #3 0x7f4c77d2c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7f4c75560700 (LWP 11659)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x953c40 , alertable=alertable@entry=1) at mono-semaphore.c:107 #2 0x005a1736 in finalizer_thread (unused=) at gc.c:1096 #3 0x005841e9 in start_wrapper_internal (data=) at threads.c:725 #4 start_wrapper (data=) at threads.c:772 #5 0x00621026 in inner_start_thread (arg=0x7fffda0d42e0) at mono-threads-posix.c:97 #6 0x7f4c77ffedf5 in start_thread (arg=0x7f4c75560700) at pthread_create.c:308 #7 0x7f4c77d2c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f4c74fd3700 (LWP 11660)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c68000940, alertable=alertable@entry=0) at mono-semaphore.c:107 #2 0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110 #3 mono_thread_info_end_self_suspend () at mono-threads.c:692 #4 0x00583d2c in self_suspend_internal (thread=0x7f4c78a74330) at threads.c:4546 #5 mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74330) at threads.c:4050 #6 0x00583f50 in mono_thread_interruption_checkpoint_request (bypass_abort_protection=bypass_abort_protection@entry=1) at threads.c:4184 #7 0x00584dae in mono_thread_force_interruption_checkpoint () at threads.c:4221 Thread 2 (Thread 0x7f4c746ff700 (LWP 11661)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c6c000940, alertable=alertable@entry=0) at mono-semaphore.c:107 #2 0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110 #3 mono_thread_info_end_self_suspend () at mono-threads.c:692 #4 0x00583d2c in self_suspend_internal (thread=0x7f4c78a74430) at threads.c:4546 #5 mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74430) at threads.c:4050 #6 0x005851f9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=1000) at threads.c:1206 #17 0x7f4c756fa2e7 in System_Threading_Thread_Sleep_int (millisecondsTimeout=1000) at /home/cswiedler/mono-main/Extern/mono/source/external/referencesource/mscorlib/system/threading/thread.cs:715 Thread 1 (Thread 0x7f4c78b34780 (LWP 11657)): #0 0x7f4c7800599d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00632a5a in monoeg_g_usleep (microseconds=1935890) at gdate-unix.c:53 #2 0x0061f2bb in suspend_sync_nolock (id=id@entry=139966347736832, interrupt_kernel=interrupt_kernel@entry=1) at mono-threads.c:870 #3 0x0061f732 in mono_thread_info_safe_suspend_and_run (id=139966347736832, interrupt_kernel=interrupt_kernel@entry=1, callback=callback@entry=0x5813c0 , user_data=user_data@entry=0x7fffda0d3a10) at mono-threads.c:891 #4 0x00581d29 in suspend_thread_internal (thread=0x7f4c78a74430, interrupt=1) at threads.c:4531 #5 0x00
Re: [Mono-dev] Shutdown hangs
" 0x7f5eb186112d in poll () at ../sysdeps/unix/syscall-template.S:81 8Thread 0x7f5e71a7b700 (LWP 3531) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 7Thread 0x7f5e7187a700 (LWP 3532) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 6Thread 0x7f5e706a3700 (LWP 3562) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 5Thread 0x7f5e704a2700 (LWP 3563) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 4Thread 0x7f5e579fc700 (LWP 3594) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 3Thread 0x7f5e577fb700 (LWP 3595) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 2Thread 0x7f5e565f2700 (LWP 3876) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 1Thread 0x7f5eb266f7c0 (LWP 2758) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 -Original Message- From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Chris Swiedler Sent: Monday, June 13, 2016 9:55 AM To: mono-devel-list Subject: [Mono-dev] Shutdown hangs I'm getting semi-reliable shutdown hangs in a mono command-line app using mono 4.2.3 on Centos 7 64-bit. The app does different things at different times, and the shutdowns only seem to happen when it connects to our SQL server database (using the Mono database bindings, not ODBC). We've also seen an occasional segfault, also seemingly only when doing database work, though those may have gone away since upgrading to 4.2.3. Does anyone have any thoughts? Pstack output below. Thanks, chris Thread 5 (Thread 0x7f4c777ff700 (LWP 11658)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x005f786c in thread_func (thread_data=0x0) at sgen-thread-pool.c:118 #2 0x7f4c77ffedf5 in start_thread (arg=0x7f4c777ff700) at pthread_create.c:308 #3 0x7f4c77d2c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7f4c75560700 (LWP 11659)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x953c40 , alertable=alertable@entry=1) at mono-semaphore.c:107 #2 0x005a1736 in finalizer_thread (unused=) at gc.c:1096 #3 0x005841e9 in start_wrapper_internal (data=) at threads.c:725 #4 start_wrapper (data=) at threads.c:772 #5 0x00621026 in inner_start_thread (arg=0x7fffda0d42e0) at mono-threads-posix.c:97 #6 0x7f4c77ffedf5 in start_thread (arg=0x7f4c75560700) at pthread_create.c:308 #7 0x7f4c77d2c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f4c74fd3700 (LWP 11660)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c68000940, alertable=alertable@entry=0) at mono-semaphore.c:107 #2 0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110 #3 mono_thread_info_end_self_suspend () at mono-threads.c:692 #4 0x00583d2c in self_suspend_internal (thread=0x7f4c78a74330) at threads.c:4546 #5 mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74330) at threads.c:4050 #6 0x00583f50 in mono_thread_interruption_checkpoint_request (bypass_abort_protection=bypass_abort_protection@entry=1) at threads.c:4184 #7 0x00584dae in mono_thread_force_interruption_checkpoint () at threads.c:4221 Thread 2 (Thread 0x7f4c746ff700 (LWP 11661)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c6c000940, alertable=alertable@entry=0) at mono-semaphore.c:107 #2 0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110 #3 mono_thread_info_end_self_suspend () at mono-threads.c:692 #4 0x00583d2c in self_suspend_internal (thread=0x7f4c78a74430) at threads.c:4546 #5 mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74430) at threads.c:4050 #6 0x005851f9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=1000) at threads.c:1206 #17 0x7f4c756fa2e7 in System_Threading_Thread_Sleep_int (millisecondsTimeout=1000) at /home/cswiedler/mono-main/Extern/mono/source/external/referencesource/mscorlib/system/threading/thread.cs:715 Thread 1 (Thread 0x7f4c78b34780 (LWP 11657)): #0 0x7f4c7800599d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00632a5a in monoeg_g_usleep (microseconds=1935890) at gdate-unix.c:53 #2 0x0061f2bb in suspend_sync_nolock (id=id@entry=139966347736832, interrupt_kernel=interrupt_kernel@entry=1) at mono-threads.c:870 #3 0x0061f732 in
Re: [Mono-dev] Shutdown hangs
I saw something similar when I was testing 4.2.2 awhile back, a rare hang on Exit with the main thread waiting like yours. I haven't tried 4.2.3 to see if our hang reproduces yet. We're still on 4.0.4 and we don't see it there. In the hang we saw, I also had one thread in pthread_cond_wait like yours, though we had two threads also in poll() with libusb_handle_events_timeout_completed () further up the stack. (gdb) info threads Id Target Id Frame 16 Thread 0x7f5eb0bff700 (LWP 2763) "mono" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 15 Thread 0x7f5eae7e1700 (LWP 2765) "Finalizer" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 14 Thread 0x7f5ead26b700 (LWP 2784) "Timer-Scheduler" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 13 Thread 0x7f5eacf2f700 (LWP 2806) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 12 Thread 0x7f5eacd2a700 (LWP 2807) "Top" 0x7f5eb1b48b9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 11 Thread 0x7f5e8e07b700 (LWP 2840) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 * 10 Thread 0x7f5e72fa5700 (LWP 2980) "Top" 0x7f5eb186112d in poll () at ../sysdeps/unix/syscall-template.S:81 9Thread 0x7f5e727a4700 (LWP 2981) "Top" 0x7f5eb186112d in poll () at ../sysdeps/unix/syscall-template.S:81 8Thread 0x7f5e71a7b700 (LWP 3531) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 7Thread 0x7f5e7187a700 (LWP 3532) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 6Thread 0x7f5e706a3700 (LWP 3562) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 5Thread 0x7f5e704a2700 (LWP 3563) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 4Thread 0x7f5e579fc700 (LWP 3594) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 3Thread 0x7f5e577fb700 (LWP 3595) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 2Thread 0x7f5e565f2700 (LWP 3876) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 1Thread 0x7f5eb266f7c0 (LWP 2758) "Top" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 -Original Message- From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Chris Swiedler Sent: Monday, June 13, 2016 9:55 AM To: mono-devel-list Subject: [Mono-dev] Shutdown hangs I'm getting semi-reliable shutdown hangs in a mono command-line app using mono 4.2.3 on Centos 7 64-bit. The app does different things at different times, and the shutdowns only seem to happen when it connects to our SQL server database (using the Mono database bindings, not ODBC). We've also seen an occasional segfault, also seemingly only when doing database work, though those may have gone away since upgrading to 4.2.3. Does anyone have any thoughts? Pstack output below. Thanks, chris Thread 5 (Thread 0x7f4c777ff700 (LWP 11658)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x005f786c in thread_func (thread_data=0x0) at sgen-thread-pool.c:118 #2 0x7f4c77ffedf5 in start_thread (arg=0x7f4c777ff700) at pthread_create.c:308 #3 0x7f4c77d2c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7f4c75560700 (LWP 11659)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x953c40 , alertable=alertable@entry=1) at mono-semaphore.c:107 #2 0x005a1736 in finalizer_thread (unused=) at gc.c:1096 #3 0x005841e9 in start_wrapper_internal (data=) at threads.c:725 #4 start_wrapper (data=) at threads.c:772 #5 0x00621026 in inner_start_thread (arg=0x7fffda0d42e0) at mono-threads-posix.c:97 #6 0x7f4c77ffedf5 in start_thread (arg=0x7f4c75560700) at pthread_create.c:308 #7 0x7f4c77d2c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f4c74fd3700 (LWP 11660)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c68000940, alertable=alertable@entry=0) at mono-semaphore.c:107 #2 0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110 #3 mono_thread_info_end_self_suspend () at mono-threads.c:692 #4 0x00583d2c in self_suspend_internal (thread=0x7f4c78a74330) at threads.c:4546 #5 mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74330) at threads.c:4050 #6 0x000
[Mono-dev] Shutdown hangs
I'm getting semi-reliable shutdown hangs in a mono command-line app using mono 4.2.3 on Centos 7 64-bit. The app does different things at different times, and the shutdowns only seem to happen when it connects to our SQL server database (using the Mono database bindings, not ODBC). We've also seen an occasional segfault, also seemingly only when doing database work, though those may have gone away since upgrading to 4.2.3. Does anyone have any thoughts? Pstack output below. Thanks, chris Thread 5 (Thread 0x7f4c777ff700 (LWP 11658)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x005f786c in thread_func (thread_data=0x0) at sgen-thread-pool.c:118 #2 0x7f4c77ffedf5 in start_thread (arg=0x7f4c777ff700) at pthread_create.c:308 #3 0x7f4c77d2c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7f4c75560700 (LWP 11659)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x953c40 , alertable=alertable@entry=1) at mono-semaphore.c:107 #2 0x005a1736 in finalizer_thread (unused=) at gc.c:1096 #3 0x005841e9 in start_wrapper_internal (data=) at threads.c:725 #4 start_wrapper (data=) at threads.c:772 #5 0x00621026 in inner_start_thread (arg=0x7fffda0d42e0) at mono-threads-posix.c:97 #6 0x7f4c77ffedf5 in start_thread (arg=0x7f4c75560700) at pthread_create.c:308 #7 0x7f4c77d2c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f4c74fd3700 (LWP 11660)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c68000940, alertable=alertable@entry=0) at mono-semaphore.c:107 #2 0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110 #3 mono_thread_info_end_self_suspend () at mono-threads.c:692 #4 0x00583d2c in self_suspend_internal (thread=0x7f4c78a74330) at threads.c:4546 #5 mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74330) at threads.c:4050 #6 0x00583f50 in mono_thread_interruption_checkpoint_request (bypass_abort_protection=bypass_abort_protection@entry=1) at threads.c:4184 #7 0x00584dae in mono_thread_force_interruption_checkpoint () at threads.c:4221 Thread 2 (Thread 0x7f4c746ff700 (LWP 11661)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x0061aa37 in mono_sem_wait (sem=sem@entry=0x7f4c6c000940, alertable=alertable@entry=0) at mono-semaphore.c:107 #2 0x0061f5ba in mono_thread_info_wait_for_resume (info=) at mono-threads.c:110 #3 mono_thread_info_end_self_suspend () at mono-threads.c:692 #4 0x00583d2c in self_suspend_internal (thread=0x7f4c78a74430) at threads.c:4546 #5 mono_thread_execute_interruption (thread=thread@entry=0x7f4c78a74430) at threads.c:4050 #6 0x005851f9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=1000) at threads.c:1206 #17 0x7f4c756fa2e7 in System_Threading_Thread_Sleep_int (millisecondsTimeout=1000) at /home/cswiedler/mono-main/Extern/mono/source/external/referencesource/mscorlib/system/threading/thread.cs:715 Thread 1 (Thread 0x7f4c78b34780 (LWP 11657)): #0 0x7f4c7800599d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00632a5a in monoeg_g_usleep (microseconds=1935890) at gdate-unix.c:53 #2 0x0061f2bb in suspend_sync_nolock (id=id@entry=139966347736832, interrupt_kernel=interrupt_kernel@entry=1) at mono-threads.c:870 #3 0x0061f732 in mono_thread_info_safe_suspend_and_run (id=139966347736832, interrupt_kernel=interrupt_kernel@entry=1, callback=callback@entry=0x5813c0 , user_data=user_data@entry=0x7fffda0d3a10) at mono-threads.c:891 #4 0x00581d29 in suspend_thread_internal (thread=0x7f4c78a74430, interrupt=1) at threads.c:4531 #5 0x005834ff in mono_thread_suspend_all_other_threads () at threads.c:3154 #6 0x00538176 in ves_icall_System_Environment_Exit (result=0) at icall.c:6006 #7 0x40731832 in ?? () ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
