Re: [Mono-dev] FW: Random hangs while running mono app
So that CUE Card was printed from the stack trace I sent earlier today. I’ll repost it here. I was running 3600 mono processes (each performing the same task). Of the 3600, I believe 8 processes hung. This probably corresponds to the 8 CUE CARD’s you see below. The way the system works, I cannot see STDOUT until I kill each of remaining 8 processes, at which point these CUE CARDS were spit out. The backtrace you see is from only one of the hung processes. However, I did connect to all 8 hung process, performed a backtrace, and they were identical except for pid values. Looks like the cue cards that are relevant to the bt below are for thread id’s : 0x7fffec637700 0x7fffedae7780 Hope that helps. CUE CARDS: STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async
Re: [Mono-dev] FW: Random hangs while running mono app
Hi, The cue card output on itself is not useful. I need the cue card output with the matched thread dump as that allows me to see what's going on on the problematic thread. -- Rodrigo On Fri, Jun 3, 2016 at 12:42 PM, George, Glover E ERDC-RDE-ITL-MS CIV < glover.e.geo...@erdc.dren.mil> wrote: > Hi Rodrigo, > > I didn¹t realize our previous emails were off thread. I was able to get > the STAT CUE CARD output. See below: > > STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any > number) > 0x0 - starting (GOOD, unless the thread is running managed > code) > 0x1 - running (BAD, unless it's the gc thread) > 0x2 - detached (GOOD, unless the thread is running managed > code) > 0x?03 - async suspended (GOOD) > 0x?04 - self suspended (GOOD) > 0x?05 - async suspend requested (BAD) > 0x?06 - self suspend requested (BAD) > 0x*07 - blocking (GOOD) > 0x?08 - blocking with pending suspend (GOOD) > --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 > --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR > WAITING for 1 threads, got 0 suspended > suspend_thread suspend took 200 ms, which is more than the allowed 200 ms > STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any > number) > 0x0 - starting (GOOD, unless the thread is running managed > code) > 0x1 - running (BAD, unless it's the gc thread) > 0x2 - detached (GOOD, unless the thread is running managed > code) > 0x?03 - async suspended (GOOD) > 0x?04 - self suspended (GOOD) > 0x?05 - async suspend requested (BAD) > 0x?06 - self suspend requested (BAD) > 0x*07 - blocking (GOOD) > 0x?08 - blocking with pending suspend (GOOD) > --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 > --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR > WAITING for 1 threads, got 0 suspended > suspend_thread suspend took 200 ms, which is more than the allowed 200 ms > STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any > number) > 0x0 - starting (GOOD, unless the thread is running managed > code) > 0x1 - running (BAD, unless it's the gc thread) > 0x2 - detached (GOOD, unless the thread is running managed > code) > 0x?03 - async suspended (GOOD) > 0x?04 - self suspended (GOOD) > 0x?05 - async suspend requested (BAD) > 0x?06 - self suspend requested (BAD) > 0x*07 - blocking (GOOD) > 0x?08 - blocking with pending suspend (GOOD) > --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 > --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR > WAITING for 1 threads, got 0 suspended > suspend_thread suspend took 200 ms, which is more than the allowed 200 ms > STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any > number) > 0x0 - starting (GOOD, unless the thread is running managed > code) > 0x1 - running (BAD, unless it's the gc thread) > 0x2 - detached (GOOD, unless the thread is running managed > code) > 0x?03 - async suspended (GOOD) > 0x?04 - self suspended (GOOD) > 0x?05 - async suspend requested (BAD) > 0x?06 - self suspend requested (BAD) > 0x*07 - blocking (GOOD) > 0x?08 - blocking with pending suspend (GOOD) > --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 > --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR > WAITING for 1 threads, got 0 suspended > suspend_thread suspend took 200 ms, which is more than the allowed 200 ms > STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any > number) > 0x0 - starting (GOOD, unless the thread is running managed > code) > 0x1 - running (BAD, unless it's the gc thread) > 0x2 - detached (GOOD, unless the thread is running managed > code) > 0x?03 - async suspended (GOOD) > 0x?04 - self suspended (GOOD) > 0x?05 - async suspend requested (BAD) > 0x?06 - self suspend requested (BAD) > 0x*07 - blocking (GOOD) > 0x?08 - blocking with pending suspend (GOOD) > --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 > --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR > WAITING for 1 threads, got 0 suspended > suspend_thread suspend took 200 ms, which is more than the allowed 200 ms > STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any > number) > 0x0 - starting (GOOD, unless the thread is running managed > code) > 0x1 - running (BAD, unless it's the gc thread) > 0x2 - detached (GOOD, unless the thread is running managed > code) > 0x?03 - async suspended (GOOD) > 0x?04 - self suspended (GOOD) > 0x?05 - async suspend requested (BAD)
Re: [Mono-dev] Loader Optimization with mono
Mono doesn't implement LoaderOptimization. On Thu, Jun 2, 2016 at 5:59 AM, techi eth wrote: > Does [LoaderOptimization(LoaderOptimization.MultiDomainHost)] work as > desired on mono? > > I am using Mono 4.2 version on ubuntu to test. > > > > I had following observation > > > > 1. Exe with this attribute and without this attribute take same memory. > Even I had check shared memory also is same and residual - shared is also > same . > > 2. Is there any tool on linux like Process explorer in windows to verify > that all GAC assembly go to shares memory after this attribute is applied. > > > > ___ > Mono-devel-list mailing list > Mono-devel-list@lists.ximian.com > http://lists.ximian.com/mailman/listinfo/mono-devel-list > > ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] FW: Random hangs while running mono app
Hi Rodrigo, I didn¹t realize our previous emails were off thread. I was able to get the STAT CUE CARD output. See below: STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the thread is running managed code) 0x1 - running (BAD, unless it's the gc thread) 0x2 - detached (GOOD, unless the thread is running managed code) 0x?03 - async suspended (GOOD) 0x?04 - self suspended (GOOD) 0x?05 - async suspend requested (BAD) 0x?06 - self suspend requested (BAD) 0x*07 - blocking (GOOD) 0x?08 - blocking with pending suspend (GOOD) --thread 0xa38b30 id 0x7fffec637700 [(nil)] state 105 --thread 0x9d8640 id 0x7fffedae7780 [(nil)] state 1 GC INITIATOR WAITING for 1 threads, got 0 suspended suspend_thread suspend took 200 ms, which is more than the allowed 200 ms STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number) 0x0 - starting (GOOD, unless the th
Re: [Mono-dev] FW: Random hangs while running mono app
I¹ve been trying to reproduce the case where my jobs get a core dump when the finalizer doesn¹t return in time, but I¹ve only been able to reproduce the previously posted stack trace which causes the job to hang in ³Sl² state (using ps). Posting the stack trace again because I somehow didn¹t pasted on of the threads entire stacktrace: Rodrigo, can you please follow up on the email I posted about not seeing anything in STDOUT/STDERR? My code doesn¹t seem to make it to where you are referring to. Burkhard, what file system are you guys using on your cluster? NFS, Gluster, Lustre? (gdb) thread apply all bt Thread 3 (Thread 0x7fffebfff700 (LWP 2269)): #0 0x7fffeccca66c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0060c873 in mono_os_cond_wait (mutex=0x97e640 , cond=0x97e600 ) at ../../mono/utils/mono-os-mutex.h:105 #2 thread_func (thread_data=0x0) at sgen-thread-pool.c:118 #3 0x7fffeccc6806 in start_thread () from /lib64/libpthread.so.0 #4 0x7fffec80a9bd in clone () from /lib64/libc.so.6 #5 0x in ?? () Thread 2 (Thread 0x7fffec637700 (LWP 2272)): #0 0x7fffec75ec8b in sigsuspend () from /lib64/libc.so.6 #1 0x0063cda6 in suspend_signal_handler (_dummy=, info=, context=0x7fffec633f80) at mono-threads-posix-signals.c:209 #2 #3 0x7fffed8faf97 in open64 () from /lib64/ld-linux-x86-64.so.2 #4 0x7fffed8ea82d in open_verify () from /lib64/ld-linux-x86-64.so.2 #5 0x7fffed8ecca1 in _dl_map_object () from /lib64/ld-linux-x86-64.so.2 #6 0x7fffed8f7400 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2 #7 0x7fffed8f2e86 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2 #8 0x7fffed8f6e3b in _dl_open () from /lib64/ld-linux-x86-64.so.2 #9 0x7fffecedcf9b in dlopen_doit () from /lib64/libdl.so.2 #10 0x7fffed8f2e86 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2 #11 0x7fffecedd33c in _dlerror_run () from /lib64/libdl.so.2 #12 0x7fffecedcf01 in dlopen@@GLIBC_2.2.5 () from /lib64/libdl.so.2 #13 0x00631345 in mono_dl_open_file (file=, flags=) at mono-dl-posix.c:67 #14 0x00630b79 in mono_dl_open (name=name@entry=0x19839c0 "/p/home/apps/unsupported/NAVAIR/build/mono-4.3.2/lib/libSystem.Data.dll.so ", flags=flags@entry=1, error_msg=error_msg@entry=0x7fffec634e80) at mono-dl.c:150 #15 0x0054b9f0 in cached_module_load (name=name@entry=0x19839c0 "/p/home/apps/unsupported/NAVAIR/build/mono-4.3.2/lib/libSystem.Data.dll.so ", err=err@entry=0x7fffec634e80, flags=1) at loader.c:1398 Python Exception zero length field name in format: #16 0x0054cc78 in mono_lookup_pinvoke_call (method=method@entry=, exc_class=exc_class@entry=0x7fffec635f00, exc_arg=exc_arg@entry=0x7fffec635f08) at loader.c:1641 Python Exception zero length field name in format: #17 0x00562ce6 in mono_marshal_get_native_wrapper (method=method@entry=, check_exceptions=check_exceptions@entry=1, aot=0) at marshal.c:7396 Python Exception zero length field name in format: #18 0x00452912 in mono_method_to_ir (cfg=cfg@entry=0x1984120, method=method@entry=, start_bblock=, start_bblock@entry=0x0, end_bblock=, end_bblock@entry=0x0, return_var=return_var@entry=0x0, inline_args=inline_args@entry=0x0, inline_offset=0, is_virtual_call=0) at method-to-ir.c:9280 Python Exception zero length field name in format: #19 0x005097d9 in mini_method_compile (method=method@entry=, opts=opts@entry=37023, domain=domain@entry=0x9d9e00, flags=flags@entry=JIT_FLAG_RUN_CCTORS, parts=parts@entry=0, aot_method_index=aot_method_index@entry=-1) at mini.c:3608 Python Exception zero length field name in format: #20 0x0050afb5 in mono_jit_compile_method_inner (method=method@entry=, target_domain=target_domain@entry=0x9d9e00, opt=opt@entry=37023, jit_ex=jit_ex@entry=0x7fffec636678) at mini.c:4263 Python Exception zero length field name in format: #21 0x00428458 in mono_jit_compile_method_with_opt (method=method@entry=, opt=37023, ex=ex@entry=0x7fffec636678) at mini-runtime.c:1952 Python Exception zero length field name in format: #22 0x00428c1b in mono_jit_compile_method (method=) at mini-runtime.c:2008 Python Exception zero length field name in format: #23 0x004ad743 in common_call_trampoline_inner (regs=regs@entry=0x7fffec636890, code=code@entry=0x40244e34 "\270\001", m=m@entry=, vt=vt@entry=0x0, vtable_slot=, vtable_slot@entry=0x0) at mini-trampolines.c:694 Python Exception zero length field name in format: #24 0x004adea0 in common_call_trampoline (regs=0x7fffec636890, code=0x40244e34 "\270\001", m=, vt=0x0, vtable_slot=0x0) at mini-trampolines.c:808 #25 0x4289 in ?? () #26 0x00a35cc5 in ?? () #27 0x40244e34 in ?? () #28 0x00a35cc5 in ?? () #29 0x7fffec6369d0 in ?? () #30 0x7fffec636890 in ?? () #31 0x7fffec637698 in ?? () #32 0x7fffec67a188 in ?? () #33 0x7fffec67a1a0 in ?? () #34 0x7fffec63
Re: [Mono-dev] High threadpool CPU usage
Well, looking at some pstack+top output, I’m not sure it’s GC anymore. I ran pstack and top as close to simultaneously as I could, and the high-cpu threads were just in pthread_cond_timedwait. Is it possible we get into a scenario where mono_cond_timedwait_ms is sleeping with a timeout_ms that's extremely short? Is there an easy way to force the compiler (#pragma or something?) to not optimize out the timeout_ms value? I'd rather not build all of mono with optimizations off, but I could disable optimizations for a few files. chris PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 17369 hydrogen 20 0 6491744 157928 17216 S 53.3 0.1 0:05.08 Threadpool work 17385 hydrogen 20 0 6491744 157928 17216 S 46.7 0.1 0:09.38 Threadpool work Thread 8 (Thread 0x7f52c65fc700 (LWP 17369)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00618817 in mono_cond_timedwait_ms (cond=cond@entry=0x7f52c65fbdc0, mutex=0x2887808, timeout_ms=) at mono-mutex.c:181 #2 0x00586f28 in worker_park () at threadpool-ms.c:509 Thread 4 (Thread 0x7f528a7fe700 (LWP 17385)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00618817 in mono_cond_timedwait_ms (cond=cond@entry=0x7f528a7fddc0, mutex=0x2887808, timeout_ms=) at mono-mutex.c:181 #2 0x00586f28 in worker_park () at threadpool-ms.c:509 From: Rodrigo Kumpera [mailto:kump...@gmail.com] Sent: Friday, May 27, 2016 5:07 PM To: Chris Swiedler Cc: Ludovic Henry ; mono-devel-list Subject: Re: [Mono-dev] High threadpool CPU usage Chris, Can you at least enable GC logging? That will tell you how frequently and for how long the collector is running. On Fri, May 27, 2016 at 11:04 AM, Chris Swiedler wrote: Thanks. I’ll try to use the profiler, but this problem doesn’t happen at startup, and running the application under profiling for eight hours is probably infeasible. I see that at one point mono support signal-based enabling/disabling of the profiler, like how SIGUSR2 toggles the trace mechanism on and off. Is that sort of functionality ever going to be re-added? Currently there’s the heapshot option via the TCP listener, perhaps there could be enable/disable commands via that mechanism? chris From: Ludovic Henry [mailto:ludo...@xamarin.com] Sent: Friday, May 27, 2016 1:27 AM To: Chris Swiedler ; mono-devel-list Subject: Re: [Mono-dev] High threadpool CPU usage Hi Chris, The signal handler you are seeing is the GC signalling every thread to suspend them. So yes you are right, that's the sgen collector stopping the world. To track down the large number of allocation, I would encourage you to try the log profiler (documentation: http://www.mono-project.com/docs/debug+profile/profile/profiler/ ). That should give you some more insights in where the allocations are coming from. Thank you, Ludovic On Thu, May 26, 2016 at 8:20 PM Chris Swiedler wrote: Thanks, Ludovic. I’m using mono-4.2.3. The ‘massive amounts of GC’ idea makes sense based on what I’m seeing. When I run pstack, I get a number of threadpool threads in stacks with: #0 0x7fdff1c6a952 in do_sigsuspend (set=0x954220 ) at ../sysdeps/unix/sysv/linux/sigsuspend.c:32 #1 __GI___sigsuspend (set=set@entry=0x954220 ) at ../sysdeps/unix/sysv/linux/sigsuspend.c:46 #2 0x005c7534 in suspend_thread (info=0x7fdf480008c0, context=context@entry=0x7fde997ea1c0) at sgen-os-posix.c:126 #3 0x005c771f in suspend_handler (_dummy=, _info=, context=0x7fde997ea1c0) at sgen-os-posix.c:153 #4 I thought this was related to GDB / pstack attaching, but it’s actually the suspend handling for the sgen collector, right? Is a thread suspending itself CPU-intensive? I see lots of threads with high CPU at any point, which seems to indicate that this wouldn’t just be the CPU usage of the collector thread itself. Do you have any suggestions for how to track down the large numbers of allocations? This isn’t very easy to reproduce, but now that I know what to look for, I might be able to get it to happen in a test environment. chris From: Ludovic Henry [mailto:ludo...@xamarin.com] Sent: Thursday, May 26, 2016 8:43 AM To: Chris Swiedler ; mono-devel-list Subject: Re: [Mono-dev] High threadpool CPU usage Hi Chris, The first stacktrace you are observing is for threadpool thread parking. We use this technique for threads that are currently not doing anything, to keep them around for a little while (5-60 seconds) so if we have burst of usage, we do not end up destroying and creating threads all the time. The second stacktrace you are observing is, as you noted, when performing a garbage collection, when the GC thread is suspending all the running threads. So if you are witnessing this trace very often, it means a thread is allocating memory very quickl
