Re: [Mono-list] Garbage collection and memory usage
Hi Miguel,
Thanks for the answer.
> We looked at the problem, and it boils down to the fact that Mono does
> not have a compacting GC.
After making some tests and googling a lot, I came to the same
conclusion. But this only explains why the memory usage is so big, and
keeps growing, it doesn't explain why mono "hangs" and .aspx pages are
not processed when there is still free memory on the machine.
> This test is generating some very large datasets that then get their
> viewstate serialized, it is something that Microsoft explicitly states
> is a bad idea.
Ofcourse it is, but it helped to quickly reproduce the problem ;-)
About the memory fragmentation, we managed to narrow down a little the
memory usage by implemeting Dispose() here and there, but I still
can't quite understand how and why this works, so here is a testcase:
http://hosting.telecoms.bg/~moskov/memtest.cs
In the first case we do the following:
while(true)
{
Console.WriteLine("new iteration");
string s = new String('a', 2000);
s = null;
//testString s = new testString();
//s.Dispose();
}
When I run the program with the desc-heap profiler, I see the following:
new iteration
Checkpoint at 2379 for gc
System.String : 40002024
new iteration
Checkpoint at 2433 for gc
System.String : 40002024
new iteration
Checkpoint at 2491 for gc
System.String : 80002038
heap resized to 120594432 @ 2491
Checkpoint at 2491 for heap-resize
System.String : 80002038
new iteration
heap resized to 160727040 @ 2601
Checkpoint at 2601 for heap-resize
System.String : 120002052
new iteration
Checkpoint at 2697 for gc
System.String : 40002024
new iteration
new iteration
new iteration
Checkpoint at 2884 for gc
System.String : 40002024
And from now on, the GC is called on every 4th iteration, and
allocated memory never drops below 160Mb and often reaches 240Mb and
sometimes more. I surpose that calling GC more rarely is some kind of
optimization, but it doesn't help a lot here, because we don't have
problems with the speed of execution, but with the memory usage.
In the other scenario I use testString which implements IDisposable
and basicaly does the same thing. But in this case it takes some time
untill GC starts to be called on every 2nd itaration, and even more
time to start beeing called on every 4th, and allocated memory never
exceeds 160Mb.
I tested in the same way one of our most frequently called and
obviously memory fragmenting classes. In the first case, when run in a
while(true) loop for 5 minutes, it allocated 39Mb of memory, and in
the second case when we implemented Dispose() it allocated 12Mb of
memory.
I hope that I haven't gotten something totaly wrong and confused the
situation more ;-) Any feedback is highly apreciated.
Regards,
Georgi Moskov
___
Mono-list maillist - Mono-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-list
Re: [Mono-list] Garbage collection and memory usage
Hi Miguel, > Ok, alternatively get the stack traces for each thread. > > Do this by enumerating all the threads: > > info threads > > And selecting one by one: > > thread 1 > > Then running the command. > > This is a better version of the "t a a bt" command. > Here is the output: http://hosting.telecoms.bg/~moskov/gdb-mono_backtrace_15.txt.gz We made some tests and here is an test app that seems to reproduce our problem: http://hosting.telecoms.bg/~moskov/WebTest.tar.gz It has 10 user control which are loaded dynamicly after clicking one of 10 buttons. Every user control contains a data grid with 1 rows. On my test machine which has 256Mb RAM the problem shows after loading the third user control, and top looks like this: Mem:255756k total, 248120k used, 7636k free, 3544k buffers Swap:72252k total, 5616k used,66636k free,24212k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 3388 www-data 15 0 208m 191m 7792 S 0.0 76.8 0:34.05 mono Regards Georgi Moskov ___ Mono-list maillist - Mono-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-list
Re: [Mono-list] Garbage collection and memory usage
Hi Miguel, > Hello, > > >Here is the output from the gdb macro: > > There are two macros, the other macro will the traces for all threads. > Your email only contains a more readable version of the same thread > (which was not useful, we know about that one). Indeed there are two macros, but for the second one I read "Starting with 1.1.13.4, you can add the following gdb macro to your .gdbinit file.", and since I think I mentioned that I am running mono 1.1.13.2 I never tried it. > >Since I am pretty sure that the problem has something to do with > > JIT compilation and memory, and the gdb trace from the macro doesn't > > look much different from the previous one, I surpose we could put up > > something similar to our application to reproduce the problem. > > This would be very useful. I'll write you back when I reproduce the problem with simpler sources and(or) when I gain better knowledge of gdb ;-) Regards, Georgi Moskov ___ Mono-list maillist - Mono-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-list
Re: [Mono-list] Re: Garbage collection and memory usage
Hello, I will skip the comppression, since the output is not so big ... 9 Thread -1219900496 (LWP 2849) 0xb7c966c6 in semop () from /lib/tls/libc.so.6 8 Thread -1219982416 (LWP 2879) 0xb7d22440 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0 7 Thread -1223263312 (LWP 2881) 0xb7c966c6 in semop () from /lib/tls/libc.so.6 6 Thread -1229239376 (LWP 2912) 0xb7d22440 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0 5 Thread -1230496848 (LWP 2913) 0xb7d24dfc in __nanosleep_nocancel () from /lib/tls/libpthread.so.0 4 Thread -1234740304 (LWP 2921) 0xb7c966c6 in semop () from /lib/tls/libc.so.6 3 Thread -1224311888 (LWP 3172) 0xb7c966c6 in semop () from /lib/tls/libc.so.6 2 Thread -1233683536 (LWP 3231) 0xb7c966c6 in semop () from /lib/tls/libc.so.6 1 Thread -1212431040 (LWP 2809) 0xb7c966c6 in semop () from /lib/tls/libc.so.6 Thread 9 (Thread -1219900496 (LWP 2849)): Program received signal SIGSEGV, Segmentation fault. 0xb7ec6e39 in mono_jit_info_table_find () from /usr/lib/libmono.so.0 The program being debugged was signaled while in a function called from GDB. GDB remains in the frame where the signal was received. To change this behavior use "set unwindonsignal on" Evaluation of the expression containing the function (mono_pmip) will be abandoned. #0 0xb7c966c6 in semop () from /lib/tls/libc.so.6 This happens at "t a a mono_backtrace 100", with or without "set unwindonsignal on". Regards, Georgi Moskov ___ Mono-list maillist - Mono-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-list
Re: [Mono-list] Garbage collection and memory usage
Hello again, Here is the output from the gdb macro: (gdb) mono_backtrace 15 [Switching to Thread -1212431040 (LWP 2809)] #0 0xb7c966c6 in semop () from /lib/tls/libc.so.6 #1 0xb7ef5bb7 in _wapi_shm_sem_lock () from /usr/lib/libmono.so.0 #2 0xb7eeb567 in _wapi_handle_count_signalled_handles () from /usr/lib/libmono.so.0 #3 0xb7ef9ca6 in SignalObjectAndWait () from /usr/lib/libmono.so.0 #4 0xb7ef9ed2 in WaitForMultipleObjectsEx () from /usr/lib/libmono.so.0 #5 0xb7eacb3b in mono_install_thread_callbacks () from /usr/lib/libmono.so.0 #6 0xb7eace8a in mono_thread_manage () from /usr/lib/libmono.so.0 #7 0xb7e320da in mono_main () from /usr/lib/libmono.so.0 #8 0x0804857b in ?? () #9 0x0005 in ?? () #10 0xbfeab0d4 in ?? () #11 0xbfeab0a8 in ?? () #12 0xb7bd2974 in __libc_start_main () from /lib/tls/libc.so.6 #13 0xb7bd2974 in __libc_start_main () from /lib/tls/libc.so.6 warning: Unable to restore previously selected frame. #0 0xb7c966c6 in semop () from /lib/tls/libc.so.6 (gdb) This time I limited the phisical memory of the machine from 192Mb to 96Mb, so the problem can show faster. top: top - 20:47:45 up 5 min, 1 user, load average: 0.29, 0.46, 0.23 Tasks: 65 total, 1 running, 64 sleeping, 0 stopped, 0 zombie Cpu(s): 1.0% us, 2.0% sy, 0.0% ni, 96.7% id, 0.0% wa, 0.3% hi, 0.0% si Mem: 93508k total,91008k used, 2500k free, 1076k buffers Swap:72252k total,50408k used,21844k free,12932k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2809 www-data 15 0 86424 56m 6960 S 1.7 62.3 0:37.23 mono Since I am pretty sure that the problem has something to do with JIT compilation and memory, and the gdb trace from the macro doesn't look much different from the previous one, I surpose we could put up something similar to our application to reproduce the problem. Regards, Georgi Moskov On 4/7/06, Miguel de Icaza <[EMAIL PROTECTED]> wrote: > Hello, > > Good news and bad news. The good news is that it seems that the > bug is easy to trigger, which means we can do something about it. > > The bad news is that the gdb stack trace you provided is only for > one thread, which is not the problematic one. > > Please try the two gdb macros that we have in our site to get all > the information: > > http://www.mono-project.com/Debugging > > Alternatively, if you could share your application with us, and the > steps to reproduce it, we could do the work ourselves. > > If your software is proprietary, we can sign an NDA to have someone > at Novell look at it. > > Miguel > ___ > Mono-list maillist - Mono-list@lists.ximian.com > http://lists.ximian.com/mailman/listinfo/mono-list > ___ Mono-list maillist - Mono-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-list
Re: [Mono-list] Garbage collection and memory usage
Hi Miguel, I am experiencing the same problem with my web application and the root ot the problem seems to be in the mono process that runs mod-mono-server. The curious thing is that it seems to hang every time when it consumes around 50% from the system memory. Here are some traces, I hope they help a little: top: Tasks: 66 total, 2 running, 64 sleeping, 0 stopped, 0 zombie Cpu(s): 4.3% us, 3.7% sy, 0.0% ni, 91.7% id, 0.0% wa, 0.0% hi, 0.3% si Mem:190940k total, 185772k used, 5168k free, 3080k buffers Swap:72252k total,35984k used,36268k free,38264k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 16019 www-data 15 0 124m 93m 8876 S 6.6 50.3 8:17.75 mono gdb: 0xb7c706c6 in semop () from /lib/tls/libc.so.6 (gdb) bt #0 0xb7c706c6 in semop () from /lib/tls/libc.so.6 #1 0xb7ecfbb7 in _wapi_shm_sem_lock () from /usr/lib/libmono.so.0 #2 0xb7ec5567 in _wapi_handle_count_signalled_handles () from /usr/lib/libmono.so.0 #3 0xb7ed3ca6 in SignalObjectAndWait () from /usr/lib/libmono.so.0 #4 0xb7ed3ed2 in WaitForMultipleObjectsEx () from /usr/lib/libmono.so.0 #5 0xb7e86b3b in mono_install_thread_callbacks () from /usr/lib/libmono.so.0 #6 0xb7e86e8a in mono_thread_manage () from /usr/lib/libmono.so.0 #7 0xb7e0c0da in mono_main () from /usr/lib/libmono.so.0 #8 0x0804857b in ?? () #9 0x0005 in ?? () #10 0xbfc84ec4 in ?? () #11 0xbfc84e98 in ?? () #12 0xb7bac974 in __libc_start_main () from /lib/tls/libc.so.6 #13 0xb7bac974 in __libc_start_main () from /lib/tls/libc.so.6 strace: # strace -p 16019 Process 16019 attached - interrupt to quit semop(3538953, 0xbfc84b90, 1 Mono version is 1.1.13.2 mod_mono version is 1.1.13 Regards, Georgi Moskov On 4/7/06, Miguel de Icaza <[EMAIL PROTECTED]> wrote: > Hello, > > > Assuming this is the case, what on earth can I do to troubleshoot this? > > Is there something we can do in code to help the GC? I thought the > > whole point of languages like C# was to get /away/ from awful days of > > malloc + free'ing memory like in C :( > > Thanks for posting the message, we need a few more data points here. > > You mention that the application will go down within twenty minutes. By > the time it goes down, what is growing out of proportion? Is it the > Apache process with mod_mono or is it the Mono process that runs > mod-mono-server.exe? > > You mentioned that requests were being received, but they were not > getting a reply. What was the size of Mono then? > > Am curious in this particular case if this is a case of Mono leaking > memory or if this is a case of Mono running out of threads to process > responses. > > Your initial description sounds like Mono is hanging, not that Mono is > leaking. > > This could be caused by a number of reasons: the requests might not be > completing; You might be using some library that has a threadpool > leak; You might be starving and deadlocking the threadpool internally. > > Miguel. > ___ > Mono-list maillist - Mono-list@lists.ximian.com > http://lists.ximian.com/mailman/listinfo/mono-list > ___ Mono-list maillist - Mono-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-list
Re: [Mono-list] [ANN] X-develop 1.2 with better Mono debugging support
Hi, This is what I get when debugging: Cannot read DWARF frame info from symbol file `/opt/x-develop-1.1/lib/debugmono/mono-debugger-mini-wrapper': System.OverflowException: This isn't a 64bits machine. in [0x0003c] System.IntPtr:.ctor (Int64 i64) in [0x00040] (at /tmp/scratch/BUILD/mono-debugger-0.11/arch/Bfd.cs:610) Mono.Debugger.Backends.Bfd:create_frame_reader () in [0xd] (at /tmp/scratch/BUILD/mono-debugger-0.11/arch/Bfd.cs:632) Mono.Debugger.Backends.Bfd:load_dwarf () Regards, Georgi Moskov On 12/20/05, Hans Kratz <[EMAIL PROTECTED]> wrote: > Hi! > > We have just release X-develop 1.2 build 1006, which supports the latest > Mono debugger (version 0.11) and Mono 1.1.12_0 as installed using the > "Linux Installer for x86" out of the box. > > The Mono debugger library seems to be much more stable with the new > release. There are still a few issues both in X-develop's Mono debugger > interface as well as the Mono.Debugger library but I am optimistic that > those can be resolved. > > More information and a free trial version of X-develop are available > from http://www.omnicore.com. > > If you give it a try please let me know if the debugger integration > works for you. > > > Best regards, > > > Hans > -- > Hans Kratz > Omnicore Software > http://www.omnicore.com > ___ > Mono-list maillist - Mono-list@lists.ximian.com > http://lists.ximian.com/mailman/listinfo/mono-list > ___ Mono-list maillist - Mono-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-list
[Mono-list] Dropping privileges in linux
Hi all, I am looking for a way to drop privileges of an assembly started as 'root' to a normal user. I found two possible solutions, but didn't succeed with either of them ... a) Using Syscall Syscall.setgid(1000); Syscall.setuid(1000); b) Using WindowsIdentity the way it is described here: http://pages.infinit.net/ctech/20040405-1133.html In either way I get a 'Segmentation fault' when I execute the assembly. Can anyone give me some hint what I'm doing wrong, or a way to surroud the problem? Thanks, Georgi Moskov ___ Mono-list maillist - Mono-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-list
