Re: [Caml-list] Re: memory usage
Yeah, I flushed the output and I used /dev/zero - I still see no messages. That's strange. Anyway, it looks like a memory leak, isn't it? 20 threads * 1KB max = 20 KB of memory for data storage should be used (plus 20KB*500% for heap overhead = 100KB). I changed delay to 1 ms: ignore (Unix.select [] [] [] 0.001); After 2 minutes of work on FreeBSD, 19MB(!) was used: # ps auxw | grep a.out | grep -v grep root 35136 30.1 0.2 60348 19304 p5 L+1:36PM 2:08.07 ./a.out # pmap PID: Address Kbytes RSS Shared Priv Mode Mapped File 0040204 360 - 360 r-x /tmp/a.out 00532000 80 76 - 80 rw- /tmp/a.out 00546000 28 16 - 28 rw- [ anon ] 40532000148 108148- r-x /libexec/ld-elf.so.1 40557000132 108 - 132 rw- [ anon ] 40657000 28 28 - 28 rw- /libexec/ld-elf.so.1 4065E000 24 20 - 24 rw- [ anon ] 40664000 92 76 92- r-x /lib/libm.so.5 4067B000 44 -4 r-x /lib/libm.so.5 4067C000 10240 1024- r-x /lib/libm.so.5 4077C000 88 -8 rw- /lib/libm.so.5 4077E000 64 64 64- r-x /lib/libthr.so.3 4078E000 44 -4 r-x /lib/libthr.so.3 4078F000 1024 12 1024- r-x /lib/libthr.so.3 4088F000 12 12 - 12 rw- /lib/libthr.so.3 40892000 88 -8 rw- [ anon ] 40894000932 452932- r-x /lib/libc.so.7 4097D000 44 -4 r-x /lib/libc.so.7 4097E000 10200 1020- r-x /lib/libc.so.7 40A7D000116 116 - 116 rw- /lib/libc.so.7 40A9A000 92 36 - 92 rw- [ anon ] 40B0 44032 13976- 44032 rw- [ anon ] 785A4000128 12 - 128 rw- [ anon ] 787A5000128 12 - 128 rw- [ anon ] 789A6000128 12 - 128 rw- [ anon ] 78BA7000128 12 - 128 rw- [ anon ] 78DA8000128 12 - 128 rw- [ anon ] 78FA9000128 12 - 128 rw- [ anon ] ... (gdb) info thr 18 Thread 0x40b01120 (LWP 101068) 0x408e4d4c in _umtx_op () from /lib/libc.so.7 17 Thread 0x40b01290 (LWP 100489) 0x40971cfc in select () from /lib/libc.so.7 16 Thread 0x40b01b30 (LWP 100079) 0x408e4d4c in _umtx_op () from /lib/libc.so.7 15 Thread 0x40b04d80 (LWP 100858) 0x40971cfc in select () from /lib/libc.so.7 14 Thread 0x40b02260 (LWP 101096) 0x408e4d4c in _umtx_op () from /lib/libc.so.7 13 Thread 0x40b04200 (LWP 101128) 0x40971cfc in select () from /lib/libc.so.7 12 Thread 0x40b03960 (LWP 101175) 0x408e4d4c in _umtx_op () from /lib/libc.so.7 11 Thread 0x40b02820 (LWP 101189) 0x408e4d4c in _umtx_op () from /lib/libc.so.7 10 Thread 0x40b03680 (LWP 101217) 0x408e4d4c in _umtx_op () from /lib/libc.so.7 9 Thread 0x40b03f20 (LWP 100057) 0x408e4d4c in _umtx_op () from /lib/libc.so.7 8 Thread 0x40b020f0 (LWP 100066) 0x40971cfc in select () from /lib/libc.so.7 7 Thread 0x40b05ec0 (LWP 100535) 0x40971cfc in select () from /lib/libc.so.7 6 Thread 0x40b04370 (LWP 100564) 0x408e4d4c in _umtx_op () from /lib/libc.so.7 5 Thread 0x40b04c10 (LWP 100698) 0x40971cfc in select () from /lib/libc.so.7 4 Thread 0x40b02b00 (LWP 101224) 0x40971cfc in select () from /lib/libc.so.7 3 Thread 0x40b03ad0 (LWP 101227) 0x40971cfc in select () from /lib/libc.so.7 2 Thread 0x40b037f0 (LWP 101258) 0x408e4d4c in _umtx_op () from /lib/libc.so.7 * 1 Thread 0x40b05620 (LWP 100101) 0x409643dc in open () from /lib/libc.so.7 (gdb) I also added special thread to check GC: let print_stats _ = while true do Gc.compact (); Gc.print_stat stdout; flush stdout; Unix.sleep 10; done Thread.create print_stats (); GC statistics after 1 minute: minor_words: 21929847-- ? promoted_words: 2474840 major_words: 2939105 minor_collections: 1655 major_collections: 635 heap_words: 61440 heap_chunks: 1 top_heap_words: 61440 live_words: 1024 live_blocks: 115 free_words: 60416 free_blocks: 1 largest_free: 60416 fragments: 0 compactions: 7 ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
[Caml-list] Re: memory usage
On 12-01-2009, John Lepikhin j...@ispsystem.com wrote: Starting point should be to call this periodically: Gc.compact (); let stat = Gc.stat () in let live_words = stat.Gc.live_words in eprintf live words %d\n%! live_words; which will tell you how many words (ie 4 or 8 byte chunks) are reachable according to the garbage collector. Richard, here is result (statistics was saved every 10 seconds): live words - RSS: 186980 - 12380KB -- after first 10 seconds of work 154156 - 18232KB 153923 - 19648KB ... after 10 minutes of work: 203842 - 33436KB 170559 - 33528KB 187018 - 33664KB 71626 - 33592KB Sometimes live words drops down to 40.000. But RSS always stay near 30-50MB. To get real memory used, (Sys.word_size * live_word / 8). Do you use out-of-heap datastructure that can use memory ? (malloc-ed datastructure). Regards, Sylvain Le Gall ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Re: memory usage
The only specific module is ocaml-fd (send file descriptors over pipes). What version of OCaml is this? 3.10.2. I'm afraid to say that you'll have to post a short reproducer here before I can look at this further ... Yes, it could be the simplest solution. I have this issue only under the high load, on production server, with real client applications. At this time, I have no idea how to reduce code from big commercial application to something short. May be I'll create some stand-alone GC test later. P.S. Interesting thing I found on other machine, where application works with another load: after the same number of connections processed, only 5MB was allocated (feel the difference: 5MB and 40MB, both after processing 250.000 connections). The only difference is the first machine had processed 250.000 connections in 5 hours (up to ~20 concurrent threads), the second in 24 hours (up to ~10 concurrent threads). ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
[Caml-list] Re: memory usage
On 12-01-2009, John Lepikhin j...@ispsystem.com wrote: Each thread is killed after work is done How do you kill the threads? I hope this is just a figure of speech for I do an orderly shutdown of each thread after work is done. Well, that was consequence of my bad English :-) Threads finish their work and exit. I also made a simple wrapper to Thread.create to be sure that all work inside threads is done: module MyThread = let create f p = let dowork _ = (* log thread creation *) f p; (* log thread shutdown *) in Thread.create dowork () end Do you use some kind of Thread.join ? Regards Sylvain Le Gall ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Re: memory usage
I'm afraid to say that you'll have to post a short reproducer here before I can look at this further ... I'm not sure this is the same issue. Look at this test code: let threads_counter = ref 0 let counter_mutex = Mutex.create () let send_data data size = let fd = Unix.openfile /dev/null [ Unix.O_WRONLY ] 0o644 in ignore (Unix.write fd data 0 size); Unix.close fd let read_data _ = let fd = Unix.openfile /dev/random [] 0o644 in let buffer = String.create 1024 in let br = Unix.read fd buffer 0 (Random.int 1000) in Unix.close fd; ignore (Unix.select [] [] [] 0.1); send_data buffer br let do_work _ = read_data (); Mutex.lock counter_mutex; decr threads_counter; Mutex.unlock counter_mutex let _ = while true do Mutex.lock counter_mutex; if !threads_counter 20 then begin incr threads_counter; ignore (Thread.create do_work ()); end; Mutex.unlock counter_mutex; done; Unix.sleep 100 It checks if thread count is less than 20. If so, new thread is created and threads_counter incremented. After thread is done, threads_counter is decremented. After 5 minutes of work, RSS usage was grown from 3300KB to 5670KB on FreeBSD 7.0 and from 976KB to 1200KB on Linux. In both cases Ocaml 3.10.2 was used. ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
[Caml-list] Re: memory usage
Can you make sure that all your function terminate and are joined (Thread.join). I think this will help to make sure that the thread exit and call thread_kill (see OCaml source code). If you take a look at thread_kill there is a function stat_free + set to NULL things called stack_low, stack_high... Maybe all the data you are seeing come from this... I am not sure that Thread.join will free anything, but it will help you to be sure that your thread has exited correctly. Regards, Sylvain Le Gall ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
Re: [Caml-list] Re: memory usage
On Mon, Jan 12, 2009 at 11:03:10PM +0800, John Lepikhin wrote: The only specific module is ocaml-fd (send file descriptors over pipes). What version of OCaml is this? 3.10.2. Low level parts of the GC were rewritten in 3.11. Well, the way that it gets blocks of memory off the operating system anyhow ... https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9 http://caml.inria.fr/pub/ml-archives/caml-list/2008/05/9c24581520a98afa2e11185845b5458a.en.html So maybe upgrading to 3.11 is an option for you? I don't know whether it will make a difference, but it's possible it will change the behaviour and 3.11 is quite stable so it's unlikely to introduce any regressions. Rich. -- Richard Jones Red Hat ___ Caml-list mailing list. Subscription management: http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list Archives: http://caml.inria.fr Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
