Hi, I'm trying to debug and understand an issue on several of my Haraka servers after a recent update.
I'm seeing large amounts of memory being used and never returned back to the operating system causing the machine to swap and eventually the Linux OOM killer shooting the processes. I thought that there must be a memory leak somewhere - but I've used node-heapdump and taken multiple snapshots and compared them and whilst I'm seeing a small growth I'm not seeing anything major. However the heapdump reports the size at ~97Mb whereas the RSS of the process at the time of the dump was 1.1Gb which is what I really don't understand at all. Would such a disparity between heap and RSS indicate that an external C++ module that is being used is leaking memory?? If so, what is the best way of finding out which? I've tried the mtrace module - but it caused the process to completely hang after only a few minutes of running so I had to kill it. The output that I got from it is attached - I'm not really sure it tells me anything though. Any suggestions as to how to work out what is going on here and how to fix it would be gratefully received. Kind regards, Steve. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
unrecognized type on line 156: "antEiNS0_4Code4KindEiNS0_6HandleINS0_6StringEEENS4_INS0_6ObjectEEENS4_INS0_8JSObjectEEENS4_INS0_10JSFunctionEEE+0x26d)[0x89f6bd] - 0x1f7bccc0" regex fail on line 1375478: "@ /lib64/libc.so.6:(__nss_lookup_function+0x" parsed 1375478 lines Addr Size Count Traffic Module Symbol -------------- --------- ----- ------- -------------------------------- ------------------------------------------------ 0x30220f455a 0 0 2 /lib64/libc.so.6 getifaddrs 0x3022097b06 0 0 2 /lib64/libc.so.6 getgrnam 0x30220dfdf2 0 0 2 /lib64/libc.so.6 argp_parse 0x3022026ef8 0 0 2 /lib64/libc.so.6 new_composite_name 0x302202853d 0 0 2 /lib64/libc.so.6 _nl_load_locale_from_archive 0x302202d97e 0 0 2 /lib64/libc.so.6 _nl_normalize_codeset 0x5e87a7 0 0 2 /usr/bin/node uv_cpu_info 0x5e85f1 0 0 2 /usr/bin/node uv_interface_addresses 0x2b52db4c81e7 0 0 2 /usr/lib/node_modules/Haraka/... sdsnewlen 0x5d8dff 0 0 4 /usr/bin/node eio_fstat 0x2b52da5d8dac 0 0 4 /usr/lib/node_modules/Haraka/... (anonymous namespace)::Iconv::Convert(char*, ... 0x5d99e3 0 0 4 /usr/bin/node eio_open 0x5d90cf 0 0 4 /usr/bin/node eio_close 0x5d8f64 0 0 4 /usr/bin/node eio_read 0x5d81e2 0 0 4 /usr/bin/node No symbol 0x30220617be 0 0 4 /lib64/libc.so.6 getdelim 0x30220bc16b 0 0 4 /lib64/libc.so.6 gaih_inet 0x2b52da5d884d 0 0 4 /usr/lib/node_modules/Haraka/... libiconv_open 0x5d20c5 0 0 6 Haraka (worker) cn=1203 cc=10... ares_parse_ptr_reply 0x8dae6b 0 0 6 Haraka (worker) cn=5 cc=5 cps... v8::internal::Deoptimizer::DoComputeOsrOutput... 0x3022078ce2 0 0 8 /lib64/libc.so.6 strndup 0x63e840 0 0 8 /usr/bin/node CRYPTO_realloc 0x30220f42fb 0 0 10 /lib64/libc.so.6 __netlink_request 0x5d1b3e 0 0 16 Haraka (worker) cn=3 cc=3 cps... ares_parse_ns_reply 0x5d1d2b 0 0 16 Haraka (worker) cn=3 cc=3 cps... ares_parse_ns_reply 0x5d1d3d 0 0 16 Haraka (worker) cn=3 cc=3 cps... ares_parse_ns_reply 0x2b52da5d8b97 0 0 20 /usr/lib/node_modules/Haraka/... (anonymous namespace)::grow(char**, unsigned ... 0x3022027d1d 0 0 24 /lib64/libc.so.6 _nl_intern_locale_data 0x5d27f9 0 0 28 Haraka (worker) cn=1 cc=1 cps... ares_parse_txt_reply 0x30220327dd 0 0 44 /lib64/libc.so.6 qsort 0x5d1c62 0 0 44 Haraka (worker) cn=3 cc=3 cps... ares_parse_ns_reply 0x5ccbdd 0 0 102 Haraka (worker) cn=1 cc=1 cps... ares_malloc_data 0x6fce58 0 0 330 /usr/bin/node v8::internal::Deoptimizer::Deoptimizer(v8::in... 0x8da8e9 0 0 362 /usr/bin/node v8::internal::Deoptimizer::DoComputeJSFrame(v... 0x5d12db 0 0 3442 /usr/bin/node ares_parse_a_reply 0x5d1724 0 0 3442 /usr/bin/node ares_parse_a_reply 0x5d1627 0 0 3442 /usr/bin/node ares_parse_a_reply 0x5d1608 0 0 3442 /usr/bin/node ares_parse_a_reply 0x302206847d 0 0 6790 /lib64/libc.so.6 open_memstream 0x3022068438 0 0 6790 /lib64/libc.so.6 open_memstream 0x5d21c8 0 0 9608 Haraka (worker) cn=0 cc=0 cps... ares_parse_ptr_reply 0x5d21ab 0 0 9608 Haraka (worker) cn=0 cc=0 cps... ares_parse_ptr_reply 0x5d21ec 0 0 9608 Haraka (worker) cn=0 cc=0 cps... ares_parse_ptr_reply 0x5d2195 0 0 9608 Haraka (worker) cn=0 cc=0 cps... ares_parse_ptr_reply 0x5d1ed7 0 0 9614 Haraka (worker) cn=0 cc=0 cps... ares_parse_ptr_reply 0x5d205f 0 0 10080 Haraka (worker) cn=0 cc=0 cps... ares_parse_ptr_reply 0x30220609aa 0 0 11836 /lib64/libc.so.6 __fopen_internal 0x5ea62d 0 0 17234 /usr/bin/node ares_mkquery 0x5ea42a 0 0 73438 /usr/bin/node ares_expand_name 0x5ce196 4 1 35405 Haraka (worker) cn=0 cc=0 cps... ares__get_hostent 0x30220e5c38 16 1 3 /lib64/libc.so.6 __nss_database_lookup 0x5ce1db 16 1 35405 Haraka (worker) cn=0 cc=0 cps... ares__get_hostent 0x5ce170 16 1 35405 Haraka (worker) cn=0 cc=0 cps... ares__get_hostent 0x30220e5015 24 1 3 /lib64/libc.so.6 __nss_lookup_function 0x3021c07d32 25 1 3 /lib64/ld-linux-x86-64.so.2 _dl_map_object 0x5ce12e 32 1 35401 Haraka (worker) cn=0 cc=0 cps... ares__get_hostent 0x5d5d0a 48 1 5 Haraka (worker) cn=4 cc=4 cps... uv_getaddrinfo 0x30220e4f00 48 3 5 /lib64/libc.so.6 __nss_lookup_function 0x5d49f2 64 8 17226 /usr/bin/node ares_send 0x5e5cea 88 1 1 /usr/bin/node uv_tcp_listen 0x5d6f56 96 3 3 Haraka (worker) cn=5 cc=5 cps... No symbol 0x302208c3b4 122 6 6 /lib64/libc.so.6 __tzstring 0x30220d360e 128 4 6 /lib64/libc.so.6 tsearch 0x5d4223 128 8 17226 /usr/bin/node ares_query 0x5d8139 144 1 18201 /usr/bin/node No symbol 0x5d8773 176 1 18449 /usr/bin/node No symbol 0x2b52db2bee09 192 8 6790 /usr/lib/node_modules/Haraka/... node::Syslog::log(v8::Arguments const&) 0x3022068564 225 1 6797 /lib64/libc.so.6 _IO_mem_finish 0x3021c0703c 230 3 3 /lib64/ld-linux-x86-64.so.2 expand_dynamic_string_token 0x5cb463 232 1 157 /usr/bin/node No symbol 0x3021c0a1f1 255 4 6 /lib64/ld-linux-x86-64.so.2 _dl_new_object 0x30220e5da8 362 15 45 /lib64/libc.so.6 __nss_database_lookup 0x5d49d5 367 8 17226 /usr/bin/node ares_send 0x3021c0bbab 416 4 6 /lib64/ld-linux-x86-64.so.2 _dl_map_object_deps 0x5cd495 512 8 11792 Haraka (worker) cn=0 cc=0 cps... ares_gethostbyaddr 0x3021c0ea96 528 4 6 /lib64/ld-linux-x86-64.so.2 _dl_check_map_versions 0x63e793 528 10 1588 /usr/bin/node CRYPTO_malloc 0x2b52db4c5580 560 1 1 /usr/lib/node_modules/Haraka/... redisReaderCreate 0x3021c10163 864 3 3 /lib64/ld-linux-x86-64.so.2 _dl_allocate_tls 0x5d42a1 896 7 47167 Haraka (worker) cn=0 cc=0 cps... ares__read_line 0x30220e5301 1034 19 57 /lib64/libc.so.6 nss_parse_service_list 0x5d49ba 1600 8 17226 /usr/bin/node ares_send 0x5d855f 1760 10 6796 /usr/bin/node eio_custom 0x3022078c82 1874 33 203977 /lib64/libc.so.6 strdup 0x302208d8a5 1912 1 1 /lib64/libc.so.6 __tzfile_read 0x3021c09f8b 4888 4 6 /lib64/ld-linux-x86-64.so.2 _dl_new_object 0x2b52db4c7bd5 4971 1 3 /usr/lib/node_modules/Haraka/... sdsMakeRoomFor 0x93e4c6 23200 12 278570 /usr/bin/node v8::internal::Malloced::New(unsigned long) 0x5dbd40 47064 4 30 /usr/bin/node No symbol 0x5cbc71 342002 591 9861 /usr/bin/node uv_fs_poll_start 0x30258bd25d 13428625 2181 365125 /usr/lib64/libstdc++.so.6 operator new(unsigned long) -------------- --------- ----- ------- -------------------------------- ------------------------------------------------ Totals 13866272 2984 1375068
