Re: [OpenSIPS-Users] Possible memory leak in PKG
Hi all, As a quick update, thanks to Arto's help and troubleshooting, a mem leak was indeed identified. The fix is now available on all versions. https://github.com/OpenSIPS/opensips/commit/f07e90aac92f33d3321d73ed69ca19850b7e10b4 Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Bootcamp 5-16 Dec 2022, online https://www.opensips.org/training/OpenSIPS_eBootcamp_2022/ On 10/10/22 4:55 PM, Arto Kuiri wrote: Hi, Yes, that is where memory leak seams to be. This happens in all SIP udp receiver processes. I can provide logs and config etc offlist. -Arto *Lähettäjä:* Bogdan-Andrei Iancu *Lähetetty:* maanantai 10. lokakuuta 2022 16.33 *Vastaanottaja:* OpenSIPS users mailling list ; Arto Kuiri *Aihe:* Re: [OpenSIPS-Users] Possible memory leak in PKG Hi Arto, Thanks for the report here. So, the 3.2.7 suffers of a mem leak which DOES NOT exist in 3.1.1, mainly this 109600 : 4858 x [dlg_vals.c: fetch_dlg_value, line 176] right ? Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com <https://www.opensips-solutions.com> OpenSIPS Summit 27-30 Sept 2022, Athens https://www.opensips.org/events/Summit-2022Athens/ <https://www.opensips.org/events/Summit-2022Athens/> On 10/5/22 10:16 AM, Arto Kuiri wrote: Hi, I think I have stumbled to some kind of memory leak. I made new opensips server used same opensips.cfg (changed only ip address) as in my older servers and after some time I started to get these to log file: /usr/sbin/opensips[1145854]: ERROR:core:fm_malloc: not enough free pkg memory (2312 bytes left, need 2472), please increase the "-M" command line parameter! /usr/sbin/opensips[1145854]: ERROR:core:receive_msg: no pkg mem left for sip_msg Older servers works fine with much higher load. Old servers are with opensips 3.1.1 and new server is 3.2.7 : opensips -V version: opensips 3.1.1 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on with gcc 9 opensips -V version: opensips 3.2.7 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on with gcc 11 I allocated more memory and changed private memory allocator to F_MALLOC_DBG After while I checked what process had highest memory useage and I did: opensips-cli -x mi mem_pkg_dump (PID was "SIP receiver udp") ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Possible memory leak in PKG
Hi Arto, Thanks for the report here. So, the 3.2.7 suffers of a mem leak which DOES NOT exist in 3.1.1, mainly this 109600 : 4858 x [dlg_vals.c: fetch_dlg_value, line 176] right ? Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 27-30 Sept 2022, Athens https://www.opensips.org/events/Summit-2022Athens/ On 10/5/22 10:16 AM, Arto Kuiri wrote: Hi, I think I have stumbled to some kind of memory leak. I made new opensips server used same opensips.cfg (changed only ip address) as in my older servers and after some time I started to get these to log file: /usr/sbin/opensips[1145854]: ERROR:core:fm_malloc: not enough free pkg memory (2312 bytes left, need 2472), please increase the "-M" command line parameter! /usr/sbin/opensips[1145854]: ERROR:core:receive_msg: no pkg mem left for sip_msg Older servers works fine with much higher load. Old servers are with opensips 3.1.1 and new server is 3.2.7 : opensips -V version: opensips 3.1.1 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on with gcc 9 opensips -V version: opensips 3.2.7 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on with gcc 11 I allocated more memory and changed private memory allocator to F_MALLOC_DBG After while I checked what process had highest memory useage and I did: opensips-cli -x mi mem_pkg_dump (PID was "SIP receiver udp") ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
[OpenSIPS-Users] Possible memory leak in PKG
Hi, I think I have stumbled to some kind of memory leak. I made new opensips server used same opensips.cfg (changed only ip address) as in my older servers and after some time I started to get these to log file: /usr/sbin/opensips[1145854]: ERROR:core:fm_malloc: not enough free pkg memory (2312 bytes left, need 2472), please increase the "-M" command line parameter! /usr/sbin/opensips[1145854]: ERROR:core:receive_msg: no pkg mem left for sip_msg Older servers works fine with much higher load. Old servers are with opensips 3.1.1 and new server is 3.2.7 : opensips -V version: opensips 3.1.1 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on with gcc 9 opensips -V version: opensips 3.2.7 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on with gcc 11 I allocated more memory and changed private memory allocator to F_MALLOC_DBG After while I checked what process had highest memory useage and I did: opensips-cli -x mi mem_pkg_dump (PID was "SIP receiver udp") from log file: dumping summary of all alloc'ed. fragments: +--- total_bytes | num_allocations x [file: func, line] +--- 40 : 1 x [acc.c: init_acc_evi, line 1137] 8 : 1 x [drouting.c: dr_init, line 1819] 200 : 7 x [script_var.c: set_var_value, line 101] 128 : 13 x [script_var.c: add_var, line 59] 1432 : 176 x [sip_i.c: pv_parse_isup_param_name, line 242] 335544 : 1 x [io_wait.c: init_io_wait, line 568] 109600 : 4858 x [dlg_vals.c: fetch_dlg_value, line 176] 16 : 1 x [cfg.y: yyparse, line 510] 32 : 1 x [rr_cb.c: register_rrcb, line 57] 1160 : 1 x [my_con.c: db_mysql_new_connection, line 158] 48 : 1 x [parser/parse_to.c: parse_to_param, line 289] 80 : 1 x [sr_module_deps.c: _alloc_module_dep, line 60] 312 : 9 x [sipmsgops.c: fixup_parse_hname, line 670] 1056 : 33 x [transformations.c: tr_parse_nparam, line 2629] 360 : 18 x [map.c: map_get, line 155] 1904 : 34 x [route_struct.c: mk_exp, line 53] 512 : 8 x [route.c: fix_expr, line 1056] 24 : 1 x [ul_evi.c: ul_event_init, line 155] 1342176 : 1 x [io_wait.c: init_io_wait, line 559] 48 : 1 x [ipc.c: ipc_register_handler, line 154] 168 : 3 x [script_var.c: set_var_value, line 112] 48 : 1 x [transformations.c: tr_eval_uri, line 1028] 16 : 1 x [dp_db.c: dp_add_connection, line 848] 624 : 13 x [script_var.c: add_var, line 52] 48 : 1 x [dlg_hash.c: state_changed_event_init, line 1024] 32 : 1 x [map.c: map_create, line 84] 24 : 1 x [ul_evi.c: ul_event_init, line 132] 8 : 1 x [socket_info.c: fix_socket_list, line 615] 96 : 1 x [mi/mi_trace.c: try_load_trace_api, line 53] 160 : 5 x [transformations.c: tr_parse_string, line 2906] 912 : 1 x [transformations.c: tr_eval_nameaddr, line 2375] 640 : 4 x [db/db.c: db_do_init, line 351] 144 : 3 x [sr_module_deps.c: solve_module_dependencies, line 293] 832 : 13 x [mod_fix.c: fixup_regcomp, line 55] 128 : 1 x [prefix_tree.c: init_prefix_tree, line 53] 503320 : 1 x [io_wait.c: init_io_wait, line 621] 40 : 1 x [transformations.c: tr_add_extra, line 109] 15456 : 276 x [route_struct.c: mk_elem, line 69] 112 : 1 x [route.c: fix_actions, line 1338] 88 : 2 x [flags.c: get_flag_id_by_name, line 202] 272 : 2 x [db/db.c: db_do_init, line 314] 41528 : 324 x [pvar.c: pv_parse_format, line 4716] 24 : 1 x [socket_info.c: fix_socket_list, line 760] 400 : 11 x [acc_extra.c: add_extra, line 155] 16 : 1 x [socket_info.c: new_sock_info, line 127] 4104 : 1 x [xlog.c: buf_init, line 124] 24 : 1 x [ul_evi.c: ul_event_init, line 269] 32 : 1 x [acc.c: init_acc_evi, line 1151] 10448 : 1 x [route.c: new_sroutes_holder, line 104] 200 : 5 x [script_cb.c: add_callback, line 60] 320 : 20 x [sipmsgops.c: fixup_method, line 728] 432 : 9 x [sr_module_deps.c: solve_module_dependencies, line 335] 240 : 1 x [acc_extra.c: add_tag, line 110] 8 : 1 x [pvar.c: pv_init_extra_list, line 5223] 5016 : 153 x [mod_fix.c: alloc_gp, line 72] 24 : 1 x [cfg.y: yyparse, line 1301]
