Re: [OpenSIPS-Users] Opensips Crashed
Hi Dragomir, Thanks for the report, but please open a ticket (if not already) reporting the crash. Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com OpenSIPS Summit 2018 http://www.opensips.org/events/Summit-2018Amsterdam On 03/26/2018 02:54 PM, Dragomir Haralambiev wrote: Hello, I use stable LTS: version: opensips 2.2.6 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: 6a921f1 main.c compiled on 01:08:40 Mar 10 2018 with gcc 4.4.7 Opensips crased. Here is dump log: https://pastebin.com/VA41R7Y9 Best regards, Dragomir ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
[OpenSIPS-Users] Opensips Crashed
Hello, I use stable LTS: version: opensips 2.2.6 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: 6a921f1 main.c compiled on 01:08:40 Mar 10 2018 with gcc 4.4.7 Opensips crased. Here is dump log: https://pastebin.com/VA41R7Y9 Best regards, Dragomir ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPs crashed
Is there updates on this? Date: Mon, 16 Jan 2017 09:51:13 -0500 > From: Ahmed Munir <ahmedmunir...@gmail.com> > To: OpenSIPs Users <users@lists.opensips.org> > Subject: Re: [OpenSIPS-Users] OpenSIPs crashed > Message-ID: >
Re: [OpenSIPS-Users] OpenSIPs crashed
See details below;
[root@qorblpsisprxyd1 ~]# opensips -V
version: opensips 2.2.2 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
main.c compiled on 11:09:47 Jan 13 2017 with gcc 4.4.7
[root@qorblpsisprxyd1 ~]# opensipsctl ps
Process:: ID=0 PID=8269 Type=attendant
Process:: ID=1 PID=8271 Type=MI FIFO
Process:: ID=2 PID=8272 Type=time_keeper
Process:: ID=3 PID=8274 Type=timer
Process:: ID=4 PID=8275 Type=SIP receiver udp:10.3.120.94:5060
Process:: ID=5 PID=8278 Type=SIP receiver udp:10.3.120.94:5060
Process:: ID=6 PID=8279 Type=SIP receiver udp:10.3.120.94:5060
Process:: ID=7 PID=8281 Type=SIP receiver udp:10.3.120.94:5060
Process:: ID=8 PID=8283 Type=SIP receiver udp:10.3.120.94:5060
Process:: ID=9 PID=8285 Type=SIP receiver udp:10.3.120.94:5060
Process:: ID=10 PID=8287 Type=SIP receiver udp:10.3.120.94:5060
Process:: ID=11 PID=8289 Type=SIP receiver udp:10.3.120.94:5060
Process:: ID=12 PID=8291 Type=Timer handler
[root@qorblpsisprxyd1 ~]# lscpu
Architecture: x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):8
On-line CPU(s) list: 0-7
Thread(s) per core:1
Core(s) per socket:4
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family:6
Model: 44
Stepping: 2
CPU MHz: 1197.000
BogoMIPS: 4266.58
Virtualization:VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-3
NUMA node1 CPU(s): 4-7
In /etc/default/opensips config, declaring shared and pkg memory as server
memory is 64 GB;
# Amount of shared memory to allocate for the running OpenSIPS server (in
Mb)
S_MEMORY=256
# Amount of pkg memory to allocate for the running OpenSIPS server (in Mb)
P_MEMORY=32
Let me know any other info needed from my end.
Date: Mon, 16 Jan 2017 10:49:37 +0200
> From: Răzvan Crainea <raz...@opensips.org>
> To: users@lists.opensips.org
> Subject: Re: [OpenSIPS-Users] OpenSIPs crashed
> Message-ID: <1584806e-a154-a5ad-a464-4eef60915...@opensips.org>
> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>
> Hi, Ahmed!
>
> Can you tell us exactly what revision of OpenSIPS you are using? Please
> provide the output of the following commands:
> opensips -V
> opensipsctl ps
>
> Also, during startup, is there a process who's "eating" a lot of CPU? If
> so, can you pinpoint the PID to see what type of process is that?
>
> Regarding the avp_db_query() issue, did you define a db_url parameter
> for it? Also I am not sure you can do something like $var(res) =
> avp_db_query(...). But anyways, this is something completely different,
> so please open a different topic for it.
>
> Best regards,
>
> Răzvan Crainea
> OpenSIPS Solutions
> www.opensips-solutions.com
>
> On 01/14/2017 12:24 AM, Ahmed Munir wrote:
> > Hi,
> >
> > I've just installed new version of opensips 2.2.2 on the test box and
> > updated by routing script, the issue currently I'm seeing alot warning
> > messages while starting opensips service below;
> >
> > /usr/sbin/opensips[6902]: WARNING:core:handle_timer_job: utimer job
> > has a 283 us delay in execution
> >
> > Number of children running on that server is 8 as it is 8 core processor.
> >
> > I would like to know what steps do I need to take to fix this issue.
> > Btw, warnings only occurred during the time of starting opensips
> > service but not during calls.
> >
> >
> > Further added, a issue I face using avp_db_query () function i.e. when
> > using it as
> >
> > $var(res) = avp_db_query("SELECT Outpulse_number,setid FROM
> > Prefix_data where Program_prefix = $var(pg_prefix)", "$avp(outpluse),
> > $avp(trunkid)");
> >
> > failed to start opensips service due to errors below;
> >
> > ERROR:avpops:__fixup_db_query_avp: no db url defined to be used by
> > this function
> > ERROR:core:fix_actions: fixing failed (code=-6) at
> > //etc/opensips/opensips.cfg:207
> > CRITICAL:core:fix_expr: fix_actions error
> > ERROR:core:main: failed to fix configuration with err code -6
> >
> >
> > If I add this line: avp_db_query("SELECT 1"); above to my $var(res) db
> > query, opensips service starts successfully.
> >
> > Please advise the steps do I need to take to fix above issues.
> >
> >
> >
>
>
--
Regards,
Ahmed Munir Chohan
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPs crashed
Hi, Ahmed!
Can you tell us exactly what revision of OpenSIPS you are using? Please
provide the output of the following commands:
opensips -V
opensipsctl ps
Also, during startup, is there a process who's "eating" a lot of CPU? If
so, can you pinpoint the PID to see what type of process is that?
Regarding the avp_db_query() issue, did you define a db_url parameter
for it? Also I am not sure you can do something like $var(res) =
avp_db_query(...). But anyways, this is something completely different,
so please open a different topic for it.
Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 01/14/2017 12:24 AM, Ahmed Munir wrote:
Hi,
I've just installed new version of opensips 2.2.2 on the test box and
updated by routing script, the issue currently I'm seeing alot warning
messages while starting opensips service below;
/usr/sbin/opensips[6902]: WARNING:core:handle_timer_job: utimer job
has a 283 us delay in execution
Number of children running on that server is 8 as it is 8 core processor.
I would like to know what steps do I need to take to fix this issue.
Btw, warnings only occurred during the time of starting opensips
service but not during calls.
Further added, a issue I face using avp_db_query () function i.e. when
using it as
$var(res) = avp_db_query("SELECT Outpulse_number,setid FROM
Prefix_data where Program_prefix = $var(pg_prefix)", "$avp(outpluse),
$avp(trunkid)");
failed to start opensips service due to errors below;
ERROR:avpops:__fixup_db_query_avp: no db url defined to be used by
this function
ERROR:core:fix_actions: fixing failed (code=-6) at
//etc/opensips/opensips.cfg:207
CRITICAL:core:fix_expr: fix_actions error
ERROR:core:main: failed to fix configuration with err code -6
If I add this line: avp_db_query("SELECT 1"); above to my $var(res) db
query, opensips service starts successfully.
Please advise the steps do I need to take to fix above issues.
From: Răzvan Crainea <raz...@opensips.org
<mailto:raz...@opensips.org>>
To: users@lists.opensips.org <mailto:users@lists.opensips.org>
Subject: Re: [OpenSIPS-Users] OpenSIPs crashed
Message-ID: <40f6dada-e121-a2da-b283-69dff891c...@opensips.org
<mailto:40f6dada-e121-a2da-b283-69dff891c...@opensips.org>>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Hi, Ahmed!
OpenSIPS 1.6.3 is no longer supported (since 2013), so there's not
much
we can do right now. Try upgrading your install to the latest 1.6.4
version and see if your problem is solved. Otherwise, upgrade to a
newer, supported version, preferably the latest stable release, 2.2.2.
Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com <http://www.opensips-solutions.com>
On 01/12/2017 11:55 PM, Ahmed Munir wrote:
> Found coredump on one of the server, see some partial message below
> while taking the back trace;
>
>
>
> Core was generated by `/usr/sbin/opensips -P
/var/run/opensips.pid -m
> 64 -u opensips -g opensips'.
>
> Program terminated with signal 11, Segmentation fault.
>
> #0 0x7f650687a069 in sip_msg_cloner () from
> /usr/lib64/opensips/modules/tm.so
>
> Missing separate debuginfos, use: debuginfo-install
> opensips-1.6.3-notls.x86_64
>
>
>
> Please advise what might be the reason causing opensips to crash.
>
> --
> Regards,
>
> Ahmed Munir Chohan
>
>
>
>
> --
> Regards,
>
> Ahmed Munir Chohan
>
--
Regards,
Ahmed Munir Chohan
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPs crashed
Hi,
I've just installed new version of opensips 2.2.2 on the test box and
updated by routing script, the issue currently I'm seeing alot warning
messages while starting opensips service below;
/usr/sbin/opensips[6902]: WARNING:core:handle_timer_job: utimer job
has a 283 us delay in execution
Number of children running on that server is 8 as it is 8 core processor.
I would like to know what steps do I need to take to fix this issue. Btw,
warnings only occurred during the time of starting opensips service but not
during calls.
Further added, a issue I face using avp_db_query () function i.e. when
using it as
$var(res) = avp_db_query("SELECT Outpulse_number,setid FROM Prefix_data
where Program_prefix = $var(pg_prefix)", "$avp(outpluse), $avp(trunkid)");
failed to start opensips service due to errors below;
ERROR:avpops:__fixup_db_query_avp: no db url defined to be used by this
function
ERROR:core:fix_actions: fixing failed (code=-6) at
//etc/opensips/opensips.cfg:207
CRITICAL:core:fix_expr: fix_actions error
ERROR:core:main: failed to fix configuration with err code -6
If I add this line: avp_db_query("SELECT 1"); above to my $var(res) db
query, opensips service starts successfully.
Please advise the steps do I need to take to fix above issues.
> From: Răzvan Crainea <raz...@opensips.org>
> To: users@lists.opensips.org
> Subject: Re: [OpenSIPS-Users] OpenSIPs crashed
> Message-ID: <40f6dada-e121-a2da-b283-69dff891c...@opensips.org>
> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>
> Hi, Ahmed!
>
> OpenSIPS 1.6.3 is no longer supported (since 2013), so there's not much
> we can do right now. Try upgrading your install to the latest 1.6.4
> version and see if your problem is solved. Otherwise, upgrade to a
> newer, supported version, preferably the latest stable release, 2.2.2.
>
> Best regards,
>
> Răzvan Crainea
> OpenSIPS Solutions
> www.opensips-solutions.com
>
> On 01/12/2017 11:55 PM, Ahmed Munir wrote:
> > Found coredump on one of the server, see some partial message below
> > while taking the back trace;
> >
> >
> >
> > Core was generated by `/usr/sbin/opensips -P /var/run/opensips.pid -m
> > 64 -u opensips -g opensips'.
> >
> > Program terminated with signal 11, Segmentation fault.
> >
> > #0 0x7f650687a069 in sip_msg_cloner () from
> > /usr/lib64/opensips/modules/tm.so
> >
> > Missing separate debuginfos, use: debuginfo-install
> > opensips-1.6.3-notls.x86_64
> >
> >
> >
> > Please advise what might be the reason causing opensips to crash.
> >
> > --
> > Regards,
> >
> > Ahmed Munir Chohan
> >
> >
> >
> >
> > --
> > Regards,
> >
> > Ahmed Munir Chohan
> >
>
--
Regards,
Ahmed Munir Chohan
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPs crashed
The version currently running is 1.6.3. Today again we got the opensips crashed issues i.e. 5 out of 8 were crashed due to below message (common on all 5); Jan 12 10:07:36 QORCLPSIPROXY02 kernel: opensips[2820]: segfault at 0 ip 004a2936 sp 7fff5cfaa430 error 6 in opensips[40+13a000] Jan 12 10:07:36 QORCLPSIPROXY02 abrt[39302]: Can't open 'core.2820': Permission denied Jan 12 10:07:36 QORCLPSIPROXY02 abrt[39302]: Saved core dump of pid 2820 (/usr/sbin/opensips) to /var/spool/abrt/ccpp-2017-01-12-10:07:36-2820 (70246400 bytes) Jan 12 10:07:36 QORCLPSIPROXY02 abrtd: Directory 'ccpp-2017-01-12-10:07:36-2820' creation detected Jan 12 10:07:38 QORCLPSIPROXY02 kernel: Bridge firewalling registered Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Sending an email... Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: /usr/sbin/sendmail: No such file or directory Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: . . . message not sent. Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Error running '/bin/mailx' Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2017-01-12-10:07:36-2820' exited with 1 Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2017-01-12-10:07:36-2820' We would like to know why we are getting this segmentation fault? Is there a way to backtrace the issue as don't have the core dump? Further added, did someone faced this similar issue(s) in past and got this fixed without upgrade? Note: we are running opensips for quite a while didn't faced this kind of issue and there is no changes made on opensips routing script. > From: Răzvan Crainea <raz...@opensips.org> >> To: users@lists.opensips.org >> Subject: Re: [OpenSIPS-Users] OpenSIPs crashed >> Message-ID: <2b35e2dc-6e11-1787-b87e-33bd29a32...@opensips.org> >> Content-Type: text/plain; charset="utf-8"; Format="flowed" >> >> Hi, Ahmed! >> >> Make sure OpenSIPS is run as root and it is allowed to write in the >> /var/spool/abrt/ directory, otherwise it is unable to write the core >> dump, therefore we can't inspect it to say what is happening. If this >> does not work, make OpenSIPS write the core dump in a writeble directory >> by changing the /proc/sys/kernel/core_pattern settings. >> Also, please let us know the version of OpenSIPS you are running. >> >> Best regards, >> >> Răzvan Crainea >> OpenSIPS Solutions >> www.opensips-solutions.com >> >> >> On 01/11/2017 11:10 PM, Ahmed Munir wrote: >> > Hi, >> > >> > Our OpenSIPs service crashed with below error; >> > >> > Jan 11 12:16:19 QORBLPSIPROXY05 abrtd: Directory >> > 'ccpp-2017-01-11-12:16:19-2807' creation detected >> > Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Saved core dump of pid >> > 2807 (/usr/sbin/opensips) to >> > /var/spool/abrt/ccpp-2017-01-11-12:16:19-2807 (70225920 bytes) >> > Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Can't open 'core.2807': >> > Permission denied >> > Jan 11 12:16:19 QORBLPSIPROXY05 kernel: opensips[2807]: segfault at 29 >> > ip 004bae7a sp 7fffdb7734d0 error 4 in >> opensips[40+13a000] >> > >> > >> > We would like to know, what might be the reason for the crash. >> > >> > Further added, there is another server we are running OpenSIPs, the >> > opensips child processes utilizing 100% of CPU and the system load >> > average reach around 'load average: 20.01, 18.03, 24.00' as normally >> > it is below 1 (load average). >> > >> > After looking into logs, unable to find the info what might causing >> > the CPU to spike. >> > >> > Please advise what useful steps to take for narrowing down this issue. >> > >> > >> > -- >> > Regards, >> > >> > Ahmed Munir Chohan >> > >> >> > > > -- > Regards, > > Ahmed Munir Chohan > > -- Regards, Ahmed Munir Chohan ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPs crashed
The version currently running is 1.6.3. Will try to enable core dump and share the info if run into the issues again. From: Răzvan Crainea <raz...@opensips.org> > To: users@lists.opensips.org > Subject: Re: [OpenSIPS-Users] OpenSIPs crashed > Message-ID: <2b35e2dc-6e11-1787-b87e-33bd29a32...@opensips.org> > Content-Type: text/plain; charset="utf-8"; Format="flowed" > > Hi, Ahmed! > > Make sure OpenSIPS is run as root and it is allowed to write in the > /var/spool/abrt/ directory, otherwise it is unable to write the core > dump, therefore we can't inspect it to say what is happening. If this > does not work, make OpenSIPS write the core dump in a writeble directory > by changing the /proc/sys/kernel/core_pattern settings. > Also, please let us know the version of OpenSIPS you are running. > > Best regards, > > Răzvan Crainea > OpenSIPS Solutions > www.opensips-solutions.com > > On 01/11/2017 11:10 PM, Ahmed Munir wrote: > > Hi, > > > > Our OpenSIPs service crashed with below error; > > > > Jan 11 12:16:19 QORBLPSIPROXY05 abrtd: Directory > > 'ccpp-2017-01-11-12:16:19-2807' creation detected > > Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Saved core dump of pid > > 2807 (/usr/sbin/opensips) to > > /var/spool/abrt/ccpp-2017-01-11-12:16:19-2807 (70225920 bytes) > > Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Can't open 'core.2807': > > Permission denied > > Jan 11 12:16:19 QORBLPSIPROXY05 kernel: opensips[2807]: segfault at 29 > > ip 004bae7a sp 7fffdb7734d0 error 4 in > opensips[40+13a000] > > > > > > We would like to know, what might be the reason for the crash. > > > > Further added, there is another server we are running OpenSIPs, the > > opensips child processes utilizing 100% of CPU and the system load > > average reach around 'load average: 20.01, 18.03, 24.00' as normally > > it is below 1 (load average). > > > > After looking into logs, unable to find the info what might causing > > the CPU to spike. > > > > Please advise what useful steps to take for narrowing down this issue. > > > > > > -- > > Regards, > > > > Ahmed Munir Chohan > > > > -- Regards, Ahmed Munir Chohan ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
[OpenSIPS-Users] OpenSIPs crashed
Hi, Our OpenSIPs service crashed with below error; Jan 11 12:16:19 QORBLPSIPROXY05 abrtd: Directory 'ccpp-2017-01-11-12:16:19-2807' creation detected Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Saved core dump of pid 2807 (/usr/sbin/opensips) to /var/spool/abrt/ccpp-2017-01-11-12:16:19-2807 (70225920 bytes) Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Can't open 'core.2807': Permission denied Jan 11 12:16:19 QORBLPSIPROXY05 kernel: opensips[2807]: segfault at 29 ip 004bae7a sp 7fffdb7734d0 error 4 in opensips[40+13a000] We would like to know, what might be the reason for the crash. Further added, there is another server we are running OpenSIPs, the opensips child processes utilizing 100% of CPU and the system load average reach around 'load average: 20.01, 18.03, 24.00' as normally it is below 1 (load average). After looking into logs, unable to find the info what might causing the CPU to spike. Please advise what useful steps to take for narrowing down this issue. -- Regards, Ahmed Munir Chohan ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
I've modified my init.d file with /usr/local/sbin/opensips -P /var/run/opensips/opensips.pid *-m 256 -M 8* -u opensips -g opensips -f /usr/local/etc/opensips/opensips.cfg wait and see ;) 2014-09-16 15:12 GMT+02:00 Mickael Hubert m.hub...@hexanet.fr: Hi all, after upgrade Opensips, I've faced same memory error (see below). Have you a help for me please ? Thanks in advance Sep 16 15:05:23 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 395 Sep 16 15:05:23 asbc2 /usr/local/sbin/opensips[15776]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 388 Sep 16 15:05:24 asbc2 /usr/local/sbin/opensips[15777]: ERROR:core:build_req_buf_from_sip_req: out of pkg memory Sep 16 15:05:24 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 433 Sep 16 15:05:24 asbc2 /usr/local/sbin/opensips[15776]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 427 Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:parse_from_header: out of pkg_memory Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:parse_from_header: out of pkg_memory Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 355 Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:parse_from_header: out of pkg_memory Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 531 Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15773]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 355 Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15773]: ERROR:core:new_credentials: no pkg memory left Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15773]: ERROR:core:new_credentials: no pkg memory left version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: c529bb1 main.c compiled on 16:58:09 Sep 11 2014 with gcc 4.7 2014-09-10 16:16 GMT+02:00 Mickael Hubert m.hub...@hexanet.fr: Ok thanks for your response, I'll have upgrade my opensips immediately. but for me, my memory conf is wrong. -m 64 -M 4 with 20 childrens: 4Mo * 20 childrens = 80 Mo shared memory, but with -m option with 64Mo it's impossible no ? thanks 2014-09-10 13:27 GMT+02:00 Vlad Paiu vladp...@opensips.org: Hello, Can you please upgrade to the latest 1.11 from GIT ? There has been a fix related to this on GIT starting from https://github.com/OpenSIPS/opensips/commit/ea9191b6993a36b6190eb218f318d0a876421311 Best Regards, Vlad Paiu OpenSIPS Developerhttp://www.opensips-solutions.com On 10.09.2014 12:58, Mickael Hubert wrote: Hi Bogdan, in addition, we use these options: -m 64 -M 4 and in opensips.cfg children=20 private memory: 4 Mo Share memory: 64Mo We use a server with 16Go of ram it's good conf ? Thanks a lot for your help. 2014-09-10 11:44 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, Here's the version of opensips we use : [root@asbc2b kemathy]# /usr/local/sbin/opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: ff23ed0 main.c compiled on 19:24:42 Jul 23 2014 with gcc 4.4.7 About those points : - the out of mem was reported for pkg memory (and not for shm) == The out of mem was reported for pkg memory - you sent the SIGUSR1 signal to the process complaining about lack of mem == I don't know if I made it well... So, here are two links pointing to other memdumps, taken on processes which seem to lack of mem Sep 10 05:34:05 asbc2b /usr/local/sbin/opensips[920]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 522 Sep 10 05:34:04 asbc2b /usr/local/sbin/opensips[923]: ERROR:core:received_builder: out of pkg memory Sep 10 05:33:57 asbc2b /usr/local/sbin/opensips[937]: ERROR:core:parse_headers: pkg memory allocation failed Sep 10 05:33:57 asbc2b /usr/local/sbin/opensips[937]: ERROR:core:add_lump_rpl: no free pkg memory ! https://www.dropbox.com/s/p62hb16wpk0i86e/memdump_process_920.txt?dl=0 https://www.dropbox.com/s/uim49o6zvj6zvu3/memdump_process_923.txt?dl=0 https://www.dropbox.com/s/c6tksw4j4lgk5d3/memdump_process_937.txt?dl=0 So now, I'm sure that the memdumps I've taken concern processes impacted by the lack of pkg mem...
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi all, after upgrade Opensips, I've faced same memory error (see below). Have you a help for me please ? Thanks in advance Sep 16 15:05:23 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 395 Sep 16 15:05:23 asbc2 /usr/local/sbin/opensips[15776]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 388 Sep 16 15:05:24 asbc2 /usr/local/sbin/opensips[15777]: ERROR:core:build_req_buf_from_sip_req: out of pkg memory Sep 16 15:05:24 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 433 Sep 16 15:05:24 asbc2 /usr/local/sbin/opensips[15776]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 427 Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:parse_from_header: out of pkg_memory Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:parse_from_header: out of pkg_memory Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 355 Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:parse_from_header: out of pkg_memory Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15783]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 531 Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15773]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 355 Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15773]: ERROR:core:new_credentials: no pkg memory left Sep 16 15:05:25 asbc2 /usr/local/sbin/opensips[15773]: ERROR:core:new_credentials: no pkg memory left version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: c529bb1 main.c compiled on 16:58:09 Sep 11 2014 with gcc 4.7 2014-09-10 16:16 GMT+02:00 Mickael Hubert m.hub...@hexanet.fr: Ok thanks for your response, I'll have upgrade my opensips immediately. but for me, my memory conf is wrong. -m 64 -M 4 with 20 childrens: 4Mo * 20 childrens = 80 Mo shared memory, but with -m option with 64Mo it's impossible no ? thanks 2014-09-10 13:27 GMT+02:00 Vlad Paiu vladp...@opensips.org: Hello, Can you please upgrade to the latest 1.11 from GIT ? There has been a fix related to this on GIT starting from https://github.com/OpenSIPS/opensips/commit/ea9191b6993a36b6190eb218f318d0a876421311 Best Regards, Vlad Paiu OpenSIPS Developerhttp://www.opensips-solutions.com On 10.09.2014 12:58, Mickael Hubert wrote: Hi Bogdan, in addition, we use these options: -m 64 -M 4 and in opensips.cfg children=20 private memory: 4 Mo Share memory: 64Mo We use a server with 16Go of ram it's good conf ? Thanks a lot for your help. 2014-09-10 11:44 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, Here's the version of opensips we use : [root@asbc2b kemathy]# /usr/local/sbin/opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: ff23ed0 main.c compiled on 19:24:42 Jul 23 2014 with gcc 4.4.7 About those points : - the out of mem was reported for pkg memory (and not for shm) == The out of mem was reported for pkg memory - you sent the SIGUSR1 signal to the process complaining about lack of mem == I don't know if I made it well... So, here are two links pointing to other memdumps, taken on processes which seem to lack of mem Sep 10 05:34:05 asbc2b /usr/local/sbin/opensips[920]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 522 Sep 10 05:34:04 asbc2b /usr/local/sbin/opensips[923]: ERROR:core:received_builder: out of pkg memory Sep 10 05:33:57 asbc2b /usr/local/sbin/opensips[937]: ERROR:core:parse_headers: pkg memory allocation failed Sep 10 05:33:57 asbc2b /usr/local/sbin/opensips[937]: ERROR:core:add_lump_rpl: no free pkg memory ! https://www.dropbox.com/s/p62hb16wpk0i86e/memdump_process_920.txt?dl=0 https://www.dropbox.com/s/uim49o6zvj6zvu3/memdump_process_923.txt?dl=0 https://www.dropbox.com/s/c6tksw4j4lgk5d3/memdump_process_937.txt?dl=0 So now, I'm sure that the memdumps I've taken concern processes impacted by the lack of pkg mem... Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-10 10:04 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Kevin, This is stranger as nothing in the dump points to a leak/issue - do you get the out of mem error during a SB
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Kevin, Any chance to get the mem dump after waiting ? Thanks Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 04.09.2014 13:09, Kevin Mathy wrote: Hi Bogdan, So, I've faced the problem once again, and followed the Determining the cause part of the link you gave me. I've waited more than 20 minutes (near an hour), and re-sent some voice trafic to the problematic opensips server : memory errors just came after that. So, I think it's a memory leak, no ? Now, what can I do to solve this ? Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-08-20 17:22 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, So it is more a mem leak than a crash. Any chances to get a memdump after memory is exhausted ? See: http://www.opensips.org/Documentation/TroubleShooting-OutOfMem Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 20.08.2014 15:35, Kevin Mathy wrote: Hi Bogdan, In fact, opensips didn't crashed : we got the previous error messages in the logs, but then we switched our voice traffic to another server, and opensips on the 1st one never crashed... We restarted the service in the following night, to be sure to make it rework well. So, is there a way in this case to get some backtrace, even if it didn't crashed ? Maybe something generated at startup ? Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-08-19 20:34 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, Try to extract at least a backtrace, to see what kind of problem we are facing. Then we can decide if a mem dump is required or not. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 19.08.2014 12:03, Kevin Mathy wrote: Hi Bogdan, Coming back to you after some holidays, and unfortunately, new Opensips crashes... Some weeks ago, we upgraded our servers to OpenSIPS 1.11, but we faced a new crash, always with the out of pkg memory error, last week. Here is the exact version : root@asbc2:/home/kemathy# opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: d41eb41 main.c compiled on 19:23:35 Jul 23 2014 with gcc 4.7 Here are an extract of logs we get at the crash-time (I masked phone numbers and IPs for security purposes) : Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contacts: failed to parse contact parameters Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:contact_parser: failed to parse contacts Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contact: failed to parse contact Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:pv_get_contact_body: failed to parse contact hdr Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 526 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:sl:sl_send_reply_helper: response building failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:auth:challenge: failed to send the response Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, Got the same errors on my second opensips server (second node of the cluster). Here is the memdump : https://www.dropbox.com/s/d2gacgrxik3xs1t/memdump_server2_1.txt?dl=0 Thanks for your help, Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-09 14:56 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, I've got now a memdump ! https://www.dropbox.com/s/5hb678r0ym9vtt1/memdump1.txt?dl=0 And if you need the list of the opensips processes running at the moment : https://www.dropbox.com/s/hqurr3ilamc0a0g/opensips_processes_list.txt?dl=0 I hope it'll be ok... If you need more things, I didn't have restarted the service yet, so I'm able to get other memdumps. Thanks for your help, Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-09 13:06 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, yes, that's the right way of getting a mem dump during runtime. Also you will automatically get one during the shutdown procedure. Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com -- ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Kevin, First just checking if: - the out of mem was reported for pkg memory (and not for shm) - you sent the SIGUSR1 signal to the process complaining about lack of mem IF you look at the dump: used= 1630504, used+overhead=4188560, free=5744 but there are only 64 fragments with avg len 100, so not more than 100K or so it is a bit strange. I remember some fixes in that area (where the stats for available mem were kept) - what exact version of OpenSIPS are you using ? Thanks and regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 09.09.2014 15:56, Kevin Mathy wrote: Hi Bogdan, I've got now a memdump ! https://www.dropbox.com/s/5hb678r0ym9vtt1/memdump1.txt?dl=0 And if you need the list of the opensips processes running at the moment : https://www.dropbox.com/s/hqurr3ilamc0a0g/opensips_processes_list.txt?dl=0 I hope it'll be ok... If you need more things, I didn't have restarted the service yet, so I'm able to get other memdumps. Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-09-09 13:06 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, yes, that's the right way of getting a mem dump during runtime. Also you will automatically get one during the shutdown procedure. Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, Here's the version of opensips we use : [root@asbc2b kemathy]# /usr/local/sbin/opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: ff23ed0 main.c compiled on 19:24:42 Jul 23 2014 with gcc 4.4.7 About those points : - the out of mem was reported for pkg memory (and not for shm) == The out of mem was reported for pkg memory - you sent the SIGUSR1 signal to the process complaining about lack of mem == I don't know if I made it well... So, here are two links pointing to other memdumps, taken on processes which seem to lack of mem Sep 10 05:34:05 asbc2b /usr/local/sbin/opensips[920]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 522 Sep 10 05:34:04 asbc2b /usr/local/sbin/opensips[923]: ERROR:core:received_builder: out of pkg memory Sep 10 05:33:57 asbc2b /usr/local/sbin/opensips[937]: ERROR:core:parse_headers: pkg memory allocation failed Sep 10 05:33:57 asbc2b /usr/local/sbin/opensips[937]: ERROR:core:add_lump_rpl: no free pkg memory ! https://www.dropbox.com/s/p62hb16wpk0i86e/memdump_process_920.txt?dl=0 https://www.dropbox.com/s/uim49o6zvj6zvu3/memdump_process_923.txt?dl=0 https://www.dropbox.com/s/c6tksw4j4lgk5d3/memdump_process_937.txt?dl=0 So now, I'm sure that the memdumps I've taken concern processes impacted by the lack of pkg mem... Thanks for your help, Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-10 10:04 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Kevin, This is stranger as nothing in the dump points to a leak/issue - do you get the out of mem error during a SB query ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 10.09.2014 09:53, Kevin Mathy wrote: Hi Bogdan, Got the same errors on my second opensips server (second node of the cluster). Here is the memdump : https://www.dropbox.com/s/d2gacgrxik3xs1t/memdump_server2_1.txt?dl=0 Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-09 14:56 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, I've got now a memdump ! https://www.dropbox.com/s/5hb678r0ym9vtt1/memdump1.txt?dl=0 And if you need the list of the opensips processes running at the moment : https://www.dropbox.com/s/hqurr3ilamc0a0g/opensips_processes_list.txt?dl=0 I hope it'll be ok... If you need more things, I didn't have restarted the service yet, so I'm able to get other memdumps. Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-09 13:06 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, yes, that's the right way of getting a mem dump during runtime. Also you will automatically get one during the shutdown procedure. Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com -- ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, in addition, we use these options: -m 64 -M 4 and in opensips.cfg children=20 private memory: 4 Mo Share memory: 64Mo We use a server with 16Go of ram it's good conf ? Thanks a lot for your help. 2014-09-10 11:44 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, Here's the version of opensips we use : [root@asbc2b kemathy]# /usr/local/sbin/opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: ff23ed0 main.c compiled on 19:24:42 Jul 23 2014 with gcc 4.4.7 About those points : - the out of mem was reported for pkg memory (and not for shm) == The out of mem was reported for pkg memory - you sent the SIGUSR1 signal to the process complaining about lack of mem == I don't know if I made it well... So, here are two links pointing to other memdumps, taken on processes which seem to lack of mem Sep 10 05:34:05 asbc2b /usr/local/sbin/opensips[920]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 522 Sep 10 05:34:04 asbc2b /usr/local/sbin/opensips[923]: ERROR:core:received_builder: out of pkg memory Sep 10 05:33:57 asbc2b /usr/local/sbin/opensips[937]: ERROR:core:parse_headers: pkg memory allocation failed Sep 10 05:33:57 asbc2b /usr/local/sbin/opensips[937]: ERROR:core:add_lump_rpl: no free pkg memory ! https://www.dropbox.com/s/p62hb16wpk0i86e/memdump_process_920.txt?dl=0 https://www.dropbox.com/s/uim49o6zvj6zvu3/memdump_process_923.txt?dl=0 https://www.dropbox.com/s/c6tksw4j4lgk5d3/memdump_process_937.txt?dl=0 So now, I'm sure that the memdumps I've taken concern processes impacted by the lack of pkg mem... Thanks for your help, Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-10 10:04 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Kevin, This is stranger as nothing in the dump points to a leak/issue - do you get the out of mem error during a SB query ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 10.09.2014 09:53, Kevin Mathy wrote: Hi Bogdan, Got the same errors on my second opensips server (second node of the cluster). Here is the memdump : https://www.dropbox.com/s/d2gacgrxik3xs1t/memdump_server2_1.txt?dl=0 Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-09 14:56 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, I've got now a memdump ! https://www.dropbox.com/s/5hb678r0ym9vtt1/memdump1.txt?dl=0 And if you need the list of the opensips processes running at the moment : https://www.dropbox.com/s/hqurr3ilamc0a0g/opensips_processes_list.txt?dl=0 I hope it'll be ok... If you need more things, I didn't have restarted the service yet, so I'm able to get other memdumps. Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-09 13:06 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, yes, that's the right way of getting a mem dump during runtime. Also you will automatically get one during the shutdown procedure. Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com -- Cordialement HUBERT Mickaël Ingénieur VOIP - Hexanet -- ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Ok thanks for your response, I'll have upgrade my opensips immediately. but for me, my memory conf is wrong. -m 64 -M 4 with 20 childrens: 4Mo * 20 childrens = 80 Mo shared memory, but with -m option with 64Mo it's impossible no ? thanks 2014-09-10 13:27 GMT+02:00 Vlad Paiu vladp...@opensips.org: Hello, Can you please upgrade to the latest 1.11 from GIT ? There has been a fix related to this on GIT starting from https://github.com/OpenSIPS/opensips/commit/ea9191b6993a36b6190eb218f318d0a876421311 Best Regards, Vlad Paiu OpenSIPS Developerhttp://www.opensips-solutions.com On 10.09.2014 12:58, Mickael Hubert wrote: Hi Bogdan, in addition, we use these options: -m 64 -M 4 and in opensips.cfg children=20 private memory: 4 Mo Share memory: 64Mo We use a server with 16Go of ram it's good conf ? Thanks a lot for your help. 2014-09-10 11:44 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, Here's the version of opensips we use : [root@asbc2b kemathy]# /usr/local/sbin/opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: ff23ed0 main.c compiled on 19:24:42 Jul 23 2014 with gcc 4.4.7 About those points : - the out of mem was reported for pkg memory (and not for shm) == The out of mem was reported for pkg memory - you sent the SIGUSR1 signal to the process complaining about lack of mem == I don't know if I made it well... So, here are two links pointing to other memdumps, taken on processes which seem to lack of mem Sep 10 05:34:05 asbc2b /usr/local/sbin/opensips[920]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 522 Sep 10 05:34:04 asbc2b /usr/local/sbin/opensips[923]: ERROR:core:received_builder: out of pkg memory Sep 10 05:33:57 asbc2b /usr/local/sbin/opensips[937]: ERROR:core:parse_headers: pkg memory allocation failed Sep 10 05:33:57 asbc2b /usr/local/sbin/opensips[937]: ERROR:core:add_lump_rpl: no free pkg memory ! https://www.dropbox.com/s/p62hb16wpk0i86e/memdump_process_920.txt?dl=0 https://www.dropbox.com/s/uim49o6zvj6zvu3/memdump_process_923.txt?dl=0 https://www.dropbox.com/s/c6tksw4j4lgk5d3/memdump_process_937.txt?dl=0 So now, I'm sure that the memdumps I've taken concern processes impacted by the lack of pkg mem... Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-10 10:04 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Kevin, This is stranger as nothing in the dump points to a leak/issue - do you get the out of mem error during a SB query ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 10.09.2014 09:53, Kevin Mathy wrote: Hi Bogdan, Got the same errors on my second opensips server (second node of the cluster). Here is the memdump : https://www.dropbox.com/s/d2gacgrxik3xs1t/memdump_server2_1.txt?dl=0 Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-09 14:56 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, I've got now a memdump ! https://www.dropbox.com/s/5hb678r0ym9vtt1/memdump1.txt?dl=0 And if you need the list of the opensips processes running at the moment : https://www.dropbox.com/s/hqurr3ilamc0a0g/opensips_processes_list.txt?dl=0 I hope it'll be ok... If you need more things, I didn't have restarted the service yet, so I'm able to get other memdumps. Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-09 13:06 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, yes, that's the right way of getting a mem dump during runtime. Also you will automatically get one during the shutdown procedure. Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com -- Cordialement HUBERT Mickaël Ingénieur VOIP - Hexanet ___ Users mailing listUsers@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users -- Cordialement HUBERT Mickaël Ingénieur VOIP - Hexanet -- ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Kevin, yes, that's the right way of getting a mem dump during runtime. Also you will automatically get one during the shutdown procedure. Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 08.09.2014 10:02, Kevin Mathy wrote: Hi Bogdan, Unfortunately, no... I'll try to get that next time it'll happen ! I can do that with kill -SIGUSR1 OPENSIPS_PID, that's right ? Thanks, * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-09-04 20:10 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, Any chance to get the mem dump after waiting ? Thanks Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 04.09.2014 13:09, Kevin Mathy wrote: Hi Bogdan, So, I've faced the problem once again, and followed the Determining the cause part of the link you gave me. I've waited more than 20 minutes (near an hour), and re-sent some voice trafic to the problematic opensips server : memory errors just came after that. So, I think it's a memory leak, no ? Now, what can I do to solve this ? Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, I've got now a memdump ! https://www.dropbox.com/s/5hb678r0ym9vtt1/memdump1.txt?dl=0 And if you need the list of the opensips processes running at the moment : https://www.dropbox.com/s/hqurr3ilamc0a0g/opensips_processes_list.txt?dl=0 I hope it'll be ok... If you need more things, I didn't have restarted the service yet, so I'm able to get other memdumps. Thanks for your help, Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-09-09 13:06 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, yes, that's the right way of getting a mem dump during runtime. Also you will automatically get one during the shutdown procedure. Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com -- ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, So, I've faced the problem once again, and followed the Determining the cause part of the link you gave me. I've waited more than 20 minutes (near an hour), and re-sent some voice trafic to the problematic opensips server : memory errors just came after that. So, I think it's a memory leak, no ? Now, what can I do to solve this ? Thanks for your help, Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-08-20 17:22 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, So it is more a mem leak than a crash. Any chances to get a memdump after memory is exhausted ? See: http://www.opensips.org/Documentation/TroubleShooting-OutOfMem Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 20.08.2014 15:35, Kevin Mathy wrote: Hi Bogdan, In fact, opensips didn't crashed : we got the previous error messages in the logs, but then we switched our voice traffic to another server, and opensips on the 1st one never crashed... We restarted the service in the following night, to be sure to make it rework well. So, is there a way in this case to get some backtrace, even if it didn't crashed ? Maybe something generated at startup ? Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-08-19 20:34 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, Try to extract at least a backtrace, to see what kind of problem we are facing. Then we can decide if a mem dump is required or not. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 19.08.2014 12:03, Kevin Mathy wrote: Hi Bogdan, Coming back to you after some holidays, and unfortunately, new Opensips crashes... Some weeks ago, we upgraded our servers to OpenSIPS 1.11, but we faced a new crash, always with the out of pkg memory error, last week. Here is the exact version : root@asbc2:/home/kemathy# opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: d41eb41 main.c compiled on 19:23:35 Jul 23 2014 with gcc 4.7 Here are an extract of logs we get at the crash-time (I masked phone numbers and IPs for security purposes) : Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contacts: failed to parse contact parameters Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:contact_parser: failed to parse contacts Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contact: failed to parse contact Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:pv_get_contact_body: failed to parse contact hdr Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 526 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:sl:sl_send_reply_helper: response building failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:auth:challenge: failed to send the response Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:core:db_allocate_rows: no memory left Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_convert_result: error while converting rows Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_store_result: error while
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, In fact, opensips didn't crashed : we got the previous error messages in the logs, but then we switched our voice traffic to another server, and opensips on the 1st one never crashed... We restarted the service in the following night, to be sure to make it rework well. So, is there a way in this case to get some backtrace, even if it didn't crashed ? Maybe something generated at startup ? Thanks for your help, Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-08-19 20:34 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, Try to extract at least a backtrace, to see what kind of problem we are facing. Then we can decide if a mem dump is required or not. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 19.08.2014 12:03, Kevin Mathy wrote: Hi Bogdan, Coming back to you after some holidays, and unfortunately, new Opensips crashes... Some weeks ago, we upgraded our servers to OpenSIPS 1.11, but we faced a new crash, always with the out of pkg memory error, last week. Here is the exact version : root@asbc2:/home/kemathy# opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: d41eb41 main.c compiled on 19:23:35 Jul 23 2014 with gcc 4.7 Here are an extract of logs we get at the crash-time (I masked phone numbers and IPs for security purposes) : Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contacts: failed to parse contact parameters Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:contact_parser: failed to parse contacts Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contact: failed to parse contact Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:pv_get_contact_body: failed to parse contact hdr Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 526 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:sl:sl_send_reply_helper: response building failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:auth:challenge: failed to send the response Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:core:db_allocate_rows: no memory left Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_convert_result: error while converting rows Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_store_result: error while converting result Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:core:db_do_query: error while storing result for query [select contact,expires,q,callid,cseq,flags,cflags,user_agent,received,path,socket,methods,last_modified,sip_instance,attr from location where username='0326xx' order by q] Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:usrloc:db_load_urecord: db_query failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: CRITICAL:db_mysql:wrapper_single_mysql_stmt_execute: driver error (1062): Duplicate entry ' 0326xx--sip:0326xxx...@188.94.xx.xxx:5060;transport=udp;line' for key 'account_contact_idx' Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:usrloc:db_insert_ucontact: inserting contact in db failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]:
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Kevin, So it is more a mem leak than a crash. Any chances to get a memdump after memory is exhausted ? See: http://www.opensips.org/Documentation/TroubleShooting-OutOfMem Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 20.08.2014 15:35, Kevin Mathy wrote: Hi Bogdan, In fact, opensips didn't crashed : we got the previous error messages in the logs, but then we switched our voice traffic to another server, and opensips on the 1st one never crashed... We restarted the service in the following night, to be sure to make it rework well. So, is there a way in this case to get some backtrace, even if it didn't crashed ? Maybe something generated at startup ? Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-08-19 20:34 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, Try to extract at least a backtrace, to see what kind of problem we are facing. Then we can decide if a mem dump is required or not. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 19.08.2014 12:03, Kevin Mathy wrote: Hi Bogdan, Coming back to you after some holidays, and unfortunately, new Opensips crashes... Some weeks ago, we upgraded our servers to OpenSIPS 1.11, but we faced a new crash, always with the out of pkg memory error, last week. Here is the exact version : root@asbc2:/home/kemathy# opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: d41eb41 main.c compiled on 19:23:35 Jul 23 2014 with gcc 4.7 Here are an extract of logs we get at the crash-time (I masked phone numbers and IPs for security purposes) : Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contacts: failed to parse contact parameters Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:contact_parser: failed to parse contacts Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contact: failed to parse contact Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:pv_get_contact_body: failed to parse contact hdr Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 526 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:sl:sl_send_reply_helper: response building failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:auth:challenge: failed to send the response Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 mailto:sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 mailto:sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:core:db_allocate_rows: no memory left Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_convert_result: error while converting rows Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_store_result: error while converting result
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, Coming back to you after some holidays, and unfortunately, new Opensips crashes... Some weeks ago, we upgraded our servers to OpenSIPS 1.11, but we faced a new crash, always with the out of pkg memory error, last week. Here is the exact version : root@asbc2:/home/kemathy# opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: d41eb41 main.c compiled on 19:23:35 Jul 23 2014 with gcc 4.7 Here are an extract of logs we get at the crash-time (I masked phone numbers and IPs for security purposes) : Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contacts: failed to parse contact parameters Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:contact_parser: failed to parse contacts Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contact: failed to parse contact Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:pv_get_contact_body: failed to parse contact hdr Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 526 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:sl:sl_send_reply_helper: response building failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:auth:challenge: failed to send the response Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:core:db_allocate_rows: no memory left Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_convert_result: error while converting rows Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_store_result: error while converting result Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:core:db_do_query: error while storing result for query [select contact,expires,q,callid,cseq,flags,cflags,user_agent,received,path,socket,methods,last_modified,sip_instance,attr from location where username='0326xx' order by q] Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:usrloc:db_load_urecord: db_query failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: CRITICAL:db_mysql:wrapper_single_mysql_stmt_execute: driver error (1062): Duplicate entry '0326xx--sip:0326xxx...@188.94.xx.xxx:5060;transport=udp;line' for key 'account_contact_idx' Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:usrloc:db_insert_ucontact: inserting contact in db failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:usrloc:insert_ucontact: failed to insert in database Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 373 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: ERROR:sl:sl_send_reply_helper: response building failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: ERROR:options:opt_reply: failed to send 200 via send_reply Unfortunately, we don't have the memory dump :-s ... So, I don't know if there's something we can try to avoid those errors to happen again; maybe we made something wrong in our configuration file ? We could provide you anything you want to help us find where's the problem. Thanks a lot for your help, Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-07-08 8:43 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, Sounds really good :-) !
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Kevin, Try to extract at least a backtrace, to see what kind of problem we are facing. Then we can decide if a mem dump is required or not. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 19.08.2014 12:03, Kevin Mathy wrote: Hi Bogdan, Coming back to you after some holidays, and unfortunately, new Opensips crashes... Some weeks ago, we upgraded our servers to OpenSIPS 1.11, but we faced a new crash, always with the out of pkg memory error, last week. Here is the exact version : root@asbc2:/home/kemathy# opensips -V version: opensips 1.11.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. git revision: d41eb41 main.c compiled on 19:23:35 Jul 23 2014 with gcc 4.7 Here are an extract of logs we get at the crash-time (I masked phone numbers and IPs for security purposes) : Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contacts: failed to parse contact parameters Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:contact_parser: failed to parse contacts Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:parse_contact: failed to parse contact Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:pv_get_contact_body: failed to parse contact hdr Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 526 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:sl:sl_send_reply_helper: response building failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: ERROR:auth:challenge: failed to send the response Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: iy4l7y1r@pbx -- New REGISTER received from 188.94.XXX.XXX with Contact : sip:0326xxx...@188.94.xxx.xxx:5060;transport=udp;line=6364d3f0 -- IP 188.94.XXX.XXX -- PORT 5060 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:core:db_allocate_rows: no memory left Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_convert_result: error while converting rows Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:db_mysql:db_mysql_store_result: error while converting result Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:core:db_do_query: error while storing result for query [select contact,expires,q,callid,cseq,flags,cflags,user_agent,received,path,socket,methods,last_modified,sip_instance,attr from location where username='0326xx' order by q] Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:usrloc:db_load_urecord: db_query failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: CRITICAL:db_mysql:wrapper_single_mysql_stmt_execute: driver error (1062): Duplicate entry '0326xx--sip:0326xxx...@188.94.xx.xxx:5060;transport=udp;line' for key 'account_contact_idx' Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:usrloc:db_insert_ucontact: inserting contact in db failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: ERROR:usrloc:insert_ucontact: failed to insert in database Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 373 Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: ERROR:sl:sl_send_reply_helper: response building failed Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]:
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, We've recompiled OpenSIPS with requested modules (I think) : root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, *DBG_QM_MALLOC*, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 22:12:10 Jul 3 2014 with gcc 4.7 Now we'll keep an eye on our server to check if everything is OK, and if the memory error still occur; as we upgraded from 1.9.1 to 1.9.2... I'll get back to you with some logs if needed ;-) Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-07-02 9:37 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, Hummm, right, opensips doesn't seem to have been compiled with the requested modules for memory debugging... root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 So I think I'll have to re-compile opensips with QM_DBG_MALLOC, and try again to export the memdump log... I'll get back to you when done. Thanks a lot for your help ! Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, Unfortunately the logs are not correct - are you sure you properly compiled the mem debug ? like adding the QM_DBG_MALLOC and removing the FM_MALLOC flags ? As the logs show the standard memory manager (without debugging). Check it with opensips -V to see the list of compiled flags. I tried to get some ideas by only looking at the available memory and how many fragments were allocated in each process - indeed, there are some processes using maybe like 2 or 3 times more PKG, but not sure if a leak. Getting the proper logs (which will be huge) will tell us more. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 01.07.2014 18:11, Kevin Mathy wrote: Hi Bogdan, I have now a memdump log, as we restarted opensips this afternoon for a configuration maintenance... But the file is too big, even if I try to put it to pastebin.com ... So, here is the file; I don't want to give the link on the mailing-list :-) [removed] I hope this will help understanding our problem's cause :-) ... Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-30 16:30 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, Ooops, I thought my two first mails have been cancelled :-) I prefer waiting till there's no traffic, so I'll send a SIGUSR1 comme this evening, and reply to this topic with the log. I'll try working with MI statistics to make some memory usage graphs better than with Cacti... I'll come back to you with logs; thanks for all ! Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, There is no need to send your email three times ;). One time is enough. Waiting and taking the dump when there is not traffic is good (but not a must) - the idea is to be sure that all temporary memory (used for processing the current traffic) was freed - so what you still have in memory is configuration data or possible leaks. If you do not have the luxury of waiting, you can do it whenever you can. Once again, do not look at the memory usage reported by OS - it is irrelevant as OpenSIPS is doing its own internal memory management. Check the memory usage via MI, see the mem related statistics: http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11 Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 30.06.2014 12:01, Kevin Mathy wrote: Hi Bogdan, If I want to send a SIGUSR1, may I have to wait 20 minutes after the last call ? 20 minutes without any call ? I don't understand well this sentence : It is highly recommended to do this after waiting about 20 minutes to be sure that as much as possile memory is freed - all temporary memory used during processing is freed by lack of load on the proxy Also, last week-end, the traffic reduced a lot, and between last friday, when the free system's memory was around 170M, and this morning, the free memory seems to have increased : this
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Kevin, Indeed, it looks compiled ok now (with the memory debugger). Keep in mind to set (in this order): mem_log=4 mem_dump=1 to avoid runtime logs, but to get the mem dumps. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 04.07.2014 12:41, Kevin Mathy wrote: Hi Bogdan, We've recompiled OpenSIPS with requested modules (I think) : root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, *DBG_QM_MALLOC*, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 22:12:10 Jul 3 2014 with gcc 4.7 Now we'll keep an eye on our server to check if everything is OK, and if the memory error still occur; as we upgraded from 1.9.1 to 1.9.2... I'll get back to you with some logs if needed ;-) Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-07-02 9:37 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr mailto:k.ma...@hexanet.fr: Hi Bogdan, Hummm, right, opensips doesn't seem to have been compiled with the requested modules for memory debugging... root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 So I think I'll have to re-compile opensips with QM_DBG_MALLOC, and try again to export the memdump log... I'll get back to you when done. Thanks a lot for your help ! Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, Unfortunately the logs are not correct - are you sure you properly compiled the mem debug ? like adding the QM_DBG_MALLOC and removing the FM_MALLOC flags ? As the logs show the standard memory manager (without debugging). Check it with opensips -V to see the list of compiled flags. I tried to get some ideas by only looking at the available memory and how many fragments were allocated in each process - indeed, there are some processes using maybe like 2 or 3 times more PKG, but not sure if a leak. Getting the proper logs (which will be huge) will tell us more. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 01.07.2014 18:11, Kevin Mathy wrote: Hi Bogdan, I have now a memdump log, as we restarted opensips this afternoon for a configuration maintenance... But the file is too big, even if I try to put it to pastebin.com http://pastebin.com ... So, here is the file; I don't want to give the link on the mailing-list :-) [removed] I hope this will help understanding our problem's cause :-) ... Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-30 16:30 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr mailto:k.ma...@hexanet.fr: Hi Bogdan, Ooops, I thought my two first mails have been cancelled :-) I prefer waiting till there's no traffic, so I'll send a SIGUSR1 comme this evening, and reply to this topic with the log. I'll try working with MI statistics to make some memory usage graphs better than with Cacti... I'll come back to you with logs; thanks for all ! Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, There is no need to send your email three times ;). One time is enough. Waiting and taking the dump when there is not traffic is good (but not a must) - the idea is to be sure that all temporary memory (used for processing the
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, Hummm, right, opensips doesn't seem to have been compiled with the requested modules for memory debugging... root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 So I think I'll have to re-compile opensips with QM_DBG_MALLOC, and try again to export the memdump log... I'll get back to you when done. Thanks a lot for your help ! Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, Unfortunately the logs are not correct - are you sure you properly compiled the mem debug ? like adding the QM_DBG_MALLOC and removing the FM_MALLOC flags ? As the logs show the standard memory manager (without debugging). Check it with opensips -V to see the list of compiled flags. I tried to get some ideas by only looking at the available memory and how many fragments were allocated in each process - indeed, there are some processes using maybe like 2 or 3 times more PKG, but not sure if a leak. Getting the proper logs (which will be huge) will tell us more. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 01.07.2014 18:11, Kevin Mathy wrote: Hi Bogdan, I have now a memdump log, as we restarted opensips this afternoon for a configuration maintenance... But the file is too big, even if I try to put it to pastebin.com ... So, here is the file; I don't want to give the link on the mailing-list :-) [removed] I hope this will help understanding our problem's cause :-) ... Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-30 16:30 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr: Hi Bogdan, Ooops, I thought my two first mails have been cancelled :-) I prefer waiting till there's no traffic, so I'll send a SIGUSR1 comme this evening, and reply to this topic with the log. I'll try working with MI statistics to make some memory usage graphs better than with Cacti... I'll come back to you with logs; thanks for all ! Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, There is no need to send your email three times ;). One time is enough. Waiting and taking the dump when there is not traffic is good (but not a must) - the idea is to be sure that all temporary memory (used for processing the current traffic) was freed - so what you still have in memory is configuration data or possible leaks. If you do not have the luxury of waiting, you can do it whenever you can. Once again, do not look at the memory usage reported by OS - it is irrelevant as OpenSIPS is doing its own internal memory management. Check the memory usage via MI, see the mem related statistics: http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11 Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 30.06.2014 12:01, Kevin Mathy wrote: Hi Bogdan, If I want to send a SIGUSR1, may I have to wait 20 minutes after the last call ? 20 minutes without any call ? I don't understand well this sentence : It is highly recommended to do this after waiting about 20 minutes to be sure that as much as possile memory is freed - all temporary memory used during processing is freed by lack of load on the proxy Also, last week-end, the traffic reduced a lot, and between last friday, when the free system's memory was around 170M, and this morning, the free memory seems to have increased : this morning, it was around 700M, before the traffic comes back. So, opensips seems to well free the memory, isn't it ? Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, There is no need to wait for a crash. From time to time, you can send a SIGUSR1 to a worker process (or a process you suspect as running out of pkg mem) - the process will do a pkg dump to the log. Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9 is no longer maintained and there were some fixes in the memory manager since then. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 27.06.2014 10 27.06.2014%2010:36, Kevin Mathy wrote: Hi Bogdan, I've set given options, and now I'm waiting for a new crash of
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Kevin, Unfortunately the logs are not correct - are you sure you properly compiled the mem debug ? like adding the QM_DBG_MALLOC and removing the FM_MALLOC flags ? As the logs show the standard memory manager (without debugging). Check it with opensips -V to see the list of compiled flags. I tried to get some ideas by only looking at the available memory and how many fragments were allocated in each process - indeed, there are some processes using maybe like 2 or 3 times more PKG, but not sure if a leak. Getting the proper logs (which will be huge) will tell us more. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 01.07.2014 18:11, Kevin Mathy wrote: Hi Bogdan, I have now a memdump log, as we restarted opensips this afternoon for a configuration maintenance... But the file is too big, even if I try to put it to pastebin.com http://pastebin.com ... So, here is the file; I don't want to give the link on the mailing-list :-) [removed] I hope this will help understanding our problem's cause :-) ... Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-30 16:30 GMT+02:00 Kevin Mathy k.ma...@hexanet.fr mailto:k.ma...@hexanet.fr: Hi Bogdan, Ooops, I thought my two first mails have been cancelled :-) I prefer waiting till there's no traffic, so I'll send a SIGUSR1 comme this evening, and reply to this topic with the log. I'll try working with MI statistics to make some memory usage graphs better than with Cacti... I'll come back to you with logs; thanks for all ! Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, There is no need to send your email three times ;). One time is enough. Waiting and taking the dump when there is not traffic is good (but not a must) - the idea is to be sure that all temporary memory (used for processing the current traffic) was freed - so what you still have in memory is configuration data or possible leaks. If you do not have the luxury of waiting, you can do it whenever you can. Once again, do not look at the memory usage reported by OS - it is irrelevant as OpenSIPS is doing its own internal memory management. Check the memory usage via MI, see the mem related statistics: http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11 Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 30.06.2014 12:01, Kevin Mathy wrote: Hi Bogdan, If I want to send a SIGUSR1, may I have to wait 20 minutes after the last call ? 20 minutes without any call ? I don't understand well this sentence : It is highly recommended to do this after waiting about 20 minutes to be sure that as much as possile memory is freed - all temporary memory used during processing is freed by lack of load on the proxy Also, last week-end, the traffic reduced a lot, and between last friday, when the free system's memory was around 170M, and this morning, the free memory seems to have increased : this morning, it was around 700M, before the traffic comes back. So, opensips seems to well free the memory, isn't it ? Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, There is no need to wait for a crash. From time to time, you can send a SIGUSR1 to a worker process (or a process you suspect as running out of pkg mem) - the process will do a pkg dump to the log. Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9 is no longer maintained and there were some fixes in the memory manager since then. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 27.06.2014 10 tel:27.06.2014%2010:36, Kevin Mathy wrote: Hi Bogdan, I've set given options, and now I'm waiting for a new crash of the service... Where the memdump will be located ? In another logfile than opensips.log, or in the same ? Thanks * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP *
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, If I want to send a SIGUSR1, may I have to wait 20 minutes after the last call ? 20 minutes without any call ? I don't understand well this sentence : It is highly recommended to do this after waiting about 20 minutes to be sure that as much as possile memory is freed - all temporary memory used during processing is freed by lack of load on the proxy Also, last week-end, the traffic reduced a lot, and between last friday, when the free system's memory was around 170M, and this morning, the free memory seems to have increased : this morning, it was around 700M, before the traffic comes back. So, opensips seems to well free the memory, isn't it ? Thanks for your help, Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, There is no need to wait for a crash. From time to time, you can send a SIGUSR1 to a worker process (or a process you suspect as running out of pkg mem) - the process will do a pkg dump to the log. Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9 is no longer maintained and there were some fixes in the memory manager since then. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 27.06.2014 10:36, Kevin Mathy wrote: Hi Bogdan, I've set given options, and now I'm waiting for a new crash of the service... Where the memdump will be located ? In another logfile than opensips.log, or in the same ? Thanks * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Kevin, Restarting should not make you loose ongoing calls (even if you use the dialog module), do do not worry on that. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com -- ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Kevin, There is no need to send your email three times ;). One time is enough. Waiting and taking the dump when there is not traffic is good (but not a must) - the idea is to be sure that all temporary memory (used for processing the current traffic) was freed - so what you still have in memory is configuration data or possible leaks. If you do not have the luxury of waiting, you can do it whenever you can. Once again, do not look at the memory usage reported by OS - it is irrelevant as OpenSIPS is doing its own internal memory management. Check the memory usage via MI, see the mem related statistics: http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11 Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 30.06.2014 12:01, Kevin Mathy wrote: Hi Bogdan, If I want to send a SIGUSR1, may I have to wait 20 minutes after the last call ? 20 minutes without any call ? I don't understand well this sentence : It is highly recommended to do this after waiting about 20 minutes to be sure that as much as possile memory is freed - all temporary memory used during processing is freed by lack of load on the proxy Also, last week-end, the traffic reduced a lot, and between last friday, when the free system's memory was around 170M, and this morning, the free memory seems to have increased : this morning, it was around 700M, before the traffic comes back. So, opensips seems to well free the memory, isn't it ? Thanks for your help, Kevin * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, There is no need to wait for a crash. From time to time, you can send a SIGUSR1 to a worker process (or a process you suspect as running out of pkg mem) - the process will do a pkg dump to the log. Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9 is no longer maintained and there were some fixes in the memory manager since then. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 27.06.2014 10 tel:27.06.2014%2010:36, Kevin Mathy wrote: Hi Bogdan, I've set given options, and now I'm waiting for a new crash of the service... Where the memdump will be located ? In another logfile than opensips.log, or in the same ? Thanks * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Kevin, Restarting should not make you loose ongoing calls (even if you use the dialog module), do do not worry on that. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, I've set given options, and now I'm waiting for a new crash of the service... Where the memdump will be located ? In another logfile than opensips.log, or in the same ? Thanks *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Kevin, Restarting should not make you loose ongoing calls (even if you use the dialog module), do do not worry on that. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 26.06.2014 18:31, Kevin Mathy wrote: Yes, I'll set those options, and wait 'till the trafic on the server reduces a lot (tonight I think) to restart opensips. Unless if you have another method to reload the config without losing calls ? Thanks a lot, * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-26 17:20 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Kevin, The trick is to set (in this order): memlog = 4 memdump = 1 This will do no logging at runtime, but it will log the mem dump (at shutdown or at signal). Could you try that ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 26.06.2014 18:09, Kevin Mathy wrote: Hi Bogdan, We already have compiled opensips as described, and set memlog=1 in the configuration file. But as we had a lot of debug messages about memory usage, we unset memlog=1 in the config, so we don't have any report now. Is there a way to get those messages in a different log file, instead of inside the classical opensips.log ? I've found an option like memdump that can be set in opensips.cfg, but what about it ? And as cacti doesn't seems to be relevant to check opensips's mem usage, do you have any other solution to check it ? Thanks for your help, * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-26 16:46 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, For debugging potential mem leaks, please look into: http://www.opensips.org/Documentation/TroubleShooting-OutOfMem If you need help with that, let me know. Also, cacti is monitoring the system memory which is irrelevant for OpenSIPS (as OpenSIPS is pre-allocating - from OS - all the needed memory at startup). Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 26.06.2014 15:04, Kevin Mathy wrote: Hi List, I'm facing a memory usage issue with OpenSIPS 1.9.2 on a Debian 7 server root@asbc2:/home/kemathy# uname -a Linux asbc2 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 The server handles something like 7 to 9 calls per day, with around 20 to 40 Calls Per Second As you can see on the following Cacti graphs, system's free memory decreases constantly to finally reach the lower value of 160M : https://www.dropbox.com/s/np3fnc6ni4vuoet/ASBC2_Memory_Usage_Evolution.jpg The system can stay 2-3 days working in this state, but for example, yesterday evening (2014-06-25 18:50:00), opensips crashed, after showing those log messages : Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:received_builder: out of pkg memory Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:add_lump_rpl: no free pkg memory ! Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:db_allocate_rows: no memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Kevin, There is no need to wait for a crash. From time to time, you can send a SIGUSR1 to a worker process (or a process you suspect as running out of pkg mem) - the process will do a pkg dump to the log. Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9 is no longer maintained and there were some fixes in the memory manager since then. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 27.06.2014 10:36, Kevin Mathy wrote: Hi Bogdan, I've set given options, and now I'm waiting for a new crash of the service... Where the memdump will be located ? In another logfile than opensips.log, or in the same ? Thanks * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Kevin, Restarting should not make you loose ongoing calls (even if you use the dialog module), do do not worry on that. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 26.06.2014 18 tel:26.06.2014%2018:31, Kevin Mathy wrote: Yes, I'll set those options, and wait 'till the trafic on the server reduces a lot (tonight I think) to restart opensips. Unless if you have another method to reload the config without losing calls ? Thanks a lot, * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-26 17:20 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Kevin, The trick is to set (in this order): memlog = 4 memdump = 1 This will do no logging at runtime, but it will log the mem dump (at shutdown or at signal). Could you try that ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 26.06.2014 18 tel:26.06.2014%2018:09, Kevin Mathy wrote: Hi Bogdan, We already have compiled opensips as described, and set memlog=1 in the configuration file. But as we had a lot of debug messages about memory usage, we unset memlog=1 in the config, so we don't have any report now. Is there a way to get those messages in a different log file, instead of inside the classical opensips.log ? I've found an option like memdump that can be set in opensips.cfg, but what about it ? And as cacti doesn't seems to be relevant to check opensips's mem usage, do you have any other solution to check it ? Thanks for your help, * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-26 16:46 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, For debugging potential mem leaks, please look into: http://www.opensips.org/Documentation/TroubleShooting-OutOfMem If you need help with that, let me know. Also, cacti is monitoring the system memory which is irrelevant for OpenSIPS (as OpenSIPS is pre-allocating - from OS - all the needed memory at startup). Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 26.06.2014 15 tel:26.06.2014%2015:04, Kevin Mathy wrote: Hi List, I'm facing a memory usage issue with OpenSIPS 1.9.2 on a Debian 7 server root@asbc2:/home/kemathy# uname -a Linux asbc2 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 The server handles something like 7 to 9 calls per day, with around 20 to 40 Calls Per Second As you can see on the following Cacti graphs, system's free memory decreases constantly to finally reach the lower value of 160M : https://www.dropbox.com/s/np3fnc6ni4vuoet/ASBC2_Memory_Usage_Evolution.jpg The system can stay 2-3 days
[OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi List, I'm facing a memory usage issue with OpenSIPS 1.9.2 on a Debian 7 server root@asbc2:/home/kemathy# uname -a Linux asbc2 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 The server handles something like 7 to 9 calls per day, with around 20 to 40 Calls Per Second As you can see on the following Cacti graphs, system's free memory decreases constantly to finally reach the lower value of 160M : https://www.dropbox.com/s/np3fnc6ni4vuoet/ASBC2_Memory_Usage_Evolution.jpg The system can stay 2-3 days working in this state, but for example, yesterday evening (2014-06-25 18:50:00), opensips crashed, after showing those log messages : Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:received_builder: out of pkg memory Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:add_lump_rpl: no free pkg memory ! Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:db_allocate_rows: no memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:add_lump_rpl: no free pkg memory ! And after the service's restart, everything was OK : Jun 25 19:12:39 asbc2 /usr/local/sbin/opensips[29351]: INFO:core:main: using 64 Mb shared memory Jun 25 19:12:39 asbc2 /usr/local/sbin/opensips[29351]: INFO:core:main: using 4 Mb private memory per process For information, here are the memory values at this moment; opensips has handled 5000 calls since his bootup : root@asbc2:/home/kemathy# free -m total used free sharedbuffers cached Mem: 16082 12887 3195 0339 11393 -/+ buffers/cache: 1154 14928 Swap:32755 0 32755 For now, OpenSIPS is launched with S_MEMORY = 64 and P_MEMORY = 4 My first idea would be to increase the P_MEMORY value to 8 for example, as it seems to be a lack of PKG Memory, but in case it's a memory-freeing problem, it'll not solve our issue (I think). My questions are : How can we calculate the appropriate value for S_MEMORY and P_MEMORY ? And how can we solve our out of pkg memory problem ? If you need further informations, or anything else, feel free to ask me ! Thanks a lot for your help, *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP -- ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Kevin, For debugging potential mem leaks, please look into: http://www.opensips.org/Documentation/TroubleShooting-OutOfMem If you need help with that, let me know. Also, cacti is monitoring the system memory which is irrelevant for OpenSIPS (as OpenSIPS is pre-allocating - from OS - all the needed memory at startup). Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 26.06.2014 15:04, Kevin Mathy wrote: Hi List, I'm facing a memory usage issue with OpenSIPS 1.9.2 on a Debian 7 server root@asbc2:/home/kemathy# uname -a Linux asbc2 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 The server handles something like 7 to 9 calls per day, with around 20 to 40 Calls Per Second As you can see on the following Cacti graphs, system's free memory decreases constantly to finally reach the lower value of 160M : https://www.dropbox.com/s/np3fnc6ni4vuoet/ASBC2_Memory_Usage_Evolution.jpg The system can stay 2-3 days working in this state, but for example, yesterday evening (2014-06-25 18:50:00), opensips crashed, after showing those log messages : Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:received_builder: out of pkg memory Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:add_lump_rpl: no free pkg memory ! Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:db_allocate_rows: no memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:add_lump_rpl: no free pkg memory ! And after the service's restart, everything was OK : Jun 25 19:12:39 asbc2 /usr/local/sbin/opensips[29351]: INFO:core:main: using 64 Mb shared memory Jun 25 19:12:39 asbc2 /usr/local/sbin/opensips[29351]: INFO:core:main: using 4 Mb private memory per process For information, here are the memory values at this moment; opensips has handled 5000 calls since his bootup : root@asbc2:/home/kemathy# free -m total used free shared buffers cached Mem: 16082 12887 3195 0 339 11393 -/+ buffers/cache: 1154 14928 Swap:32755 0 32755 For now, OpenSIPS is launched with S_MEMORY = 64 and P_MEMORY = 4 My first idea would be to increase the P_MEMORY value to 8 for example, as it seems to be a lack of PKG Memory, but in case it's a memory-freeing problem, it'll not solve our issue (I think). My questions are : How can we calculate the appropriate value for S_MEMORY and P_MEMORY ? And how can we solve our out of pkg memory problem ? If you need further informations, or anything else, feel free to ask me ! Thanks a lot for your help, * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Hi Bogdan, We already have compiled opensips as described, and set memlog=1 in the configuration file. But as we had a lot of debug messages about memory usage, we unset memlog=1 in the config, so we don't have any report now. Is there a way to get those messages in a different log file, instead of inside the classical opensips.log ? I've found an option like memdump that can be set in opensips.cfg, but what about it ? And as cacti doesn't seems to be relevant to check opensips's mem usage, do you have any other solution to check it ? Thanks for your help, *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-26 16:46 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, For debugging potential mem leaks, please look into: http://www.opensips.org/Documentation/TroubleShooting-OutOfMem If you need help with that, let me know. Also, cacti is monitoring the system memory which is irrelevant for OpenSIPS (as OpenSIPS is pre-allocating - from OS - all the needed memory at startup). Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 26.06.2014 15:04, Kevin Mathy wrote: Hi List, I'm facing a memory usage issue with OpenSIPS 1.9.2 on a Debian 7 server root@asbc2:/home/kemathy# uname -a Linux asbc2 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 The server handles something like 7 to 9 calls per day, with around 20 to 40 Calls Per Second As you can see on the following Cacti graphs, system's free memory decreases constantly to finally reach the lower value of 160M : https://www.dropbox.com/s/np3fnc6ni4vuoet/ASBC2_Memory_Usage_Evolution.jpg The system can stay 2-3 days working in this state, but for example, yesterday evening (2014-06-25 18:50:00), opensips crashed, after showing those log messages : Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:received_builder: out of pkg memory Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:add_lump_rpl: no free pkg memory ! Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:db_allocate_rows: no memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:add_lump_rpl: no free pkg memory ! And after the service's restart, everything was OK : Jun 25 19:12:39 asbc2 /usr/local/sbin/opensips[29351]: INFO:core:main: using 64 Mb shared memory Jun 25 19:12:39 asbc2 /usr/local/sbin/opensips[29351]: INFO:core:main: using 4 Mb private memory per process For information, here are the memory values at this moment; opensips has handled 5000 calls since his bootup : root@asbc2:/home/kemathy# free -m total used free sharedbuffers cached Mem: 16082 12887 3195 0339 11393 -/+ buffers/cache: 1154 14928 Swap:32755 0 32755 For now, OpenSIPS is launched with S_MEMORY = 64 and P_MEMORY = 4 My first idea would be to increase the P_MEMORY value to 8 for example, as it seems to be a lack of PKG Memory, but in case it's a memory-freeing problem, it'll not solve our issue (I think). My questions are : How can we calculate the appropriate value for S_MEMORY and P_MEMORY ? And how can we solve our out of pkg memory problem ? If you need further informations, or anything else, feel free to ask me ! Thanks a lot for your help, * Bien cordialement, Best Regards, **Kevin
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Kevin, The trick is to set (in this order): memlog = 4 memdump = 1 This will do no logging at runtime, but it will log the mem dump (at shutdown or at signal). Could you try that ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 26.06.2014 18:09, Kevin Mathy wrote: Hi Bogdan, We already have compiled opensips as described, and set memlog=1 in the configuration file. But as we had a lot of debug messages about memory usage, we unset memlog=1 in the config, so we don't have any report now. Is there a way to get those messages in a different log file, instead of inside the classical opensips.log ? I've found an option like memdump that can be set in opensips.cfg, but what about it ? And as cacti doesn't seems to be relevant to check opensips's mem usage, do you have any other solution to check it ? Thanks for your help, * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-26 16:46 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, For debugging potential mem leaks, please look into: http://www.opensips.org/Documentation/TroubleShooting-OutOfMem If you need help with that, let me know. Also, cacti is monitoring the system memory which is irrelevant for OpenSIPS (as OpenSIPS is pre-allocating - from OS - all the needed memory at startup). Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 26.06.2014 15 tel:26.06.2014%2015:04, Kevin Mathy wrote: Hi List, I'm facing a memory usage issue with OpenSIPS 1.9.2 on a Debian 7 server root@asbc2:/home/kemathy# uname -a Linux asbc2 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 The server handles something like 7 to 9 calls per day, with around 20 to 40 Calls Per Second As you can see on the following Cacti graphs, system's free memory decreases constantly to finally reach the lower value of 160M : https://www.dropbox.com/s/np3fnc6ni4vuoet/ASBC2_Memory_Usage_Evolution.jpg The system can stay 2-3 days working in this state, but for example, yesterday evening (2014-06-25 18:50:00), opensips crashed, after showing those log messages : Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:received_builder: out of pkg memory Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:add_lump_rpl: no free pkg memory ! Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:db_allocate_rows: no memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:add_lump_rpl: no free pkg memory ! And after the service's restart, everything was OK : Jun 25 19:12:39 asbc2 /usr/local/sbin/opensips[29351]: INFO:core:main: using 64 Mb shared memory Jun 25 19:12:39 asbc2 /usr/local/sbin/opensips[29351]: INFO:core:main: using 4 Mb private memory per process For information, here are the memory values at this moment; opensips has handled 5000 calls since his bootup : root@asbc2:/home/kemathy# free
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Yes, I'll set those options, and wait 'till the trafic on the server reduces a lot (tonight I think) to restart opensips. Unless if you have another method to reload the config without losing calls ? Thanks a lot, *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-26 17:20 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Kevin, The trick is to set (in this order): memlog = 4 memdump = 1 This will do no logging at runtime, but it will log the mem dump (at shutdown or at signal). Could you try that ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 26.06.2014 18:09, Kevin Mathy wrote: Hi Bogdan, We already have compiled opensips as described, and set memlog=1 in the configuration file. But as we had a lot of debug messages about memory usage, we unset memlog=1 in the config, so we don't have any report now. Is there a way to get those messages in a different log file, instead of inside the classical opensips.log ? I've found an option like memdump that can be set in opensips.cfg, but what about it ? And as cacti doesn't seems to be relevant to check opensips's mem usage, do you have any other solution to check it ? Thanks for your help, * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-06-26 16:46 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org: Hi Kevin, For debugging potential mem leaks, please look into: http://www.opensips.org/Documentation/TroubleShooting-OutOfMem If you need help with that, let me know. Also, cacti is monitoring the system memory which is irrelevant for OpenSIPS (as OpenSIPS is pre-allocating - from OS - all the needed memory at startup). Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developerhttp://www.opensips-solutions.com On 26.06.2014 15:04, Kevin Mathy wrote: Hi List, I'm facing a memory usage issue with OpenSIPS 1.9.2 on a Debian 7 server root@asbc2:/home/kemathy# uname -a Linux asbc2 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 The server handles something like 7 to 9 calls per day, with around 20 to 40 Calls Per Second As you can see on the following Cacti graphs, system's free memory decreases constantly to finally reach the lower value of 160M : https://www.dropbox.com/s/np3fnc6ni4vuoet/ASBC2_Memory_Usage_Evolution.jpg The system can stay 2-3 days working in this state, but for example, yesterday evening (2014-06-25 18:50:00), opensips crashed, after showing those log messages : Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:received_builder: out of pkg memory Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:add_lump_rpl: no free pkg memory ! Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:db_allocate_rows: no memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11356]: ERROR:core:add_lump_rpl: no free pkg memory ! And after the service's restart, everything was OK : Jun 25 19:12:39 asbc2 /usr/local/sbin/opensips[29351]: INFO:core:main: using 64 Mb shared memory Jun 25 19:12:39 asbc2 /usr/local/sbin/opensips[29351]: INFO:core:main: using 4 Mb private memory per process For information, here are the memory values at this moment; opensips has handled 5000 calls since his bootup : root@asbc2:/home/kemathy# free -m total used free sharedbuffers cached Mem:
Re: [OpenSIPS-Users] OpenSIPS crashed after out of pkg memory
Kevin, Restarting should not make you loose ongoing calls (even if you use the dialog module), do do not worry on that. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 26.06.2014 18:31, Kevin Mathy wrote: Yes, I'll set those options, and wait 'till the trafic on the server reduces a lot (tonight I think) to restart opensips. Unless if you have another method to reload the config without losing calls ? Thanks a lot, * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-26 17:20 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Kevin, The trick is to set (in this order): memlog = 4 memdump = 1 This will do no logging at runtime, but it will log the mem dump (at shutdown or at signal). Could you try that ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 26.06.2014 18 tel:26.06.2014%2018:09, Kevin Mathy wrote: Hi Bogdan, We already have compiled opensips as described, and set memlog=1 in the configuration file. But as we had a lot of debug messages about memory usage, we unset memlog=1 in the config, so we don't have any report now. Is there a way to get those messages in a different log file, instead of inside the classical opensips.log ? I've found an option like memdump that can be set in opensips.cfg, but what about it ? And as cacti doesn't seems to be relevant to check opensips's mem usage, do you have any other solution to check it ? Thanks for your help, * Bien cordialement, Best Regards, **Kevin MATHY* |**Ingénieur VoIP * * 2014-06-26 16:46 GMT+02:00 Bogdan-Andrei Iancu bog...@opensips.org mailto:bog...@opensips.org: Hi Kevin, For debugging potential mem leaks, please look into: http://www.opensips.org/Documentation/TroubleShooting-OutOfMem If you need help with that, let me know. Also, cacti is monitoring the system memory which is irrelevant for OpenSIPS (as OpenSIPS is pre-allocating - from OS - all the needed memory at startup). Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 26.06.2014 15 tel:26.06.2014%2015:04, Kevin Mathy wrote: Hi List, I'm facing a memory usage issue with OpenSIPS 1.9.2 on a Debian 7 server root@asbc2:/home/kemathy# uname -a Linux asbc2 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux root@asbc2:/home/kemathy# opensips -V version: opensips 1.9.2-notls (x86_64/linux) flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: unknown @(#) $Id$ main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 The server handles something like 7 to 9 calls per day, with around 20 to 40 Calls Per Second As you can see on the following Cacti graphs, system's free memory decreases constantly to finally reach the lower value of 160M : https://www.dropbox.com/s/np3fnc6ni4vuoet/ASBC2_Memory_Usage_Evolution.jpg The system can stay 2-3 days working in this state, but for example, yesterday evening (2014-06-25 18:50:00), opensips crashed, after showing those log messages : Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11390]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 412 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11380]: ERROR:core:received_builder: out of pkg memory Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11364]: ERROR:core:add_lump_rpl: no free pkg memory ! Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]: ERROR:core:build_res_buf_from_sip_req: out of pkg memory ; needs 355 Jun 25 18:51:18 asbc2 /usr/local/sbin/opensips[11362]:
[OpenSIPS-Users] OpenSIPS Crashed!!
Hi, Right now I am using OpenSIPS 1.5.3 no-tls version in production. Suddenly OpenSIPS got crashed. I did check coredump but can not understand it. Please help me out to interpret it. Here is the pastebin link which contains output of bt. http://pastebin.com/m49520853 Thanks in advance!!! -- VoipExpert ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Opensips crashed - problem with MD5?
Hi Carlo, The backtrace looks really broken.it might be a core overwrite. What revision on 1.5.0 are you using (do opensips -V) ? there are several mysql related bugs fixed since the 1.5.0 release. If you have a SVN copy, do a SVN update to get the latest fixes. Otherwise I recomand upgrading to 1.5.1 (see the download section on the web site). Regards, Bogdan Carlo Dimaggio wrote: Hi all, My opensips 1.5.0 has crashed twice in the last 5 days. It had low load (few registrations)... In opensipsctlrc I have: MD5=md5sum STORE_PLAINTEXT_PW=0 in opensips.cfg: modparam(auth_db, calculate_ha1, 0) modparam(auth_db, password_column, ha1) modparam(auth_db, password_column_2, ha1b) Below the backtrace. Is it a bug? Thank you! Carlo Dimaggio -- ca...@sip:~$ sudo gdb opensips core.opensips.sig11.22690 GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type show copying and show warranty for details. This GDB was configured as i486-linux-gnu... Cannot access memory at address 0xf830e900 (gdb) bt #0 0xb794781f in ?? () #1 0xb7af3894 in ?? () #2 0xb7af5c8b in ?? () #3 0xb7aebb79 in ?? () #4 0x08056ee2 in do_action (a=0x81d2b90, msg=0x8203db8) at action.c:426 #5 0x0805a38d in destroy_black_lists () at mem/../fastlock.h:88 #6 0x080aac21 in mk_action (type=136129528, n=136330680, elem=0x0, line=0) at route_struct.c:124 #7 0x080aa9c4 in mk_action (type=136129568, n=136330680, elem=0x0, line=3) at route_struct.c:113 #8 0x080aa9ec in mk_action (type=136129608, n=136330680, elem=0x0, line=0) at route_struct.c:113 #9 0x080570de in do_action (a=0x81d2ed8, msg=0x8203db8) at action.c:674 #10 0x0805a38d in destroy_black_lists () at mem/../fastlock.h:88 #11 0x08059013 in do_action (a=0x81d4710, msg=0x8203db8) at parser/../ ip_addr.h:385 #12 0x0805a38d in destroy_black_lists () at mem/../fastlock.h:88 #13 0x0805840d in do_action (a=0x81d1d68, msg=0x8203db8) at parser/../ ip_addr.h:378 #14 0x0805a38d in destroy_black_lists () at mem/../fastlock.h:88 #15 0x08059013 in do_action (a=0x81d1e38, msg=0x8203db8) at parser/../ ip_addr.h:385 #16 0x0805a38d in destroy_black_lists () at mem/../fastlock.h:88 #17 0x0805a75a in delete_expired_routine (ticks=136098008, param=0x8203db8) at mem/../fastlock.h:88 #18 0x0809a2c3 in dns_a_parser ( rdata=0x819dba0 REGISTER sip:sip.xxx.it SIP/2.0\r\nVia: SIP/2.0/ UDP 10.0.6.83:5060;branch=z9hG4bK2f7b8b41;rport\r\nMax-Forwards: 69\r \nFrom: sip:vp...@sip.xxx.it;tag=as24ad03ef\r\nTo: sip:vpbx1..., end=0x298 Address 0x298 out of bounds) at resolve.c:314 #19 0x080e027b in parse_avp_name (name=0xbfa8ca58, type=0x15, avp_name=0x81f52d8) at usr_avp.c:552 #20 0x0807229f in MD5Transform (state=0x0, block=value optimized out) at md5.c:268 #21 0xb7c59450 in ?? () #22 0x08055761 in do_assign (msg=Cannot access memory at address 0x8 ) at action.c:242 Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Opensips crashed - problem with MD5?
Il giorno 15/apr/09, alle ore 13:36, Bogdan-Andrei Iancu ha scritto: Hi Carlo, The backtrace looks really broken.it might be a core overwrite. What revision on 1.5.0 are you using (do opensips -V) ? there are several mysql related bugs fixed since the 1.5.0 release. If you have a SVN copy, do a SVN update to get the latest fixes. Otherwise I recomand upgrading to 1.5.1 (see the download section on the web site). Ok Bogdan, I have upgraded to 1.5.1. One question: why I see version 1.4.3 from opensips -V and 1.5.1 from osipsconsole (monitor)? Thanks and Regards, Carlo ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Opensips crashed - problem with MD5?
hmmm... how did you do the upgrade ? maybe you have a mixture of installations between the versions This is what I get for the 1.5.1 SVN : $ ./opensips -V version: opensips 1.5.1-notls (i386/linux) flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. svnrevision: 2:5637 @(#) $Id: main.c 5469 2009-03-18 12:43:10Z bogdan_iancu $ main.c compiled on 16:03:06 Apr 15 2009 with gcc 4.1.3 Regards, Bogdan Carlo Dimaggio wrote: Il giorno 15/apr/09, alle ore 13:36, Bogdan-Andrei Iancu ha scritto: Hi Carlo, The backtrace looks really broken.it might be a core overwrite. What revision on 1.5.0 are you using (do opensips -V) ? there are several mysql related bugs fixed since the 1.5.0 release. If you have a SVN copy, do a SVN update to get the latest fixes. Otherwise I recomand upgrading to 1.5.1 (see the download section on the web site). Ok Bogdan, I have upgraded to 1.5.1. One question: why I see version 1.4.3 from opensips -V and 1.5.1 from osipsconsole (monitor)? ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Opensips crashed - problem with MD5?
Carlo Dimaggio wrote: Il giorno 15/apr/09, alle ore 15:05, Bogdan-Andrei Iancu ha scritto: hmmm... how did you do the upgrade ? maybe you have a mixture of installations between the versions This is what I get for the 1.5.1 SVN : I have downloaded the 1.5.1 src and compiled/installed with sudo make prefix=/ all sudo make prefix=/ install after having stopped the old version. What is the best practice to upgrade opensips? this should be enough, but as there is no uninstall, there is the risk of keeping some old files that were not removed by previous installation. What I do (when having more versions on the same server) is to install the whole stuff (for a version) under a single directory: make prefix=/usr/local/opensips-X.Y install In this way there is no way of having overlaps. In your case, I suspect the binary file (opensips) is still from the old installation - check the creation time. Regards, Bogdan ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
