Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

The libcrypto crash has also reproduced on the latest 2.4. I collected the 
backtrace as well as the information you requested below. There was no curl 
error, just this one from OpenSIPS:

Jun 27 13:06:10 [5992] CRITICAL:core:qm_free: freeing already freed pointer, 
first free: tls.h: os_free(119) - aborting

Backtrace: https://pastebin.com/SqtafdxR

# opensips -V
version: opensips 2.4.6 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 4e258b17f
main.c compiled on 18:21:00 Jun 26 2019 with gcc 7

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, June 26, 2019 at 9:12 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Oh, just do in gdb:
  f 2
  p *((struct qm_frag*)((char*)(p)-sizeof(struct qm_frag)))

In regards to async, I was more talking about the script async() (to try to 
remove it) , not about the async in the protocols.

Best regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/26/2019 04:00 PM, Ben Newlin wrote:
Bogdan,

Thanks for the response. I will get the info you need. When you say to print in 
frame 2, is there some command I must issue to put the current context into 
frame 2 (or 7 or 8) or do I just issue the command you provided? Apologies, I 
use GDB very seldomly.

Also, while we are using async support in this proxy, none of our current TLS 
traffic is using any async functions. Do you think other calls using async 
could be interfering with the TLS calls?

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Wednesday, June 26, 2019 at 5:25 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

In any of the corefile, in frame 2 could you print
p *((struct qm_frag*)((char*)(p)-sizeof(struct qm_frag)))


Now, I see that the backtrace via libcurl & libcrypto may go via some error 
cleanup - see frame #7 and #8 - so , do you see any curl error prior to the 
crash (ideally from the same process) ?

If you can reproduce it easily, could you try without the async support, just 
as a PoC only.

Thanks,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/25/2019 10:04 PM, Ben Newlin wrote:
Yes, we are using tls_mgm and proto_tls. We have the following version of 
openssl-devel installed:

# yum info openssl-devel
Name: openssl-devel
Arch: x86_64
Epoch   : 1
Version : 1.0.2k
Release : 16.amzn2.1.1
Size: 3.1 M
Repo: installed
From repo   : amzn2-core
Summary : Files for development of applications which will use OpenSSL
URL : http://www.openssl.org/<http://www.openssl.org/>
License : OpenSSL
Description : OpenSSL is a toolkit for supporting cryptography. The 
openssl-devel
: package contains include files needed to develop applications 
which
: support various cryptographic algorithms and protocols.


Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Tuesday, June 25, 2019 at 2:54 PM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Thank you Ben :)

The backtraces report a double free, but the mem ops are performed via 
libcrypto ( opensips registers mem-specific functions to lib crypto).

Are you using in your opensips cfg any other module that might require 
ssl/crtypto, like tls_mgm, proto_wss ?

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/25/2019 06:49 PM, Ben Newlin wrote:
Bogdan,

You have good timing. :) I was just collecting the data from a crash this 
morning with the extra flags set.

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: 

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

I forgot to provide the version information:

# opensips -V
version: opensips 2.4.6 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 4e258b17f
main.c compiled on 13:16:44 Jul 11 2019 with gcc 7

Ben Newlin

From: Ben Newlin 
Date: Thursday, July 11, 2019 at 4:14 PM
To: OpenSIPS devel mailling list , Bogdan-Andrei 
Iancu 
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi,

We experienced an issue today in our testing environment where OpenSIPS was 
crashing pretty much constantly. We are investigating some changes we made to 
our TLS config, but I wanted to also provide some of the backtraces here. Due 
to the sheer number of core dumps (over 360) I cannot provide them all, so I 
tried to take a relative sampling. I expected the backtraces to all be similar 
but they do appear to be substantially different, although on closer inspection 
they may be the same cause. It’s also not clear to me whether any of these 
crashes are the same as the ones I have already reported.

Server 1:
https://pastebin.com/5gmfd0KU

Server 2:
https://pastebin.com/r8vNdA8W

Server 3:
https://pastebin.com/cyg9F4Z5
https://pastebin.com/r4fvLA91

Server 4:
https://pastebin.com/V5MWrUWh
https://pastebin.com/FShUNc6z


Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Tuesday, July 9, 2019 at 5:20 PM
To: OpenSIPS devel mailling list , Bogdan-Andrei 
Iancu 
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Any updates on this? Would you like me to open a ticket or does one already 
exist for the issue?

Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Wednesday, June 26, 2019 at 11:58 AM
To: Bogdan-Andrei Iancu , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Bogdan,

I am glad the cause is known at least. :)

Is there a ticket I can follow for more information and to be notified of 
resolution?

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, June 26, 2019 at 7:37 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out a way to 
get it fixed in a more generic way; what you experience is just one face of the 
problem (with many faces :P) and there are other related reports.

Regards,





Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/26/2019 02:34 AM, Ben Newlin wrote:
Bogdan,

I have now been able to reproduce this crash reliably, I believe. Please see 
the backtrace [1]. Is there some other information you would like me to collect 
when this occurs?

[1] https://pastebin.com/n0Ph8XH5

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,






Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: https://pastebin.com/6bttsSVD.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 S

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Any updates on this TLS crash? I see there is a ticket open on Github for a TLS 
crash. I wonder if that is the same issue?

Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Thursday, June 27, 2019 at 6:42 AM
To: Bogdan-Andrei Iancu , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Bogdan,

I was able to reproduce this crash on the newer 2.4 commit.

Backtrace: https://pastebin.com/RaMXTZA4

# opensips -V
version: opensips 2.4.6 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 714afb97f
main.c compiled on 17:22:40 Jun 26 2019 with gcc 7

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, June 26, 2019 at 11:34 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

As per ticket, the issue you reported was already fixed, so going for the 
latest version in 2.4 branch should be fine with you.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/26/2019 06:21 PM, Ben Newlin wrote:
Bogdan,

I will test both with the latest and with that specific commit, but I cannot 
use the latest in production due to the issue I reported in 2.4.6. [1] Perhaps 
that specific commit is before whatever change is causing that issue.

[1] https://github.com/OpenSIPS/opensips/issues/1736

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Wednesday, June 26, 2019 at 10:56 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

While re-checking some of some data related to your report, I noticed that your 
opensips version (revision d025b4f61) is a bit old and it is missing some fixes 
that might be relevant for you (mainly 
3a994ceeafc6830a1c3b3f2699a7dc1ea915c83a).

Could you grab the latest 2.4 from git and give it a try please.

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/26/2019 02:37 PM, Bogdan-Andrei Iancu wrote:
Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out a way to 
get it fixed in a more generic way; what you experience is just one face of the 
problem (with many faces :P) and there are other related reports.

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/26/2019 02:34 AM, Ben Newlin wrote:
Bogdan,

I have now been able to reproduce this crash reliably, I believe. Please see 
the backtrace [1]. Is there some other information you would like me to collect 
when this occurs?

[1] https://pastebin.com/n0Ph8XH5

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,





Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: https://pastebin.com/6bttsSVD.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] Ope

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Any updates on this? Would you like me to open a ticket or does one already 
exist for the issue?

Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Wednesday, June 26, 2019 at 11:58 AM
To: Bogdan-Andrei Iancu , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Bogdan,

I am glad the cause is known at least. :)

Is there a ticket I can follow for more information and to be notified of 
resolution?

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, June 26, 2019 at 7:37 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out a way to 
get it fixed in a more generic way; what you experience is just one face of the 
problem (with many faces :P) and there are other related reports.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/26/2019 02:34 AM, Ben Newlin wrote:
Bogdan,

I have now been able to reproduce this crash reliably, I believe. Please see 
the backtrace [1]. Is there some other information you would like me to collect 
when this occurs?

[1] https://pastebin.com/n0Ph8XH5

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: https://pastebin.com/6bttsSVD.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 Service 
Unavailable" (which triggers the crash) - 10.32.20.60 ?? - a really close (from 
net delay perspective) and fast to answer party ?

Regards,





Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/05/2019 10:02 PM, Ben Newlin wrote:
We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 4978, 
id: 8


Please let me know if any further information can be useful.

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Friday, May 10, 2019 at 6:31 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp 7ffcaa850c80 
error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Friday, May 10, 2019 at 5:44 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: [OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx

# o

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

OK, thanks for the confirmation. Let me work out an additional fix for 
this;)


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 08:55 PM, Ben Newlin wrote:


Bogdan,

I was able to reproduce this crash on the newer 2.4 commit.

Backtrace: https://pastebin.com/RaMXTZA4 <https://pastebin.com/RaMXTZA4>

# opensips -V

version: opensips 2.4.6 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, 
QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT


ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535


poll method support: poll, epoll, sigio_rt, select.

git revision: 714afb97f

main.c compiled on 17:22:40 Jun 26 2019 with gcc 7

Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Wednesday, June 26, 2019 at 11:34 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

As per ticket, the issue you reported was already fixed, so going for 
the latest version in 2.4 branch should be fine with you.


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 06:21 PM, Ben Newlin wrote:

Bogdan,

I will test both with the latest and with that specific commit,
but I cannot use the latest in production due to the issue I
reported in 2.4.6. [1] Perhaps that specific commit is before
whatever change is causing that issue.

[1] https://github.com/OpenSIPS/opensips/issues/1736
<https://github.com/OpenSIPS/opensips/issues/1736>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Wednesday, June 26, 2019 at 10:56 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

While re-checking some of some data related to your report, I
noticed that your opensips version (revision d025b4f61) is a bit
old and it is missing some fixes that might be relevant for you
(mainly 3a994ceeafc6830a1c3b3f2699a7dc1ea915c83a).

Could you grab the latest 2.4 from git and give it a try please.

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 02:37 PM, Bogdan-Andrei Iancu wrote:

Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to
find out a way to get it fixed in a more generic way; what you
experience is just one face of the problem (with many faces
:P) and there are other related reports.

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 02:34 AM, Ben Newlin wrote:

Bogdan,

I have now been able to reproduce this crash reliably, I
believe. Please see the backtrace [1]. Is there some other
information you would like me to collect when this occurs?

[1] https://pastebin.com/n0Ph8XH5
<https://pastebin.com/n0Ph8XH5>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Friday, June 7, 2019 at 9:15 AM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling
list 
<mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ?
Brainstorming with Razvan, we suspect a race (on the msg
save in shmem in transaction) between the process doing
the cleanup after the async resume and the process running
the failure route (due th 503).

But this is just a supposition, eventually you can
validate it or not by removing the async ??

And on the double ACK - I'm not 100% it is a actually a
double one, as the second has a smaller MF value (69,
versus the 70 on the first ACK).

Regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/07/2019 03:52 PM, Ben Newlin wro

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

I was able to reproduce this crash on the newer 2.4 commit.

Backtrace: https://pastebin.com/RaMXTZA4

# opensips -V
version: opensips 2.4.6 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 714afb97f
main.c compiled on 17:22:40 Jun 26 2019 with gcc 7

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, June 26, 2019 at 11:34 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

As per ticket, the issue you reported was already fixed, so going for the 
latest version in 2.4 branch should be fine with you.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/26/2019 06:21 PM, Ben Newlin wrote:
Bogdan,

I will test both with the latest and with that specific commit, but I cannot 
use the latest in production due to the issue I reported in 2.4.6. [1] Perhaps 
that specific commit is before whatever change is causing that issue.

[1] 
https://github.com/OpenSIPS/opensips/issues/1736<https://github.com/OpenSIPS/opensips/issues/1736>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Wednesday, June 26, 2019 at 10:56 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

While re-checking some of some data related to your report, I noticed that your 
opensips version (revision d025b4f61) is a bit old and it is missing some fixes 
that might be relevant for you (mainly 
3a994ceeafc6830a1c3b3f2699a7dc1ea915c83a).

Could you grab the latest 2.4 from git and give it a try please.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/26/2019 02:37 PM, Bogdan-Andrei Iancu wrote:
Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out a way to 
get it fixed in a more generic way; what you experience is just one face of the 
problem (with many faces :P) and there are other related reports.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/26/2019 02:34 AM, Ben Newlin wrote:
Bogdan,

I have now been able to reproduce this crash reliably, I believe. Please see 
the backtrace [1]. Is there some other information you would like me to collect 
when this occurs?

[1] https://pastebin.com/n0Ph8XH5<https://pastebin.com/n0Ph8XH5>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: 
https://pastebin.com/6bttsSVD<https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Da

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

I see now. The commit you linked was against another branch. This fix was only 
backported to 2.4 after the 2.4.6 release. I will test with that.

Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Wednesday, June 26, 2019 at 11:49 AM
To: Bogdan-Andrei Iancu , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Bogdan,

Strange, I got no notification of your update on that ticket. Still I cannot 
see how it was fixed a month ago? I was using the 2.4.6 release when I 
encountered the crash. 2.4.6 was only released 15 days ago.

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, June 26, 2019 at 11:34 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

As per ticket, the issue you reported was already fixed, so going for the 
latest version in 2.4 branch should be fine with you.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/26/2019 06:21 PM, Ben Newlin wrote:
Bogdan,

I will test both with the latest and with that specific commit, but I cannot 
use the latest in production due to the issue I reported in 2.4.6. [1] Perhaps 
that specific commit is before whatever change is causing that issue.

[1] https://github.com/OpenSIPS/opensips/issues/1736

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Wednesday, June 26, 2019 at 10:56 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

While re-checking some of some data related to your report, I noticed that your 
opensips version (revision d025b4f61) is a bit old and it is missing some fixes 
that might be relevant for you (mainly 
3a994ceeafc6830a1c3b3f2699a7dc1ea915c83a).

Could you grab the latest 2.4 from git and give it a try please.

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/26/2019 02:37 PM, Bogdan-Andrei Iancu wrote:
Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out a way to 
get it fixed in a more generic way; what you experience is just one face of the 
problem (with many faces :P) and there are other related reports.

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/26/2019 02:34 AM, Ben Newlin wrote:
Bogdan,

I have now been able to reproduce this crash reliably, I believe. Please see 
the backtrace [1]. Is there some other information you would like me to collect 
when this occurs?

[1] https://pastebin.com/n0Ph8XH5

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,





Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: https://pastebin.com/6bttsSVD.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 Service 
Unavailable" (which triggers the crash) - 10

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Let's see if the update solves the problem (btw, it is the same thread 
as the one related to #1736)


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 06:56 PM, Ben Newlin wrote:


Bogdan,

I am glad the cause is known at least. :)

Is there a ticket I can follow for more information and to be notified 
of resolution?


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Wednesday, June 26, 2019 at 7:37 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out 
a way to get it fixed in a more generic way; what you experience is 
just one face of the problem (with many faces :P) and there are other 
related reports.


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 02:34 AM, Ben Newlin wrote:

Bogdan,

I have now been able to reproduce this crash reliably, I believe.
Please see the backtrace [1]. Is there some other information you
would like me to collect when this occurs?

[1] https://pastebin.com/n0Ph8XH5 <https://pastebin.com/n0Ph8XH5>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Friday, June 7, 2019 at 9:15 AM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ?
Brainstorming with Razvan, we suspect a race (on the msg save in
shmem in transaction) between the process doing the cleanup after
the async resume and the process running the failure route (due th
503).

But this is just a supposition, eventually you can validate it or
not by removing the async ??

And on the double ACK - I'm not 100% it is a actually a double
one, as the second has a smaller MF value (69, versus the 70 on
the first ACK).

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/07/2019 03:52 PM, Ben Newlin wrote:

Bogdan,

Sorry, I should have thought to actually look at the trace and
examine this call.

1) Yes

2) The Called Party is 10.32.20.60, which is another OpenSIPS
instance. The crashed instance received the "503 Service
Unavailable" approximately 8-10 ms after sending the INVITE.

There is a SIP trace of the exchange here:
https://pastebin.com/6bttsSVD <https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send
(or at least siptrace) the ACK twice.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Thursday, June 6, 2019 at 11:42 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the
"503 Service Unavailable" (which triggers the crash) -
10.32.20.60 ?? - a really close (from net delay perspective)
and fast to answer party ?

Regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/05/2019 10:02 PM, Ben Newlin wrote:

We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS
<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in
process pid: 4978, id: 8

Please let me know if any further information can be useful.

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben
Newlin 
<mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Date: *Friday, May 10, 2019 at 6:31 PM
*To: *OpenSIPS devel mailling list
   

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

I am glad the cause is known at least. :)

Is there a ticket I can follow for more information and to be notified of 
resolution?

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, June 26, 2019 at 7:37 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out a way to 
get it fixed in a more generic way; what you experience is just one face of the 
problem (with many faces :P) and there are other related reports.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/26/2019 02:34 AM, Ben Newlin wrote:
Bogdan,

I have now been able to reproduce this crash reliably, I believe. Please see 
the backtrace [1]. Is there some other information you would like me to collect 
when this occurs?

[1] https://pastebin.com/n0Ph8XH5<https://pastebin.com/n0Ph8XH5>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: 
https://pastebin.com/6bttsSVD<https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 Service 
Unavailable" (which triggers the crash) - 10.32.20.60 ?? - a really close (from 
net delay perspective) and fast to answer party ?

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/05/2019 10:02 PM, Ben Newlin wrote:
We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 4978, 
id: 8


Please let me know if any further information can be useful.

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Friday, May 10, 2019 at 6:31 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp 7ffcaa850c80 
error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Friday, May 10, 2019 at 5:44 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: [OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash toda

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

Strange, I got no notification of your update on that ticket. Still I cannot 
see how it was fixed a month ago? I was using the 2.4.6 release when I 
encountered the crash. 2.4.6 was only released 15 days ago.

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, June 26, 2019 at 11:34 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

As per ticket, the issue you reported was already fixed, so going for the 
latest version in 2.4 branch should be fine with you.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/26/2019 06:21 PM, Ben Newlin wrote:
Bogdan,

I will test both with the latest and with that specific commit, but I cannot 
use the latest in production due to the issue I reported in 2.4.6. [1] Perhaps 
that specific commit is before whatever change is causing that issue.

[1] 
https://github.com/OpenSIPS/opensips/issues/1736<https://github.com/OpenSIPS/opensips/issues/1736>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Wednesday, June 26, 2019 at 10:56 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

While re-checking some of some data related to your report, I noticed that your 
opensips version (revision d025b4f61) is a bit old and it is missing some fixes 
that might be relevant for you (mainly 
3a994ceeafc6830a1c3b3f2699a7dc1ea915c83a).

Could you grab the latest 2.4 from git and give it a try please.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/26/2019 02:37 PM, Bogdan-Andrei Iancu wrote:
Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out a way to 
get it fixed in a more generic way; what you experience is just one face of the 
problem (with many faces :P) and there are other related reports.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/26/2019 02:34 AM, Ben Newlin wrote:
Bogdan,

I have now been able to reproduce this crash reliably, I believe. Please see 
the backtrace [1]. Is there some other information you would like me to collect 
when this occurs?

[1] https://pastebin.com/n0Ph8XH5<https://pastebin.com/n0Ph8XH5>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: 
https://pastebin.com/6bttsSVD<https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for t

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

As per ticket, the issue you reported was already fixed, so going for 
the latest version in 2.4 branchshould be fine with you.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 06:21 PM, Ben Newlin wrote:


Bogdan,

I will test both with the latest and with that specific commit, but I 
cannot use the latest in production due to the issue I reported in 
2.4.6. [1] Perhaps that specific commit is before whatever change is 
causing that issue.


[1] https://github.com/OpenSIPS/opensips/issues/1736 
<https://github.com/OpenSIPS/opensips/issues/1736>


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Wednesday, June 26, 2019 at 10:56 AM
*To: *OpenSIPS devel mailling list , Ben 
Newlin 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

While re-checking some of some data related to your report, I noticed 
that your opensips version (revision d025b4f61) is a bit old and it is 
missing some fixes that might be relevant for you (mainly 
3a994ceeafc6830a1c3b3f2699a7dc1ea915c83a).


Could you grab the latest 2.4 from git and give it a try please.

Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 02:37 PM, Bogdan-Andrei Iancu wrote:

Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find
out a way to get it fixed in a more generic way; what you
experience is just one face of the problem (with many faces :P)
and there are other related reports.

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 02:34 AM, Ben Newlin wrote:

Bogdan,

I have now been able to reproduce this crash reliably, I
believe. Please see the backtrace [1]. Is there some other
information you would like me to collect when this occurs?

[1] https://pastebin.com/n0Ph8XH5 <https://pastebin.com/n0Ph8XH5>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Friday, June 7, 2019 at 9:15 AM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ?
Brainstorming with Razvan, we suspect a race (on the msg save
in shmem in transaction) between the process doing the cleanup
after the async resume and the process running the failure
route (due th 503).

But this is just a supposition, eventually you can validate it
or not by removing the async ??

And on the double ACK - I'm not 100% it is a actually a double
one, as the second has a smaller MF value (69, versus the 70
on the first ACK).

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/07/2019 03:52 PM, Ben Newlin wrote:

Bogdan,

Sorry, I should have thought to actually look at the trace
and examine this call.

1) Yes

2) The Called Party is 10.32.20.60, which is another
OpenSIPS instance. The crashed instance received the "503
Service Unavailable" approximately 8-10 ms after sending
the INVITE.

There is a SIP trace of the exchange here:
https://pastebin.com/6bttsSVD <https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to
send (or at least siptrace) the ACK twice.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Thursday, June 6, 2019 at 11:42 AM
*To: *OpenSIPS devel mailling list

<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating
the "503 Service Unavailable" (which triggers the crash) -
10.32.20.60 ?? - a really close (from net delay
perspective) and fast to answer party ?

  

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

I will test both with the latest and with that specific commit, but I cannot 
use the latest in production due to the issue I reported in 2.4.6. [1] Perhaps 
that specific commit is before whatever change is causing that issue.

[1] https://github.com/OpenSIPS/opensips/issues/1736

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, June 26, 2019 at 10:56 AM
To: OpenSIPS devel mailling list , Ben Newlin 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

While re-checking some of some data related to your report, I noticed that your 
opensips version (revision d025b4f61) is a bit old and it is missing some fixes 
that might be relevant for you (mainly 
3a994ceeafc6830a1c3b3f2699a7dc1ea915c83a).

Could you grab the latest 2.4 from git and give it a try please.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/26/2019 02:37 PM, Bogdan-Andrei Iancu wrote:
Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out a way to 
get it fixed in a more generic way; what you experience is just one face of the 
problem (with many faces :P) and there are other related reports.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/26/2019 02:34 AM, Ben Newlin wrote:
Bogdan,

I have now been able to reproduce this crash reliably, I believe. Please see 
the backtrace [1]. Is there some other information you would like me to collect 
when this occurs?

[1] https://pastebin.com/n0Ph8XH5<https://pastebin.com/n0Ph8XH5>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: 
https://pastebin.com/6bttsSVD<https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 Service 
Unavailable" (which triggers the crash) - 10.32.20.60 ?? - a really close (from 
net delay perspective) and fast to answer party ?

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/05/2019 10:02 PM, Ben Newlin wrote:
We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 4978, 
id: 8


Please let me know if any further information can be useful.

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

While re-checking some of some data related to your report, I noticed 
that your opensips version (revision d025b4f61) is a bit old and it is 
missing some fixes that might be relevant for you (mainly 
3a994ceeafc6830a1c3b3f2699a7dc1ea915c83a).


Could you grab the latest 2.4 from git and give it a try please.

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 02:37 PM, Bogdan-Andrei Iancu wrote:

Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out 
a way to get it fixed in a more generic way; what you experience is 
just one face of the problem(with many faces :P)and there are other 
related reports.


Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/26/2019 02:34 AM, Ben Newlin wrote:


Bogdan,

I have now been able to reproduce this crash reliably, I believe. 
Please see the backtrace [1]. Is there some other information you 
would like me to collect when this occurs?


[1] https://pastebin.com/n0Ph8XH5 <https://pastebin.com/n0Ph8XH5>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Friday, June 7, 2019 at 9:15 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? 
Brainstorming with Razvan, we suspect a race (on the msg save in 
shmem in transaction) between the process doing the cleanup after the 
async resume and the process running the failure route (due th 503).


But this is just a supposition, eventually you can validate it or not 
by removing the async ??


And on the double ACK - I'm not 100% it is a actually a double one, 
as the second has a smaller MF value (69, versus the 70 on the first 
ACK).


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/07/2019 03:52 PM, Ben Newlin wrote:

Bogdan,

Sorry, I should have thought to actually look at the trace and
examine this call.

1) Yes

2) The Called Party is 10.32.20.60, which is another OpenSIPS
instance. The crashed instance received the "503 Service
Unavailable" approximately 8-10 ms after sending the INVITE.

There is a SIP trace of the exchange here:
https://pastebin.com/6bttsSVD <https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send (or
at least siptrace) the ACK twice.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Thursday, June 6, 2019 at 11:42 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the
"503 Service Unavailable" (which triggers the crash) -
10.32.20.60 ?? - a really close (from net delay perspective) and
fast to answer party ?

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/05/2019 10:02 PM, Ben Newlin wrote:

We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS
<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in
process pid: 4978, id: 8

Please let me know if any further information can be useful.

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben
Newlin  <mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Date: *Friday, May 10, 2019 at 6:31 PM
*To: *OpenSIPS devel mailling list 
    <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp
7ffcaa850c80 error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben
Newlin  <mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Oh, just do in gdb:
  f 2
  p *((struct qm_frag*)((char*)(p)-sizeof(struct qm_frag)))

In regards to async, I was more talking about the script async() (to try 
to remove it) , not about the async in the protocols.


Best regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 04:00 PM, Ben Newlin wrote:


Bogdan,

Thanks for the response. I will get the info you need. When you say to 
print in frame 2, is there some command I must issue to put the 
current context into frame 2 (or 7 or 8) or do I just issue the 
command you provided? Apologies, I use GDB very seldomly.


Also, while we are using async support in this proxy, none of our 
current TLS traffic is using any async functions. Do you think other 
calls using async could be interfering with the TLS calls?


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Wednesday, June 26, 2019 at 5:25 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

In any of the corefile, in frame 2 could you print
p *((struct qm_frag*)((char*)(p)-sizeof(struct qm_frag)))


Now, I see that the backtrace via libcurl & libcrypto may go via some 
error cleanup - see frame #7 and #8 - so , do you see any curl error 
prior to the crash (ideally from the same process) ?


If you can reproduce it easily, could you try without the async 
support, just as a PoC only.


Thanks,


Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/25/2019 10:04 PM, Ben Newlin wrote:

Yes, we are using tls_mgm and proto_tls. We have the following
version of openssl-devel installed:

# yum info openssl-devel

Name: openssl-devel

Arch: x86_64

Epoch   : 1

Version : 1.0.2k

Release : 16.amzn2.1.1

Size: 3.1 M

Repo: installed

From repo   : amzn2-core

Summary : Files for development of applications which will use
OpenSSL

URL : http://www.openssl.org/ <http://www.openssl.org/>

License : OpenSSL

Description : OpenSSL is a toolkit for supporting cryptography.
The openssl-devel

: package contains include files needed to develop
applications which

: support various cryptographic algorithms and protocols.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Tuesday, June 25, 2019 at 2:54 PM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Thank you Ben :)

The backtraces report a double free, but the mem ops are performed
via libcrypto ( opensips registers mem-specific functions to lib
crypto).

Are you using in your opensips cfg any other module that might
require ssl/crtypto, like tls_mgm, proto_wss ?

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/25/2019 06:49 PM, Ben Newlin wrote:

Bogdan,

You have good timing. :) I was just collecting the data from a
crash this morning with the extra flags set.

# opensips -V

version: opensips 2.4.5 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535

poll method support: poll, epoll, sigio_rt, select.

git revision: 60097425d

main.c compiled on 22:27:12 Jun 24 2019 with gcc 7

[1] https://pastebin.com/aj1mvk0N <https://pastebin.com/aj1mvk0N>

[2] https://pastebin.com/kfSz9Ks0 <https://pastebin.com/kfSz9Ks0>

[3] https://pastebin.com/hptH9yiA <https://pastebin.com/hptH9yiA>

[4] https://pastebin.com/GfYYZxzB <https://pastebin.com/GfYYZxzB>

[5] https://pastebin.com/CzYiNAch <https://pastebin.com/CzYiNAch>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Tuesday, June 25, 2019 at 11:39 AM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
     <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The backtrace + corefile should do it ;).

Best regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https:

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

Thanks for the response. I will get the info you need. When you say to print in 
frame 2, is there some command I must issue to put the current context into 
frame 2 (or 7 or 8) or do I just issue the command you provided? Apologies, I 
use GDB very seldomly.

Also, while we are using async support in this proxy, none of our current TLS 
traffic is using any async functions. Do you think other calls using async 
could be interfering with the TLS calls?

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, June 26, 2019 at 5:25 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

In any of the corefile, in frame 2 could you print
p *((struct qm_frag*)((char*)(p)-sizeof(struct qm_frag)))


Now, I see that the backtrace via libcurl & libcrypto may go via some error 
cleanup - see frame #7 and #8 - so , do you see any curl error prior to the 
crash (ideally from the same process) ?

If you can reproduce it easily, could you try without the async support, just 
as a PoC only.

Thanks,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/25/2019 10:04 PM, Ben Newlin wrote:
Yes, we are using tls_mgm and proto_tls. We have the following version of 
openssl-devel installed:

# yum info openssl-devel
Name: openssl-devel
Arch: x86_64
Epoch   : 1
Version : 1.0.2k
Release : 16.amzn2.1.1
Size: 3.1 M
Repo: installed
From repo   : amzn2-core
Summary : Files for development of applications which will use OpenSSL
URL : http://www.openssl.org/<http://www.openssl.org/>
License : OpenSSL
Description : OpenSSL is a toolkit for supporting cryptography. The 
openssl-devel
: package contains include files needed to develop applications 
which
: support various cryptographic algorithms and protocols.


Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Tuesday, June 25, 2019 at 2:54 PM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Thank you Ben :)

The backtraces report a double free, but the mem ops are performed via 
libcrypto ( opensips registers mem-specific functions to lib crypto).

Are you using in your opensips cfg any other module that might require 
ssl/crtypto, like tls_mgm, proto_wss ?

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/25/2019 06:49 PM, Ben Newlin wrote:
Bogdan,

You have good timing. :) I was just collecting the data from a crash this 
morning with the extra flags set.

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 60097425d
main.c compiled on 22:27:12 Jun 24 2019 with gcc 7

[1] https://pastebin.com/aj1mvk0N<https://pastebin.com/aj1mvk0N>
[2] https://pastebin.com/kfSz9Ks0<https://pastebin.com/kfSz9Ks0>
[3] https://pastebin.com/hptH9yiA<https://pastebin.com/hptH9yiA>
[4] https://pastebin.com/GfYYZxzB<https://pastebin.com/GfYYZxzB>
[5] https://pastebin.com/CzYiNAch<https://pastebin.com/CzYiNAch>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Tuesday, June 25, 2019 at 11:39 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The backtrace + corefile should do it ;).

Best regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/20/2019 07:19 PM, Ben Newlin wrote:
Bogdan,

Yes I will enable those. This seems to be reproducing fairly often so hopefully 
we will get some more info quickly. Is there anything else I need to collect at 
time of the crash with this enabled or is the core dump sufficient?

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 20, 2019 at 12:15 PM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.op

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Thank you Ben,

This backtrace confirms the initial suspicion - I'm trying to find out a 
way to get it fixed in a more generic way; what you experience is just 
one face of the problem(with many faces :P)and there are other related 
reports.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/26/2019 02:34 AM, Ben Newlin wrote:


Bogdan,

I have now been able to reproduce this crash reliably, I believe. 
Please see the backtrace [1]. Is there some other information you 
would like me to collect when this occurs?


[1] https://pastebin.com/n0Ph8XH5 <https://pastebin.com/n0Ph8XH5>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Friday, June 7, 2019 at 9:15 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? 
Brainstorming with Razvan, we suspect a race (on the msg save in shmem 
in transaction) between the process doing the cleanup after the async 
resume and the process running the failure route (due th 503).


But this is just a supposition, eventually you can validate it or not 
by removing the async ??


And on the double ACK - I'm not 100% it is a actually a double one, as 
the second has a smaller MF value (69, versus the 70 on the first ACK).


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/07/2019 03:52 PM, Ben Newlin wrote:

Bogdan,

Sorry, I should have thought to actually look at the trace and
examine this call.

1) Yes

2) The Called Party is 10.32.20.60, which is another OpenSIPS
instance. The crashed instance received the "503 Service
Unavailable" approximately 8-10 ms after sending the INVITE.

There is a SIP trace of the exchange here:
https://pastebin.com/6bttsSVD <https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send (or
at least siptrace) the ACK twice.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Thursday, June 6, 2019 at 11:42 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503
Service Unavailable" (which triggers the crash) - 10.32.20.60 ?? -
a really close (from net delay perspective) and fast to answer party ?

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/05/2019 10:02 PM, Ben Newlin wrote:

We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS
<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in
process pid: 4978, id: 8

Please let me know if any further information can be useful.

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben
Newlin  <mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Date: *Friday, May 10, 2019 at 6:31 PM
*To: *OpenSIPS devel mailling list 
    <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp
7ffcaa850c80 error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben
Newlin  <mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Date: *Friday, May 10, 2019 at 5:44 PM
    *To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
*Subject: *[OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx

# opensips -V

version: opensips 2.4.5 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT

ADAPTIVE_WAIT_LOOPS=1024, MAX_RE

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

In any of the corefile, in frame 2 could you print
p *((struct qm_frag*)((char*)(p)-sizeof(struct qm_frag)))


Now, I see that the backtrace via libcurl & libcrypto may go via some 
error cleanup- see frame #7 and #8- so , do you see any curl error prior 
to the crash (ideally from the same process) ?


If you can reproduce it easily, could you try without the async support, 
just as a PoC only.


Thanks,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/25/2019 10:04 PM, Ben Newlin wrote:


Yes, we are using tls_mgm and proto_tls. We have the following version 
of openssl-devel installed:


# yum info openssl-devel

Name: openssl-devel

Arch: x86_64

Epoch   : 1

Version : 1.0.2k

Release : 16.amzn2.1.1

Size: 3.1 M

Repo: installed

From repo   : amzn2-core

Summary : Files for development of applications which will use OpenSSL

URL : http://www.openssl.org/

License : OpenSSL

Description : OpenSSL is a toolkit for supporting cryptography. The 
openssl-devel


: package contains include files needed to develop 
applications which


: support various cryptographic algorithms and protocols.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Tuesday, June 25, 2019 at 2:54 PM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Thank you Ben :)

The backtraces report a double free, but the mem ops are performed via 
libcrypto ( opensips registers mem-specific functions to lib crypto).


Are you using in your opensips cfg any other module that might require 
ssl/crtypto, like tls_mgm, proto_wss ?


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/25/2019 06:49 PM, Ben Newlin wrote:

Bogdan,

You have good timing. :) I was just collecting the data from a
crash this morning with the extra flags set.

# opensips -V

version: opensips 2.4.5 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
16, MAX_URI_SIZE 1024, BUF_SIZE 65535

poll method support: poll, epoll, sigio_rt, select.

git revision: 60097425d

main.c compiled on 22:27:12 Jun 24 2019 with gcc 7

[1] https://pastebin.com/aj1mvk0N <https://pastebin.com/aj1mvk0N>

[2] https://pastebin.com/kfSz9Ks0 <https://pastebin.com/kfSz9Ks0>

[3] https://pastebin.com/hptH9yiA <https://pastebin.com/hptH9yiA>

[4] https://pastebin.com/GfYYZxzB <https://pastebin.com/GfYYZxzB>

[5] https://pastebin.com/CzYiNAch <https://pastebin.com/CzYiNAch>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Tuesday, June 25, 2019 at 11:39 AM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The backtrace + corefile should do it ;).

Best regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/20/2019 07:19 PM, Ben Newlin wrote:

Bogdan,

Yes I will enable those. This seems to be reproducing fairly
often so hopefully we will get some more info quickly. Is
there anything else I need to collect at time of the crash
with this enabled or is the core dump sufficient?

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Thursday, June 20, 2019 at 12:15 PM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
     <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

This looks like very generic - a crash due a previous memory
corruption - is it possible for you to run with QM_MALLOC +
DBG_MALLOC ? otherwise it is rather impossible to identify the
original source of corruption.

Thanks and regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/20/2019 05:34 PM, Ben Newlin wrote:

We have had another crash this morning:
https://pastebin.com/dmXYbhzC

OpenSIPS version is the same as below.

B

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

I have now been able to reproduce this crash reliably, I believe. Please see 
the backtrace [1]. Is there some other information you would like me to collect 
when this occurs?

[1] https://pastebin.com/n0Ph8XH5

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: 
https://pastebin.com/6bttsSVD<https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 Service 
Unavailable" (which triggers the crash) - 10.32.20.60 ?? - a really close (from 
net delay perspective) and fast to answer party ?

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/05/2019 10:02 PM, Ben Newlin wrote:
We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 4978, 
id: 8


Please let me know if any further information can be useful.

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Friday, May 10, 2019 at 6:31 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp 7ffcaa850c80 
error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Friday, May 10, 2019 at 5:44 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: [OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx<https://pastebin.com/QbRJimwx>

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: d025b4f61
main.c compiled on 20:58:31 May  9 2019 with gcc 7

Ben Newlin





___

Devel mailing list

Devel@lists.opensips.org<mailto:Devel@lists.opensips.org>

http://lists.opensips.org/cgi-bin/mailman/listinfo/devel<http://lists.opensips.org/cgi-bin/mailman/listinfo/devel>





___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Yes, we are using tls_mgm and proto_tls. We have the following version of 
openssl-devel installed:

# yum info openssl-devel
Name: openssl-devel
Arch: x86_64
Epoch   : 1
Version : 1.0.2k
Release : 16.amzn2.1.1
Size: 3.1 M
Repo: installed
From repo   : amzn2-core
Summary : Files for development of applications which will use OpenSSL
URL : http://www.openssl.org/
License : OpenSSL
Description : OpenSSL is a toolkit for supporting cryptography. The 
openssl-devel
: package contains include files needed to develop applications 
which
: support various cryptographic algorithms and protocols.


Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Tuesday, June 25, 2019 at 2:54 PM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Thank you Ben :)

The backtraces report a double free, but the mem ops are performed via 
libcrypto ( opensips registers mem-specific functions to lib crypto).

Are you using in your opensips cfg any other module that might require 
ssl/crtypto, like tls_mgm, proto_wss ?

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/25/2019 06:49 PM, Ben Newlin wrote:
Bogdan,

You have good timing. :) I was just collecting the data from a crash this 
morning with the extra flags set.

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 60097425d
main.c compiled on 22:27:12 Jun 24 2019 with gcc 7

[1] https://pastebin.com/aj1mvk0N<https://pastebin.com/aj1mvk0N>
[2] https://pastebin.com/kfSz9Ks0<https://pastebin.com/kfSz9Ks0>
[3] https://pastebin.com/hptH9yiA<https://pastebin.com/hptH9yiA>
[4] https://pastebin.com/GfYYZxzB<https://pastebin.com/GfYYZxzB>
[5] https://pastebin.com/CzYiNAch<https://pastebin.com/CzYiNAch>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Tuesday, June 25, 2019 at 11:39 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The backtrace + corefile should do it ;).

Best regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/20/2019 07:19 PM, Ben Newlin wrote:
Bogdan,

Yes I will enable those. This seems to be reproducing fairly often so hopefully 
we will get some more info quickly. Is there anything else I need to collect at 
time of the crash with this enabled or is the core dump sufficient?

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 20, 2019 at 12:15 PM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

This looks like very generic - a crash due a previous memory corruption - is it 
possible for you to run with QM_MALLOC + DBG_MALLOC ? otherwise it is rather 
impossible to identify the original source of corruption.

Thanks and regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/20/2019 05:34 PM, Ben Newlin wrote:
We have had another crash this morning: 
https://pastebin.com/dmXYbhzC<https://pastebin.com/dmXYbhzC>

OpenSIPS version is the same as below.

Ben Newlin

From: Ben Newlin <mailto:ben.new...@genesys.com>
Date: Wednesday, June 19, 2019 at 2:38 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Bogdan-Andrei 
Iancu <mailto:bog...@opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

The Github issue has the version info for that, for all the other crashes the 
version is below.

$ opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 60097425d
main.c compiled on 

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Thank you Ben :)

The backtraces report a double free, but the memops are performed via 
libcrypto ( opensips registers mem-specific functions to lib crypto).


Are you using in your opensips cfg any other module that might require 
ssl/crtypto, like tls_mgm, proto_wss ?


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/25/2019 06:49 PM, Ben Newlin wrote:


Bogdan,

You have good timing. :) I was just collecting the data from a crash 
this morning with the extra flags set.


# opensips -V

version: opensips 2.4.5 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, 
QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT


ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535


poll method support: poll, epoll, sigio_rt, select.

git revision: 60097425d

main.c compiled on 22:27:12 Jun 24 2019 with gcc 7

[1] https://pastebin.com/aj1mvk0N <https://pastebin.com/aj1mvk0N>

[2] https://pastebin.com/kfSz9Ks0 <https://pastebin.com/kfSz9Ks0>

[3] https://pastebin.com/hptH9yiA <https://pastebin.com/hptH9yiA>

[4] https://pastebin.com/GfYYZxzB <https://pastebin.com/GfYYZxzB>

[5] https://pastebin.com/CzYiNAch <https://pastebin.com/CzYiNAch>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Tuesday, June 25, 2019 at 11:39 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The backtrace + corefile should do it ;).

Best regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/20/2019 07:19 PM, Ben Newlin wrote:

Bogdan,

Yes I will enable those. This seems to be reproducing fairly often
so hopefully we will get some more info quickly. Is there anything
else I need to collect at time of the crash with this enabled or
is the core dump sufficient?

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Thursday, June 20, 2019 at 12:15 PM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

This looks like very generic - a crash due a previous memory
corruption - is it possible for you to run with QM_MALLOC +
DBG_MALLOC ? otherwise it is rather impossible to identify the
original source of corruption.

Thanks and regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/20/2019 05:34 PM, Ben Newlin wrote:

We have had another crash this morning:
https://pastebin.com/dmXYbhzC

OpenSIPS version is the same as below.

Ben Newlin

*From: *Ben Newlin 
<mailto:ben.new...@genesys.com>
*Date: *Wednesday, June 19, 2019 at 2:38 PM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Bogdan-Andrei Iancu
     <mailto:bog...@opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

The Github issue has the version info for that, for all the
other crashes the version is below.

$ opensips -V

version: opensips 2.4.5 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535

poll method support: poll, epoll, sigio_rt, select.

git revision: 60097425d

main.c compiled on 18:06:35 Jun 13 2019 with gcc 7

Ben Newlin








___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

You have good timing. :) I was just collecting the data from a crash this 
morning with the extra flags set.

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 60097425d
main.c compiled on 22:27:12 Jun 24 2019 with gcc 7

[1] https://pastebin.com/aj1mvk0N
[2] https://pastebin.com/kfSz9Ks0
[3] https://pastebin.com/hptH9yiA
[4] https://pastebin.com/GfYYZxzB
[5] https://pastebin.com/CzYiNAch

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Tuesday, June 25, 2019 at 11:39 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The backtrace + corefile should do it ;).

Best regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/20/2019 07:19 PM, Ben Newlin wrote:
Bogdan,

Yes I will enable those. This seems to be reproducing fairly often so hopefully 
we will get some more info quickly. Is there anything else I need to collect at 
time of the crash with this enabled or is the core dump sufficient?

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 20, 2019 at 12:15 PM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

This looks like very generic - a crash due a previous memory corruption - is it 
possible for you to run with QM_MALLOC + DBG_MALLOC ? otherwise it is rather 
impossible to identify the original source of corruption.

Thanks and regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/20/2019 05:34 PM, Ben Newlin wrote:
We have had another crash this morning: 
https://pastebin.com/dmXYbhzC<https://pastebin.com/dmXYbhzC>

OpenSIPS version is the same as below.

Ben Newlin

From: Ben Newlin <mailto:ben.new...@genesys.com>
Date: Wednesday, June 19, 2019 at 2:38 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Bogdan-Andrei 
Iancu <mailto:bog...@opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

The Github issue has the version info for that, for all the other crashes the 
version is below.

$ opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 60097425d
main.c compiled on 18:06:35 Jun 13 2019 with gcc 7

Ben Newlin






___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

The backtrace + corefile should do it ;).

Best regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/20/2019 07:19 PM, Ben Newlin wrote:


Bogdan,

Yes I will enable those. This seems to be reproducing fairly often so 
hopefully we will get some more info quickly. Is there anything else I 
need to collect at time of the crash with this enabled or is the core 
dump sufficient?


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Thursday, June 20, 2019 at 12:15 PM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

This looks like very generic - a crash due a previous memory 
corruption - is it possible for you to run with QM_MALLOC + DBG_MALLOC 
? otherwise it is rather impossible to identify the original source of 
corruption.


Thanks and regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/20/2019 05:34 PM, Ben Newlin wrote:

We have had another crash this morning: https://pastebin.com/dmXYbhzC

OpenSIPS version is the same as below.

Ben Newlin

*From: *Ben Newlin 
<mailto:ben.new...@genesys.com>
*Date: *Wednesday, June 19, 2019 at 2:38 PM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Bogdan-Andrei Iancu
 <mailto:bog...@opensips.org>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

The Github issue has the version info for that, for all the other
crashes the version is below.

$ opensips -V

version: opensips 2.4.5 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
16, MAX_URI_SIZE 1024, BUF_SIZE 65535

poll method support: poll, epoll, sigio_rt, select.

git revision: 60097425d

main.c compiled on 18:06:35 Jun 13 2019 with gcc 7

Ben Newlin





___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

Yes I will enable those. This seems to be reproducing fairly often so hopefully 
we will get some more info quickly. Is there anything else I need to collect at 
time of the crash with this enabled or is the core dump sufficient?

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Thursday, June 20, 2019 at 12:15 PM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

This looks like very generic - a crash due a previous memory corruption - is it 
possible for you to run with QM_MALLOC + DBG_MALLOC ? otherwise it is rather 
impossible to identify the original source of corruption.

Thanks and regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/20/2019 05:34 PM, Ben Newlin wrote:
We have had another crash this morning: 
https://pastebin.com/dmXYbhzC<https://pastebin.com/dmXYbhzC>

OpenSIPS version is the same as below.

Ben Newlin

From: Ben Newlin <mailto:ben.new...@genesys.com>
Date: Wednesday, June 19, 2019 at 2:38 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Bogdan-Andrei 
Iancu <mailto:bog...@opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

The Github issue has the version info for that, for all the other crashes the 
version is below.

$ opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 60097425d
main.c compiled on 18:06:35 Jun 13 2019 with gcc 7

Ben Newlin



___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

This looks like very generic - a crash due a previous memory corruption 
- is it possible for you to run with QM_MALLOC + DBG_MALLOC ? otherwise 
it is rather impossible to identify the original source of corruption.


Thanks and regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/20/2019 05:34 PM, Ben Newlin wrote:


We have had another crash this morning: https://pastebin.com/dmXYbhzC

OpenSIPS version is the same as below.

Ben Newlin

*From: *Ben Newlin 
*Date: *Wednesday, June 19, 2019 at 2:38 PM
*To: *OpenSIPS devel mailling list , 
Bogdan-Andrei Iancu 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

The Github issue has the version info for that, for all the other 
crashes the version is below.


$ opensips -V

version: opensips 2.4.5 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, 
F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT


ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535


poll method support: poll, epoll, sigio_rt, select.

git revision: 60097425d

main.c compiled on 18:06:35 Jun 13 2019 with gcc 7

Ben Newlin




___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

We have had another crash this morning: https://pastebin.com/dmXYbhzC

OpenSIPS version is the same as below.

Ben Newlin

From: Ben Newlin 
Date: Wednesday, June 19, 2019 at 2:38 PM
To: OpenSIPS devel mailling list , Bogdan-Andrei 
Iancu 
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

The Github issue has the version info for that, for all the other crashes the 
version is below.

$ opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 60097425d
main.c compiled on 18:06:35 Jun 13 2019 with gcc 7

Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Wednesday, June 19, 2019 at 2:30 PM
To: Bogdan-Andrei Iancu , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Bogdan,

I’m continuing to try to reproduce the timing for that crash. In the meantime, 
we’ve had quite a few more crashes, but they don’t seem to be the same cause.

I opened an issue on Github for a reproducible, consistent crash that is new on 
2.4.6 involving Dialog pinging. [1]

We also had several of our servers crash the last few days due to what may be a 
double memory free? That is just a guess, I’m not great at reading backtraces. 
[2] [3] [4] [5] [6] [7]

Finally, we had another crash that seems to be in TLS processing. These 
backtraces don’t show much, so I don’t know if they will be helpful. [8] [9]


[1] https://github.com/OpenSIPS/opensips/issues/1736
[2] https://pastebin.com/HeRPs5wt
[3] https://pastebin.com/Fs6iUD7b
[4] https://pastebin.com/EkRNi2iM
[5] https://pastebin.com/9ZAurMwa
[6] https://pastebin.com/QyWhygvf
[7] https://pastebin.com/vEUm4UtK
[8] https://pastebin.com/0VaQfX5B
[9] https://pastebin.com/LYUW0AqH


Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Monday, June 10, 2019 at 2:41 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Ben,

With what we have so far it is too less data to get to a conclusion. Let's wait 
more and see if the crash reproduces.

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/07/2019 04:27 PM, Ben Newlin wrote:
Bogdan,

I no longer have the original backtrace I posted in May, but if it was the same 
issue then it has only happened those two times, both when under load. I have 
not been able to reproduce it reliably or with single calls.

For the double ACK, so if the MF value is changed then it is not just traced 
twice it was actually sent twice. But what scenario would cause retransmission 
of a hop-by-hop ACK? I can’t think of one. So it still seems strange. But it 
may be a symptom of the issue and not a cause.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,





Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: https://pastebin.com/6bttsSVD.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 Service 
Unavailable" (whic

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

The Github issue has the version info for that, for all the other crashes the 
version is below.

$ opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 60097425d
main.c compiled on 18:06:35 Jun 13 2019 with gcc 7

Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Wednesday, June 19, 2019 at 2:30 PM
To: Bogdan-Andrei Iancu , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Bogdan,

I’m continuing to try to reproduce the timing for that crash. In the meantime, 
we’ve had quite a few more crashes, but they don’t seem to be the same cause.

I opened an issue on Github for a reproducible, consistent crash that is new on 
2.4.6 involving Dialog pinging. [1]

We also had several of our servers crash the last few days due to what may be a 
double memory free? That is just a guess, I’m not great at reading backtraces. 
[2] [3] [4] [5] [6] [7]

Finally, we had another crash that seems to be in TLS processing. These 
backtraces don’t show much, so I don’t know if they will be helpful. [8] [9]


[1] https://github.com/OpenSIPS/opensips/issues/1736
[2] https://pastebin.com/HeRPs5wt
[3] https://pastebin.com/Fs6iUD7b
[4] https://pastebin.com/EkRNi2iM
[5] https://pastebin.com/9ZAurMwa
[6] https://pastebin.com/QyWhygvf
[7] https://pastebin.com/vEUm4UtK
[8] https://pastebin.com/0VaQfX5B
[9] https://pastebin.com/LYUW0AqH


Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Monday, June 10, 2019 at 2:41 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Ben,

With what we have so far it is too less data to get to a conclusion. Let's wait 
more and see if the crash reproduces.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/07/2019 04:27 PM, Ben Newlin wrote:
Bogdan,

I no longer have the original backtrace I posted in May, but if it was the same 
issue then it has only happened those two times, both when under load. I have 
not been able to reproduce it reliably or with single calls.

For the double ACK, so if the MF value is changed then it is not just traced 
twice it was actually sent twice. But what scenario would cause retransmission 
of a hop-by-hop ACK? I can’t think of one. So it still seems strange. But it 
may be a symptom of the issue and not a cause.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: https://pastebin.com/6bttsSVD.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 Service 
Unavailable" (which triggers the crash) - 10.32.20.60 ?? - a really close (from 
net delay perspective) and fast to answer party ?

Regards,





Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-20

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

I’m continuing to try to reproduce the timing for that crash. In the meantime, 
we’ve had quite a few more crashes, but they don’t seem to be the same cause.

I opened an issue on Github for a reproducible, consistent crash that is new on 
2.4.6 involving Dialog pinging. [1]

We also had several of our servers crash the last few days due to what may be a 
double memory free? That is just a guess, I’m not great at reading backtraces. 
[2] [3] [4] [5] [6] [7]

Finally, we had another crash that seems to be in TLS processing. These 
backtraces don’t show much, so I don’t know if they will be helpful. [8] [9]


[1] https://github.com/OpenSIPS/opensips/issues/1736
[2] https://pastebin.com/HeRPs5wt
[3] https://pastebin.com/Fs6iUD7b
[4] https://pastebin.com/EkRNi2iM
[5] https://pastebin.com/9ZAurMwa
[6] https://pastebin.com/QyWhygvf
[7] https://pastebin.com/vEUm4UtK
[8] https://pastebin.com/0VaQfX5B
[9] https://pastebin.com/LYUW0AqH


Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Monday, June 10, 2019 at 2:41 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Ben,

With what we have so far it is too less data to get to a conclusion. Let's wait 
more and see if the crash reproduces.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/07/2019 04:27 PM, Ben Newlin wrote:
Bogdan,

I no longer have the original backtrace I posted in May, but if it was the same 
issue then it has only happened those two times, both when under load. I have 
not been able to reproduce it reliably or with single calls.

For the double ACK, so if the MF value is changed then it is not just traced 
twice it was actually sent twice. But what scenario would cause retransmission 
of a hop-by-hop ACK? I can’t think of one. So it still seems strange. But it 
may be a symptom of the issue and not a cause.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: 
https://pastebin.com/6bttsSVD<https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 Service 
Unavailable" (which triggers the crash) - 10.32.20.60 ?? - a really close (from 
net delay perspective) and fast to answer party ?

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/05/2019 10:02 PM, Ben Newlin wrote:
We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 4978, 
id: 8


Please let me know if any further information can be us

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Ben,

With what we have so farit is too less data to get to a conclusion. 
Let's wait more and see if the crash reproduces.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/07/2019 04:27 PM, Ben Newlin wrote:


Bogdan,

I no longer have the original backtrace I posted in May, but if it was 
the same issue then it has only happened those two times, both when 
under load. I have not been able to reproduce it reliably or with 
single calls.


For the double ACK, so if the MF value is changed then it is not just 
traced twice it was actually sent twice. But what scenario would cause 
retransmission of a hop-by-hop ACK? I can’t think of one. So it still 
seems strange. But it may be a symptom of the issue and not a cause.


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Friday, June 7, 2019 at 9:15 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? 
Brainstorming with Razvan, we suspect a race (on the msg save in shmem 
in transaction) between the process doing the cleanup after the async 
resume and the process running the failure route (due th 503).


But this is just a supposition, eventually you can validate it or not 
by removing the async ??


And on the double ACK - I'm not 100% it is a actually a double one, as 
the second has a smaller MF value (69, versus the 70 on the first ACK).


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/07/2019 03:52 PM, Ben Newlin wrote:

Bogdan,

Sorry, I should have thought to actually look at the trace and
examine this call.

1) Yes

2) The Called Party is 10.32.20.60, which is another OpenSIPS
instance. The crashed instance received the "503 Service
Unavailable" approximately 8-10 ms after sending the INVITE.

There is a SIP trace of the exchange here:
https://pastebin.com/6bttsSVD <https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send (or
at least siptrace) the ACK twice.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Thursday, June 6, 2019 at 11:42 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503
Service Unavailable" (which triggers the crash) - 10.32.20.60 ?? -
a really close (from net delay perspective) and fast to answer party ?

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   https://www.opensips-solutions.com

OpenSIPS Summit 2019

   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/05/2019 10:02 PM, Ben Newlin wrote:

We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS
<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in
process pid: 4978, id: 8

Please let me know if any further information can be useful.

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben
Newlin  <mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Date: *Friday, May 10, 2019 at 6:31 PM
*To: *OpenSIPS devel mailling list 
    <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp
7ffcaa850c80 error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben
Newlin  <mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Date: *Friday, May 10, 2019 at 5:44 PM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
*Subject: *[OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx

# opensips -V

version: opensips 2.4.5 (x86_64/linux)

   

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

I no longer have the original backtrace I posted in May, but if it was the same 
issue then it has only happened those two times, both when under load. I have 
not been able to reproduce it reliably or with single calls.

For the double ACK, so if the MF value is changed then it is not just traced 
twice it was actually sent twice. But what scenario would cause retransmission 
of a hop-by-hop ACK? I can’t think of one. So it still seems strange. But it 
may be a symptom of the issue and not a cause.

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Friday, June 7, 2019 at 9:15 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming with 
Razvan, we suspect a race (on the msg save in shmem in transaction) between the 
process doing the cleanup after the async resume and the process running the 
failure route (due th 503).

But this is just a supposition, eventually you can validate it or not by 
removing the async ??

And on the double ACK - I'm not 100% it is a actually a double one, as the 
second has a smaller MF value (69, versus the 70 on the first ACK).

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/07/2019 03:52 PM, Ben Newlin wrote:
Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: 
https://pastebin.com/6bttsSVD<https://pastebin.com/6bttsSVD>.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 Service 
Unavailable" (which triggers the crash) - 10.32.20.60 ?? - a really close (from 
net delay perspective) and fast to answer party ?

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/05/2019 10:02 PM, Ben Newlin wrote:
We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 4978, 
id: 8


Please let me know if any further information can be useful.

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Friday, May 10, 2019 at 6:31 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp 7ffcaa850c80 
error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Friday, May 10, 2019 at 5:44 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: [OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx<https://pastebin.com/QbRJimwx>

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: d025b4f61
main.c compiled on 20:58:31 May  9 2019 with gcc 7

Ben Newlin





___

Devel mailing list

Devel@lists.opensips.org<mailto:Devel@lists.opensips.org>

http://lists.opensips.org/cgi-bin/mailman/listinfo/devel<http://lists.opensips.org/cgi-bin/mai

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

How often/easy is to reproduce this crash (if possible) ? Brainstorming 
with Razvan, we suspect a race(on the msg save in shmem in 
transaction)between the process doing the cleanup after the async resume 
and the process running the failure route (due th 503).


But this is just a supposition, eventually you can validate it or not by 
removing the async ??


And on the double ACK - I'm not 100% it is a actually a double one, as 
the second has a smaller MF value (69, versus the 70 on the first ACK).


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/07/2019 03:52 PM, Ben Newlin wrote:


Bogdan,

Sorry, I should have thought to actually look at the trace and examine 
this call.


1) Yes

2) The Called Party is 10.32.20.60, which is another OpenSIPS 
instance. The crashed instance received the "503 Service Unavailable" 
approximately 8-10 ms after sending the INVITE.


There is a SIP trace of the exchange here: https://pastebin.com/6bttsSVD.

One oddity I saw is that the crashed process appears to send (or at 
least siptrace) the ACK twice.


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Thursday, June 6, 2019 at 11:42 AM
*To: *OpenSIPS devel mailling list , Ben 
Newlin 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 
Service Unavailable" (which triggers the crash) - 10.32.20.60 ?? - a 
really close (from net delay perspective) and fast to answer party ?


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/05/2019 10:02 PM, Ben Newlin wrote:

We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS
<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process
pid: 4978, id: 8

Please let me know if any further information can be useful.

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben Newlin
 <mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Date: *Friday, May 10, 2019 at 6:31 PM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp
7ffcaa850c80 error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben Newlin
 <mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Date: *Friday, May 10, 2019 at 5:44 PM
*To: *OpenSIPS devel mailling list 
    <mailto:devel@lists.opensips.org>
*Subject: *[OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx

# opensips -V

version: opensips 2.4.5 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
16, MAX_URI_SIZE 1024, BUF_SIZE 65535

poll method support: poll, epoll, sigio_rt, select.

git revision: d025b4f61

main.c compiled on 20:58:31 May  9 2019 with gcc 7

Ben Newlin




___

Devel mailing list

Devel@lists.opensips.org <mailto:Devel@lists.opensips.org>

http://lists.opensips.org/cgi-bin/mailman/listinfo/devel





___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

Sorry, I should have thought to actually look at the trace and examine this 
call.

1) Yes
2) The Called Party is 10.32.20.60, which is another OpenSIPS instance. The 
crashed instance received the "503 Service Unavailable" approximately 8-10 ms 
after sending the INVITE.

There is a SIP trace of the exchange here: https://pastebin.com/6bttsSVD.

One oddity I saw is that the crashed process appears to send (or at least 
siptrace) the ACK twice.

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Thursday, June 6, 2019 at 11:42 AM
To: OpenSIPS devel mailling list , Ben Newlin 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 Service 
Unavailable" (which triggers the crash) - 10.32.20.60 ?? - a really close (from 
net delay perspective) and fast to answer party ?

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  
https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/05/2019 10:02 PM, Ben Newlin wrote:
We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS<https://pastebin.com/q4RQC7kS>

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 4978, 
id: 8


Please let me know if any further information can be useful.

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Friday, May 10, 2019 at 6:31 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp 7ffcaa850c80 
error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Friday, May 10, 2019 at 5:44 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: [OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx<https://pastebin.com/QbRJimwx>

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: d025b4f61
main.c compiled on 20:58:31 May  9 2019 with gcc 7

Ben Newlin




___

Devel mailing list

Devel@lists.opensips.org<mailto:Devel@lists.opensips.org>

http://lists.opensips.org/cgi-bin/mailman/listinfo/devel<http://lists.opensips.org/cgi-bin/mailman/listinfo/devel>


___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Dan,

Yes, good observation that c pointer is invalid - but it is not because 
of an overflow, but it rather seems that the msg->contact->parsed (where 
the "c" is read from) was populated with a pkg pointer in a different 
process.


Regards

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/06/2019 05:01 PM, Dan Pascu wrote:

Looks like buffer overflow. That c variable in the first frame should be a memory address, but 
instead it contains "lo EYB", which I guess is "BYE ol" on little endian 
machines. Looks like some parsed part of the message spilled over and overwrote memory pointers.

On 5 Jun 2019, at 22:02, Ben Newlin wrote:


We have had another crash today.
  
Backtrace is here: https://pastebin.com/q4RQC7kS
  
I found this in the log at the time of the crash:
  
Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 4978, id: 8
  
  
Please let me know if any further information can be useful.
  
Ben Newlin
  
From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Friday, May 10, 2019 at 6:31 PM
To: OpenSIPS devel mailling list 
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash
  
I found this in the log at the time of the crash:
  
kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp 7ffcaa850c80 error 6 in tm.so[7fbd4c887000+8e000]
  
Ben Newlin
  
From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Friday, May 10, 2019 at 5:44 PM
To: OpenSIPS devel mailling list 
Subject: [OpenSIPS-Devel] OpenSIPS Crash
  
Hello,
  
We had a crash today of our OpenSIPS instance.
  
Backtrace is here: https://pastebin.com/QbRJimwx
  
# opensips -V

version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: d025b4f61
main.c compiled on 20:58:31 May  9 2019 with gcc 7
  
Ben Newlin

___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel


--
Dan





___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel



___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

Thanks for "another" report :).

Questions:
1) do you do any async for the INVITE in this crash ?
2) if it is an YES to (1), is the caller party generating the "503 
Service Unavailable" (which triggers the crash) - 10.32.20.60 ?? - a 
really close (from net delay perspective) and fast to answer party ?


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 06/05/2019 10:02 PM, Ben Newlin wrote:


We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS 
<https://pastebin.com/q4RQC7kS>


I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 
4978, id: 8


Please let me know if any further information can be useful.

Ben Newlin

*From: *Devel  on behalf of Ben 
Newlin 

*Reply-To: *OpenSIPS devel mailling list 
*Date: *Friday, May 10, 2019 at 6:31 PM
*To: *OpenSIPS devel mailling list 
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp 
7ffcaa850c80 error 6 in tm.so[7fbd4c887000+8e000]


Ben Newlin

*From: *Devel  on behalf of Ben 
Newlin 

*Reply-To: *OpenSIPS devel mailling list 
*Date: *Friday, May 10, 2019 at 5:44 PM
*To: *OpenSIPS devel mailling list 
*Subject: *[OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx

# opensips -V

version: opensips 2.4.5 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, 
F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT


ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535


poll method support: poll, epoll, sigio_rt, select.

git revision: d025b4f61

main.c compiled on 20:58:31 May  9 2019 with gcc 7

Ben Newlin



___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel


___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Dan Pascu]

Looks like buffer overflow. That c variable in the first frame should be a 
memory address, but instead it contains "lo EYB", which I guess is "BYE ol" on 
little endian machines. Looks like some parsed part of the message spilled over 
and overwrote memory pointers.

On 5 Jun 2019, at 22:02, Ben Newlin wrote:

> We have had another crash today.
>  
> Backtrace is here: https://pastebin.com/q4RQC7kS
>  
> I found this in the log at the time of the crash:
>  
> Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 4978, 
> id: 8
>  
>  
> Please let me know if any further information can be useful.
>  
> Ben Newlin
>  
> From: Devel  on behalf of Ben Newlin 
> 
> Reply-To: OpenSIPS devel mailling list 
> Date: Friday, May 10, 2019 at 6:31 PM
> To: OpenSIPS devel mailling list 
> Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash
>  
> I found this in the log at the time of the crash:
>  
> kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp 
> 7ffcaa850c80 error 6 in tm.so[7fbd4c887000+8e000]
>  
> Ben Newlin
>  
> From: Devel  on behalf of Ben Newlin 
> 
> Reply-To: OpenSIPS devel mailling list 
> Date: Friday, May 10, 2019 at 5:44 PM
> To: OpenSIPS devel mailling list 
> Subject: [OpenSIPS-Devel] OpenSIPS Crash
>  
> Hello,
>  
> We had a crash today of our OpenSIPS instance.
>  
> Backtrace is here: https://pastebin.com/QbRJimwx
>  
> # opensips -V
> version: opensips 2.4.5 (x86_64/linux)
> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
> FAST_LOCK-ADAPTIVE_WAIT
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
> MAX_URI_SIZE 1024, BUF_SIZE 65535
> poll method support: poll, epoll, sigio_rt, select.
> git revision: d025b4f61
> main.c compiled on 20:58:31 May  9 2019 with gcc 7
>  
> Ben Newlin
> ___
> Devel mailing list
> Devel@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel


--
Dan





___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

We have had another crash today.

Backtrace is here: https://pastebin.com/q4RQC7kS

I found this in the log at the time of the crash:

Jun  5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault in process pid: 4978, 
id: 8


Please let me know if any further information can be useful.

Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Friday, May 10, 2019 at 6:31 PM
To: OpenSIPS devel mailling list 
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp 7ffcaa850c80 
error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Friday, May 10, 2019 at 5:44 PM
To: OpenSIPS devel mailling list 
Subject: [OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: d025b4f61
main.c compiled on 20:58:31 May  9 2019 with gcc 7

Ben Newlin
___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

I found this in the log at the time of the crash:

kernel: opensips[5003]: segfault at 30 ip 7fbd4c8f59d0 sp 7ffcaa850c80 
error 6 in tm.so[7fbd4c887000+8e000]

Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Friday, May 10, 2019 at 5:44 PM
To: OpenSIPS devel mailling list 
Subject: [OpenSIPS-Devel] OpenSIPS Crash

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: d025b4f61
main.c compiled on 20:58:31 May  9 2019 with gcc 7

Ben Newlin
___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Hello,

We had a crash today of our OpenSIPS instance.

Backtrace is here: https://pastebin.com/QbRJimwx

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, 
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: d025b4f61
main.c compiled on 20:58:31 May  9 2019 with gcc 7

Ben Newlin
___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Yes, Liviu provided the specific commit in another message and I have upgraded.

Thanks again!

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Tuesday, November 20, 2018 at 7:38 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The fix is now available on GIT repo, so simply update and it should be fine.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/19/2018 05:31 PM, Ben Newlin wrote:
Bogdan,

That is excellent news! Thank you very much for your work on this and Liviu as 
well!

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Monday, November 19, 2018 at 9:03 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

With some additional information that we got from a different feed, we managed 
to reproduce the crash and Liviu is working on a fix as we speak. I will keep 
you posted, but many thanks for your help in investigating this issue.

Best regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/13/2018 07:41 PM, Ben Newlin wrote:
Bogdan,

Yes, we are setting acc_extra variables in our branch routes, which are 
sometimes (but not always) called from failure route. Are acc_extra variables 
not available for use in branch_routes?

We don’t currently use drop_accounting anywhere in our script. If I call it 
before that branch_route then it will stop accounting for that call, right? We 
need to have accounting records for the call, so I’m not sure how that would 
resolve the issue?

Ben Newlin



___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

The fix is now available on GIT repo, so simply update and it should be 
fine.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/19/2018 05:31 PM, Ben Newlin wrote:


Bogdan,

That is excellent news! Thank you very much for your work on this and 
Liviu as well!


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Monday, November 19, 2018 at 9:03 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

With some additional information that we got from a different feed, we 
managed to reproduce the crash and Liviu is working on a fix as we 
speak. I will keep you posted, but many thanks for your help in 
investigating this issue.


Best regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/13/2018 07:41 PM, Ben Newlin wrote:

Bogdan,

Yes, we are setting acc_extra variables in our branch routes,
which are sometimes (but not always) called from failure route.
Are acc_extra variables not available for use in branch_routes?

We don’t currently use drop_accounting anywhere in our script. If
I call it before that branch_route then it will stop accounting
for that call, right? We need to have accounting records for the
call, so I’m not sure how that would resolve the issue?

Ben Newlin




___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Liviu Chircu]

Hi Ben,

Not sure if you noticed yet, but the fix is already available here [1].  
Thank you for all the help!


Cheers,

[1]: https://github.com/OpenSIPS/opensips/commit/e25895624b1c815dd8

Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com

On 19.11.2018 17:31, Ben Newlin wrote:


Bogdan,

That is excellent news! Thank you very much for your work on this and 
Liviu as well!


Ben Newlin

___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

That is excellent news! Thank you very much for your work on this and Liviu as 
well!

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Monday, November 19, 2018 at 9:03 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

With some additional information that we got from a different feed, we managed 
to reproduce the crash and Liviu is working on a fix as we speak. I will keep 
you posted, but many thanks for your help in investigating this issue.

Best regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/13/2018 07:41 PM, Ben Newlin wrote:
Bogdan,

Yes, we are setting acc_extra variables in our branch routes, which are 
sometimes (but not always) called from failure route. Are acc_extra variables 
not available for use in branch_routes?

We don’t currently use drop_accounting anywhere in our script. If I call it 
before that branch_route then it will stop accounting for that call, right? We 
need to have accounting records for the call, so I’m not sure how that would 
resolve the issue?

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Tuesday, November 13, 2018 at 9:13 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for the info. The crash happens when you try to set an acc extra 
variable in branch route (when a creating a new branch via failure route, on 
timeout).

Now, do you use the drop accounting in your script ? and considering the above 
scenario, it is possible to have the drop acc before the branch route ?

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/12/2018 08:55 PM, Ben Newlin wrote:
Bogdan,

We upgraded to 2.4.3 and the crash reproduced today. Backtrace is available 
here: https://pastebin.com/CZxQnZdR<https://pastebin.com/CZxQnZdR>.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Wednesday, November 7, 2018 at 6:18 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The BT indicates a double free for the accounting context - and I noticed you 
use 2.4.1 version. And yes, there was an issue related to acc context, issue 
that was fixed starting 2.4.2. So, could you upgrade to the latest 2.4 and see 
if the crash still happens ? As I think the fix is already there.

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:
Jackpot - you get it right !! I will start digging into the trace, but please 
keep the corefile, I might need it later.

Thanks and regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/06/2018 10:24 PM, Ben Newlin wrote:
Bogdan,

I have reproduced this crash and verified this time that the flags were set.

$ opensips -V
version: opensips 2.4.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 5d042cffc
main.c compiled on 23:38:55 Nov  5 2018 with gcc 7

Backtrace is available here: 
https://pastebin.com/KTQjkCwq<https://pastebin.com/KTQjkCwq>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, November 1, 2018 at 1:19 PM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

According to the backtrace, the memory debugger was not activated. Do an 
"opensips -V" to check the resulting compile flags - do you see the DBG_MALLOC 
and QM_MALLOC ?

Regards,





Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://w

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

The one I provided is the only one I have with the memory debugging compiled 
in. I’m going to re-enable that and push it so that we will have that info when 
it recurs.

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Thursday, November 15, 2018 at 11:08 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

DO you have the backtraces from more similar crashes ? may there is a pattern 
there.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/15/2018 05:01 PM, Ben Newlin wrote:
Bogdan,

It’s happening every few days, so it is pretty frequent. There was another one 
yesterday but the DBG compile flags had been temporarily removed for that one.

We have not been able to determine a sequence to reproduce it yet.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, November 15, 2018 at 7:06 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often this crash happens ? are you able to reproduce it ?

The acc extra should work in the branch route, no problem. Out of curiosity, I 
will try to reproduce you case (timeout -> failure route -> t_relay -> 
branch_route) to see if I can reproduce it.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/13/2018 07:41 PM, Ben Newlin wrote:
Bogdan,

Yes, we are setting acc_extra variables in our branch routes, which are 
sometimes (but not always) called from failure route. Are acc_extra variables 
not available for use in branch_routes?

We don’t currently use drop_accounting anywhere in our script. If I call it 
before that branch_route then it will stop accounting for that call, right? We 
need to have accounting records for the call, so I’m not sure how that would 
resolve the issue?

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Tuesday, November 13, 2018 at 9:13 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for the info. The crash happens when you try to set an acc extra 
variable in branch route (when a creating a new branch via failure route, on 
timeout).

Now, do you use the drop accounting in your script ? and considering the above 
scenario, it is possible to have the drop acc before the branch route ?

Regards,





Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/12/2018 08:55 PM, Ben Newlin wrote:
Bogdan,

We upgraded to 2.4.3 and the crash reproduced today. Backtrace is available 
here: https://pastebin.com/CZxQnZdR<https://pastebin.com/CZxQnZdR>.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Wednesday, November 7, 2018 at 6:18 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The BT indicates a double free for the accounting context - and I noticed you 
use 2.4.1 version. And yes, there was an issue related to acc context, issue 
that was fixed starting 2.4.2. So, could you upgrade to the latest 2.4 and see 
if the crash still happens ? As I think the fix is already there.

Regards,





Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:
Jackpot - you get it right !! I will start digging into the trace, but please 
keep the corefile, I might need it later.

Thanks and regards,





Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/06/2018 10:24 PM, Ben Newlin wrote:
Bogdan,

I have reproduced this crash and verified this time that the flags were set.

$ opensips -V
version: opensips 2.4.1

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

With someadditional information that we got from a different feed, we 
managed to reproduce the crashand Liviu is working on a fix as we speak. 
I will keep you posted, but many thanks for your help in investigating 
this issue.


Best regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/13/2018 07:41 PM, Ben Newlin wrote:


Bogdan,

Yes, we are setting acc_extra variables in our branch routes, which 
are sometimes (but not always) called from failure route. Are 
acc_extra variables not available for use in branch_routes?


We don’t currently use drop_accounting anywhere in our script. If I 
call it before that branch_route then it will stop accounting for that 
call, right? We need to have accounting records for the call, so I’m 
not sure how that would resolve the issue?


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Tuesday, November 13, 2018 at 9:13 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for the info. The crash happens when you try to set an acc 
extra variable in branch route (when a creating a new branch via 
failure route, on timeout).


Now, do you use the drop accounting in your script ? and considering 
the above scenario, it is possible to have the drop acc before the 
branch route ?


Regards,


Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/12/2018 08:55 PM, Ben Newlin wrote:

Bogdan,

We upgraded to 2.4.3 and the crash reproduced today. Backtrace is
available here: https://pastebin.com/CZxQnZdR
<https://pastebin.com/CZxQnZdR>.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Wednesday, November 7, 2018 at 6:18 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The BT indicates a double free for the accounting context - and I
noticed you use 2.4.1 version. And yes, there was an issue related
to acc context, issue that was fixed starting 2.4.2. So, could you
upgrade to the latest 2.4 and see if the crash still happens ? As
I think the fix is already there.

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:

Jackpot - you get it right !! I will start digging into the
trace, but please keep the corefile, I might need it later.

Thanks and regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/06/2018 10:24 PM, Ben Newlin wrote:

Bogdan,

I have reproduced this crash and verified this time that
the flags were set.

$ opensips -V

version: opensips 2.4.1 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, QM_MALLOC, DBG_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535

poll method support: poll, epoll, sigio_rt, select.

git revision: 5d042cffc

main.c compiled on 23:38:55 Nov  5 2018 with gcc 7

Backtrace is available here: https://pastebin.com/KTQjkCwq
<https://pastebin.com/KTQjkCwq>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Thursday, November 1, 2018 at 1:19 PM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling
list 
<mailto:devel@lists.opensips.org>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

According to the backtrace, the memory debugger was not
activated. Do an "opensips -V" to check the resulting
compile flags - do you see the DBG_MALLOC and QM_MALLOC ?

Regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/31/2018 05:04 PM, Ben Newlin wrote:

Bogdan,

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

DO you have the backtraces from more similar crashes ? may there is 
apattern there.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/15/2018 05:01 PM, Ben Newlin wrote:


Bogdan,

It’s happening every few days, so it is pretty frequent. There was 
another one yesterday but the DBG compile flags had been temporarily 
removed for that one.


We have not been able to determine a sequence to reproduce it yet.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Thursday, November 15, 2018 at 7:06 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often this crash happens ? are you able to reproduce it ?

The acc extra should work in the branch route, no problem. Out of 
curiosity, I will try to reproduce you case (timeout -> failure route 
-> t_relay -> branch_route) to see if I can reproduce it.


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/13/2018 07:41 PM, Ben Newlin wrote:

Bogdan,

Yes, we are setting acc_extra variables in our branch routes,
which are sometimes (but not always) called from failure route.
Are acc_extra variables not available for use in branch_routes?

We don’t currently use drop_accounting anywhere in our script. If
I call it before that branch_route then it will stop accounting
for that call, right? We need to have accounting records for the
call, so I’m not sure how that would resolve the issue?

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Tuesday, November 13, 2018 at 9:13 AM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for the info. The crash happens when you try to set an acc
extra variable in branch route (when a creating a new branch via
failure route, on timeout).

Now, do you use the drop accounting in your script ? and
considering the above scenario, it is possible to have the drop
acc before the branch route ?

Regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/12/2018 08:55 PM, Ben Newlin wrote:

Bogdan,

We upgraded to 2.4.3 and the crash reproduced today. Backtrace
is available here: https://pastebin.com/CZxQnZdR
<https://pastebin.com/CZxQnZdR>.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Wednesday, November 7, 2018 at 6:18 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The BT indicates a double free for the accounting context -
and I noticed you use 2.4.1 version. And yes, there was an
issue related to acc context, issue that was fixed starting
2.4.2. So, could you upgrade to the latest 2.4 and see if the
crash still happens ? As I think the fix is already there.

Regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:

Jackpot - you get it right !! I will start digging into
the trace, but please keep the corefile, I might need it
later.

Thanks and regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/06/2018 10:24 PM, Ben Newlin wrote:

Bogdan,

I have reproduced this crash and verified this time
that the flags were set.

$ opensips -V

version: opensips 2.4.1 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, QM_MALLOC, DBG_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535

poll method support: poll, epoll, sigio_rt, select.

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

It’s happening every few days, so it is pretty frequent. There was another one 
yesterday but the DBG compile flags had been temporarily removed for that one.

We have not been able to determine a sequence to reproduce it yet.

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Thursday, November 15, 2018 at 7:06 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

How often this crash happens ? are you able to reproduce it ?

The acc extra should work in the branch route, no problem. Out of curiosity, I 
will try to reproduce you case (timeout -> failure route -> t_relay -> 
branch_route) to see if I can reproduce it.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/13/2018 07:41 PM, Ben Newlin wrote:
Bogdan,

Yes, we are setting acc_extra variables in our branch routes, which are 
sometimes (but not always) called from failure route. Are acc_extra variables 
not available for use in branch_routes?

We don’t currently use drop_accounting anywhere in our script. If I call it 
before that branch_route then it will stop accounting for that call, right? We 
need to have accounting records for the call, so I’m not sure how that would 
resolve the issue?

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Tuesday, November 13, 2018 at 9:13 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for the info. The crash happens when you try to set an acc extra 
variable in branch route (when a creating a new branch via failure route, on 
timeout).

Now, do you use the drop accounting in your script ? and considering the above 
scenario, it is possible to have the drop acc before the branch route ?

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/12/2018 08:55 PM, Ben Newlin wrote:
Bogdan,

We upgraded to 2.4.3 and the crash reproduced today. Backtrace is available 
here: https://pastebin.com/CZxQnZdR<https://pastebin.com/CZxQnZdR>.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Wednesday, November 7, 2018 at 6:18 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The BT indicates a double free for the accounting context - and I noticed you 
use 2.4.1 version. And yes, there was an issue related to acc context, issue 
that was fixed starting 2.4.2. So, could you upgrade to the latest 2.4 and see 
if the crash still happens ? As I think the fix is already there.

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:
Jackpot - you get it right !! I will start digging into the trace, but please 
keep the corefile, I might need it later.

Thanks and regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/06/2018 10:24 PM, Ben Newlin wrote:
Bogdan,

I have reproduced this crash and verified this time that the flags were set.

$ opensips -V
version: opensips 2.4.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 5d042cffc
main.c compiled on 23:38:55 Nov  5 2018 with gcc 7

Backtrace is available here: 
https://pastebin.com/KTQjkCwq<https://pastebin.com/KTQjkCwq>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, November 1, 2018 at 1:19 PM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

According to the backtrace, the memory debugger was not activated. Do an 
"opensips -V" to check the resulting co

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

Yes, we are setting acc_extra variables in our branch routes, which are 
sometimes (but not always) called from failure route. Are acc_extra variables 
not available for use in branch_routes?

We don’t currently use drop_accounting anywhere in our script. If I call it 
before that branch_route then it will stop accounting for that call, right? We 
need to have accounting records for the call, so I’m not sure how that would 
resolve the issue?

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Tuesday, November 13, 2018 at 9:13 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for the info. The crash happens when you try to set an acc extra 
variable in branch route (when a creating a new branch via failure route, on 
timeout).

Now, do you use the drop accounting in your script ? and considering the above 
scenario, it is possible to have the drop acc before the branch route ?

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/12/2018 08:55 PM, Ben Newlin wrote:
Bogdan,

We upgraded to 2.4.3 and the crash reproduced today. Backtrace is available 
here: https://pastebin.com/CZxQnZdR<https://pastebin.com/CZxQnZdR>.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Wednesday, November 7, 2018 at 6:18 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The BT indicates a double free for the accounting context - and I noticed you 
use 2.4.1 version. And yes, there was an issue related to acc context, issue 
that was fixed starting 2.4.2. So, could you upgrade to the latest 2.4 and see 
if the crash still happens ? As I think the fix is already there.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:
Jackpot - you get it right !! I will start digging into the trace, but please 
keep the corefile, I might need it later.

Thanks and regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/06/2018 10:24 PM, Ben Newlin wrote:
Bogdan,

I have reproduced this crash and verified this time that the flags were set.

$ opensips -V
version: opensips 2.4.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 5d042cffc
main.c compiled on 23:38:55 Nov  5 2018 with gcc 7

Backtrace is available here: 
https://pastebin.com/KTQjkCwq<https://pastebin.com/KTQjkCwq>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, November 1, 2018 at 1:19 PM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

According to the backtrace, the memory debugger was not activated. Do an 
"opensips -V" to check the resulting compile flags - do you see the DBG_MALLOC 
and QM_MALLOC ?

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 10/31/2018 05:04 PM, Ben Newlin wrote:
Bogdan,

I was able to compile with those options and the crash has occurred again. 
Backtrace is here: https://pastebin.com/dezi9xUU<https://pastebin.com/dezi9xUU>.

Even though I had `memdump=1` set in my script, there was no extra memory 
debugging information in the logs prior to or at the time of the crash. I’m not 
sure if that is expected or not.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Monday, October 29, 2018 at 8:11 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

You can change the compile flags via the Makefile.conf file - the menuconfig is 
also 

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

How often this crash happens ? are you able to reproduce it ?

The acc extra should work in the branch route, no problem. Out of 
curiosity, I will try to reproduce you case (timeout -> failure route -> 
t_relay -> branch_route) to see if I can reproduce it.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/13/2018 07:41 PM, Ben Newlin wrote:


Bogdan,

Yes, we are setting acc_extra variables in our branch routes, which 
are sometimes (but not always) called from failure route. Are 
acc_extra variables not available for use in branch_routes?


We don’t currently use drop_accounting anywhere in our script. If I 
call it before that branch_route then it will stop accounting for that 
call, right? We need to have accounting records for the call, so I’m 
not sure how that would resolve the issue?


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Tuesday, November 13, 2018 at 9:13 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

Thanks for the info. The crash happens when you try to set an acc 
extra variable in branch route (when a creating a new branch via 
failure route, on timeout).


Now, do you use the drop accounting in your script ? and considering 
the above scenario, it is possible to have the drop acc before the 
branch route ?


Regards,


Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/12/2018 08:55 PM, Ben Newlin wrote:

Bogdan,

We upgraded to 2.4.3 and the crash reproduced today. Backtrace is
available here: https://pastebin.com/CZxQnZdR
<https://pastebin.com/CZxQnZdR>.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Wednesday, November 7, 2018 at 6:18 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The BT indicates a double free for the accounting context - and I
noticed you use 2.4.1 version. And yes, there was an issue related
to acc context, issue that was fixed starting 2.4.2. So, could you
upgrade to the latest 2.4 and see if the crash still happens ? As
I think the fix is already there.

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:

Jackpot - you get it right !! I will start digging into the
trace, but please keep the corefile, I might need it later.

Thanks and regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/06/2018 10:24 PM, Ben Newlin wrote:

Bogdan,

I have reproduced this crash and verified this time that
the flags were set.

$ opensips -V

version: opensips 2.4.1 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, QM_MALLOC, DBG_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535

poll method support: poll, epoll, sigio_rt, select.

git revision: 5d042cffc

main.c compiled on 23:38:55 Nov  5 2018 with gcc 7

Backtrace is available here: https://pastebin.com/KTQjkCwq
<https://pastebin.com/KTQjkCwq>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Thursday, November 1, 2018 at 1:19 PM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling
list 
<mailto:devel@lists.opensips.org>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

According to the backtrace, the memory debugger was not
activated. Do an "opensips -V" to check the resulting
compile flags - do you see the DBG_MALLOC and QM_MALLOC ?

Regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/31/2018 05:04 PM, Ben Newlin wro

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

Thanks for the info. The crash happens when you try to set an acc extra 
variable in branch route (when a creating a new branch via failure 
route, on timeout).


Now, do you use the drop accounting in your script ? and considering the 
above scenario, it is possible to have the drop acc before the branch 
route ?


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/12/2018 08:55 PM, Ben Newlin wrote:


Bogdan,

We upgraded to 2.4.3 and the crash reproduced today. Backtrace is 
available here: https://pastebin.com/CZxQnZdR.


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Wednesday, November 7, 2018 at 6:18 AM
*To: *OpenSIPS devel mailling list , Ben 
Newlin 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The BT indicates a double free for the accounting context - and I 
noticed you use 2.4.1 version. And yes, there was an issue related to 
acc context, issue that was fixed starting 2.4.2. So, could you 
upgrade to the latest 2.4 and see if the crash still happens ? As I 
think the fix is already there.


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:

Jackpot - you get it right !! I will start digging into the trace,
but please keep the corefile, I might need it later.

Thanks and regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/06/2018 10:24 PM, Ben Newlin wrote:

Bogdan,

I have reproduced this crash and verified this time that the
flags were set.

$ opensips -V

version: opensips 2.4.1 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT,
DBG_LOCK

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535

poll method support: poll, epoll, sigio_rt, select.

git revision: 5d042cffc

main.c compiled on 23:38:55 Nov  5 2018 with gcc 7

Backtrace is available here: https://pastebin.com/KTQjkCwq
<https://pastebin.com/KTQjkCwq>

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Thursday, November 1, 2018 at 1:19 PM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

According to the backtrace, the memory debugger was not
activated. Do an "opensips -V" to check the resulting compile
flags - do you see the DBG_MALLOC and QM_MALLOC ?

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/31/2018 05:04 PM, Ben Newlin wrote:

Bogdan,

I was able to compile with those options and the crash has
occurred again. Backtrace is here:
https://pastebin.com/dezi9xUU <https://pastebin.com/dezi9xUU>.

Even though I had `memdump=1` set in my script, there was
no extra memory debugging information in the logs prior to
or at the time of the crash. I’m not sure if that is
expected or not.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Monday, October 29, 2018 at 8:11 AM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling
list 
<mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

You can change the compile flags via the Makefile.conf
file - the menuconfig is also updating that file. So
during your build you can simply push a pre-modified
Makefile.conf file with the options needed for memory
debugging.

Regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/26/2018 05:14 PM, Ben Newlin wrote:

Bogdan,

Unfortunately, we have run into a similar issue
  

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

We upgraded to 2.4.3 and the crash reproduced today. Backtrace is available 
here: https://pastebin.com/CZxQnZdR.

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Wednesday, November 7, 2018 at 6:18 AM
To: OpenSIPS devel mailling list , Ben Newlin 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The BT indicates a double free for the accounting context - and I noticed you 
use 2.4.1 version. And yes, there was an issue related to acc context, issue 
that was fixed starting 2.4.2. So, could you upgrade to the latest 2.4 and see 
if the crash still happens ? As I think the fix is already there.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:
Jackpot - you get it right !! I will start digging into the trace, but please 
keep the corefile, I might need it later.

Thanks and regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 11/06/2018 10:24 PM, Ben Newlin wrote:
Bogdan,

I have reproduced this crash and verified this time that the flags were set.

$ opensips -V
version: opensips 2.4.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 5d042cffc
main.c compiled on 23:38:55 Nov  5 2018 with gcc 7

Backtrace is available here: 
https://pastebin.com/KTQjkCwq<https://pastebin.com/KTQjkCwq>

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Thursday, November 1, 2018 at 1:19 PM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

According to the backtrace, the memory debugger was not activated. Do an 
"opensips -V" to check the resulting compile flags - do you see the DBG_MALLOC 
and QM_MALLOC ?

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 10/31/2018 05:04 PM, Ben Newlin wrote:
Bogdan,

I was able to compile with those options and the crash has occurred again. 
Backtrace is here: https://pastebin.com/dezi9xUU<https://pastebin.com/dezi9xUU>.

Even though I had `memdump=1` set in my script, there was no extra memory 
debugging information in the logs prior to or at the time of the crash. I’m not 
sure if that is expected or not.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Monday, October 29, 2018 at 8:11 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

You can change the compile flags via the Makefile.conf file - the menuconfig is 
also updating that file. So during your build you can simply push a 
pre-modified Makefile.conf file with the options needed for memory debugging.

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 10/26/2018 05:14 PM, Ben Newlin wrote:
Bogdan,

Unfortunately, we have run into a similar issue before. Our build system is 
completely automated and there is no way to inject the `make menuconfig` 
interactive step into that process. If I were to be testing this locally I 
might be able to work something out, but I could never get such a build into 
our testing environment which is where the crashes are occurring.

Do you have instructions for enabling memory debugging that do not require 
using the interactive TUI tool? What does the menuconfig program do when these 
options are selected? Are there some defines or other settings we can change 
ourselves and bypass menuconfig?

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, October 26, 2018 at 4:59 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

all the BT's points to 

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

The BT indicates a double free for the accounting context - and I 
noticed you use 2.4.1 version. And yes, there was an issue related to 
acc context, issue that was fixed starting 2.4.2. So, could you upgrade 
to the latest 2.4 and see if the crash still happens ? As I think the 
fix is already there.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:
Jackpot - you get it right !! I will start digging into the trace, but 
please keep the corefile, I might need it later.


Thanks and regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 11/06/2018 10:24 PM, Ben Newlin wrote:


Bogdan,

I have reproduced this crash and verified this time that the flags 
were set.


$ opensips -V

version: opensips 2.4.1 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, 
QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK


ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535


poll method support: poll, epoll, sigio_rt, select.

git revision: 5d042cffc

main.c compiled on 23:38:55 Nov  5 2018 with gcc 7

Backtrace is available here: https://pastebin.com/KTQjkCwq 
<https://pastebin.com/KTQjkCwq>


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Thursday, November 1, 2018 at 1:19 PM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

According to the backtrace, the memory debugger was not activated. Do 
an "opensips -V" to check the resulting compile flags - do you see 
the DBG_MALLOC and QM_MALLOC ?


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/31/2018 05:04 PM, Ben Newlin wrote:

Bogdan,

I was able to compile with those options and the crash has
occurred again. Backtrace is here: https://pastebin.com/dezi9xUU
<https://pastebin.com/dezi9xUU>.

Even though I had `memdump=1` set in my script, there was no
extra memory debugging information in the logs prior to or at the
time of the crash. I’m not sure if that is expected or not.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Monday, October 29, 2018 at 8:11 AM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

You can change the compile flags via the Makefile.conf file - the
menuconfig is also updating that file. So during your build you
can simply push a pre-modified Makefile.conf file with the
options needed for memory debugging.

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/26/2018 05:14 PM, Ben Newlin wrote:

Bogdan,

Unfortunately, we have run into a similar issue before. Our
build system is completely automated and there is no way to
inject the `make menuconfig` interactive step into that
process. If I were to be testing this locally I might be able
to work something out, but I could never get such a build
into our testing environment which is where the crashes are
occurring.

Do you have instructions for enabling memory debugging that
do not require using the interactive TUI tool? What does the
menuconfig program do when these options are selected? Are
there some defines or other settings we can change ourselves
and bypass menuconfig?

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Friday, October 26, 2018 at 4:59 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

all the BT's points to crashes while doing memory ops. I
suspect a memory corruption that randomly triggers crashes in
different parts of the code.

Could you try to re-compile with memory debugging support ?
See
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem,
the "How to handle it" section.

Regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

   

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Jackpot - you get it right !! I will start digging into the trace, but 
please keep the corefile, I might need it later.


Thanks and regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 11/06/2018 10:24 PM, Ben Newlin wrote:


Bogdan,

I have reproduced this crash and verified this time that the flags 
were set.


$ opensips -V

version: opensips 2.4.1 (x86_64/linux)

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, 
QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK


ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535


poll method support: poll, epoll, sigio_rt, select.

git revision: 5d042cffc

main.c compiled on 23:38:55 Nov  5 2018 with gcc 7

Backtrace is available here: https://pastebin.com/KTQjkCwq 
<https://pastebin.com/KTQjkCwq>


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Thursday, November 1, 2018 at 1:19 PM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

According to the backtrace, the memory debugger was not activated. Do 
an "opensips -V" to check the resulting compile flags - do you see the 
DBG_MALLOC and QM_MALLOC ?


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/31/2018 05:04 PM, Ben Newlin wrote:

Bogdan,

I was able to compile with those options and the crash has
occurred again. Backtrace is here: https://pastebin.com/dezi9xUU
<https://pastebin.com/dezi9xUU>.

Even though I had `memdump=1` set in my script, there was no extra
memory debugging information in the logs prior to or at the time
of the crash. I’m not sure if that is expected or not.

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Monday, October 29, 2018 at 8:11 AM
*To: *Ben Newlin 
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

You can change the compile flags via the Makefile.conf file - the
menuconfig is also updating that file. So during your build you
can simply push a pre-modified Makefile.conf file with the options
needed for memory debugging.

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/26/2018 05:14 PM, Ben Newlin wrote:

Bogdan,

Unfortunately, we have run into a similar issue before. Our
build system is completely automated and there is no way to
inject the `make menuconfig` interactive step into that
process. If I were to be testing this locally I might be able
to work something out, but I could never get such a build into
our testing environment which is where the crashes are occurring.

Do you have instructions for enabling memory debugging that do
not require using the interactive TUI tool? What does the
menuconfig program do when these options are selected? Are
there some defines or other settings we can change ourselves
and bypass menuconfig?

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Friday, October 26, 2018 at 4:59 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

all the BT's points to crashes while doing memory ops. I
suspect a memory corruption that randomly triggers crashes in
different parts of the code.

Could you try to re-compile with memory debugging support ?
See
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem,
the "How to handle it" section.

Regards,



Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/24/2018 04:28 AM, Ben Newlin wrote:

We have had 2 more crashes today.

Crash 2: https://pastebin.com/rMruBQcZ
<https://pastebin.com/rMruBQcZ>

This crash appears to have occurred while processing an
initial INVITE request. I could not see anything unusual
about the request. I cannot tell if this crash is related
to the others.

Crash 3

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

I have reproduced this crash and verified this time that the flags were set.

$ opensips -V
version: opensips 2.4.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, 
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 5d042cffc
main.c compiled on 23:38:55 Nov  5 2018 with gcc 7

Backtrace is available here: https://pastebin.com/KTQjkCwq

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Thursday, November 1, 2018 at 1:19 PM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

According to the backtrace, the memory debugger was not activated. Do an 
"opensips -V" to check the resulting compile flags - do you see the DBG_MALLOC 
and QM_MALLOC ?

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 10/31/2018 05:04 PM, Ben Newlin wrote:
Bogdan,

I was able to compile with those options and the crash has occurred again. 
Backtrace is here: https://pastebin.com/dezi9xUU<https://pastebin.com/dezi9xUU>.

Even though I had `memdump=1` set in my script, there was no extra memory 
debugging information in the logs prior to or at the time of the crash. I’m not 
sure if that is expected or not.

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Monday, October 29, 2018 at 8:11 AM
To: Ben Newlin <mailto:ben.new...@genesys.com>, 
OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

You can change the compile flags via the Makefile.conf file - the menuconfig is 
also updating that file. So during your build you can simply push a 
pre-modified Makefile.conf file with the options needed for memory debugging.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 10/26/2018 05:14 PM, Ben Newlin wrote:
Bogdan,

Unfortunately, we have run into a similar issue before. Our build system is 
completely automated and there is no way to inject the `make menuconfig` 
interactive step into that process. If I were to be testing this locally I 
might be able to work something out, but I could never get such a build into 
our testing environment which is where the crashes are occurring.

Do you have instructions for enabling memory debugging that do not require 
using the interactive TUI tool? What does the menuconfig program do when these 
options are selected? Are there some defines or other settings we can change 
ourselves and bypass menuconfig?

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, October 26, 2018 at 4:59 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

all the BT's points to crashes while doing memory ops. I suspect a memory 
corruption that randomly triggers crashes in different parts of the code.

Could you try to re-compile with memory debugging support ? See 
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem<http://www.opensips.org/Documentation/TroubleShooting-OutOfMem>,
 the "How to handle it" section.

Regards,




Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 10/24/2018 04:28 AM, Ben Newlin wrote:
We have had 2 more crashes today.

Crash 2: https://pastebin.com/rMruBQcZ<https://pastebin.com/rMruBQcZ>
This crash appears to have occurred while processing an initial INVITE request. 
I could not see anything unusual about the request. I cannot tell if this crash 
is related to the others.

Crash 3: https://pastebin.com/Gmk1m4NT<https://pastebin.com/Gmk1m4NT>
This crash follows the pattern of the original crash I reported.

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Monday, October 22, 2018 at 4:45 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Here is a better trace of the call: 
https:/

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

According to the backtrace, the memory debugger was not activated. Do an 
"opensips -V" to check the resulting compile flags - do you see the 
DBG_MALLOC and QM_MALLOC ?


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/31/2018 05:04 PM, Ben Newlin wrote:


Bogdan,

I was able to compile with those options and the crash has occurred 
again. Backtrace is here: https://pastebin.com/dezi9xUU.


Even though I had `memdump=1` set in my script, there was no extra 
memory debugging information in the logs prior to or at the time of 
the crash. I’m not sure if that is expected or not.


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Monday, October 29, 2018 at 8:11 AM
*To: *Ben Newlin , OpenSIPS devel mailling 
list 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

You can change the compile flags via the Makefile.conf file - the 
menuconfig is also updating that file. So during your build you can 
simply push a pre-modified Makefile.conf file with the options needed 
for memory debugging.


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/26/2018 05:14 PM, Ben Newlin wrote:

Bogdan,

Unfortunately, we have run into a similar issue before. Our build
system is completely automated and there is no way to inject the
`make menuconfig` interactive step into that process. If I were to
be testing this locally I might be able to work something out, but
I could never get such a build into our testing environment which
is where the crashes are occurring.

Do you have instructions for enabling memory debugging that do not
require using the interactive TUI tool? What does the menuconfig
program do when these options are selected? Are there some defines
or other settings we can change ourselves and bypass menuconfig?

Ben Newlin

*From: *Bogdan-Andrei Iancu 
<mailto:bog...@opensips.org>
*Date: *Friday, October 26, 2018 at 4:59 AM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin
 <mailto:ben.new...@genesys.com>
    *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

all the BT's points to crashes while doing memory ops. I suspect a
memory corruption that randomly triggers crashes in different
parts of the code.

Could you try to re-compile with memory debugging support ? See
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem,
the "How to handle it" section.

Regards,


Bogdan-Andrei Iancu

  


OpenSIPS Founder and Developer

   http://www.opensips-solutions.com

OpenSIPS Bootcamp 2018

   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/24/2018 04:28 AM, Ben Newlin wrote:

We have had 2 more crashes today.

Crash 2: https://pastebin.com/rMruBQcZ
<https://pastebin.com/rMruBQcZ>

This crash appears to have occurred while processing an
initial INVITE request. I could not see anything unusual about
the request. I cannot tell if this crash is related to the others.

Crash 3: https://pastebin.com/Gmk1m4NT
<https://pastebin.com/Gmk1m4NT>

This crash follows the pattern of the original crash I reported.

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben
Newlin  <mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Date: *Monday, October 22, 2018 at 4:45 PM
*To: *OpenSIPS devel mailling list 
    <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Here is a better trace of the call:
https://pastebin.com/gWpQR8E7 <https://pastebin.com/gWpQR8E7>

Ben Newlin

*From: *Ben Newlin 
<mailto:ben.new...@genesys.com>
*Date: *Monday, October 22, 2018 at 4:34 PM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
*Subject: *OpenSIPS Crash

Hello,

We have been having sporadic crashes and I was recently able
to recover a core dump for one. I have uploaded it here:
https://pastebin.com/ABktcYcH <https://pastebin.com/ABktcYcH>.

I picked out a Call-ID from the crash data and took a look in
our tracing. I have uploaded it here:
https://pastebin.com/ZEzUUKZ5 <https://pastebin.com/ZEzUUKZ5>.

It appears that a downstream server was extremely lagged and
failed to respond to an INVITE. We sent the INVITE to another
server and the ca

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

I was able to compile with those options and the crash has occurred again. 
Backtrace is here: https://pastebin.com/dezi9xUU.

Even though I had `memdump=1` set in my script, there was no extra memory 
debugging information in the logs prior to or at the time of the crash. I’m not 
sure if that is expected or not.

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Monday, October 29, 2018 at 8:11 AM
To: Ben Newlin , OpenSIPS devel mailling list 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

You can change the compile flags via the Makefile.conf file - the menuconfig is 
also updating that file. So during your build you can simply push a 
pre-modified Makefile.conf file with the options needed for memory debugging.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 10/26/2018 05:14 PM, Ben Newlin wrote:
Bogdan,

Unfortunately, we have run into a similar issue before. Our build system is 
completely automated and there is no way to inject the `make menuconfig` 
interactive step into that process. If I were to be testing this locally I 
might be able to work something out, but I could never get such a build into 
our testing environment which is where the crashes are occurring.

Do you have instructions for enabling memory debugging that do not require 
using the interactive TUI tool? What does the menuconfig program do when these 
options are selected? Are there some defines or other settings we can change 
ourselves and bypass menuconfig?

Ben Newlin

From: Bogdan-Andrei Iancu <mailto:bog...@opensips.org>
Date: Friday, October 26, 2018 at 4:59 AM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>, Ben Newlin 
<mailto:ben.new...@genesys.com>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

all the BT's points to crashes while doing memory ops. I suspect a memory 
corruption that randomly triggers crashes in different parts of the code.

Could you try to re-compile with memory debugging support ? See 
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem<http://www.opensips.org/Documentation/TroubleShooting-OutOfMem>,
 the "How to handle it" section.

Regards,



Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 10/24/2018 04:28 AM, Ben Newlin wrote:
We have had 2 more crashes today.

Crash 2: https://pastebin.com/rMruBQcZ<https://pastebin.com/rMruBQcZ>
This crash appears to have occurred while processing an initial INVITE request. 
I could not see anything unusual about the request. I cannot tell if this crash 
is related to the others.

Crash 3: https://pastebin.com/Gmk1m4NT<https://pastebin.com/Gmk1m4NT>
This crash follows the pattern of the original crash I reported.

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Monday, October 22, 2018 at 4:45 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Here is a better trace of the call: 
https://pastebin.com/gWpQR8E7<https://pastebin.com/gWpQR8E7>

Ben Newlin

From: Ben Newlin <mailto:ben.new...@genesys.com>
Date: Monday, October 22, 2018 at 4:34 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: OpenSIPS Crash

Hello,

We have been having sporadic crashes and I was recently able to recover a core 
dump for one. I have uploaded it here: 
https://pastebin.com/ABktcYcH<https://pastebin.com/ABktcYcH>.

I picked out a Call-ID from the crash data and took a look in our tracing. I 
have uploaded it here: 
https://pastebin.com/ZEzUUKZ5<https://pastebin.com/ZEzUUKZ5>.

It appears that a downstream server was extremely lagged and failed to respond 
to an INVITE. We sent the INVITE to another server and the call was connected, 
but then eventually the original server “caught up” and sent a burst of 200 OK 
responses. The crash seems to have occurred processing the ACK to one of these 
responses.

Ben Newlin





___

Devel mailing list

Devel@lists.opensips.org<mailto:Devel@lists.opensips.org>

http://lists.opensips.org/cgi-bin/mailman/listinfo/devel<http://lists.opensips.org/cgi-bin/mailman/listinfo/devel>





___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

You can change the compile flags via the Makefile.conf file - the 
menuconfig is also updating that file. So during your build you can 
simply push a pre-modified Makefile.conf file withthe options needed for 
memory debugging.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/26/2018 05:14 PM, Ben Newlin wrote:


Bogdan,

Unfortunately, we have run into a similar issue before. Our build 
system is completely automated and there is no way to inject the `make 
menuconfig` interactive step into that process. If I were to be 
testing this locally I might be able to work something out, but I 
could never get such a build into our testing environment which is 
where the crashes are occurring.


Do you have instructions for enabling memory debugging that do not 
require using the interactive TUI tool? What does the menuconfig 
program do when these options are selected? Are there some defines or 
other settings we can change ourselves and bypass menuconfig?


Ben Newlin

*From: *Bogdan-Andrei Iancu 
*Date: *Friday, October 26, 2018 at 4:59 AM
*To: *OpenSIPS devel mailling list , Ben 
Newlin 

*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

all the BT's points to crashes while doing memory ops. I suspect a 
memory corruption that randomly triggers crashes in different parts of 
the code.


Could you try to re-compile with memory debugging support ? See 
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem, the 
"How to handle it" section.


Regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/24/2018 04:28 AM, Ben Newlin wrote:

We have had 2 more crashes today.

Crash 2: https://pastebin.com/rMruBQcZ <https://pastebin.com/rMruBQcZ>

This crash appears to have occurred while processing an initial
INVITE request. I could not see anything unusual about the
request. I cannot tell if this crash is related to the others.

Crash 3: https://pastebin.com/Gmk1m4NT <https://pastebin.com/Gmk1m4NT>

This crash follows the pattern of the original crash I reported.

Ben Newlin

*From: *Devel 
<mailto:devel-boun...@lists.opensips.org> on behalf of Ben Newlin
 <mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
 <mailto:devel@lists.opensips.org>
*Date: *Monday, October 22, 2018 at 4:45 PM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Here is a better trace of the call: https://pastebin.com/gWpQR8E7
<https://pastebin.com/gWpQR8E7>

Ben Newlin

*From: *Ben Newlin 
<mailto:ben.new...@genesys.com>
*Date: *Monday, October 22, 2018 at 4:34 PM
*To: *OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
*Subject: *OpenSIPS Crash

Hello,

We have been having sporadic crashes and I was recently able to
recover a core dump for one. I have uploaded it here:
https://pastebin.com/ABktcYcH <https://pastebin.com/ABktcYcH>.

I picked out a Call-ID from the crash data and took a look in our
tracing. I have uploaded it here: https://pastebin.com/ZEzUUKZ5
<https://pastebin.com/ZEzUUKZ5>.

It appears that a downstream server was extremely lagged and
failed to respond to an INVITE. We sent the INVITE to another
server and the call was connected, but then eventually the
original server “caught up” and sent a burst of 200 OK responses.
The crash seems to have occurred processing the ACK to one of
these responses.

Ben Newlin




___

Devel mailing list

Devel@lists.opensips.org <mailto:Devel@lists.opensips.org>

http://lists.opensips.org/cgi-bin/mailman/listinfo/devel





___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Bogdan,

Unfortunately, we have run into a similar issue before. Our build system is 
completely automated and there is no way to inject the `make menuconfig` 
interactive step into that process. If I were to be testing this locally I 
might be able to work something out, but I could never get such a build into 
our testing environment which is where the crashes are occurring.

Do you have instructions for enabling memory debugging that do not require 
using the interactive TUI tool? What does the menuconfig program do when these 
options are selected? Are there some defines or other settings we can change 
ourselves and bypass menuconfig?

Ben Newlin

From: Bogdan-Andrei Iancu 
Date: Friday, October 26, 2018 at 4:59 AM
To: OpenSIPS devel mailling list , Ben Newlin 

Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

all the BT's points to crashes while doing memory ops. I suspect a memory 
corruption that randomly triggers crashes in different parts of the code.

Could you try to re-compile with memory debugging support ? See 
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem<http://www.opensips.org/Documentation/TroubleShooting-OutOfMem>,
 the "How to handle it" section.

Regards,


Bogdan-Andrei Iancu



OpenSIPS Founder and Developer

  http://www.opensips-solutions.com<http://www.opensips-solutions.com>

OpenSIPS Bootcamp 2018

  
http://opensips.org/training/OpenSIPS_Bootcamp_2018/<http://opensips.org/training/OpenSIPS_Bootcamp_2018/>
On 10/24/2018 04:28 AM, Ben Newlin wrote:
We have had 2 more crashes today.

Crash 2: https://pastebin.com/rMruBQcZ<https://pastebin.com/rMruBQcZ>
This crash appears to have occurred while processing an initial INVITE request. 
I could not see anything unusual about the request. I cannot tell if this crash 
is related to the others.

Crash 3: https://pastebin.com/Gmk1m4NT<https://pastebin.com/Gmk1m4NT>
This crash follows the pattern of the original crash I reported.

Ben Newlin

From: Devel 
<mailto:devel-boun...@lists.opensips.org> on 
behalf of Ben Newlin <mailto:ben.new...@genesys.com>
Reply-To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Date: Monday, October 22, 2018 at 4:45 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Here is a better trace of the call: 
https://pastebin.com/gWpQR8E7<https://pastebin.com/gWpQR8E7>

Ben Newlin

From: Ben Newlin <mailto:ben.new...@genesys.com>
Date: Monday, October 22, 2018 at 4:34 PM
To: OpenSIPS devel mailling list 
<mailto:devel@lists.opensips.org>
Subject: OpenSIPS Crash

Hello,

We have been having sporadic crashes and I was recently able to recover a core 
dump for one. I have uploaded it here: 
https://pastebin.com/ABktcYcH<https://pastebin.com/ABktcYcH>.

I picked out a Call-ID from the crash data and took a look in our tracing. I 
have uploaded it here: 
https://pastebin.com/ZEzUUKZ5<https://pastebin.com/ZEzUUKZ5>.

It appears that a downstream server was extremely lagged and failed to respond 
to an INVITE. We sent the INVITE to another server and the call was connected, 
but then eventually the original server “caught up” and sent a burst of 200 OK 
responses. The crash seems to have occurred processing the ACK to one of these 
responses.

Ben Newlin




___

Devel mailing list

Devel@lists.opensips.org<mailto:Devel@lists.opensips.org>

http://lists.opensips.org/cgi-bin/mailman/listinfo/devel<http://lists.opensips.org/cgi-bin/mailman/listinfo/devel>


___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Bogdan-Andrei Iancu]

Hi Ben,

all the BT's points to crashes while doing memory ops. I suspect a 
memory corruption that randomly triggers crashes in different parts of 
the code.


Could you try to re-compile with memory debugging support ? See 
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem, the "How 
to handle it" section.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/24/2018 04:28 AM, Ben Newlin wrote:


We have had 2 more crashes today.

Crash 2: https://pastebin.com/rMruBQcZ <https://pastebin.com/rMruBQcZ>

This crash appears to have occurred while processing an initial INVITE 
request. I could not see anything unusual about the request. I cannot 
tell if this crash is related to the others.


Crash 3: https://pastebin.com/Gmk1m4NT <https://pastebin.com/Gmk1m4NT>

This crash follows the pattern of the original crash I reported.

Ben Newlin

*From: *Devel  on behalf of Ben 
Newlin 

*Reply-To: *OpenSIPS devel mailling list 
*Date: *Monday, October 22, 2018 at 4:45 PM
*To: *OpenSIPS devel mailling list 
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash

Here is a better trace of the call: https://pastebin.com/gWpQR8E7

Ben Newlin

*From: *Ben Newlin 
*Date: *Monday, October 22, 2018 at 4:34 PM
*To: *OpenSIPS devel mailling list 
*Subject: *OpenSIPS Crash

Hello,

We have been having sporadic crashes and I was recently able to 
recover a core dump for one. I have uploaded it here: 
https://pastebin.com/ABktcYcH.


I picked out a Call-ID from the crash data and took a look in our 
tracing. I have uploaded it here: https://pastebin.com/ZEzUUKZ5.


It appears that a downstream server was extremely lagged and failed to 
respond to an INVITE. We sent the INVITE to another server and the 
call was connected, but then eventually the original server “caught 
up” and sent a burst of 200 OK responses. The crash seems to have 
occurred processing the ACK to one of these responses.


Ben Newlin



___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel


___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

We have had 2 more crashes today.

Crash 2: https://pastebin.com/rMruBQcZ
This crash appears to have occurred while processing an initial INVITE request. 
I could not see anything unusual about the request. I cannot tell if this crash 
is related to the others.

Crash 3: https://pastebin.com/Gmk1m4NT
This crash follows the pattern of the original crash I reported.

Ben Newlin

From: Devel  on behalf of Ben Newlin 

Reply-To: OpenSIPS devel mailling list 
Date: Monday, October 22, 2018 at 4:45 PM
To: OpenSIPS devel mailling list 
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Here is a better trace of the call: https://pastebin.com/gWpQR8E7

Ben Newlin

From: Ben Newlin 
Date: Monday, October 22, 2018 at 4:34 PM
To: OpenSIPS devel mailling list 
Subject: OpenSIPS Crash

Hello,

We have been having sporadic crashes and I was recently able to recover a core 
dump for one. I have uploaded it here: https://pastebin.com/ABktcYcH.

I picked out a Call-ID from the crash data and took a look in our tracing. I 
have uploaded it here: https://pastebin.com/ZEzUUKZ5.

It appears that a downstream server was extremely lagged and failed to respond 
to an INVITE. We sent the INVITE to another server and the call was connected, 
but then eventually the original server “caught up” and sent a burst of 200 OK 
responses. The crash seems to have occurred processing the ACK to one of these 
responses.

Ben Newlin
___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Here is a better trace of the call: https://pastebin.com/gWpQR8E7

Ben Newlin

From: Ben Newlin 
Date: Monday, October 22, 2018 at 4:34 PM
To: OpenSIPS devel mailling list 
Subject: OpenSIPS Crash

Hello,

We have been having sporadic crashes and I was recently able to recover a core 
dump for one. I have uploaded it here: https://pastebin.com/ABktcYcH.

I picked out a Call-ID from the crash data and took a look in our tracing. I 
have uploaded it here: https://pastebin.com/ZEzUUKZ5.

It appears that a downstream server was extremely lagged and failed to respond 
to an INVITE. We sent the INVITE to another server and the call was connected, 
but then eventually the original server “caught up” and sent a burst of 200 OK 
responses. The crash seems to have occurred processing the ACK to one of these 
responses.

Ben Newlin
___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] OpenSIPS Crash

[Ben Newlin]

Hello,

We have been having sporadic crashes and I was recently able to recover a core 
dump for one. I have uploaded it here: https://pastebin.com/ABktcYcH.

I picked out a Call-ID from the crash data and took a look in our tracing. I 
have uploaded it here: https://pastebin.com/ZEzUUKZ5.

It appears that a downstream server was extremely lagged and failed to respond 
to an INVITE. We sent the INVITE to another server and the call was connected, 
but then eventually the original server “caught up” and sent a burst of 200 OK 
responses. The crash seems to have occurred processing the ACK to one of these 
responses.

Ben Newlin
___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] Crash on MySQL ACC Insert (#821)

[Bogdan Andrei IANCU]

Hi @apsaras , please provide me, off line, the access to investigate. 

Can you reproduce the crash ?

Regards, Bogdan

---
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/821#issuecomment-197553545___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] Crash on MySQL ACC Insert (#821)

[Bogdan Andrei IANCU]

Closed #821 via d837ed865db8dbcf09ce5eac0ce0f160c2ec0e55.

---
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/821#event-595832446___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] Crash on MySQL ACC Insert (#821)

[Bogdan Andrei IANCU]

Thank you @apsaras  for the help - the bug was found and currently @apsaras is 
testing the fix.

---
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/821#issuecomment-198308415___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] [opensips] Crash on MySQL ACC Insert (#821)

[apsaras]

Hello

I get few core dumps per day on a pre-production system based on latest 2.1.2 
git revision: 6d126a1.

OS: Debian 7 64bit
MySQL: 5.5.47
Installed on VMWare.

#0  0x7fc043b0c789 in db_mysql_val2bind (v=v@entry=0x7fc041a33da0 
,
binds=binds@entry=0x7fc043fae078, i=i@entry=21) at val.c:295
#1  0x7fc043b12e2a in db_mysql_do_prepared_query 
(conn=conn@entry=0x7fc043d6b790,
v=v@entry=0x7fc041a33b00 , n=n@entry=23, uv=uv@entry=0x0, 
un=un@entry=0,
query=0x7fc043d2c9c0 ) at dbase.c:676
#2  0x7fc043b18518 in db_mysql_insert (_h=0x7fc043d6b790, _k=0x7fc041a35180 
,
_v=0x7fc041a33b00 , _n=23) at dbase.c:1265
#3  0x7fc0417fb5fa in acc_db_request (rq=rq@entry=0x7fc04233be80 
,
rpl=rpl@entry=0x, ins_list=ins_list@entry=0x7fc041a364d8 
, cdr_flag=2)
at acc.c:638
#4  0x7fc04180a06d in on_missed (code=, 
reply=0x,
req=0x7fc04233be80 , t=) at acc_logic.c:467
#5  tmcb_func (t=, type=, ps=) at 
acc_logic.c:696
#6  0x7fc0421152e6 in run_trans_callbacks (type=type@entry=32, 
trans=trans@entry=0x7fbfbb6533e0,
req=req@entry=0x7fc04233be80 , rpl=, 
code=) at t_hooks.c:209
#7  0x7fc0420cf5a2 in run_failure_handlers (t=0x7fbfbb6533e0) at 
t_reply.c:569
#8  t_should_relay_response (reply=0x, 
cancel_bitmap=0x7fc043d35b58,
should_relay=, should_store=, 
branch=, new_code=408,
Trans=0x7fbfbb6533e0) at t_reply.c:911
#9  relay_reply (t=t@entry=0x7fbfbb6533e0, 
p_msg=p_msg@entry=0x, branch=,
msg_status=msg_status@entry=408, 
cancel_bitmap=cancel_bitmap@entry=0x7fffebfaaa28) at t_reply.c:1125
#10 0x7fc04212661a in fake_reply (code=408, branch=, 
t=0x7fbfbb6533e0) at timer.c:260
#11 final_response_handler (fr_tl=) at timer.c:391
#12 timer_routine (ticks=, set=) at timer.c:1066
#13 0x004a15a1 in handle_timer_job () at timer.c:565
#14 0x0058dc0f in handle_io (fm=0x7fc043d782b0, fm=0x7fc043d782b0, 
fm=0x7fc043d782b0,
idx=, event_type=2) at net/net_udp.c:265
#15 io_wait_loop_epoll (h=, t=, repeat=) at net/../io_wait_loop.h:215
#16 udp_rcv_loop (si=si@entry=0x7fc043d35f68) at net/net_udp.c:308
#17 0x0058f837 in udp_start_processes (chd_rank=chd_rank@entry=0x84a314 
,
startup_done=startup_done@entry=0x0) at net/net_udp.c:448
#18 0x0041ad73 in main_loop () at main.c:731
#19 main (argc=, argv=) at main.c:1280

I can provide access to the box if required.

Thank you in advanse.


---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/821___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash on tm/timer.c (#723)

[Bogdan Andrei IANCU]

The fix is available on GIT repo. @46labs please update if the fix holds for you

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/723#issuecomment-171277652___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash on tm/timer.c (#723)

[Ovidiu Sas]

It is optimized out
```
(gdb) bt full
#0  0x7f3f5e0c93bd in check_and_split_time_list (uticks=122135690, 
set=) at timer.c:809
tl = 0x0
end = 0x7f3f5812b858
ret = 
#1  utimer_routine (uticks=122135690, set=) at 
timer.c:1075
tl = 
tmp_tl = 
id = 
__FUNCTION__ = "utimer_routine"
#2  0x004c8f44 in handle_timer_job () at timer.c:557
t = 0x7f3f5812b920
l = 
__FUNCTION__ = "handle_timer_job"
#3  0x0056cbca in handle_io (si=) at 
net/net_udp.c:265
read = 0
#4  io_wait_loop_epoll (si=) at ./io_wait_loop.h:190
e = 
ret = 
n = 1
r = 1
```

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/723#issuecomment-169325793___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash on tm/timer.c (#723)

[Bogdan Andrei IANCU]

@ovidiusas , in your crash, is frame 1 utimer_routine too ? if yes, can you 
check the id you have there ?

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/723#issuecomment-169319687___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash on tm/timer.c (#723)

[Bogdan Andrei IANCU]

thanks @ovidiusas 

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/723#issuecomment-169343561___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] [opensips] crash in dlg_hash +791 (#730)

[46 Labs LLC]

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7fa0585613ea in unref_dlg (dlg=0x7fa0609fcea8, cnt=1) at 
dlg_hash.c:791
791 d_entry = &(d_table->entries[dlg->h_entry]);
Traceback (most recent call last):
  File 
"/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", 
line 63, in 
from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
(gdb) bt full
#0  0x7fa0585613ea in unref_dlg (dlg=0x7fa0609fcea8, cnt=1) at 
dlg_hash.c:791
d_entry = 
__FUNCTION__ = "unref_dlg"
#1  0x7fa05c159b48 in empty_tmcb_list (head=head@entry=0x7fa06340abc0) at 
t_hooks.c:53
cbp = 0x0
cbp_tmp = 0x7fa05ed127a0
#2  0x7fa05c13bed5 in free_cell (dead_cell=0x7fa06340ab50) at h_table.c:127
b = 
i = 
rpl = 
tt = 
foo = 
p = 
#3  0x7fa05c13cefa in free_hash_table () at h_table.c:375
No locals.
#4  0x7fa05c153a9b in tm_shutdown () at t_funcs.c:91
__FUNCTION__ = "tm_shutdown"
#5  0x004c350a in destroy_modules () at sr_module.c:527
No locals.
#6  0x0043fd1a in cleanup (show_status=show_status@entry=1) at 
main.c:308
__FUNCTION__ = "cleanup"
#7  0x00440768 in handle_sigs () at main.c:503
chld = 0
chld_status = 139
overall_status = 139
i = 
do_exit = 
__FUNCTION__ = "handle_sigs"
#8  0x004192ea in main_loop () at main.c:746
startup_done = 0x0
chd_rank = 232
#9  main (argc=, argv=) at main.c:1250
cfg_log_stderr = 
---Type  to continue, or q  to quit---
cfg_stream = 
c = 
r = 
tmp = 0x7ffe0bca6e8b ""
tmp_len = 
port = 
proto = 
options = 0x5c0470 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
seed = 1918274510
__FUNCTION__ = "main"

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/730___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash on tm/timer.c (#723)

[Răzvan Crainea]

I noticed there is some code already there that tries to debug this issue. Can 
you please try to compile the ```tm``` module with the ```TM_TIMER_DEBUG``` 
flag set?

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/723#issuecomment-164814684___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash on tm/timer.c (#723)

[46 Labs LLC]

I can confirm this is happening in 2.2 as well.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/723#issuecomment-164775504___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash on tm/timer.c (#723)

[46 Labs LLC]

BT from 2.2

(gdb) bt full #0 0×7f5800362cc9 in check_and_split_time_list 
(timer_list=0×7f58017dec50, time=time@entry=4717000) at timer.c:809 tl = 
0×0 end = 0×7f58017dec80 ret =  #1 0×7f5800365751 in 
utimer_routine (uticks=4717000, set=) at timer.c:1075 tl = 
 tmp_tl =  id = 6 FUNCTION = “utimer_routine” #2 
0×004d22b2 in handle_timer_job () at timer.c:557 t = 0×7f58017dedb8 
FUNCTION = “handle_timer_job” #3 0×005957fe in handle_io 
(fm=0×7f5a01d1e668, idx=idx@entry=3, event_type=event_type@entry=1) at 
net/net_tcp_proc.c:117 ret = 0 n =  con =  s = 9 
rw =  resp =  response = {140015960231216, 1} 
FUNCTION = “handle_io” #4 0×005975d0 in io_wait_loop_epoll 
(h=, t=, repeat=) at 
net/../io_wait_loop.h:211 ret = 1 e =  n = 1 r = 3 #5 
tcp_worker_proc (unix_sock=, max_fd=) at 
net/net_tcp_proc.c:312 FUNCTION = “tcp_worker_proc” #6 0×0058ebe9 in 
tcp_start_processes (chd_rank=chd_rank@entry=0×81fb80 , 
startup_done=startup_done@entry=0×7f5801926a18) at net/net_tcp.c:1758 r = 

—-Type  to continue, or q  to quit—- reader_fd = {46, 47} pid = 
0 si =  load_p = 0×7f5801928058 FUNCTION = “tcp_start_processes” 
#7 0×00419247 in main_loop () at main.c:718 startup_done = 
0×7f5801926a18 chd_rank = 25 #8 main (argc=, argv=) at main.c:1250 cfg_log_stderr =  cfg_stream =  c =  r =  tmp = 0×7fff7b449e36 “” tmp_len = 
 port =  proto =  options = 
0×5c0470 “f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:” seed = 2148737755 
FUNCTION = “main”

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/723#issuecomment-164801074___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash if initial db connection is failing (#683)

[Ionut Ionita]

Closed #683.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/683#event-463703046___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash if initial db connection is failing (#683)

[Ionut Ionita]

fixed with 
https://github.com/OpenSIPS/opensips/commit/c4ebfc87fea74bcb3ab7b544e02fd5f68edf6e54

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/683#issuecomment-156463414___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] [opensips] crash if initial db connection is failing (#683)

[Ovidiu Sas]

latest opensips 2.1
tcp_start_process() is called with startup_done set to NULL and then when 
init_child() failed, an initialization to -1 is attempted.

```
(gdb) bt full
#0  tcp_start_processes (chd_rank=chd_rank@entry=0x84d7d0 , 
startup_done=startup_done@entry=0x0) at net/net_tcp.c:1723
r = 
reader_fd = {15, 16}
pid = 
si = 
load_p = 0x7fb12d4494a8
__FUNCTION__ = "tcp_start_processes"
#1  0x00419888 in main_loop () at main.c:728
startup_done = 0x0
chd_rank = 2
#2  main (argc=, argv=) at main.c:1259
cfg_log_stderr = 
cfg_stream = 
c = 
r = 0
tmp = 0x7ffceb4a687f ""
tmp_len = 
port = 
proto = 
options = 0x5d0c08 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
seed = 3932613069
rfd = 
__FUNCTION__ = "main"
(gdb) print startup_done
$1 = (int *) 0x0
```

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/683___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] OpenSIPS crash

[Stas Kobzar]

Hello OpenSIPS developers,

We had OpenSIPS crashed. Please, find attached crash full stack trace from
core file attached.
OpenSIPS version:
version: opensips 1.11.5-notls (x86_64/linux)
flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM,
SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
git revision: 02a7c06
main.c compiled on 18:04:23 Jul 16 2015 with gcc 4.4.7

I am not sure, but it looks like the problem is MI command ul_add. We use
datagram  MI to add/remove permanent user locations via web interface.

Before we were using XML-rpc and we had similar problem. That's why we
moved to datagram.

Please, advice.
Thank you!

-- 

Stas Kobzar

Developeur VoIP / VoIP Developer

___

Modulis­.ca Inc.

# Bureau / Office: 514-284-2020 x 246

Email: s tas.kob...@modulis.ca

https://www.modulis.com


(gdb) bt full
#0  0x7f6ca92e09ef in mem_update_ucontact (_c=0x7f6c84cdc6d8, 
_ci=0x7fff18013e70) at ucontact.c:262
ptr = 
__FUNCTION__ = "mem_update_ucontact"
#1  0x7f6ca92e39d2 in update_ucontact (_r=0x7f6c84d16b58, 
_c=0x7f6c84cdc6d8, _ci=0x7fff18013e70, is_replicated=0 '\000') at ucontact.c:923
ret = 
__FUNCTION__ = "update_ucontact"
#2  0x7f6ca92eb201 in mi_usrloc_add (cmd=, 
param=) at ul_mi.c:542
ci = {received = {s = 0x0, len = 0}, path = 0x0, expires = 0, q = 1000, 
instance = {s = 0x0, len = 0}, callid = 0x0, cseq = 1, flags = 0, cflags = 0, 
user_agent = 0x7f6ca94fbac0, sock = 0x0, methods = 4294967295, last_modified = 
0, attr = 0x0}
r = 0x7f6c84d16b58
c = 0x7f6c84cdc6d8
node = 
dom = 0x7f6c84c21cc8
aor = 0x7f6cabb0b440
contact = 
ui_val = 
n = 
#3  0x7f6ca9eeb241 in run_mi_cmd (rx_sock=7, tx_sock=7) at ../../mi/mi.h:109
ret = 
#4  mi_datagram_server (rx_sock=7, tx_sock=7) at datagram_fnc.c:520
mi_cmd = 
mi_rpl = 
hdl = 0x0
f = 0x7f6cabac4188
dtgram = {start = 0x7f6cabafb270 ":ul_add", current = 0x7f6cabafb2cc 
"", len = 0}
ret = 
len = 
__FUNCTION__ = "mi_datagram_server"
#5  0x7f6ca9eecd83 in datagram_process (rank=0) at mi_datagram.c:311
__FUNCTION__ = "datagram_process"
#6  0x004af269 in start_module_procs () at sr_module.c:586
m = 
n = 
l = 
x = 
__FUNCTION__ = "start_module_procs"
#7  0x0043a5b0 in main_loop (argc=, argv=) at main.c:852
---Type  to continue, or q  to quit---
i = 
pid = 
si = 0x0
startup_done = 0x0
chd_rank = 0
rc = 
load_p = 0x0
#8  main (argc=, argv=) at main.c:1612
cfg_log_stderr = 
cfg_stream = 
c = 
r = 0
tmp = 0x7fff18014e5e ""
tmp_len = 
port = 
proto = 
options = 0x576ca0 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
ret = -1
seed = 933295776
rfd = 
__FUNCTION__ = "main"
(gdb)___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[Bogdan Andrei IANCU]

@dbeskoek , I sent you on last Friday, by email, a patch to be tested. Any luck 
with that ?

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589#issuecomment-129476611___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[dbeskoek]

sorry, for the delay. I did several tests with the patch, all of them were 
fine, no crash appears. I reverted to GIT-version and the crash occurs again. 
Patch is working :-)

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589#issuecomment-129483061___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[Bogdan Andrei IANCU]

Great @dbeskoek , especially that the faulty line was the log line itself, not 
the useful code. I will commit and backport asap
Thanks and regards, Bogdan

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589#issuecomment-129484192___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[Bogdan Andrei IANCU]

Closed #589.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589#event-378117425___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[Bogdan Andrei IANCU]

@dbeskoek , could you do a list in gdb, to see what is your line 766 ? In my 
case it is a LM_WARN : 
LM_WARN(alias already present and points to another connection 
(%d : %d and %d : %d)\n, a-parent-id,  port, c-id, port);

Is this correct ?

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589#issuecomment-128388030___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[dbeskoek]

thats correct:
```
761 LM_ERR(no connection found for id %d\n,id);
762 return -1;
763 error_sec:
764 TCPCONN_UNLOCK(id);
765 LM_WARN(possible port hijack attempt\n);
766 LM_WARN(alias already present and points to another connection 

767 (%d : %d and %d : %d)\n, a-parent-id,  
port, c-id, port);
768 return -1;
769 }
770
```

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589#issuecomment-128392028___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[Bogdan Andrei IANCU]

Is there any chance to get direct access to the corefile (ssh?) the debugging 
gets a bit more complex at this stage.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589#issuecomment-128424797___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[Bogdan Andrei IANCU]

ok, try printing in gdb the vars: a,  a-parent, c


---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589#issuecomment-128418538___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[dbeskoek]

```
(gdb) p a
$1 = (struct tcp_conn_alias *) 0x7f31115f35d8
(gdb) p a-parent
$2 = (struct tcp_connection *) 0x0
(gdb) p c
$3 = (struct tcp_connection *) 0x7f311a308470
```

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589#issuecomment-128420950___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[dbeskoek]

During running a sipp-scenario against an opensips2.1  doing tls-udp 
traversing we are able to easily crash this opensips within seconds. As soon as 
we disable tcp_accept_aliases the crashes are gone. We are heavily generating 
new tls-sessions (around 1000 per second), so we will reuse source-ports. Maybe 
this forces this segfault.

gdb(bt full):
#0  0x0059a2bf in tcpconn_add_alias (id=error reading variable: Cannot 
access memory at address 0x7ffefb7b78b8, port=5067,
port@entry=error reading variable: Cannot access memory at address 
0x7ffefb7b78b8, proto=error reading variable: Cannot access memory at address 
0x7ffefb7b78b8) at net/net_tcp.c:766
c = 0x7ff422570658
hash = 0
a = optimized out
__FUNCTION__ = tcpconn_add_alias

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash issue in tcpconn_add_alias (2.1) (#589)

[dbeskoek]

Hi Bogdan, yes, latest 2.1code from GIT was used. requested output:

```
#0  0x0052247a in tcpconn_add_alias (id=1358, port=5063, proto=3) at 
net/net_tcp.c:766
c = 0x7f311a308470
hash = 938
a = 0x7f31115f35d8
__FUNCTION__ = tcpconn_add_alias
```

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/589#issuecomment-128135696___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] Crash with poorly given var (#529)

[Liviu Chircu]

Closed #529.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/529#event-319737566___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] [opensips] Crash with poorly given var (#529)

[Liviu Chircu]

Although core variables correctly handle this error, OpenSIPS will segfault 
when a module variable is given instead:

startup_route
{
#works
$(ru(foo)) = foo;

#crashes
$(DLG_dir(foo)) = 5;

line 9:  5074 Segmentation fault  (core dumped) ./opensips -m 64 -M 64 -w . 
-f ...


---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/529___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] Crash with nested $param variables (#412)

[Liviu Chircu]

Closed #412 via f3ed4db9903d85cba7d7d491c931b550e29932d1.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/412#event-245330670___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] [opensips] Crash with nested $param variables (#412)

[Liviu Chircu]

Present in OpenSIPS 1.9+.

route [route_b]
{
# Evaluating $param(1) leads to infinite recursion!
xlog(This should not crash: $param(1)\n);
}

route [route_a]
{
route(route_b, $param(1));
}

route
{
route(route_a, test);
...
}

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/412___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash in v1.11.3 using fifo pres_phtable_list (#382)

[Carlos Oliva]

Hi @bogdan-iancu you are right, I only need pua for pua_dialoginfo and 
pua_usrloc. Include presence modules was my mistake, are not necesary, thanks 
for the point.
Nowadays i think your patch is essential because avoid the crash, I will apply 
it.

Thank you so much

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/382#issuecomment-61609590___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash in v1.11.3 using fifo pres_phtable_list (#382)

[Carlos Oliva]

Closed #382.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/382#event-187888632___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] [opensips] crash in v1.11.3 using fifo pres_phtable_list (#382)

[telematico]

Hello Devels:

Making some tests with version 1.11.3 I've just discover how to casue a crash 
simply calling pres_phtable_list using fifo (opensipsctl fifo 
pres_phtable_list).

I execute this command in a proxy that is not acting as presence server, but 
must have the presence modules loaded because is using pua_dialoginfo to set 
dialoginfo to real presence server (another opensips).

I asume the correct behaviour in this proxy should be not return anything  
(because it really don't have anything in the tables) but never a crash

The related modules config is:

modparam(presence, mix_dialog_presence, 1)
modparam(presence, server_address, sip:XXX.XXX.XXX.XXX)
modparam(pua, db_table, pua_p3)
modparam(pua, hash_size, 11)
modparam(pua_dialoginfo, presence_server, sip:prese...@yyy.yyy.yyy.yyy)
modparam(pua_dialoginfo, caller_spec_param, $avp(callerSpec))
modparam(pua_dialoginfo, callee_spec_param, $avp(calleeSpec))

modparam(pua_usrloc, default_domain, sip.mydomain.com)
modparam(pua_usrloc, presence_server, sip:prese...@yyy.yyy.yyy.yyy)

where XXX.XXX.XXX.XXX is my own IP (the ip of the proxy that crash) and 
YYY.YYY.YYY.YYY is the IP of the real presence server.

All presence related messages are relayed to real presence server or the UACs, 
with no presence processing in the proxy, except for pua_dialoginfo and  
pua_usrloc publishing.

In next comment I'll put a backtrace of the generated core. If you need to 
reproduce the issue I can supply all the info you request.

Regards,
Carlos Oliva



---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/382___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash in v1.11.3 using fifo pres_phtable_list (#382)

[Bogdan Andrei IANCU]

Hi @telematico - I uploaded a commit to fix the crash ; the fix disables the MI 
commands in presence module when used in library mode.
Still, if you want to use pua_dialoginfo, only pua module is required; presence 
module is not (if you have another OpenSIPS acting as presence server..What 
modules does require presence module in your case ??
Thanks, Bogdan

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/382#issuecomment-61597167___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash while creating avp (#230)

[Bogdan Andrei IANCU]

Closed #230.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/230#event-134145317___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] Crash in pua_dialoginfo (#70)

[Bogdan Andrei IANCU]

Closed #70.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/70#event-122372695___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] crash while creating avp (#230)

[Bogdan Andrei IANCU]

@ovidiusas , could you email me please the complete backtrace (all stacks). 
@liviuchircu mentioned that this is related to B2B usage - is it right ?

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/230#issuecomment-43076367___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] [opensips] crash while creating avp (#230)

[Ovidiu Sas]

Latest trunk:
version: opensips 1.12.0dev-notls (x86_64/linux)
git revision: 0338917

Relevant info:
```
#0  0x005157b2 in fm_remove_free (n=0x7fb977efa590, qm=0x7fb977b1d000) 
at mem/f_malloc.c:172
172 *pf=n-u.nxt_free;
(gdb) bt
#0  0x005157b2 in fm_remove_free (n=0x7fb977efa590, qm=0x7fb977b1d000) 
at mem/f_malloc.c:172
#1  fm_malloc (qm=0x7fb977b1d000, size=72, size@entry=65) at mem/f_malloc.c:386
#2  0x00510a25 in shm_malloc (size=65) at evi/../mem/shm_mem.h:338
#3  new_avp (flags=flags@entry=2, id=1, val=...) at usr_avp.c:119
#4  0x00510f22 in add_avp (flags=flags@entry=2, name=optimized out, 
val=...) at usr_avp.c:148
#5  0x0047b486 in pv_set_avp (val=optimized out, op=optimized out, 
param=0x7fb9815e9af8, msg=optimized out) at pvar.c:2251
#6  pv_set_avp (msg=optimized out, param=0x7fb9815e9af8, op=optimized out, 
val=optimized out) at pvar.c:2186
#7  0x004777ed in pv_set_value (msg=msg@entry=0x7fb981619b70, 
sp=sp@entry=0x7fb9815e9ae0, op=optimized out, 
value=value@entry=0x7fffaa4d8550) at pvar.c:3887
...
(gdb) print pf
$1 = (struct fm_frag **) 0x0
```

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/230___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] Crash in pua_dialoginfo (#70)

[Bogdan Andrei IANCU]

@nikbyte , do you still have the corefile (available for inspection) ? or can 
you reproduce it and get a new core ? I would need some more info from the 
corefile, just to validate a theory of mine on how this crash happens.

Thanks, Bogdan

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/70#issuecomment-41221747___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel