Bogdan,
We upgraded to 2.4.3 and the crash reproduced today. Backtrace is
available here: https://pastebin.com/CZxQnZdR.
Ben Newlin
*From: *Bogdan-Andrei Iancu <bog...@opensips.org>
*Date: *Wednesday, November 7, 2018 at 6:18 AM
*To: *OpenSIPS devel mailling list <devel@lists.opensips.org>, Ben
Newlin <ben.new...@genesys.com>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
Hi Ben,
The BT indicates a double free for the accounting context - and I
noticed you use 2.4.1 version. And yes, there was an issue related to
acc context, issue that was fixed starting 2.4.2. So, could you
upgrade to the latest 2.4 and see if the crash still happens ? As I
think the fix is already there.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:
Jackpot - you get it right !! I will start digging into the trace,
but please keep the corefile, I might need it later.
Thanks and regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 11/06/2018 10:24 PM, Ben Newlin wrote:
Bogdan,
I have reproduced this crash and verified this time that the
flags were set.
$ opensips -V
version: opensips 2.4.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT,
DBG_LOCK
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 5d042cffc
main.c compiled on 23:38:55 Nov 5 2018 with gcc 7
Backtrace is available here: https://pastebin.com/KTQjkCwq
<https://pastebin.com/KTQjkCwq>
Ben Newlin
*From: *Bogdan-Andrei Iancu <bog...@opensips.org>
<mailto:bog...@opensips.org>
*Date: *Thursday, November 1, 2018 at 1:19 PM
*To: *Ben Newlin <ben.new...@genesys.com>
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list
<devel@lists.opensips.org> <mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
Hi Ben,
According to the backtrace, the memory debugger was not
activated. Do an "opensips -V" to check the resulting compile
flags - do you see the DBG_MALLOC and QM_MALLOC ?
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/31/2018 05:04 PM, Ben Newlin wrote:
Bogdan,
I was able to compile with those options and the crash has
occurred again. Backtrace is here:
https://pastebin.com/dezi9xUU <https://pastebin.com/dezi9xUU>.
Even though I had `memdump=1` set in my script, there was
no extra memory debugging information in the logs prior to
or at the time of the crash. I’m not sure if that is
expected or not.
Ben Newlin
*From: *Bogdan-Andrei Iancu <bog...@opensips.org>
<mailto:bog...@opensips.org>
*Date: *Monday, October 29, 2018 at 8:11 AM
*To: *Ben Newlin <ben.new...@genesys.com>
<mailto:ben.new...@genesys.com>, OpenSIPS devel mailling
list <devel@lists.opensips.org>
<mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
Hi Ben,
You can change the compile flags via the Makefile.conf
file - the menuconfig is also updating that file. So
during your build you can simply push a pre-modified
Makefile.conf file with the options needed for memory
debugging.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/26/2018 05:14 PM, Ben Newlin wrote:
Bogdan,
Unfortunately, we have run into a similar issue
before. Our build system is completely automated and
there is no way to inject the `make menuconfig`
interactive step into that process. If I were to be
testing this locally I might be able to work something
out, but I could never get such a build into our
testing environment which is where the crashes are
occurring.
Do you have instructions for enabling memory debugging
that do not require using the interactive TUI tool?
What does the menuconfig program do when these options
are selected? Are there some defines or other settings
we can change ourselves and bypass menuconfig?
Ben Newlin
*From: *Bogdan-Andrei Iancu <bog...@opensips.org>
<mailto:bog...@opensips.org>
*Date: *Friday, October 26, 2018 at 4:59 AM
*To: *OpenSIPS devel mailling list
<devel@lists.opensips.org>
<mailto:devel@lists.opensips.org>, Ben Newlin
<ben.new...@genesys.com> <mailto:ben.new...@genesys.com>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
Hi Ben,
all the BT's points to crashes while doing memory ops.
I suspect a memory corruption that randomly triggers
crashes in different parts of the code.
Could you try to re-compile with memory debugging
support ? See
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem,
the "How to handle it" section.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/24/2018 04:28 AM, Ben Newlin wrote:
We have had 2 more crashes today.
Crash 2: https://pastebin.com/rMruBQcZ
<https://pastebin.com/rMruBQcZ>
This crash appears to have occurred while
processing an initial INVITE request. I could not
see anything unusual about the request. I cannot
tell if this crash is related to the others.
Crash 3: https://pastebin.com/Gmk1m4NT
<https://pastebin.com/Gmk1m4NT>
This crash follows the pattern of the original
crash I reported.
Ben Newlin
*From: *Devel <devel-boun...@lists.opensips.org>
<mailto:devel-boun...@lists.opensips.org> on
behalf of Ben Newlin <ben.new...@genesys.com>
<mailto:ben.new...@genesys.com>
*Reply-To: *OpenSIPS devel mailling list
<devel@lists.opensips.org>
<mailto:devel@lists.opensips.org>
*Date: *Monday, October 22, 2018 at 4:45 PM
*To: *OpenSIPS devel mailling list
<devel@lists.opensips.org>
<mailto:devel@lists.opensips.org>
*Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
Here is a better trace of the call:
https://pastebin.com/gWpQR8E7
<https://pastebin.com/gWpQR8E7>
Ben Newlin
*From: *Ben Newlin <ben.new...@genesys.com>
<mailto:ben.new...@genesys.com>
*Date: *Monday, October 22, 2018 at 4:34 PM
*To: *OpenSIPS devel mailling list
<devel@lists.opensips.org>
<mailto:devel@lists.opensips.org>
*Subject: *OpenSIPS Crash
Hello,
We have been having sporadic crashes and I was
recently able to recover a core dump for one. I
have uploaded it here:
https://pastebin.com/ABktcYcH
<https://pastebin.com/ABktcYcH>.
I picked out a Call-ID from the crash data and
took a look in our tracing. I have uploaded it
here: https://pastebin.com/ZEzUUKZ5
<https://pastebin.com/ZEzUUKZ5>.
It appears that a downstream server was extremely
lagged and failed to respond to an INVITE. We sent
the INVITE to another server and the call was
connected, but then eventually the original server
“caught up” and sent a burst of 200 OK responses.
The crash seems to have occurred processing the
ACK to one of these responses.
Ben Newlin
_______________________________________________
Devel mailing list
Devel@lists.opensips.org
<mailto:Devel@lists.opensips.org>
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
_______________________________________________
Devel mailing list
Devel@lists.opensips.org <mailto:Devel@lists.opensips.org>
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
