Bogdan, The libcrypto crash has also reproduced on the latest 2.4. I collected the backtrace as well as the information you requested below. There was no curl error, just this one from OpenSIPS:
Jun 27 13:06:10 [5992] CRITICAL:core:qm_free: freeing already freed pointer, first free: tls.h: os_free(119) - aborting Backtrace: https://pastebin.com/SqtafdxR # opensips -V version: opensips 2.4.6 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. git revision: 4e258b17f main.c compiled on 18:21:00 Jun 26 2019 with gcc 7 Ben Newlin From: Bogdan-Andrei Iancu <bog...@opensips.org> Date: Wednesday, June 26, 2019 at 9:12 AM To: Ben Newlin <ben.new...@genesys.com>, OpenSIPS devel mailling list <devel@lists.opensips.org> Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash Oh, just do in gdb: f 2 p *((struct qm_frag*)((char*)(p)-sizeof(struct qm_frag))) In regards to async, I was more talking about the script async() (to try to remove it) , not about the async in the protocols. Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com<https://www.opensips-solutions.com> OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/> On 06/26/2019 04:00 PM, Ben Newlin wrote: Bogdan, Thanks for the response. I will get the info you need. When you say to print in frame 2, is there some command I must issue to put the current context into frame 2 (or 7 or 8) or do I just issue the command you provided? Apologies, I use GDB very seldomly. Also, while we are using async support in this proxy, none of our current TLS traffic is using any async functions. Do you think other calls using async could be interfering with the TLS calls? Ben Newlin From: Bogdan-Andrei Iancu <bog...@opensips.org><mailto:bog...@opensips.org> Date: Wednesday, June 26, 2019 at 5:25 AM To: Ben Newlin <ben.new...@genesys.com><mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list <devel@lists.opensips.org><mailto:devel@lists.opensips.org> Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash Hi Ben, In any of the corefile, in frame 2 could you print p *((struct qm_frag*)((char*)(p)-sizeof(struct qm_frag))) Now, I see that the backtrace via libcurl & libcrypto may go via some error cleanup - see frame #7 and #8 - so , do you see any curl error prior to the crash (ideally from the same process) ? If you can reproduce it easily, could you try without the async support, just as a PoC only. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com<https://www.opensips-solutions.com> OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/> On 06/25/2019 10:04 PM, Ben Newlin wrote: Yes, we are using tls_mgm and proto_tls. We have the following version of openssl-devel installed: # yum info openssl-devel Name : openssl-devel Arch : x86_64 Epoch : 1 Version : 1.0.2k Release : 16.amzn2.1.1 Size : 3.1 M Repo : installed From repo : amzn2-core Summary : Files for development of applications which will use OpenSSL URL : http://www.openssl.org/<http://www.openssl.org/> License : OpenSSL Description : OpenSSL is a toolkit for supporting cryptography. The openssl-devel : package contains include files needed to develop applications which : support various cryptographic algorithms and protocols. Ben Newlin From: Bogdan-Andrei Iancu <bog...@opensips.org><mailto:bog...@opensips.org> Date: Tuesday, June 25, 2019 at 2:54 PM To: Ben Newlin <ben.new...@genesys.com><mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list <devel@lists.opensips.org><mailto:devel@lists.opensips.org> Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash Thank you Ben :) The backtraces report a double free, but the mem ops are performed via libcrypto ( opensips registers mem-specific functions to lib crypto). Are you using in your opensips cfg any other module that might require ssl/crtypto, like tls_mgm, proto_wss ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com<https://www.opensips-solutions.com> OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/> On 06/25/2019 06:49 PM, Ben Newlin wrote: Bogdan, You have good timing. :) I was just collecting the data from a crash this morning with the extra flags set. # opensips -V version: opensips 2.4.5 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. git revision: 60097425d main.c compiled on 22:27:12 Jun 24 2019 with gcc 7 [1] https://pastebin.com/aj1mvk0N<https://pastebin.com/aj1mvk0N> [2] https://pastebin.com/kfSz9Ks0<https://pastebin.com/kfSz9Ks0> [3] https://pastebin.com/hptH9yiA<https://pastebin.com/hptH9yiA> [4] https://pastebin.com/GfYYZxzB<https://pastebin.com/GfYYZxzB> [5] https://pastebin.com/CzYiNAch<https://pastebin.com/CzYiNAch> Ben Newlin From: Bogdan-Andrei Iancu <bog...@opensips.org><mailto:bog...@opensips.org> Date: Tuesday, June 25, 2019 at 11:39 AM To: Ben Newlin <ben.new...@genesys.com><mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list <devel@lists.opensips.org><mailto:devel@lists.opensips.org> Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash Hi Ben, The backtrace + corefile should do it ;). Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com<https://www.opensips-solutions.com> OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/> On 06/20/2019 07:19 PM, Ben Newlin wrote: Bogdan, Yes I will enable those. This seems to be reproducing fairly often so hopefully we will get some more info quickly. Is there anything else I need to collect at time of the crash with this enabled or is the core dump sufficient? Ben Newlin From: Bogdan-Andrei Iancu <bog...@opensips.org><mailto:bog...@opensips.org> Date: Thursday, June 20, 2019 at 12:15 PM To: Ben Newlin <ben.new...@genesys.com><mailto:ben.new...@genesys.com>, OpenSIPS devel mailling list <devel@lists.opensips.org><mailto:devel@lists.opensips.org> Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash Hi Ben, This looks like very generic - a crash due a previous memory corruption - is it possible for you to run with QM_MALLOC + DBG_MALLOC ? otherwise it is rather impossible to identify the original source of corruption. Thanks and regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com<https://www.opensips-solutions.com> OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/> On 06/20/2019 05:34 PM, Ben Newlin wrote: We have had another crash this morning: https://pastebin.com/dmXYbhzC<https://pastebin.com/dmXYbhzC> OpenSIPS version is the same as below. Ben Newlin From: Ben Newlin <ben.new...@genesys.com><mailto:ben.new...@genesys.com> Date: Wednesday, June 19, 2019 at 2:38 PM To: OpenSIPS devel mailling list <devel@lists.opensips.org><mailto:devel@lists.opensips.org>, Bogdan-Andrei Iancu <bog...@opensips.org><mailto:bog...@opensips.org> Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash The Github issue has the version info for that, for all the other crashes the version is below. $ opensips -V version: opensips 2.4.5 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. git revision: 60097425d main.c compiled on 18:06:35 Jun 13 2019 with gcc 7 Ben Newlin
_______________________________________________ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
