Bug#969559: curl segmentation fauls on any https URL
On Fri, Sep 11, 2020 at 06:28:20PM +0200, Bernhard Übelacker wrote: > Dear Maintainer, hello Bruce Momjian, > with the last informations the issue is perfectly reproducible. > > It looks like a use after free caused by statically stored > function pointers in libengine-pkcs11-openssl / libp11. > > That led to following upstream bug: > https://github.com/OpenSC/libp11/issues/328 > > This got fixed in this commit: > > https://github.com/OpenSC/libp11/commit/e64496a198d4d2eb0310a22dc21be8b81367d319 > > This commit is not yet included in an upstream release tag. > Therefore this error is also visible in current testing. > > I hope it is ok to reassign to libengine-pkcs11-openssl. Yes, thank you for researching this and closing it. -- Bruce Momjian https://momjian.us EnterpriseDB https://enterprisedb.com The usefulness of a cup is in its emptiness, Bruce Lee
Bug#969559: curl segmentation fauls on any https URL
Dear Maintainer, hello Bruce Momjian, with the last informations the issue is perfectly reproducible. It looks like a use after free caused by statically stored function pointers in libengine-pkcs11-openssl / libp11. That led to following upstream bug: https://github.com/OpenSC/libp11/issues/328 This got fixed in this commit: https://github.com/OpenSC/libp11/commit/e64496a198d4d2eb0310a22dc21be8b81367d319 This commit is not yet included in an upstream release tag. Therefore this error is also visible in current testing. I hope it is ok to reassign to libengine-pkcs11-openssl. Kind regards, Bernhard
Processed: Re: Bug#969559: Info received (Bug#969559: curl segmentation fauls on any https URL)
Processing commands for cont...@bugs.debian.org: > forwarded 969559 https://github.com/OpenSC/libp11/issues/328 Bug #969559 [libengine-pkcs11-openssl] curl segmentation fauls on any https URL Set Bug forwarded-to-address to 'https://github.com/OpenSC/libp11/issues/328'. > tags 969559 + patch upstream fixed-upstream Bug #969559 [libengine-pkcs11-openssl] curl segmentation fauls on any https URL Added tag(s) fixed-upstream, upstream, and patch. > End of message, stopping processing here. Please contact me if you need assistance. -- 969559: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969559 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Processed: Re: Bug#969559: Info received (Bug#969559: curl segmentation fauls on any https URL)
Processing commands for cont...@bugs.debian.org: > reassign 969559 libengine-pkcs11-openssl 0.4.9-4 Bug #969559 [curl] curl segmentation fauls on any https URL Bug reassigned from package 'curl' to 'libengine-pkcs11-openssl'. No longer marked as found in versions curl/7.64.0-4+deb10u1. Ignoring request to alter fixed versions of bug #969559 to the same values previously set Bug #969559 [libengine-pkcs11-openssl] curl segmentation fauls on any https URL Marked as found in versions libp11/0.4.9-4. > affects 969559 curl Bug #969559 [libengine-pkcs11-openssl] curl segmentation fauls on any https URL Added indication that 969559 affects curl > End of message, stopping processing here. Please contact me if you need assistance. -- 969559: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969559 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#969559: Info received (Bug#969559: curl segmentation fauls on any https URL)
Oh, the kernel error message might be helpful: curl[4979] general protection ip:7f3a3da00bce sp:7fff5dc217d0 error:0 in libcrypto.so.1.1[7f3a3d8fe000+19e000] -- Bruce Momjian https://momjian.us EnterpriseDB https://enterprisedb.com The usefulness of a cup is in its emptiness, Bruce Lee
Bug#969559: curl segmentation fauls on any https URL
On Sun, Sep 6, 2020 at 02:37:22PM +0200, Bernhard Übelacker wrote: > Hello Bruce Momjian, > thanks for the details and confirmation. > > > Am 05.09.20 um 17:32 schrieb Bruce Momjian,,,: > > (gdb) print pmeth->init > > $1 = (int (*)(EVP_PKEY_CTX *)) 0xf0e0d0c0b0a0908 > > > gdb) print *pmeth > > $8 = {pkey_id = 50462976, flags = 117835012, init = 0xf0e0d0c0b0a0908, > > copy = 0x1716151413121110, cleanup = 0x1f1e1d1c1b1a1918, paramgen_init = > > 0x98c476a19fc273a5, paramgen = 0x9cc072a593ce7fa9, > > The pointer init copy and cleanup are really not looking like usual > pointers or random ... > > > I am using a pkcs11 hardware crypto device, and perhaps it is > > misconfigured, but it probably shouldn't crash. This might be a library > > bug, not sure. I will check the pkcs11's configuration now, but it used > > to work. > > But I have no knowledge about such crypto hardware, therefore > I am not sure if I can be of any more help. Maybe you could > provide the needed packages, libraries and configuration steps > that are needed to use such a device of yours when starting with > a fresh debian installation? I was just able to reproduce this failure on a fresh install of Debian 10.5/Buster. What I did was just to install pkcs11 support: apt-get install libengine-pkcs11-openssl and then modify /etc/ssl/openssl.cnf with the attached patch to use pkcs11 support; 'curl https://google.com' will then segmentation fault. This server has no pkcs11 hardware; it is an AWS instance. If you comment out the line: pkcs11 = pkcs11_section curl works again. Thanks for your research so far on this. -- Bruce Momjian https://momjian.us EnterpriseDB https://enterprisedb.com The usefulness of a cup is in its emptiness, Bruce Lee --- /etc/ssl/openssl.cnf.orig 2019-05-30 11:27:48.0 -0400 +++ /etc/ssl/openssl.cnf 2020-09-07 16:02:31.448309714 -0400 @@ -353,6 +353,7 @@ # identifier (optional, default: sha1) [default_conf] ssl_conf = ssl_sect +engines = engine_section [ssl_sect] system_default = system_default_sect @@ -360,3 +361,14 @@ [system_default_sect] MinProtocol = TLSv1.2 CipherString = DEFAULT@SECLEVEL=2 + +[engine_section] +pkcs11 = pkcs11_section + +[pkcs11_section] +# https://github.com/openssl/openssl/blob/master/README.ENGINE +engine_id = pkcs11 +# same as SO_PATH +dynamic_path = /usr/lib/x86_64-linux-gnu/engines-1.1/pkcs11.so +MODULE_PATH = opensc-pkcs11.so +init = 0
Bug#969559: curl segmentation fauls on any https URL
Hello Bruce Momjian, thanks for the details and confirmation. Am 05.09.20 um 17:32 schrieb Bruce Momjian,,,: > (gdb) print pmeth->init > $1 = (int (*)(EVP_PKEY_CTX *)) 0xf0e0d0c0b0a0908 > gdb) print *pmeth > $8 = {pkey_id = 50462976, flags = 117835012, init = 0xf0e0d0c0b0a0908, > copy = 0x1716151413121110, cleanup = 0x1f1e1d1c1b1a1918, paramgen_init = > 0x98c476a19fc273a5, paramgen = 0x9cc072a593ce7fa9, The pointer init copy and cleanup are really not looking like usual pointers or random ... > I am using a pkcs11 hardware crypto device, and perhaps it is > misconfigured, but it probably shouldn't crash. This might be a library > bug, not sure. I will check the pkcs11's configuration now, but it used > to work. But I have no knowledge about such crypto hardware, therefore I am not sure if I can be of any more help. Maybe you could provide the needed packages, libraries and configuration steps that are needed to use such a device of yours when starting with a fresh debian installation? Kind regards, Bernhard
Bug#969559: Info received (Bug#969559: curl segmentation fauls on any https URL)
I have checked my pkcs11 device and it is functioning properly, but curl still crashes. Fortunately I can just use 'wget' until this is fixed. -- Bruce Momjian https://momjian.us EnterpriseDB https://enterprisedb.com The usefulness of a cup is in its emptiness, Bruce Lee
Bug#969559: curl segmentation fauls on any https URL
On Sat, Sep 5, 2020 at 03:50:20PM +0200, Bernhard Übelacker wrote: > Dear Maintainer, > I tried to reproduce this fault, but did not get a segfault. > > However, I think the backtrace points to these lines: > > (gdb) bt > #0 0x7769dbce in int_ctx_new () at ../crypto/evp/pmeth_lib.c:160 > #1 0x7769dcfa in EVP_PKEY_CTX_new () at > ../crypto/evp/pmeth_lib.c:245 > #2 0x77698d44 in do_sigver_init () at ../crypto/evp/m_sigver.c:29 > #3 0x77698eab in EVP_DigestVerifyInit () at > ../crypto/evp/m_sigver.c:97 > #4 0x775bc7d2 in ASN1_item_verify () at > ../crypto/asn1/a_verify.c:148 > #5 0x77722490 in X509_verify () at ../crypto/x509/x_all.c:26 > ... > > > https://sources.debian.org/src/openssl/1.1.1d-0+deb10u3/crypto/evp/pmeth_lib.c/#L160 > > 159 if (pmeth->init) { > 160 if (pmeth->init(ret) <= 0) { > 161 ret->pmeth = NULL; > > As there is a check for pmeth->init being non-null, I guess > it contains for some reason an invalid pointer. > > > @Bruce Momjian, > maybe you could install the following debug symbols packages > `curl-dbgsym libcurl4-dbgsym libssl1.1-dbgsym` from the dbgsym > repository described here: > > https://wiki.debian.org/HowToGetABacktrace#Installing_the_debugging_symbols > > Then run a new gdb session and when the segfault appears > please run these commands in gdb: > print pmeth->init > bt full 5 Sure, here it is: (gdb) run https://google.com Starting program: /usr/bin/curl https://google.com [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x76730700 (LWP 30481)] [Thread 0x76730700 (LWP 30481) exited] Thread 1 "curl" received signal SIGSEGV, Segmentation fault. 0x77679bce in int_ctx_new (pkey=pkey@entry=0x556035a0, e=0x555bd8d0, e@entry=0x0, id=, id@entry=-1) at ../crypto/evp/pmeth_lib.c:160 160 ../crypto/evp/pmeth_lib.c: No such file or directory. (gdb) print pmeth->init $1 = (int (*)(EVP_PKEY_CTX *)) 0xf0e0d0c0b0a0908 (gdb) bt full 5 #0 0x77679bce in int_ctx_new (pkey=pkey@entry=0x556035a0, e=0x555bd8d0, e@entry=0x0, id=, id@entry=-1) at ../crypto/evp/pmeth_lib.c:160 ret = 0x55609810 pmeth = 0x555eaaf0 #1 0x77679cfa in EVP_PKEY_CTX_new (pkey=pkey@entry=0x556035a0, e=e@entry=0x0) at ../crypto/evp/pmeth_lib.c:245 No locals. #2 0x77674d44 in do_sigver_init (ctx=ctx@entry=0x556034c0, pctx=pctx@entry=0x0, type=type@entry=0x777b1fc0 , e=e@entry=0x0, pkey=pkey@entry=0x556035a0, ver=ver@entry=1) at ../crypto/evp/m_sigver.c:29 No locals. #3 0x77674eab in EVP_DigestVerifyInit (ctx=ctx@entry=0x556034c0, pctx=pctx@entry=0x0, type=type@entry=0x777b1fc0 , e=e@entry=0x0, pkey=pkey@entry=0x556035a0) at ../crypto/evp/m_sigver.c:97 No locals. #4 0x775987d2 in ASN1_item_verify (it=0x777c3e80 , a=a@entry=0x555ff698, signature=signature@entry=0x555ff6a8, asn=asn@entry=0x555ff610, pkey=0x556035a0) at ../crypto/asn1/a_verify.c:148 type = 0x777b1fc0 ctx = 0x556034c0 buf_in = 0x0 ret = -1 inl = 0 mdnid = 672 pknid = 6 inll = 0 (More stack frames follow...) I also got this output: gdb) print *pmeth $8 = {pkey_id = 50462976, flags = 117835012, init = 0xf0e0d0c0b0a0908, copy = 0x1716151413121110, cleanup = 0x1f1e1d1c1b1a1918, paramgen_init = 0x98c476a19fc273a5, paramgen = 0x9cc072a593ce7fa9, keygen_init = 0xdabe4402cda85116, keygen = 0xdeba4006c1a45d1a, sign_init = 0x681bf10ff0df87ae, sign = 0x6715fc03fbd58ea6, verify_init = 0x924fa56f48f1e16d, verify = 0x8d51b87353ebf875, verify_recover_init = 0x1799a7c97f8256c6, verify_recover = 0x8b59d56cec4c296f, signctx_init = 0xe7754752753ae23d, signctx = 0x39cf0754b49ebf27, verifyctx_init = 0x48097bc25f90dc0b, verifyctx = 0x2f1c87c1a44552ad, encrypt_init = 0x87d3b21760a6f545, encrypt = 0xa820a64334d0d30, decrypt_init = 0x54feb4be1cf7cf7c, decrypt = 0xdfa761d2f0bbe613, derive_init = 0x7929a8e7fefa1af0, derive = 0x40e6afb34a64a5d7, ctrl = 0x2500f59b71fe4125, ctrl_str = 0xa1c725ad5bb1388, digestsign = 0xe04ff2a999665a4e, digestverify = 0xeacdf8cdaa2b577e, check = 0xe97909bfcc79fc24, public_check = 0x36de686d3cc21a37, param_check = 0xd, digest_custom = 0x7758ac80 } (gdb) print pmeth->init[0] Cannot access memory at address 0xf0e0d0c0b0a0908 (gdb) print *(pmeth->init) Cannot access memory at address 0xf0e0d0c0b0a090
Bug#969559: curl segmentation fauls on any https URL
Dear Maintainer, I tried to reproduce this fault, but did not get a segfault. However, I think the backtrace points to these lines: (gdb) bt #0 0x7769dbce in int_ctx_new () at ../crypto/evp/pmeth_lib.c:160 #1 0x7769dcfa in EVP_PKEY_CTX_new () at ../crypto/evp/pmeth_lib.c:245 #2 0x77698d44 in do_sigver_init () at ../crypto/evp/m_sigver.c:29 #3 0x77698eab in EVP_DigestVerifyInit () at ../crypto/evp/m_sigver.c:97 #4 0x775bc7d2 in ASN1_item_verify () at ../crypto/asn1/a_verify.c:148 #5 0x77722490 in X509_verify () at ../crypto/x509/x_all.c:26 ... https://sources.debian.org/src/openssl/1.1.1d-0+deb10u3/crypto/evp/pmeth_lib.c/#L160 159 if (pmeth->init) { 160 if (pmeth->init(ret) <= 0) { 161 ret->pmeth = NULL; As there is a check for pmeth->init being non-null, I guess it contains for some reason an invalid pointer. @Bruce Momjian, maybe you could install the following debug symbols packages `curl-dbgsym libcurl4-dbgsym libssl1.1-dbgsym` from the dbgsym repository described here: https://wiki.debian.org/HowToGetABacktrace#Installing_the_debugging_symbols Then run a new gdb session and when the segfault appears please run these commands in gdb: print pmeth->init bt full 5 Kind regards, Bernhard # Buster/stable amd64 qemu VM apt update apt dist-upgrade apt install systemd-coredump curl gdb curl https://google.com dpkg -l curl libc6 libcurl4 zlib1g libssl1.1 ii curl7.64.0-4+deb10u1 amd64command line tool for transferring data with URL syntax ii libc6:amd64 2.28-10 amd64GNU C Library: Shared libraries ii libcurl4:amd64 7.64.0-4+deb10u1 amd64easy-to-use client-side URL transfer library (OpenSSL flavour) ii libssl1.1:amd64 1.1.1d-0+deb10u3 amd64Secure Sockets Layer toolkit - shared libraries ii zlib1g:amd641:1.2.11.dfsg-1 amd64compression library - runtime benutzer@debian:~$ curl https://google.com 301 Moved 301 Moved The document has moved https://www.google.com/";>here. gdb -q --args curl https://google.com b ASN1_item_verify y run disassemble ASN1_item_verify b EVP_DigestVerifyInit cont ... generate-core-file /tmp/core (gdb) bt #0 0x7769dbce in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 #1 0x77698d44 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 #2 0x775bc7d2 in ASN1_item_verify () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 #3 0x7771cfb4 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 #4 0x7771edd6 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 #5 0x7771f416 in X509_verify_cert () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 #6 0x7782fb88 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1 #7 0x778510f3 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1 #8 0x778536c5 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1 #9 0x7784d143 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1 #10 0x77838f34 in SSL_do_handshake () from /lib/x86_64-linux-gnu/libssl.so.1.1 #11 0x77fa3240 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4 #12 0x77fa53f0 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4 #13 0x77fa61da in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4 #14 0x77f4d462 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4 #15 0x77f6f6fe in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4 #16 0x77f70aa9 in curl_multi_perform () from /lib/x86_64-linux-gnu/libcurl.so.4 #17 0x77f67642 in curl_easy_perform () from /lib/x86_64-linux-gnu/libcurl.so.4 #18 0x55569f30 in ?? () #19 0x5556b42a in ?? () #20 0xd8c4 in ?? () #21 0x77b5c09b in __libc_start_main (main=0xd770, argc=2, argv=0x7fffe608, init=, fini=, rtld_fini=, stack_end=0x7fffe5f8) at ../csu/libc-start.c:308 #22 0xd9da in ?? () apt install curl-dbgsym libcurl4-dbgsym libssl1.1-dbgsym gdb -q /usr/bin/curl --core /tmp/core set width 0 set pagination off (gdb) bt #0 0x7769dbce in int_ctx_new (pkey=pkey@entry=0x55601a10, e=e@entry=0x0, id=, id@entry=-1) at ../crypto/evp/pmeth_lib.c:160 #1 0x7769dcfa in EVP_PKEY_CTX_new (pkey=pkey@entry=0x55601a10, e=e@entry=0x0) at ../crypto/evp/pmeth_lib.c:245 #2 0x77698d44 in do_sigver_init (ctx=ctx@entry=0x55601930, pctx=pctx@entry=0x0, type=type@entry=0x777d5fc0 , e=e@entry=0x0, pkey=pkey@entry=0x55601a10, ver=ver@entry=1) at ../crypto/evp/m_sigver.c:29 #3 0x77698eab in EVP_DigestVerifyInit (ctx=ctx@entry=0x55601930, pctx=pctx@entry=0x0, type=type@entry=0x777d5fc0 , e=e@entry=0x0, pkey=pkey@entry=0x55601a10) at ../crypto/evp/m_sigver.c:97 #4 0x775bc7d2 in ASN1_item_verify (it=0x777e7e80 , a=a@entry=0x555fda18, signature=signature@entry=0x555fda28, asn=asn
Bug#969559: curl segmentation fauls on any https URL
Package: curl Version: 7.64.0-4+deb10u1 Severity: grave Justification: renders package unusable Dear Maintainer, *** Reporter, please consider answering these questions, where appropriate *** Simply type: $ curl https://google.com Segmentation fault or use any https URL. Here is a backtrace: 0x77679bce in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (gdb) bt #0 0x77679bce in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #1 0x77674d44 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #2 0x775987d2 in ASN1_item_verify () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #3 0x776f8fb4 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #4 0x776fadd6 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #5 0x776fb416 in X509_verify_cert () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #6 0x7780bb88 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #7 0x7782d0f3 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #8 0x7782f6c5 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #9 0x77829143 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #10 0x77814f34 in SSL_do_handshake () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #11 0x77f7f240 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #12 0x77f813f0 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #13 0x77f821da in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #14 0x77f29462 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #15 0x77f4b6fe in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #16 0x77f4caa9 in curl_multi_perform () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #17 0x77f43642 in curl_easy_perform () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 #18 0x55569f30 in ?? () #19 0x5556b42a in ?? () #20 0xd8c4 in ?? () #21 0x77b3809b in __libc_start_main (main=0xd770, argc=2, argv=0x7fffded8, init=, fini=, rtld_fini=, stack_end=0x7fffdec8) at ../csu/libc-start.c:308 #22 0xd9da in ?? () *** End of the template - remove these template lines *** -- System Information: Debian Release: 10.5 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.19.0-10-amd64 (SMP w/16 CPU cores) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages curl depends on: ii libc6 2.28-10 ii libcurl4 7.64.0-4+deb10u1 ii zlib1g1:1.2.11.dfsg-1 curl recommends no packages. curl suggests no packages. -- no debconf information