Re: crash from curl with pkcs12 certs and threads
On Thu, 2010-07-08 at 18:54 +0200, Dr. Stephen Henson wrote: On Thu, Jul 08, 2010, Brian Makin wrote: Ahh, got it. in crypto/evp/evp_pbe.c:EvP_PBE_alg_add pbe_tmp isn't initialized which means sometimes it has a bogus value. 119c119 EVP_PBE_CTL *pbe_tmp = NULL, pbelu; --- EVP_PBE_CTL *pbe_tmp, pbelu; Is that all that is needed or is this indicative of a lower level problem? Ah, yes that is the problem. When there are no existing PBE algorithms pbe_tmp is uninitialised. I've just committed a fix. Let me know if that fixes your problems. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org Testing with the latest development snapshot looks good. It appears that is the fix. Thank you very much for your assistance! -- BRIAN MAKIN Senior Software Engineer ma...@vivisimo.com Vivisimo [Search Done Right™] 1710 Murray Avenue Pittsburgh, PA 15217 USA tel: +1.412.422.2499 vivisimo.com __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
On Wed, 2010-07-07 at 21:28 +0200, Dr. Stephen Henson wrote: On Wed, Jul 07, 2010, Brian Makin wrote: The snapshots all seem to be failing in make test... Is this a known issue or might it be a problem on my end? Testing key generation with NIST Binary-Curve K-571 ok Testing key generation with NIST Binary-Curve B-571 ok sh ./testenc cat ./testenc: line 10: 26677 Segmentation fault (core dumped) $cmd enc $test $test.cipher ./testenc: line 11: 26684 Segmentation fault (core dumped) $cmd enc $test.cipher $test.clear cmp: EOF on ./p.clear Just tested it under Linux and don't get any crashes. Does reverting that change I made stop it crashing? Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org Ahh, got it. in crypto/evp/evp_pbe.c:EvP_PBE_alg_add pbe_tmp isn't initialized which means sometimes it has a bogus value. 119c119 EVP_PBE_CTL *pbe_tmp = NULL, pbelu; --- EVP_PBE_CTL *pbe_tmp, pbelu; Is that all that is needed or is this indicative of a lower level problem? -- BRIAN MAKIN Senior Software Engineer ma...@vivisimo.com Vivisimo [Search Done Right™] 1710 Murray Avenue Pittsburgh, PA 15217 USA tel: +1.412.422.2499 vivisimo.com __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
On Thu, Jul 08, 2010, Brian Makin wrote: Ahh, got it. in crypto/evp/evp_pbe.c:EvP_PBE_alg_add pbe_tmp isn't initialized which means sometimes it has a bogus value. 119c119 EVP_PBE_CTL *pbe_tmp = NULL, pbelu; --- EVP_PBE_CTL *pbe_tmp, pbelu; Is that all that is needed or is this indicative of a lower level problem? Ah, yes that is the problem. When there are no existing PBE algorithms pbe_tmp is uninitialised. I've just committed a fix. Let me know if that fixes your problems. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
The snapshots all seem to be failing in make test... Is this a known issue or might it be a problem on my end? Testing key generation with NIST Binary-Curve K-571 ok Testing key generation with NIST Binary-Curve B-571 ok sh ./testenc cat ./testenc: line 10: 26677 Segmentation fault (core dumped) $cmd enc $test $test.cipher ./testenc: line 11: 26684 Segmentation fault (core dumped) $cmd enc $test.cipher $test.clear cmp: EOF on ./p.clear On Sat, 2010-06-26 at 14:56 +0200, Dr. Stephen Henson wrote: On Thu, Jun 24, 2010, Brian Makin wrote: On Thu, 2010-06-24 at 15:22 +0200, Dr. Stephen Henson wrote: On Tue, Jun 22, 2010, Brian Makin wrote: snip Can you check to see if PKCS12_PBE_add() is called multiple times using the debugger? It is only supposed to be called once before threads are started but a bug means if it is called more than once you get multiple table entries per PBE algorithm (instead of no-op or replacing) and the subsequent sort operations can result in a race condition. I'll look into fixing that. OpenSSL 1.0.0 doesn't have this problem because the builtin PBE algorithms are in a static table. It appears the PKCS12_PBE_add is called many times in my test program. Please try the next snapshot or apply this patch and see if it fixes the problem: http://cvs.openssl.org/chngview?cn=19721 Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
On Wed, Jul 07, 2010, Brian Makin wrote: The snapshots all seem to be failing in make test... Is this a known issue or might it be a problem on my end? Testing key generation with NIST Binary-Curve K-571 ok Testing key generation with NIST Binary-Curve B-571 ok sh ./testenc cat ./testenc: line 10: 26677 Segmentation fault (core dumped) $cmd enc $test $test.cipher ./testenc: line 11: 26684 Segmentation fault (core dumped) $cmd enc $test.cipher $test.clear cmp: EOF on ./p.clear Just tested it under Linux and don't get any crashes. Does reverting that change I made stop it crashing? Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
Hah! reverted EVP_PBE_alg_add $ make $ make test everything passed :) I'll try to get at least a stack. On Wed, 2010-07-07 at 21:28 +0200, Dr. Stephen Henson wrote: On Wed, Jul 07, 2010, Brian Makin wrote: The snapshots all seem to be failing in make test... Is this a known issue or might it be a problem on my end? Testing key generation with NIST Binary-Curve K-571 ok Testing key generation with NIST Binary-Curve B-571 ok sh ./testenc cat ./testenc: line 10: 26677 Segmentation fault (core dumped) $cmd enc $test $test.cipher ./testenc: line 11: 26684 Segmentation fault (core dumped) $cmd enc $test.cipher $test.clear cmp: EOF on ./p.clear Just tested it under Linux and don't get any crashes. Does reverting that change I made stop it crashing? Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
On Thu, Jun 24, 2010, Brian Makin wrote: On Thu, 2010-06-24 at 15:22 +0200, Dr. Stephen Henson wrote: On Tue, Jun 22, 2010, Brian Makin wrote: snip Can you check to see if PKCS12_PBE_add() is called multiple times using the debugger? It is only supposed to be called once before threads are started but a bug means if it is called more than once you get multiple table entries per PBE algorithm (instead of no-op or replacing) and the subsequent sort operations can result in a race condition. I'll look into fixing that. OpenSSL 1.0.0 doesn't have this problem because the builtin PBE algorithms are in a static table. It appears the PKCS12_PBE_add is called many times in my test program. Please try the next snapshot or apply this patch and see if it fixes the problem: http://cvs.openssl.org/chngview?cn=19721 Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
On Thu, 2010-06-24 at 15:22 +0200, Dr. Stephen Henson wrote: On Tue, Jun 22, 2010, Brian Makin wrote: snip Can you check to see if PKCS12_PBE_add() is called multiple times using the debugger? It is only supposed to be called once before threads are started but a bug means if it is called more than once you get multiple table entries per PBE algorithm (instead of no-op or replacing) and the subsequent sort operations can result in a race condition. I'll look into fixing that. OpenSSL 1.0.0 doesn't have this problem because the builtin PBE algorithms are in a static table. It appears the PKCS12_PBE_add is called many times in my test program. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
On Tue, Jun 22, 2010, Brian Makin wrote: On Tue, 2010-06-22 at 19:00 +0200, Dr. Stephen Henson wrote: On Tue, Jun 22, 2010, Brian Makin wrote: Using curl 7.16.1 and 7.20.0 with openssl 0.9.8l and 0.9.8o I am getting intermittent crashes. Apache is setup with a pkcs12 cert and when this fails it is always early in the process ie: initialization of the threads. I'm guessing either curl or ssl has a missing lock but I'm not sure which and where. Any assistance is greatly appreciated. The callstack usually looks something like the following or ends up in the CRYPTO_malloc functions. See if adding sk_sort(pbe_algs) at the end of EVP_PBE_alg_add() in crypto/evp/evp_pbe.c fixes this. still crashes... although now it sometimes crashes in sk_sort :) Looks like it is generally one of these two stacks. #6 0x7f3df3695780 in CRYPTO_realloc () #7 0x7f3df36ee507 in sk_insert () #8 0x7f3df36fbf19 in EVP_PBE_alg_add () #9 0x7f3df373713e in PKCS12_PBE_add () #10 0x7f3df3d42b72 in cert_stuff (conn=0xdba9e0, sockindex=value optimized out) at ssluse.c:377 #0 0x7fc6fe101e78 in pbe_cmp () #1 0x7fc6fe3efb15 in msort_with_tmp (p=value optimized out, b=value optimized out, n=value optimized out) at msort.c:83 ... #7 0x7fc6fe3f009c in *__GI_qsort_r (b=value optimized out, n=value optimized out, s=8, cmp=0x7fc6fe101e70 pbe_cmp, arg=0x0) at msort.c:294 #8 0x7fc6fe0f4317 in sk_sort () #9 0x7fc6fe101f25 in EVP_PBE_alg_add () #10 0x7fc6fe13d13e in PKCS12_PBE_add () #11 0x7fc6fe748b72 in cert_stuff (conn=0x7fc6d8005930, sockindex=value optimized out) at ssluse.c:377 OK, try calling EVP_PKEY_alg_add() (with the sk_sort addition) before curl is initialised, i.e. when you setup thread locks. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
On Wed, Jun 23, 2010, Dr. Stephen Henson wrote: On Tue, Jun 22, 2010, Brian Makin wrote: On Tue, 2010-06-22 at 19:00 +0200, Dr. Stephen Henson wrote: On Tue, Jun 22, 2010, Brian Makin wrote: Using curl 7.16.1 and 7.20.0 with openssl 0.9.8l and 0.9.8o I am getting intermittent crashes. Apache is setup with a pkcs12 cert and when this fails it is always early in the process ie: initialization of the threads. I'm guessing either curl or ssl has a missing lock but I'm not sure which and where. Any assistance is greatly appreciated. The callstack usually looks something like the following or ends up in the CRYPTO_malloc functions. See if adding sk_sort(pbe_algs) at the end of EVP_PBE_alg_add() in crypto/evp/evp_pbe.c fixes this. still crashes... although now it sometimes crashes in sk_sort :) Looks like it is generally one of these two stacks. #6 0x7f3df3695780 in CRYPTO_realloc () #7 0x7f3df36ee507 in sk_insert () #8 0x7f3df36fbf19 in EVP_PBE_alg_add () #9 0x7f3df373713e in PKCS12_PBE_add () #10 0x7f3df3d42b72 in cert_stuff (conn=0xdba9e0, sockindex=value optimized out) at ssluse.c:377 #0 0x7fc6fe101e78 in pbe_cmp () #1 0x7fc6fe3efb15 in msort_with_tmp (p=value optimized out, b=value optimized out, n=value optimized out) at msort.c:83 ... #7 0x7fc6fe3f009c in *__GI_qsort_r (b=value optimized out, n=value optimized out, s=8, cmp=0x7fc6fe101e70 pbe_cmp, arg=0x0) at msort.c:294 #8 0x7fc6fe0f4317 in sk_sort () #9 0x7fc6fe101f25 in EVP_PBE_alg_add () #10 0x7fc6fe13d13e in PKCS12_PBE_add () #11 0x7fc6fe748b72 in cert_stuff (conn=0x7fc6d8005930, sockindex=value optimized out) at ssluse.c:377 OK, try calling EVP_PKEY_alg_add() (with the sk_sort addition) before curl is initialised, i.e. when you setup thread locks. Ooops, sorry I meant call PKCS12_PBE_add() Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
On Tue, Jun 22, 2010, Brian Makin wrote: Using curl 7.16.1 and 7.20.0 with openssl 0.9.8l and 0.9.8o I am getting intermittent crashes. Apache is setup with a pkcs12 cert and when this fails it is always early in the process ie: initialization of the threads. I'm guessing either curl or ssl has a missing lock but I'm not sure which and where. Any assistance is greatly appreciated. The callstack usually looks something like the following or ends up in the CRYPTO_malloc functions. See if adding sk_sort(pbe_algs) at the end of EVP_PBE_alg_add() in crypto/evp/evp_pbe.c fixes this. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: crash from curl with pkcs12 certs and threads
On Tue, 2010-06-22 at 19:00 +0200, Dr. Stephen Henson wrote: On Tue, Jun 22, 2010, Brian Makin wrote: Using curl 7.16.1 and 7.20.0 with openssl 0.9.8l and 0.9.8o I am getting intermittent crashes. Apache is setup with a pkcs12 cert and when this fails it is always early in the process ie: initialization of the threads. I'm guessing either curl or ssl has a missing lock but I'm not sure which and where. Any assistance is greatly appreciated. The callstack usually looks something like the following or ends up in the CRYPTO_malloc functions. See if adding sk_sort(pbe_algs) at the end of EVP_PBE_alg_add() in crypto/evp/evp_pbe.c fixes this. still crashes... although now it sometimes crashes in sk_sort :) Looks like it is generally one of these two stacks. #6 0x7f3df3695780 in CRYPTO_realloc () #7 0x7f3df36ee507 in sk_insert () #8 0x7f3df36fbf19 in EVP_PBE_alg_add () #9 0x7f3df373713e in PKCS12_PBE_add () #10 0x7f3df3d42b72 in cert_stuff (conn=0xdba9e0, sockindex=value optimized out) at ssluse.c:377 #0 0x7fc6fe101e78 in pbe_cmp () #1 0x7fc6fe3efb15 in msort_with_tmp (p=value optimized out, b=value optimized out, n=value optimized out) at msort.c:83 ... #7 0x7fc6fe3f009c in *__GI_qsort_r (b=value optimized out, n=value optimized out, s=8, cmp=0x7fc6fe101e70 pbe_cmp, arg=0x0) at msort.c:294 #8 0x7fc6fe0f4317 in sk_sort () #9 0x7fc6fe101f25 in EVP_PBE_alg_add () #10 0x7fc6fe13d13e in PKCS12_PBE_add () #11 0x7fc6fe748b72 in cert_stuff (conn=0x7fc6d8005930, sockindex=value optimized out) at ssluse.c:377 Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
