RE: Memory issues with ssl handshake
Try plugging your code to exit after the first OpenSSL function, then after the second, and so forth, and see if you can get down to the simplest case. Charles -Original Message- From: owner-openssl-us...@openssl.org [mailto:owner-openssl-us...@openssl.org] On Behalf Of Thomas Sent: Monday, September 17, 2012 11:36 AM To: Michel Cc: openssl-users@openssl.org Subject: Re: Memory issues with ssl handshake Hi again, I've changed the code to reuse the SSL contexts but in terms of memory consumption/release it did not change much - if anything at all. By the way, is there a way to unload a certificate once it has been loaded into a SSL context via SSL_CTX_use_certificate() ? I didn't find anything in the docs and simply specifying NULL as cert parameter caused a crash in OpenSSL. The only places left that cause memory leaks are reported inside OpenSSL as in at 0x68EAC8B: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==27041==by 0x6C472DB: default_malloc_ex (mem.c:79) ==27041==by 0x6C4795F: CRYPTO_malloc (mem.c:306) ==27041==by 0x6C73940: bn_expand_internal (bn_lib.c:336) ==27041==by 0x6C73AE0: bn_expand2 (bn_lib.c:451) ==27041==by 0x6C73BB2: BN_set_bit (bn_lib.c:730) ==27041==by 0x6C7E16E: BN_MONT_CTX_set (bn_mont.c:514) ==27041==by 0x6C7E402: BN_MONT_CTX_set_locked (bn_mont.c:552) ==27041==by 0x6C95B56: RSA_eay_mod_exp (rsa_eay.c:782) ==27041==by 0x6C96422: RSA_eay_private_decrypt (rsa_eay.c:565) ==27041==by 0x6C97EDF: RSA_private_decrypt (rsa_lib.c:303) ==27041==by 0x6942918: ssl3_get_client_key_exchange (s3_srvr.c:2038) ==27041==by 0x6946693: ssl3_accept (s3_srvr.c:529) ==27041==by 0x69513CA: ssl3_read_bytes (s3_pkt.c:941) ==27041==by 0x694C688: ssl3_read_internal (s3_lib.c:3274) ==27041==by 0x69642E8: SSL_read (ssl_lib.c:954) Sometimes these are flagged still reachable and sometimes indirectly lost, usually both types are reported as I get a large amount of these traces. One thing I noticed is that all goes well if I cause the code to run sequentially (e.g. cause requests to come one ater another). Yet it starts eating up memory like crazy if I cause several (HTTPS) requests to come at once. I'm at a loss here. Valgrind insists the leaks happen in OpenSSL code. I'll be happy to supply more information if anyone has an idea of how to approach this. Regards, Thomas On 09/13/2012 12:30 PM, Michel wrote: Hi again Thomas, Do you really need to free your context each time you free your TLS session ? I believe it is not needed and at least not usual. If you need several *DIFFERENT* contexts, implying different TLS configurations/setup, wich, I think, is not so common, you can keep them 'alive' during all your app 'run', even in multi-threaded programs. It would allow you to access some activity informations like the ones documented in : http://www.openssl.org/docs/ssl/SSL_CTX_sess_number.html Hope this helps, Regards Le 13/09/2012 10:39, Thomas a écrit : Hi Michel, Thanks for trying to help, I really appreciate it :-) Does your app setup and free a context each time a client is connecting ? The context is created only when a client requests a HTTPS connection and is destroyed together with the SSL session once the connection goes down. It is rather related to connections then to clients since one client can open several connections but I think you implied one connection per client and then the answer is 'yes'. I will try freeing the session before the context and come back with the results. Regards, Thomas __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: Memory issues with ssl handshake
Hi again, I've changed the code to reuse the SSL contexts but in terms of memory consumption/release it did not change much - if anything at all. By the way, is there a way to unload a certificate once it has been loaded into a SSL context via SSL_CTX_use_certificate() ? I didn't find anything in the docs and simply specifying NULL as cert parameter caused a crash in OpenSSL. The only places left that cause memory leaks are reported inside OpenSSL as in at 0x68EAC8B: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==27041==by 0x6C472DB: default_malloc_ex (mem.c:79) ==27041==by 0x6C4795F: CRYPTO_malloc (mem.c:306) ==27041==by 0x6C73940: bn_expand_internal (bn_lib.c:336) ==27041==by 0x6C73AE0: bn_expand2 (bn_lib.c:451) ==27041==by 0x6C73BB2: BN_set_bit (bn_lib.c:730) ==27041==by 0x6C7E16E: BN_MONT_CTX_set (bn_mont.c:514) ==27041==by 0x6C7E402: BN_MONT_CTX_set_locked (bn_mont.c:552) ==27041==by 0x6C95B56: RSA_eay_mod_exp (rsa_eay.c:782) ==27041==by 0x6C96422: RSA_eay_private_decrypt (rsa_eay.c:565) ==27041==by 0x6C97EDF: RSA_private_decrypt (rsa_lib.c:303) ==27041==by 0x6942918: ssl3_get_client_key_exchange (s3_srvr.c:2038) ==27041==by 0x6946693: ssl3_accept (s3_srvr.c:529) ==27041==by 0x69513CA: ssl3_read_bytes (s3_pkt.c:941) ==27041==by 0x694C688: ssl3_read_internal (s3_lib.c:3274) ==27041==by 0x69642E8: SSL_read (ssl_lib.c:954) Sometimes these are flagged still reachable and sometimes indirectly lost, usually both types are reported as I get a large amount of these traces. One thing I noticed is that all goes well if I cause the code to run sequentially (e.g. cause requests to come one ater another). Yet it starts eating up memory like crazy if I cause several (HTTPS) requests to come at once. I'm at a loss here. Valgrind insists the leaks happen in OpenSSL code. I'll be happy to supply more information if anyone has an idea of how to approach this. Regards, Thomas On 09/13/2012 12:30 PM, Michel wrote: Hi again Thomas, Do you really need to free your context each time you free your TLS session ? I believe it is not needed and at least not usual. If you need several *DIFFERENT* contexts, implying different TLS configurations/setup, wich, I think, is not so common, you can keep them 'alive' during all your app 'run', even in multi-threaded programs. It would allow you to access some activity informations like the ones documented in : http://www.openssl.org/docs/ssl/SSL_CTX_sess_number.html Hope this helps, Regards Le 13/09/2012 10:39, Thomas a écrit : Hi Michel, Thanks for trying to help, I really appreciate it :-) Does your app setup and free a context each time a client is connecting ? The context is created only when a client requests a HTTPS connection and is destroyed together with the SSL session once the connection goes down. It is rather related to connections then to clients since one client can open several connections but I think you implied one connection per client and then the answer is 'yes'. I will try freeing the session before the context and come back with the results. Regards, Thomas __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: Memory issues with ssl handshake
Hi Thomas, I don't know if it makes a real difference concerning memory leaks, but I would have freed the TLS session *BEFORE* freing the context : SSL_free(ssl); ... SSL_CTX_free(ssl-ctx); Does your app setup and free a context each time a client is connecting ? Le 12/09/2012 16:58, Thomas Eckert a écrit : Hi, Valgrind gives me a *lot* of messages like this ==19021== 2,056 bytes in 2 blocks are indirectly lost in loss record 186 of 190 ==19021==at 0x68EAC8B: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==19021==by 0x6C472DB: default_malloc_ex (mem.c:79) ==19021==by 0x6C4795F: CRYPTO_malloc (mem.c:306) ==19021==by 0x6C73940: bn_expand_internal (bn_lib.c:336) ==19021==by 0x6C73AE0: bn_expand2 (bn_lib.c:451) ==19021==by 0x6C73BB2: BN_set_bit (bn_lib.c:730) ==19021==by 0x6C7E16E: BN_MONT_CTX_set (bn_mont.c:514) ==19021==by 0x6C7E402: BN_MONT_CTX_set_locked (bn_mont.c:552) ==19021==by 0x6C97053: RSA_eay_public_decrypt (rsa_eay.c:693) ==19021==by 0x6C97F1F: RSA_public_decrypt (rsa_lib.c:309) ==19021==by 0x6C98BEB: int_rsa_verify (rsa_sign.c:182) ==19021==by 0x6C98F57: RSA_verify (rsa_sign.c:284) ==19021==by 0x6949E16: ssl3_get_key_exchange (s3_clnt.c:1562) ==19021==by 0x694BF5F: ssl3_connect (s3_clnt.c:335) ==19021==by 0x6964AB9: SSL_connect (ssl_lib.c:933) ==19021==by 0x6953FDD: ssl23_connect (s23_clnt.c:693) ==19021==by 0x6963FBC: SSL_do_handshake (ssl_lib.c:2368) ==19021==by 0x8060D14: ssl_connect (ssl.c:1078) Compared to the above they vary only in one line, e.g. ==19021== 1,024 bytes in 2 blocks are indirectly lost in loss record 181 of 190 ==19021==at 0x68EAC8B: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==19021==by 0x6C472DB: default_malloc_ex (mem.c:79) ==19021==by 0x6C4795F: CRYPTO_malloc (mem.c:306) ==19021==by 0x6C73940: bn_expand_internal (bn_lib.c:336) ==19021==by 0x6C73AE0: bn_expand2 (bn_lib.c:451) ==19021==by 0x6C73D53: BN_copy (bn_lib.c:506) ==19021==by 0x6C7DFCA: BN_MONT_CTX_set (bn_mont.c:421) ==19021==by 0x6C7E402: BN_MONT_CTX_set_locked (bn_mont.c:552) ==19021==by 0x6C97053: RSA_eay_public_decrypt (rsa_eay.c:693) ==19021==by 0x6C97F1F: RSA_public_decrypt (rsa_lib.c:309) ==19021==by 0x6C98BEB: int_rsa_verify (rsa_sign.c:182) ==19021==by 0x6C98F57: RSA_verify (rsa_sign.c:284) ==19021==by 0x6949E16: ssl3_get_key_exchange (s3_clnt.c:1562) ==19021==by 0x694BF5F: ssl3_connect (s3_clnt.c:335) ==19021==by 0x6964AB9: SSL_connect (ssl_lib.c:933) ==19021==by 0x6953FDD: ssl23_connect (s23_clnt.c:693) ==19021==by 0x6963FBC: SSL_do_handshake (ssl_lib.c:2368) ==19021==by 0x8060D14: ssl_connect (ssl.c:1078) The application in question is a http proxy and is leaking memory heavily. Actually, Valgrind also reports many entries as still reachable instead of indirectly lost but even those regions are mentioned by Valgrind like the above two blocks. I doubt OpenSSL has such blatant memory leaks but I have been searching for the leaks for some time now and just cannot find anything - also due to the fact that Valgrind reports all the leaks like above, which is not really helpful. I do realize that for detailed help I would need to give some code on how the connections are set up but that code is pretty large. So best would be some hint as to where to look for or what regions of code to supply. Looking for hints as to what is going wrong I went through some of the OpenSSL functions mentioned above and after leaving my code and entering OpenSSL code the only thing being passed around the functions is the actual SSL object - and no buffers of any kind. So I suppose the source of the problem lies in setting up the object, it's context or what ever is stuck to it. The first thing I checked on was how the SSL connections are torn down in my application: ERR_remove_state(0); SSL_CTX_free(ssl-ctx); SSL_free(ssl); The first line was added only after starting the search for the leaks and I'm still not sure if that line doesn't break anything. I did find some minor leaks in my code along the way (e.g. a forgotten ASN1_INTEGER_free()) but they were neither easy to find - because Valgrind also reported them like the above two blocks, meaning in the wrong place! - nor did they change much. Does anyone have some tips for me ? __ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: Memory issues with ssl handshake
Hi again Thomas, Do you really need to free your context each time you free your TLS session ? I believe it is not needed and at least not usual. If you need several *DIFFERENT* contexts, implying different TLS configurations/setup, wich, I think, is not so common, you can keep them 'alive' during all your app 'run', even in multi-threaded programs. It would allow you to access some activity informations like the ones documented in : http://www.openssl.org/docs/ssl/SSL_CTX_sess_number.html Hope this helps, Regards Le 13/09/2012 10:39, Thomas a écrit : Hi Michel, Thanks for trying to help, I really appreciate it :-) Does your app setup and free a context each time a client is connecting ? The context is created only when a client requests a HTTPS connection and is destroyed together with the SSL session once the connection goes down. It is rather related to connections then to clients since one client can open several connections but I think you implied one connection per client and then the answer is 'yes'. I will try freeing the session before the context and come back with the results. Regards, Thomas __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: Memory issues with ssl handshake
Hi Michel, Thanks for trying to help, I really appreciate it :-) Does your app setup and free a context each time a client is connecting ? The context is created only when a client requests a HTTPS connection and is destroyed together with the SSL session once the connection goes down. It is rather related to connections then to clients since one client can open several connections but I think you implied one connection per client and then the answer is 'yes'. I will try freeing the session before the context and come back with the results. Regards, Thomas On 09/13/2012 10:09 AM, Michel wrote: Hi Thomas, I don't know if it makes a real difference concerning memory leaks, but I would have freed the TLS session *BEFORE* freing the context : SSL_free(ssl); ... SSL_CTX_free(ssl-ctx); Does your app setup and free a context each time a client is connecting ? Le 12/09/2012 16:58, Thomas a écrit : Hi, Valgrind gives me a *lot* of messages like this ==19021== 2,056 bytes in 2 blocks are indirectly lost in loss record 186 of 190 ==19021==at 0x68EAC8B: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==19021==by 0x6C472DB: default_malloc_ex (mem.c:79) ==19021==by 0x6C4795F: CRYPTO_malloc (mem.c:306) ==19021==by 0x6C73940: bn_expand_internal (bn_lib.c:336) ==19021==by 0x6C73AE0: bn_expand2 (bn_lib.c:451) ==19021==by 0x6C73BB2: BN_set_bit (bn_lib.c:730) ==19021==by 0x6C7E16E: BN_MONT_CTX_set (bn_mont.c:514) ==19021==by 0x6C7E402: BN_MONT_CTX_set_locked (bn_mont.c:552) ==19021==by 0x6C97053: RSA_eay_public_decrypt (rsa_eay.c:693) ==19021==by 0x6C97F1F: RSA_public_decrypt (rsa_lib.c:309) ==19021==by 0x6C98BEB: int_rsa_verify (rsa_sign.c:182) ==19021==by 0x6C98F57: RSA_verify (rsa_sign.c:284) ==19021==by 0x6949E16: ssl3_get_key_exchange (s3_clnt.c:1562) ==19021==by 0x694BF5F: ssl3_connect (s3_clnt.c:335) ==19021==by 0x6964AB9: SSL_connect (ssl_lib.c:933) ==19021==by 0x6953FDD: ssl23_connect (s23_clnt.c:693) ==19021==by 0x6963FBC: SSL_do_handshake (ssl_lib.c:2368) ==19021==by 0x8060D14: ssl_connect (ssl.c:1078) Compared to the above they vary only in one line, e.g. ==19021== 1,024 bytes in 2 blocks are indirectly lost in loss record 181 of 190 ==19021==at 0x68EAC8B: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==19021==by 0x6C472DB: default_malloc_ex (mem.c:79) ==19021==by 0x6C4795F: CRYPTO_malloc (mem.c:306) ==19021==by 0x6C73940: bn_expand_internal (bn_lib.c:336) ==19021==by 0x6C73AE0: bn_expand2 (bn_lib.c:451) ==19021==by 0x6C73D53: BN_copy (bn_lib.c:506) ==19021==by 0x6C7DFCA: BN_MONT_CTX_set (bn_mont.c:421) ==19021==by 0x6C7E402: BN_MONT_CTX_set_locked (bn_mont.c:552) ==19021==by 0x6C97053: RSA_eay_public_decrypt (rsa_eay.c:693) ==19021==by 0x6C97F1F: RSA_public_decrypt (rsa_lib.c:309) ==19021==by 0x6C98BEB: int_rsa_verify (rsa_sign.c:182) ==19021==by 0x6C98F57: RSA_verify (rsa_sign.c:284) ==19021==by 0x6949E16: ssl3_get_key_exchange (s3_clnt.c:1562) ==19021==by 0x694BF5F: ssl3_connect (s3_clnt.c:335) ==19021==by 0x6964AB9: SSL_connect (ssl_lib.c:933) ==19021==by 0x6953FDD: ssl23_connect (s23_clnt.c:693) ==19021==by 0x6963FBC: SSL_do_handshake (ssl_lib.c:2368) ==19021==by 0x8060D14: ssl_connect (ssl.c:1078) The application in question is a http proxy and is leaking memory heavily. Actually, Valgrind also reports many entries as still reachable instead of indirectly lost but even those regions are mentioned by Valgrind like the above two blocks. I doubt OpenSSL has such blatant memory leaks but I have been searching for the leaks for some time now and just cannot find anything - also due to the fact that Valgrind reports all the leaks like above, which is not really helpful. I do realize that for detailed help I would need to give some code on how the connections are set up but that code is pretty large. So best would be some hint as to where to look for or what regions of code to supply. Looking for hints as to what is going wrong I went through some of the OpenSSL functions mentioned above and after leaving my code and entering OpenSSL code the only thing being passed around the functions is the actual SSL object - and no buffers of any kind. So I suppose the source of the problem lies in setting up the object, it's context or what ever is stuck to it. The first thing I checked on was how the SSL connections are torn down in my application: ERR_remove_state(0); SSL_CTX_free(ssl-ctx); SSL_free(ssl); The first line was added only after starting the search for the leaks and I'm still not sure if that line doesn't break anything. I did find some minor leaks in my code along the way (e.g. a forgotten ASN1_INTEGER_free()) but they were neither easy to find - because Valgrind also reported them like the above two blocks, meaning in the wrong place! - nor did they change much.
Re: Memory issues with ssl handshake
On 12.09.2012 16:58, Thomas Eckert wrote: Hi, Valgrind gives me a *lot* of messages like this [snip] Does anyone have some tips for me ? Have you seen the FAQ entry regarding valgrind: http://www.openssl.org/support/faq.html#PROG14 ? Regards, Dominik __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
RE: Memory issues with ssl handshake
Not sure if it help at all but FWIW I am using Windows pre-built OpenSSL dll's with server client certificates, CAs, DH keys, and a CRL, and I am seeing no memory leaks whatsoever in Visual Studio 2010. Charles -Original Message- From: owner-openssl-us...@openssl.org [mailto:owner-openssl-us...@openssl.org] On Behalf Of Thomas Eckert Sent: Wednesday, September 12, 2012 7:59 AM To: openssl-users@openssl.org Subject: Memory issues with ssl handshake Hi, Valgrind gives me a *lot* of messages like this ==19021== 2,056 bytes in 2 blocks are indirectly lost in loss record 186 of 190 ==19021==at 0x68EAC8B: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==19021==by 0x6C472DB: default_malloc_ex (mem.c:79) ==19021==by 0x6C4795F: CRYPTO_malloc (mem.c:306) ==19021==by 0x6C73940: bn_expand_internal (bn_lib.c:336) ==19021==by 0x6C73AE0: bn_expand2 (bn_lib.c:451) ==19021==by 0x6C73BB2: BN_set_bit (bn_lib.c:730) ==19021==by 0x6C7E16E: BN_MONT_CTX_set (bn_mont.c:514) ==19021==by 0x6C7E402: BN_MONT_CTX_set_locked (bn_mont.c:552) ==19021==by 0x6C97053: RSA_eay_public_decrypt (rsa_eay.c:693) ==19021==by 0x6C97F1F: RSA_public_decrypt (rsa_lib.c:309) ==19021==by 0x6C98BEB: int_rsa_verify (rsa_sign.c:182) ==19021==by 0x6C98F57: RSA_verify (rsa_sign.c:284) ==19021==by 0x6949E16: ssl3_get_key_exchange (s3_clnt.c:1562) ==19021==by 0x694BF5F: ssl3_connect (s3_clnt.c:335) ==19021==by 0x6964AB9: SSL_connect (ssl_lib.c:933) ==19021==by 0x6953FDD: ssl23_connect (s23_clnt.c:693) ==19021==by 0x6963FBC: SSL_do_handshake (ssl_lib.c:2368) ==19021==by 0x8060D14: ssl_connect (ssl.c:1078) Compared to the above they vary only in one line, e.g. ==19021== 1,024 bytes in 2 blocks are indirectly lost in loss record 181 of 190 ==19021==at 0x68EAC8B: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==19021==by 0x6C472DB: default_malloc_ex (mem.c:79) ==19021==by 0x6C4795F: CRYPTO_malloc (mem.c:306) ==19021==by 0x6C73940: bn_expand_internal (bn_lib.c:336) ==19021==by 0x6C73AE0: bn_expand2 (bn_lib.c:451) ==19021==by 0x6C73D53: BN_copy (bn_lib.c:506) ==19021==by 0x6C7DFCA: BN_MONT_CTX_set (bn_mont.c:421) ==19021==by 0x6C7E402: BN_MONT_CTX_set_locked (bn_mont.c:552) ==19021==by 0x6C97053: RSA_eay_public_decrypt (rsa_eay.c:693) ==19021==by 0x6C97F1F: RSA_public_decrypt (rsa_lib.c:309) ==19021==by 0x6C98BEB: int_rsa_verify (rsa_sign.c:182) ==19021==by 0x6C98F57: RSA_verify (rsa_sign.c:284) ==19021==by 0x6949E16: ssl3_get_key_exchange (s3_clnt.c:1562) ==19021==by 0x694BF5F: ssl3_connect (s3_clnt.c:335) ==19021==by 0x6964AB9: SSL_connect (ssl_lib.c:933) ==19021==by 0x6953FDD: ssl23_connect (s23_clnt.c:693) ==19021==by 0x6963FBC: SSL_do_handshake (ssl_lib.c:2368) ==19021==by 0x8060D14: ssl_connect (ssl.c:1078) The application in question is a http proxy and is leaking memory heavily. Actually, Valgrind also reports many entries as still reachable instead of indirectly lost but even those regions are mentioned by Valgrind like the above two blocks. I doubt OpenSSL has such blatant memory leaks but I have been searching for the leaks for some time now and just cannot find anything - also due to the fact that Valgrind reports all the leaks like above, which is not really helpful. I do realize that for detailed help I would need to give some code on how the connections are set up but that code is pretty large. So best would be some hint as to where to look for or what regions of code to supply. Looking for hints as to what is going wrong I went through some of the OpenSSL functions mentioned above and after leaving my code and entering OpenSSL code the only thing being passed around the functions is the actual SSL object - and no buffers of any kind. So I suppose the source of the problem lies in setting up the object, it's context or what ever is stuck to it. The first thing I checked on was how the SSL connections are torn down in my application: ERR_remove_state(0); SSL_CTX_free(ssl-ctx); SSL_free(ssl); The first line was added only after starting the search for the leaks and I'm still not sure if that line doesn't break anything. I did find some minor leaks in my code along the way (e.g. a forgotten ASN1_INTEGER_free()) but they were neither easy to find - because Valgrind also reported them like the above two blocks, meaning in the wrong place! - nor did they change much. Does anyone have some tips for me ? __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
