RE: Question regarding to memory leak
Thanks all of you for providing the inputs, and with your help, I think I found out the problem. In my test program I am calling the function X509_STORE_get_by_subject(ctx, X509_LU_CRL, issuer, obj) to get the CRL list. But after this function call, I didn't use the X509_OBJECT_free_contents() to free the contents of the returned object obj. It seems this is the cause of the memory leak. By adding X509_OBJECT_free_contents(obj) function, the Purify does not report any memory leaks. Thanks Bob -Original Message- From: owner-openssl-us...@openssl.org [mailto:owner-openssl-us...@openssl.org] On Behalf Of Chris Dodd Sent: Friday, June 24, 2011 5:30 PM To: openssl-users@openssl.org Subject: Re: Question regarding to memory leak On Fri, Jun 24, 2011, Yan, Bob wrote: Hi, I have used IBM purify to check my test program which invokes openssl library. There are some memory leaks reported by Purify, please see below. Could somebody point to me from which function those leaks were generated, and how to avoid those leaks? Thanks, Bob MLK: 1104 bytes leaked in 46 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_STRING_type_new [libcrypto.so.1.0.0] ASN1_primitive_new [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 24 bytes (46 times); last block at 0x2c0f7218 I've used valgrind with OpenSSL based programs to find memory leaks, and with it, you can use a --num-callers=N option to get deeper stack traces in the leak reports. You need a depth of 15-20 to get far enough to see where your code is calling into the OpenSSL code in most cases. There's probably a similar option for IBM purify. -chris __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Question regarding to memory leak
Hi, I have used IBM purify to check my test program which invokes openssl library. There are some memory leaks reported by Purify, please see below. Could somebody point to me from which function those leaks were generated, and how to avoid those leaks? Thanks, Bob MLK: 1104 bytes leaked in 46 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_STRING_type_new [libcrypto.so.1.0.0] ASN1_primitive_new [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 24 bytes (46 times); last block at 0x2c0f7218 MLK: 836 bytes leaked in 2 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] asn1_enc_save [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] asn1_template_ex_d2i [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] ASN1_item_d2i [libcrypto.so.1.0.0] Block of 432 bytes at 0x2c093668 Block of 404 bytes at 0x2c009d58 MLK: 800 bytes leaked in 20 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_OBJECT_new [libcrypto.so.1.0.0] c2i_ASN1_OBJECT [libcrypto.so.1.0.0] asn1_ex_c2i[libcrypto.so.1.0.0] asn1_d2i_ex_primitive [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 40 bytes (20 times); last block at 0x2c097a78 MLK: 600 bytes leaked in 15 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] asn1_template_ex_d2i [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 40 bytes (15 times); last block at 0x2c0f6e98 MLK: 376 bytes leaked in 33 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_STRING_set [libcrypto.so.1.0.0] asn1_ex_c2i[libcrypto.so.1.0.0] asn1_d2i_ex_primitive [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] asn1_template_ex_d2i [libcrypto.so.1.0.0] Block of 14 bytes (21 times); last block at 0x2c0f7168 Block of 12 bytes (2 times); last block at 0x2c096828 Block of 11 bytes (2 times); last block at 0x2c096338 Block of 5 bytes (6 times); last block at 0x2c0976f8 Block of 3 bytes (2 times); last block at 0x2c095e58 . . .
Re: Question regarding to memory leak
As a general comment not all memory leaks reported by these tools are a bad thing. I often write code that has these type of leaks on purpose for performance reasons. For example a function that is called often and malloc's memory ... rather than malloc and free each time (causing context switching and generally slow) I just make the pointer static and a size variable static. I use the pointer until it is too small, then I realloc to a larger size. In modern systems often the leak is worth the performance gain. I run on IBM AIX. Having said that, I have not dug in to your specifics which may just be bugs, an error with the tool, or deliberate. Eric At 11:58 AM 6/24/2011, Yan, Bob wrote: Hi, I have used IBM purify to check my test program which invokes openssl library. There are some memory leaks reported by Purify, please see below. Could somebody point to me from which function those leaks were generated, and how to avoid those leaks? Thanks, Bob MLK: 1104 bytes leaked in 46 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_STRING_type_new [libcrypto.so.1.0.0] ASN1_primitive_new [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 24 bytes (46 times); last block at 0x2c0f7218 MLK: 836 bytes leaked in 2 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] asn1_enc_save [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] asn1_template_ex_d2i [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] ASN1_item_d2i [libcrypto.so.1.0.0] Block of 432 bytes at 0x2c093668 Block of 404 bytes at 0x2c009d58 MLK: 800 bytes leaked in 20 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_OBJECT_new [libcrypto.so.1.0.0] c2i_ASN1_OBJECT [libcrypto.so.1.0.0] asn1_ex_c2i[libcrypto.so.1.0.0] asn1_d2i_ex_primitive [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 40 bytes (20 times); last block at 0x2c097a78 MLK: 600 bytes leaked in 15 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] asn1_template_ex_d2i [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 40 bytes (15 times); last block at 0x2c0f6e98 MLK: 376 bytes leaked in 33 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_STRING_set [libcrypto.so.1.0.0] asn1_ex_c2i[libcrypto.so.1.0.0] asn1_d2i_ex_primitive [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] asn1_template_ex_d2i [libcrypto.so.1.0.0] Block of 14 bytes (21 times); last block at 0x2c0f7168 Block of 12 bytes (2 times); last block at 0x2c096828 Block of 11 bytes (2 times); last block at 0x2c096338 Block of 5 bytes (6 times); last block at 0x2c0976f8 Block of 3 bytes (2 times); last block at 0x2c095e58 . . . Eric S. Eberhard (928) 567-3727 Voice (928) 567-6122 Fax (928) 301-7537 Cell Vertical Integrated Computer Systems, LLC Metropolis Support, LLC For Metropolis support and VICS MBA Supporthttp://www.vicsmba.com Pictures of Snake in Spring http://www.facebook.com/album.php?aid=115547id=1409661701l=1c375e1f49 Pictures of Camp Verde http://www.facebook.com/album.php?aid=12771id=1409661701l=fc0e0a2bcf Pictures of Land Cruiser in Sedona http://www.facebook.com/album.php?aid=50953id=1409661701 Pictures of Flagstaff area near our cabin http://www.facebook.com/album.php?aid=12750id=1409661701 Pictures of Cheryl in a Horse Show http://www.facebook.com/album.php?aid=32484id=1409661701 Pictures of the AZ Desert
Re: Question regarding to memory leak
I have suggested this before. Write your own memtools.
http://www.terralogic.net/developer/developer.html
I tossed up a couple poor boy examples.
Note the calls:
struct pfa_ControlBlock chain1 = { ipfa_ControlBlock }
, chain2 = { ipfa_ControlBlock };
pfa_Init( chain1, 8172 );
pfa_Init( chain2, 8172 );
p = pfa_alloc( chain1, 10, ierr );
In OpenSSL for instance each connection is identified by an fd (for the socket)
and there are a fixed number of these.
One can set up an array of pointers and allocate a page of memory for each
connection at the time the connection is opened.
One wants a 1:1 correspondence between the fd flags from select() and the
pfa_Control_Block pointer array.
Then when _ANY_ memory is needed simply use pfa_alloc() with the approriate
control block of which the address can easily be looked up from the pointer
array.
At any time if the connection is lost... forget it. Close the socket and the
next time the fd is used the old pages will be fetched from virtual memory as
required. No leaks! Also all memory for a connection will be in a small group
of pages which will be flushed to the swap file and they will stay there
because there will never be any competing connections using that pool of pages!
(malloc will shuffle memory for many connections into a given set of pages).
If we want to actally free the memory because we are closing the fd then its
simple... follow the linked list and free the pages.
-
Next it really doesn't matter if malloc and pfa_alloc() are both used. Any
structres which use pfa_blah will live in a world of their own which is totally
compatible with anything malloc() might do. We just wouldn't want a struct
allocated with pfa_blah() pointed to by something allocated with malloc()
because if we call pfa_init() then all those pointers in the malloc()ed structs
are invalid. However by definition if we call pfa_init() then that fd is dead
so anything that points to anything allocated for that dead fd is likely
invalid anyways.
The thing is this. One can free and reallocate in a pfa_blah page pool. I
didn't need to do this so I never bothered. But it can be done. Anything
which is not actually freed can be re-initialized at any time by calling
pfa_init(). At that point all pages are wiped and ready for re-use. One
doesn't have to go through all sorts of linked structures and determine peice
by piece what was and what was not allocated and whether something has been
freed. There is one master and simple linked list structure and that is found
from the fd which identifies the socket.
-
As I said I offered this a few years ago. I was told there is already some
sort of memory management being used. But I keep seeing comments on leaks so
I'm not sure what is done because I think if one uses the fd one can sort of
allocate a pool of pages for each connection and as long as all malloc() is
done within that pool then we shouldn't have leaks.
Also if all malloc()s for an fd are in their own pool of pages then the machine
can swap that connection out if required and if the connection times out and
dies jsut re-initialize it when the fd is re-opened (on another connection).
On Fri, Jun 24, 2011 at 01:20:42PM -0700, Eric S. Eberhard wrote:
As a general comment not all memory leaks reported by these tools are
a bad thing. I often write code that has these type of leaks on
purpose for performance reasons. For example a function that is
called often and malloc's memory ... rather than malloc and free each
time (causing context switching and generally slow) I just make the
pointer static and a size variable static. I use the pointer until
it is too small, then I realloc to a larger size. In modern systems
often the leak is worth the performance gain. I run on IBM
AIX. Having said that, I have not dug in to your specifics which may
just be bugs, an error with the tool, or deliberate. Eric
At 11:58 AM 6/24/2011, Yan, Bob wrote:
Hi,
I have used IBM purify to check my test program which invokes
openssl library. There are some memory leaks reported by Purify,
please see below. Could somebody point to me from which function
those leaks were generated, and how to avoid those leaks? Thanks, Bob
MLK: 1104 bytes leaked in 46 blocks
This memory was allocated from:
malloc [rtlib.o]
CRYPTO_malloc [libcrypto.so.1.0.0]
ASN1_STRING_type_new [libcrypto.so.1.0.0]
ASN1_primitive_new [libcrypto.so.1.0.0]
asn1_item_ex_combine_new [libcrypto.so.1.0.0]
asn1_item_ex_combine_new [libcrypto.so.1.0.0]
ASN1_item_ex_d2i [libcrypto.so.1.0.0]
asn1_template_noexp_d2i [libcrypto.so.1.0.0]
Block of 24 bytes (46 times); last block at 0x2c0f7218
MLK: 836 bytes
RE: Question regarding to memory leak
Thank you and Eric for the inputs. I will look at them and see what I should do
with this.
Have a nice weekend.
Bob
-Original Message-
From: owner-openssl-us...@openssl.org [mailto:owner-openssl-us...@openssl.org]
On Behalf Of t...@terralogic.net
Sent: Friday, June 24, 2011 4:52 PM
To: openssl-users@openssl.org
Subject: Re: Question regarding to memory leak
I have suggested this before. Write your own memtools.
http://www.terralogic.net/developer/developer.html
I tossed up a couple poor boy examples.
Note the calls:
struct pfa_ControlBlock chain1 = { ipfa_ControlBlock }
, chain2 = { ipfa_ControlBlock };
pfa_Init( chain1, 8172 );
pfa_Init( chain2, 8172 );
p = pfa_alloc( chain1, 10, ierr );
In OpenSSL for instance each connection is identified by an fd (for the socket)
and there are a fixed number of these.
One can set up an array of pointers and allocate a page of memory for each
connection at the time the connection is opened.
One wants a 1:1 correspondence between the fd flags from select() and the
pfa_Control_Block pointer array.
Then when _ANY_ memory is needed simply use pfa_alloc() with the approriate
control block of which the address can easily be looked up from the pointer
array.
At any time if the connection is lost... forget it. Close the socket and the
next time the fd is used the old pages will be fetched from virtual memory as
required. No leaks! Also all memory for a connection will be in a small group
of pages which will be flushed to the swap file and they will stay there
because there will never be any competing connections using that pool of pages!
(malloc will shuffle memory for many connections into a given set of pages).
If we want to actally free the memory because we are closing the fd then its
simple... follow the linked list and free the pages.
-
Next it really doesn't matter if malloc and pfa_alloc() are both used. Any
structres which use pfa_blah will live in a world of their own which is totally
compatible with anything malloc() might do. We just wouldn't want a struct
allocated with pfa_blah() pointed to by something allocated with malloc()
because if we call pfa_init() then all those pointers in the malloc()ed structs
are invalid. However by definition if we call pfa_init() then that fd is dead
so anything that points to anything allocated for that dead fd is likely
invalid anyways.
The thing is this. One can free and reallocate in a pfa_blah page pool. I
didn't need to do this so I never bothered. But it can be done. Anything
which is not actually freed can be re-initialized at any time by calling
pfa_init(). At that point all pages are wiped and ready for re-use. One
doesn't have to go through all sorts of linked structures and determine peice
by piece what was and what was not allocated and whether something has been
freed. There is one master and simple linked list structure and that is found
from the fd which identifies the socket.
-
As I said I offered this a few years ago. I was told there is already some
sort of memory management being used. But I keep seeing comments on leaks so
I'm not sure what is done because I think if one uses the fd one can sort of
allocate a pool of pages for each connection and as long as all malloc() is
done within that pool then we shouldn't have leaks.
Also if all malloc()s for an fd are in their own pool of pages then the machine
can swap that connection out if required and if the connection times out and
dies jsut re-initialize it when the fd is re-opened (on another connection).
On Fri, Jun 24, 2011 at 01:20:42PM -0700, Eric S. Eberhard wrote:
As a general comment not all memory leaks reported by these tools are
a bad thing. I often write code that has these type of leaks on
purpose for performance reasons. For example a function that is
called often and malloc's memory ... rather than malloc and free each
time (causing context switching and generally slow) I just make the
pointer static and a size variable static. I use the pointer until it
is too small, then I realloc to a larger size. In modern systems
often the leak is worth the performance gain. I run on IBM AIX.
Having said that, I have not dug in to your specifics which may just
be bugs, an error with the tool, or deliberate. Eric
At 11:58 AM 6/24/2011, Yan, Bob wrote:
Hi,
I have used IBM purify to check my test program which invokes openssl
library. There are some memory leaks reported by Purify, please see
below. Could somebody point to me from which function those leaks
were generated, and how to avoid those leaks? Thanks, Bob
MLK: 1104 bytes leaked in 46 blocks
This memory was allocated from:
malloc [rtlib.o]
CRYPTO_malloc [libcrypto.so.1.0.0]
ASN1_STRING_type_new [libcrypto.so.1.0.0
Re: Question regarding to memory leak
On Fri, Jun 24, 2011, Yan, Bob wrote: Hi, I have used IBM purify to check my test program which invokes openssl library. There are some memory leaks reported by Purify, please see below. Could somebody point to me from which function those leaks were generated, and how to avoid those leaks? Thanks, Bob MLK: 1104 bytes leaked in 46 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_STRING_type_new [libcrypto.so.1.0.0] ASN1_primitive_new [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 24 bytes (46 times); last block at 0x2c0f7218 MLK: 836 bytes leaked in 2 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] asn1_enc_save [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] asn1_template_ex_d2i [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] ASN1_item_d2i [libcrypto.so.1.0.0] Block of 432 bytes at 0x2c093668 Block of 404 bytes at 0x2c009d58 MLK: 800 bytes leaked in 20 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_OBJECT_new [libcrypto.so.1.0.0] c2i_ASN1_OBJECT [libcrypto.so.1.0.0] asn1_ex_c2i[libcrypto.so.1.0.0] asn1_d2i_ex_primitive [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 40 bytes (20 times); last block at 0x2c097a78 MLK: 600 bytes leaked in 15 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] asn1_template_ex_d2i [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 40 bytes (15 times); last block at 0x2c0f6e98 MLK: 376 bytes leaked in 33 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_STRING_set [libcrypto.so.1.0.0] asn1_ex_c2i[libcrypto.so.1.0.0] asn1_d2i_ex_primitive [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] asn1_template_ex_d2i [libcrypto.so.1.0.0] Block of 14 bytes (21 times); last block at 0x2c0f7168 Block of 12 bytes (2 times); last block at 0x2c096828 Block of 11 bytes (2 times); last block at 0x2c096338 Block of 5 bytes (6 times); last block at 0x2c0976f8 Block of 3 bytes (2 times); last block at 0x2c095e58 . . You can enable the internal leak checking by doing something like: CRYPTO_memcheck_ctrl(CRYPTO_MEM_CHECK_ON); then later on: CRYPTO_mem_leaks(stderr); Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: Question regarding to memory leak
Bob,
Just a little more explanation here. Suppose we have a bunch of structures
allocted into the pages of memory and a connection goes sour and is timing out.
Suppose our server is under substantial memory pressure. In a situation like
this if _any_ structure on any active connection happens to live in a memory
page otherwise or even largely used by the dead connection, then the machine
CANNOT swap that page. The way malloc() works this will be the normal
situation because stuctures are alloacted temporaly (in the time domain) and by
size. It becomes a real problem when many small structs are malloc()ed.
next, in order to optimize allocation and freeing malloc() actually does quite
a lot of work.
When I wrote that poor boy code I looked for as simple a solution as possible.
In OpenSSL I would make the assumption that the high water mark on pretty much
any connection will be about as high as the high water mark on any other. This
is measured by the total amount of memory needed for all structures relavent to
the given connection. But if this is not so its not really all that terrible
because if the pool of pages is reused then it just grows back into the old
page set as required. Memory pages sitting in a swap file are something I
consider free. It might require more I/O to free them up than to just let them
fester on disk.
Note that I didn't bother with a free space chain. Depending on the nature of
the code if something like the pfa_blah() functions are to be used then a free
chain might be necessary and it might make sense to organise some sort of a
stack as well. Maybe they did this. I never looked. What I am thinking here
is that whenever a function is called it gets a stack marker and automatic
variables are allocated into the stack... but in a forked or threaded model I'm
not sure if each fork() or thread() gets its own stack and how this might
interact with a pfa_blah() scheme indexed by an fd.
Regardless if an fd is not known and is not relevent to a function then I
suppose one could index via a pid? This of course depends on the specfic way
the code is written and a lot of assumptions may not be valid.
To me however I think one observation has to be correct but this will have to
be confirmed. We should be able to collect all malloc()s into a pool of pages
organised around an fd which is what the operating system uses to identify the
connection. If so then we should NOT need to care if that fd is alive or dead.
Somehow the o/s and the code knows about this and hopefully will eventually
close the file if the connection dies. regardless of how this might happen.
If a connection dies then the O/S will collect all control blocks associated
with the fd and somehow make the memory and the fd number available for another
connections.
What this means is that in the code, select() will re-use a dead fd#. If so
then all one really needs to do is call pfa_init() at the time one calls
fopen(). pfa_init() will reset the page pool for the fd or initializes one if
there is none there.
Note the maximum number of file descriptors is FD_SETSIZE which is defined in
sys/select.h.
So it should work and should be easy to layer in.
On Fri, Jun 24, 2011 at 04:35:57PM -0700, Yan, Bob wrote:
Thank you and Eric for the inputs. I will look at them and see what I should
do with this.
Have a nice weekend.
Bob
-Original Message-
From: owner-openssl-us...@openssl.org
[mailto:owner-openssl-us...@openssl.org] On Behalf Of t...@terralogic.net
Sent: Friday, June 24, 2011 4:52 PM
To: openssl-users@openssl.org
Subject: Re: Question regarding to memory leak
I have suggested this before. Write your own memtools.
http://www.terralogic.net/developer/developer.html
I tossed up a couple poor boy examples.
Note the calls:
struct pfa_ControlBlock chain1 = { ipfa_ControlBlock }
, chain2 = { ipfa_ControlBlock };
pfa_Init( chain1, 8172 );
pfa_Init( chain2, 8172 );
p = pfa_alloc( chain1, 10, ierr );
In OpenSSL for instance each connection is identified by an fd (for the
socket) and there are a fixed number of these.
One can set up an array of pointers and allocate a page of memory for each
connection at the time the connection is opened.
One wants a 1:1 correspondence between the fd flags from select() and the
pfa_Control_Block pointer array.
Then when _ANY_ memory is needed simply use pfa_alloc() with the approriate
control block of which the address can easily be looked up from the pointer
array.
At any time if the connection is lost... forget it. Close the socket and the
next time the fd is used the old pages will be fetched from virtual memory as
required. No leaks! Also all memory for a connection will be in a small
group of pages which will be flushed to the swap file and they will stay
there because there will never be any competing
Re: Question regarding to memory leak
On Fri, Jun 24, 2011, Yan, Bob wrote: Hi, I have used IBM purify to check my test program which invokes openssl library. There are some memory leaks reported by Purify, please see below. Could somebody point to me from which function those leaks were generated, and how to avoid those leaks? Thanks, Bob MLK: 1104 bytes leaked in 46 blocks This memory was allocated from: malloc [rtlib.o] CRYPTO_malloc [libcrypto.so.1.0.0] ASN1_STRING_type_new [libcrypto.so.1.0.0] ASN1_primitive_new [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] asn1_item_ex_combine_new [libcrypto.so.1.0.0] ASN1_item_ex_d2i [libcrypto.so.1.0.0] asn1_template_noexp_d2i [libcrypto.so.1.0.0] Block of 24 bytes (46 times); last block at 0x2c0f7218 I've used valgrind with OpenSSL based programs to find memory leaks, and with it, you can use a --num-callers=N option to get deeper stack traces in the leak reports. You need a depth of 15-20 to get far enough to see where your code is calling into the OpenSSL code in most cases. There's probably a similar option for IBM purify. -chris __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
