Re: avoiding child death by size limit
On Saturday 12 December 2009 00:24:24 Perrin Harkins wrote:
Perl will keep the memory and reuse it for that exact same lexical
variable the next time you enter that section of code. It's a
performance optimization that is usually a good thing, unless you put
a lot of data in one lexical in some code that you rarely run.
perhaps an example can help to shed some light. To understand it you have to
know that many malloc implementations allocate large chunks of memory via
mmap() while smaller pieces are allocated via brk(). The mmap-allocated
blocks can later be returned to the OS while the brk-allocated can not.
So, here is the example. It creates a large string (10 mbytes) and assigns it
to the variable $x. This means another 10mb chunk is allocated to keep the
variable. Devel::Peek::Dump is used to show the address of the string assigned
to the variable.
Now look what is when freed:
strace perl -MDevel::Peek -e '{my $x=xx(10*1024*1024); $x=; Dump $x; undef
$x; warn 1;} warn 2'
Here 2 memory blocks are allocated. One is the xx(10*1024*1024) string the
other is the PV-member of $x. It seems to be logical that the second one is
$x but let's see ...
mmap(NULL, 10489856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7ff87f08a000
mmap(NULL, 10489856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7ff87e689000
These write()s come from Dump $x. The PV=0x7ff87e689028 line below is of
particular interest. It shows that the memory block at 0x7ff87e689000
in fact belongs to $x.
write(2, SV = , 5SV = )= 5
write(2, PV(0x78bcb0) at 0x78ef78\n REFCN..., 65PV(0x78bcb0) at 0x78ef78
REFCNT = 1
FLAGS = (PADMY,POK,pPOK)
) = 65
write(2, PV = 0x7ff87e689028 , 22 PV = 0x7ff87e689028 ) = 22
write(2, 0, 4\0) = 4
write(2, \n, 1
) = 1
write(2, CUR = 0\n, 10 CUR = 0
) = 10
write(2, LEN = 10485768\n, 17 LEN = 10485768
) = 17
The next syscall comes from undef $x. Here the second allocated block is
freed and returned to the OS.
munmap(0x7ff87e689000, 10489856)= 0
This write() is the warn 1
write(2, 1 at -e line 1.\n, 161 at -e line 1.
) = 16
And this the warn 2. You see the memory block allocated to keep the
xx(10*1024*1024) string is still kept by the program even though the
scope where it has been allocated is finished.
write(2, 2 at -e line 1.\n, 162 at -e line 1.
) = 16
Only right before finishing the program the xx(10*1024*1024) string
is freed.
munmap(0x7ff87f08a000, 10489856)= 0
exit_group(0) = ?
Now, omit the undef $x and see when the memory allocated to the variable
is freed.
Torsten
Re: avoiding child death by size limit
On Thu, Dec 10, 2009 at 11:28 AM, E R pc88m...@gmail.com wrote: Hi, I have a problem where a mod_perl handler will allocate a lot of memory when processing a request, and this causes Apache to kill the child due to exceeding the configure child size limit. Are there any special techniques people use to avoid this situation? Does SizeLimit count actual memory used or does it just look at the process size? You can make sure the variables that the memory was malloc'd for have gone out of scope, and there are not trailing references to them. Perl will then reuse that memory. Occasionaly Perl will free memory it's not using, but why, where and when can be hard to determine. It's not always as clear cut as undef'ing the variable. In terms of managing memory in your mod_perl process you can use any of the standard techniques: * change/fix the code (fix memory leak, Tie structure to filesystem/db, ...) * recycle your children after N number of request are completed * offload the memory intensive work into a different process space (another instance of Apache/mod_perl or even just Perl) * use a system set resource limit (not so good because it can kill the proc mid-request) * Apache2::SizeLimit (kill the proc after it's done serving the request when it grows too big) These were off the top of my head. THere may be other techniques I missed. -wjt
Re: avoiding child death by size limit
E R wrote:
Hi,
I have a problem where a mod_perl handler will allocate a lot of
memory when processing a request, and this causes Apache to kill the
child due to exceeding the configure child size limit.
Chances are that a child does not exceed this memory right away, at the
first request. More likely, it uses more and more memory at each
request it processes, and finally after a number of requests, it exceeds
the maximum memory and gets killed.
In other words, it is leaking.
So, paraphrasing someone else : don't treat the symptom, treat the cause.
However, the memory allocated will get freed up or re-used by the next
request -
If it is a real leak, then no, it will not.
I think the memory is just fragmented enough to be
automatically reclaimed by the memory allocator (I've heard that some
mallocs can return memory to the OS in 1 MB chunks.)
See William's answer : unlikely.
Are there any special techniques people use to avoid this situation?
Does SizeLimit count actual memory used or does it just look at the
process size?
In a previous similar exercise, in despair I used the module Devel::Leak
as follows :
use Devel::Leak;
my $DEBUGMem = 1;
my ($SVTable,$prevSVCount,$lastSVCount);
if ($DEBUGMem) {
$prevSVCount = Devel::Leak::NoteSV($SVTable);
warn [$$] before something, total SVs : $prevSVCount;
}
do_something(); # .. which could be leaking
if ($DEBUGMem) {
$lastSVCount = Devel::Leak::CheckSV($SVTable);
warn [$$] after something : total SVs : $lastSVCount;
warn [$$] new SVs : ,($lastSVCount - $prevSVCount);
}
It doesn't require any specially-compiled perl.
It does not actually print the memory size used. It just provides a
count of new things that have been allocated and not freed by
do_something(). It is very rough, but it was very helpful to me to find
out what exact piece of code was leaking things, which is basically an
alias for memory.
The point is, if it keeps on growing around the same piece of code each
time you process a request, then you at least know where bad things happen.
If it happens in your code, then when you know where, it should be
possible to fix it. If it happens in someone else's module that you are
using, there are usually several alternative modules for just about
anything on CPAN. If there aren't and you cannot do without, /then/
maybe you should considering limiting the number of requests that each
child handles before it gets killed. But that should not be the first
choice, because it is the least efficient.
Re: avoiding child death by size limit
On Fri, Dec 11, 2009 at 11:37 AM, William T dietbud...@gmail.com wrote: You can make sure the variables that the memory was malloc'd for have gone out of scope, and there are not trailing references to them. Perl will then reuse that memory. It will keep the memory allocated to the out-of-scope variables unless you undef them. There's a summary of many PerlMonks discussions on this topic here: http://www.perlmonks.org/?node_id=803515 - Perrin
Re: avoiding child death by size limit
Perrin Harkins wrote: On Fri, Dec 11, 2009 at 11:37 AM, William T dietbud...@gmail.com wrote: You can make sure the variables that the memory was malloc'd for have gone out of scope, and there are not trailing references to them. Perl will then reuse that memory. It will keep the memory allocated to the out-of-scope variables unless you undef them. There's a summary of many PerlMonks discussions on this topic here: http://www.perlmonks.org/?node_id=803515 Perrin, that is in the end, in my personal opinion, a rather confusing discussion. So, to an extent, is your phrase above. When you say It (perl) will keep the memory, do you mean that - the perl interpreter embedded in this Apache child will keep the memory (and not return it to the OS), but will re-use it if possible for other variable allocations happening within its lifetime ? or - the perl interpreter embedded in this Apache child will keep the memory (and not return it to the OS), and never re-use it again for any other variable allocation happening within its lifetime (in other words, this is a leak) ? Does any guru care to provide some simple real-world examples of when memory once allocated to a variable is/is not being re-used, in a mod_perl handler context ? Or pointers to ditto ? (Maybe at first independently of whether the memory also may, or not, be returned by Perl to the OS) Maybe on this last subject, what I gather from this and other discussions I've seen, is that once the Perl interpreter obtained some memory from the OS, it rarely returns it (before it exits); and if it does, it is in any case not practically predictable in a cross-platform way, so one cannot rely on it. Is that a fair interpretation ? Would it be preferable/easier if I construct some simple examples myself and present them here asking if memory allocated to my $a is being leaked or not ?
Re: avoiding child death by size limit
On Fri, Dec 11, 2009 at 5:56 PM, André Warnier a...@ice-sa.com wrote: When you say It (perl) will keep the memory, do you mean that - the perl interpreter embedded in this Apache child will keep the memory (and not return it to the OS), but will re-use it if possible for other variable allocations happening within its lifetime ? or - the perl interpreter embedded in this Apache child will keep the memory (and not return it to the OS), and never re-use it again for any other variable allocation happening within its lifetime (in other words, this is a leak) ? Option 3: Perl will keep the memory and reuse it for that exact same lexical variable the next time you enter that section of code. It's a performance optimization that is usually a good thing, unless you put a lot of data in one lexical in some code that you rarely run. It's not a leak. Perl tracks the memory and will reuse it, just not for any other variables. Does any guru care to provide some simple real-world examples of when memory once allocated to a variable is/is not being re-used, in a mod_perl handler context ? It's simple to see. Slurp a file into a lexical. Let it go out of scope. There are many posts on this subject in the archives here as well as on PerlMonks and the p5p archives. Maybe on this last subject, what I gather from this and other discussions I've seen, is that once the Perl interpreter obtained some memory from the OS, it rarely returns it (before it exits); and if it does, it is in any case not practically predictable in a cross-platform way, so one cannot rely on it. Is that a fair interpretation ? Yes, you can't expect to get memory back. - Perrin
avoiding child death by size limit
Hi, I have a problem where a mod_perl handler will allocate a lot of memory when processing a request, and this causes Apache to kill the child due to exceeding the configure child size limit. However, the memory allocated will get freed up or re-used by the next request - I think the memory is just fragmented enough to be automatically reclaimed by the memory allocator (I've heard that some mallocs can return memory to the OS in 1 MB chunks.) Are there any special techniques people use to avoid this situation? Does SizeLimit count actual memory used or does it just look at the process size? Thanks, ER
Re: avoiding child death by size limit
On Thu, Dec 10, 2009 at 2:28 PM, E R pc88m...@gmail.com wrote: However, the memory allocated will get freed up or re-used by the next request The memory won't get freed unless you undef all the variables manually. Perl keeps that memory otherwise, even when they go out of scope. If you know it will get reused on the next request, then set your size limit higher. If it's only being used for the current request and the odds of reuse on the following requests is low, the best thing is to kill the process as SizeLimit is doing. - Perrin
Re: avoiding child death by size limit
E R wrote: Hi, I have a problem where a mod_perl handler will allocate a lot of memory when processing a request, and this causes Apache to kill the child due to exceeding the configure child size limit. However, the memory allocated will get freed up or re-used by the next request - I think the memory is just fragmented enough to be automatically reclaimed by the memory allocator (I've heard that some mallocs can return memory to the OS in 1 MB chunks.) Are there any special techniques people use to avoid this situation? Does SizeLimit count actual memory used or does it just look at the process size? This is not a direct answer to your question, and it begs for a more authoritative answer. I have had problems similar to yours, which I solved by turning what was original part of my mod_perl handler (or script), into a separate process. I know it is not elegant, but it seems to work well in my case. One of the problems is that, as far as I know, once perl has obtained some memory from the OS, it will never give it back until perl itself exits. And since by definition with mod_perl normally the perl interpreter lives as long as the Apache process it is inside of, that means almost never. But if perl runs as an external process for a request, then of course it exits at the end of it, and the memory is returned. The exact problem I had, was that some processing I had to do involved parsing XML, and some XML parsing module in the chain (XML::Twig ?) was leaking a chunk of memory at each request. I ended up with multi-megabyte Apache children all the time. So I off-loaded this parsing into a separate process, which wrote (to disk) its results in the form of a Storable structure. When the external process was done, the main mod_perl handler sucked in the data back from the Storable file and deleted it. Not elegant, but it's been working flawlessly for several years now.
