mod_perl memory
Hi, I have a perl script running in mod_perl that needs to write a large amount of data to the client, possibly over a long period. The behavior that I observe is that once I print and flush something, the buffer memory is not reclaimed even though I rflush (I know this cant be reclaimed back by the OS). Is that how mod_perl operates and is there a way that I can force it to periodically free the buffer memory, so that I can use that for new buffers instead of taking more from the OS?
Re: confirm subscribe to modperl@perl.apache.org
Please add me to the mailing list On Tue, Mar 16, 2010 at 12:58 PM, modperl-h...@perl.apache.org wrote: Hi! This is the ezmlm program. I'm managing the modperl@perl.apache.org mailing list. To confirm that you would like matty.r...@gmail.com added to the modperl mailing list, please send a short reply to this address: modperl-sc.1268724497.hmamdfldddmbopdailgb-matty.rnld=gmail@perl.apache.org Usually, this happens when you just hit the reply button. If this does not work, simply copy the address and paste it into the To: field of a new message. This confirmation serves two purposes. First, it verifies that I am able to get mail through to you. Second, it protects you in case someone forges a subscription request in your name. Some mail programs are broken and cannot handle long addresses. If you cannot reply to this request, instead send a message to modperl-requ...@perl.apache.org and put the entire address listed above into the Subject: line. --- Administrative commands for the modperl list --- I can handle administrative requests automatically. Please do not send them to the list address! Instead, send your message to the correct command address: To subscribe to the list, send a message to: modperl-subscr...@perl.apache.org To remove your address from the list, send a message to: modperl-unsubscr...@perl.apache.org Send mail to the following for info and FAQ for this list: modperl-i...@perl.apache.org modperl-...@perl.apache.org Similar addresses exist for the digest list: modperl-digest-subscr...@perl.apache.org modperl-digest-unsubscr...@perl.apache.org To get messages 123 through 145 (a maximum of 100 per request), mail: modperl-get.123_...@perl.apache.org To get an index with subject and author for messages 123-456 , mail: modperl-index.123_...@perl.apache.org They are always returned as sets of 100, max 2000 per request, so you'll actually get 100-499. To receive all messages with the same subject as message 12345, send a short message to: modperl-thread.12...@perl.apache.org The messages should contain one line or word of text to avoid being treated as s...@m, but I will ignore their content. Only the ADDRESS you send to is important. You can start a subscription for an alternate address, for example j...@host.domain, just add a hyphen and your address (with '=' instead of '@') after the command word: modperl-subscribe-john=host.dom...@perl.apache.org To stop subscription for this address, mail: modperl-unsubscribe-john=host.dom...@perl.apache.org In both cases, I'll send a confirmation message to that address. When you receive it, simply reply to it to complete your subscription. If despite following these instructions, you do not get the desired results, please contact my owner at modperl-ow...@perl.apache.org. Please be patient, my owner is a lot slower than I am ;-) --- Enclosed is a copy of the request I received. Return-Path: matty.r...@gmail.com Received: (qmail 51100 invoked by uid 99); 16 Mar 2010 07:28:17 - Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Mar 2010 07:28:17 + X-ASF-Spam-Status: No, hits=-2.0 required=10.0 tests=ASF_LIST_OPS,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of matty.r...@gmail.com designates 209.85.216.171 as permitted sender) Received: from [209.85.216.171] (HELO mail-px0-f171.google.com) (209.85.216.171) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Mar 2010 07:28:10 + Received: by pxi1 with SMTP id 1so2642977pxi.10 for modperl-subscr...@perl.apache.org; Tue, 16 Mar 2010 00:27:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=; b=mNkT13R474B+zvUP+jaEki2A0nr/rGgY2wdE16LXyr+e25chPkJlNN0Aj/iq5aDIuc tHs2v7P344nmDXShAHt9LMmqY9NsxN8MlT2MjJXISOq12KZ1JZPqJK5rw4ojz6/jBZ1u uX1OZjuQx+Z3LjFupPh8xmmSIhCyPiYH0kXbM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=gE9t/Ct7gWjBHSaLePtCk1JPFNNC02j1KLzHM1+7LCDMDTWFt/7k5W/DHwttXS5vHA pE+EPyrI9NwqrqHmZY9gWp2rgZykRFVcmXcYzex9RJbCPwRmFIfbvvgKptlzvcn5YkB/ pGgNKX/keR2mukyalvub6mzXiPjoT5qj91nSw= MIME-Version: 1.0 Received: by 10.115.36.31 with SMTP id o31mr5960815waj.79.1268724470376; Tue, 16 Mar 2010 00:27:50 -0700 (PDT) Date: Tue, 16 Mar 2010 12:57:50 +0530 Message-ID: 22782ece1003160027s258c0daat3578c75d04214...@mail.gmail.com Subject: subscribe to the list From: Matty Ronald matty.r...@gmail.com To: modperl-subscr...@perl.apache.org
Help with mod_perl
Hi, Can any one please tell me how to disable perl's malloc in mod_perl? I donot want to recompile perl. Is there any way that i can disable perl's malloc in mod_perl only? Regards, Matty
Re: mod_perl memory
On Mon, Mar 15, 2010 at 11:26 PM, Pavel Georgiev pa...@3tera.com wrote: I have a perl script running in mod_perl that needs to write a large amount of data to the client, possibly over a long period. The behavior that I observe is that once I print and flush something, the buffer memory is not reclaimed even though I rflush (I know this cant be reclaimed back by the OS). Is that how mod_perl operates and is there a way that I can force it to periodically free the buffer memory, so that I can use that for new buffers instead of taking more from the OS? That is how Perl operates. Mod_Perl is just Perl embedded in the Apache Process. You have a few options: * Buy more memory. :) * Delegate resource intensive work to a different process (I would NOT suggest a forking a child in Apache). * Tie the buffer to a file on disk, or db object, that can be explicitly reclaimed * Create a buffer object of a fixed size and loop. * Use compression on the data stream that you read into a buffer. You could also architect your system to mitigate resource usage if the large data serve is not a common operation: * Proxy those requests to a different server which is optimized to handle large data serves. * Execute the large data serves with CGI rather than Mod_Perl. I'm sure there are probably other options as well. -wjt
Re: Help with mod_perl
On Tue, Mar 16, 2010 at 3:33 AM, Matty Ronald matty.r...@gmail.com wrote: Can any one please tell me how to disable perl's malloc in mod_perl? I donot want to recompile perl. Is there any way that i can disable perl's malloc in mod_perl only? No, sorry. Since mod_perl is just embedding your perl in apache, you have to change your perl to accomplish this. You do have the option of compiling a separate perl just for mod_perl to use and keeping your system perl untouched. If you want to do that, look at the perl compile options for setting an install location and then use your newly built perl to run the mod_perl configuration scripts. - Perrin
Re: mod_perl memory
You could use Apache2::SizeLimit (because size does matter) which evaluates the size of Apache httpd processes when they complete HTTP Requests, and kills those that grow too large. (Note that Apache2::SizeLimit can only be used for non-threaded MPMs, such as prefork.) Since it operates at the end of a Request, SizeLimit has the advantage that it doesn't interrupt Request processing and the disadvantage that it won't prevent a process from becoming oversized while processing a Request. To reduce the regular load of Apache2::SizeLimit it can be configured to check the size intermittently by setting the parameter CHECK_EVERY_N_REQUESTS. These parameters can be configured in a Perl section in httpd.conf, or a Perl start-up file. That way, if your script allocates too much memory the process will be killed when it finishes handling the request. The MPM will eventually start another process if necessary. BR A On Mar 16, 2010, at 9:30 AM, William T wrote: On Mon, Mar 15, 2010 at 11:26 PM, Pavel Georgiev pa...@3tera.com wrote: I have a perl script running in mod_perl that needs to write a large amount of data to the client, possibly over a long period. The behavior that I observe is that once I print and flush something, the buffer memory is not reclaimed even though I rflush (I know this cant be reclaimed back by the OS). Is that how mod_perl operates and is there a way that I can force it to periodically free the buffer memory, so that I can use that for new buffers instead of taking more from the OS? That is how Perl operates. Mod_Perl is just Perl embedded in the Apache Process. You have a few options: * Buy more memory. :) * Delegate resource intensive work to a different process (I would NOT suggest a forking a child in Apache). * Tie the buffer to a file on disk, or db object, that can be explicitly reclaimed * Create a buffer object of a fixed size and loop. * Use compression on the data stream that you read into a buffer. You could also architect your system to mitigate resource usage if the large data serve is not a common operation: * Proxy those requests to a different server which is optimized to handle large data serves. * Execute the large data serves with CGI rather than Mod_Perl. I'm sure there are probably other options as well. -wjt Arthur P. Goldberg, PhD Research Scientist in Bioinformatics Plant Systems Biology Laboratory www.virtualplant.org Visiting Academic Computer Science Department Courant Institute of Mathematical Sciences www.cs.nyu.edu/artg a...@cs.nyu.edu New York University 212 995-4918 Coruzzi Lab 8th Floor Silver Building 1009 Silver Center 100 Washington Sq East New York NY 10003-6688
Re: mod_perl memory
Pavel You're welcome. You are correct about the limitations of Apache2::SizeLimit. Processes cannot be 'scrubbed'; rather they should be killed and restarted. Rapid memory growth should be prevented by prohibiting processes from ever growing large than a preset limit. On Unix systems, the system call setrlimit sets process resource limits. These limits are inherited by children of the process. These limits can view and set with the bash command rlimit. Many resources can be limited, but I'm focusing on process size, which is controlled by resource RLIMIT_AS, the maximum size of a process's virtual memory (address space) in bytes. (Some operating systems control RLIMIT_DATA, The maximum size of the process's data segment, but Linux doesn't.) When a process tries to exceeds a resource limit, the system call that requested the resource fails and returns an error. The type of error depends on which resource's limit is violated (see man page for setrlimit). In the case of virtual memory, the RLIMIT_AS can be exceeded by any call that asks for additional virtual memory, such as brk(2), which sets the end of the data segment. Perl manages memory via either the system's malloc or its own malloc. If asking for virtual memory fails, then malloc will fail, which will typically cause the Perl process to write Out of Memory! to STDERR and die. RLIMIT_AS can be set in many ways. One direct way an Apache/mod_perl process can set it is via Apache2::Resource. For example, these commands can be added to httpd.conf: PerlModule Apache2::Resource # set child memory limit to 100 megabytes # RLIMIT_AS (address space) will work to limit the size of a process PerlSetEnv PERL_RLIMIT_AS 100 PerlChildInitHandler Apache2::Resource The PerlSetEnv line sets the Perl environment variable PERL_RLIMIT_AS. The PerlChildInitHandler line directs Apache to load Apache2::Resource each time it creates an httpd process. Apache2::Resource then reads PERL_RLIMIT_AS and sets the RLIMIT_AS limit to 100 (megabytes). Any httpd that tries to grow bigger than 100 MB will fail. (Also, PERL_RLIMIT_AS can be set to soft_limit:hard_limit, where soft_limit is the limit at which the resource request will fail. At any time the soft_limit can be adjusted up to the hard_limit.) I recommend against setting this limit for a threaded process, because if one Request handler gets the process killed then all threads handling requests will fail. When the process has failed it is difficult to output an error message to the web user, because Perl calls die and the process exits. As I wrote yesterday, failure of a mod_perl process with Out of Memory!, as occurs when the softlimit of RLIMIT_AS is exceeded, does not trigger an Apache ErrorDocument 500. A mod_perl process that exits (actually CORE::exit() must be called) doesn't trigger an ErrorDocument 500 either. Second, if Apache detects a server error it can redirect to a script as discussed in Custom Error Response. It can access the REDIRECT environment variables but doesn't know anything else about the HTTP Request. At this point I think that the best thing to do is use MaxRequestsPerChild and Apache2::SizeLimit to handle most memory problems, and simply let processes that blow up die without feedback to users. Not ideal, but they should be extremely rare events. BR A On Mar 16, 2010, at 2:31 PM, Pavel Georgiev wrote: Thank you both for the quick replies! Arthur, Apache2::SizeLimit is no solution for my problem as I'm looking for a way to limit the size each requests take, the fact that I can scrub the process after the request is done (or drop the requests if the process reaches some limit, although my understanding is that Apache2::SizeLimit does its job after the requests is done) does not help me. William, Let me make I'm understanding this right - I'm not using any buffers myself, all I do is sysread() from a unix socked and print(), its just that I need to print a large amount of data for each request. Are you saying that there is no way to free the memory after I've done print() and rflush()? BTW thanks for the other suggestions, switching to cgi seems like the only reasonable thing for me, I just want to make sure that this is how mod_perl operates and it is not me who is doing something wrong. Thanks, Pavel On Mar 16, 2010, at 11:18 AM, ARTHUR GOLDBERG wrote: You could use Apache2::SizeLimit (because size does matter) which evaluates the size of Apache httpd processes when they complete HTTP Requests, and kills those that grow too large. (Note that Apache2::SizeLimit can only be used for non-threaded MPMs, such as prefork.) Since it operates at the end of a Request, SizeLimit has the advantage that it doesn't interrupt Request processing and the disadvantage that it won't prevent a process from becoming oversized while processing a Request. To reduce the regular
Re: mod_perl memory
Pavel Georgiev wrote: ... Let me make I'm understanding this right - I'm not using any buffers myself, all I do is sysread() from a unix socked and print(), its just that I need to print a large amount of data for each request. ... Taking the issue at the source : can you not arrange to sysread() and/or print() in smaller chunks ? There exists something in HTTP named chunked response encoding (forgive me for not remembering the precise technical name). It consists of sending the response to the browser without an overall Content-Length response header, but indicating that the response is chunked. Then each chunk is sent with its own length, and the sequence ends with (if I remember correctly) a last chunk of size zero. The browser receives each chunk in turn, and re-assembles them. I have never had the problem myself, so I never looked deeply into it. But it just seems to me that before going off in more complicated solutions, it might be worth investigating.
Re: mod_perl memory
Andre,
That is what I'm currently doing:
$request-content_type(multipart/x-mixed-replace;boundary=\$this-{boundary}\;);
and then each chuck of prints looks like this (no length specified):
for (some condition) {
$request-print(--$this-{boundary}\n);
$request-print(Content-type: text/html; charset=utf-8;\n\n);
$request-print($data\n\n);
$request-rflush;
}
And the result is endless memory growth in the apache process. Is that what you
had in mind?
On Mar 16, 2010, at 12:50 PM, André Warnier wrote:
Pavel Georgiev wrote:
...
Let me make I'm understanding this right - I'm not using any buffers myself,
all I do is sysread() from a unix socked and print(),
its just that I need to print a large amount of data for each request.
...
Taking the issue at the source : can you not arrange to sysread() and/or
print() in smaller chunks ?
There exists something in HTTP named chunked response encoding
(forgive me for not remembering the precise technical name). It
consists of sending the response to the browser without an overall
Content-Length response header, but indicating that the response is
chunked. Then each chunk is sent with its own length, and the
sequence ends with (if I remember correctly) a last chunk of size zero.
The browser receives each chunk in turn, and re-assembles them.
I have never had the problem myself, so I never looked deeply into it.
But it just seems to me that before going off in more complicated
solutions, it might be worth investigating.
Re: mod_perl memory
Pavel Georgiev wrote:
Andre,
That is what I'm currently doing:
$request-content_type(multipart/x-mixed-replace;boundary=\$this-{boundary}\;);
I don't think so. What you show above is a multipart message body,
which is not the same (and not the same level).
What you are looking for is a header like
Transfer-encoding: chunked
And this chunked transfer encoding happens last in the chain. It is just
a way of sending the data to the browser in smaller pieces, like one
would do for example to send a whole multi-megabyte video file.
This link may help :
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6
and this
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.41
Also search Google for mod_perl +chunked
The first item in the list contains a phrase :
mod_perl can transparently generate chunked encoding on recent versions
of Apache
I personally don't know how, but it sounds intriguing enough to look
further into it.
Maybe one of the gurus on this list knows more about it, and can give a
better opinion than mine as to whether this might help you.
Re: mod_perl memory
André Warnier wrote:
Pavel Georgiev wrote:
Andre,
That is what I'm currently doing:
$request-content_type(multipart/x-mixed-replace;boundary=\$this-{boundary}\;);
I don't think so. What you show above is a multipart message body,
which is not the same (and not the same level).
What you are looking for is a header like
Transfer-encoding: chunked
And this chunked transfer encoding happens last in the chain. It is just
a way of sending the data to the browser in smaller pieces, like one
would do for example to send a whole multi-megabyte video file.
This link may help :
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6
and this
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.41
Also search Google for mod_perl +chunked
The first item in the list contains a phrase :
mod_perl can transparently generate chunked encoding on recent versions
of Apache
I personally don't know how, but it sounds intriguing enough to look
further into it.
Maybe one of the gurus on this list knows more about it, and can give a
better opinion than mine as to whether this might help you.
In one of the following hits in Google, I found this demo CGI script :
#!/usr/bin/perl -w
use CGI;
my $cgi = new CGI();
print $cgi-header(-type = 'application/octet-stream',
-content_disposition = 'attachment; filename=data.raw', -connection =
'close', -transfer_encoding = 'chunked');
open (my $fh, '', '/dev/urandom') || die $!\n;
for (1..100) {
my $data;
read($fh, $data, 1024*1024);
print $data;
}
That should be easily transposable to mod_perl.
Of course above they read and send in chunks of 1 MB, which may not be
the small buffers you are looking for..
;-)
Can i use mod_perl to filter/auth the request before forwarding the request by mod_proxy?
Hi, I want to use mod_proxy_http(reverse proxy) to forward the permissive requests. can i use mod_perl to check the requests before forward them by mod_proxy_http? Here below are my work environment. Apache 2.2 + Mod_perl2 + mod_proxy + mod_proxy_http Any advice will be appreciated. Thanks. -jl
