Re: suggestions for dynamic run-time configuration?
2011/11/1 Torsten Förtsch : > On Monday, 31 October 2011 22:28:32 E R wrote: > ... >> Any suggestions / recommendations or alternatives for a mod-perl >> environment? > > Have a look at MMapDB on CPAN. ... That's a very interesting module - thanks for the pointer!
suggestions for dynamic run-time configuration?
I was wondering if anyone had suggestions / recommendations for a run-time application configuration mechanism for mod-perl apps. What I am thinking of is simply a way to communicate new application settings to all of the mod-perl children. The only solution I can think of is to have all of the mod-perl child processes periodically check a file to see if it has changed, and if so have the child process reload their application configuration. By "application configuration" I mean configuration data at the application level and above - not necessarily httpd-level configuration data (i.e. such as found in httpd.conf.) Typical application configuration data includes database credentials, flags to enable/disable parts of the application, urls for web services used by the application, etc. The idea is to have a capability something like Facebook's Gatekeeper as described in one of their tech talks ( http://techcrunch.com/2011/05/30/facebook-source-code/ ) To reduce the impact this checking would have on serving requests I was thinking of checking only every N requests (i.e. N = 10) and/or performing the check as part of the cleanup handler. Any suggestions / recommendations or alternatives for a mod-perl environment?
cleanup handler - is it "free" time?
I am wondering how time spent in a cleanup handler affects page response time. For instance, if I do a sleep(1) in a cleanup handler, will my page delivery times be increased by a second? Or will other available mod-perl child processes pick up pending requests until the cleanup handler returns?
modperl and attributes
Hi all,
I've been playing around with modperl and attributes. Below is the
module I am experimenting with.
When I start up the server I see the warnings coming from the
MODIFY_CODE_ATTRIBUTES routine.
However, when I fetch the url associated with the handler, it reports
that ATTRS = {}.
Moreover, the code addresses for sub1, sub2, and sub3 do not match
what is printed in the warning messages.
Example warnings messages from MODIFY_CODE_ATTRIBUTES:
setting ATTRS for CODE(0x90e44ec) to A1
setting ATTRS for CODE(0x90e44cc) to A2
setting ATTRS for CODE(0x90e442c) to A3
Example output from the handler:
ATTRS = $VAR1 = {};
ADDR sub1 = CODE(0x90da764)
ADDR sub2 = CODE(0x90da744)
ADDR sub3 = CODE(0x90da6a4)
I'm really curious to know what's going on!
Thanks,
ER
package Foo4;
use Apache2::Const qw(OK);
use Data::Dumper;
my $ATTRS = {};
sub MODIFY_CODE_ATTRIBUTES {
my ($package, $sub, $attr) = @_;
warn "setting ATTRS for $sub to $attr\n";
$ATTRS->{$sub} = $attr;
return;
}
sub handler {
my $r = shift;
$r->content_type('text/plain');
print "ATTRS = ", Dumper($ATTRS), "\n";
for my $x (qw(sub1 sub2 sub3)) {
print "ADDR $x = ", \&$x, "\n";
}
return OK;
}
sub sub1 : A1 { }
sub sub2 : A2 { }
sub sub3 : A3 { }
Re: proper use of $r->read?
Hi Torsten, So is this what you are suggesting... Define $MAX_SIZE to be the largest length of POSTed data you will accept. For modperl <= 2.0.4 just use a single read and hope for the best: $r->read($buf, $MAX_SIZE); For modperl > 2.0.4 ( >= 2.0.5?) using this while-loop works: my $buf = ""; 1 while (length($buf) < $MAX_SIZE) && $r->read($buf, $MAX_SIZE, length($buf)); Thanks, ER
proper use of $r->read?
Hi,
What's the proper way to read in the posted content from a request?
Using this Google code search:
http://www.google.com/codesearch?q=lang%3Aperl+%5C%24r-%3Eread.*length&hl=en
I see instances of:
$r->read($line, $r->headers_in->get('Content-length'));
and also loops like:
while ( $r->read( $buffer, $content_length ) ) {
$content .= $buffer;
}
What's the best/proper way to slurp in all of the POST content?
Thanks,
ER
Re: Apache2::SizeLimit and x64_86
On Mon, Apr 11, 2011 at 3:43 AM, Dave Hodgkinson wrote: > Are you using the A::S that's in the repo? It has fixes for calculating > sizes... Using RSS-SHARED instead of VSIZE-SHARED should fix the x86_64 issue I'm seeing. But I've got some questions about Apache2::SizeLimit... 1. Are the docs accurate? The example in the POD says you can do this: require 'Apache2/SizeLimit.pm'; ... my ($size, $shared) = $Apache2::SizeLimit->_check_size(); But when I try this I get "Can't call method "_check_size" on an undefined value". Note - I've been using: my ($size, $shared) = $Apache2::SizeLimit::HOW_BIG_IS_IT->(); to get the current program size. I don't even remember how I came to use this. 2. I extracted version Apache::SizeLimit 0.96 out of the latest mod-perl snapshot (modperl-2.0_20110411151604.tar.gz) and used it with mod-perl 2.0.4. From the change log I was hoping that it would compute the program size as RSS - SHARED for linux systems, but it didn't seem to work. Do I need other stuff from the mod-perl dist for the RSS-SHARED patch? Do I need to upgrade to mod-perl 2.0.5 (or later) for this fix? Thanks, ER
Apache2::SizeLimit and x64_86
Hi all, On x86_64 I noticed that perl .so files (XS modules) appear to use a lot more memory than they do on 32-bit architectures. In particular, /proc/$$/smaps reports a ~ 2MB segment associated with the .so library that is not reported in the 32-bit case. For more details, see my posting to the Centos mailing list: http://lists.centos.org/pipermail/centos/2011-April/109530.html My questions are: - Does this represent real (physical) memory usage? - Apache2::SizeLimit seems to be including these segments in its size calculation. Is there a way to have Apache2::SizeLimit ignore them? Thanks, ER
Re: How to reference %Location outside of ...?
On Mon, Feb 28, 2011 at 4:41 PM, Jeff Nokes wrote:
> I've seen it done where you can make a home-grown module that has a second
> package declaration for Apache[2]::ReadConfig. Something like ...
Thanks, I've found that this works:
use Some::Module;
Some::Module::init(__PACKAGE__);
package Some::Module;
sub init {
my $package = shift;
*Redirect = *{ $package . "::Redirect" };
push(@Redirect, [ '/to-cnn', 'http://cnn.com' ]);
...
}
Modperl2 uses a different package name for each block.
How to reference %Location outside of ...?
I have the following block in my httpd.conf file which installs
mod-perl handlers for a list of urls:
...
for my $uri (...) {
$Location{$uri} = ...;
}
If I wanted to move this code to a normal perl module, how should I
reference %Location?
Thanks,
ER
use Apache::Constants / mod_perl2 / command line
Here's the problem I'm trying to solve: I have a lot of mod_perl 1.x source code that uses Apache::Constants. We have just moved to mod_perl2, and I'd like to be able to compile-check the code on the command line. In our httpd.conf we are using Apache2::compat so we don't have to change all the constant references to Apache2::Const. However, to compile check the code I can't use Apache2::compat since that seems to require that you are running in a mod_perl environment. Are there any other quick-n-dirty solutions to this problem that I've overlooked? I'm about to create MyOwnApache::Constants package which emulates Apache::Constants using Apache2::Const. It seems that Apache2::Const is usable outside of the mod_perl environment. Running perl -MApache2::compat results in the error: Undefined subroutine &Apache2::ServerUtil::restart_count called at /Apache2/compat.pm line 76.
Log4perl with and without mod_perl
I was wondering if anyone has come up with a solution to this problem: 1. You have a mod_perl application and use Log4perl for logging. 2. There is a module in your mod_perl app that you want to use outside of the mod_perl environment (e.g. in a script run from the command line) 3. That module uses Log4perl for logging, so you need to initialize the logging environment. 4. However, there are some aspects of the Log4perl configuration that are not relevant or you don't want to use when using the module outside of the mod_perl environment. One example for #4 would be the filename property for the appender Log::Log4perl::Appender::File. In your mod_perl app you make want a particular path configured, but when running a script from the command line you may want all logging output to go to stdout/stderr. On the other hand, there are some aspects of the Log4perl configuration which would be common to both scenarios - like the layout. This is really more of a Log4perl question than a mod_perl one, but I figure mod_perl users might have a better chance of running into this problem. Any good solutions out there? Thanks, ER
IPC3::Open3 and mod_perl
I've just run into this bug: http://rt.perl.org/rt3/Public/Bug/Display.html?id=66224 Apparently it's been a problem for a while - first being reported in 2001: http://marc.info/?l=apache-modperl&m=98839218229712&w=2 What would it take to resolve this - either by changing IPC::Open3 or mod_perl or by some other remedy? It would be really nice to have things 'just work'. Thanks, ER
mod_perl vs. mod_rewrite precedence?
I'm trying to figure out how mod_perl and mod_rewrite work together.
This is with Apache 1.3.41, mod_perl 1.30, perl 5.8,9.
It seems that if there is a mod_perl handler installed at a location,
a mod_rewrite rule rewriting that location to another doesn't have any
effect.
Below is a sample httpd.conf file. It installs mod perl handlers at
/perl1 and /perl2. The mod_rewrite rules rewrite /perl2 -> /foo.txt (a
file which exists), /bar -> /foo.txt and /baz -> /perl1.
Here is what I am seeing:
GET /perl2 -> returns the output of the mod-perl handler
GET /bar -> returns the contents of the file foo.txt (as expected)
GET /baz -> returns a 404 error; inspecting the rewrite log file shows
that it is trying to serve the file DOCUMENT_ROOT/perl1
Can anyone explain what is going on?
Thanks,
ER
RewriteEngine On
RewriteLog /tmp/rewrite
RewriteCond %{REQUEST_URI} ^/perl2 [NC]
RewriteRule .* /foo.txt [L]
RewriteCond %{REQUEST_URI} ^/bar [NC]
RewriteRule .* /foo.txt [L]
RewriteCond %{REQUEST_URI} ^/baz [NC]
RewriteRule .* /perl1 [L]
RewriteLogLevel 9
package MyHandler;
use Apache::Constants qw(OK);
sub handler {
my $r = shift;
$r->content_type("text/plain");
$r->print("Now is @{[scalar(localtime(time))]}\n");
return OK;
}
package ApacheReadConfig;
my $app = {
SetHandler => 'perl-script',
PerlHandler => 'MyHandler',
};
$Location{'/perl1'} = $app;
$Location{'/perl2'} = $app;
building mod_perl-1.30 - mod_perl.h not getting installed?
Hello, I'm building ancient versions of apache and mod_perl (with perl 5.8.9) in the following way: tar zxf apache_1.3.41.tar.gz cd apache_1.3.41 ./configure --prefix=/APACHE-PREFIX --enable-module=so --enable-module=access (... more --enable-module) \ --disable-module=imap --disable-module=userdir --disable-rule=EXPAT make make install cd .. tar zxf mod_perl-1.30.tar.gz perl Makefile.PL EVERYTHING=1 PREFIX=/PERL-PREFIX \ USE_APXS=1 \ WITH_APXS=/APACHE-PREFIX/bin/apxs APACHE_SRC=../apache_1.3.41/src make make install The problem is that include files like mod_perl.h are not getting installed in the perl tree. When I try to build libapreq-1.34 I'll get compilation errors like: Request.xs:41:22: mod_perl.h: No such file or directory What do I need to do to get the mod_perl include files installed? Thanks, ER
modperl and comet?
I'm interested in adding a Comet capability to a (rather large) mod-perl application. To get around XSS limitations, the Comet service will have the same host and port as the web service. However, I don't want a large mod-perl process tied up performing the Comet service. Is there a way that a web request can be passed off to another process without tying up the Apache process? For instance, can the Apache process pass the file descriptors for the HTTP connection to another process which would free itself up for handling the next Apache request? Is there another way to implement this? Thanks, ER
Re: protecting internal redirects
Thanks for all of the suggestions. Looking for REDIRECT_* environment variables seems like it will work for me. 2010/3/19 Torsten Förtsch : > On Thursday 18 March 2010 21:59:26 E R wrote: >> To serve the file the CGI script issues an internal redirect to a url >> which points to the cached results. >> >> My question is: can the url which points to the cached results be >> protected so that it cannot be directly accessed by external clients? >> > When it creates the new redirected request (can I say "redirectee"?) apache > copies the environment variables of the original request to the new one. All > variable names are prefixed with "REDIRECT_". mod_rewrite should be able to > check the presence of one of them. > > Torsten Förtsch > > -- > Need professional modperl support? Hire me! (http://foertsch.name) > > Like fantasy? http://kabatinte.net >
protecting internal redirects
Hi all, This is not exactly a mod_perl question - in fact I hope there is a solution which does not use mod_perl. I have a CGI script which generates a lot of output. Because it takes a lot of time to the output, the results are cached in case the same request is made again. To serve the file the CGI script issues an internal redirect to a url which points to the cached results. My question is: can the url which points to the cached results be protected so that it cannot be directly accessed by external clients? For example: 1. user makes a request 2. CGI script handles request. It computes a file name for the results, generates the results and places the result in that file. 3. The CGI script then emits an internal redirect to a url which will map to the file name determined in step 2. 4. Apache will process the internal redirect and serve the contents of the file to the client. I want to prevent the clients from accessing the file directly by figuring out what the url is in step 3. I know that clients will not see the internal redirect, but I also want to prevent them from guessing it. Is there an Apache configuration I can use to accomplish this, or do I need to use mod_perl? Thanks, ER
bytes malloc-ed less bytes free-ed
as kind of a followup to my last question... is there a way to determine this quantity: the amount of memory malloc-ed minus the amount of memory free-ed? It seems that this would be easy for malloc()/free() to keep track of. I would like to compare that value with the process size to get an idea of how much memory my program needs vs. how much it is actually taking up. Thanks, ER
avoiding child death by size limit
Hi, I have a problem where a mod_perl handler will allocate a lot of memory when processing a request, and this causes Apache to kill the child due to exceeding the configure child size limit. However, the memory allocated will get freed up or re-used by the next request - I think the memory is just fragmented enough to be automatically reclaimed by the memory allocator (I've heard that some mallocs can return memory to the OS in 1 MB chunks.) Are there any special techniques people use to avoid this situation? Does SizeLimit count actual memory used or does it just look at the process size? Thanks, ER
Re: code in sections executed twice in same process?
The sections don't seem to be executed in the children. Is there a way to determine which pass you are in? For large mod_perl apps, is there a way to avoid loading your code in twice? Note I am using mod_perl 1.3.41. Thanks, ER On Mon, Nov 16, 2009 at 4:41 PM, André Warnier wrote: > E R wrote: > ... >> >> Is this "normal", and what can I do so that the code in >> sections is only executed once in the parent process? >> > I believe it is "normal", in the sense that Apache actually parses its > configuration at least twice : one time "just for checking", then it throws > everything away and parses it a second time "for real". > > Then it will even (probably) run your section again, each time it starts a > new child process. > > You probably really want to read the following 2 pages : > http://perl.apache.org/docs/2.0/user/handlers/server.html > http://perl.apache.org/docs/2.0/user/config/custom.html > >
code in sections executed twice in same process?
Hi -
I am using mod-perl 1.3.41 with perl 5.8.9.
The only perl I have in my config file is:
warn "This is a section in process $$ at time @{[time]}\n";
sleep(5);
warn "Done sleeping\n";
When I start it up in single process mode (-X), I'll something like
this on stderr (to the console):
This is a section in process 25597 at time 1258408992
Done sleeping
And also in the error log I'll see:
This is a section in process 25597 at time 1258408998
Done sleeping
The pids will be the same, but the times will be different (about 5
secs. apart.)
Is this "normal", and what can I do so that the code in
sections is only executed once in the parent process?
Thanks,
ER
finding memory leaks
Hi,
What's the best way to find memory leaks in a mod_perl app? I am
trying to use Apache::Leak with mod_perl 1.41
Some notes:
1. My mod_perl app is *HUGE*. Apache::Leak reports over 500,000 SVs. I
think the time it takes to traverse the symbol tables is long enough
to time out my browser (I'm seeing the output from what looks like a
second request or an error handler.)
2. Should I compile perl with any special options, like DEBUG?
Apache::Leak reports I am leaking ~50 SVs, and I see a lot of "old"
messages, but not a lot of other useful information to track down the
leakage.
3. I'm putting the leak detection code right around my handler
function (which I've renamed to 'real_handler')
sub handler {
my @args = @_;
my $response;
leak_test {
$response = real_handler(@args);
};
$response;
}
sub real_handler { ... } # original handler routine
Any issues with that?
Thanks,
ER
Re: is there a TELL method for mod_perl's STDOUT and STDERR
it seems to work just fine. For now I've just replaced all references to tell(STD(ERR|OUT)) with 0. It's only used for formatting anyway. I've just started to use it, though. Can you think of any issues with using source filters in mod_perl? On Wed, Jul 15, 2009 at 2:28 PM, Perrin Harkins wrote: > On Wed, Jul 15, 2009 at 3:23 PM, E R wrote: >> I'm trying to use Smart::Comments with mod_perl. The problem is that >> STDOUT and STDERR are blessed into the Apache class and they do not >> have a TELL method. > > I would have thought that the problem would be that it uses a source > filter. I think you're going to be out of luck getting that one to > work. > > - Perrin >
is there a TELL method for mod_perl's STDOUT and STDERR
I'm trying to use Smart::Comments with mod_perl. The problem is that STDOUT and STDERR are blessed into the Apache class and they do not have a TELL method. The Smart::Comments module has several references to tell(STDOUT) and tell(STDERR). Is there a way to implement a TELL method for these file handles?
graceful restarts, modperl and pre-loaded modules
Hi, I was wondering how graceful restarts of apache interact with modperl and "pre-loaded" modules. Specifically, if the apache parent process loads module Foo.pm, and I modify Foo.pm, can I use a graceful restart to get the apache parent to read the new version of Foo.pm? Thanks, ER
Re: how did I get here?
On Wed, Apr 8, 2009 at 3:23 PM, Andy Armstrong wrote: > > On Solaris and Mac OS you could actually use a dtrace-enabled Perl. You only > get sub entry / exit but you can infer a call graph from that. Thanks for the pointer to dtrace perl on Mac OS - now I've got a good reason to upgrade to Leopard.
how did I get here?
When diving into a large code base and trying to understand how it works, one thing that would be very helpful to know is how control gets to a particular point in the code. That is, for a given subroutine, what does the stack look like when the subroutine is executed? This is similar to what dtrace on Solaris can do for system calls, but I am only looking to instrument perl subroutines (or points in perl code.) Also, like dtrace, it would be nice to do this on a running perl/mod_perl process. It's kinda like being able to attach the perl debugger to a running process. I realize I'm probably just dreaming right now, but I figured I'd at least bring up the idea with other perl users.
Re: sending all of stderr to a process
Hi Torsten, On Sun, Apr 5, 2009 at 4:13 AM, Torsten Foertsch wrote: > > And later on it does not check the return code when writing the log: > > httpd-2.2.11/server/log.c:682: > apr_file_puts(errstr, logf); > apr_file_flush(logf); My concern is about what happens in this scenario: 1. Set everything up (named pipe, logger process, etc.) 2. Now suppose the logger process dies (it crashes or I need to restart it) 3. Any writes through existing connections to the named pipe will just fail, and I'll lose those writes. I'm pretty sure about #3, but I might be wrong. Perhaps I can't do what I want to do at the application level and I need to go to the kernel level - like write my own device driver or something, especially since I want to capture all output sent through stderr (ie. from Apache and from mod_perl and from any XS modules.)
Re: sending all of stderr to a process
One operational issue with this is that if the logger process dies, existing httpd processes will get a SIGPIPE signal on subsequent writes. Restarting the logger process does not restore the lost connections. On Fri, Apr 3, 2009 at 12:38 PM, Torsten Foertsch wrote: > On Fri 03 Apr 2009, E R wrote: >> I want to send all error output to a process for specialized logging. >> I know that you can use CustomLog with a pipe. However, will that >> capture output from things like 'print STDERR ...' or if an XS module >> directly writes to stderr? I need that output to also go to my >> logging process. >> >> Does anyone know if there an Apache module which does this? > > Give a named pipe's name to ErrorLog outside any virtual host. Then have > your process read from that pipe. > > At shell level: > > mkfifo /path/to/PIPE > your_program > in httpd.conf: > > ErrorLog /path/to/PIPE > > I'd very much recommend to monitor "your_program" and have it restarted > automatically if it crashes. The simplest way to do that is to start it > via /etc/inittab. > > Torsten > > -- > Need professional mod_perl support? > Just hire me: torsten.foert...@gmx.net >
sending all of stderr to a process
I want to send all error output to a process for specialized logging. I know that you can use CustomLog with a pipe. However, will that capture output from things like 'print STDERR ...' or if an XS module directly writes to stderr? I need that output to also go to my logging process. Does anyone know if there an Apache module which does this?
Re: Cannot get UTF-8 out of mod_perl
On Nov 16, 2007 10:34 AM, Michael Lackhoff <[EMAIL PROTECTED]> wrote: > On 16.11.2007 17:08 Michael Peters wrote: > > binmode(STDOUT, ':utf8'); > > that did the trick, many thanks! > > - Michael Michael - what version of mod_perl are you using?
Re: Cannot get UTF-8 out of mod_perl
On Nov 16, 2007 10:08 AM, Michael Peters <[EMAIL PROTECTED]> wrote:
> Michael Lackhoff wrote:
> ...
> I'm not sure if that works when using $r->print() though.
mod_perl doesn't use Perl's filter layers. $r->print is an octet
sequence interface, so you have to perform the encoding yourself
before call it, e.g.:
$r->print(Encode::encode('utf-8', $html));
Re: Cannot get UTF-8 out of mod_perl
On Nov 16, 2007 10:42 AM, Michael Lackhoff <[EMAIL PROTECTED]> wrote: > > On 16.11.2007 17:39 E R wrote: > > > On Nov 16, 2007 10:34 AM, Michael Lackhoff <[EMAIL PROTECTED]> wrote: > >> On 16.11.2007 17:08 Michael Peters wrote: > > > >> > binmode(STDOUT, ':utf8'); > >> > >> that did the trick, many thanks! > >> > >> - Michael > > > > Michael - what version of mod_perl are you using? > > 2.02 - the one that came with Ubuntu 7.10 server. > Should I build 2.03 myself instead? I can't advise on that point, but it's good to know that mod_perl 2.x does use Perl's file handle layers (at least when using STDOUT). That's not the case in 1.x.
Re: error log analyzer/reporter
I have a need to develop such an error-log analyzer but with real-time analysis. The approach I am currently considering is to implement it as a specialized web server so it can asynchronously process the log messages and respond to requests for gathered statistics. I'd like to write it all in perl, but I would consider other solutions if they are easier to maintain. Let me know if you would like to collaborate on this. ER
Re: simple profiling capability
On Nov 12, 2007 11:25 AM, Jonathan Vanasco <[EMAIL PROTECTED]> wrote: > i'm just tossing this idea out... Thanks for the ideas. The main issue I'm trying to find a good solution for is the collection of the profiling info from all the child processes because occasionally I want to poll the stats in real time to see what they are. The more I think about it, the more I think this might be a good solution: 1. a call to $profile->mark(...) will emit a special record to the log file 2. on server start up, spawn a single process to monitor the log file and collect profile records 3. have the monitoring process listen on a socket for requests for the info 4. implement some method of keeping the monitoring process alive I especially like the idea that logging won't lock up the httpd process. And Apache makes sure that log writes are atomic, right? Log files can get rolled, but a period check on the inode can detect that, right? Has something like this been implemented already?
Re: simple profiling capability
On Nov 12, 2007 9:51 AM, Geoffrey Young <[EMAIL PROTECTED]> wrote: > this should help: > > http://search.cpan.org/dist/Devel-Profiler/lib/Devel/Profiler/Apache.pm The two problems I see with this package are: 1. I don't want to profile every single subroutine. In fact, I want to profile only specific locations within my code, and a subroutine may have more than one profiling location. 2. Can I get real-time results?
simple profiling capability
Hi,
I have the need for a simple profiling capability for mod_perl applications.
At a few (< 50) points in my code I want to call something like:
$profiler->mark("some identifier");
and increment the counter for "some identifier". Later I want to be able to get
(through a web page) a summary (grouped by identifiers) of all the
profiling calls
made (of course, made by all Apache child processes.) For instance, if I want
to profile the number of times a particular subroutine was called, I would use:
sub mysubroutine {
$profiler->mark("in mysubroutine");
...
}
This is very similar to logging, except for the summary part. Does anyone
have ideas of a good way to implement this? Shared memory? An external
daemon to record these calls? Has someone already implemented this?
Robust and simple are good.
Thanks,
ER
