Hi all,
I have a problem with segmentation faults which I've been unsuccessful
at solving so far. These only occure when running the handler+filter in
a threaded worker-mpm setup, a preforked environment does not have any
problems.
First, the environment. I am running on Debian Etch, with Apache 2.2.3,
Perl 5.8.8 and the latest (on 31.5.) mod_perl from trunk with the two
patches posted by Torsten to mod-perl-dev on May.
The mod_perl handler/filter is fairly complex, consisting of:
- a PerlPostReadRequestHandler
- a PerlFixupHandler
- and a HTTP request inputfilter
The setup implements:
- on-the-fly mod_proxy configuration to select the correct backend
server based on the client request (PostReadRequestHandler + FixupHandler)
- POST to GET conversion for some requests (PostReadRequestHandler)
- POST request pass-through for others (InputFilter to re-post body)
- Regular GET request parsing and validation (PostReadRequestHandler)
- Backend-server response caching by mod_cache (for GET requests)
As said, when running with a preforked mpm, everything works great - no
segfaults at all. Also, when running on a worker mpm setup with only
*one* perl interpreter per process, the setup also works without any
problems. The problems start only when two or more interpreters per
process is running - unfortunately, this is exactly what I would need
for scalability and performance.
In addition to the segfaults (and quite often a bit prior to them), I
see one or more of the following type of error in the apache error log:
[error] lookup of 'MyApache::MyModule::' failed
[error] lookup of 'MyApache::::' failed
[error] lookup of 'MyApache::MyM\xf6\xe1\xa1le::rewrite_handler' failed
[error] lookup of '\x88\x8a\xa5\b\x10' failed
The above are examples only, and the "corruption" in the strings is
random. These errors may also appear with no segfault following them. No
other suspicious erros are visible in the logs.
The setup handles a large volume of requests - anywhere from 10-20
requests/s upto 100-200 requests/s at peaks. With these volumes, I
usually see a segmentation fault on average every couple minutes.
MaxRequestPerChild is set to 5000. Lowering this will make the faults a
bit less infrequent, but they are there still none the less.
PerlInterpMaxRequests is currently at 500 - lowering could also make the
segfaults less infrequent, but would not remove them entirely.
I've done my best to ensure that all the perl code is thread safe, and
cannot find any problems with it myself. The PostReadRequestHandler uses
sockets to communicate with a seperate process, but I've tested the code
with all socket communications removed, and it still segfaulted. The
rest of the code is pretty much standard and straighforward processing
of the request, and I cannot see how that could cause problems with
threading - only functions and methods provided by Apache2:: or APR::
are used.
The code does use the $r->notes quite a bit to store content between the
handlers and filters, and I've somehow been suspecting that maybe the
memory corruption (or what atleast looks like that) comes from somewhere
over there..
Any thoughts, ideas, solutions or further clarifying questions on the
subject are welcome!
--
Jani
