AW: print throwing intermittent Segfaults
Hi Willian,
Thanks for your checklist, I've run through it, segfaults still there...
Right now it takes less then a minute from apache restart to the first segfault.
This is from the error_log from the RedHat 5 Production machine:
Apache2::RequestIO::print: (103) Software caused connection abort at
The guys from rackspace are saying that I should recompile all my perl modules
installed directly from CPAN ( see above ) , do you think this would help?
Or has someone another hint?
Thanks
Denis
-Ursprüngliche Nachricht-
Von: William T [mailto:dietbud...@gmail.com]
Gesendet: Samstag, 21. November 2009 19:28
An: Denis Banovic
Cc: modperl@perl.apache.org
Betreff: Re: print throwing intermittent Segfaults
This is the list of stuff I usually start with when I get a problem that
doesn't seem to be tied to a particular code path.
* code path - perhaps a particular code path is only being exercised rarely,
and it has a bug
* forking - when child dies, all open descriptors in it's name space also get
closed
* eval - always a good thing to look at when weird things happen
* persistancy - globals, closures, persistant objects, serialized/restored
objects, shared memory, shared objects (between
processes)
-wjt
Von: Denis Banovic [mailto:denis.bano...@ncm.at]
Gesendet: Samstag, 21. November 2009 10:43
An: modperl@perl.apache.org
Betreff: print throwing intermittent Segfaults
Hi Everybody,
I'm having big problems with mod_perl throwing intermittent Segmentation faults
our production machines on RHEL 4 & 5.
To be able to produce a core dump on this segfaults I've installed mod_dumpcore
from this tutorial:
http://mituzas.lt/2009/09/26/getting-apache-core-dumps-in-linux/
gdb /usr/sbin/httpd core.1 produces following output:
#0 0x00b29f4b in XS_Apache__RequestRec_content_type () from
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/auto/Apache/RequestRec/RequestRec.so
and very rarely ( 1 in 15 )
#0 0x003830b9 in apr_palloc () from /usr/lib/libapr-0.so.0
The content-type is set by
$r->content_type("text/html; charset=iso-8859-1") but this is not what is
causing him to segfault...
By try and error I've figured out that the segfault happens when I do a
$r->print($mypagecontent);
I've even tried to do a
unless($r->connection->aborted) {
$r->print($mypagecontent);
}
but this didn't help either.
The segfault happens randomly, between 30 and 250 mod_perl requests. There is
no specific request URL or script that causes him to segfault, it just happens
after some time.
More load on the server means more segfaults.
>From my Apache Config:
StartServers 8
MinSpareServers5
MaxSpareServers 20
ServerLimit 256
MaxClients 200
MaxRequestsPerChild 15
There are some additional Perl Modules that I've build from CPAN:
Compress-Zlib-2.004
Digest-MD5-2.39
Email-MIME-1.861
Email-MIME-ContentType-1.014
Email-MIME-Encodings-1.311
Email-Simple-2.004
Encode-Detect-1.01
ExtUtils-CBuilder-0.23
File-Slurp-.12
IO-Compress-Zlib-2.004
MIME-Base64-3.07
MIME-Types-1.24
Module-Build-0.2808
Pod-Escapes-1.04
Pod-Simple-3.07
String-Similarity-1.03
Template-Plugin-XML-Escape-0.02
Test-Pod-1.26
Test-Simple-0.80
Has anyone a hint where to start looking and what to do next to figure out why
this segfault is happening?
Thanks
Denis
Re: AW: print throwing intermittent Segfaults
Denis Banovic wrote: Hi Willian, Thanks for your checklist, I've run through it, segfaults still there... Right now it takes less then a minute from apache restart to the first segfault. This is from the error_log from the RedHat 5 Production machine: Apache2::RequestIO::print: (103) Software caused connection abort at The guys from rackspace are saying that I should recompile all my perl modules installed directly from CPAN ( see above ) , do you think this would help? Or has someone another hint? Just my grain of salt : in my own experience, 99% of the "segfault" cases I have encountered, was when Apache or Perl tried to run a piece of code not meant for this machine (such as a library meant for another machine or another OS version). Maybe one of the modules you are using installed a wrong library ? In that sense, the guys from rackspace may be right, although I believe that the CPAN modules don't generally contain object-code libraries, or else they do compile them at installation. So maybe it is a library from the RHEL repository which is wrong.
RE: AW: print throwing intermittent Segfaults
Hi I've had a similar error I "fixed" it by adding an eval block around the offending code which was tracked back to MASON. http://rt.cpan.org/Public/Bug/Display.html?id=49031 We compile everything from scratch apache,perl,mod_perl,mason, all modules by an automated build script. Earlier when we run on mod_perl1.99 and the redhat stack it worked fine. But then we had other worries :-) -- Morten Bjoernsvik, Developer, Decision Analytics -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: 23. november 2009 09:46 To: mod_perl list Subject: Re: AW: print throwing intermittent Segfaults Denis Banovic wrote: > Hi Willian, > > Thanks for your checklist, I've run through it, segfaults still there... > Right now it takes less then a minute from apache restart to the first > segfault. > This is from the error_log from the RedHat 5 Production machine: > > Apache2::RequestIO::print: (103) Software caused connection abort at > > The guys from rackspace are saying that I should recompile all my perl > modules installed directly from CPAN ( see above ) , do you think this would > help? > Or has someone another hint? > Just my grain of salt : in my own experience, 99% of the "segfault" cases I have encountered, was when Apache or Perl tried to run a piece of code not meant for this machine (such as a library meant for another machine or another OS version). Maybe one of the modules you are using installed a wrong library ? In that sense, the guys from rackspace may be right, although I believe that the CPAN modules don't generally contain object-code libraries, or else they do compile them at installation. So maybe it is a library from the RHEL repository which is wrong.
AW: print throwing intermittent Segfaults [ solved ]
Hi Morten, Thanks a lot, By putting an eval around the code I found out, that the segfault was produced by next request to the same child after the $r->print failed. $r->print is still failing from time to time, but it's not producing segfaults anymore! Thanks Denis -Ursprüngliche Nachricht- Von: Morten Bjørnsvik [mailto:morten.bjorns...@experian-da.no] Gesendet: Montag, 23. November 2009 10:16 An: mod_perl list Betreff: RE: AW: print throwing intermittent Segfaults Hi I've had a similar error I "fixed" it by adding an eval block around the offending code which was tracked back to MASON. http://rt.cpan.org/Public/Bug/Display.html?id=49031 We compile everything from scratch apache,perl,mod_perl,mason, all modules by an automated build script. Earlier when we run on mod_perl1.99 and the redhat stack it worked fine. But then we had other worries :-) -- Morten Bjoernsvik, Developer, Decision Analytics -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: 23. november 2009 09:46 To: mod_perl list Subject: Re: AW: print throwing intermittent Segfaults Denis Banovic wrote: > Hi Willian, > > Thanks for your checklist, I've run through it, segfaults still there... > Right now it takes less then a minute from apache restart to the first > segfault. > This is from the error_log from the RedHat 5 Production machine: > > Apache2::RequestIO::print: (103) Software caused connection abort at > > The guys from rackspace are saying that I should recompile all my perl > modules installed directly from CPAN ( see above ) , do you think this would > help? > Or has someone another hint? > Just my grain of salt : in my own experience, 99% of the "segfault" cases I have encountered, was when Apache or Perl tried to run a piece of code not meant for this machine (such as a library meant for another machine or another OS version). Maybe one of the modules you are using installed a wrong library ? In that sense, the guys from rackspace may be right, although I believe that the CPAN modules don't generally contain object-code libraries, or else they do compile them at installation. So maybe it is a library from the RHEL repository which is wrong.
FW: mod-perl child process
Hi The below method used to kill the child process after the successful execution of web request. $r->child_terminate(); Can anyone suggest me where and how do I call this method in the httpd.conf file. -Raja -Original Message- From: Kulasekaran, Raja Sent: Thursday, October 29, 2009 7:36 PM To: Perrin Harkins Cc: mod_perl list Subject: RE: mod-perl child process Hi, The below method used to kill the child process after the successful execution of web request. $r->child_terminate(); How do I get the status that particular child process has been killed ?. Any suggestion on this ? Raja -Original Message- From: Kulasekaran, Raja Sent: Wednesday, October 28, 2009 10:02 AM To: Perrin Harkins Cc: mod_perl list Subject: RE: mod-perl child process So, How to I control this ?. Is it possible to reuse the existing connection ?. Raja -Original Message- From: Perrin Harkins [mailto:phark...@gmail.com] Sent: Tuesday, October 27, 2009 11:47 PM To: Kulasekaran, Raja Cc: mod_perl list Subject: Re: mod-perl child process On Tue, Oct 27, 2009 at 8:33 AM, Kulasekaran, Raja wrote: > I have configured the mod_perl with oracle persistent connection through Apache::DBI module. On every web page request It creates a process > something like below and It never be killed automatically when the request has completed. That is the intended behavior. You should get one Oracle connection for each apache child process. They will stay connected for the life of the process. - Perrin
Re: mod-perl child process
Kulasekaran, Raja wrote: > The below method used to kill the child process after the successful > execution of web request. > > $r->child_terminate(); > > Can anyone suggest me where and how do I call this method in the > httpd.conf file. You don't, you put it in your application code. However you should not be calling this under normal circumstances as all it'll do is cause the current apache child process to exit and for the main apache process to fork another child. Which will massively increase the load on your server for zero gain. Specifically having read your previous posts it will not reduce the number of DB connections you're seeing, as the newly-forked replacement process will make a new DB connection. You'll also increase the load on your DB server as Oracle will be constantly closing down connections and opening new connections - which is relatively expensive in Oracle and the reason that modules such as Apache::DBI exist in the first place. Assuming that your problem is the number of Oracle processes, then you may be better switching to multithreaded Oracle. You may also be able to reduce the number of connection by checking your code base to ensure that the same options are used whenever you request a DB handle. Finally you'll be able to limit the number of connection by limiting the number of Apache child processes (MaxClients in httpd.conf) - however all that you're likely to achieve is pushing the bottleneck closer to the client. As Perrin has already suggested if you're not proxying or dealing with static content in another manner you need to ensure that these requests aren't going through to your mod_perl server. Carl
Dynamically setting PerlVars in Apache per-request
WAS: A better way to handle multiple client authentication? Yeah I use something similar in another application, but in this application I actually need to change the Auth_DBI_data_source variable since the "FROM pwd_table" would actually need to be "FROM clientA.pwd_table" and I can't see how to set this on the fly. I could probably also set the: Auth_DBI_pwd_table variable as well, but again the per-request setting is what's throwing me off. PerlSetVar Auth_DBI_data_source DBI:mysql:clientA or PerlSetVar Auth_DBI_pwd_table clientA.pwd_table Which is why I thought: RewriteRule ^/(.+)/$ PerlSetVar Auth_DBI_data_source DBI:mysql:$1 I was hoping a SetEnvIf or IfDefine would work but after reading more about Apache configuration I see it won't. Anyway, this is straying too far into Apache territory so I guess I will just set those variables within a modified Apache::AuthDBI I guess if anyone already knows an auth module that does that above that would be awesome, or if anyone knows how to easily change PerlVars on the fly within the Apache config/htaccess space that's be great, otherwise it's a small change to the above module. Thanks again! Tosh William T wrote: The documentation alludes to the variable 'pwd_whereclause'. If this variable is set it will be used in the passwd query. I would try and set it per client so that the query gets an additional where clause: SELECT pwd_field FROM pwd_table WHERE uid_field = user AND client = clientA I havn't actually tried this so I don't know if there are any caveats, but from the docs at least it seems possible. The only trick is making sure you can reset the pwd_whereclause with each different client url, and make client an additional column in your pwd_table. -- -wjt -- McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
Re: Dynamically setting PerlVars in Apache per-request
My suggestion would be to subclass AuthDBI to make the constructor fiddle with the dir_config entries that AuthDBI uses. See the docs for dir_config (the perl interface to PerlSetVar variables: http://perl.apache.org/docs/2.0/api/Apache2/ServerUtil.html#C_dir_config_ I have no idea how subclass friendly AuthDBI is or isn't. Adam Tosh Cooey wrote: WAS: A better way to handle multiple client authentication? Yeah I use something similar in another application, but in this application I actually need to change the Auth_DBI_data_source variable since the "FROM pwd_table" would actually need to be "FROM clientA.pwd_table" and I can't see how to set this on the fly. I could probably also set the: Auth_DBI_pwd_table variable as well, but again the per-request setting is what's throwing me off. PerlSetVar Auth_DBI_data_source DBI:mysql:clientA or PerlSetVar Auth_DBI_pwd_table clientA.pwd_table Which is why I thought: RewriteRule ^/(.+)/$ PerlSetVar Auth_DBI_data_source DBI:mysql:$1 I was hoping a SetEnvIf or IfDefine would work but after reading more about Apache configuration I see it won't. Anyway, this is straying too far into Apache territory so I guess I will just set those variables within a modified Apache::AuthDBI I guess if anyone already knows an auth module that does that above that would be awesome, or if anyone knows how to easily change PerlVars on the fly within the Apache config/htaccess space that's be great, otherwise it's a small change to the above module. Thanks again! Tosh William T wrote: The documentation alludes to the variable 'pwd_whereclause'. If this variable is set it will be used in the passwd query. I would try and set it per client so that the query gets an additional where clause: SELECT pwd_field FROM pwd_table WHERE uid_field = user AND client = clientA I havn't actually tried this so I don't know if there are any caveats, but from the docs at least it seems possible. The only trick is making sure you can reset the pwd_whereclause with each different client url, and make client an additional column in your pwd_table. -- -wjt
Announcing: Server::Control and apachectlp
Server::Control allows you to control servers ala apachectl, but with better diagnostics and many more features. It includes both a drop-in replacement for apachectl (”apachectlp”) and an OO interface. Though it was designed with Apache in mind, there are also subclasses for HTTP::Server::Simple and Net::Server, and it’s general enough to use with any server that has a pid file and listens on a port. Features include: * Checks server status via both pid file and port connect * Detects and handles corrupt or out-of-date pid files * Reports what is using a port when it is busy (via lsof) * Tails the error log when server fails to start * Supports sudo usage for restricted (< 1024) port * Easy to customize for your environment with start/stop hooks, etc. Coming soon: plugins for common tasks associated with starting and stopping servers, such as generating conf files from templates and auto-restarting a server on file change. If you try this and have problems getting it to work, or suggestions for improvement, please let me know! Jon
Re: Announcing: Server::Control and apachectlp
Perhaps it would be prudent to add some logic that when "lsof" is not available, check for (and possibly use) "fstat" instead. Do note that the syntax of these two commands (lsof vs. fstat) are not the same and additional logic may need to be added. Just thought I'd mention "fstat" because not all systems have "lsof" (for example, FreeBSD vs. Linux). -- Devin On Mon, 2009-11-23 at 12:14 -0800, Jonathan Swartz wrote: > Server::Control allows you to control servers ala apachectl, but with > better diagnostics and many more features. It includes both a drop-in > replacement for apachectl (”apachectlp”) and an OO interface. > > Though it was designed with Apache in mind, there are also subclasses > for HTTP::Server::Simple and Net::Server, and it’s general enough to > use with any server that has a pid file and listens on a port. > > Features include: > > * Checks server status via both pid file and port connect > * Detects and handles corrupt or out-of-date pid files > * Reports what is using a port when it is busy (via lsof) > * Tails the error log when server fails to start > * Supports sudo usage for restricted (< 1024) port > * Easy to customize for your environment with start/stop hooks, > etc. > > Coming soon: plugins for common tasks associated with starting and > stopping servers, such as generating conf files from templates and > auto-restarting a server on file change. > > If you try this and have problems getting it to work, or suggestions > for improvement, please let me know! > > Jon > -- Cheers, Devin Teske -> CONTACT INFORMATION <- Field Engineer FIS - Vicor Business Unit 626-573-6040 Office 510-735-5650 Mobile devin.te...@metavante.com -> LEGAL DISCLAIMER <- This message contains confidential and proprietary information of the sender, and is intended only for the person(s) to whom it is addressed. Any use, distribution, copying or disclosure by any other person is strictly prohibited. If you have received this message in error, please notify the e-mail sender immediately, and delete the original message without making a copy. -> END TRANSMISSION <-
Apache2/mod_perl2 test in module
Hi. I created a perl object module which can be used by mod_perl2 handlers, cgi scripts, or stand-alone perl programs. It has a new() method which creates an object $obj, which is later used via standard $obj->method() calls. What would be a recommended/elegant mechanism/idiom, to have this module log warnings and errors appropriately, depending on where it is currently being used, e.g. : - to the Apache2 error log when it is used within Apache/mod_perl2 - warn() if it is used by a stand-alone program - ?? when used in a cgi script Thanks for ideas/examples/pointers.
[mp2] OS X snow leopard compile issue
Hello,
I tried to compile mod_perl 2.0.4 on mac os x 10.6.2
httpd 2.2.14
./configure --prefix=/tmp/q
make ; make install
mod_perl 2.0.4 and latest from SVN
perl Makefile.PL MP_APXS=/tmp/q/bin/apxs PREFIX=/tmp/q2
system perl: perl -V
Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
Platform:
osname=darwin, osvers=10.0, archname=darwin-thread-multi-2level
uname='darwin b66.apple.com 10.0 darwin kernel version 10.0.0d8: tue may 5
19:29:59 pdt 2009; root:xnu-1437.2~2release_i386 i386 '
config_args='-ds -e -Dprefix=/usr -Dccflags=-g -pipe -Dldflags=
-Dman3ext=3pm -Duseithreads -Duseshrplib -Dinc_version_list=none -Dcc=gcc-4.2'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_64_BIT_ALL
USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
USE_PERLIO USE_REENTRANT_API
Locally applied patches:
/Library/Perl/Updates/ comes before system perl directories
installprivlib and installarchlib points to the Updates directory
Built under darwin
Compiled at Jul 9 2009 15:07:54
@INC:
/Library/Perl/Updates/5.10.0/darwin-thread-multi-2level
/Library/Perl/Updates/5.10.0
/System/Library/Perl/5.10.0/darwin-thread-multi-2level
/System/Library/Perl/5.10.0
/Library/Perl/5.10.0/darwin-thread-multi-2level
/Library/Perl/5.10.0
/Network/Library/Perl/5.10.0/darwin-thread-multi-2level
/Network/Library/Perl/5.10.0
/Network/Library/Perl
/System/Library/Perl/Extras/5.10.0/darwin-thread-multi-2level
/System/Library/Perl/Extras/5.10.0
.
after make and make install I got .so file.
When I try to start apache now I get an error:
./apachectl start
httpd: Syntax error on line 53 of /tmp/q/conf/httpd.conf: Cannot load
/tmp/q/modules/mod_perl.so into server: dlopen(/tmp/q/modules/mod_perl.so, 10):
Symbol not found: _ap_add_input_filter\n Referenced from:
/tmp/q/modules/mod_perl.so\n Expected in: flat namespace\n in
/tmp/q/modules/mod_perl.so
I decided to perform static installation but was not successful too:
I tried to build via:
perl Makefile.PL MP_USE_STATIC=1 MP_AP_PREFIX=/tmp/httpd-2.2.14
MP_AP_CONFIGURE="--with-mpm=prefork" PREFIX=/tmp/q3
...
mod_perl.c:768: error: storage class specified for parameter
'modperl_destruct_level'
mod_perl.c:768: error: parameter 'modperl_destruct_level' is initialized
mod_perl.c:771: error: expected '=', ',', ';', 'asm' or '__attribute__' before
'{' token
mod_perl.c:778: error: expected ';', ',' or ')' before '*' token
mod_perl.c:788: error: expected '=', ',', ';', 'asm' or '__attribute__' before
'{' token
mod_perl.c:823: error: expected ')' before '*' token
mod_perl.c:833: error: expected ')' before '*' token
mod_perl.c:904: error: expected '=', ',', ';', 'asm' or '__attribute__' before
'modperl_cmds'
mod_perl.c:961: error: expected declaration specifiers before ';' token
mod_perl.c:963: error: expected ')' before '*' token
mod_perl.c:986: error: expected ')' before '*' token
mod_perl.c:994: error: expected ')' before '*' token
mod_perl.c:1016: error: expected ')' before '*' token
mod_perl.c:1057: error: expected ')' before '*' token
mod_perl.c:1140: error: expected '=', ',', ';', 'asm' or '__attribute__' before
'{' token
mod_perl.c:1145: error: expected declaration specifiers before 'module'
mod_perl.c:1153: error: expected declaration specifiers before ';' token
mod_perl.c:1153: error: old-style parameter declarations in prototyped function
definition
/tmp/httpd-2.2.14/srclib/apr/include/apr_errno.h:52: error: parameter name
omitted
mod_perl.c:1153: error: expected '{' at end of input
make[1]: *** [mod_perl.o] Error 1
make: *** [modperl_lib] Error 2
any ideas how to make step further?
--
Problems running mod_perl install tests - where should these be posted to
Regards, Greg George IT Shared Services Phone: +613-9091-2492 3/100 Victoria Prd, Melbourne Please consider the environment before printing this e-mail *** Please consider the environment before printing this e-mail. This message is intended solely for the individual(s) and entity(s) addressed. It is confidential and may contain legally privileged information. The use, copying or distribution of this message or any information it contains, by anyone other than the addressee, is prohibited. If you have received this message in error, please notify postmas...@orica.com. The mailbox address from which this message has been sent is for business mail only. Mail sent to it may be subject to security scanning and delivery on non-business messages sent to this address may not occur. Thank you. ***
