maintaining shared memory size (was: Re: swamped with connection?)
I think I have to reword the question: How do I maintain the size of the shared memory between apache children? What cause a memory page to be copied (not shared) from perl's point of view? This brings the question of how to increase shared memory usage. I've tried to load every modules upfront. But even before any request comes in, the shared memory is only 7 Mb. What makes it so small? Thanks... --- Badai Aqrandista Cheepy (?) _ Your opinion counts..for your chance to win a Mini Cooper click here http://www.qualifiedopinions.com/joinup.php?source=hotmail
Re: swamped with connection?
On Tue, 2005-08-23 at 15:52 +1000, Badai Aqrandista wrote: RAM = 700 Mb Per process total size = 40 Mb Shared memory = 7 Mb So, the number of processes = (700 - 7) / 33 = 21 processes So, does that mean it can only accept up to 21 connections? Yes. If you are running a reverse proxy in front of it, that should be enough to handle a lot of traffic. However, you should be aware that a few months back we discovered that our methods for measuring shared memory didn't work very well on Linux 2.4 kernels and don't really work at all on 2.6 kernels, so there may be more sharing (via copy-on-write) going on than you can see here. Looking at the total free memory on your machine when 21 processes are running may be more useful than just doing the calculation. - Perrin
Re: maintaining shared memory size (was: Re: swamped with connection?)
On Tue, 2005-08-23 at 17:23 +1000, Badai Aqrandista wrote: How do I maintain the size of the shared memory between apache children? What cause a memory page to be copied (not shared) from perl's point of view? Anything that writes to memory -- modifying any variable (even just reading one in a different context) or compiling some code are the most common things. There's a bit more here: http://modperlbook.com/html/ch10_01.html - Perrin
Re: Can't get SOAP to work under mod_perl
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: | As Perrin mentioned, this may not be a problem with mod_perl | specifically, so it may be worth asking on the soap-lite | mailing list. Just to verify this, I've placed at | http://people.apache.org/~randyk/ | a file, bug-reporting-skeleton-mp1.tar.gz, which can be | used to test your mod_perl SOAP installation. To use it: | perl Makefile.PL | make | make test | If the tests pass, can you see differences between the | modules and/or configuration used here and the one you | have? If the tests don't pass, is there anything useful | in the error log? Or does | perl t/TEST -v | provide any clues? | It doesn't appear I can run this. It uses ModPerl::MM, which appears to be part of mod_perl 2, but as I mentioned, I'm using 1.29. I thought I'd start with the modperl list, since this all works under CGI, and I assumed the problem would be something I have misconfigured in mod_perl or something I have to do differently with mod_perl. I'll go pester the SOAP people now. :-) Thanks for the help everyone! - -- Steve Baker AOL Configuration Management kiku wa ittoki no haji kikanu wa matsudai no haji -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.4 (Darwin) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDCyGiz179gX3oLkwRAjnFAJ9inRKDWODvRNKyjEdrCWVTL8h+RgCfeTMl fv+tH8uELViePDPqhxsD2Yo= =T/Yn -END PGP SIGNATURE-
Re: swamped with connection?
On Tuesday 23 August 2005 14:23, Perrin Harkins wrote: However, you should be aware that a few months back we discovered that our methods for measuring shared memory didn't work very well on Linux 2.4 kernels and don't really work at all on 2.6 kernels, so there may be more sharing (via copy-on-write) going on than you can see here. See also http://marc.theaimsgroup.com/?l=apache-modperlm=112343986910467w=2 Torsten pgptlfOwFDQgb.pgp Description: PGP signature
'make test' fails for mp2 under HP-UX
Sorry, the subject on my first post probably obscured the fact that this was a new problem post. -8-- Start Bug Report 8-- 1. Problem Description: make test fails on t/modperl/request_rec_tie_api.t . # Running under perl version 5.006002 for hpux # Current time local: Mon Aug 22 16:22:54 2005 # Current time GMT: Mon Aug 22 20:22:54 2005 # Using Test.pm version 1.24 # Using Apache/Test.pm version 1.25 FAILED before any test output arrived Failed Test Stat Wstat Total Fail Failed List of Failed --- t/modperl/request_rec_tie_api.t ?? ?? % ?? Failed 1/1 test scripts, 0.00% okay. 3/3 subtests failed, 0.00% okay. [warning] server localhost:8529 shutdown [ error] error running tests (please examine t/logs/error_log) [ error] oh crap, server dumped core I've tinkered around with request_rec_tie_api.pm and the line it dies on looks innocent enough: my $fileno = fileno STDOUT; fileno on STDERR or STDIN return the expected results. The same code under plain perl works fine. If I change the line to: my $fileno = 1; just to get past the problem everything works fine. 2. Used Components and their Configuration: *** mod_perl version 2.01 *** using /opt/local/wrfopt/build/mod_perl-2.0.1/lib/Apache2/BuildConfig.pm *** Makefile.PL options: MP_APR_LIB = aprext MP_APXS= /usr/opt/httpd-2.0.54/bin/apxs MP_CCOPTS = -I/usr/opt/mod_perl-2.0.1/include -I/opt/default/include +z +DA1.1 MP_COMPAT_1X = 1 MP_DEBUG = 1 MP_GENERATE_XS = 1 MP_LIBNAME = mod_perl MP_TRACE = 1 MP_USE_DSO = 1 *** /usr/opt/httpd-2.0.54/bin/httpd -V Server version: Apache/2.0.54 Server built: Aug 22 2005 15:11:53 Server's Module Magic Number: 20020903:9 Architecture: 32-bit Server compiled with -D APACHE_MPM_DIR=server/mpm/prefork -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D HTTPD_ROOT=/usr/opt/httpd-2.0.54 -D SUEXEC_BIN=/usr/opt/httpd-2.0.54/bin/suexec -D DEFAULT_PIDLOG=logs/httpd.pid -D DEFAULT_SCOREBOARD=logs/apache_runtime_status -D DEFAULT_LOCKFILE=logs/accept.lock -D DEFAULT_ERRORLOG=logs/error_log -D AP_TYPES_CONFIG_FILE=conf/mime.types -D SERVER_CONFIG_FILE=conf/httpd.conf *** (apr|apu)-config linking info -L/usr/opt/httpd-2.0.54/lib -laprutil-0 -lc -ldb-4.3 -lexpat -liconv -L/usr/opt/httpd-2.0.54/lib -lapr-0 -lm -lnsl -lpthread -ldld *** /usr/opt/perl-5.6.2/bin/perl -V Summary of my perl5 (revision 5.0 version 6 subversion 2) configuration: Platform: osname=hpux, osvers=11.11, archname=PA-RISC1.1 uname='hp-ux cars-l b.11.11 u 9000800 134921527 unlimited-user license ' config_args='-O -Ubincompat5005 -Uuselargefiles -Uuseithreads -Darchname=PA-RISC1.1 -Accflags=+z -Accflags=-DPERL_POLLUTE -Accflags=+DA1.1 -Dcccdlflags=+z -Dprefix=/usr/opt/perl-5.6.2 -Doptimize=-g -Dloclibpth=/usr/opt/perl-5.6.2/lib /lib /usr/lib /usr/ccs/lib /usr/opt/libiconv-1.9.1/lib /usr/opt/readline-5.0/lib /usr/opt/openssl-0.9.8/lib /usr/opt/ncurses-5.4/lib /usr/opt/gettext-0.12.1/lib /usr/opt/jpeg.v6b/lib /opt/informix/lib /usr/opt/zlib-1.2.3/lib /usr/opt/db-4.3.27/lib /usr/opt/expat-1.95.8/lib /usr/opt/gzip-1.2.4/lib /usr/opt/freetds-0.63RC10/lib /usr/opt/libpng-1.2.8/lib /usr/opt/bzip2-1.0.2/lib /usr/opt/gdbm-1.8.3/lib /usr/opt/freetype-2.1.7/lib /usr/opt/gd-2.0.15/lib /usr/opt/tiff-3.7.1/lib -Dlocincpth=/usr/opt/perl-5.6.2/include /include /usr/include /usr/ccs/include /usr/opt/libiconv-1.9.1/include /usr/opt/readline-5.0/include /usr/opt/openssl-0.9.8/include /usr/opt/ncurses-5.4/include /usr/opt/gettext-0.12.1/include /usr/opt/jpeg.v6b/include /opt/informix/incl /usr/opt/zlib-1.2.3/include /usr/opt/db-4.3.27/include /usr/opt/expat-1.95.8/include /usr/opt/gzip-1.2.4/include /usr/opt/freetds-0.63RC10/include /usr/opt/libpng-1.2.8/include /usr/opt/bzip2-1.0.2/include /usr/opt/gdbm-1.8.3/include /usr/opt/freetype-2.1.7/include /usr/opt/gd-2.0.15/include /usr/opt/tiff-3.7.1/include -Dcc=/opt/ansic/bin/cc -Dlibs= -lnm -lndbm -lgdbm -ldb -lmalloc -ldld -lm -lcrypt -lsec -lpthread -lc -lcl -lCsup -e -s -d' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef useperlio=undef d_sfio=undef uselargefiles=undef usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef Compiler: cc='/opt/ansic/bin/cc', ccflags ='-Ae -D_HPUX_SOURCE -Wl,+vnocompatwarnings +z -DPERL_POLLUTE +DA1.1 -DDEBUGGING -I/usr/opt/libiconv-1.9.1/include -I/usr/opt/readline-5.0/include -I/usr/opt/openssl-0.9.8/include -I/usr/opt/ncurses-5.4/include -I/usr/opt/gettext-0.12.1/include -I/usr/opt/jpeg.v6b/include -I/opt/informix/incl -I/usr/opt/zlib-1.2.3/include
Re: Can't get SOAP to work under mod_perl
On Tue, 23 Aug 2005, Steve Baker wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [EMAIL PROTECTED] wrote: | As Perrin mentioned, this may not be a problem with mod_perl | specifically, so it may be worth asking on the soap-lite | mailing list. Just to verify this, I've placed at | http://people.apache.org/~randyk/ | a file, bug-reporting-skeleton-mp1.tar.gz, which can be | used to test your mod_perl SOAP installation. To use it: | perl Makefile.PL | make | make test | If the tests pass, can you see differences between the | modules and/or configuration used here and the one you | have? If the tests don't pass, is there anything useful | in the error log? Or does | perl t/TEST -v | provide any clues? | It doesn't appear I can run this. It uses ModPerl::MM, which appears to be part of mod_perl 2, but as I mentioned, I'm using 1.29. Sorry about that - I forgot to change that section in the Makefile.PL. I replaced the archive in the link above with one that has the appropriate change. -- best regards, randy
MP2 easy cookie interface available
Hi folks, I found the available cookie methods a bit tricky to use (I think the blessed scalars or overloading were confusing me!), so I wrote a simple cookie interface to use in place of Apache2::Cookie. Most of the code is copied from that. I haven't got time to CPAN-ify it at the mo, but there is some POD at the bottom and there are no licence conditions attached - I guess it should inherit the Apache2::Cookie license? Its main feature is automatic use of MD5 hashes to protect the cookie from being modified by the user, so you can store things in it safely. Grab it here: http://www.versatilia.com/downloads/Validated.pm and install it as Apache2/Cookie/Validated.pm somewhere in @INC. cheers John
Re: swamped with connection?
Torsten Foertsch wrote: On Tuesday 23 August 2005 14:23, Perrin Harkins wrote: However, you should be aware that a few months back we discovered that our methods for measuring shared memory didn't work very well on Linux 2.4 kernels and don't really work at all on 2.6 kernels, so there may be more sharing (via copy-on-write) going on than you can see here. See also http://marc.theaimsgroup.com/?l=apache-modperlm=112343986910467w=2 Speaking of which, is there any reason not to commit this? I can finally verify that it works(doesn't break anything) locally on a new box I've got. One comment might be to add an option to turn it off even if the newer Smaps support is present? -- END What doesn't kill us can only make us stronger. Nothing is impossible. Philip M. Gollucci ([EMAIL PROTECTED]) 301.254.5198 Consultant / http://p6m7g8.net/Resume/ Senior Developer / Liquidity Services, Inc. http://www.liquidityservicesinc.com http://www.liquidation.com http://www.uksurplus.com http://www.govliquidation.com http://www.gowholesale.com
Re: swamped with connection?
On Tuesday 23 August 2005 21:08, Philip M. Gollucci wrote: One comment might be to add an option to turn it off even if the newer Smaps support is present? I have also thought of it. Sounds sound. I'll send a patch soon. Torsten pgpKu6J3Mu1CO.pgp Description: PGP signature
Re: MP2 easy cookie interface available
Having it validate the cookie based on a hash on data+ server secret is a great addition. I've been doing that already on a custom abstraction class for Apache::Cookie as well -- i needed to build something that would switch the cookie baking code to either use headers_out or bake on the fly, to get around a since fixed bug. Anyways, since i know many people who have subclassed or manipulated cookies to act like that, Maybe instead of your module being cpanned as a standalone, it could be incorporated into the main Apache2::Cookie ? ie: patch Apache2::Cookie to add a 'validation_require' flag, a 'validation_serversecret' string for hashing/matching, and have fetch return an error if validation is required but the hash doesn't match. I guess i should talk about that on the libapreq list though - there might be large opposition to it there and any patching i do will be hours of wasted time. For the past few months, I've been thinking about making a CookieFactory module that is kind of like the BBC's anytemplate -- just a simple abstraction class with simple configuration that can get/send via Apache::Cookie ( or another backend like cgi) , include the server-secret hack, but let it be a little easier for configuration (ie, i hate putting stuff in httpd.conf - i like to consolidate as much of my configuration in startup.pl or in the main handler.) if the main apache::cookie handled this, then i could just forget about my stupid idea and never think of it again. I really wanted a module called Cookie::Factory though.. the pod could be so amazingly funny.
Re: MP2 easy cookie interface available
Sounds great, it'd be a nice addition to Apache2::Cookie. Getting the
secret through to freeze and thaw is the tricky bit - I guess it could
just be supplied to fetch() and new().
There are a couple of extras that I'm adding:
- a delete method which deletes the cookie by name (ie. you don't have
to create an object)
- an expiry time validator - if you absolutely need cookies to expire on
time, store the expiry time in the validated cookie and check it,
instead of relying on the user to expire it.
I really wanted a module called Cookie::Factory though.. the pod could
be so amazingly funny.
Hehe! Perl is punny sometimes... the other day I wrote sub merge {...
...back to the code for me, I've started using Set-Cookie2 and firefox
is ignoring them :(
John
Re: maintaining shared memory size (was: Re: swamped withconnection?)
On Tue, 2005-08-23 at 17:23 +1000, Badai Aqrandista wrote: How do I maintain the size of the shared memory between apache children? What cause a memory page to be copied (not shared) from perl's point of view? Anything that writes to memory -- modifying any variable (even just reading one in a different context) or compiling some code are the most common things. There's a bit more here: http://modperlbook.com/html/ch10_01.html Oh my, I should've kept memory consumption in mind when designing the code in the first place. Anyway, to fix this, I'm trying to make my onw shared memory with Apache::SharedMem. But it seems that shared memory is just a memory area that any process can read from or write to, not a memory that can become part of any process's memory space. I'd like to put the read only data in a memory area that can become part of any process's memory space, like threads sharing memory space. Is there any way I can do that? How about mmap? I have never looked into that. Does this sound like fixing the wrong problem? Thank you... --- Badai Aqrandista Cheepy (?) _ On the road to retirement? Check out MSN Life Events for advice on how to get there! http://lifeevents.msn.com/category.aspx?cid=Retirement
Re: mp1 to mp2: server info in startup.pl
Philip M. Gollucci wrote:
In mp2 I am doing this:
---
use Apache2::ServerRec ();
use Apache2::ServerUtil ();
my $s = Apache2::ServerUtil-server();
warn port = ,$s-port(),\n;
Try
use Apache2::RequestUtil ();
Apache2::RequestUtil-request()-get_server_port()
Though I don't see why your method doesn't work. I'll give it a spin in
a bit.
Okay, I'll verify that neither method works from startup.pl whether I pull it
in via
PerlRequire or PerlPostConfigRequire.
I will say that the following handler gives me mixed Results:
package TEST::Port;
use strict;
use warnings FATAL = 'all';
use Carp;
use Apache2::RequestUtil ();
use Apache2::ServerUtil ();
use Apache2::Const -compile = qw (OK);
sub handler {
my $r = shift;
$r-content_type('text/html');
my $s = $r-server();
my $port1 = $s-port();
my $port2 = $r-get_server_port();
my $port3 = Apache2::ServerUtil-server()-port();
$r-print(Ports: $port1, $port2, $port3 br /\n);
return Apache2::Const::OK;
}
Gives me:
Ports: 0, 8000, 0
I checked in httpd/server/core.c:ap_get_server_port() via gdb and yes,
it does return 8000. So that verifies #2 works.
As far as I can tell 1, 3 don't come through this function.. but I can't for
the life of me figure out where the hell
they do go. Any ideas?
I also can't see where $s-port() is tested in the test suite. Only
$r-get_server_port is which works :)
I would assume the end up calling the same function though.
--
END
What doesn't kill us can only make us stronger.
Nothing is impossible.
Philip M. Gollucci ([EMAIL PROTECTED]) 301.254.5198
Consultant / http://p6m7g8.net/Resume/
Senior Developer / Liquidity Services, Inc.
http://www.liquidityservicesinc.com
http://www.liquidation.com
http://www.uksurplus.com
http://www.govliquidation.com
http://www.gowholesale.com
Re: MP2 easy cookie interface available
Fowarding to apreq-dev (at) apache.org Did you really need to override all those methods? Granted I haven't looked at it that closely. The bigest thing I would say is you should subclass perhaps APR::Request::Cookie instead as that is the API we recommend these days. John ORourke wrote: Hi folks, I found the available cookie methods a bit tricky to use (I think the blessed scalars or overloading were confusing me!), so I wrote a simple cookie interface to use in place of Apache2::Cookie. Most of the code is copied from that. I haven't got time to CPAN-ify it at the mo, but there is some POD at the bottom and there are no licence conditions attached - I guess it should inherit the Apache2::Cookie license? Its main feature is automatic use of MD5 hashes to protect the cookie from being modified by the user, so you can store things in it safely. Grab it here: http://www.versatilia.com/downloads/Validated.pm and install it as Apache2/Cookie/Validated.pm somewhere in @INC. cheers John -- END What doesn't kill us can only make us stronger. Nothing is impossible. Philip M. Gollucci ([EMAIL PROTECTED]) 301.254.5198 Consultant / http://p6m7g8.net/Resume/ Senior Developer / Liquidity Services, Inc. http://www.liquidityservicesinc.com http://www.liquidation.com http://www.uksurplus.com http://www.govliquidation.com http://www.gowholesale.com
Re: MP2 easy cookie interface available
Thanks for the feedback Philip. I believe it does subclass APR::Request::Cookie but I just left in all the 'use' statements from Apache2::Cookie. I figured the APR:: namespace would be the wrong place for it too. It need to override so many methods because they need to get the RequestRec object so the get_secret method can read the secret from dir_config - I couldn't think of an easier way but it does seem a bit much. cheers John Philip M. Gollucci wrote: Fowarding to apreq-dev (at) apache.org Did you really need to override all those methods? Granted I haven't looked at it that closely. The bigest thing I would say is you should subclass perhaps APR::Request::Cookie instead as that is the API we recommend these days. John ORourke wrote: http://www.versatilia.com/downloads/Validated.pm
Re: Apache, Mod_Perl Upgrade
Tom Schindl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I went back to my 2.0.40 install of Apache. The 2.0.54 was not playing nice with plesk. How do I update the @INC path and/or mod_perl for the new perl 5.8.7? I try using CPAN with install Bundle::Apache and no go. Any ideas? -Jon I not sure if there have been API changes since 2.0.40 and 2.0.54 but compiling mod-perl-2.x with apache 2.0.54 and use it with 2.0.40 this will certainly lead to very strange situations you. As in the message before there's nothing wrong when running 2.0.40 if your plesk system needs it and install your own apache-2.0.54 and run it on a different port (e.g. 81). As already said before als compiling Apache-2.0.x is straight forward and nothing you could not manage without reading the bundled INSTALL-File(s). Default-Install-Procedure: - --8-- tar xzvf stable.tar.gz cd perl-5.8.7/ sh ./Configure -de -Dprefix=/opt/myperl make make test su make install exit cd .. tar xjvf httpd-2.0.54.tar.bz2 cd httpd-2.0.54 ./configure --prefix=/opt/myapache make make install vi /opt/myapache/conf/httpd.conf # search for Listen and set it to 81 cd .. cd mod_perl-2.0.1/ /opt/myperl/bin/perl Makefile.PL MP_APXS=/opt/myapache/bin/apxs make make test make install vi /opt/myapache/conf/httpd.conf # Add LoadModule perl_module modules/mod_perl.so /opt/myapache/bin/apachectl start - --8-- I've timed it and it took my now exactly 20 Minutes to get a setup with: - - perl 5.8.7 - - httpd-2.0.54 - - mod-perl 2.0.1 When it comes to php I have no idea but I think its also that straight forward. As I've always said I'm not a friend of those rpm's and it really does take that long to get a running system. At least less time than it took you: - - find what's going wrong with your installation - - work with ancient software like apache-2.0.40 causing you headache because you are runing it long standing fixed bugs Tom -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Using GnuPG with Mandriva - http://enigmail.mozdev.org iD8DBQFDBX0jkVPeOFLgZFIRAg63AJ485/bL6i5wgJxcPtbXG4Raw9VffgCgniot hkzZ4lKj1xLzk36cKQ6xB6M= =dINX -END PGP SIGNATURE- I am currently using these install notes. I have run into another problem. This must have been caused by one of my first perl installs? /opt/perl/bin/perl Makefile.PL MP_APXS=/opt/httpd/bin/apxs Reading Makefile.PL args from @ARGV MP_APXS = /opt/httpd/bin/apxs no conflicting prior mod_perl version found - good. [ error] '/opt/httpd/bin/apxs -q INCLUDEDIR' failed: [ error] Can't locate strict.pm in @INC (@INC contains: /usr/local/lib/perl5/5.8.7/i686-linux /usr/local/lib/perl5/5.8.7 /usr/local/lib/perl5/site_perl/5.8.7/i686-linux /usr/local/lib/perl5/site_perl/5.8.7 /usr/local/lib/perl5/site_perl .) at /opt/httpd/bin/apxs line 19. BEGIN failed--compilation aborted at /opt/httpd/bin/apxs line 19. [ error] Unable to determine server version, aborting. [ error] Invalid MP_APXS specified? I have installed perl and apache as described above. Include should be /opt/perl? Any ideas? Thanks in advance. Jon
Re: maintaining shared memory size (was: Re: swamped withconnection?)
On Wed, 2005-08-24 at 10:31 +1000, Badai Aqrandista wrote: Anyway, to fix this, I'm trying to make my onw shared memory with Apache::SharedMem. Don't use that module. It's very inefficient. But it seems that shared memory is just a memory area that any process can read from or write to, not a memory that can become part of any process's memory space. I'd like to put the read only data in a memory area that can become part of any process's memory space, like threads sharing memory space. Is there any way I can do that? You can share read-only data by loading it into normal variables during startup. You can share read/write data by keeping it in a database (including things like BerkeleyDB or Cache::FastMmap) and only reading the small parts of it that you need in each process. There is no way to access data in perl without the size of the process you read the data from allocating memory to hold that data while you use it. In other words, shared memory is not useful for reducing the size of existing processes unless those processes are currently each holding a lot of data that they never use. Does this sound like fixing the wrong problem? Yes. Put a reverse proxy in front of your server, tune MaxClients so you won't go into swap, and then benchmark to see how much load you can handle. Then think about tuning. - Perrin
