Re: [mp2 Patch] BUG with mod_deflate and $|=1 (20014:Error string not specified)
Try this patch: [...] Feel free to submit this bug report and the fix to httpd-dev. Please let me know if you do that, so I won't duplicate it. But I'd prefer that you do it so you can make sure that it gets fixed in the next release, since you need it working. I've just verified that your patch fixes my problem. I've never submitted a but report and fix to httpd-dev, but I'm willing to do it. How do I do it? -bill
Re: [mp2 Patch] BUG with mod_deflate and $|=1 (20014:Error string not specified)
Please report to the list the bug id so we can document this issue for those who have the same problem with older httpds. Thanks. OK, I've posted it. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=22259 Thanks for the fix! -bill
Re: [mp2 Patch] BUG with mod_deflate and $|=1 (20014:Error string not specified)
I can measure it myself if you can provide me with URLs to your resources and identify them in terms of which one is mod_CGI and which is mod_perl. This is the mod_cgi one that works fine, no errors: http://shevek.kenyonhill.com/cgi/test.pl This is the mod_perl one (same script) that generates the 20014:Error in the error_log. Also, the page doesn't display correctly (it seems to erase itself): http://shevek.kenyonhill.com/perl/test.pl This is the contents of test.pl: --- #!/usr/bin/perl $|=1; print Content-Type: text/html\n\n; print hello worldP; # This line causes the error print ; --- Let me know if you need anything more.
Re: [mp2 Patch] BUG with mod_deflate and $|=1 (20014:Error string not specified)
We can see that mod_cgi bufferizes the output and sends it with Content-Length HTTP header (to mod_deflate). Indeed mod_perl generates chunked response. Finally we have the same result. I don't see any problem at this moment. Well, the problem is that I get this error in my error_log: [Mon Jul 21 14:18:55 2003] [error] 4297: ModPerl::RegistryBB: 20014:Error string not specified yet at /var/www/perl/test.pl line 6. Also, more important, the script seems to be terminating and/or any output following the 'print ' is not sent to the client as far as I can tell. You would probably wish to append your script with additional output after the empty string? Something like: #!/usr/bin/perl $|=1; print Content-Type: text/html\n\n; print hello worldP; # This line causes the error (?) print ; print hello againP; --- When I do this, the mod_perl variant of the script fails to print hello againP. mod_cgi prints everything just fine and gets no errors. I changed my test script to print a bunch of `date`'s http://shevek.kenyonhill.com/cgi/test.pl http://shevek.kenyonhill.com/perl/test.pl It may cause a problem for chunked output if mod_deflate does not care to keep internal buffer and check its size when flushing... I may not be understanding the output you sent or what you're saying, but I still don't follow why this would be a mod_deflate bug if mod_cgi with the same script has no problem.
Re: [mp2 Patch] BUG with mod_deflate and $|=1 (20014:Error string not specified)
At 04:24 AM 7/15/2003, Stas Bekman wrote: Philippe M. Chiasson wrote: On Thu, 2003-07-03 at 01:24, Bill Marrs wrote: This fixed the bug for me. Great! Will commit it in the near future. (Can't seem to access the cvs server right now, crappy internet cafe) -1, this is a wrong solution. print ; should flush just like it did in mod_perl 1.0 if $| == 1; Consider this script: print Content-type: text/plain\n\n; local $| = 0; print start\n\n; local $| = 1; print ; sleep 5; local $| = 0; print end\n; print , must immediately flush the buffered data, since $| has changed from 0 to 1. This may be naive, but might it not flush the output buffer at the 4th line (local $| = 1;)? ...or does the flush only happen when print is called. Having to call print seems cumbersome to do a flush, but maybe that's just the way Perl works? One thing that could help is if someone could take the time to write a test for this bug. Unfortunately I don't seem to be able to reproduce the problem, so I can't debug the problem. It could be a bug on the mod_deflate's behalf as well. Philippe, were you able to reproduce the problem with Bill's setup? I was writing a test, but couldn't get it to fail... may be because i was using 2.0.47. Bill, do you have the problem with the latest apache version? Did you see my note in the original post about this working fine under mod_cgi, but causing the 20014:Error only under mod_perl? This seemed to point the finger squarely at mod_perl. I could upgrade to 2.0.47, but it seems unlikely that it would fix this. Are you sure you're running a mod_perl without Philippe's fix (in Apache__RequestIO.h), I assumed he eventually checked it in. Also Bill, why do you have this setup: Location /perl AddOutputFilterByType DEFLATE text/* SetOutputFilter DEFLATE /Location why adding it twice? You need only the latter inside Location, or alternatively only the former outside Location if you want it to be set globally: http://httpd.apache.org/docs-2.0/mod/core.html#addoutputfilterbytype Ah, I misunderstood the mod_deflate docs. I think at the time, it didn't seem to work with just one of them in-place, so I added the other. *SHRUG* I can't say I'm a pro at Apache config files, I just tinker until it works. I assume this is irrelevant to the bug, though. -bill
Re: [mp2 Patch] BUG with mod_deflate and $|=1 (20014:Error string not specified)
I could upgrade to 2.0.47, but it seems unlikely that it would fix
this. Are you sure you're running a mod_perl without Philippe's fix (in
Apache__RequestIO.h), I assumed he eventually checked it in.
No, Philippe hasn't committed it, neither I have used it. If you can test
with 2.9.47 that will help. Otherwise I'll later try with .46 as well.
I just upgraded to Apache 2.0.47 and the latest CVS of mp2 and I'm
reproducing it, same as I originally reported.
I tried a few variations to see if I could find other factors, but didn't
have much luck. The server I'm testing on is live (runs a small site), so
I'm somewhat limited in what I can do. But, I tried removing mod_rewrite,
and mod_ssl from my server config and I still got the error in my tests.
I also tried varying the mod_deflate config (as you pointed out it was
redundant/wrong)...
Also Bill, why do you have this setup:
Location /perl
AddOutputFilterByType DEFLATE text/*
SetOutputFilter DEFLATE
/Location
why adding it twice? You need only the latter inside Location, or
alternatively only the former outside Location if you want it to be
set globally:
http://httpd.apache.org/docs-2.0/mod/core.html#addoutputfilterbytype
Ah, I misunderstood the mod_deflate docs. I think at the time, it didn't
seem to work with just one of them in-place, so I added the
other. *SHRUG* I can't say I'm a pro at Apache config files, I just
tinker until it works. I assume this is irrelevant to the bug, though.
I'm not sure if your config doesn't insert the filter twice. Need to check
whether SetOutputFilter overrides AddOutputFilterByType as well.
In my config, I have a deflate log:
DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio
LogFormat '%r %{outstream}n/%{instream}n (%{ratio}n%%) %{User-agent}i'
deflate
CustomLog /var/log/httpd/deflate_log deflate
I was using this to verify that deflate was active or not. What I found
was that using AddOutputFilterByType DEFLATE text/* doesn't seem to
activate deflate, no matter where I put it (in a Location or at the
top-level of the config). My deflate log shows that no compression is
occurring. SetOutputFilter DEFLATE does activate deflate globally (for
all locations), no matter where I put it. I'm afraid I don't find the
mod_deflate docs very clear on placement.
When you were trying to reproduce, I don't know if you used my (redundant)
config before, but if you only used AddOutputFilterByType DEFLATE text/*,
and not SetOutputFilter DEFLATE, that might explain why you were not able
to reproduce this, I don't think that activates deflate (at least that's
what my deflate_log shows for me).
Otherwise, if you're still not reproducing this, I would assume there's
some difference in the way we are building apache or our apache config
files or perl?
Here's my Apache Configure line:
./configure --enable-modules=all --enable-mods-shared=all --enable-deflate
--with-mpm=prefork --enable-rewrite --enable-ssl
Here are the Modules I load:
LoadModule cgi_module modules/mod_cgi.so
LoadModule access_module modules/mod_access.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule asis_module modules/mod_asis.so
LoadModule info_module modules/mod_info.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule dir_module modules/mod_dir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule perl_module modules/mod_perl.so
LoadModule ssl_module modules/mod_ssl.so
mod_cgi, mod_ssl and mod_rewrite seem to be innocent (when I removed them,
it still failed), but maybe you're using a module that I'm not and this is
causing your config to work better? I think that I'm loading less modules
than what is typical, I removed as many as I could to reduce memory usage a
while ago.
*SHRUG*
One thing that may be relevant is tracking down where the 20014:Error is
coming from. Web searching seems to associate it with Berkeley DB code,
which seems odd to me, is Apache using Berkeley DB code somewhere (via a
filter?)?
This isn't a very important issue for me. I have a decent workaround
(print instead of ), plus I'm not really able to use mod_deflate that
much anyway because it puts too much load on my server (mod_gzip with
Apache 1.3 worked better for me).
Thanks
-bill
Re: Memoize.pm and mod_perl
I don't know anything about Memoize, but perhap db-level caching would work for you? If you user MySQL, Mysql 4.0.1 and beyond has Query Caching capabilities built into it. http://www.mysql.com/documentation/mysql/bychapter/manual_Reference.html#Query_Cache -=bill
[mp2] BUG with mod_deflate and $|=1 (20014:Error string not specified)
When I use Apache 2.0.46, mod_deflate with mod_perl-1.99_09 (or the latest mod_perl-2.0 CVS), perl buffering is off ($|=1), and my perl script prints nothing (e.g. 'print ;'), I get the following error: [Wed Jul 02 10:10:00 2003] [error] 19513: ModPerl::RegistryBB: 20014:Error string not specified yet at /var/www/perl/test.pl line 6. If I switch to running the script under mod_cgi or if I remove the $|=1; line, I do not get an error. Here is my script: #!/usr/bin/perl $|=1; print Content-Type: text/html\n\n; print hello worldP; # This line causes the error print ; httpd.conf snipit: Alias /perl/ /var/www/perl/ Location /perl AddOutputFilterByType DEFLATE text/* SetOutputFilter DEFLATE AllowOverride None SetHandler perl-script PerlHandlerModPerl::RegistryBB PerlSendHeader On Options+ExecCGI /Location I've worked-around this problem by changing my print to print . It's not a major issue for me, I'm just letting you know. Let me know if you need any more info. -bill
Re: [mp2 Patch] BUG with mod_deflate and $|=1 (20014:Error string not specified)
This fixed the bug for me.
At 10:48 AM 7/2/2003, you wrote:
#define mpxs_output_flush(r, rcfg) \
/* if ($|) */ \
-if (IoFLUSH(PL_defoutgv)) { \
+if (bytes 0 IoFLUSH(PL_defoutgv)) { \
MP_FAILURE_CROAK(modperl_wbucket_flush(rcfg-wbucket, TRUE)); \
}
Best compression for mod_perl application?
I used to use mod_gzip under Apache 1.3, and it worked great, saving my over 50% of my bandwidth for my mod_perl generated pages. But, it appears that mod_gzip doesn't work with Apache 2. Apache 2 has a built-in mod_deflate, but I've had some trouble with it (seemed to cause a load spike on my server + errors if I print ). I recall there used to be alternatives to mod_gzip out there, but I'm not sure if they apply to Apache 2. Are any of you use compression on your mod_perl pages? Do you recommend any compression schemes for the Apache 2/mp2 environment? TIA, -=bill
Re: Best compression for mod_perl application?
That would be of my real interest to know as many details of Bill's
experience with mod_deflate as he can provide.
Since I posted my first message, I've been snooping around the 'net to find
more info on mod_gzip and mod_deflate. Here's what I came up with:
The general recommendation seems to be migration from mod_gzip to
mod_deflate when you switch to Apache 2.0. mod_gzip seems to have lost
most of its support going forward while mod_deflate is part of the Apache
source code and has active development.
There is a Apache 2.0 compatible version of mod_gzip, here:
http://www.gknw.de/development/apache/httpd-2.0/unix/modules/
When I tried it, it didn't work for me. It caused my site to spit out
blank pages and garbage. I had used my old Apache 1.3 mod_gzip config with
it. I read that there's some odd timing issues where the Apache 2.0
version of mod_gzip branched a long time ago and thus doesn't have some of
the modern mod_gzip 1.3.x features. I didn't get config errors, though,
just blanks and garbage. So, I decided to back away slowly for mod_gzip on
Apache 2.0. There is more discussion of it here:
http://freshmeat.net/projects/mod_gzip/?topic_id=90
There a good mod_gzip info page here, though little is said about a 2.0
version:
http://www.schroepl.net/projekte/mod_gzip/index.htm
The mod_gzip mailing list has some good info. Here's a 26 Jun 2003 post by
someone who seems to know well what's going on (I think the author of the
above page):
Subject: [Mod_gzip] gzip vs deflate on Apache
http://lists.over.net/pipermail/mod_gzip/2003-June/007130.html
So, I decided to try harder to move ahead with mod_deflate. I'm using a
built from scratch, Apache/2.0.46 mod_perl_1.99_09. Work is being done on
mod_deflate, some recent directives has been added (I hear). One of which
is DeflateCompressionLevel. Along with this addition in 2.0.44 came a
better default for this compression level. It's now 6, the same thing that
gzip and zlib uses by default. Apparently, it had been 1 before that,
which is fast but doesn't compress very well. There's some discussion of
this here:
http://www.webcompression.org/deflate-compress.html
My own personal experience with mod_deflate (in Apache/2.0.46) is that it
tends to spike my server's load. My server (gametz.com) is dual 800Mhz,
1.5GB ram, Linux, doing about 70K pages/day. Last night, I happened to be
watching it while the load jumped up a few points during my site's prime
time, so I pulled mod_deflate out of the config file and that fixed it.
So, today, I'm trying a lower DeflateCompressionLevel. I'm using 4 now
(instead of the default 6). This seems better, though the load is still a
little higher than it should be and I'm not quote at prime time
yet. Still, I am getting decent compression. I'm going to keep an eye on
it, I suspect I'll be at 3 later this evening.
I never had any trouble with load when I used mod_gzip and Apache 1.3.
The other odd problem I got was that if anywhere in my perl code I printed
nothing (e.g. print or $foo=;print $foo), I'd get this error:
error: 20014:Error string not specified yet at /my/perl/code.pl line 123
This error was both blurted to the error_log and to the web page (screwing
up the page and truncating further output).
I changed my code to print instead of (HTML ignores extra
white-space, so no biggie), and the errors all went away. So, I see this
as an annoyance more than a serious bug.
I really should try to tell the author of mod_deflate about these problems.
Here's the config I'm using for mod_deflate:
#
## Deflate
#
LoadModule deflate_module modules/mod_deflate.so
AddOutputFilterByType DEFLATE text/*
SetOutputFilter DEFLATE
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
DeflateBufferSize 8096
# DeflateCompressionLevel 6
DeflateCompressionLevel 4
DeflateMemLevel 9
DeflateWindowSize 15
DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio
LogFormat '%r %{outstream}n/%{instream}n (%{ratio}n%%) %{User-agent}i'
defl\
ate
CustomLog /var/log/httpd/deflate_log deflate
All of which I cribbed from the Apache 2.0 manual:
http://httpd.apache.org/docs-2.0/mod/mod_deflate.html
I sort of got forced into upgrading to Apache 2.0/mp2/etc. by RedHat. They
announced they would pull support for old releases (all that used Apache
1.3) by the end of the year. Apparently, this may be an intentional
(evil?) business move by them to motivate more customers to move to their
Enterprise OS (which is very expensive, but has more stable software like
good old Apache 1.3 mp1).
I did try to go back at one point, builing Apache 1.3 from source, but it
had some other problem (maybe, because I used Perl 5.8.0?). But, then I
waffled and decided there's also a lot of value in staying current. So,
I'm back to 2.0 land, and I'm surviving so far.
-=bill
Re: Best compression for mod_perl application?
1. Are you using any Cascaded Stile Sheets and/of JavaScript libraries linked to your main web pages? I'm not, but... I think mod_defalte's hook is after all that is processed, so it's not especially relevant. 2. If yes, how do you turn compression off for those files in case of Netscape-4 originated request? http://httpd.apache.org/docs-2.0/mod/mod_deflate.html says to use this sort of thing: # Netscape 4.x has some problems... BrowserMatch ^Mozilla/4 gzip-only-text/html # Netscape 4.06-4.08 have some more problems BrowserMatch ^Mozilla/4\.0[678] no-gzip # MSIE masquerades as Netscape, but it is fine BrowserMatch \bMSIE !no-gzip !gzip-only-text/html # Don't compress images SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary # Make sure proxies don't deliver the wrong content Header append Vary User-Agent env=!dont-vary
submit input truncation
I recently moved to Apache 2 + mod_perl2 and things have been working fairly well. But, I'm getting an intermittent problem with POSTs where the input data is being truncated. This causes havoc, especially in my forum system. I use CGI.pm, mod_perl 2, Apache 2, and the standard RH9 Perl (which is threaded) - I think that's all the relevant stuff. Has anyone else seen this? Is there some fix for it? I'm not even sure where the problem is yet. Thanks in Advance. -bill
Re: submit input truncation
At 04:48 PM 6/25/2003, Haroon Rafique wrote: I think you should take a look at the list archives for this thread: http://mathforum.org/epigone/modperl/bindondrei Thank You! This is very likely the problem I'm having. Secondly, you should also post the version numbers for all of the above-mentioned packages. As mentioned in one of the emails this was fixed in mod_perl 1.99_09. I'm using what comes with RH9: httpd-2.0.40-21.3 mod_perl-1.99_07-5 perl-5.8.0-88 plus, this which is the latest from cpan: CGI.pm-2.97 Clearly, the first thing I should try is mod_perl 1.99_09. Now, I'm wondering if someone has made a RH9 friendly mod_perl-1.99_09 rpm... As always, I'm trying desperately to avoid rebuilding apache/perl/mod_perl/etc. on my systems. -bill
mod_perl-1.99_09 for Redhat 9
I'm looking for a Redhat 9 compatible mod_perl-1.99_09 rpm. If anyone has one or knows where I can get one, let me know. Thanks, -bill p.s. I did find a Rawhide (bleeding edge Red Hat release, I think) mod_perl-1.99_09, but it doesn't seem to be compatible (I got an error from Apache).
2nd perl install?
If I can't find the mod_perl rpm I need, it's looking like I might need to build it from source. I believe this would mean also building Perl and Apache from source. Now, I have a redhat-installed Perl already. From past discussions, the best idea seemed to be to create a 2nd perl installation for Apache/mod_perl to use, perhaps located in /usr/local. The thing I'm fuzzy on is how to tell my mod_perl programs to use this mod_perl. Is it enough to have the LoadModule line point to my build-from-scratch mod_perl.so? Also, if I want to use this 2nd perl for scripts, is it as simple as changing the top line to #!/usr/local/bin/perl. Do I have to worry about library pathes or environment variables? I assume I'll need to keep separate lib/perl5 trees as well and I'll need to install the various cpan modules I use twice. Any tips would be appreciated. -bill
Re: Apache::VMonitor not showing requests (might be Apache::Scoreboard problem)
Maybe adding this to your httpd.conf will help: ExtendedStatus On ?
Re: 'Pinning' the root apache process in memory with mlockall
At 10:53 PM 3/22/2002, Stas Bekman wrote: top and libgtop use the same source of information, so it has nothing to do with these tools. 'top' has the ability to display SWAP on a per-process basis (you have to change the defaults to see it, but it's there). I didn't find this per-process SWAP value in Gtop.pm anywhere. If GTop.pm had it, I could fix GTopLimit's bug. -bill
Re: 'Pinning' the root apache process in memory with mlockall
Stas, Thanks for tracking that down. So, the problem is our tools. For me, that's GTopLimit (but also SizeLimit). I would think it must be possible to cajole these two into realizing their error. top seems to know how much a process has swapped. If GTopLimit could know that, the number could be subtracted from the total used in calculating the amount of sharing (and the new unshared), then this bug would be resolved, right? I looked, but didn't see anything in Gtop.pm that gives swap per process, though. So, it's not going to be easy. I guess I'll turn of me deswapper... ...and GTopLimit as well. for now... hmm, maybe I could just avoid using the share-related trigger values in GTopLimit, and just use the SIZE one. That would be an acceptable compromise, though not the best. -bill
RE: loss of shared memory in parent httpd
The reason turning off swap works is because it forces the memory from the parent process that was swapped out to be swapped back in. It will not fix those processes that have been sired after the shared memory loss, as of Linux 2.2.15 and Solaris 2.6. (I have not checked since then for behavior in this regard, nor have I checked on other OSes.) In my case, I'm using Linux 2.4.17, when I turn off swap and turn it back on again, it restores the shared memory of both the parent and the children Apache processes. This seems counter-intuitive, as it would seem the kernel memory manager would have to bend over backwards to accomplish this re-binding of the swapped-out shared memory pages. Thus, it leads ones to wonder if some of our assumptions or tools used to monitor memory are inaccurate or we're misinterpreting them. -bill
RE: loss of shared memory in parent httpd
It's copy-on-write. The swap is a write-to-disk. There's no such thing as sharing memory between one process on disk(/swap) and another in memory. agreed. What's interesting is that if I turn swap off and back on again, the sharing is restored! So, now I'm tempted to run a crontab every 30 minutes that turns the swap off and on again, just to keep the httpds shared. No Apache restart required! Seems like a crazy thing to do, though. You'll also want to look into tuning your paging algorithm. Yeah... I'll look into it. If I had a way to tell the kernel to never swap out any httpd process, that would be a great solution. The kernel is making a bad choice here. By swapping, it triggers more memory usage because sharing removed on the httpd process group (thus multiplied)... I've got MaxClients down to 8 now and it's still happening. I think my best course of action may be a crontab swap flusher. -bill
Re: Apache and Perl with Virtual Host
At 04:02 AM 3/14/2002, Matt Phelps wrote: Forgive me if I'm posting to the wrong group. Ive got apache 1.3.22 running several virtual webs. I can get perl scripts to run under the default web but not in the others. All the webs point to the same script folder. If I try to run the script under a virtual web, all I get is text display. Any help would be great. Well, I use mod_perl with VituralHosts... My config looks something like: VirtualHost gametz.com ServerAdmin [EMAIL PROTECTED] DocumentRoot /home/tz/html ServerName gametz.com DirectoryIndex /perl/gametz.pl # The live area Alias /perl/ /home/tz/perl/ Location /perl AllowOverride None SetHandler perl-script PerlHandler Apache::RegistryBB PerlSendHeader On Options+ExecCGI /Location /VirtualHost VirtualHost surveycentral.org ServerAdmin [EMAIL PROTECTED] DocumentRoot /projects/web/survey-central ServerName surveycentral.org DirectoryIndex /perl/survey.pl Alias /perl/ /projects/web/survey-central/perl/ Location /perl SetHandler perl-script PerlHandlerApache::RegistryBB PerlSendHeader On Options+ExecCGI /Location /VirtualHost
Re: [OT]RE: loss of shared memory in parent httpd
How is it even remotely possible that turning off swap restores memory shared between processes? Is the Linux kernel going from process to process comparing pages of memory as they re-enter RAM? Oh, those two look identical, they'll get shared? This is a good point. I really have no clue how the kernel deals with swapping/sharing, so I can only speculate. I could imagine that it's possible for it to do this, if the pages are marked properly, they could be restored. But, I'll admit, it seems unlikely. ...and, I had this thought before. Maybe this apparent loss of shared memory is an illusion. It appears to make the amount of memory that the httpds use grow very high, but perhaps it is a kind of shared-swap, and thus the calculation I'm using to determine overall memory usage would need to also factor out swap. ...in which case, there's no problem at all. But, I do see an albeit qualitative performance increase and CPU load lowering when I get the httpds to stay shared (and unswapped). So, I think it does matter. Though, if you think about it, it sort of makes sense. Some portion of the shared part of the httpd is also not being used much, so it gets swapped out to disk. But, if those pages really aren't being used, then there shouldn't be a performance hit. If they are being used, then they'd get swapped back in. ...which sort of disproves my qualitative reasoning that swap/unshared is bad. my head hurts, maybe I should join a kernel mailing list and see is someone there can help me (and if I can understand them). -bill
Re: loss of shared memory in parent httpd
I just wanted to mention that the theory that my loss of shared memory in the parent is related to swapping seems to be correct. When the lack of sharing occurs, it is correlated with my httpd processes showing a SWAP (from top/ps) of 7.5MB, which is roughly equal to the amount of sharing that I lose. I've been lowering my MaxClients setting (from 25 to 10, so far) in hopes of finding a new balance where SWAP is not used, and more RAM is on order. Thanks -bill
loss of shared memory in parent httpd
I'm a heavy mod_perl user, running 3 sites as virtual servers, all with lots of custom Perl code. My httpd's are huge(~50mb), but with the help of a startup file I'm able to get them sharing most of their memory(~43). With the help of GTopLimit, I'm able to keep the memory usage under control. But... recently, something happened, and things have changed. After some random amount of time (1 to 40 minutes or so, under load), the parent httpd suddenly loses about 7-10mb of share between it and any new child it spawns. As you can imagine, the memory footprint of my httpds skyrockets and the delicate balance I set up is disturbed. Also, GTopLimit is no help in this case - it actually causes flailing because each new child starts with memory sharing that is out of bounds and is thus killed very quickly. Restarting Apache resets the memory usage and restores the server to smooth operation. Until, it happens again. Using GTop() to get the shared memory of each child before and after running my perl for each page load showed that it wasn't my code causing the jump, but suddenly the child, after having a good amount of shared memory in use, loses a 10MB chunk and from then on the other children follow suit. So, something I did on the server (I'm always doing stuff!) has caused this change to happen, but I've been pulling my hair out for days trying to track it down. I am now getting desperate. One of the recent things I did was to run tux (another web server) to serve my images, but I don't see how that could have any effect on this. If anyone has any ideas what might cause the httpd parent (and new children) to lose a big chunk of shared memory between them, please let me know. Thanks in advance, -bill
Re: loss of shared memory in parent httpd
Thanks for all the great advice. A number of you indicated that it's likely due to my apache processes being partially swapped to disk. That seems likely to me. I haven't had a chance to prove that point, but when it does it again and I'm around, I plan to test it with free/top (top has a SWAP column which should show if my apaches are swapped out at all). I am in the process of getting a memory upgrade, so that should ease this problem. Meanwhile, I can set MaxClients lower and see if that keeps me out of trouble as well. I suspect adding the tux server disrupted the balance I had (apparently, I was tuned pretty close to my memory limits!) Yes, I am running on Linux... One more piece of advice: I find it easier to tune memory control with a single parameter. Setting up a maximum size and a minumum shared size is not as effective as setting up a maximum *UNSHARED* size. After all, it's the amount of real memory being used by each child that you care about, right? Apache::SizeLimit has this now, and it would be easy to add to GTopLimit (it's just $SIZE - $SHARED). Doing it this way helps avoid unnecessary process turnover. I agree. For me, with my ever more bloated Perl code, I find this unshared number to be easier to keep a lid on. I keep my apache children under 10MB each unshared as you say. That number is more stable that the SIZE/SHARED numbers that GTopLimmit offers. But, I have the GTopLimit sources, so I plan to tweak them to allow for an unshared setting. I think I bugged Stas about this a year ago and he had a reason why I was wrong to think this way, but I never understood it. -bill
trouble with GTop and
When I install the recent Redhat 7.2 updates for glibc: glibc-2.2.4-19.3.i386.rpm glibc-common-2.2.4-19.3.i386.rpm glibc-devel-2.2.4-19.3.i386.rpm It breaks my Apache GTop-based Perl modules, in a way that I don't understand. Here is are the error messages from my httpd/error_log: [Tue Mar 12 15:44:37 2002] [error] Can't load '/usr/lib/perl5/site_perl/5.6.0/i386-linux/auto/GTop/GTop.so' for module GTop: /usr/lib/libgdbm.so.2: undefined symbol: gdbm_errno at /usr/lib/perl5/5.6.1/i386-linux/DynaLoader.pm line 206. at /usr/lib/perl5/site_perl/5.6.0/i386-linux/GTop.pm line 12 Compilation failed in require at /usr/lib/perl5/site_perl/5.6.0/Apache/GTopLimit.pm line 144. BEGIN failed--compilation aborted at /usr/lib/perl5/site_perl/5.6.0/Apache/GTopLimit.pm line 144. Compilation failed in require at /home/httpd/startup.pl line 32. BEGIN failed--compilation aborted at /home/httpd/startup.pl line 32. Compilation failed in require at (eval 1) line 1. Syntax error on line 1017 of /etc/httpd/conf/httpd.conf: Can't load '/usr/lib/perl5/site_perl/5.6.0/i386-linux/auto/GTop/GTop.so' for module GTop: /usr/lib/libgdbm.so.2: undefined symbol: gdbm_errno at /usr/lib/perl5/5.6.1/i386-linux/DynaLoader.pm line 206. at /usr/lib/perl5/site_perl/5.6.0/i386-linux/GTop.pm line 12 Compilation failed in require at /usr/lib/perl5/site_perl/5.6.0/Apache/GTopLimit.pm line 144. BEGIN failed--compilation aborted at /usr/lib/perl5/site_perl/5.6.0/Apache/GTopLimit.pm line 144. Compilation failed in require at /home/httpd/startup.pl line 32. BEGIN failed--compilation aborted at /home/httpd/startup.pl line 32. Compilation failed in require at (eval 1) line 1. Anyone have a clue about what I'd need to do to get this working? I am able to force the old glibc rpms back on to fix the problem. The previous versions that work for me are: glibc-common-2.2.4-13 glibc-devel-2.2.4-13 glibc-2.2.4-13 -bill
Re: PerlModule/Require didn't work ??
This is the same problem I've been struggling with. What has fixed it for me is building Apache and mod_perl from the tar archives and linking Apache statically (no .so's). I haven't investigated much further, but even after I built Perl, Apache, and mod_perl by hand it still happened. Only when I switch to a non-DSO Apache did it work for me. -bill At 03:39 PM 12/16/99 +, [EMAIL PROTECTED] wrote: hi, I was wondering why the PerlModule and perlRequire didn't work for me. I have a fresh installation of : apache 1.3.9 + mod_perl 1.21 (that is bundled in RedHat6.1), what was my surprise to see that when I try to use something like : PerlRequire .../startup.pl or PerlModule Apache::DBI Apache::ASP the Apache won't start. When I start the apache i.e. : /etc/rc.d/init.d/httpd start I get .. [OK] , but nothing started nothing even in the error_log :"( (LogLevel debug). I missed something or WHAT ?!? I thought that it was 'cause now by default if Apache find suEXEC it runs in suEXEC-mode, but I moved suEXEC and it dosen't use it then. Another surprice was that if I go .../perl-status .. Loaded Modules ... I see CGI there, but I didn't point to it in .conf files.?!?! Thanx alot in advance = iVAN [EMAIL PROTECTED] =
Re: silent failure when I try to use PerlRequire in httpd.conf
At 10:14 AM 12/11/99 -0800, Cliff Rayman wrote: Stas Bekman wrote: On Sat, 11 Dec 1999, Bill Marrs wrote: Hi, I've been using mod_perl successfully albeit minimally for a couple months now. I'm trying to tune performance and memory usage by consolidating some common Perl "use" directives in a PerlRequire statement. However, when I add this line to my Apache config file (/etc/httpd/conf/httpd.conf), the server fails to come up - and also give no indication that I can find of a problem. So, I don't know what's going wrong. Here's what I'm adding to my httpd.conf file: PerlRequire /home/ugtz/startup.pl I have the usual stuff in my /home/ugtz/startup.pl file, actually I tried commenting everything but the #!/usr/local/bin/perl and the "1;" at the end, and it still failed. So, I don't think the contents of the file are at fault. My httpd error log doesn't say anything is wrong, it actually doesn't log anything - like it never got that far... make sure no server is running on port 80 or use an alternate config with an alternate port if you are on a production server. cd /usr/local/apache/bin # where ever your binary is gdb httpd set args -X -f /path/to/alternate/serverconfig_ifneeded.conf run it should blow up! then, bt this will show you a backtrace of what was happening when it blew up. post that to the list and we may be able to come up with some better guesses as to the problem. make sure to include versions of apache, mod perl and perl. I tried this, but the server is just happily and quietly exiting - no blow up, no indication of why it's failing (unless I don't know where to look). [root@ugtz /root]# cd /usr/sbin/ [root@ugtz sbin]# gdb httpd GNU gdb 4.17.0.11 with Linux support Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux"... (no debugging symbols found)... (gdb) run Starting program: /usr/sbin/httpd Program exited normally. (gdb) quit # httpd -V Server version: Apache/1.3.9 (Unix) (Red Hat/Linux) Server built: Sep 21 1999 10:46:27 Server's Module Magic Number: 19990320:6 Server compiled with -D HAVE_MMAP -D HAVE_SHMGET -D USE_SHMGET_SCOREBOARD -D USE_MMAP_FILES -D USE_FCNTL_SERIALIZED_ACCEPT -D HTTPD_ROOT="/usr" -D SUEXEC_BIN="/usr/sbin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="/var/run/httpd.scoreboard" -D DEFAULT_LOCKFILE="/var/run/httpd.lock" -D DEFAULT_XFERLOG="/var/log/httpd/access_log" -D DEFAULT_ERRORLOG="/var/log/httpd/error_log" -D TYPES_CONFIG_FILE="/etc/httpd/conf/mime.types" -D SERVER_CONFIG_FILE="/etc/httpd/conf/httpd.conf" -D ACCESS_CONFIG_FILE="/etc/httpd/conf/access.conf" -D RESOURCE_CONFIG_FILE="/etc/httpd/conf/srm.conf" I believe the mod_perl version I'm running is: mod_perl/1.21 (I'm not sure how to officially get this info) # httpd -l Compiled-in modules: http_core.c mod_so.c
Re: silent failure when I try to use PerlRequire in httpd.conf
I'm sure there are no other servers. I'm actually stopping my live server to try this. I stop it, change the config file and start it - and it doesn't come up and no server is running. So, I put it back and start it and its OK again. I tried "set args -X" in gdb, and the result was that the server worked fine. So, I guess the problem is related to starting the child servers. Prakash Kailasa's reply talks about what is probably the same thing that is happening to me. He said: I started apache with strace and the output shoed me that apache is blowing up when mod_perl is trying to load the first shared object (.so file), which in my case happened to be IO.so. This only happens in the first forked child process. The root apache process loads everything just fine. But I think when the root process gets a signal that the child has died, it is also dying silently. I believe that this is the same thing that is happening to me. Cliff - are you using PerlRequire successfully? What version of Apache and mod_perl are you using? Prakash Kailasa also said: I have accidentally found a workaround. It may or may not work for you. - Comment the PerlRequire line. - Run: "apachectl stop" and then "apachectl start" - Uncomment the PerlRequire line. - Run: "apachectl graceful" I don't have an "apachectl" installed, I use /etc/rc.d/init.d/httpd, and it doesn't have "graceful" as an option (only stop|start|restart|status|reload). Maybe "graceful" is the same as "reload"? I'll try that... When I use reload in this way it seems to do what I wanted. My httpd processes seem to be using more memory to start (from the Perl "use" lines pre-loading stuff) - but as I understand it, that's probably shared memory (the whole point of using PerlRequire). So, are some people successfully using PerlRequire out there? What versions of Apache and mod_perl are you using? What OS? ...something is wrong for a couple of us out here. -bill At 02:00 PM 12/13/99 , Cliff Rayman wrote: are you sure that there are no other webservers currently running and on port 80? do the 'set args' gdb command with at least the -X parameter before executing run. when you start apache and it blows up, is apache using any other command line arguments such as -DSSL? if so, make sure you use that on the set args line as well so you are running the program with the same enviroment in gdb as when you run with apachectl. cliff rayman genwax.com Bill Marrs wrote: At 10:14 AM 12/11/99 -0800, Cliff Rayman wrote: Stas Bekman wrote: On Sat, 11 Dec 1999, Bill Marrs wrote: Hi, I've been using mod_perl successfully albeit minimally for a couple months now. I'm trying to tune performance and memory usage by consolidating some common Perl "use" directives in a PerlRequire statement. However, when I add this line to my Apache config file (/etc/httpd/conf/httpd.conf), the server fails to come up - and also give no indication that I can find of a problem. So, I don't know what's going wrong. Here's what I'm adding to my httpd.conf file: PerlRequire /home/ugtz/startup.pl I have the usual stuff in my /home/ugtz/startup.pl file, actually I tried commenting everything but the #!/usr/local/bin/perl and the "1;" at the end, and it still failed. So, I don't think the contents of the file are at fault. My httpd error log doesn't say anything is wrong, it actually doesn't log anything - like it never got that far... make sure no server is running on port 80 or use an alternate config with an alternate port if you are on a production server. cd /usr/local/apache/bin # where ever your binary is gdb httpd set args -X -f /path/to/alternate/serverconfig_ifneeded.conf run it should blow up! then, bt this will show you a backtrace of what was happening when it blew up. post that to the list and we may be able to come up with some better guesses as to the problem. make sure to include versions of apache, mod perl and perl. I tried this, but the server is just happily and quietly exiting - no blow up, no indication of why it's failing (unless I don't know where to look). [root@ugtz /root]# cd /usr/sbin/ [root@ugtz sbin]# gdb httpd GNU gdb 4.17.0.11 with Linux support Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux"... (no debugging symbols found)... (gdb) run Starting program: /usr/sbin/httpd Pro
silent failure when I try to use PerlRequire in httpd.conf
Hi, I've been using mod_perl successfully albeit minimally for a couple months now. I'm trying to tune performance and memory usage by consolidating some common Perl "use" directives in a PerlRequire statement. However, when I add this line to my Apache config file (/etc/httpd/conf/httpd.conf), the server fails to come up - and also give no indication that I can find of a problem. So, I don't know what's going wrong. Here's what I'm adding to my httpd.conf file: PerlRequire /home/ugtz/startup.pl I have the usual stuff in my /home/ugtz/startup.pl file, actually I tried commenting everything but the #!/usr/local/bin/perl and the "1;" at the end, and it still failed. So, I don't think the contents of the file are at fault. My httpd error log doesn't say anything is wrong, it actually doesn't log anything - like it never got that far... I do: % /etc/rc.d/init.d/httpd start Starting httpd: [ OK ] % ps -ef |grep httpd [no httpd processes] % tail /etc/httpd/logs/error_log [ nothing new in log ] Does anyone have any clues for me to track down on this, I'm stumped. -bill
Sending requests for *.html files to a mod_perl script
Hello,
I run a couple web sites that use mod_perl to generate every page. The
site is working well, but the CGI syntax of the links I use (e.g. pages
looking like "http://host/myscript.pl?name=valuesession=foo") has some
drawbacks.
My understanding is that search engines tend to not follow such links. So,
my web sites have little coverage by search engines, which is too bad
because there's lots of good content behind those CGI links.
There would also be the added benefit of supporting some old links to flat
html pages that I've since moved into the mod_perl script.
There are probably a number of ways to solve this problem, and if you can
think of a better way to do this, let me know. I'd like to avoid
pre-generating pages though as my content is fairly dynamic.
What I'd like to do now, is have my site (when in [session-less]
guest-mode; how search engines would see it) use URLs that don't have the
's and ?'s in them, and instead have paths that look more normal (e.g.
"/users/bill.html" instead of "/myscript.pl?user=bill").
If I can change a request for /users/bill.html into a call to my mod_perl
script with extra path_info of ("/users/bill.html"), my script could
examine the path_info and figure out that it should load the "user=bill" page.
I figure this is an Apache config file issue, but given that I'm using
mod_perl, that config may be trickier.
I've been reading the docs for mod_rewrite, but frankly I'm confused. I'm
not sure this is the right direction to go in either.
I'm using very basic mod_perl now, just Apache::Registry. But, I'm very
happy with the performance boost it has given me.
Have any of you done this sort of thing?
Here's my VirtualHost snippet from the httpd.conf file (in case that matters):
VirtualHost ugtz.com
DocumentRoot /home/ugtz/html
ServerName ugtz.com
DirectoryIndex /perl/ugtz.pl
Alias /perl/ /home/ugtz/perl/
Location /perl
SetHandler perl-script
PerlHandlerApache::Registry
PerlSendHeader On
Options+ExecCGI
/Location
/VirtualHost
Any help would be appreciated, feel free to just point me elsewhere if this
isn't appropriate on the modperl mailing list (I'm new to the list).
Bill Marrs
Sending requests for *.html files to a mod_perl script
Hello,
I run a couple web sites that use mod_perl to generate every page. The
site is working well, but the CGI syntax of the links I use (e.g. pages
looking like "http://host/myscript.pl?name=valuesession=foo") has some
drawbacks.
My understanding is that search engines tend to not follow such links. So,
my web sites have little coverage by search engines, which is too bad
because there's lots of good content behind those CGI links.
There would also be the added benefit of supporting some old links to flat
html pages that I've since moved into the mod_perl script.
There are probably a number of ways to solve this problem, and if you can
think of a better way to do this, let me know. I'd like to avoid
pre-generating pages though as my content is fairly dynamic.
What I'd like to do now, is have my site (when in [session-less]
guest-mode; how search engines would see it) use URLs that don't have the
's and ?'s in them, and instead have paths that look more normal (e.g.
"/users/bill.html" instead of "/myscript.pl?user=bill").
If I can change a request for /users/bill.html into a call to my mod_perl
script with extra path_info of ("/users/bill.html"), my script could
examine the path_info and figure out that it should load the "user=bill" page.
I figure this is an Apache config file issue, but given that I'm using
mod_perl, that config may be trickier.
I've been reading the docs for mod_rewrite, but frankly I'm confused. I'm
not sure this is the right direction to go in either.
I'm using very basic mod_perl now, just Apache::Registry. But, I'm very
happy with the performance boost it has given me.
Have any of you done this sort of thing?
Here's my VirtualHost snippet from the httpd.conf file (in case that matters):
VirtualHost ugtz.com
DocumentRoot /home/ugtz/html
ServerName ugtz.com
DirectoryIndex /perl/ugtz.pl
Alias /perl/ /home/ugtz/perl/
Location /perl
SetHandler perl-script
PerlHandlerApache::Registry
PerlSendHeader On
Options+ExecCGI
/Location
/VirtualHost
Any help would be appreciated, feel free to just point me elsewhere if this
isn't appropriate on the modperl mailing list (I'm new to the list).
Bill Marrs
