Re: News regarding mod_perl returning empty pages
I think that I finally have a clue as to why those empty pages were returned. I have perlbal as front, and it was set to maintain persistent connections with the apache backend listening on localhost. I also have some configuration of apache that would essentially deny access to certain user agents that were known to abuse my site before, such as wget, RapidDownloader, ia_archiver, and a few more. (I know that wget can be run with diff. user agents. The people who run wget against my site usually are clueless. If they had enough clue to change user agent, they would also realize that my site is dynamic and very deep. ) So, I am conjecturing, what happened was that perlbal selected some worker for a persistent connection, and the first user agent to connect was one of those bad guys. Then the worker would reject all subsequent queries coming on the same TCP connection, which would have the unfortunate effect that all queries were rejected. Since the time that I disabled persistent connections (which should not matter too much on localhost), I have never had this problem where my server would start returning empty pages. I also verified with ab, that I have not had any performance hit due to that. The stock Ubuntu Hardy mod_perl is solid as a rock, now. I want to thank everyone. It was a tough one because regular stress testing would not trigger it. As for those spiders, some of them are sort of legitimate, like ia_archiver (which I thought was a rogue bot at some point as it would not provide a webpage), but their expense in terms of traffic is not worth the benefit that I get from them. Igor
Re: News regarding mod_perl returning empty pages
On Tue, Aug 25, 2009 at 7:20 PM, Igor Chudovichu...@gmail.com wrote: My new perlbal.conf is included: REATE POOL dynamic pool dynamic add 127.0.0.1:10080 CREATE SERVICE balancer SET listen = 0.0.0.0:80 SET role = reverse_proxy SET pool = dynamic SET persist_client = on SET persist_backend = off SET verify_backend = on With verify_backend = on, perlbal makes an OPTIONS request to the mod_perl server. That request usually returns an empty 200, which in other scenarios could be viewed as a blank page. How perlbal would send that back to the client is something I don't know. But I would consider disabling that option as it doubles the number of total requests to the backend mod_perl server. That's just my opinion though, changing that failure could cause other failure modes to surface. If you want to use that option, you should write a mod_perl handler that answers to a verify_backend url (see the perlbal docs) and returns a few characters of text back to perlbal. If you see those characters in your blank screens - that's the problem. Although I'm skeptical that is what is happening.
Re: News regarding mod_perl returning empty pages
Fred, thanks. I am afraid that verify_backend is more of an expensive distraction, than something actually useful. At this point in time I will try setting both persist_backend and verify_backend to off. I am also considering changing MaxRequestsPerChild and setting it to something like 1,000. None of this really solves the problem, but it might alleviate it very considerably. Thank you very much. Your help was invaluable and assisted me in keeping a level head in dealing with this problem. .
Re: News regarding mod_perl returning empty pages
Igor, Why don't you try logging the request size from your mod_perl server? If it turns out that it knows when a request is zero bytes, you can just kill the process in a cleanup handler. Also, if you identify the PID of the broken process in this way, you can look back through the logs to see what it did on the request before it was broken. - Perrin
Re: News regarding mod_perl returning empty pages
On Wed, Aug 26, 2009 at 8:58 AM, Perrin Harkins phark...@gmail.com wrote: Igor, Why don't you try logging the request size from your mod_perl server? If it turns out that it knows when a request is zero bytes, you can just kill the process in a cleanup handler. Do you refer to the response size, as opposed to request size? If so... I like your idea. How would I do that? Also, if you identify the PID of the broken process in this way, you can look back through the logs to see what it did on the request before it was broken. Yes, it is a very good idea indeed.
Re: News regarding mod_perl returning empty pages
On Wed, Aug 26, 2009 at 11:12 AM, Igor Chudovichu...@gmail.com wrote: Do you refer to the response size, as opposed to request size? Yes. If so... I like your idea. How would I do that? Just read the logging section of the apache docs. It's a common part of the access log. - Perrin
Re: News regarding mod_perl returning empty pages
On Wed, Aug 26, 2009 at 8:16 AM, Perrin Harkinsphark...@gmail.com wrote: On Wed, Aug 26, 2009 at 11:12 AM, Igor Chudovichu...@gmail.com wrote: Do you refer to the response size, as opposed to request size? Yes. If so... I like your idea. How would I do that? Just read the logging section of the apache docs. It's a common part of the access log. This is a good reference: http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler
News regarding mod_perl returning empty pages
I have an update on this story. I set up a shell script that would notice this condition (empty pages returned) and alert me immediately via wall within 10 seconds. So I caught this in progress, before my 5 minute restarter would intervene, and experimented quickly. As I mentioned before, I have perlbal running in front on port 80, with apache listening on localhost port 10080 and serving perlbal. Since we discussed perlbal, I tried to access http://localhost.algebra.com:10080/ instead of www.algebra.com. ( localhost.algebra.com resolves to 127.0.0.1 and is an alias for www.algebra.com). Results were materially different: instead of 100% failures that I had with perlbal, I would only fail one out of many times. The failure became intermittent instead of constant. I believe that I have an explanation: only one worker happens to be messed up. When I access apache directly, I would connect to a random worker, so my failure would be intermittent. However, perlbal was set (in perlbal.conf) to maintain persistent connection, so it would mostly bang on one worker instead of randomly hitting them. As a result, with perlbal, if that worker goes bad, then the whole website is consistently not working. In other words, perlbal is not the issue, as such, but it exacerbated the problem due to the way I set it up. I changed perlbal.conf to set persist_backend = off, which, I hope, will have an effect of perlbal creating a separate connection for every request. Should not have much effect, speed wise, since everything runs on localhost, I hope. My new perlbal.conf is included: REATE POOL dynamic pool dynamic add 127.0.0.1:10080 CREATE SERVICE balancer SET listen = 0.0.0.0:80 SET role = reverse_proxy SET pool = dynamic SET persist_client = on SET persist_backend = off SET verify_backend = on ENABLE balancer # always good to keep an internal management port open: CREATE SERVICE mgmt SET role = management SET listen = 127.0.0.1:6 ENABLE mgmt
