I've got here exactly the same problem, but without any php modules. My server serves only static resources and It's flooded with reading state request sometime. Sometime there's more than 80 request waiting in reading state ..
The only turnaround I found is to lower the server timeout (to 10s) and increase the number of available processes. As soon as I'm serving only static stuff, it seems to be ok. On Nov 11, 2007 4:49 PM, Andrew Rosolino <[EMAIL PROTECTED]> wrote: > > Im using this for php.. > LoadModule php4_module modules/libphp4.so > > not sure if that is mod_php or what > > I tried using strace but my server was going extremly slow when I started it > probably because we get so much traffic so i couldnt debug it... > > I did run the gdb on a process and got this.. > > #0 0x003307a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 > #1 0x00408dbd in poll () from /lib/tls/libc.so.6 > #2 0x00af595f in apr_poll (aprset=0xbfe55310, num=1, nsds=0xbfe5530c, > timeout=300000) at poll.c:130 > #3 0x00af5d5f in apr_wait_for_io_or_timeout (f=0x0, s=0x959b4a8, > for_read=1) > at waitio.c:54 > #4 0x00aec19c in apr_socket_recv (sock=0x959b4a8, > buf=0x9624688 "\n# AddHandler application/x-httpd-php .php > .sc\r\nIndexIgnore .htaccess */.??* *~ *# */HEADER* */README* > */_vti*\r\nDirectoryIndex home.gc index.gc index.php\r\n\r\n<Limit GET > POST>\r\n#The next line modified b"..., > len=0xbfe553e8) at sendrecv.c:87 > #5 0x00fcc0dc in socket_bucket_read (a=0x95c0468, str=0xbfe553e4, > len=0xbfe553e8, block=APR_BLOCK_READ) at apr_buckets_socket.c:36 > #6 0x00fcd06a in apr_brigade_split_line (bbOut=0x95d0f50, bbIn=0x959b9a0, > block=APR_BLOCK_READ, maxbytes=8192) at apr_brigade.c:289 > #7 0x080b9a9b in core_input_filter (f=0x959b968, b=0x95d0f50, > mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core.c:3802 > #8 0x0806e8db in logio_in_filter (f=0x959b928, bb=0x95d0f50, > mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at > mod_logio.c:115 > #9 0x080b3ae3 in ap_rgetline_core (s=0x95d02a0, n=8192, read=0xbfe55528, > r=0x95d0288, fold=0, bb=0x95d0f50) at protocol.c:230 > #10 0x080b4577 in ap_read_request (conn=0x959b580) at protocol.c:586 > #11 0x0808bc6e in ap_process_http_connection (c=0x959b580) at > http_core.c:271 > ---Type <return> to continue, or q <return> to quit--- > #12 0x080b0a62 in ap_run_process_connection (c=0x959b580) at connection.c:43 > #13 0x080a5ad1 in child_main (child_num_arg=Variable "child_num_arg" is not > available. > ) at prefork.c:610 > #14 0x080a5cfb in make_child (s=Variable "s" is not available. > ) at prefork.c:704 > #15 0x080a5d8c in startup_children (number_to_start=5) at prefork.c:722 > #16 0x080a645f in ap_mpm_run (_pconf=0x8c2c0a8, plog=0x8c6c1a8, s=0x8c32ed0) > at prefork.c:941 > #17 0x080ab803 in main (argc=4, argv=0xbfe558c4) at main.c:638 > > havent tried netstat yet because the problem isnt occuring right now.. > Here is some info on the Apache we are using: > Server Version: Apache/2.0.61 (Unix) mod_ssl/2.0.61 OpenSSL/0.9.7a > mod_auth_passthrough/2.1 FrontPage/5.0.2.2635 mod_bwlimited/1.4 > > We are running REDHAT Enterprise 4 i686 almost all our pages are dynamic > php.. and its running about 200 websites.. we dont have a load balancer.. > > All the requests are not from the same ip address very random. > > > > Alf Høgemark wrote: > > > > Hi > > > > This seems to be quite similar to what I have seen, and others have also > > posted about the last couple of years. > > I've done a lot of investigation, but have not found a real solution. > > My problem is that the threads gets stuck in "W" / "Sending reply" state. > > > > Are you using mod_php ? If so, it might be that one thread is doing > > work, while the other threads are waiting to acquire a lock on the php > > session file. > > When PHP session files are used, the request are processed sequentially, > > and not in parallel for a user. > > > > I suggest you also use "strace" unix command to see what the different > > threads are doing, that might be easier than using gdb. > > When I use strace, I see that one thread is hanging on "poll" system > > call, until the timeout (Timeout Apache directive - default 300 seconds) > > occur. Then I see that the next thread hangs in "poll" and so on. > > So if you set the "Timeout" lower, it might improve your situation. > > > > You should also run "netstat", and see what the status of the sockets > > are. If they are almost all in CLOSE_WAIT, I think you are seeing > > something similar to me. > > > > Here are some links to similar issues, I'm not sure if it will help you : > > http://www.nabble.com/RE:-requests-time-out-under-load,-no-warnings-in-logs-p13065495.html > > http://mail-archives.apache.org/mod_mbox/httpd-users/200611.mbox/[EMAIL > > PROTECTED] > > ( and the other posts to that thread) > > http://www.linuxforums.org/forum/servers/27519-alright-im-dying-here-apache-help-mysql-php.html > > http://osdir.com/ml/apache.mod-ssl.user/2004-10/msg00009.html > > > > I have also seen other posts from people with very similar problems, > > lots of thread stuck in a state. > > > > What is your environment ? I.e. what version of Apache, what operating > > system and version, are you generating dynamic pages, and if so, what > > tools (for example php) are you using ? > > Do you have a firewall or a load balancer in front of your web server ? > > > > Are almost all the requests from the same IP adresse ? Are almost all > > the request for the same URL ? > > > > What my theory is at the moment, after throwing away some other > > theories, are that somehow a client creates several hundred requests for > > the same URL. > > Since in my case this happens fairly often, several times a day on a > > uncontroversial high traffic site, from different users, I do not think > > it is someone doing a DoS attack. > > > > I think it might be a problem with a firewall or a proxy or maybe the > > browser (I have seen it happen for people using both Windows and Mac, > > both Firefox and IE), that sits between the web server and the end user. > > Somehow the closing of the socket/connection by the browser is not > > picked up by the web server. And somehow the browser issues lots of > > requests. > > I have not completely ruled out that it might be a Apache web server bug > > as well. > > But since I have no way of reproducing this, apart from observing it > > happening quite often, it is very hard to get any further. > > So any help would be greatly appreciated. > > > > Regards > > Alf Hogemark > > > > > > > > Sander Temme wrote: > >> > >> On Nov 7, 2007, at 9:31 AM, Andrew Rosolino wrote: > >> > >>> I restarted apache and all was fine.. but then 20 minutes later they > >>> went > >>> back all into a reading state.. it appears as if slowly each > >>> processes goes > >>> into the reading state?? I dont understand what the problem is. > >> > >> If you have gdb on the box (and have some acumen using it), you can > >> attach to one of the processes and see if it is hanging in something > >> obvious. > >> > >> If not, and your traffic is plaintext, I recommend the tcpdump > >> approach to see what is going on traffic-wise, and what triggers the > >> issue. I'd dump the traffic to disk (tcpdump -i ethX -s 0 -w > >> /wherever/traffic.dump port 80) and take it over to a box with > >> Ethereal/Wireshark for some cozy analysis. > >> > >> If you're being DOS attacked by trickle requests, you could try > >> setting a very low timeout (default is 5 minutes which doesn't seem to > >> be working for you) and perhaps use mod_evasive or somesuch to flag > >> and firewall the bad clients. > >> > >> S. > >> > > > > --------------------------------------------------------------------- > > The official User-To-User support forum of the Apache HTTP Server Project. > > See <URL:http://httpd.apache.org/userslist.html> for more info. > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > " from the digest: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > -- > View this message in context: > http://www.nabble.com/Apache-Hangs..-Server-Status-shows-all-Reading-tf4766110.html#a13693182 > Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > > The official User-To-User support forum of the Apache HTTP Server Project. > See <URL:http://httpd.apache.org/userslist.html> for more info. > To unsubscribe, e-mail: [EMAIL PROTECTED] > " from the digest: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- Jerome Eteve. [EMAIL PROTECTED] http://jerome.eteve.free.fr/
