George, I have something similar...
I have been debugging an issue where I have seen processes growing to 800Mb on a forward proxy configuration using the worker model. Perhaps interestingly on reverse proxy configurations I get 100% CPU states occasionally as well. What I have noticed is that these are almost always associated with the TCP state condition of CLOSE_WAIT. What is netstat saying on your system when the problems occur? Just today I had a win where I was able to truss the process on a forward proxy reading data from a server but had nowhere to write as the socket to the client was closed/half closed. The error returned (EPIPE) was not being caught by apache so there was this continual read then failure to write. While this process did not result in a large increase of memory or CPU or CLOSE_WAIT states it did verify for me something I had suspected for a long time, that the apache code is not checking it's socket calls at some point (in this case writev), and/or does not catch a close on the socket. This trace was on an ftp connection so the results may be different for an http connection e.g. memory usage. I will be discussing these issues on the dev mailing list in the near future but it would be good to see if we are seeing anything in common first. My hunch (which is a very long one) has been for a while now that when the client breaks the connection, the process somehow misses the close and continues to read but cannot write as verified by the above. It appears that in some instances the memory consumption may be caused by the buffering of data or bad memory management which is due to socket issues i.e. I have noticed that the larger the download file such as an iso image the larger the process grows. As the processes handle additional connections the memory is not freed and it grows. Note at this point it is speculation on my part but I consistantly get these stateless connections in conjunction with high CPU or memory usage so something is going on. I have noticed that processes will always creep in size with time so I believe there may be memory leaks but the cause of large memory consumption may be caused by weird socket states. If your clients are downloading 18Mb files over slow links they may keep trying the connection, breaking the original therefore leaving you with multiple connections to the same file from the same client. If there is a memory leak under half close conditions your process grows and your processes will handle 5000 connections before they are cycled as per your configuration. But why does this not appear to affect many other people? Firewalls perhaps, do you have any network or host based firewalls which may be preventing proper shutdown of connections? If so do they drop or reject packets? Which firewalls are they? I work in an Internet Gateway environment so I have firewalls all over the place and have added these as a variable to my list of possibilities. Your 20 concurrent connections are limited by MaxClients. I assume you are keeping this small because of the size they are growing to as you should be able to get to approx 175-200 of the top of my head using prefork with 1Gb of memory. I would have thought this would max out pretty quickly with many 18Mb downloads as they will take time. As a workaround you may try to lower the MaxRequestsPerChild to turnover processes which may be affected by memory leakage and raise the MaxClients to handle more concurrent connections. Say initially MaxClients 150 and MaxRequestsPerChild 100 or more agressively 10. This will produce more CPU overhead in forking processes but modern CPUs are pretty fast. Or go for a threaded model such as worker and you should be able to get 10-15 times as many concurrent connections (based on proxy configurations - I have never used apache as a web server). But another model may simply have the same issues if it is socket related. Funnily I was considering going to a prefork model to eliminate the possibility of threading and mutex issues - won't be doing that for a while. This may not help but I would be interested in whether there are similarities in network states or hardware etc. to what I have. Regards, Jon On Wednesday 10 August 2005 01:05, George Adams wrote: > I read an earlier thread on memory consumption (http://tinyurl.com/bly4d), > which may be related to my problem... but because of some differences, I'm > not so sure. Any help would be appreciated! > > I have an Apache 2.0.54 server on a Gentoo Linux (2.6.11) box which has > 1Gig RAM and an additional 1Gig swap space. The server handles a lot of > people downloading sermons from our church website (which are no larger > than 18Meg MP3 files), but I can't figure out how to keep the server from > running out of memory. > > Here's my Apache2 prefork configuration: > ------------------------------------------------------------------------ > <IfModule prefork.c> > StartServers 5 > MinSpareServers 5 > MaxSpareServers 10 > MaxClients 20 > MaxRequestsPerChild 5000 > </IfModule> > > > And here's what the Apache "/server-status" URL showed earlier today (I had > just restarted the server, but it immediately filled up with download > requests, all from the same guy, apparently using a download accelerator > judging by the duplicate requests): > ------------------------------------------------------------------------ > Srv PID M CPU Req Request > 0-0 15822 W 0.48 0 GET /out/181.mp3 HTTP/1.1 > 1-0 15823 W 0.00 1742573500 GET /out/388.mp3 HTTP/1.1 > 2-0 15824 W 0.00 1742573499 GET /out/238.mp3 HTTP/1.1 > 3-0 15825 W 0.00 1742573499 GET /out/504.mp3 HTTP/1.1 > 4-0 15826 W 0.00 1742573496 GET /out/388.mp3 HTTP/1.1 > 5-0 15832 W 0.00 1742572495 GET /out/801.mp3 HTTP/1.1 > 6-0 15834 W 0.00 1742571493 GET /out/504.mp3 HTTP/1.1 > 7-0 15835 W 0.00 1742571489 GET /out/504.mp3 HTTP/1.1 > 8-0 15838 W 0.00 1742570476 GET /out/388.mp3 HTTP/1.1 > 9-0 15839 W 0.00 1742570484 GET /out/504.mp3 HTTP/1.1 > 10-0 15840 W 0.60 0 GET /out/238.mp3 HTTP/1.1 > 11-0 15841 W 0.00 1742570477 GET /out/388.mp3 HTTP/1.1 > 12-0 15846 W 0.25 0 GET /out/181.mp3 HTTP/1.1 > 13-0 15847 W 0.00 1742569347 GET /out/181.mp3 HTTP/1.1 > 14-0 15848 W 0.00 1742568761 GET /out/801.mp3 HTTP/1.1 > 15-0 15849 W 0.00 1742568761 GET /out/801.mp3 HTTP/1.1 > 16-0 15852 W 0.19 0 GET /out/181.mp3 HTTP/1.1 > 17-0 15853 W 0.17 0 GET /out/801.mp3 HTTP/1.1 > 18-0 15854 W 0.22 0 GET /out/504.mp3 HTTP/1.1 > 19-0 15855 W 0.28 0 GET /server-status HTTP/1.1 > > > And here's a portion of what "top" showed at the same time: > ------------------------------------------------------------------------ > top - 18:09:59 up 64 days, 7:08, 3 users, load avg: 21.62, 10.57, 4.70 > Tasks: 154 total, 1 running, 143 sleeping, 1 stopped, 9 zombie > Cpu(s): 0.8% us, 2.3% sy, 0.0% ni, 0.0% id, 96.3% wa, 0.3% hi, 0.2% si > Mem: 1034276k total, 1021772k used, 12504k free, 6004k buffers > Swap: 1030316k total, 985832k used, 44484k free, 83812k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 15846 apache 16 0 132m 89m 1968 S 0.3 8.9 0:01.46 apache2 > 15840 apache 17 0 130m 83m 2008 D 0.0 8.3 0:00.90 apache2 > 15849 apache 16 0 120m 82m 1968 S 0.3 8.1 0:01.02 apache2 > 15852 apache 16 0 120m 81m 1968 S 0.3 8.1 0:00.91 apache2 > 15848 apache 16 0 109m 73m 2008 S 0.3 7.2 0:00.85 apache2 > 15855 apache 16 0 107m 70m 2076 D 0.3 7.0 0:00.76 apache2 > 15822 apache 17 0 179m 55m 1968 D 0.3 5.5 0:00.88 apache2 > 15854 apache 16 0 98024 55m 1968 D 0.0 5.5 0:00.58 apache2 > 15853 apache 18 0 98.9m 53m 2000 S 0.0 5.3 0:00.51 apache2 > 15847 apache 17 0 86884 52m 1968 D 0.0 5.2 0:00.42 apache2 > 15841 apache 17 0 110m 36m 1964 D 0.3 3.6 0:00.64 apache2 > 15826 apache 17 0 173m 20m 1968 D 0.0 2.0 0:00.57 apache2 > 15825 apache 16 0 97.7m 19m 1968 D 0.0 1.9 0:00.36 apache2 > 15834 apache 16 0 117m 14m 1968 D 0.3 1.5 0:00.42 apache2 > 15839 apache 17 0 115m 12m 1968 D 0.0 1.2 0:00.40 apache2 > 15838 apache 15 0 182m 12m 1968 D 0.0 1.2 0:00.59 apache2 > 15823 apache 16 0 180m 11m 1968 D 0.0 1.1 0:00.65 apache2 > 15824 apache 15 0 103m 9980 1968 D 0.0 1.0 0:00.27 apache2 > 15832 apache 16 0 116m 9112 1968 D 0.0 0.9 0:00.29 apache2 > 15835 apache 16 0 162m 8844 1968 D 0.0 0.9 0:00.41 apache2 > (everything else listed on "top" below this was less than 0.5 for %MEM) > > > The memory usage swelled very fast as the download requests came in, and > based on previous experience, the server would have slowed to a crawl and > possible crashed as it tried to save itself if I hadn't run "killall > apache2" at this point. > > So it seems like this guy's 19 download requests are enough to pretty much > exhaust my 1 Gig of physical RAM and 1 Gig of swap space. That just > doesn't seem right. EVEN IF something weird was happening where every > Apache child loaded an entire MP3 file into RAM before serving it, that > still only accounts for 20 servers * 18Meg files = 360Meg RAM - a lot, but > nowhere near 2 Gig. Yet these 20 processes have consumed almost 2 Gig. > > What am I doing wrong that so few download requests can bring the server to > its knees? How can I fix this configuration? > > Thanks to anyone who can help! > > > > --------------------------------------------------------------------- > The official User-To-User support forum of the Apache HTTP Server Project. > See <URL:http://httpd.apache.org/userslist.html> for more info. > To unsubscribe, e-mail: [EMAIL PROTECTED] > " from the digest: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: [EMAIL PROTECTED] " from the digest: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
