It's been a few days without a crash so I think I've got it beat. It was a bug in my code where I had a RandomAccessFile that I failed to close. What threw me off was that Apache failed and not the app. Once I used lsof on the app instances I saw the growing list of open files.
Anyway thanks for all your help! Johnny On Apr 6, 2011, at 9:36 AM, Pascal Robert wrote: > > Le 2011-04-06 à 15:26, Chuck Hill a écrit : > >> You are confusing number of Apache processes with file handles/sockets. The >> error message >>>>>>> Too many open files in system:Some/path/on/system >> refers to the latter, the number of connections is the former. > > And using lsof on Some/path/on/system will tell which processes are holding > files on that directory. Example: > > # sudo lsof -P +D /private/tmp/ > > Will find all files opened in /private/tmp > >> >> Chuck >> >> >> On Apr 6, 2011, at 12:10 PM, Johnny Miller wrote: >> >>> Mssr. Robert, >>> >>> Thank you for the suggestion. If I run the command you suggested I get a >>> fluctuating count between 3 and 9. While that is way below the threshold >>> you suggested I'm not sure I'm convinced that is the whole story. >>> Otherwise why would I be having a problem? >>> >>> Best, >>> >>> Johnny >>> >>> On Apr 6, 2011, at 8:54 AM, Pascal Robert wrote: >>> >>>> >>>> Le 2011-04-06 à 14:38, Johnny Miller a écrit : >>>> >>>>> Hi Johan, >>>>> >>>>> Yeah, I'm not doing anything unusual like creating a DB connection per >>>>> server. >>>>> >>>>> If I run from the CLI: >>>>> >>>>> sudo lsof | grep httpd | wc -l >>>>> >>>>> The count kind of bounces up and down between 500 - 1,500 >>>>> >>>>> If I just look at the contents instead of the count I see a bunch of >>>>> redundant connections to access_log and the various modules (see below) >>>>> >>>>> To be honest I'm out of my element here, so I'm not sure if this is >>>>> unusual or not. I've updated the apache's launch daemon plist to >>>>> increase the number of connections apache is allowed to 2048 >>>> >>>> MaxClient and ServerLimit set to 2048?? Are you sure that you need that >>>> much clients? I think I never set that value pass 384. >>>> >>>> How many httpd processes do you have? You can find out like this: >>>> >>>> # lsof -P | grep "/usr/sbin/httpd" | wc -l >>>> >>>>> and I've gone a couple of days without a failure. But I did do an >>>>> intentional restart last night. Prior to altering the plist I was seeing >>>>> apache going into the unresponsive state about every 24 hours. >>>>> >>>>> When I called Apple they told me to run netstat -m and look at this >>>>> section: >>>>> >>>>> 0 requests for memory denied >>>>> 0 requests for memory delayed >>>>> >>>>> If that goes over 0 you can increase the memory buffer per this article: >>>>> http://support.apple.com/kb/HT3512 But I've yet to see that number go >>>>> over zero. >>>>> >>>>> (this prints out several times) >>>>> httpd 6511 _www txt REG 14,2 1996352 >>>>> 17733353 /usr/sbin/httpd >>>>> httpd 6511 _www txt REG 14,2 393792 >>>>> 17733502 /usr/lib/libaprutil-1.0.2.7.dylib >>>>> httpd 6511 _www txt REG 14,2 651248 >>>>> 15133 /usr/lib/libexpat.1.5.0.dylib >>>>> httpd 6511 _www txt REG 14,2 4147008 >>>>> 15324 /usr/lib/libiconv.2.dylib >>>>> httpd 6511 _www txt REG 14,2 578656 >>>>> 17733501 /usr/lib/libapr-1.0.2.7.dylib >>>>> httpd 6511 _www txt REG 14,2 53888 >>>>> 17733291 /usr/libexec/apache2/mod_authn_file.so >>>>> httpd 6511 _www txt REG 14,2 66500 >>>>> 17733295 /usr/libexec/apache2/mod_authz_host.so >>>>> httpd 6511 _www txt REG 14,2 201360 >>>>> 17733299 /usr/libexec/apache2/mod_cache.so >>>>> httpd 6511 _www txt REG 14,2 70660 >>>>> 17733308 /usr/libexec/apache2/mod_dumpio.so >>>>> httpd 6511 _www txt REG 14,2 126528 >>>>> 17733321 /usr/libexec/apache2/mod_mem_cache.so >>>>> httpd 6511 _www txt REG 14,2 117296 >>>>> 17733307 /usr/libexec/apache2/mod_disk_cache.so >>>>> httpd 6511 _www txt REG 14,2 105944 >>>>> 17733311 /usr/libexec/apache2/mod_ext_filter.so >>>>> httpd 6511 _www txt REG 14,2 245316 >>>>> 17733316 /usr/libexec/apache2/mod_include.so >>>>> httpd 6511 _www txt REG 14,2 91784 >>>>> 17733312 /usr/libexec/apache2/mod_filter.so >>>>> httpd 6511 _www txt REG 14,2 121380 >>>>> 17733305 /usr/libexec/apache2/mod_deflate.so >>>>> httpd 6511 _www txt REG 14,2 126208 >>>>> 17733318 /usr/libexec/apache2/mod_log_config.so >>>>> httpd 6511 _www txt REG 14,2 54200 >>>>> 17733320 /usr/libexec/apache2/mod_logio.so >>>>> httpd 6511 _www txt REG 14,2 53956 >>>>> 17733309 /usr/libexec/apache2/mod_env.so >>>>> httpd 6511 _www txt REG 14,2 71040 >>>>> 17733310 /usr/libexec/apache2/mod_expires.so >>>>> httpd 6511 _www txt REG 14,2 100768 >>>>> 17733313 /usr/libexec/apache2/mod_headers.so >>>>> httpd 6511 _www txt REG 14,2 66568 >>>>> 17733314 /usr/libexec/apache2/mod_ident.so >>>>> httpd 6511 _www txt REG 14,2 70940 >>>>> 17733334 /usr/libexec/apache2/mod_setenvif.so >>>>> httpd 6511 _www txt REG 14,2 378832 >>>>> 17733325 /usr/libexec/apache2/mod_proxy.so >>>>> httpd 6511 _www txt REG 14,2 168484 >>>>> 17733330 /usr/libexec/apache2/mod_proxy_http.so >>>>> httpd 6511 _www txt REG 14,2 138088 >>>>> 17733327 /usr/libexec/apache2/mod_proxy_balancer.so >>>>> httpd 6511 _www txt REG 14,2 786308 >>>>> 17733336 /usr/libexec/apache2/mod_ssl.so >>>>> httpd 6511 _www txt REG 14,2 915632 >>>>> 17731671 /usr/lib/libssl.0.9.7.dylib >>>>> httpd 6511 _www txt REG 14,2 100588 >>>>> 17733322 /usr/libexec/apache2/mod_mime.so >>>>> httpd 6511 _www txt REG 14,2 120612 >>>>> 17733337 /usr/libexec/apache2/mod_status.so >>>>> httpd 6511 _www txt REG 14,2 176016 >>>>> 17733298 /usr/libexec/apache2/mod_autoindex.so >>>>> httpd 6511 _www txt REG 14,2 54000 >>>>> 17733284 /usr/libexec/apache2/mod_asis.so >>>>> httpd 6511 _www txt REG 14,2 101080 >>>>> 17733317 /usr/libexec/apache2/mod_info.so >>>>> httpd 6511 _www txt REG 14,2 127384 >>>>> 17733301 /usr/libexec/apache2/mod_cgi.so >>>>> httpd 6511 _www txt REG 14,2 70560 >>>>> 17733343 /usr/libexec/apache2/mod_vhost_alias.so >>>>> httpd 6511 _www txt REG 14,2 167388 >>>>> 17733324 /usr/libexec/apache2/mod_negotiation.so >>>>> httpd 6511 _www txt REG 14,2 66448 >>>>> 17733306 /usr/libexec/apache2/mod_dir.so >>>>> httpd 6511 _www txt REG 14,2 95568 >>>>> 17733315 /usr/libexec/apache2/mod_imagemap.so >>>>> httpd 6511 _www txt REG 14,2 58204 >>>>> 17733282 /usr/libexec/apache2/mod_actions.so >>>>> httpd 6511 _www txt REG 14,2 71144 >>>>> 17733335 /usr/libexec/apache2/mod_speling.so >>>>> httpd 6511 _www txt REG 14,2 66460 >>>>> 17733340 /usr/libexec/apache2/mod_userdir.so >>>>> httpd 6511 _www txt REG 14,2 79308 >>>>> 17733283 /usr/libexec/apache2/mod_alias.so >>>>> httpd 6511 _www txt REG 14,2 448160 >>>>> 698607 /System/Library/WebObjects/Adaptors/Apache2.2/mod_WebObjects.so >>>>> httpd 6511 _www txt REG 14,2 305712 >>>>> 17733333 /usr/libexec/apache2/mod_rewrite.so >>>>> httpd 6511 _www txt REG 14,2 290736 >>>>> 3429086 /usr/lib/libexslt.0.dylib >>>>> httpd 6511 _www txt REG 14,2 1297968 >>>>> 15260 /usr/lib/libiodbc.2.dylib >>>>> httpd 6511 _www txt REG 14,2 1046544 >>>>> 17731388 /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP >>>>> httpd 6511 _www txt REG 14,2 958864 >>>>> 17733803 /usr/lib/libcurl.4.dylib >>>>> httpd 6511 _www txt REG 14,2 401216 >>>>> 17733947 /usr/lib/libsasl2.2.dylib >>>>> httpd 6511 _www txt REG 14,2 88356 >>>>> 172780 /usr/libexec/apache2/mod_auth_apple.so >>>>> httpd 6511 _www txt REG 14,2 101284 >>>>> 172785 /usr/libexec/apache2/mod_spnego_apple.so >>>>> httpd 6511 _www txt REG 14,2 105824 >>>>> 172775 >>>>> /System/Library/PrivateFrameworks/SpnegoApache.framework/Versions/A/SpnegoApache >>>>> httpd 6511 _www txt REG 14,2 684032 >>>>> 17731066 >>>>> /System/Library/Frameworks/DirectoryService.framework/Versions/A/DirectoryService >>>>> httpd 6511 _www txt REG 14,2 135724 >>>>> 172782 /usr/libexec/apache2/mod_digest_apple.so >>>>> httpd 6511 _www txt REG 14,2 66280 >>>>> 174412 /usr/libexec/apache2/mod_hfs_apple.so >>>>> httpd 6511 _www txt REG 14,2 1380352 >>>>> 18089764 /private/tmp/WOAdaptorState >>>>> httpd 6511 _www txt REG 14,2 31641660 >>>>> 17733495 /usr/libexec/apache2/libphp5.so >>>>> httpd 6511 _www txt REG 14,2 1380352 >>>>> 18089793 /private/tmp/WOAdaptorState >>>>> httpd 6511 _www txt REG 14,2 1059792 >>>>> 8952176 /usr/lib/dyld >>>>> httpd 6511 _www txt REG 14,2 98283520 >>>>> 17736835 /private/var/db/dyld/dyld_shared_cache_x86_64 >>>>> httpd 6511 _www 0r CHR 3,2 0t0 >>>>> 345489284 /dev/null >>>>> >>>>> (same here this repeats many times) >>>>> httpd 6511 _www 11w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 12w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 13w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 14w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 15w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 16w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 17w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 18w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 19w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 20w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 21w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 22w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 23w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 24w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 25w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 26w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 27w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 28w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 29w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 30w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 31w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 32w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 33w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 34w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 35w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 36w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> httpd 6511 _www 37w REG 14,2 5055271697 >>>>> 474173 /private/var/log/apache2/access_log >>>>> >>>>> >>>>> >>>>> On Apr 6, 2011, at 7:53 AM, Johan Henselmans wrote: >>>>> >>>>>> >>>>>> On 5 apr 2011, at 02:40, Johnny Miller wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I was wondering if anyone else had experienced the same thing. I have >>>>>>> a X-Serve running OS X 10.5.8 whose apache server is going into a >>>>>>> frozen state where it stops serving requests and can only be restarted >>>>>>> by rebooting the machine. All other services seem to being running >>>>>>> normally. >>>>>>> >>>>>>> If I look at the error_log I see many variations of this error: >>>>>>> >>>>>>> Too many open files in system:Some/path/on/system >>>>>>> >>>>>>> I've seen this thread on the Apple discussion board: >>>>>>> http://discussions.apple.com/thread.jspa?threadID=1449787&start=15&tstart=0 >>>>>>> And this article in the knowledge base: >>>>>>> http://support.apple.com/kb/HT3512 >>>>>>> >>>>>>> Does anyone have more wisdom to share? >>>>>>> >>>>>> >>>>>> I once had a similar problem when I started my own objectstore >>>>>> coordinator for each session, along the line of: >>>>>> // this causes all kinds of mayhem: do not do this at home, it creates a >>>>>> new database connection for every session >>>>>> >>>>>> // EOObjectStoreCoordinator coord = new EOObjectStoreCoordinator(); >>>>>> // myEc = ERXEC.newEditingContext(coord); >>>>>> // myEc().setDelegate(this); >>>>>> >>>>>> >>>>>> After that, the database would have so many open connections after a >>>>>> while that the database would have too many open files, and would break >>>>>> down. >>>>>> >>>>>> I also seem to remember that some apache configuration would freeze with >>>>>> a specific log-rotation setup. (that was apache 1.3 on a FreeBSD machine) >>>>>> >>>>>> Furthermore I would look for something in apache that opens a file but >>>>>> never closes these. You might use opensnoop for that, see man opensnoop, >>>>>> or lsof might give you some clues. >>>>>> >>>>>>> Thanks in advance, >>>>>>> >>>>>>> Johnny Miller >>>>>>> Kahalawai Media Corp >>>>>>> http://www.kahalawai.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) >>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>> http://lists.apple.com/mailman/options/webobjects-dev/johan%40netsense.nl >>>>>>> >>>>>>> This email sent to jo...@netsense.nl >>>>>> >>>>>> Johan Henselmans >>>>>> jo...@netsense.nl >>>>>> >>>>>> >>>>>> >>>>> >>>>> Johnny Miller >>>>> Kahalawai Media Corp >>>>> http://www.kahalawai.com >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Do not post admin requests to the list. They will be ignored. >>>>> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) >>>>> Help/Unsubscribe/Update your Subscription: >>>>> http://lists.apple.com/mailman/options/webobjects-dev/probert%40macti.ca >>>>> >>>>> This email sent to prob...@macti.ca >>>> >>> >>> Johnny Miller >>> Kahalawai Media Corp >>> http://www.kahalawai.com >>> >>> >>> >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> http://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net >>> >>> This email sent to ch...@global-village.net >> >> -- >> Chuck Hill Senior Consultant / VP Development >> >> Practical WebObjects - for developers who want to increase their overall >> knowledge of WebObjects or who are trying to solve specific problems. >> http://www.global-village.net/products/practical_webobjects >> >> >> >> >> >> >> > Johnny Miller Kahalawai Media Corp http://www.kahalawai.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
