It ran for 18+ hours without httpd.webservices failing again, so it appears that AcceptMutex fixed it. Can I suggest that this be made a permanent change for distribution in future CentOS builds?
One thing that I did notice in the other person's thread is that I did *not* receive an error for MaxClients. (Apache seems to be shutting down extra httpd.webservices tasks a few minutes after boot.) This is why I'm not 100% certain that the same underlying problems caused both of our problems. While looking in the portal_error_log file, I noticed these emergency notices that correspond with every PF restart or server reboot: portal_error_log:[Thu Mar 05 17:13:28 2015] [emerg] mod_qos(007): could not determine MaxClients/MaxRequestWorkers! You MUST set this directive within the Apache configuration file. portal_error_log:[Thu Mar 05 17:13:30 2015] [emerg] mod_qos(007): could not determine MaxClients/MaxRequestWorkers! You MUST set this directive within the Apache configuration file. Checking the portal's config file reveals these for where Apache is trying to fill in those values: $MaxClients = pf::services::manager::httpd::calculate_max_clients(pf::services::manager:: httpd::get_total_system_memory()); $StartServers = pf::services::manager::httpd::calculate_start_servers($MaxClients); $MinSpareServers = pf::services::manager::httpd::calculate_min_spare_servers($MaxClients); Anyway, I was wondering if there is a race condition on startup that prevents whatever provides the above values from being loaded and ready to provide the answers? Something else that I did notice is that multiple pfdns and/or pfdhcplistener jobs seem to be starting on server boot. I don't know if this is a symptom of the underlying problem that I was having, or potentially the cause? I can say that an overly-aggressive service watch cron schedule (like 2 minutes) combined with having PF restart failed services creates a race condition where during boot the cron job starts some services before the PF startup script, which may explain some of these. (I cloned my running PF VM and booted the clone over a dozen times, and had to kill the cron entry to get a clean boot with an aggressive cron schedule.) Service watch may need some locking mechanism so that it knows not to restart services during boot, because even with a longer cron schedule it could still potentially happen to anyone. Maybe also a conditional on the pfdhcplistener so that it doesn't start one if another one is running (no matter what the PID file mechanism says)? I'm also wondering if multiple running listeners isn't the cause of my other observation from months ago that PF seems to be creating duplicate node entries for some new machines??? -Arthur ------------------------------------------------------------------------- Arthur Emerson III Email: [email protected] Network Administrator InterNIC: AE81 Mount Saint Mary College MaBell: (845) 561-0800 Ext. 3109 330 Powell Ave. Fax: (845) 562-6762 Newburgh, NY 12550 SneakerNet: Aquinas Hall Room 11 On 3/5/15, 9:50 PM, "Arthur Emerson" <[email protected]> wrote: >Louis Munro <[email protected]> wrote: >> >> Yes, that’s possible, especially if pfdhcplistener takes a long time to >>stop. >> >> Did you change the httpd AcceptMutex type as mentioned in the thread? > >I put it in the config file, waiting for the next failure so that >service watch will restart it with that setting. Of course, it is >behaving itself now that I *want* it to fail! :-( > >We are on spring break this week, and the network load is only 5% of >normal. I changed service watch's schedule in cron to a safer 5 minutes, >and have it set to restart any failed services. At this point, I'm >inclined to leave it run overnight and see if/how many times it >restarts... > >-Arthur > >------------------------------------------------------------------------- >Arthur Emerson III Email: [email protected] >Network Administrator InterNIC: AE81 >Mount Saint Mary College MaBell: (845) 561-0800 Ext. 3109 >330 Powell Ave. Fax: (845) 562-6762 >Newburgh, NY 12550 SneakerNet: Aquinas Hall Room 11 > >-------------------------------------------------------------------------- >---- >Dive into the World of Parallel Programming The Go Parallel Website, >sponsored >by Intel and developed in partnership with Slashdot Media, is your hub >for all >things parallel software development, from weekly thought leadership >blogs to >news, videos, case studies, tutorials and more. Take a look and join the >conversation now. http://goparallel.sourceforge.net/ >_______________________________________________ >PacketFence-users mailing list >[email protected] >https://lists.sourceforge.net/lists/listinfo/packetfence-users ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ PacketFence-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/packetfence-users
