A small update: We thought the load average was a little high, in the 70+ range. We observed massive CPU usage from the httpd processes, but IO wise the machine was fine -- tons of IO bandwidth to go around -- it seemed to be something spinning inside httpd itself.
At first I thought it might of been the event mpm bug, so with the awesome MPMs as a module, we switched to worker. Didn't help. I dug out ktrace, and started seeing what we were doing. Every httpd processes I looked at was spinning on sendfile: 33893 httpd CALL sendfile(0x33,0x23,0x1d3000,0xbe6201,0x7ffffd1e9bd0,0x7ffffd1e9bf8,<invalid=0>) 33893 httpd RET sendfile -1 errno 35 Resource temporarily unavailable ktrace/dtrace dumps available to any committer who wants to take a look. A similar problem is described in this thread from 2002, about running 2.0.x on Daedalus: <http://www.mail-archive.com/[email protected]/msg04928.html> I've started tuning up the FreeBSD sysctls for network buffers etc, but it didn't seem to have a significant effect. We have now disabled Sendfile on apache.org, and the load average dropped from ~80 to 0.35. On Sun, Jul 18, 2010 at 3:08 AM, Paul Querna <[email protected]> wrote: > www.apache.org is now running trunk @ r965127, using the Event MPM on > FreeBSD 8.1, on a new x86 box. > > Previously it was running on Solaris 10, with 2.2.x on sparc t2000s. > > I have also enabled OCSP stapling on the SSL side: > <https://www.apache.org/> > > Anyways, I've made a list of the 2.3-ish issues seen during the upgrade. > > httpd itself: > - Using the Define directive to enable SSL was a bad idea. The SSL > init code was only being called once in post_config instead of twice > This is only 'new' because the Define directive is new, so I tried to > use it. Example of how it was used: > <IfDefine MACHINE_NAME> > Define SSL > Listen 1.2.3.4: > </IfDefine> > > <IfDefine SSL> > LoadModule ssl_module .. > </IfDefine> > > We do -D MACHINE_NAME at the command line, and I was hoping to just > enable SSL from within the config file, but that didn't work. Adding > -D SSL to the command line fixed the issue. > > - Lots of warnings in our configurations about "Ignoring deprecated > use of DefaultType". > > - Infra already had lots of <IfVersion blocks for most 2.3 authn/z > changes, and those seem to stil be working fine. > > 3rd party: > > - APR's detection of BDB was funky again, I had --with-dbm=db48, which > it was happy about, but until I added --with-berkeley-db=/usr/local, > it wouldn't actually compile up the shared objects for the DB DBM, but > configure reported success. > > - mod_wsgi: Needed to comment out 2 lines debug logging, that were > accessing server_rec->loglevel (which has been removed from 2.3.x). > > - mod_mbox/mod_allowmethods/mod_asf_cgi: Needed build system > adjustments to not look for apr-util. > > Surprisingly painless upgrade after I figured out the mod_ssl config > issue; Most of the work was around moving Solaris -> FreeBSD for our > various rsyncs and crontabs, of course let infra know if you see > anything busted! > > Thanks, > > Paul >
