Now we are developing new replacement content for large parts of this site on a pair of development only servers which are trying to catch up by using higher versions of all three; apache, modperl, and perl. These machines have very low load... no more than 4 or 5 people developing and testing, and rarely all at the same time. However, we have noticed some pretty scary behavior on these machines, and feel that it is linked somehow to the newer versions... since in most cases, the new mp/perl modules we're writing are better made and more efficient than the old... but despite this, there is an intrinsic decay in BOTH of the dev machines that is not present under the public machine which takes the vast brunt of the labor without a problem.
The versions on the two dev machines are: httpd-2.0.52-9, mod_perl-1.99_16-4, perl-5.8.5-12.1
At first, content works as expected on the dev machines... then, after a couple days or so, some of the error screens we have built in (like, 'we're sorry, the database is out of reach at the moment, please try again' and various 'you are not authorized to be here' screens) start creeping in, even when we can observe our database and warns from the code and see no problems... what is odd is that when these screens appear, there is NO HIT to either the error or access logs for the site. This behavior continues to decay to the point where sometimes you have to click a link 2 or 3 times to get past the error screens, and submitted forms have great difficulty getting through. After a bit of this, a rare case occurs where an ultimately unrelated screen appears ... a search page when you click to an administration link, and so on... no possible connected logic between the two, and yet it appears (though this is rare even under the worst conditions we've seen)... and then finally the system begins either to ignore the user's cookie, or sees some phantom cookie, and all pages that check it react as if it is wrong, no matter how many times you refresh. By the time it does this, no action will cure it... logging out and in again (destroy,recreate cookie), no effect, still acts as if their either is none or it stores info other than what it physically contains... getting error screens 3/4s of the clicks, etc.
And what cures this technoschizophrenic server behavior? A restart of apache (httpd) ... then everything is peachy keen again and the decay begins again, taking a day or two to accumulate.
We have an underlaying set of modules which interact with apache through modperl ($r), interpreting authentification, setting up higher level modules based on URI request, database-to-object, etc... then tons of upper level modules which are all the various URI sections of the site, using module-per-table data structures to be wholly OO, etc. This is how it works on the older versioned but wholly behaving machine, and how we've continued to think on the newer machines...
Has anyone encountered similar problems? Does anyone have a deeper understanding of the whole apache/mp/perl setup to just ~know~ what these symptoms are indicitive of? I'm struggling since I'm more on the upper perl end and less an OS person. Our OS guys feel like it's gotta be a code problem, while I feel that perhaps our apache and/or modperl installs on these machines are not ideal, or that in the move from mp1 to mp2, some new capacities in modperl2 are allowing a breakdown of otherwise good structures under mp1.
Any ideas? suggestions? ... heck, even psychic vibes??
