The backend/frontend setup works well, but for the following problem
that I think is a risk for loaded sites:
I set MaxChildRequests to 500, to effectively clean house every
now and again, as is recommended.. I have about 12 backend
modperl servers, and that handles the load fine. (about 250 front
ends).
What happens is, though, they all tick up towards MaxChildRequests,
and they pretty much start getting cloe to 500 over the period of
less than a minute. (about 10-20 minutes after a server restart).
What I think happens is the children die after their last request,
and apache does not kick off a new child straight away.. MinFree is
set to 2 .. as 12 becomes 11 becomes 10 becomes 9, my backend is
getting less and less powerful and more and more swamped. When
Apache does wakeup and spawn a new child, it takes many seconds
to interpret all the (large amount) of perl code and modules that
are not in the parent... up to 10 seconds now since its only getting
a fraction of the box.. this gives the unlucky user a dead browser.
Worse, the remaining alive servers are dieing faster now as they
handle more and more of the load, and rush towards 500 to contribute
to the same traffic jam of booting children.
The effect is essentially that what starts out as random child
death/restart collapses to all backend processes rebooting at the
same time, and an effectively dead server for about 20 seconds.
Any easy fixes to this? am I the first to find this? Am i missing
something obvious? Perhaps having my own ChildRequest counter and
dieing myself, but more randomly, is a better way?
thanks
-Justin
dslreports.com
