Scott Hess wrote: > On Thu, 21 Mar 2002, Stas Bekman wrote: > >>>On Wed, 20 Mar 2002, Stas Bekman wrote: >>> >>> >>>>mod_perl child processes save a lot of memory when they can share >>>>memory with the parent process and quite often we get reports from >>>>people that they lose that shared memory when the system decides to >>>>page out the parent's memory pages because they are LRU (least >>>>recently used, the algorithm used by many memory managers). >>>> >>>> >>>I'm fairly certain that this is not an issue. If a page was shared >>>COW before being paged out, I expect it will be shared COW when paged >>>back in, at least for any modern OS. >>> >>But if the system needs to page things out, most of the parent process's >>pages will be scheduled to go first, no? So we are talking about a >>constant page-in/page-out from/to the parent process as a performance >>degradation rather than memory unsharing. Am I correct? >> > > The system is going to page out an approximation of the > least-recently-used pages. If the children are using those pages, then > they won't be paged out, regardless of what the parent is doing. [If the > children _aren't_ using those pages, then who cares?]
mod_perl users do care. let's say you've preloaded a bunch of modules which aren't in use now but may be used any moment. Ideally we don't want any of the pages to be swapped out. Of course this is all possible if the demand does go over the supply. >>I'm just repeating the reports posted to the mod_perl list. I've never >>seen such a problem myself, since I try hard to have close to zero swap >>usage. >> > > :-). In my experience, you can get some really weird stuff happening when > you start swapping mod_perl. It seems to be stable in memory usage, > though, so long as you have MaxClients set low enough that your maximum > amount of committed memory is appropriate. Also, I've seen people run > other heavyweight processes, like mysql, on the same system, so that when > the volume spikes, mod_perl spikes AND mysql spikes. A sure recipe for > disaster. Well, we (mod_perl users) use tools like Apache::GTopLimit and Apache::SizeLimit which monitor the child process' memory and will kill the process after a request if the memory size crossed a total memory used limit, or lost too much of a shared memory. I think that's where the problem lays, the memory usage tools report that the sharing went down and the process gets killed when the parent's pages are getting paged out. Even if in reality the sharing is still there. The memory reporting tools don't reflect the reality. >>[Yes, please let me know your methodology for testing this] >> > > OK, two programs. bigshare.c: thanks Scott, I've forwarded this code to the folks who are having the trouble with. and it'll stay in the archives for others to re-use. >>>>Therefore my question is there any reason for not using mlockall(2) in >>>>the parent process on systems that support it and when the parent >>>>httpd is started as root (mlock* works only within root owned >>>>processes). >>>> >>>I don't think mlockall is appropriate for something with the heft of >>>mod_perl. >>> >>>Why are the pages being swapped out in the first place? Presumably >>>there's a valid reason. >>> >>Well, the system coming close to zero of real memory available. The >>parent process starts swapping like crazy because most of its pages are >>LRU, slowing the whole system down and if the load doesn't go away, the >>system takes a whirl down to a halt. >> > > I can think of a couple possible causes. One is the MaxClients setting. > Say you have MaxClients set to 50, but on a standard day you never need > more than 15 servers. Then you get listed on slashdot. As you pass, oh, > 30 simultaneous servers, you start thrashing, so requests take longer to > process, so you immediately spike to your MaxClients of 50 and the server > goes right down the drain. If you shut things down and start them back > up, it's likely you'll immediately spike to 50 again, and back down the > drain it goes. > > I've found that it's _super_ important to make certain you've pushed > mod_perl to MaxClients under your standard conditions. Once you start > swapping, you're doomed, unless the traffic was a very short spike. > > Another possible cause is that you have another heavyweight server running > on the same server. As I indicated above, I've seen people do this with > things like mysql. Since high mod_perl traffic implies high mysql > traffic, it's just like having MaxClients set too high, but twice as bad! > > Another possible cause is that the OS is aggressively grabbing pages for > the filesystem cache. It's possible that tuning down the size of the > filesystem cache would be appropriate - many webservers have a very large > maximum amount of data they might server, but a very small working set. > > Really, though, all of this really tends to come down to MaxClients. The > database load is proportional to the number of clients, the filesystem > load is proportional to the number of clients, everything is proportional > to the number of clients. MaxClients is as close to a silver bullet as > I've seen. People tend to set MaxClients based on their expected load, > rather than on how much load their server can handle. You just have to > arrange for your maximum committed memory usage to appropriately reflect > the memory available, or you're doomed, there's nothing the OS can do to > help. As mentioned above we restrict the memory usage and then we know the exact value for MaxClients that won't lead to swapping. Here is how we do the calculation: http://perl.apache.org/guide/performance.html#Choosing_MaxClients Of course as you said a different process may lead to swapping, but for heavy servers it's best to separate the database from the webserver, so no complaints here. One new issue that you've raised here is the filesystem cache. Is that really important to tune this component to improve performance? If so what are the guidelines for doing this, RTFM links are very welcome. >>>Doing mlockall on your mod_perl would result in restricting the memory >>>available to the rest of the system. Whatever is causing mod_perl to >>>page out would then start thrashing. Worse, since mlockall will lock >>>down mod_perl pages indiscriminately, the resulting thrashing will >>>probably be even worse than what they're seeing right now. >>> >>Possible, I've never tried this myself and therefore asked. Someone has >>suggested using P_SYSTEM flag which is supposed to tell the system not >>to page out certain pages, but seems to be a *BSD thingy. >> > > Really, the problem is that it's very hard to figure out which pages of > mod_perl really need this treatment. Heck, it's very hard with any > program, but with mod_perl you have to deal with the obfuscating Perl > virtual machine layer. That's pretty tough. If you could just lock down > whatever is needed to keep running, that would be great... indeed ;) Thanks for your insightful comments Scott. I've forwarded these to the mod_perl list so others can learn from these. I've many of the issues you've mentioned documented here: http://perl.apache.org/guide/performance.html#Know_Your_Operating_System http://perl.apache.org/guide/performance.html#Performance_Tuning_by_Tweaking_A but there is always space for improvement :) _____________________________________________________________________ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
