Re: performance: using mlock(2) on httpd parent process
Hi Scott, On Wed, 20 Mar 2002, Stas Bekman wrote: mod_perl child processes save a lot of memory when they can share memory with the parent process and quite often we get reports from people that they lose that shared memory when the system decides to page out the parent's memory pages because they are LRU (least recently used, the algorithm used by many memory managers). I'm fairly certain that this is not an issue. If a page was shared COW before being paged out, I expect it will be shared COW when paged back in, at least for any modern OS. But if the system needs to page things out, most of the parent process's pages will be scheduled to go first, no? So we are talking about a constant page-in/page-out from/to the parent process as a performance degradation rather than memory unsharing. Am I correct? [To verify that I wasn't talking through my hat, here, I just verified this using RedHat 7.2 running kernel 2.4.9-21. If you're interested in my methodology, drop me an email.] I suppose that this could vary from one kernel version to another. I'm just repeating the reports posted to the mod_perl list. I've never seen such a problem myself, since I try hard to have close to zero swap usage. [Yes, please let me know your methodology for testing this] I believe that this applies to all httpd modules and httpd itself, the more we can share the less memory resources are needed, and usually it leads to a better performance. I'm absolutely _certain_ that unmodified pages from executable files will be backed by the executable, and will thus be shared by default. Therefore my question is there any reason for not using mlockall(2) in the parent process on systems that support it and when the parent httpd is started as root (mlock* works only within root owned processes). I don't think mlockall is appropriate for something with the heft of mod_perl. Why are the pages being swapped out in the first place? Presumably there's a valid reason. Well, the system coming close to zero of real memory available. The parent process starts swapping like crazy because most of its pages are LRU, slowing the whole system down and if the load doesn't go away, the system takes a whirl down to a halt. Doing mlockall on your mod_perl would result in restricting the memory available to the rest of the system. Whatever is causing mod_perl to page out would then start thrashing. Worse, since mlockall will lock down mod_perl pages indiscriminately, the resulting thrashing will probably be even worse than what they're seeing right now. Possible, I've never tried this myself and therefore asked. Someone has suggested using P_SYSTEM flag which is supposed to tell the system not to page out certain pages, but seems to be a *BSD thingy. I've also seen madvise(2) - give advice about use of memory but again no personal experience here. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: performance: using mlock(2) on httpd parent process
On Thu, Mar 21, 2002 at 02:14:04AM +0800, Stas Bekman wrote: But if the system needs to page things out, most of the parent process's pages will be scheduled to go first, no? So we are talking about a constant page-in/page-out from/to the parent process as a performance degradation rather than memory unsharing. Am I correct? [...] Well, the system coming close to zero of real memory available. The parent process starts swapping like crazy because most of its pages are LRU, slowing the whole system down and if the load doesn't go away, the system takes a whirl down to a halt. FWIW, Solaris 8 introduced priority paging, which basicly means that pages that are non-executable will be swapped out earlier than higher-priority executable pages. This of course introduces nasty little tricks like marking a data file as executable just to ensure that it gets priority paging. I'm not sure how this affects mod_perl and it's data is runtime-compiled bytecode situation, but it means that we get it for free w/ httpd and all its dependent libraries (and DSOs). -aaron p.s. It can be enabled on Solaris 7, it's just not on by default.
Re: performance: using mlock(2) on httpd parent process
Aaron Bannert wrote: On Thu, Mar 21, 2002 at 02:14:04AM +0800, Stas Bekman wrote: But if the system needs to page things out, most of the parent process's pages will be scheduled to go first, no? So we are talking about a constant page-in/page-out from/to the parent process as a performance degradation rather than memory unsharing. Am I correct? [...] Well, the system coming close to zero of real memory available. The parent process starts swapping like crazy because most of its pages are LRU, slowing the whole system down and if the load doesn't go away, the system takes a whirl down to a halt. FWIW, Solaris 8 introduced priority paging, which basicly means that pages that are non-executable will be swapped out earlier than higher-priority executable pages. This of course introduces nasty little tricks like marking a data file as executable just to ensure that it gets priority paging. priority paging is more to do with the filesystem cache than anything else. in low memory conditions it start freeing pages used by the file cache first. without it you could swap your machine to death by copying a 1 gig file around the place. I'm not sure how this affects mod_perl and it's data is runtime-compiled bytecode situation, but it means that we get it for free w/ httpd and all its dependent libraries (and DSOs). -aaron p.s. It can be enabled on Solaris 7, it's just not on by default.
Re: performance: using mlock(2) on httpd parent process
On Wed, Mar 20, 2002 at 11:19:44AM -0800, Ian Holsman wrote: FWIW, Solaris 8 introduced priority paging, which basicly means that pages that are non-executable will be swapped out earlier than higher-priority executable pages. This of course introduces nasty little tricks like marking a data file as executable just to ensure that it gets priority paging. priority paging is more to do with the filesystem cache than anything else. in low memory conditions it start freeing pages used by the file cache first. without it you could swap your machine to death by copying a 1 gig file around the place. We are talking about a low memory condition, when the scanner is looking for a page to swap out for whatever reason (including filesystem activity). Before priority paging it would select a LRU page, which could very likely be the program that is triggering the page fault. With priority paging enabled, a non-executable page will be selected for swapping before an executable one. Therefore the httpd binary will remain in hard-wired memory longer than the anonymous pages it is operating on (including the COW pages inherited in the children and any filesystem cache pages). Without priority paging an httpd process that required access to more memory than physically present on the system could potentially cause itself to be paged out and then immediate back in as soon as the next instruction tripped a page fault. This would result in a huge amount of thrashing, for even something as simple as copying a file (as with your example). So my point is we don't have to do mlock() on solaris8 (or 7 with p.p. enabled) since we get similiar or sufficient behavior already. -aaron
Re: performance: using mlock(2) on httpd parent process
Scott Hess wrote: On Thu, 21 Mar 2002, Stas Bekman wrote: On Wed, 20 Mar 2002, Stas Bekman wrote: mod_perl child processes save a lot of memory when they can share memory with the parent process and quite often we get reports from people that they lose that shared memory when the system decides to page out the parent's memory pages because they are LRU (least recently used, the algorithm used by many memory managers). I'm fairly certain that this is not an issue. If a page was shared COW before being paged out, I expect it will be shared COW when paged back in, at least for any modern OS. But if the system needs to page things out, most of the parent process's pages will be scheduled to go first, no? So we are talking about a constant page-in/page-out from/to the parent process as a performance degradation rather than memory unsharing. Am I correct? The system is going to page out an approximation of the least-recently-used pages. If the children are using those pages, then they won't be paged out, regardless of what the parent is doing. [If the children _aren't_ using those pages, then who cares?] mod_perl users do care. let's say you've preloaded a bunch of modules which aren't in use now but may be used any moment. Ideally we don't want any of the pages to be swapped out. Of course this is all possible if the demand does go over the supply. I'm just repeating the reports posted to the mod_perl list. I've never seen such a problem myself, since I try hard to have close to zero swap usage. :-). In my experience, you can get some really weird stuff happening when you start swapping mod_perl. It seems to be stable in memory usage, though, so long as you have MaxClients set low enough that your maximum amount of committed memory is appropriate. Also, I've seen people run other heavyweight processes, like mysql, on the same system, so that when the volume spikes, mod_perl spikes AND mysql spikes. A sure recipe for disaster. Well, we (mod_perl users) use tools like Apache::GTopLimit and Apache::SizeLimit which monitor the child process' memory and will kill the process after a request if the memory size crossed a total memory used limit, or lost too much of a shared memory. I think that's where the problem lays, the memory usage tools report that the sharing went down and the process gets killed when the parent's pages are getting paged out. Even if in reality the sharing is still there. The memory reporting tools don't reflect the reality. [Yes, please let me know your methodology for testing this] OK, two programs. bigshare.c: thanks Scott, I've forwarded this code to the folks who are having the trouble with. and it'll stay in the archives for others to re-use. Therefore my question is there any reason for not using mlockall(2) in the parent process on systems that support it and when the parent httpd is started as root (mlock* works only within root owned processes). I don't think mlockall is appropriate for something with the heft of mod_perl. Why are the pages being swapped out in the first place? Presumably there's a valid reason. Well, the system coming close to zero of real memory available. The parent process starts swapping like crazy because most of its pages are LRU, slowing the whole system down and if the load doesn't go away, the system takes a whirl down to a halt. I can think of a couple possible causes. One is the MaxClients setting. Say you have MaxClients set to 50, but on a standard day you never need more than 15 servers. Then you get listed on slashdot. As you pass, oh, 30 simultaneous servers, you start thrashing, so requests take longer to process, so you immediately spike to your MaxClients of 50 and the server goes right down the drain. If you shut things down and start them back up, it's likely you'll immediately spike to 50 again, and back down the drain it goes. I've found that it's _super_ important to make certain you've pushed mod_perl to MaxClients under your standard conditions. Once you start swapping, you're doomed, unless the traffic was a very short spike. Another possible cause is that you have another heavyweight server running on the same server. As I indicated above, I've seen people do this with things like mysql. Since high mod_perl traffic implies high mysql traffic, it's just like having MaxClients set too high, but twice as bad! Another possible cause is that the OS is aggressively grabbing pages for the filesystem cache. It's possible that tuning down the size of the filesystem cache would be appropriate - many webservers have a very large maximum amount of data they might server, but a very small working set. Really, though, all of this really tends to come down to MaxClients. The database load is proportional to the number of clients, the filesystem load is proportional to the number of clients, everything is proportional to the number of
Re: performance: using mlock(2) on httpd parent process
On Wed, 20 Mar 2002, Stas Bekman wrote: mod_perl child processes save a lot of memory when they can share memory with the parent process and quite often we get reports from people that they lose that shared memory when the system decides to page out the parent's memory pages because they are LRU (least recently used, the algorithm used by many memory managers). I'm fairly certain that this is not an issue. If a page was shared COW before being paged out, I expect it will be shared COW when paged back in, at least for any modern OS. [To verify that I wasn't talking through my hat, here, I just verified this using RedHat 7.2 running kernel 2.4.9-21. If you're interested in my methodology, drop me an email.] I believe that this applies to all httpd modules and httpd itself, the more we can share the less memory resources are needed, and usually it leads to a better performance. I'm absolutely _certain_ that unmodified pages from executable files will be backed by the executable, and will thus be shared by default. Therefore my question is there any reason for not using mlockall(2) in the parent process on systems that support it and when the parent httpd is started as root (mlock* works only within root owned processes). I don't think mlockall is appropriate for something with the heft of mod_perl. Why are the pages being swapped out in the first place? Presumably there's a valid reason. Doing mlockall on your mod_perl would result in restricting the memory available to the rest of the system. Whatever is causing mod_perl to page out would then start thrashing. Worse, since mlockall will lock down mod_perl pages indiscriminately, the resulting thrashing will probably be even worse than what they're seeing right now. Later, scott
performance: using mlock(2) on httpd parent process
mod_perl child processes save a lot of memory when they can share memory with the parent process and quite often we get reports from people that they lose that shared memory when the system decides to page out the parent's memory pages because they are LRU (least recently used, the algorithm used by many memory managers). I believe that this applies to all httpd modules and httpd itself, the more we can share the less memory resources are needed, and usually it leads to a better performance. Therefore my question is there any reason for not using mlockall(2) in the parent process on systems that support it and when the parent httpd is started as root (mlock* works only within root owned processes). Of course this is relevant to 1.3 and 2.0 httpds. Thanks. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: performance: using mlock(2) on httpd parent process
On Wed, Mar 20, 2002 at 12:02:53PM +0800, Stas Bekman wrote: mod_perl child processes save a lot of memory when they can share memory with the parent process and quite often we get reports from people that they lose that shared memory when the system decides to page out the parent's memory pages because they are LRU (least recently used, the algorithm used by many memory managers). I believe that this applies to all httpd modules and httpd itself, the more we can share the less memory resources are needed, and usually it leads to a better performance. Therefore my question is there any reason for not using mlockall(2) in the parent process on systems that support it and when the parent httpd is started as root (mlock* works only within root owned processes). We don't really support httpd running as root. That is what that whole BIG_SECURITY_HOLE thing is about. If somebody is going to run as root, then let them put the mlockall() patch in themselves. Further, since mlockall() jams everything into memory, I'm wary of making this the default behavior [when running as root]. In turn, that means that we'd have to introduce a new directive to turn on the mlockall thing. So the net result is: new behavior and a new directive to support a case (httpd running as root) that we don't even want/like people doing in the first place. I'd be a heavy -0 on such a thing. Cheers, -g -- Greg Stein, http://www.lyra.org/
Re: performance: using mlock(2) on httpd parent process
Greg Stein wrote: On Wed, Mar 20, 2002 at 12:02:53PM +0800, Stas Bekman wrote: mod_perl child processes save a lot of memory when they can share memory with the parent process and quite often we get reports from people that they lose that shared memory when the system decides to page out the parent's memory pages because they are LRU (least recently used, the algorithm used by many memory managers). I believe that this applies to all httpd modules and httpd itself, the more we can share the less memory resources are needed, and usually it leads to a better performance. Therefore my question is there any reason for not using mlockall(2) in the parent process on systems that support it and when the parent httpd is started as root (mlock* works only within root owned processes). We don't really support httpd running as root. That is what that whole BIG_SECURITY_HOLE thing is about. The parent process is always running as root if you want to bind to port 80. And that's the only process that needs locking with mlockall(). We don't want any locking for child processes. If somebody is going to run as root, then let them put the mlockall() patch in themselves. Further, since mlockall() jams everything into memory, I'm wary of making this the default behavior [when running as root]. In turn, that means that we'd have to introduce a new directive to turn on the mlockall thing. So the net result is: new behavior and a new directive to support a case (httpd running as root) that we don't even want/like people doing in the first place. If this improves the httpd's performance what's so wrong with adding a new directive? I'd be a heavy -0 on such a thing. -- _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
