Re: 'Pinning' the root apache process in memory with mlockall
Ed Grimm wrote: Danger: Rant ahead. Proceed with caution. [my summary of mlocks discussion removed] In the discussion you referred to, all of the people saying this was a bad idea were using terms like, I think. None of them had the situation themselves, so have a difficult time coming to terms with it. None of them had related former experience using this. Something like this really needs to be tested by someone who has the issue, and has the ability to do benchmarks with real data streams. If they find it seems to work well, then they should test it on production systems. Anyone else talking about it is simply that much hot air, myself included. (I *could* test it, but I don't have enough of a problem to put a priority on it. If we were waiting for me to get time, we'd be waiting a long time.) Right, therefore until someone comes up with real numbers and real testing this is all words. I've summarized the discussions here and at the httpd-dev lists. I've never used mlock() and friends myself, so I can only rely on other users' experiences. I cannot do the testing myself now, too many things on my plate already. If you think my summary is not so good (which is definitely can be) you are welcome to improve it. If someone can do the real testing and share the results with the list, everybody will benefit. [the rest of the interesting rant removed] __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: 'Pinning' the root apache process in memory with mlockall
Bill Marrs wrote: At 10:53 PM 3/22/2002, Stas Bekman wrote: top and libgtop use the same source of information, so it has nothing to do with these tools. 'top' has the ability to display SWAP on a per-process basis (you have to change the defaults to see it, but it's there). yeah, that's a cool feature! I didn't find this per-process SWAP value in Gtop.pm anywhere. If GTop.pm had it, I could fix GTopLimit's bug. GTop.pm is a Perl glue to the libgtop Gnome C library. If somebody adds the SWAP/process feature this will be available in the GTop.pm. Any takers? __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: 'Pinning' the root apache process in memory with mlockall
At 10:53 PM 3/22/2002, Stas Bekman wrote: top and libgtop use the same source of information, so it has nothing to do with these tools. 'top' has the ability to display SWAP on a per-process basis (you have to change the defaults to see it, but it's there). I didn't find this per-process SWAP value in Gtop.pm anywhere. If GTop.pm had it, I could fix GTopLimit's bug. -bill
'Pinning' the root apache process in memory with mlockall
Recently on this list the idea of 'pinning' or locking the root apache process in memory has been discussed with some interest. The reason being was that some users have experienced the situtaion where a server becomes loaded, and the root apache process gets swapped out, and in the process loses some of its shared memory. Future child processes that are forked also share in the loss of shared memory, so methods like using GtopLimit to 'recyle' child processes when their shared memory becomes too low cease to work because when they come up, they are already too low on shared memory. In our systems we had attempted this but it always came down to the same problem --the root process would lose its shared memory, to the point that any child process would come up, serve a request, find that it was beyond the threshold for shared memory and die. The only help was to restart Apache altogether. So in scouring the list I found someone mentioning using the mlockall C function to lock the pages of the core apache process in memory. Some handy .xs code was provided, so I built a module, Sys::Mman, which wraps mlockall, and makes it available to Perl. We installed this on our servers, and call mlockall right at the end of our preload stuff, i.e., the end of the 'startup.pl'-style script called from httpd.conf. The result has been very encouraging. The core apache process is able then to maintain all its shared memory, and child processes that are forked are able to start with high amounts of shared memory, all making for a much happier system. Now I also read that probably better than this would be to ensure that you never swap by tuning MaxClients, as well as examining our Perl code to make it less prone to lose shared memory. We're working on that sort of tuning, but in volatile environments like ours, where we serve a very large amount of data, and new code is coming out almost daily here and there, locking the core httpd in memory has been very helpful. I just thought I would let others know on the list that it is feasible, and works well in our environment. If there's enough interest I might put the module up on CPAN, but it's really very simple. h2xs did most of the work for me. And thanks to Doug MacEachern for posting the .xs code. It worked like a charm. FWIW, -- Dan Hanks Daniel Hanks - Systems/Database Administrator About Inc., Web Services Division
Re: 'Pinning' the root apache process in memory with mlockall
Daniel Hanks wrote: Recently on this list the idea of 'pinning' or locking the root apache process in memory has been discussed with some interest. The reason being was that some users have experienced the situtaion where a server becomes loaded, and the root apache process gets swapped out, and in the process loses some of its shared memory. Future child processes that are forked also share in the loss of shared memory, so methods like using GtopLimit to 'recyle' child processes when their shared memory becomes too low cease to work because when they come up, they are already too low on shared memory. In our systems we had attempted this but it always came down to the same problem --the root process would lose its shared memory, to the point that any child process would come up, serve a request, find that it was beyond the threshold for shared memory and die. The only help was to restart Apache altogether. So in scouring the list I found someone mentioning using the mlockall C function to lock the pages of the core apache process in memory. Some handy .xs code was provided, so I built a module, Sys::Mman, which wraps mlockall, and makes it available to Perl. We installed this on our servers, and call mlockall right at the end of our preload stuff, i.e., the end of the 'startup.pl'-style script called from httpd.conf. The result has been very encouraging. The core apache process is able then to maintain all its shared memory, and child processes that are forked are able to start with high amounts of shared memory, all making for a much happier system. Now I also read that probably better than this would be to ensure that you never swap by tuning MaxClients, as well as examining our Perl code to make it less prone to lose shared memory. We're working on that sort of tuning, but in volatile environments like ours, where we serve a very large amount of data, and new code is coming out almost daily here and there, locking the core httpd in memory has been very helpful. I just thought I would let others know on the list that it is feasible, and works well in our environment. If there's enough interest I might put the module up on CPAN, but it's really very simple. h2xs did most of the work for me. And thanks to Doug MacEachern for posting the .xs code. It worked like a charm. See the discussion on the [EMAIL PROTECTED] list, http://marc.theaimsgroup.com/?t=10165973081r=1w=2 where it was said that it's a very bad idea to use mlock and variants. Moreover the memory doesn't get unshared when the parent pages are paged out, it's the reporting tools that report the wrong information and of course mislead the the size limiting modules which start killing the processes. As a conclusion to this thread I've added the following section to the performance chapter of the guide: =head3 Potential Drawbacks of Memory Sharing Restriction It's very important that the system won't be heavily engaged in swapping process. Some systems do swap in and out every so often even if they have plenty of real memory available and it's OK. The following applies to conditions when there is hardly any free memory available. So if the system uses almost all of its real memory (including the cache), there is a danger of parent's process memory pages being swapped out (written to a swap device). If this happens the memory usage reporting tools will report all those swapped out pages as non-shared, even though in reality these pages are still shared on most OSs. When these pages are getting swapped in, the sharing will be reported back to normal after a certain amount of time. If a big chunk of the memory shared with child processes is swapped out, it's most likely that CApache::SizeLimit or CApache::GTopLimit will notice that the shared memory floor threshold was crossed and as a result kill those processes. If many of the parent process' pages are swapped out, and the newly created child process is already starting with shared memory below the limit, it'll be killed immediately after serving a single request (assuming that we the C$CHECK_EVERY_N_REQUESTS is set to one). This is a very bad situation which will eventually lead to a state where the system won't respond at all, as it'll be heavily engaged in swapping process. This effect may be less or more severe depending on the memory manager's implementation and it certainly varies from OS to OS, and different kernel versions. Therefore you should be aware of this potential problem and simply try to avoid situations where the system needs to swap at all, by adding more memory, reducing the number of child servers or spreading the load across more machines, if reducing the number of child servers is not an options because of the request rate demands. __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide ---
Re: 'Pinning' the root apache process in memory with mlockall
On Sat, 23 Mar 2002, Stas Bekman wrote: See the discussion on the [EMAIL PROTECTED] list, http://marc.theaimsgroup.com/?t=10165973081r=1w=2 where it was said that it's a very bad idea to use mlock and variants. Moreover the memory doesn't get unshared when the parent pages are paged out, it's the reporting tools that report the wrong information and of course mislead the the size limiting modules which start killing the processes. As a conclusion to this thread I've added the following section to the performance chapter of the guide: Are we saying then that libgtop is erroneous in its reporting under these circumstances? And in the case of Linux, I'm asusming libgtop just reads its info straight from /proc. Is /proc erroneous then? -- Dan Daniel Hanks - Systems/Database Administrator About Inc., Web Services Division
Re: 'Pinning' the root apache process in memory with mlockall
Stas, Thanks for tracking that down. So, the problem is our tools. For me, that's GTopLimit (but also SizeLimit). I would think it must be possible to cajole these two into realizing their error. top seems to know how much a process has swapped. If GTopLimit could know that, the number could be subtracted from the total used in calculating the amount of sharing (and the new unshared), then this bug would be resolved, right? I looked, but didn't see anything in Gtop.pm that gives swap per process, though. So, it's not going to be easy. I guess I'll turn of me deswapper... ...and GTopLimit as well. for now... hmm, maybe I could just avoid using the share-related trigger values in GTopLimit, and just use the SIZE one. That would be an acceptable compromise, though not the best. -bill
Re: 'Pinning' the root apache process in memory with mlockall
Stas Bekman wrote: Moreover the memory doesn't get unshared when the parent pages are paged out, it's the reporting tools that report the wrong information and of course mislead the the size limiting modules which start killing the processes. Apache::SizeLimit just reads /proc on Linux. Is that going to report a shared page as an unshared page if it has been swapped out? Of course you can void these issues if you tune your machine not to swap. The trick is, you really have to tune it for the worst case, i.e. look at the memory usage while beating it to a pulp with httperf or http_load and tune for that. That will result in MaxClients and memory limit settings that underutilize the machine when things aren't so busy. At one point I was thinking of trying to dynamically adjust memory limits to allow processes to get much bigger when things are slow on the machine (giving better performance for the people who are on at that time), but I never thought of a good way to do it. - Perrin
RE: 'Pinning' the root apache process in memory with mlockall
Stas Bekman wrote: Moreover the memory doesn't get unshared when the parent pages are paged out, it's the reporting tools that report the wrong information and of course mislead the the size limiting modules which start killing the processes. Apache::SizeLimit just reads /proc on Linux. Is that going to report a shared page as an unshared page if it has been swapped out? Of course you can void these issues if you tune your machine not to swap. The trick is, you really have to tune it for the worst case, i.e. look at the memory usage while beating it to a pulp with httperf or http_load and tune for that. That will result in MaxClients and memory limit settings that underutilize the machine when things aren't so busy. At one point I was thinking of trying to dynamically adjust memory limits to allow processes to get much bigger when things are slow on the machine (giving better performance for the people who are on at that time), but I never thought of a good way to do it. Ooh... neat idea, but then that leads to a logical set of questions: Is MaxClients that can be changed at runtime? If not, would it be possible to see about patches to set this? :-) L8r Rob #!/usr/bin/perl -w use Disclaimer qw/:standard/;
Re: 'Pinning' the root apache process in memory with mlockall
Danger: Rant ahead. Proceed with caution. On Sat, 23 Mar 2002, Stas Bekman wrote: See the discussion on the [EMAIL PROTECTED] list, http://marc.theaimsgroup.com/?t=10165973081r=1w=2 where it was said that it's a very bad idea to use mlock and variants. Moreover the memory doesn't get unshared when the parent pages are paged out, it's the reporting tools that report the wrong information and of course mislead the the size limiting modules which start killing the processes. As a conclusion to this thread I've added the following section to the performance chapter of the guide: =head3 Potential Drawbacks of Memory Sharing Restriction It's very important that the system won't be heavily engaged in swapping process. Some systems do swap in and out every so often even if they have plenty of real memory available and it's OK. The following applies to conditions when there is hardly any free memory available. So if the system uses almost all of its real memory (including the cache), there is a danger of parent's process memory pages being swapped out (written to a swap device). If this happens the memory usage reporting tools will report all those swapped out pages as non-shared, even though in reality these pages are still shared on most OSs. When these pages are getting swapped in, the sharing will be My Solaris 2.6 box, while in this situation, was swapping hard, as measured by my ears, by iostat, and by top (both iowait and the memory stats). Note that mlockall does not restrict memory sharing, it restricts swapping a certain portion of memory. This will prevent this memory from ever being needlessly unshared. In the discussion you referred to, all of the people saying this was a bad idea were using terms like, I think. None of them had the situation themselves, so have a difficult time coming to terms with it. None of them had related former experience using this. Something like this really needs to be tested by someone who has the issue, and has the ability to do benchmarks with real data streams. If they find it seems to work well, then they should test it on production systems. Anyone else talking about it is simply that much hot air, myself included. (I *could* test it, but I don't have enough of a problem to put a priority on it. If we were waiting for me to get time, we'd be waiting a long time.) Yes, I agree, it's better to never swap. But if we take the attitude that we won't use tools to help us when times are tight, then get rid of swap entirely. Locking memory is all about being selective about what you will and won't swap. Yes, I agree, it'd be better to mlock those bits of memory that you really care about, but that's hard to do when that memory is allocated by software you didn't write. (In this case, I'd really like to mlock all the memory that perl allocated but did not free in BEGIN sections (including, of course, use blocks). I would also like to compact that first, but that could be even more difficult.) As far as the logic regarding 'let the OS decide' - the admin of the system has the ability to have a much better understanding of how the system resources are used. If I have one section of memory which is used 95% of the time by 75% of my active processes, I really don't want that memory to swap out just because another program that'll only run for a minute wants a bit more memory, if it can take that memory from anywhere else. When doing individual page-ins, memory managers tend to worry only about those processes that they are trying to make runable now; they're not going to go and load that page back on to every other page map that shares it just because they also use it. So even though that memory is loaded back into memory, all those processes will still have to swap it back. For them to do otherwise would be irresponsible, unless the system administrator clearly doesn't know how to system administrate, or has chosen not to. The OS is supposed to handle the typical case; having one segment of memory used by dozens of processes actively is not the typical case. This does not happen on end-user machines; this only happens on servers. Theoretically speaking, servers are run by people who can analyze and tune; mlock and mlockall are tools available to them to do such tuning. reported back to normal after a certain amount of time. If a big chunk of the memory shared with child processes is swapped out, it's most likely that CApache::SizeLimit or CApache::GTopLimit will notice that the shared memory floor threshold was crossed and as a result kill those processes. If many of the parent process' pages are swapped out, and the newly created child process is already starting with shared memory below the limit, it'll be killed immediately after serving a single request (assuming that we the C$CHECK_EVERY_N_REQUESTS is set to one). This is a very bad situation which will eventually lead to a state where the system won't
Re: 'Pinning' the root apache process in memory with mlockall
Daniel Hanks wrote: On Sat, 23 Mar 2002, Stas Bekman wrote: See the discussion on the [EMAIL PROTECTED] list, http://marc.theaimsgroup.com/?t=10165973081r=1w=2 where it was said that it's a very bad idea to use mlock and variants. Moreover the memory doesn't get unshared when the parent pages are paged out, it's the reporting tools that report the wrong information and of course mislead the the size limiting modules which start killing the processes. As a conclusion to this thread I've added the following section to the performance chapter of the guide: Are we saying then that libgtop is erroneous in its reporting under these circumstances? And in the case of Linux, I'm asusming libgtop just reads its info straight from /proc. Is /proc erroneous then? As people have pointed out it's not libgtop, it's /proc. You have the same problem with top(1). It's not erroneous, it just doesn't reflect the immidiate change, the /proc will be updated when pages in question will be accessed which for performance reasons doesn't happen immediately. But this could be too late for the processes that are going to be killed. I've posted the C code to test this earlier this week here: http://marc.theaimsgroup.com/?l=apache-modperlm=101667859909389w=2 You are welcome to run more tests and report back. __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: 'Pinning' the root apache process in memory with mlockall
Perrin Harkins wrote: Stas Bekman wrote: Moreover the memory doesn't get unshared when the parent pages are paged out, it's the reporting tools that report the wrong information and of course mislead the the size limiting modules which start killing the processes. Apache::SizeLimit just reads /proc on Linux. Is that going to report a shared page as an unshared page if it has been swapped out? That's what people report. Try the code here: http://marc.theaimsgroup.com/?l=apache-modperlm=101667859909389w=2 to reproduce the phenomena in a few easy steps Of course you can void these issues if you tune your machine not to swap. The trick is, you really have to tune it for the worst case, i.e. look at the memory usage while beating it to a pulp with httperf or http_load and tune for that. That will result in MaxClients and memory limit settings that underutilize the machine when things aren't so busy. At one point I was thinking of trying to dynamically adjust memory limits to allow processes to get much bigger when things are slow on the machine (giving better performance for the people who are on at that time), but I never thought of a good way to do it. This can be done in the following way: move the variable that controls the limit into a shared memory. Now run a special monitor process that will adjust this variable, or let each child process to do that in the cleanup stage. To dynamically change MaxClients one need to re-HUP the server. __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
