Re: [RFC] Swapping Prevention
On Tue, 20 Jun 2000, Joshua Chamas wrote: your machine. Therefore you should configure the server, so that the maximum number of possible processes will be small enough using the CMaxClients directive. This will ensure that at the peak hours the system won't swap. Remember that swap space is an emergency pool, not a resource to be used routinely. If you are low on memory and you badly need it, buy it or reduce a number of processes to prevent swapping. One common mistake that people make is to not load test against a server to trigger the full MaxClients in production. In order to prevent swapping, one must simulate the server at its point of greatest RAM stress, and one can start to do this by running ab against a program so that each of the MaxClient httpd processes goes through its MaxRequests. So lets say MaxRequests is set to 1000 and MaxClients is set to 100, then in order to see the system go through a full cycle fully maxed out in RAM, one might ab -c 100 -n 10 http://yoursite.loadtest/modperl/ramhog Then fire up top, sit back, and enjoy the show! In summary this aids in swapping prevention, by load testing the server under highest RAM stress preproduction, and seeing if it swaps. Then one can tune their parameters to avoid swapping under this highest load, by decreasing MaxClients or MaxRequests. Hmm, good points Joshua. But just running the above example is not enough. Remember that when running the production server code become unshared, more memory is used, therefore the above test won't protect ensure that swapping won't happen. Unless you can reproduce a production server load, including different queries that will lead to the very close to real simulation of the server usage your advice is definitely helps but not as much as one would want. I think that it'd be really nice to have a post processing handler that will log all the requests including the query data and feed it into a database on the constant basis, merging with the previously identicall inputs and bumping out the count. So when you will want to reproduce the real load of your server, the other program will analyze this database and will generate the output like this: input % -- URL1?query1 46.1% URL1?query2 12.1% URL2?query37.2% URL2?query44.2% ... URLN?queryN 0.2% --- 100% so this input can be fed into the program that runs ab. So if you want to simulate 1 requests, you will run 1*0.461 URLs of the request in the first row above, 1*0.121 of the second, etc... The you will get to tune your server to the maximum performance given that the behavior of the users is not about to change. The older your database is the more correct statistics you will get. Cith help of real usage statistics, in addition ability to fine tune the MaxClient directive, you will learn what sections of code are mostly used and therefore optimize them. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: [RFC] Swapping Prevention
A nit: the distinction between paging and swapping doesn't seem clear to me. You describe the paging process, then talk about how you should never swap. Or maybe that's too detailed for your intended audience. - Barrie
Re: [RFC] Swapping Prevention
your machine. Therefore you should configure the server, so that the maximum number of possible processes will be small enough using the CMaxClients directive. This will ensure that at the peak hours the system won't swap. Remember that swap space is an emergency pool, not a resource to be used routinely. If you are low on memory and you badly need it, buy it or reduce a number of processes to prevent swapping. One common mistake that people make is to not load test against a server to trigger the full MaxClients in production. In order to prevent swapping, one must simulate the server at its point of greatest RAM stress, and one can start to do this by running ab against a program so that each of the MaxClient httpd processes goes through its MaxRequests. So lets say MaxRequests is set to 1000 and MaxClients is set to 100, then in order to see the system go through a full cycle fully maxed out in RAM, one might ab -c 100 -n 10 http://yoursite.loadtest/modperl/ramhog Then fire up top, sit back, and enjoy the show! In summary this aids in swapping prevention, by load testing the server under highest RAM stress preproduction, and seeing if it swaps. Then one can tune their parameters to avoid swapping under this highest load, by decreasing MaxClients or MaxRequests. -- Joshua _ Joshua Chamas Chamas Enterprises Inc. NodeWorks free web link monitoring Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051 Stas Bekman wrote: I've rewritten the guide's swapping section now delving into quite technical details. Please take a moment to review this section and comment on it. I have tried to stay on focus to deliver as much as possible relevant details without actually delving into the fine details of memory management techniques like virtual memory, sharing and alike, therefore this version is a simplified one. I hope that it'll be still educating and clear for folks with no CS background. The goal is to helps to understand the reasoning behind desired swapping prevention under mod_perl. Submitting of the additional techniques/details and possible problems/solutions that I've not covered or covered incorrectly is very welcome. Remember that I'm not trying to reproduce the operating systems course but provide only the relevant details. Thanks a lot! So here we go: =head2 Swapping Prevention Before we delve into swapping process details, let's refresh our memory management knowledge. The computer memory is called RAM, which stands for Random Access Memory. Reading and writing to RAM is by a few orders faster than doing the same operations with a hard disk, the former uses the non-movable memory cells, while the latter uses the rotating magnetic media. On most operating systems a swap memory is used as an extension for RAM and not as a duplication of it. So if your OS is one of those, if you have 128MB of RAM and 256MB swap partition, you have a total of 384MB of memory available for OS. You should never count this extra memory when you decide on the maximum number of processes to be run, and we will show why in the moment. The swapping memory can be built of a number of hard disk partitions and swap files formatted to be used as a swap memory. When you need more swap memory you can always extend it on demand as long as you have some free disk space (for more information see the Imkswap and Iswapon manpages). System memory is quantified in units called memory pages. Usually the size of the memory page is 4KB or 8KB. So if you have 256MB of RAM installed on your machine and the page size is 4KB your system has 64,000 memory pages to work with. When the system is started all memory pages are available for use by the programs (processes). When a process asks for a specific memory page, the system checks whether this page is already loaded in memory and if it's not, the Ipage fault event occurs, which requires the system to allocate a free memory page and load this page into memory. Each program asks a system for a certain number of the RAM memory pages when it started. For example if a program needs 512KB of memory when it starts and the memory page size is 4KB, the system will have to allocate 128 memory pages for this program. Note that if some of the pages can be shared with other processes it's possible that less pages will be needed. For examples pages that are occupied by all kind of Ilibc libraries are generally shared between the processes that use these libraries. After the process was started it might ask the OS for additional memory pages when it needs them. While there are memory pages available, the OS allocates these demanded memory pages and delivers them to the process that asks for them. Most programs, particularly Perl programs, on most modern OSs don't return memory pages
Re: [RFC] Swapping Prevention
On Tue, 20 Jun 2000, Joshua Chamas wrote:
your machine. Therefore you should configure the server, so that the
maximum number of possible processes will be small enough using the
CMaxClients directive. This will ensure that at the peak hours the
system won't swap. Remember that swap space is an emergency pool, not
a resource to be used routinely. If you are low on memory and you
badly need it, buy it or reduce a number of processes to prevent
swapping.
One common mistake that people make is to not load
test against a server to trigger the full MaxClients
in production. In order to prevent swapping, one must
simulate the server at its point of greatest RAM stress,
and one can start to do this by running ab against
a program so that each of the MaxClient httpd processes
goes through its MaxRequests.
attached is a little perl script (3.5k) that I use to "replay" a log file
as fast as it will go... the script uses Sys-5 message queues and forks
off the number of children specified by -n to read these messages and make
the given request... Since we use it for benchmarking machines hosting
lots of virtual servers, the input file format is actually
host uri status size
(space separated) and then it whines about responses that don't match the
status or have a vastly differing size...
This version isn't as polished as the previous one was... but I left that
under /tmp/ for too long :-(
Anyhow, my beef with ab is that it just pounds on one page (but it's
very good at that).
Anyhow, I haven't generally used it for testing MaxClients settings, but
that would just mean increasing the -n parameter... it is
_simple_ and doesn't deal with stuff like keep alives, and it's
not the fastest client, so it would be plain stupid to try to use
this to benchmark the number of GIFs per second a faster CPU
could serve. Basically, we generally use for correctness testing rather
than all out performance.
-Tom
p.s. Options...
# -n = num threads;
# -v = verbose
# -d = debug
# -s = allowed size variance (bytes)
# -p = allowed size variance (%)
# -w = warn if response takes more than this seconds
#!/usr/bin/perl
#
# script to read in a file of "host uri status size"
# lines, and feed the requests to a pool of child processes via Sys5
# message queue.
# constants
my $MSG_NORM = 2;
my $MSG_EXIT = 1;
my $MSG_SIZE = 1;
my $KEY = 1984; # use fixed key instead of IPC_PRIVATE
my $TIMEOUT = 10;
# use init
use strict;
use IPC::SysV qw(IPC_CREAT S_IRWXU S_IRWXG S_IRWXO IPC_NOWAIT);
use IPC::Msg;
use IO::Socket::INET;
use Getopt::Std;
use vars qw($opt_n $opt_v $opt_d $opt_s $opt_p $opt_w);
getopts("n:vdw:s:p:");
# -n = num threads;
# -v = verbose
# -d = debug
# -s = allowed size variance (bytes)
# -p = allowed size variance (%)
# -w = warn if response takes more than this seconds
$opt_v++ if ($opt_d);
# defaults
$opt_n = 10 unless ($opt_n);
$opt_p = 5 unless (defined $opt_p);
$opt_s = 50 unless (defined $opt_s);
$opt_w = 0 unless (defined $opt_w);
my %DNS_HASH = ();
##
my $msg = new IPC::Msg($KEY, IPC_CREAT | S_IRWXU | S_IRWXG | S_IRWXO)
or die "message queue creation failed! ($!)";
# drain queue, since it might have old messages in it.
{ my $i=1e6;
my $buffer;
while ($i-- 0 $msg-rcv($buffer, $MSG_SIZE, $MSG_NORM, IPC_NOWAIT) ) {
# drop it ;-)
}
while ($i-- 0 $msg-rcv($buffer, $MSG_SIZE, $MSG_EXIT, IPC_NOWAIT) ) {
# drop it ;-)
}
}
my $pid;
for (my $i=$opt_n; $i 0; $i--) {
$pid=fork();
die "fork failed ($!)" if ($pid 0);
last if ($pid ==0); # child
}
if ($pid != 0) { # parent
# send messages
while (defined (my $line = )) {
$msg-snd($MSG_NORM,$line,undef); # tough huh?
}
for (my $i=$opt_n; $i 0; $i--) {
$msg-snd($MSG_EXIT,"goodbye",undef); # one for each child
}
while ( wait() 0 ) {
# reap children.
}
$msg-remove; # remove the message queue
print "parent finished\n" if ($opt_v);
} else { # children
child:
while ( 1 ) { # loop until
my $buffer = '';
my $timeout = $TIMEOUT;
while ($timeout 0) {
if ( $msg-rcv($buffer, $MSG_SIZE, $MSG_NORM, IPC_NOWAIT) ) {
print "$$: $buffer\n" if ($opt_d);
get_request($buffer);
next child;
} else {
last child if( $msg-rcv($buffer, $MSG_SIZE, $MSG_EXIT,
IPC_NOWAIT));
sleep 1;
$timeout--;
}
}
}
print "child $$: finished\n" if ($opt_v);
}
