Re: killing of greater than MaxSpareServers
Cliff,
Here's the diff of src/main/http_main.c No children are spawned if the
file /tmp/spawn.lock is readable by the root webserver.
I also post yet another watchdog script. It touches the spawnlock file and
kills/shuts down too large apache children.
It uses the /proc filesystem heavily, so might not be too portable.
Enjoy,
-Balazs
[balazs@felix apache_1.3.14]$ diff src/main/http_main.c.ORIG src/main/http_main.c
4419a4420,4423
> /* BALAZSD */
> int spawn_lock=0;
> FILE *SPAWN;
> /* BALAZSD */
4483a4488,4495
>
> /* BALAZSD */
> if( SPAWN=fopen( "/tmp/spawn.lock", "r" )){
> fclose( SPAWN );
> spawn_lock = 1;
> }
> /* BALAZSD */
>
4506a4519,4526
> /* BALAZSD */
> else if ( spawn_lock ) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, server_conf,
>"Server would've spawned, but lockfile found");
>
> idle_spawn_rate = 1;
> }
> /* BALAZSD */
#!/usr/bin/perl
use strict;
# = D E C L A R A T I O N S ===
#These are the variables that are worth changing through USR2
use vars qw( $STOPPED $INTERVAL $DEFAULT $USR1LIM $TERMLIM $SPAWNLIM );
# Bad globals...
use vars qw( $opt_i $opt_s $FREEMEM $LOADAVG $CPUUTIL $CHILDINFO $SPAWNLOCK_SET
@SIGKILLNEXT);
# C O N F I G =
# File to log to using append
my $LOGFILE= "/tmp/balazsd.log";
# File that stops apache from spawning news kids
my $SPAWNLOCK = "/tmp/spawn.lock";
require 'getopts.pl';
&Getopts('i:s:');
# Default interval at which processes are checked
$INTERVAL = $opt_i || 3;
# === E N D O F C O N F I G =
# Detach
exit if fork();
`renice -20 $$`;
# Or have some other way to run this as max priority
# For example start as root and then change user to web
$| = 1;
my $scriptname = $opt_s || usage();
logme ("Monitoring <$scriptname> at $INTERVAL seconds with PID #$$\n");
$SIG{TERM} = sub { $STOPPED = 1 };
$SIG{USR1} = sub { $DEFAULT = 1 };
my $default_action = \&autopilot_action;
$SIG{USR2} = \&handle_usr2;
# Default values for the limits
$USR1LIM = 45_000_000; # send SIGUSR1 to a process if uses more mem then
$USR1LIM
$TERMLIM = 80_000_000; # || SIGTERM ||
$SPAWNLIM = 30_000_000; # Stop apache spawning new kids if free mem <
$SPAWNLIM
my ($oldstarttime, $oldstartproc, $starttime, $startproc);
while( not $STOPPED ){
( $oldstarttime, $oldstartproc ) = ($starttime, $startproc);
($starttime, $startproc) = getproctime();
$CPUUTIL = 100 * (1-( $startproc-$oldstartproc ) / (
$starttime-$oldstarttime ));
gather_data();
if( $DEFAULT ){
logme("!!! AUTOPILOT MODE !!!");
&$default_action();
}
else {
eval {action()};
logme("action ERROR: $@") if $@;
}
my $currtime;
($currtime, undef) = getproctime();
my $sleeptime = $starttime + $INTERVAL - $currtime;
select undef, undef, undef, $sleeptime if $sleeptime > 0;
}
logme ("Finished successfully\n");
exit 0;
# === S U B S ===
sub usage {
warn "Usage: $0 -s scriptname [ -i interval ]\n";
exit 1;
}
sub logme {
my $extralinefeed = $_[$#_] =~ /\n$/ ? "" : "\n";
open LOGFILE, ">>$LOGFILE";
print LOGFILE "balazsd: ", scalar localtime, ": @_$extralinefeed";
close LOGFILE;
}
sub getproctime {
unless ( open (UPTIME, "/proc/uptime")){
logme ("getproctime failed: $!\n");
return ( 0 , 0 );
}
my ($rtime, $utime) = split " ", ;
close UPTIME;
($rtime, $utime);
}
sub handle_usr2 {
do "/tmp/balazsd.eval";
# FIXME
`(echo; echo;echo -n "At ";date;cat /tmp/balazsd.eval;echo =) >>
/tmp/balazsd.eval.log`;
if( $@ ){
logme("handle_usr2 ERROR $@");
}
else {
logme("Caught USR2 and evaled code successfully.");
}
}
# This is the piece of code that is always executed when
# $DEFAULT == 1 no matter what was evaled.
sub autopilot_action {
my $lockfile = $FREEMEM < $SPAWNLIM ? 1 : 0;
if( $lockfile ){
open SPAWNLOCK , ">$SPAWNLOCK";
print SPAWNLOCK ".";
close SPANLOCK;
logme("Setting SPANLOCK") if ! $SPAWNLOCK_SET;
$SPAWNLOCK_SET = 1;
}
else {
unlink $SPAWNLOCK;
logme("Releasing SPANLOCK") if $SPAWNLOCK_SET;
$SPAWNLOCK_SET = 0;
}
for ( @SIGKILLNEXT ){
if( kill 0, $_){
kill 9, $_ ;
logme (" $_ SIGKILLED");
}
}
@SIGKILLNEXT = ();
for my $pid ( keys %{$CHILDINFO->{children}} ){
if( $CHILDINFO->{children}{$pid}{mem} > $TERMLIM ){
kill 15, $pid;
push @SIGKILLNEXT, $pid;
logme(" $pid SIGTERM-ed");
}
elsif ( $CHILDINFO->{children}{$pid}{mem} > $USR1LIM ){
kill 10, $pid;
logme(" $pid SIGUSR1-ed");
}
}
}
# The default for action is of course autopilot_action
sub action {
autopilot_action();
}
sub gather_data {
&read_loadavg();
&read_freemem();
&read_childinfo();
logme("MEM: $FREEMEM LOADAVG: $LOADAVG CPUUTIL: $CPUUTIL");
}
Re: killing of greater than MaxSpareServers
i think tuning like this is available already. probably something like: MinSpareServers 100 MaxSpareServers 110 StartServers 100 MaxClients 110 i haven't checked the source code to see if these values might drive apache a little crazy (such a small range to work in) , but it seems like the above would keep apache between 100-110 servers at all times. you'd have to make sure Apache::GtopLimit wasn't killing processes off too early or their would be lots of killing and respawning of processes. -- ___cliff [EMAIL PROTECTED]http://www.genwax.com/ Jayme Frye wrote: > This may not be quite on topic. I wonder in the case of a dedicated > Apache/Mod_Perl server if it wouldn't be best to spawn up to a maximum > number of child procs that the server can handle at startup. Maximum > child procs calculated based on the amount of core memory minus amount > of real memory used by each proc (Stas Bekman mod_perl guide Performance > Tuning Section) with a cushion of free core memory to allow for some > process growth and handle misc. house keeping duties of the server. Then > use the Apache::GtopLimit module to prune child procs based on memory > size. If a child proc grows too large because of memory leaks for > example it is killed and a fresh proc spawned in its place. In this way > you always have as many procs available as your server can handle for > periods of high load. No wasted time spawning up to meet the load. > > Jayme Frye > System Administrator > Inventive Comm.
Re: killing of greater than MaxSpareServers
Perrin Harkins wrote: > On Wed, 17 Jan 2001, ___cliff rayman___ wrote: > >> i and others have written on the list before, that pushing apache >> children into swap causes a rapid downward spiral in performance. I >> don't think that MaxClients is the right way to limit the # of >> children. i think MaxSpareCoreMemory would make more sense. You >> could set this to 1K if your server was designated for Apache only, or >> set it to a higher value if it were a multipurpose machine. > > > I've thought about that too. The trick is, Apache would need to know > things about your application to do that right. It would need to know how > big your processes were likely to be and how big they could get before > they die. Otherwise, it has no way of knowing whether or not there's > enough room for another process. > > A combination of Apache::SizeLimit and a dynamically changing MaxClients > could possibly accomplish this, but you wouldn't want to run it too close > to the edge since you don't want to have to axe a process that's in the > middle of doing something just because it got a little too big (i.e. no > hard limits on per-process memory usage). > > You can't change MaxClients while the server is running, can you? > > - Perrin This may not be quite on topic. I wonder in the case of a dedicated Apache/Mod_Perl server if it wouldn't be best to spawn up to a maximum number of child procs that the server can handle at startup. Maximum child procs calculated based on the amount of core memory minus amount of real memory used by each proc (Stas Bekman mod_perl guide Performance Tuning Section) with a cushion of free core memory to allow for some process growth and handle misc. house keeping duties of the server. Then use the Apache::GtopLimit module to prune child procs based on memory size. If a child proc grows too large because of memory leaks for example it is killed and a fresh proc spawned in its place. In this way you always have as many procs available as your server can handle for periods of high load. No wasted time spawning up to meet the load. Jayme Frye System Administrator Inventive Comm.
Re: killing of greater than MaxSpareServers
i think its worth posting to the list. it will be forever in the archives when someone needs it. thanks! Balazs Rauznitz wrote: > On Wed, 17 Jan 2001, ___cliff rayman___ wrote: > > > i and others have written on the list before, that pushing apache > > children into swap causes a rapid downward spiral in performance. > > I don't think that MaxClients is the right way to limit the # of children. i think > > MaxSpareCoreMemory would make more sense. You could > > set this to 1K if your server was designated for Apache > > only, or set it to a higher value if it were a multipurpose machine. > > mod_perl/apache and paging/swapping just don't mix. > > Once I wrote a patch to apache so that it would not spawn new children if > a certain file was present in the filesystem. You can then have a watchdog > process touch or delete that file based on any criteria you want. Imo > having a separate and flexible process is better than apache trying to > make these decisions... > I'll dig it up if interested. > > -Balazs -- ___cliff [EMAIL PROTECTED]http://www.genwax.com/
Re: killing of greater than MaxSpareServers
On Wed, 17 Jan 2001, ___cliff rayman___ wrote: > i and others have written on the list before, that pushing apache > children into swap causes a rapid downward spiral in performance. > I don't think that MaxClients is the right way to limit the # of children. i think > MaxSpareCoreMemory would make more sense. You could > set this to 1K if your server was designated for Apache > only, or set it to a higher value if it were a multipurpose machine. > mod_perl/apache and paging/swapping just don't mix. Once I wrote a patch to apache so that it would not spawn new children if a certain file was present in the filesystem. You can then have a watchdog process touch or delete that file based on any criteria you want. Imo having a separate and flexible process is better than apache trying to make these decisions... I'll dig it up if interested. -Balazs
Re: killing of greater than MaxSpareServers
if you are able to determine how much core memory is left, you may also be able to determine average apache process size and variance. then, apache can determine whether or not to start up any additional children. i'm not sure how much processor time would be taken to determine free core memory. might not be something worth doing on every scoreboard cycle. there is an additional bonus for determining child size mean and variance. if it is noted that a process is growing exceptionally large compared to other processes, it could be slated for death based on its growth size and rate, rather than the number of requests that it served. Perrin Harkins wrote: > On Wed, 17 Jan 2001, ___cliff rayman___ wrote: > > i and others have written on the list before, that pushing apache > > children into swap causes a rapid downward spiral in performance. I > > don't think that MaxClients is the right way to limit the # of > > children. i think MaxSpareCoreMemory would make more sense. You > > could set this to 1K if your server was designated for Apache only, or > > set it to a higher value if it were a multipurpose machine. > > I've thought about that too. The trick is, Apache would need to know > things about your application to do that right. It would need to know how > big your processes were likely to be and how big they could get before > they die. Otherwise, it has no way of knowing whether or not there's > enough room for another process. > > A combination of Apache::SizeLimit and a dynamically changing MaxClients > could possibly accomplish this, but you wouldn't want to run it too close > to the edge since you don't want to have to axe a process that's in the > middle of doing something just because it got a little too big (i.e. no > hard limits on per-process memory usage). > > You can't change MaxClients while the server is running, can you? > > - Perrin -- ___cliff [EMAIL PROTECTED]http://www.genwax.com/
Re: killing of greater than MaxSpareServers
On Wed, 17 Jan 2001, ___cliff rayman___ wrote: > i and others have written on the list before, that pushing apache > children into swap causes a rapid downward spiral in performance. I > don't think that MaxClients is the right way to limit the # of > children. i think MaxSpareCoreMemory would make more sense. You > could set this to 1K if your server was designated for Apache only, or > set it to a higher value if it were a multipurpose machine. I've thought about that too. The trick is, Apache would need to know things about your application to do that right. It would need to know how big your processes were likely to be and how big they could get before they die. Otherwise, it has no way of knowing whether or not there's enough room for another process. A combination of Apache::SizeLimit and a dynamically changing MaxClients could possibly accomplish this, but you wouldn't want to run it too close to the edge since you don't want to have to axe a process that's in the middle of doing something just because it got a little too big (i.e. no hard limits on per-process memory usage). You can't change MaxClients while the server is running, can you? - Perrin
Re: killing of greater than MaxSpareServers
i and others have written on the list before, that pushing apache children into swap causes a rapid downward spiral in performance. I don't think that MaxClients is the right way to limit the # of children. i think MaxSpareCoreMemory would make more sense. You could set this to 1K if your server was designated for Apache only, or set it to a higher value if it were a multipurpose machine. mod_perl/apache and paging/swapping just don't mix. Perrin Harkins wrote: > On Wed, 17 Jan 2001, ___cliff rayman___ wrote: > > > here is an excerpt from httpd.h: > > Good reading. Thanks. > > It looks as if Apache should find the right number of servers for a steady > load over time, but it could jump up too high for a bit when the load > spike first comes in, pushing into swap if MaxClients is not configured > correctly. That may be what Sam was seeing. > > - Perrin -- ___cliff [EMAIL PROTECTED]http://www.genwax.com/
Re: killing of greater than MaxSpareServers
On Wed, 17 Jan 2001, ___cliff rayman___ wrote: > here is an excerpt from httpd.h: Good reading. Thanks. It looks as if Apache should find the right number of servers for a steady load over time, but it could jump up too high for a bit when the load spike first comes in, pushing into swap if MaxClients is not configured correctly. That may be what Sam was seeing. - Perrin
