Re: [analog-help] server crashing down / big cache file

analog-help Tue, 11 May 2004 07:04:07 -0700

The Analog documentation mentions low memory
at "http://www.analog.cx/docs/lowmem.html";.

HTH,

-- Duke

Andreas Leppert wrote:

hello analog list,

sorry for posting in german, i'm posting again in german.

I've a webserver with several domains on it, everything work pretty fine, but the major site on the server which produces 30GB traffic / month doesn't get along with analog. the nightly cronjobs run a script, which runs analog commands.


here my analog.conf:
# Configuration file for analog 4.03
# Extra options used for generating the log data for OpCenterWeb
CACHEOUTFILE none
DNS NONE
#WARNINGS OFF
#APACHELOGFORMAT ("%h %l %u %t \"%r\" %>s %b \"%{User-Agent}i\"")
#LOGFORMAT (%S %l %j [%d/%M/%Y:%h:%n:%j] \"%j%w%r%wHTTP%j\" %c %b \"%B\")
LOGFORMAT COMBINED
LOGFILE none
OUTPUT COMPUTER
OUTFILE -
GENERAL ON
MONTHLY ON
MONTHCOLS RrPp
WEEKLY OFF
FULLDAILY OFF
DAILY ON
DAYCOLS RrPp
FULLHOURLY OFF
HOURLY ON
HOURCOLS RrPp
QUARTER OFF
FIVE OFF
HOST ON
HOSTCOLS RrPp
HOSTFLOOR 1:r
ORGANISATION ON
ORGCOLS RrPp
ORGFLOOR 1:r
DOMAIN ON
DOMCOLS RrPp
DOMSORTBY REQUESTS
DOMFLOOR 1:r
SUBDOMFLOOR 1:r
SUBDOMSORTBY REQUESTS
REQUEST OFF
DIRECTORY ON
DIRCOLS Rrb
FILETYPE ON
TYPECOLS Rrb
SIZE OFF
PROCTIME OFF
REDIR OFF
FAILURE ON
FAILCOLS Rr
REFERER OFF
REFSITE OFF
SEARCHQUERY OFF
SEARCHWORD OFF
REDIRREF OFF
FAILREF OFF
FULLBROWSER OFF
BROWSER ON
BROWFLOOR 1:r
BROWCOLS Rr
BROWSORTBY REQUESTS
SUBBROWSORTBY REQUESTS
SUBBROWFLOOR 1:r
OSREP OFF
VHOST OFF
USER OFF
FAILUSER OFF
STATUS OFF
HOSTLOWMEM 3

the script which calls analog:

#!/usr/bin/perl -w
use integer;
my($oneday) = 86400;
my($oneweek) = 604800;

sub analyze {
# print STDERR "$_[0]\n";
if ($ENV{'OSTYPE'} =~ /^solaris/i) {
system ("/usr/local/analog4.03/analog $_[0]");
} else {
system ("/usr/bin/analog $_[0]");
}
}

#Process the command line arguments
if ($#ARGV != 1) {
die "The number of arguments is wrong\nUsage updatestats statsdir logfile\n";
}

my($statdir,$logfile) = @ARGV;
my($analogconffile) = "$statdir/analog.cfg";
my($time) = time;
my($thisweekstat) = "$statdir/week".&getweeksbefore(0,$time);
my(@stattodelete) = (); #we sometimes do garbage collection
my($timeoption,$logoption);
my($weeklyoption) = "+g$analogconffile +C\"MONTHLY OFF\"";

my($cachefile) = "$statdir/cache";
my($runningoption) = "+g$analogconffile +C\"CACHEFILE $cachefile\"";

unless (stat ("$thisweekstat")) {
# build the statistics for last week one more time
my ($lastweekstat) = "$statdir/week".&getweeksbefore(1,$time);
@stattodelete = ("$statdir/week".&getweeksbefore(4,$time),
"$statdir/week".&getweeksbefore(5,$time));
#just playing it safe, normally the first should be enough
$timeoption = "+F".&firstdayofweek(1,$time)." +T".&lastdayofweek(1,$time);
$logoption = "+C\"LOGFILE $logfile\" +C\"LOGFILE $logfile.1\" +C\"LOGFILE 
$logfile.2\"";
#normally the last two should be enough
analyze("$weeklyoption $timeoption $logoption >$lastweekstat");
if ( -e "$lastweekstat" ) {
chmod(0640,"$lastweekstat") ;
}
#Get rid of "false ampty reports"
my ($hastogo)="";
if (open (LASTWEEK,$lastweekstat)) {
my ($line);
while ($line=<LASTWEEK>) {
next if ($line !~ /^x\sSR\s(\d+)/);
last if ($1 != 0);
last if (stat($logfile.1));
$hastogo=1;
last;
}
close(LASTWEEK);
if ($hastogo) {unlink($lastweekstat)}
}
}

#build statistics for this week
$timeoption = "+F".&firstdayofweek(0,$time)." +T".&lastdayofweek(0,$time);
$logoption = "+C\"LOGFILE $logfile\" +C\"LOGFILE $logfile.1\"";
analyze("$weeklyoption $timeoption $logoption >$thisweekstat");
if ( -e "$thisweekstat" ) {
chmod(0640,"$thisweekstat");
}

if (@stattodelete) {
map {unlink($_)} @stattodelete;
}
#Now we deal with the running total
#Update the cache if the logs have been rotated since last call
my (@statinfo);
if (@statinfo = stat ("$logfile.1")) {
my ($logmtime,$cachemtime);
$logmtime = $statinfo[9];
if(@statinfo = stat ("$cachefile")) {
$cachemtime = $statinfo[9];
} else {
$cachemtime = 0;
}
if ($cachemtime != $logmtime) {
#build new cache
analyze("$runningoption +C\"LOGFILE $logfile.1\" +C\"CACHEOUTFILE $cachefile.new\" +C\"OUTPUT 
NONE\"");
utime($logmtime,$logmtime,"$cachefile.new");
rename("$cachefile.new","$cachefile");
if ( -e "$cachefile" ) {
chmod(0640,"$cachefile");
}
}
}

analyze("$runningoption +C\"LOGFILE $logfile\" >$statdir/running");
if ( -e "$statdir/running" ) {
chmod(0640,"$statdir/running") ;
}

#returns the number of the week of the year we were in x weeks before $time
sub getweeksbefore {
my ($diff,$time) = @_;
$time -= $oneweek*$diff; # $diff weeks ago
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)= localtime($time);
if ($wday > $yday) { #this week started last year, so we count it there
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)= localtime($time-$oneweek);
$yday+=7;
}
return 1+($yday-$wday)/7; #1..53
}
sub firstdayofweek {
my ($diff,$time) = @_;
$time -= $oneweek*$diff; # $diff weeks ago
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)= localtime($time);
$time -= $oneday*$wday; # last Sunday
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)= localtime($time);
$year = ($year>=100 ? $year%100 : $year);
return sprintf ("%02d%02d%02d",$year,$mon+1,$mday);
}

sub lastdayofweek {
my ($diff,$time) = @_;
$time -= $oneweek*$diff; # $diff weeks ago
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)= localtime($time);
$time += $oneday*(6-$wday); # this Saturday
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)= localtime($time);
$year = ($year>=100 ? $year%100 : $year);
return sprintf ("%02d%02d%02d",$year,$mon+1,$mday);
}

you can see that this is calling analog with several runtime-parameters, i.e. the cachefile parameter. the cache file for this domain is really big (and i think here is the problem):

-rw-r----- 1 root root 385M May 2 06:01 cache

while analog is working on this domain, i look at the output of top and ps faux. i can see, that analog needs a lot of memory, and the system is crashing. httpd and sshd are not responding after a couple of time. (~ 1 hour). kswapd is running, that means, linux is swapping out, which means that analog needs too much memory space.

the system: 2.0 ghz celeron, 256 ram + 768 MB swap...

what can i do?

thanks in advance
andi leppert

p.s.: Michael S Hill gave me the hint to increase the memory space: this isn't possible because my provider won't do that. ____________________________________________________________________ Der WEB.DE Virenschutz schuetzt Ihr Postfach vor dem Wurm Sober.A-F! Kostenfrei fuer FreeMail Nutzer. http://f.web.de/?mc=021158

+------------------------------------------------------------------------
|  TO UNSUBSCRIBE from this list:
|    http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  Digest version: http://lists.isite.net/listgate/analog-help-digest/
|  Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
|  List archives:  http://www.analog.cx/docs/mailing.html#listarchives
+------------------------------------------------------------------------


+------------------------------------------------------------------------
|  TO UNSUBSCRIBE from this list:
|    http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  Digest version: http://lists.isite.net/listgate/analog-help-digest/
|  Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
|  List archives:  http://www.analog.cx/docs/mailing.html#listarchives
+------------------------------------------------------------------------

Re: [analog-help] server crashing down / big cache file

Reply via email to