Bug#485413: apache2/lenny/amd64 eats up all RAM
On Mon, 15 Aug 2011, Stefan Fritsch wrote: > - How much memory does your machine have, how much is actually > available for apache (i.e. not used by other processes). This one has 12 GiB in total. Memory is mostly shared between Apache, Sonar (Jenkins Plugin) and Jenkins. Normally, Apache has 11 processes, each using less than 5 MiB (ps ax -O rss). > - Which MPM do you use? ii apache2-mpm-prefork 2.2.9-10+lenny9.0tarent1 Apache HTTP Server - traditional non-threade > - What are your settings for StartServers, MaxRequestsPerChild, > ThreadsPerChild, MinSpareThreads/MinSpareServers, > MaxSpareThreads/MaxSpareServers Used to be the defaults. When the problems started, I tried tweaking to these (no success): /etc/apache2/apache2.conf:StartServers 5 /etc/apache2/apache2.conf:MinSpareServers 5 /etc/apache2/apache2.conf:MaxSpareServers 10 /etc/apache2/apache2.conf:MaxRequestsPerChild 3000 On another box, I have these: /etc/apache2/apache2.conf:StartServers 5 /etc/apache2/apache2.conf:MinSpareServers 5 /etc/apache2/apache2.conf:MaxSpareServers 10 /etc/apache2/apache2.conf:MaxRequestsPerChild 20 /etc/apache2/apache2.conf:MaxMemFree 512 Will monitor whether it occurs there, too – a bit futile since we know of nothing to trigger the problem, it just happens every once in a while. > - How large do the apache2 processes grow? (RSS) How quickly do they > grow? As could be seen, they eat up _all_ available RAM and swap. This happens very quickly because we usually notice the problem because Jenkins is terminated by the OOM Kernel thing when Apache grows; Jenkins is in use by our developers, so this occurs suddenly. > - How many apache2 are running during normal operation? Eleven, as seen above. > If that doesn't help, try setting 'MaxMemFree 4'. If that doesn't help > enough, add 'export MALLOC_MMAP_THRESHOLD_=8000' to > /etc/apache2/envvars (the trailing underscore is not a typo). These > options lower performance, though. Will do. First thing is lowering MaxRequestsPerChild for all our Jenkins instances, I’d say. > Please report back about the results. OK. Thanks! bye, //mirabilos -- tarent solutions GmbH Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/ Tel: +49 228 54881-393 • Fax: +49 228 52675-25 HRB AG Bonn 5168 • USt-ID (VAT): DE122264941 Geschäftsführer: Boris Esser, Elmar Geese -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#485413: apache2/lenny/amd64 eats up all RAM
On Monday 15 August 2011, Thorsten Glaser wrote: > Can please someone recommend a different ulimit, > or some other suggestion? You could provide some more information: - How much memory does your machine have, how much is actually available for apache (i.e. not used by other processes). - Which MPM do you use? - What are your settings for StartServers, MaxRequestsPerChild, ThreadsPerChild, MinSpareThreads/MinSpareServers, MaxSpareThreads/MaxSpareServers - How large do the apache2 processes grow? (RSS) How quickly do they grow? - How many apache2 are running during normal operation? Tune the above settings. If processes grow slowly, you may be hitting a memory leak. Setting MaxRequestsPerChild to non-zero (e.g. to 500) may help. If that doesn't help, try setting 'MaxMemFree 4'. If that doesn't help enough, add 'export MALLOC_MMAP_THRESHOLD_=8000' to /etc/apache2/envvars (the trailing underscore is not a typo). These options lower performance, though. Please report back about the results. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#485413: apache2/lenny/amd64 eats up all RAM
Dixi quod… > Trying with the attached patch now. > Let’s see whether this helps to work around the problem. It doesn’t: root@dev-hudson:/proc/4480 # cat limits Limit Soft Limit Hard Limit Units Max cpu time unlimitedunlimitedms Max file size unlimitedunlimitedbytes Max data size 268435456268435456bytes Max stack size8388608 unlimitedbytes Max core file size0unlimitedbytes Max resident set unlimitedunlimitedbytes Max processes 102400 102400 processes Max open files1024 1024 files Max locked memory 3276832768bytes Max address space unlimitedunlimitedbytes Max file locksunlimitedunlimitedlocks Max pending signals 102400 102400 signals Max msgqueue size 819200 819200 bytes Max nice priority 00 Max realtime priority 00 Max realtime timeout unlimitedunlimitedus root@dev-hudson:/proc/4480 # cat status Name: apache2 State: S (sleeping) Tgid: 4480 Pid:4480 PPid: 7113 TracerPid: 0 Uid:33 33 33 33 Gid:33 33 33 33 FDSize: 64 Groups: 33 VmPeak: 13022820 kB VmSize: 13022820 kB VmLck: 0 kB VmHWM: 11278624 kB VmRSS: 10980224 kB VmData: 12901596 kB VmStk:88 kB VmExe: 392 kB VmLib: 11840 kB VmPTE: 25436 kB Threads:1 SigQ: 0/102400 SigPnd: ShdPnd: SigBlk: SigIgn: 1000 SigCgt: 0001880046eb CapInh: CapPrm: CapEff: CapBnd: Cpus_allowed: 0007 Cpus_allowed_list: 0-2 Mems_allowed: ,0001 Mems_allowed_list: 0 voluntary_ctxt_switches:51843420 nonvoluntary_ctxt_switches: 303994690 Can please someone recommend a different ulimit, or some other suggestion? Thanks, //mirabilos -- tarent solutions GmbH Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/ Tel: +49 228 54881-393 • Fax: +49 228 52675-25 HRB AG Bonn 5168 • USt-ID (VAT): DE122264941 Geschäftsführer: Boris Esser, Elmar Geese -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#485413: apache2/lenny/amd64 eats up all RAM
Trying with the attached patch now. Let’s see whether this helps to work around the problem. bye, //mirabilos -- tarent solutions GmbH Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/ Tel: +49 228 54881-393 • Fax: +49 228 52675-25 HRB AG Bonn 5168 • USt-ID (VAT): DE122264941 Geschäftsführer: Boris Esser, Elmar Geesediff -u apache2-2.2.9/debian/apache2.2-common.apache2.default apache2-2.2.9/debian/apache2.2-common.apache2.default --- apache2-2.2.9/debian/apache2.2-common.apache2.default +++ apache2-2.2.9/debian/apache2.2-common.apache2.default @@ -1,3 +1,10 @@ +### apache2 settings ### + +## uncomment this to change the datasize ulimit +## commented out means do not set it at all +## otherwise this is the ulimit in Kibibytes +#APACHE2_DATASIZE=262144 + ### htcacheclean settings ### ## run htcacheclean: yes, no, auto diff -u apache2-2.2.9/debian/changelog apache2-2.2.9/debian/changelog --- apache2-2.2.9/debian/changelog +++ apache2-2.2.9/debian/changelog @@ -1,3 +1,9 @@ +apache2 (2.2.9-10+lenny9.0tarent1) local; urgency=low + + * Allow setting the datasize ulimit for the Apache 2 Web Server + + -- Thorsten Glaser Thu, 11 Aug 2011 13:09:19 +0200 + apache2 (2.2.9-10+lenny9) stable-security; urgency=high * Add the new SSLInsecureRenegotiation directive to configure if clients diff -u apache2-2.2.9/debian/apache2.2-common.apache2.init apache2-2.2.9/debian/apache2.2-common.apache2.init --- apache2-2.2.9/debian/apache2.2-common.apache2.init +++ apache2-2.2.9/debian/apache2.2-common.apache2.init @@ -18,6 +18,7 @@ #echo "To add and enable a host, use addhost and enhost." && exit 0 #edit /etc/default/apache2 to change this. +APACHE2_DATASIZE="" HTCACHECLEAN_RUN=auto HTCACHECLEAN_MODE=daemon HTCACHECLEAN_SIZE=300M @@ -148,7 +149,11 @@ case $1 in start) log_daemon_msg "Starting web server" "apache2" - if $APACHE2CTL start; then + if ( + test -n "$APACHE2_DATASIZE" && \ + ulimit -d "$APACHE2_DATASIZE" + exec $APACHE2CTL start + ); then if check_htcacheclean ; then log_progress_msg htcacheclean start_htcacheclean || log_end_msg 1
Bug#485413: apache2/lenny/amd64 eats up all RAM
Hi, this also happens on an installation without PHP, but mod_jk and SSL enabled (Jenkins frontend). bye, //mirabilos -- tarent solutions GmbH Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/ Tel: +49 228 54881-393 • Fax: +49 228 52675-25 HRB AG Bonn 5168 • USt-ID (VAT): DE122264941 Geschäftsführer: Boris Esser, Elmar Geese -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
