Re: Memory footprint of spamd 3.0
Howdy. Following up on this thread... My spamd children don't seem to be sharing much memory with their parents: PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 14692 alias 17 0 40620 34M 2092 S 8.3 0.7 24:24 1 spamd (this is an example spamd child) In reviewing http://aspn.activestate.com/ASPN/Mail/Message/perl5-porters/2197769, it seems like others are reporting 60% of spamd children's memory as shared. I'm getting 10%. Am I missing something? I'm running spamd with: /usr/bin/perl -T -w /usr/bin/spamd -u alias -s mail -i 127.0.0.1 -A 127.0.0.1 -x -m 50 --max-conn-per-child=5000 --syslog=stderr -H /etc/mail/spamassassin/home On: Linux 2.4.21-20.ELsmp #1 SMP Wed Aug 18 20:46:40 EDT 2004 i686 i686 i386 GNU/Linux [EMAIL PROTECTED] 19529]# perl -V Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration: Platform: osname=linux, osvers=2.4.21-14.elsmp, archname=i386-linux-thread-multi uname='linux twe' config_args='-des -Doptimize=-O2 -g -pipe -march=i386 -mcpu=i686 -Dmyhostname=localhost [EMAIL PROTECTED] -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Dotherlibdirs=/usr/lib/perl5/5.8.0 -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr' hint=recommended, useposix=true, d_sigaction=define usethreads=define use5005threads=undef' useithreads=define usemultiplicity= useperlio= d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=un uselongdouble= usemymalloc=, bincompat5005=undef Compiler: cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm', optimize='', cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm' ccversion='', gccversion='3.2.3 20030502 (Red Hat Linux 3.2.3-37)', gccosandvers='' gccversion='3.2.3 200305' intsize=o, longsize=s, ptrsize=l, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long' k', ivsize=4' ivtype='long' known_extensi, nvtype='double', nvsize=, Off_t='', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='gcc' l', ldflags ='' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt -lutil perllibs= libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libper gnulibc_version='2.3.2' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so', d_dlsymun=undef, ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE' cccdlflags='-fPIC' ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5', lddlflags='s Unicode/Normalize XS/A' Characteristics of this binary (from libperl): Compile-time options: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES PERL_IMPLICIT_CONTEXT Locally applied patches: MAINT18379 Built under linux Compiled at Jun 28 2004 14:32:58 @INC: /usr/lib/perl5/5.8.0/i386-linux-thread-multi /usr/lib/perl5/5.8.0 /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.0 /usr/lib/perl5/vendor_perl /usr/lib/perl5/5.8.0/i386-linux-thread-multi /usr/lib/perl5/5.8.0 . Thanks in advance for any insight. - Jason Parsons
Re: Memory footprint of spamd 3.0
yep -- various versions of the Linux kernel do not measure shared memory in the same way. vanilla 2.4.18/19: reports shared correctly 2.4.x with Red Hat patches: incorrect 2.6.x: incorrect Is there any way to get at the actual amount of shared memory? Perhaps comparing the /proc/*/map output for each process? Also, would rebuilding PERL with a shared libperl.so help at all? Thank you. - Jason Parsons
Re: Memory footprint of spamd 3.0
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jason Parsons writes: yep -- various versions of the Linux kernel do not measure shared memory in the same way. vanilla 2.4.18/19: reports shared correctly 2.4.x with Red Hat patches: incorrect 2.6.x: incorrect Is there any way to get at the actual amount of shared memory? Perhaps comparing the /proc/*/map output for each process? run the test on a vanilla 2.4.x kernel, that's about it. Also, would rebuilding PERL with a shared libperl.so help at all? nope :( - --j. -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Exmh CVS iD8DBQFBdrGnMJF5cimLx9ARAilWAKCr2+vUs4L/k9ItFUHJP+aM6I0I7QCgnSA6 tDyhoL/1BpIWwrDyZLuOHhw= =vrQs -END PGP SIGNATURE-
Re: Memory footprint of spamd 3.0
I use ok_locales es en On Sat, 9 Oct 2004 22:03:21 -0700 (PDT), Jerry Glomph Black [EMAIL PROTECTED] wrote: In the default /usr/share/spamassasin/10_misc.cf file, I have ok_locales all ok_languagesall Nothing related in the personalized files in /etc/mail/spamassassin, or elsewhere. On Fri, 8 Oct 2004, Michael Parker wrote: On Tue, Oct 05, 2004 at 12:25:45PM -0500, Michael Parker wrote: On Tue, Oct 05, 2004 at 10:22:42AM -0700, Morris Jones wrote: I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort. Better question. Of all the folks seeing memory issues, are you using ok_languages in your config somewhere? If not, please speak up as well. Thanks Michael -- - GNU-GPL: May The Source Be With You... -
Re: Memory footprint of spamd 3.0
On Thu, 7 Oct 2004, Jon Trulson wrote: On Thu, 7 Oct 2004, Michael Parker wrote: On Thu, Oct 07, 2004 at 10:53:30AM -0600, Jon Trulson wrote: FWIW, in our case a child would go to 320MB and just stay there until the child was terminated (even after finishing a message). We do use AWL and bayes. Is it possible to try and find the msgs that was being scanned at that point in time? If so, can you reproduce by re-processing that message? Also, if you can, do an sa-learn --dump magic shortly after the jump happens and see what it says for the last expiry atime value. Does it happen to match when you saw the memory jump? I'll give that a shot this weekend when I'll have time to try to watch for it to happen. I missed the actual message (though it was only 2.5K, rejected at MTA), but I did happen to see one of the blowups happen this morning - to 325MB. The last expiry atime did indeed correspond with the time of the blowup. FWIW, here is the output of 'sa-learn --dump magic' [ pulsar ] sa-learn --dump magic 0.000 0 3 0 non-token data: bayes db version 0.000 0 175877 0 non-token data: nspam 0.000 0 132455 0 non-token data: nham 0.000 0 149125 0 non-token data: ntokens 0.000 0 1097168279 0 non-token data: oldest atime 0.000 0 1097513975 0 non-token data: newest atime 0.000 0 0 0 non-token data: last journal sync atime 0.000 0 1097513872 0 non-token data: last expiry atime 0.000 0 345600 0 non-token data: last expire atime delta 0.000 0 15505 0 non-token data: last expire reduction count The time 1097513872 (Mon Oct 11 10:57:52 2004) - matches when the the blowup started. The scan for this particular message lasted 125.9 seconds, and ran the CPU at 99% until the child exited. I am using '--max-conn-per-child=1' option to spamd. I am also running Perl 5.8.0, if that makes a diff. -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: Memory footprint of spamd 3.0
On Fri, 8 Oct 2004, Michael Parker wrote: On Tue, Oct 05, 2004 at 12:25:45PM -0500, Michael Parker wrote: On Tue, Oct 05, 2004 at 10:22:42AM -0700, Morris Jones wrote: I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort. Better question. Of all the folks seeing memory issues, are you using ok_languages in your config somewhere? If not, please speak up as well. I am using 'ok_locales en'. -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: Memory footprint of spamd 3.0
In the default /usr/share/spamassasin/10_misc.cf file, I have ok_locales all ok_languagesall Nothing related in the personalized files in /etc/mail/spamassassin, or elsewhere. On Fri, 8 Oct 2004, Michael Parker wrote: On Tue, Oct 05, 2004 at 12:25:45PM -0500, Michael Parker wrote: On Tue, Oct 05, 2004 at 10:22:42AM -0700, Morris Jones wrote: I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort. Better question. Of all the folks seeing memory issues, are you using ok_languages in your config somewhere? If not, please speak up as well. Thanks Michael
Re: Memory footprint of spamd 3.0
On Thu, 7 Oct 2004, Jeff Tucker wrote: I captured an exact copy of one of the messages that was being scanned when this happened. [ ... ] Rescanning the same message by calling spamc didn't cause the problem. The scan completed in just a couple of seconds. I did exactly the same experiment with the same result. You have Bayes turned off, I have it on. I'm also using auto-whitelist. Clearly there's nothing significant about the triggering message. I'm doing fine with --max-conn-per-child=10 for now. Cheers, Mojo -- Morris Jones * Monrovia, CA [EMAIL PROTECTED] http://www.whiteoaks.com
Re: Memory footprint of spamd 3.0
Michael Parker wrote: On Tue, Oct 05, 2004 at 12:25:45PM -0500, Michael Parker wrote: On Tue, Oct 05, 2004 at 10:22:42AM -0700, Morris Jones wrote: I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort. Better question. Of all the folks seeing memory issues, are you using ok_languages in your config somewhere? If not, please speak up as well. Thanks Michael I use ok_locales. -- Jim Gifford [EMAIL PROTECTED]
Re: Memory footprint of spamd 3.0
On Wed, 6 Oct 2004, Michael Parker wrote: On Wed, Oct 06, 2004 at 10:19:17AM -0300, Luis Hernán Otegui wrote: In my specific case, the ponit isn't only woth the big memory usage jumps, but with SA keeping the memory, and never releasing it. Highwater marks, common in most perl applicatios, don't concern me as much as these HUGE jumps in memory that folks are seeing. Jumps that just keep chewing memory without stopping. spamd opts: -c -d -m 20 --max-conn-per-child=1 There are places in the code where we could use memory a little more efficiently, I found one yesterday in fact, and we will work on these over time. In 2.x, the fork-on-demand model allowed us to be much more liberal with our data structures. We just need to wrangle that in a little with the 3.0 pre-fork code. FWIW, in our case a child would go to 320MB and just stay there until the child was terminated (even after finishing a message). We do use AWL and bayes. Michael -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: Memory footprint of spamd 3.0
On Thu, Oct 07, 2004 at 10:53:30AM -0600, Jon Trulson wrote: FWIW, in our case a child would go to 320MB and just stay there until the child was terminated (even after finishing a message). We do use AWL and bayes. Is it possible to try and find the msgs that was being scanned at that point in time? If so, can you reproduce by re-processing that message? Also, if you can, do an sa-learn --dump magic shortly after the jump happens and see what it says for the last expiry atime value. Does it happen to match when you saw the memory jump? Thanks Michael
Re: Memory footprint of spamd 3.0
On Thu, 7 Oct 2004, Michael Parker wrote: On Thu, Oct 07, 2004 at 10:53:30AM -0600, Jon Trulson wrote: FWIW, in our case a child would go to 320MB and just stay there until the child was terminated (even after finishing a message). We do use AWL and bayes. Is it possible to try and find the msgs that was being scanned at that point in time? If so, can you reproduce by re-processing that message? Also, if you can, do an sa-learn --dump magic shortly after the jump happens and see what it says for the last expiry atime value. Does it happen to match when you saw the memory jump? I'll give that a shot this weekend when I'll have time to try to watch for it to happen. -- Jon Trulsonmailto:[EMAIL PROTECTED] ID: 1A9A2B09, FP: C23F328A721264E7 B6188192EC733962 PGP keys at http://radscan.com/~jon/PGPKeys.txt #include std/disclaimer.h I am Nomad. -Nomad
Re: Memory footprint of spamd 3.0
I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort. It seems a solution here might be to have a spamd child that notices it needs to to maintenance should either pass that off to another child created specifically for that purpose, or else die after performing the maintenance . Of course, that is based on the speculation that the maintenance is causing the extra memory usage, which is still speculation. Loren
Re: Memory footprint of spamd 3.0
On Tue, Oct 05, 2004 at 05:30:32PM -0700, Loren Wilton wrote: It seems a solution here might be to have a spamd child that notices it needs to to maintenance should either pass that off to another child created specifically for that purpose, or else die after performing the maintenance Yeah, I had actually thought about doing that in the future. Right now, that's not possible due to the way the code gets run. The plan, BTW, was for the parent spamd to be more involved in things in the future, and the child would report to the parent that some work needs doing (like expire user foo's bayes tokens), and it would then fork off a process to do it. -- Randomly Generated Tagline: Oh, go and eat my alarm clock. - Matt pgpb7TskVPVTp.pgp Description: PGP signature
Re: Memory footprint of spamd 3.0
From: Loren Wilton [EMAIL PROTECTED] Sent: Tuesday, October 05, 2004 7:42 AM Any chance in going back to something that actually worked? I tried running a 2.64 version of spamd, but got a mountain of bayes-related errors. There is an option to only run a single child, which is claimed to be equivalent to the 2.6x implementation. I don't recall the option (something=1), but Theo posted it within the last day here. And I'm almost positive it is in the docs somewhere. Just watching what people have been reporting, I've come to several tentative conclusions on 3.0 as it currently stands: 1.I'm about 70% convinced there is an undiscvered memory leak or other resource leak that has the equivalent result. 2.The copying of the config back and forth with preforking has a few minor but serious problems. 3.There is a problem with spamd children getting hung out to dry on a read that never completes. I had this problem and I'm checking on the moment if I found the bug. The message was deleted before spamc was killed so with a sleep in my spamc call script I hope to be able to close my problem 4.I suspect that 3.0 is inherently less memory efficient than 2.6x; but probably not by a huge amount. I think that when the first three problems are addressed and solved that 3.0 will become a whole lot more generally usable. With kind regards, Met vriendelijke groet, Maurice Lucas TAOS-IT
Re: Memory footprint of spamd 3.0
On Tue, 5 Oct 2004 12:25:45 -0500, Michael Parker [EMAIL PROTECTED] wrote: On Tue, Oct 05, 2004 at 10:22:42AM -0700, Morris Jones wrote: I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort. For folks that are seeing huge jumps in memory, instead of gradual growth, how are you calling SA? Thanks, Michael I'm calling it via spamass-milter. In the Spamd init script I have this options: OPTIONS=-d -x -u spamd -H /var/spool/spamassassin/ and in the spamassmilter init script, this ones: SM_EXTRA_FLAGS=-r 15 -m -u spamd -fb [EMAIL PROTECTED] -i 163.10.18.0/32 -i 163.10.7.0/32 In my specific case, the ponit isn't only woth the big memory usage jumps, but with SA keeping the memory, and never releasing it. -- - GNU-GPL: May The Source Be With You... -
Re: Memory footprint of spamd 3.0
On Wed, Oct 06, 2004 at 10:19:17AM -0300, Luis Hernán Otegui wrote: jumps, but with SA keeping the memory, and never releasing it. Well, there's really no way to do that at the application level -- it's up to the OS. Typically what happens is: - process wants X memory - process gets X memory allocated from kernel - process uses X memory - process frees (free()) X memory - kernel keeps memory allocated to process figuring it'll use it again (and to avoid all the overhead of cleanup/reallocation, etc.) So the goal needs to be: don't use the memory in the first place. Anything after that is only fixable by restarting the process. -- Randomly Generated Tagline: But these go to 11... - Spinal Tap pgpVzEp5Ony1P.pgp Description: PGP signature
Re: Memory footprint of spamd 3.0
Good enough argument, but the new version has killed off two machines that have successfully run SA/spamd with no problems since Jan 2002. The machines leak into oblivion, and Something Bad happens (major daemons killed off, etc.). There is definitely some kind of memory resource problem (if not an outright leak) going on.Which reverting to the old fork-on-demand -might- help. I always marvelled at how SA was different from most perl programs, since they generally hog a machine's resources if running long enough. Perhaps this is finally happening with this persistent-process model. Any chance in going back to something that actually worked? I tried running a 2.64 version of spamd, but got a mountain of bayes-related errors. On Mon, 4 Oct 2004, Justin Mason wrote: Jerry Glomph Black writes: spamd 3.0 does preforking of the child processes. Nothing wrong with that, but WHY do the children have such enormous RSS numbers already when started (20 Meg per process)? To me, this makes no sense. 3.0 has rendered two decent machines of mine useless by snarfing up all the RAM. Can this be put back to the old fork-on-demand model? I'm convinced that the current forking scheme is broken. Is the old 2.6x spamd code compatible enough to run with all the 3.0 perl apparatus? most of this memory is shared. spamd preloads as much as possible upfront, to maximise memory sharing -- including all the rules, compiled into perl code etc. recent investigation (can't recall bug #) is showing that this is working, too...
Re: Memory footprint of spamd 3.0
Any chance in going back to something that actually worked? I tried running a 2.64 version of spamd, but got a mountain of bayes-related errors. There is an option to only run a single child, which is claimed to be equivalent to the 2.6x implementation. I don't recall the option (something=1), but Theo posted it within the last day here. And I'm almost positive it is in the docs somewhere. Just watching what people have been reporting, I've come to several tentative conclusions on 3.0 as it currently stands: 1.I'm about 70% convinced there is an undiscvered memory leak or other resource leak that has the equivalent result. 2.The copying of the config back and forth with preforking has a few minor but serious problems. 3.There is a problem with spamd children getting hung out to dry on a read that never completes. 4.I suspect that 3.0 is inherently less memory efficient than 2.6x; but probably not by a huge amount. I think that when the first three problems are addressed and solved that 3.0 will become a whole lot more generally usable. Loren
Re: Memory footprint of spamd 3.0
On Tue, 5 Oct 2004, Theo Van Dinter wrote: Well, it's not running a single child, it's that each child should only run 1 message before dying. It's right in the spamd docs, but --max-conn-per-child=1 is what you're looking for. I set mine to --max-conn-per-child=10 as a compromise, and it's doing very well for me. At least if a 250MB spamd pops up, it won't live very long. I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort. One solution might be to have a child terminate after one of these events to reclaim the memory. Of course, here I am speculating about it without actually diving into the code to propose a solution. Regards, Mojo -- Morris Jones * Monrovia, CA [EMAIL PROTECTED] http://www.whiteoaks.com
Re: Memory footprint of spamd 3.0
On Tue, Oct 05, 2004 at 10:22:42AM -0700, Morris Jones wrote: I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort. For folks that are seeing huge jumps in memory, instead of gradual growth, how are you calling SA? Thanks, Michael
Re: Memory footprint of spamd 3.0
On Tue, 5 Oct 2004, Michael Parker wrote: On Tue, Oct 05, 2004 at 10:22:42AM -0700, Morris Jones wrote: I watched a spamd child grow to 250MB yesterday on a single message. I have a suspicion that the memory usage growth is happening on a whitelist or bayes database maintenance event of some sort. For folks that are seeing huge jumps in memory, instead of gradual growth, how are you calling SA? Thanks, Michael I'm running spamd -d -c --max-conn-per-child=10 In /etc/procmailrc I have: :0fw * 256000 | /usr/bin/spamc Mojo -- Morris Jones * Monrovia, CA [EMAIL PROTECTED] http://www.whiteoaks.com
Re: Memory footprint of spamd 3.0
Morris Jones wrote on Tue, 5 Oct 2004 10:22:42 -0700 (PDT): I watched a spamd child grow to 250MB yesterday on a single message. This can happen sometimes (rarely) with 2.6x as well! I have already seen 900 MB spamds. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com IE-Center: http://ie5.de http://msie.winware.org
