spamd issues
Hi, I recently put my first spamd installation into production and am quite impressed with the results, good work, folks. Nevertheless I have some questions: * it seems that when spamd scans it's database in /var/db/spamd (which is currently ~160MB of size) it doesn't accept any new requests on it's port (at least it let's the clients wait). That sucks. I see 2 spamd processes process states hanging in: biowait, pipewr - I understand that while the database is being scanned (and maybe locked) new requests maybe can't easily be written to the db, is there a plan to improve that (by creating a queue-log or something)? * Due to the fact that spamd only seems to insert pf-rules into the pf spamd-white table when doing a db-scan it seems that it creates some more delays than necessary in the greylisted mta's. Let's say I have 2 mx'es - mx1 and mx2 - my client is connecting to mx1, get's greylisted, connects to mx2 (protected by the same spamd instance), still keeps being greylisted - so my client adds a penalty time of let's say 2x5 mins. After 10 minutes it connects to spamd again, tries mx1, still gets redirected to spamd, but spamd decides to whitelist that host, and writes that into the spamdb. After being refused for mx1, my client tries mx2, and due to the fact that maybe spamd didn't scan the spamdb yet the pf-rules aren't in place yet, so it get's redirected to spamd once again, creating quite some penalty time of let's say 2x10 minutes, which wouldn't be necessary if spamd would insert that ip directly into pf on writing the whitelist-entry into the database. Are there plans to improve that? I hope this is a question that still belongs to misc and not developer, but I'm not too sure about that :) btw: I recorded 1886206 connections within the first 24 hours to spamd for that particular mai provider :) thanks for any replies; Wolfgang -- http://www.wogri.com
Re: spamd issues
Darrin Chandler wrote: On Tue, Jan 06, 2009 at 08:26:37PM -0500, Frank Bax wrote: I notice that one example line was removed from pf.conf: table spamd persist I guess I can delete that line from my file too? Er, you'll still need that unless something's happened that I totally missed. I'm thinking that line remembers something during reboot (but I'm not sure what that is). Does removing it forget GREY or WHITE or both? If you are running spamd on your mail server then it's a bit simpler: no rdr on $ext_if proto tcp from spamd-mywhite to any port smtp no rdr on $ext_if proto tcp from spamd-white to any port smtp rdr pass on $ext_if proto tcp from any to any port smtp \ - 127.0.0.1 port spamd Seems to be working just fine, thanks. GREY/WHITE issue is still there though.
Re: spamd issues
table spamd persist I guess I can delete that line from my file too? Er, you'll still need that unless something's happened that I totally missed. I'm thinking that line remembers something during reboot (but I'm not sure what that is). Does removing it forget GREY or WHITE or both? Tables and 'persist' are covered nicely in the man page. Persist keeps the table even if it's empty. It's nothing to do with reboots, and that it handled by spamd's database. If you are running spamd on your mail server then it's a bit simpler: no rdr on $ext_if proto tcp from spamd-mywhite to any port smtp no rdr on $ext_if proto tcp from spamd-white to any port smtp rdr pass on $ext_if proto tcp from any to any port smtp \ - 127.0.0.1 port spamd Seems to be working just fine, thanks. GREY/WHITE issue is still there though. I'm glad it's working. If it were really a problem then you'd have a bazillion GREY entries and/or no email would get through. It'll stop being an issue when you stop worrying about it ;) -- Darrin Chandler| Phoenix BSD User Group | MetaBUG dwchand...@stilyagin.com | http://phxbug.org/ | http://metabug.org/ http://www.stilyagin.com/ | Daemons in the Desert | Global BUG Federation [demime 1.01d removed an attachment of type application/pgp-signature]
spamd issues
I've been using spamd since 3.5 or 3.6 - It seems to be working great, so mostly I just let it do it's thing and ignore it. Today I was having some issues sending mail through a local ISP to my system (4.4 release). Some investigation showed that spamdb reports the ip address of the ISP's smtp server as both WHITE and GREY? This should not be possible, should it? $ sudo spamdb | grep 64.7.153.18 WHITE|64.7.153.18|||1231252840|1231254379|1234364784|9|0 GREY|64.7.153.18|smarthost1.sentex.ca|x...@clgw.ca|x...@clgw.ca|1231252840|1231254390|1231267240|10|0 Is it possible to remove the GREY entry (spamdb -d only removes WHITE entires)? I'm trying to remember how many config files need to be included for this; hopefully, I don't miss any. = = /etc/pf.conf ext_if=rl0 in_mx=127.0.0.1 table spamd persist table spamd-white persist table spamd-mywhite persist scrub in rdr pass on $ext_if proto tcp from spamd-mywhite to port smtp \ - $in_mx port smtp rdr pass on $ext_if proto tcp from spamd to port smtp \ - 127.0.0.1 port spamd rdr pass on $ext_if proto tcp from spamd-white to port smtp \ - $in_mx port smtp rdr pass on $ext_if proto tcp from !spamd-white to port smtp \ - 127.0.0.1 port spamd pass in log on $ext_if proto tcp to ($ext_if) port smtp keep state pass out log on $ext_if proto tcp from ($ext_if) to port smtp keep state all:\ :myblack:mywhite:uatraps:nixspam:china:korea: uatraps:\ :black:\ :msg=Your address %A has sent mail to a ualberta.ca spamtrap\n\ within the last 24 hours:\ :method=http:\ :file=www.openbsd.org/spamd/traplist.gz nixspam:\ :black:\ :msg=Your address %A is in the nixspam list\n\ See http://www.heise.de/ix/nixspam/dnsbl_en/ for details:\ :method=http:\ :file=www.openbsd.org/spamd/nixspam.gz china:\ :black:\ :msg=SPAM. Your address %A appears to be from China\n\ See http://www.okean.com/asianspamblocks.html for more details:\ :method=http:\ :file=www.openbsd.org/spamd/chinacidr.txt.gz: korea:\ :black:\ :msg=SPAM. Your address %A appears to be from Korea\n\ See http://www.okean.com/asianspamblocks.html for more details:\ :method=http:\ :file=www.openbsd.org/spamd/koreacidr.txt.gz: myblack:\ :black:\ :msg=SPAM: %A has been blacklisted.:\ :method=file:\ :file=/etc/mail/spamd_black.txt: mywhite:\ :white:\ :method=file:\ :file=/etc/mail/spamd_white.txt: = = = /etc/mail/spamd_white.txt Adapted from http://cvs.puremagic.com/viewcvs/greylisting/schema/whitelist_ip.txt Site seems to be down at the moment; but it hadn't changed content in sometime.
Re: spamd issues
On Tue, Jan 06, 2009 at 12:58:00PM -0500, Frank Bax wrote: Today I was having some issues sending mail through a local ISP to my system (4.4 release). Some investigation showed that spamdb reports the ip address of the ISP's smtp server as both WHITE and GREY? This should not be possible, should it? $ sudo spamdb | grep 64.7.153.18 WHITE|64.7.153.18|||1231252840|1231254379|1234364784|9|0 GREY|64.7.153.18|smarthost1.sentex.ca|x...@clgw.ca|x...@clgw.ca|1231252840| 1231254390|1231267240|10|0 Is it possible to remove the GREY entry (spamdb -d only removes WHITE entires)? WHITE is seen first, so the GREY has no effect. This behavior started a few releases ago, and it's been discussed several times. There's no need to remove the GREY entries. They expire on their own in short order. -- Darrin Chandler| Phoenix BSD User Group | MetaBUG dwchand...@stilyagin.com | http://phxbug.org/ | http://metabug.org/ http://www.stilyagin.com/ | Daemons in the Desert | Global BUG Federation [demime 1.01d removed an attachment of type application/pgp-signature]
Re: spamd issues
Darrin Chandler wrote: On Tue, Jan 06, 2009 at 12:58:00PM -0500, Frank Bax wrote: Today I was having some issues sending mail through a local ISP to my system (4.4 release). Some investigation showed that spamdb reports the ip address of the ISP's smtp server as both WHITE and GREY? This should not be possible, should it? $ sudo spamdb | grep 64.7.153.18 WHITE|64.7.153.18|||1231252840|1231254379|1234364784|9|0 GREY|64.7.153.18|smarthost1.sentex.ca|x...@clgw.ca|x...@clgw.ca|1231252840| 1231254390|1231267240|10|0 Is it possible to remove the GREY entry (spamdb -d only removes WHITE entires)? WHITE is seen first, so the GREY has no effect. This behavior started a few releases ago, and it's been discussed several times. There's no need to remove the GREY entries. They expire on their own in short order. Ah, my bad; sorry. Thanks for being kind enough to post the same answer a second time. Now I've done a bit of homework. http://marc.info/?l=openbsd-miscm=118755082205516w=2 http://marc.info/?l=openbsd-miscm=120033441815022w=2 http://marc.info/?l=openbsd-miscm=120507275423154w=2 The basic response was that this behaviour is normal; but I started to wonder about this when 2 of 3 posts did not contain a pf.conf file. In the only thread (first one) where a pf.conf file was included; Edgars happened to mention: I've always had my spamd-white list match on a no rdr before any of the other rules. I don't have a no rdr rule and neither did the OP that included his pf.conf file. Going through the changelog for pf.conf, I notice this change Feb.2007: http://www.openbsd.org/cgi-bin/cvsweb/src/etc/pf.conf.diff?r1=1.33;r2=1.34 I'm afraid I barely understood how spamd worked when I first did the setup back in 3.5; and now I don't fully understand the impact of these changes made to examples in pf.conf; but I do notice a few things. First, I notice that other people started having soon after this change was committed. Is it possible that a change to my pf.conf would get rid of the duplicate ip addresses (once the GREY's had expired)? I notice that one example line was removed: table spamd persist I guess I can delete that line from my file too? I notice that the two example rules that were changed match my 2nd and 4th rules. That cannot be coincidence. in_mx=127.0.0.1 rdr pass on $ext_if proto tcp from spamd-mywhite to port smtp \ - $in_mx port smtp rdr pass on $ext_if proto tcp from spamd to port smtp \ - 127.0.0.1 port spamd rdr pass on $ext_if proto tcp from spamd-white to port smtp \ - $in_mx port smtp rdr pass on $ext_if proto tcp from !spamd-white to port smtp \ - 127.0.0.1 port spamd Should I change my file to: rdr pass on $ext_if proto tcp from spamd-mywhite to port smtp \ - $in_mx port smtp no rdr on $ext_if proto tcp from spamd-white to any port smtp rdr pass on $ext_if proto tcp from spamd-white to port smtp \ - $in_mx port smtp rdr pass on $ext_if proto tcp from any to any port smtp \ - 127.0.0.1 port spamd I'm thinking my 3rd rule is now redundant - is this correct? Frank
Re: spamd issues
On Tue, Jan 06, 2009 at 08:26:37PM -0500, Frank Bax wrote: I notice that one example line was removed: table spamd persist I guess I can delete that line from my file too? Er, you'll still need that unless something's happened that I totally missed. I notice that the two example rules that were changed match my 2nd and 4th rules. That cannot be coincidence. in_mx=127.0.0.1 rdr pass on $ext_if proto tcp from spamd-mywhite to port smtp \ - $in_mx port smtp rdr pass on $ext_if proto tcp from spamd to port smtp \ - 127.0.0.1 port spamd rdr pass on $ext_if proto tcp from spamd-white to port smtp \ - $in_mx port smtp rdr pass on $ext_if proto tcp from !spamd-white to port smtp \ - 127.0.0.1 port spamd Should I change my file to: rdr pass on $ext_if proto tcp from spamd-mywhite to port smtp \ - $in_mx port smtp no rdr on $ext_if proto tcp from spamd-white to any port smtp rdr pass on $ext_if proto tcp from spamd-white to port smtp \ - $in_mx port smtp rdr pass on $ext_if proto tcp from any to any port smtp \ - 127.0.0.1 port spamd I'm thinking my 3rd rule is now redundant - is this correct? I'm a little confused. Do you have separate firewall mail server, and are running spamd on the firewall? If so I think the following (untested) should work: rdr pass on $ext_if proto tcp from spamd-mywhite to any port smtp \ - $in_mx port smtp rdr pass on $ext_if proto tcp from spamd-white to any port smtp \ - $in_mx port smtp rdr pass on $ext_if proto tcp from any to any port smtp \ - 127.0.0.1 port spamd If you are running spamd on your mail server then it's a bit simpler: no rdr on $ext_if proto tcp from spamd-mywhite to any port smtp no rdr on $ext_if proto tcp from spamd-white to any port smtp rdr pass on $ext_if proto tcp from any to any port smtp \ - 127.0.0.1 port spamd -- Darrin Chandler| Phoenix BSD User Group | MetaBUG dwchand...@stilyagin.com | http://phxbug.org/ | http://metabug.org/ http://www.stilyagin.com/ | Daemons in the Desert | Global BUG Federation [demime 1.01d removed an attachment of type application/pgp-signature]
