Score on sender domain by country
Hi, One of our clients has a purely local business and wants any mail coming from a foreign domain to be given a score for spam I would like to reduce the spam threshold , and then give a negative score for every mail with sender domain in India Is there a possibility of identifying the country where a domain is registered. Identyfying by tld seems in-correct Thanks Ram
spamd child hangs
Hi, I am running SA 3.3.1 on FreeBSD 8 amd64 Some of the child process use %100 CPU. When I check the process activity with truss -p PID, the result is absolutely nothing. lsof displays CLOSED or CLOSE_WAIT status connections. Here is an example lsof output from a hang spamd child. If I restart spamd or kill this process. the cpu load decreases. There is no error in the log files. this is happening randonly 2-3 times in a day. PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 43021 root1 1180 178M 147M CPU33 121:28 100.00% perl # lsof -p 43021 COMMAND PIDUSER FD TYPE DEVICE SIZE/OFFNODE NAME perl43021 mail rtd VDIR 0,90 512 2 / perl43021 mail txt VREG 0,92 7152 213112 /usr/local/bin/perl5.10.1 perl43021 mail txt VREG 0,90 246776 800769 /libexec/ld-elf.so.1 perl43021 mail txt VREG 0,92 1636236 237075 /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so perl43021 mail txt VREG 0,90 154320 471045 /lib/libm.so.5 perl43021 mail txt VREG 0,9033792 471043 /lib/libcrypt.so.5 perl43021 mail txt VREG 0,9064856 471050 /lib/libutil.so.8 perl43021 mail txt VREG 0,90 1295416 471042 /lib/libc.so.7 perl43021 mail txt VREG 0,9229907 237545 /usr/local/lib/perl5/5.10.1/mach/auto/Socket/Socket.so perl43021 mail txt VREG 0,9224660 237345 /usr/local/lib/perl5/5.10.1/mach/auto/IO/IO.so perl43021 mail txt VREG 0,9228857 285378 /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Socket6/Socket6.so perl43021 mail txt VREG 0,9221204 237327 /usr/local/lib/perl5/5.10.1/mach/auto/Fcntl/Fcntl.so perl43021 mail txt VREG 0,92 122478 237364 /usr/local/lib/perl5/5.10.1/mach/auto/POSIX/POSIX.so perl43021 mail txt VREG 0,9229249 354309 /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Time/HiRes/HiRes.so perl43021 mail txt VREG 0,9211255 237575 /usr/local/lib/perl5/5.10.1/mach/auto/Sys/Hostname/Hostname.so perl43021 mail txt VREG 0,9219446 237354 /usr/local/lib/perl5/5.10.1/mach/auto/MIME/Base64/Base64.so perl43021 mail txt VREG 0,9228745 237330 /usr/local/lib/perl5/5.10.1/mach/auto/File/Glob/Glob.so perl43021 mail txt VREG 0,9234634 401718 /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/NetAddr/IP/Util/Util.so perl43021 mail txt VREG 0,9269083 401705 /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/HTML/Parser/Parser.so perl43021 mail txt VREG 0,9212408 285467 /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Net/DNS/DNS.so perl43021 mail txt VREG 0,9242618 237290 /usr/local/lib/perl5/5.10.1/mach/auto/Data/Dumper/Dumper.so perl43021 mail txt VREG 0,9215190 237284 /usr/local/lib/perl5/5.10.1/mach/auto/Cwd/Cwd.so perl43021 mail txt VREG 0,9235205 237351 /usr/local/lib/perl5/5.10.1/mach/auto/List/Util/Util.so perl43021 mail txt VREG 0,9256047 237302 /usr/local/lib/perl5/5.10.1/mach/auto/Digest/SHA/SHA.so perl43021 mail txt VREG 0,9251901 237286 /usr/local/lib/perl5/5.10.1/mach/auto/DB_File/DB_File.so perl43021 mail txt VREG 0,9228186 285392 /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Digest/SHA1/SHA1.so perl43021 mail txt VREG 0,9222327 237578 /usr/local/lib/perl5/5.10.1/mach/auto/Sys/Syslog/Syslog.so perl43021 mail txt VREG 0,9225741 285502 /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Razor2/Preproc/deHTMLxs/deHTMLxs.so perl43021 mail txt VREG 0,90 2498211 424313 /var/db/spamassassin/compiled/5.010/3.003001/auto/Mail/SpamAssassin/CompiledRegexps/body_0/body_0.so perl43021 mail txt VREG 0,9011641 424585 /var/db/spamassassin/compiled/5.010/3.003001/auto/Mail/SpamAssassin/CompiledRegexps/body_500/body_500.so perl43021 mail txt VREG 0,9239291 237543 /usr/local/lib/perl5/5.10.1/mach/auto/SDBM_File/SDBM_File.so perl43021 mail0r VCHR 0,29 0t0 29 /dev/null perl43021 mail1u PIPE 0xff000e53f9e00 -0xff000e53f888 perl43021 mail2u PIPE 0xff000e53f9e00 -0xff000e53f888 perl43021 mail3r VREG 0,92 108282 216586 /usr/local/bin/spamd perl43021 mail4u PIPE 0xff000e53f9e00 -0xff000e53f888 perl43021 mail5u IPv4 0xff01672cc000 0t0 TCP localhost.localdomain:783 (LISTEN) perl43021 mail6u unix 0xff01677ab000 0t0 -(none) perl43021 mail
[OT] If you can read french please...
If you can read french please look in this message http://devel.debian.tamay-dogan.net/tmp/joke_spam.001.txt read the Subject: and then the SA results. :-D How big must this idiots be? :-/ Note: They have bombed my domains tamay-dogan.net and tdwave.net with more then 3000 identical messages on 2100 accounts. Unfortunately this message was not stoped on SMTP-Level because I use only zen.spamhaus.org here. Thanks, Greetings and nice Day/Evening Michelle Konzack -- # Debian GNU/Linux Consultant ## Development of Intranet and Embedded Systems with Debian GNU/Linux itsystems@tdnet France EURL itsystems@tdnet UG (limited liability) Owner Michelle KonzackOwner Michelle Konzack Apt. 917 (homeoffice) 50, rue de Soultz Kinzigstraße 17 67100 Strasbourg/France 77694 Kehl/Germany Tel: +33-6-61925193 mobil Tel: +49-177-9351947 mobil Tel: +33-9-52705884 fix http://www.itsystems.tamay-dogan.net/ http://www.flexray4linux.org/ http://www.debian.tamay-dogan.net/ http://www.can4linux.org/ Jabber linux4miche...@jabber.ccc.de Linux-User #280138 with the Linux Counter, http://counter.li.org/ signature.pgp Description: Digital signature
Re: [OT] If you can read french please...
On 11/04/2011 8:58 AM, Michelle Konzack wrote: If you can read french please look in this message http://devel.debian.tamay-dogan.net/tmp/joke_spam.001.txt read the Subject: and then the SA results. :-D How big must this idiots be? :-/ Note: They have bombed my domainstamay-dogan.net andtdwave.net with more then 3000 identical messages on 2100 accounts. Unfortunately this message was not stoped on SMTP-Level because I use onlyzen.spamhaus.org here. Thanks, Greetings and nice Day/Evening Michelle Konzack Server not found Firefox can't find the server at devel.debian.tamay-dogan.net. Regards, Rick
Re: Score on sender domain by country
On Mon, 11 Apr 2011, Ramprasad wrote: One of our clients has a purely local business and wants any mail coming from a foreign domain to be given a score for spam I would like to reduce the spam threshold , and then give a negative score for every mail with sender domain in India Is there a possibility of identifying the country where a domain is registered. Identyfying by tld seems in-correct It's also weak in the face of forgery. Much more useful is identifying the countries where the MTAs are located. Take a look at the RelayCountry plugin. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Gun Control laws cannot reduce violent crime, because gun control laws assume a violent criminal will obey the law. --- 2 days until Thomas Jefferson's 268th Birthday
Re: RCVD_IN_SORBS_DUL on my own emails to self
Am 2011-04-09 15:50:36, hacktest Du folgendes herunter: Does your header definitely include an ESMTP marker as per the RFC? Mine didn't; that was the real issue. We didn't find a bug in this rule. So I guess SpamAssassin doesn't have a way to find out that you were authenticated and that it was your own message. On Apr 9, 2011, at 5:59 PM, Michelle Konzack wrote: Yes, look into my previous message... However, I find SORBS too errorprone and not very reliabel! Thanks, Greetings and nice Day/Evening Michelle Konzack On 10.04.11 15:30, Jonathan Nichols wrote: Sadly, I have to agree and have been dealing with that for a while. in fact, I wonder if this message will ever make it to the list or if apache.org will bounce it because of SORBS.. :/ back on topic... is there a way to lower the score for a particular ruleset for certain hosts/clients? there's trusted_networks setting that will make SA skip checking of those IP's. However blacklists like PBL and DUL are only being checked on internal_networks boundary, that is, only for machines that deliver mail to your network. If the problem lies in dialup machines sending mail directly to your mailhost without authentication (or your mailhost does not mark authenticated mail the way SA understands), trusted_networks should help here. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Spam = (S)tupid (P)eople's (A)dvertising (M)ethod
Re: Retrieve specific word, phrases or sentances from mail body and subject
On 4/10/2011 4:43 PM, John Hardin wrote: On Sun, 10 Apr 2011, rokdominko wrote: The problem is, we want to tell our client exactly which words, phrases or sentences are problematic, so we need Spamassassin to return the list of these words, phrases or sentences, so that we can tell our client what exactly is wrong with their message. We have written a PHP script, which connects to spamd process on our server (on port 783) and it checks the message with no problems and if it's spam it doesn't allow sending it. That level of detail isn't available via spamd. You'd have to run spamassassin in debug mode with rules tracing and then parse the results. Take a look at the output from: spamassassin -t --debug area=rules your_message_file Keep in mind that when using the '-t' flag, spamassassin will always claim the mail is spam. You will need to ignore this and focus on the score instead. -- Bowie
RelayCountry plugin: make it capable to use IP::Country alternatives [Was: Score on sender domain by country]
John Hardin jhar...@impsec.org wrote: [...] Much more useful is identifying the countries where the MTAs are located. Take a look at the RelayCountry plugin. BTW It would be nice (and IMHO simple) to make RelayCountry plugin capable to use IP::Country *OR* (e.g.) Geo::IPfree modules. WHY: Debian does not provide libip-country-perl package (debianized IP::Country module). -- [plen: Andrew] Andrzej Adam Filip : a...@onet.eu In the long run, every program becomes rococco, and then rubble. -- Alan Perlis
Re: Score on sender domain by country
Is there a possibility of identifying the country where a domain is registered. Identyfying by tld seems in-correct ifplugin Mail::SpamAssassin::Plugin::RelayCountry header RELAY_IN X-Relay-Countries =~ /\bIN\b/ describeRELAY_IN Relayed through India score RELAY_IN 1.0 header RELAY_STAR X-Relay-Countries =~ /\*\*/ describeRELAY_STAR Relayed through RFC1918 score RELAY_STAR 0.1 # Note that the X-Relay-Countries header is by default a pseudo header # that isn't actually added to the message, but can be matched by rules # and used by bayes. # See also: # Docs for Relay Country # http://spamassassin.apache.org/full/3.1.x/dist/doc/Mail_SpamAssassin_Plugin_RelayCountry.html # Docs for IP::Country # http://www.annocpan.org/~NWETTERS/IP-Country-2.22/lib/IP/Country.pm # List of ISO 3166 2-character country codes # http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 endif # Mail::SpamAssassin::Plugin::RelayCountry more strict rule meta it with spf or dkim on sender domain exsample headers for RU tld # header __HRD_SENDER_RU From:addr =~ /@((([a-zA-Z0-9])|\.|\-)+)\.ru(\.?)$/i # header __HRD_RECIPIENT_RU From:addr =~ /@((([a-zA-Z0-9])|\.|\-)+)\.ru(\.?)$/i # header __HDR_ENVFROM_RU EnvelopeFrom:addr =~ /@((([a-zA-Z0-9])|\.|\-)+)\.ru(\.?)$/i # header __HDR_RCVD_RU Received:raw =~ /from([[:blank:]]+((([a-zA-Z0-9])|\.|\-)+)\.ru(\.?)[[:blank:]])/i # meta HDR_CCTLD_RU __HRD_SENDER_RU || __HRD_RECIPIENT_RU || __HDR_ENVFROM_RU || __HDR_RCVD_RU # score HDR_CCTLD_RU 0.01 if your mta is postfix then check Return-Path header
Re: Score on sender domain by country
Hi Ram, At 23:34 10-04-2011, Ramprasad wrote: One of our clients has a purely local business and wants any mail coming from a foreign domain to be given a score for spam I would like to reduce the spam threshold , and then give a negative score for every mail with sender domain in India Is there a possibility of identifying the country where a domain is registered. Identyfying by tld seems in-correct No. You mentioned that using the ccTLD for negative scoring isn't what you want. If you assume that senders will be sending the mail from an IP address (or ASN) generally used within the country, you can put in a score for such a rule. You may have to allow some exceptions (e.g. by domain name). Regards, -sm
Re: [OT] If you can read french please...
Hello Rick Macdougall, Am 2011-04-11 09:11:04, hacktest Du folgendes herunter: Server not found Firefox can't find the server at devel.debian.tamay-dogan.net. Sorry, my fault! I had a PosgreSQL replication problem... :-/ because dns1 is one of my new servers and is currently not running correctly. (had to disable DNSSEC) Now it should work... 8-- [michelle.konzack@michelle1:~] dig ANY devel.debian.tamay-dogan.net @dns1.tamay-dogan.net devel.debian.tamay-dogan.net. 3600 IN CNAME mail.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns2.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns3.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns1.tamay-dogan.net. dns1.tamay-dogan.net. 3600IN A 78.47.104.44 dns2.tamay-dogan.net. 3600IN A 217.147.94.23 dns3.tamay-dogan.net. 3600IN A 78.47.247.21 [michelle.konzack@michelle1:~] dig ANY devel.debian.tamay-dogan.net @dns2.tamay-dogan.net devel.debian.tamay-dogan.net. 3600 IN CNAME mail.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns3.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns1.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns2.tamay-dogan.net. dns1.tamay-dogan.net. 3600IN A 78.47.104.44 dns2.tamay-dogan.net. 3600IN A 217.147.94.23 dns3.tamay-dogan.net. 3600IN A 78.47.247.21 [michelle.konzack@michelle1:~] dig ANY devel.debian.tamay-dogan.net @dns2.tamay-dogan.net devel.debian.tamay-dogan.net. 3600 IN CNAME mail.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns2.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns3.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns1.tamay-dogan.net. dns1.tamay-dogan.net. 3600IN A 78.47.104.44 dns2.tamay-dogan.net. 3600IN A 217.147.94.23 dns3.tamay-dogan.net. 3600IN A 78.47.247.21 [michelle.konzack@michelle1:~] dig ANY devel.debian.tamay-dogan.net @dns.private.tamay-dogan.net ;; Truncated, retrying in TCP mode. devel.debian.tamay-dogan.net. 3600 IN CNAME mail.tamay-dogan.net. devel.debian.tamay-dogan.net. 3600 IN RRSIG CNAME 5 4 3600 20110503204700 20110403204700 22362 debian.tamay-dogan.net. tCuMWipWLVwR3a3PSOp1Z30yY524XuqODzCT3Um20gB6zk3GhrXLheyf oCJFFRK5T+z5HAhXr67GMCQPI1c3GYr95RKYDVyXFdH4PMBzxYZN8SZ8 b7n0xDf6gy/Uq6jq36rt1Oql4NWxFNkyyYLRBF4XGuE9O23p3h/kwwrY Ops= devel.debian.tamay-dogan.net. 604800 IN NSECdocs.debian.tamay-dogan.net. CNAME RRSIG NSEC devel.debian.tamay-dogan.net. 604800 IN RRSIG NSEC 5 4 604800 20110503204700 20110403204700 22362 debian.tamay-dogan.net. snQpR+4m3Mrx+st0iOgMWvDw8TZEaxc/VkMb8oaCYz4hfq7ZT0rhd0GN GdvykWnBEo9cftflXpQF3K4SjI8NA0tfjsOvOijCy4WSZG7pQsOuZNp3 0ODDnQTlxPFKeU6zixQluH4IwM8isihlrgQ7sjsGLS9mse03iMHVsRWI lhk= debian.tamay-dogan.net. 3600IN NS dns2.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns1.tamay-dogan.net. debian.tamay-dogan.net. 3600IN NS dns3.tamay-dogan.net. dns1.tamay-dogan.net. 3600IN A 78.47.104.44 dns2.tamay-dogan.net. 3600IN A 217.147.94.23 8-- Regards, Rick Thanks, Greetings and nice Day/Evening Michelle Konzack -- # Debian GNU/Linux Consultant ## Development of Intranet and Embedded Systems with Debian GNU/Linux itsystems@tdnet France EURL itsystems@tdnet UG (limited liability) Owner Michelle KonzackOwner Michelle Konzack Apt. 917 (homeoffice) 50, rue de Soultz Kinzigstraße 17 67100 Strasbourg/France 77694 Kehl/Germany Tel: +33-6-61925193 mobil Tel: +49-177-9351947 mobil Tel: +33-9-52705884 fix http://www.itsystems.tamay-dogan.net/ http://www.flexray4linux.org/ http://www.debian.tamay-dogan.net/ http://www.can4linux.org/ Jabber linux4miche...@jabber.ccc.de ICQ#328449886 Linux-User #280138 with the Linux Counter, http://counter.li.org/ signature.pgp Description: Digital signature
Results you can expect with my IP reputation system
Without contributing any data: RCVD_IN_IPREP_100 hits 29.104% ham, 0.396% spam. S/O = 0.013. RCVD_IN_IPREP_0 hits 0.461% ham, 7.470% spam. S/O = 0.942. It looks like there are plenty of rules in active use by spamassassin which do worse. After uploading a list of which IPs from 100 emails sent spam or ham: RCVD_IN_IPREP_100 hits 63.568% ham, 0.396% spam. S/O = 0.006. RCVD_IN_IPREP_0 hits 0.461% ham, 29.259% spam. S/O = 0.984. And I don't expect many to provide data on 3,500 emails, but to show you where this goes: RCVD_IN_IPREP_100 hits 90.117% ham, 0.396% spam. S/O = 0.004 RCVD_IN_IPREP_0 hits 0.251% ham, 50.283% spam. S/O = 0.995 Detailed graph of the progression: http://www.chaosreigns.com/iprep/results.svg (Three lines for each value from three runs, variance due to random selection of training vs. testing sets.) This was the result of training on data from everyone I have data from except myself, and then testing on my own data. I split my data in half, half for training, and half for testing. I trained 1 ham and 1 spam at a time (so the numbers above assume equal amounts of ham and spam), and recalculated the score each time, using the testing half. Since the data from my email is a fairly significant portion of the data I have still, I'm hoping that others will actually get better results. Spamassassin rules to use it (currently via DNS), and instructions for contributing data, are here: http://www.chaosreigns.com/iprep/ I'm still anxious to get data from more people to increase the usefulness of this for everybody. (Just a list of IPs, time stamps, and whether they were spam or not, collected and uploaded by my script.) If anything is at all unclear, please ask. This is entirely free to everyone. S/O is a score used by spamassassin ruleqa to judge usefulness of a test. Numbers closer to 0.000 are better for finding ham, and numbers closer to 1.000 are better for finding spam. It's calculated as (% spam hits) / (% spam hits + % ham hits) hence Spam / Overall. -- You will need: a big heavy rock, something with a bit of a swing to it... perhaps Mars - How to destroy the Earth http://www.ChaosReigns.com
