Re: user_prefs questions/problem
On Mon, 20 Sep 2010, Chuck Campbell wrote:
> > enabled). Is SA integrated in your mail system in a way that it "knows"
>
> Not sure where to enable this. Will dig more in the docs.
>
> > the user name of the recipient? (some integration methods do not make that
> > info avaialble to SA so the per-user prefs don't work).
> > Have you checked to make sure that your user_prefs are available/readable
> > to the SA daemon?
>
> How do I test this?
Assuming you're running spamd with standard logging enabled, look at the
spamd logs. You should see the username associated in each log entry. EG:
Sep 20 17:39:24 server33 spamd[20757]: spamd: connection from
s-l104.engr.uiowa.edu [128.255.17.210] at port 36478
Sep 20 17:39:24 server33 spamd[20757]: spamd: checking message
<20100920163923.oyqyrtl...@mx1.whitebeek.com> for astockda:115
Sep 20 17:39:25 server33 spamd[20757]: spamd: identified spam (29.1/6.0) for
astockda:115 in 1.2 seconds, 19513 bytes.
Sep 20 17:39:25 server33 spamd[20757]: spamd: result: Y 29 -
BAYES_99,COMBINED_FROM,FS_DEGREE,FVGT_m_MULTI_ODD2,HTML_90_100,HTML_MESSAGE,HTML_TAG_BALANCE_BODY,HTML_TINY_FONT,L_CLAMAV,MY_CLAMAV,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E8_51_100,RAZOR2_CHECK,SPF_PASS,T__BOTNET_NOTRUST,T__MY_CLAMAV,URIBL_BLACK,URIBL_JP_SURBL,URIBL_OB_SURBL,URIBL_WS_SURBL
scantime=1.2,size=19513,user=astockda,uid=115,required_score=6.0,rhost=s-l104.engr.uiowa.edu,raddr=128.255.17.210,rport=36478,mid=<20100920163923.oyqyrtl...@mx1.whitebeek.com>,bayes=1,autolearn=spam
That "for astockda" and "user=astockda" part is the username that spamd
received from the milter that I use to connect sendmail to spamd
To check user_prefs readablilty, do this:
1) on the machine running spamd (or what ever SA mechanism) login (or su)
to the user in question.
2) create or obtain a test mail message, store in a text file.
3) run it thru spamassassin in debug mode:
% spamassassin -D < test-message.txt > /tmp/test.out 2>&1
Then grep for 'user' in the output file:
% grep user /tmp/test.out
[21751] dbg: config: using "/home/bill/.spamassassin" for user state dir
[21751] dbg: config: using "/home/bill/.spamassassin/user_prefs" for user prefs
file
[21751] dbg: config: read file /home/bill/.spamassassin/user_prefs
[21751] dbg: Botnet: adding (\b|\d)user(\b|\d) to botnet_clientwords
Note the line that says "read file /home/bill/.spamassassin/user_prefs"
that file should exist and be readable by your spamd process AND be the
file that you've put the user config stuff in.
--
Dave Funk University of Iowa
College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
#include
Better is not better, 'standard' is better. B{
Re: Making the Botnet plugin work with IPv6
On Mon, 20 Sep 2010 20:03:43 +0200 Yves Goergen wrote: > I'm currently testing a rather simple fix: I've added the following > line to Botnet.cf to ignore anything from IPv6 (hope it works): Alternately you can do this by rewriting the BOTNET rule as a metarule (see Botnet.variants.txt) and incorporating an IPv6 check into that. I haven't checked, but it may be that not all of the subtests fail under IPv6 and something can be salvaged. Also some of the functionality might be recreated using header rules e.g. BOTNET_NORDNS could supplemented by RDNS_NONE.
Re: user_prefs questions/problem
On Sat, Sep 18, 2010 at 06:46:18PM -0500, Dave Funk wrote: > On Sat, 18 Sep 2010, Chuck Campbell wrote: > > >I have SA set up and working (mostly) on my mail machine, however I've put > >the > >following into my user_prefs: > > > >whitelist_from *...@zyngamail.com > >and > >whitelist_from_rcvd *...@zyngamail.com zyngamail.com > >and > >whitelist_allows_relays *...@zyngamail.com > > > > > >when email comes in from them it is flagged as spam. > > > >Here is an example spam report from the headers: > > > >X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on helium.inexs.com > >X-Spam-Flag: YES > >X-Spam-Level: * > >X-Spam-Status: Yes, score=5.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, > > DKIM_VALID_AU,FSL_HELO_NON_FQDN_1,HTML_MESSAGE,MIME_HEADER_CTYPE_ONLY, > > MIME_HTML_ONLY,RDNS_NONE,SPF_FAIL,TO_NO_BRKTS_NORDNS_HTML > >+autolearn=disabled > [snip..] > >I have contacted them about their SPF failure, and am awaiting a response, > >but > >I don't see any reference to my whitelisting, is it not working as I > >anticipated? > > > >If it is there, and I just don't see it, let me know where, and how to > >increase it's score to ensure whitelisted emails are not flagged as spam. > > > >thanks, > >-chuck > First, thanks for the reply. > I assume when you say "into my user_prefs" you mean the per-user prefs > (as opposed to the site-wide config). yes, those in the /home/user/.spamassassin/user_prefs file > > Have you enabled per-user prefs in the site-wide config? (default is not > enabled). Is SA integrated in your mail system in a way that it "knows" Not sure where to enable this. Will dig more in the docs. > the user name of the recipient? (some integration methods do not make that > info avaialble to SA so the per-user prefs don't work). > Have you checked to make sure that your user_prefs are available/readable > to the SA daemon? How do I test this? > > Try putting those white-list statements in the global "local.cf" config > file (do a --lint check & what ever steps are necessary to restart SA so > that it will see the changes) and then test to see if the whitelist works. Done this in /etc/mail/spamassassin/local.cf waiting for appropriate message(s) to arrive and check headers. > If it does then there's an issue with your per-user prefs, if not then > there's an issue with the white-listing itself. I'll be back...
Making the Botnet plugin work with IPv6
Hi there, I've been upgrading to IPv6 yesterday and needed to find out that the Botnet plugin causes false positives on every message that comes in from an IPv6 address. The only information that I've found on that was a thread from January 2010 [1] that contains some debug output and a "good luck" wish. The plugin author doesn't seem to exist anymore. Now I'd definitely like to make this work again. I needed to disable the plugin for now. But unfortunately I can hardly read Perl and not write very much of it. So I'll likely break more than I repair. A problem seems to exist in Botnet.pm from line 760 on where the octets of an IPv4 address are split and computed in some way. I believe the easiest fix for now (as IPv6 isn't so widespread yet) would be to make that function return false when it is passed an IPv6 address (check for colon in the $ip variable). I'm currently testing a rather simple fix: I've added the following line to Botnet.cf to ignore anything from IPv6 (hope it works): botnet_skip_ip : Can anybody assist me with this issue? [1] http://www.mail-archive.com/users@spamassassin.apache.org/msg70589.html (and other copies) -- Yves Goergen "LonelyPixel" Visit my web laboratory at http://beta.unclassified.de
Re: explain DKIM_ADSP_DISCARD?
Michael, > these new dkim tests confuse me :-) > DKIM_ADSP_DISCARD=1.8 > seems like it decided this is a forged email? It is implemented as per RFC 5617. > seems ok to me. I can pastebin it if someone wants. > spf pass, dkim passes (dkim_valid), rcvd_in_rp_certified, > rcvd_in_dnswl_med, rdns and envelope from seem to match. > > AH.. braniacs at ebay... envelope from is members.ebay.com. > dkim signature has d=ebay.com ADSP des not care for envelope sender address. All it cares is an exact match between the From header field and the d= tag (a signing domain) in the DKIM signature. If that From is also @members.ebay.com and d is ebay.com, that would be a mismatch and ADSP would strike in. > is that what adsp_discard means? that even though the dkim signature > matched, the domain in the envelope from didn't match the domain that > the signature says it signed? Which of the ADSP outcomes (unknown/all/discardable) would you end up with depends in either the published or overridden ADSP policy. The published ADSP policy (assuming From: @members.ebay.com) would be queried as: $ host -t txt _adsp._domainkey.members.ebay.com which in this case does not exist, so it defaults to "unknown". But the 60_adsp_override_dkim.cf overrides this policy by an "discardable": adsp_override *.ebay.com $ man Mail::SpamAssassin::Plugin::DKIM Absence of this second parameter implies "discardable" which scores a conservative 1.8 points, not to penalize mistakes at the signing or the receiving side. But yes, technically it is a forgery as far as ADSP is concerned. Mark
explain DKIM_ADSP_DISCARD?
these new dkim tests confuse me :-) DKIM_ADSP_DISCARD=1.8 seems like it decided this is a forged email? seems ok to me. I can pastebin it if someone wants. spf pass, dkim passes (dkim_valid), rcvd_in_rp_certified, rcvd_in_dnswl_med, rdns and envelope from seem to match. AH.. braniacs at ebay... envelope from is members.ebay.com. dkim signature has d=ebay.com is that what adsp_discard means? that even though the dkim signature matched, the domain in the envelope from didn't match the domain that the signature says it signed? -- Michael Scheidell, CTO o: 561-999-5000 d: 561-948-2259 ISN: 1259*1300 > *| *SECNAP Network Security Corporation * Certified SNORT Integrator * 2008-9 Hot Company Award Winner, World Executive Alliance * Five-Star Partner Program 2009, VARBusiness * Best in Email Security,2010: Network Products Guide * King of Spam Filters, SC Magazine 2008 __ This email has been scanned and certified safe by SpammerTrap(r). For Information please see http://www.secnap.com/products/spammertrap/ __
Re: New plugin: DecodeShortURLs
On 20/09/10 15:28, Bowie Bailey wrote: You can get rid of the 'backslashitis' by using a different delimiter. uri URI_BITLY_BLOCKED m~^http://bit\.ly/a/warning~i You still need to escape the period, but since the tilde (~) is now the delimiter rather than the slash, you don't need to escape all the slashes. This is very useful for URI patterns! Just remember that you will now need to escape the new delimiter if it appears in the regex. Thanks for the tip; I did know about using different delimiters - but using / is force of habit ;-) I'll try and remember to use something different for uri rules. Cheers, Steve.
Re: New plugin: DecodeShortURLs
On 20/09/10 16:17, Michael Scheidell wrote: On 9/20/10 8:15 AM, Steve Freegard wrote: Caching; if desired it will now cache URLs to a SQLite database for additional speed-up and to prevent DoS of the shortener services. any anticipated write lock problems with this due to sqlite not handling multi-threaded reads/writes? No; the module handles locking just fine. If you get two writes at once; the first will block until the other completes. Being as this is a very simple database this is unlikely to cause any problems. Also; if you're really concerned about this or have reason to believe you'll have high enough concurrency on the local machine for this to be a problem, then see http://www.sqlite.org/wal.html which is new in SQLite 3.7. most (many?) SA installs already have db4. I guess maybe, hey, its open source, get out your flowchart guys and write the db4 module :-) I didn't use db4 as I'm not familiar enough with it. I also needed more than just a key->value store to handle the expiry of old data efficiently using indexed deletes and I've no idea how to do that in db4. So patches welcome if someone really wants this. Regards, Steve.
Re: New plugin: DecodeShortURLs
On 9/20/10 8:15 AM, Steve Freegard wrote: Caching; if desired it will now cache URLs to a SQLite database for additional speed-up and to prevent DoS of the shortener services. any anticipated write lock problems with this due to sqlite not handling multi-threaded reads/writes? most (many?) SA installs already have db4. I guess maybe, hey, its open source, get out your flowchart guys and write the db4 module :-) -- Michael Scheidell, CTO o: 561-999-5000 d: 561-948-2259 ISN: 1259*1300 > *| *SECNAP Network Security Corporation __ This email has been scanned and certified safe by SpammerTrap(r). For Information please see http://www.secnap.com/products/spammertrap/ __
Re: autolearn : lock_file
Cédric Jeanneret wrote: Hello, I have an error with SA using autolearn plugin: Sep 20 12:25:06 hostname spamd[6157]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /home/USER/.spamassassin/bayes.lock.host.domain.ltd.6157 for /home/USER/.spamassassin/bayes.lock: Permission denied Is it possble to define the lockfile to, say, /tmp/ ? As I don't have only one user, it can be nice to set the lockfile somewhere else on the system, where SA process can write. I didn't see anything about such a configuration variable SA runs as "vmail" user, if it can help. Thank you ! C. SA can write to the users directories. You have something mis-configured or you hosed the perms in the directory.
Re: New plugin: DecodeShortURLs
On 9/20/2010 8:15 AM, Steve Freegard wrote: > On 17/09/10 14:48, RW wrote: >> >> I think it might be better to take the "blocked page" handling out of >> the perl and turn it into an ordinary uri rule. >> > > Yeah; really don't know why I did it like that in the first place. > > I've just uploaded version 0.2 which does it this way instead and adds > the following rule: > > uri URI_BITLY_BLOCKED /^http:\/\/bit\.ly\/a\/warning/i You can get rid of the 'backslashitis' by using a different delimiter. uri URI_BITLY_BLOCKED m~^http://bit\.ly/a/warning~i You still need to escape the period, but since the tilde (~) is now the delimiter rather than the slash, you don't need to escape all the slashes. This is very useful for URI patterns! Just remember that you will now need to escape the new delimiter if it appears in the regex. -- Bowie
Re: autolearn : lock_file
On Mon, 20 Sep 2010, Cédric Jeanneret wrote: I have an error with SA using autolearn plugin: Sep 20 12:25:06 hostname spamd[6157]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /home/USER/.spamassassin/bayes.lock.host.domain.ltd.6157 for /home/USER/.spamassassin/bayes.lock: Permission denied Is it possble to define the lockfile to, say, /tmp/ ? As I don't have only one user, it can be nice to set the lockfile somewhere else on the system, where SA process can write. I didn't see anything about such a configuration variable SA runs as "vmail" user, if it can help. Having a global lock file for per-user bayes databases is not optimal... -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- The Constitution is a written instrument. As such its meaning does not alter. That which it meant when adopted, it means now. -- U.S. Supreme Court SOUTH CAROLINA v. US, 199 U.S. 437, 448 (1905) --- 88 days until TRON Legacy
Re: application/octet-stream obfuscated JPEGs
On Mon, 20 Sep 2010, Chip M. wrote: The second part is a new-ish spin: an image using "application/octet-stream" as the Content Type, but otherwise sanely constructed (i.e. it has a full filename with ".jpg", which is the ACTUAL image encoding used, unlike some of his previous morphs). Dangit, I can't believe I didn't include that in my existing obfuscation rules. Added. Sadly, I've seen this particular stupid-spammer-trick before... in ham. :( It's rare enough, and the senders broken enough, that some may feel comfortable penalizing this pattern (maybe a simple test of app/oct with an image file extension?). On the other hand, a significant percentage of the broken mailing lists that use this, do tend to have high value with their recipients. A cautious score is advisable. meta it with "not mailing list". I'll review masscheck results for that sort of thing. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- The Constitution is a written instrument. As such its meaning does not alter. That which it meant when adopted, it means now. -- U.S. Supreme Court SOUTH CAROLINA v. US, 199 U.S. 437, 448 (1905) --- 88 days until TRON Legacy
Re: Yahoo HTML Base64 Attachments
On Mon, 20 Sep 2010, Chip M. wrote:
On 19 Sep 2010, John Hardin wrote:
Adding to my sandbox for masscheck:
rawbody HTML_OBFU_ESC /document\.write\(unescape\("(?:%[0-9a-f]{2}){10}/i
It performs pretty well. It should be in the next rules update, under a
slightly different name (OBFU_JVSCR_ESC).
Shiny!
How about combining/meta-ing that with a simple Base64 HTML rule?
I vaguely recall you may already have one (Base64 rule, not (yet) a
meta).
Based on my ham data, that pairing seems extraordinarily rare.
I'll review the masscheck results for overlap, but in the masscheck
corpora OBFU_JVSCR_ESC is hitting zero ham.
I just dumped the Content Type summary lines for all 58, and if
you're interested, John, I can email them as a zip. Just eyeballing
them, there appears to be some interesting differences in the
filename distribution vs this spam campaign.
Sure, send it along. No guarantees I'll do anything with it, though...
--
John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
---
The Constitution is a written instrument. As such its meaning does
not alter. That which it meant when adopted, it means now.
-- U.S. Supreme Court
SOUTH CAROLINA v. US, 199 U.S. 437, 448 (1905)
---
88 days until TRON Legacy
Re: New plugin: DecodeShortURLs
On 17/09/10 14:48, RW wrote: I think it might be better to take the "blocked page" handling out of the perl and turn it into an ordinary uri rule. Yeah; really don't know why I did it like that in the first place. I've just uploaded version 0.2 which does it this way instead and adds the following rule: uri URI_BITLY_BLOCKED /^http:\/\/bit\.ly\/a\/warning/i describe URI_BITLY_BLOCKED Message contains a bit.ly URL that has been disabled due to abuse score URI_BITLY_BLOCKED 10.0 This new version also contains the following: 1) Improved the URI handling as I noticed it was checking some URIs that it shouldn't have been due to the URI embedding a shortener in a get variable. 2) Log file; if desired it will output the unix time, short url and decoded URL to a log file for further analysis. 3) Caching; if desired it will now cache URLs to a SQLite database for additional speed-up and to prevent DoS of the shortener services. Cheers, Steve.
autolearn : lock_file
Hello, I have an error with SA using autolearn plugin: Sep 20 12:25:06 hostname spamd[6157]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /home/USER/.spamassassin/bayes.lock.host.domain.ltd.6157 for /home/USER/.spamassassin/bayes.lock: Permission denied Is it possble to define the lockfile to, say, /tmp/ ? As I don't have only one user, it can be nice to set the lockfile somewhere else on the system, where SA process can write. I didn't see anything about such a configuration variable SA runs as "vmail" user, if it can help. Thank you ! C.
application/octet-stream obfuscated JPEGs
There's a new morph from our old nuisance, the inline PNG/RTF, and all manner of wavy image insecure-boy-drugs spammer. :( Here's a sample: http://puffin.net/software/spam/samples/0009_jpg_oct.txt It began (here) on Sep 10, and replaced his (relatively boring) "Your wife photos attached" zipped JPEG. This time, it has two parts. The first is plain text, with his often seen before anti-Bayes chunk of text from a copyright expired book. The second part is a new-ish spin: an image using "application/octet-stream" as the Content Type, but otherwise sanely constructed (i.e. it has a full filename with ".jpg", which is the ACTUAL image encoding used, unlike some of his previous morphs). Sadly, I've seen this particular stupid-spammer-trick before... in ham. :( It's rare enough, and the senders broken enough, that some may feel comfortable penalizing this pattern (maybe a simple test of app/oct with an image file extension?). On the other hand, a significant percentage of the broken mailing lists that use this, do tend to have high value with their recipients. A cautious score is advisable. On a bright note, it does have the exact same JPEG header size that I've previously reported (623 bytes). It also continues this spammer's use of random (ALWAYS wrong) Realnames in the To header. Those two tests, plus nation of origin, are my main test hits. So far, none have snuck thru my last layers of defense. - "Chip"
Re: Yahoo HTML Base64 Attachments
On 19 Sep 2010, John Hardin wrote:
>> Adding to my sandbox for masscheck:
>>
>> rawbody HTML_OBFU_ESC /document\.write\(unescape\("(?:%[0-9a-f]{2}){10}/i
>
>It performs pretty well. It should be in the next rules update, under a
>slightly different name (OBFU_JVSCR_ESC).
Shiny!
How about combining/meta-ing that with a simple Base64 HTML rule?
I vaguely recall you may already have one (Base64 rule, not (yet) a
meta).
Based on my ham data, that pairing seems extraordinarily rare.
I just checked all 2010 data for my most diverse domain (three
generations of an extended family, with a superb mix of business
plus personal ham), and found only 58 (out of 66,795) hams with
Base64 HTML.
Of those, ZERO hit any of my anti-script tests, however 49 of them
did have an existing non-trivial pass rule that skips some of those
anti-script tests (in other words, those were already well known
(to us) for their poor mailing hygiene).
I just dumped the Content Type summary lines for all 58, and if
you're interested, John, I can email them as a zip. Just eyeballing
them, there appears to be some interesting differences in the
filename distribution vs this spam campaign.
I checked a similar quantity of data for a pure business domain, and
found ZERO occurrences of Base64 HTML.
As is often the case, choosing tests and scores depends on one's ham
ecology.
>Today: Talk Like a Pirate day
... and Today: Talk Like a Browncoat Day
i.e. the 8th anniversary of the TV broadcast debut of Firefly. :)
Keep flyin',
- "Chip"
Re: New plugin: DecodeShortURLs
Steve Freegard wrote: >Hopefully it will be useful to others; you can grab it from: Thanks Steve! Suggestions (for future enhancements): 1. Consider splitting the list of shorteners between those that are well established and KNOWN to be reasonably diligent, and "all others" (e.g. the anti-pattern ably described last week in: http://www.xkcd.com/792/ ). Split them in such a way as to make it easy for users to test only ONE set (probably the better known ones), and (perhaps) add an option to score the rest without doing a DNS call. 2. Investigate BitLy's API. I've been experimenting with it for a few months, and am very pleased with the options and data it provides. I still need to add ham shortener links to my standard/automated testing (preliminary results are excellent). The only "issue" I had (at the very beginning) was signing up with a mixed case API key, then lower casing it when I used it. My BIM. 3. Please collect and share performance data. Thanks in advance! :) I still haven't deployed anything real-time (have had VERY limited quality-dev-time this year - Grrr!Argh!). Since these first became a problem, I've been auto-quarantining (except for a very short list of manually excluded newsletters and select validated Senders), then we handle the DNS tests as part of our desktop-based FP pipeline. The occurrence of shorteners in ham is low enough that that's been acceptable to our userbase, largely because they run the actual tests, so they have Complete Control. It's been my experience that not-stupid endusers who are given control are happy users. They're full participants in the process. :) - "Chip"
