spam reports

[Michael Grant]

I'm trying to create my own spam report template.

spamassassin seems to be putting in an extra return or linefeed when I do this:

clear_report_template
report _REPORT_

What I get in the headers is this:

X-Spam-Report: \r\n  *  0.6 NO_REAL_NAME From: does not include a
real name\r\n  * -0.0 SPF_PASS SPF: sender matches SPF record\r\n  *
0.0 HTML_MESSAGE BODY: HTML included in message\r\n  \r

The \n after the  seems to make Outlook separate the headers from
the body and all the rest of the report and any following headers come
out in the body of the message.

Is there some way to eliminate the superfluous linefeed or carriage return?

Is there any way to get rid of the ___ lines?

Michael Grant

Re: spam reports

[Michael Grant]

Never mind.  My problems were not coming from spamd at all.  They were
coming from milter-spamc which was calling spamd.

Michael Grant

On 8/29/07, Jari Fredriksson <[EMAIL PROTECTED]> wrote:
> > I'm trying to create my own spam report template.
> >
> > spamassassin seems to be putting in an extra return or
> > linefeed when I do this:
> >
> > clear_report_template
> > report _REPORT_
> >
> > What I get in the headers is this:
> >
> > X-Spam-Report: \r\n  *  0.6 NO_REAL_NAME From: does
> > not include a
> > real name\r\n  * -0.0 SPF_PASS SPF: sender matches SPF
> > record\r\n  *
> > 0.0 HTML_MESSAGE BODY: HTML included in message\r\n
> > \r
> >
> > The \n after the  seems to make Outlook separate the
> > headers from
> > the body and all the rest of the report and any following
> > headers come
> > out in the body of the message.
> >
> > Is there some way to eliminate the superfluous linefeed
> > or carriage return?
> >
> > Is there any way to get rid of the ___ lines?
> >
> > Michael Grant
>
> Clearly the _REPORT_ macro is meant to be used as body, not a header.
>
>
>

spamassassin and gmail

[Michael Grant]

I filter mail on my server and then forward some of it to gmail.

Does anyone out there have a trick you can share to get gmail to
filter on spamassassin's x-spam markup to put mail directly into the
spam folder (or even to set a label)?

I do not want to modify the subject because once you modify the
subject, you can't unmodify it if the mail isn't spam and it stays
that way in gmail's archive forever.

Michael Grant

Re: Razor Problems

[Michael Grant]

On Dec 12, 2007 1:09 AM, Matt Kettler <[EMAIL PROTECTED]> wrote:
> Marc Perkel wrote:
> > What causes this?
> >
> > reporter: razor2 report failed: No such file or directory report
> > requires authentication
> You didn't run razor-admin --register?

Funny, I too just got this same error and yes, I did a razor-agent
-create and -register.

[88199] warn: reporter: razor2 report failed: No such file or
directory report requires authentication at
/usr/local/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/Plugin/Razor2.pm
line 178. at 
/usr/local/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/Plugin/Razor2.pm
line 326.
[88199] info: reporter: could not report spam to Razor

Michael Grant

Re: Razor Problems

[Michael Grant]

On Dec 12, 2007 5:05 AM, Matt Kettler <[EMAIL PROTECTED]> wrote:
>
> Michael Grant wrote:
> > On Dec 12, 2007 1:09 AM, Matt Kettler <[EMAIL PROTECTED]> wrote:
> >
> >> Marc Perkel wrote:
> >>
> >>> What causes this?
> >>>
> >>> reporter: razor2 report failed: No such file or directory report
> >>> requires authentication
> >>>
> >> You didn't run razor-admin --register?
> >>
> >
> > Funny, I too just got this same error and yes, I did a razor-agent
> > -create and -register.
> >
> > [88199] warn: reporter: razor2 report failed: No such file or
> > directory report requires authentication at
> > /usr/local/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/Plugin/Razor2.pm
> > line 178. at 
> > /usr/local/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/Plugin/Razor2.pm
> > line 326.
> > [88199] info: reporter: could not report spam to Razor
> Interesting.. do you get a similar result from razor-report?

Ahh, I had to do a razor-admin like this:

su - root
# razor-admin -create
# razor-admin -register

Even though I had done this initially as just 'su', it was using my
homedir to create the .razor directory.

Michael Grant

spamc/spamd failure

[Michael Grant]

I'm running 3.2.3.  I'm noticing that spamc/spamd fails when it's
presented large messages containing rather large mime attachments
(like more than a megabyte or so).  When I run the messages through
spamc by hand, it returns immediately with a not-spam result with
headers like this:

X-Spam-Flag: NO
X-Spam-Score: 0
X-Spam-Level:

When I run other messages through spamc, it muches on it for a while
before much more normal headers like:

X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on
charm.networkguild.org
X-Spam-Level: 
...

Anyone else seen this?  Anyone have a fix?

By the way, for those running snertsoft's milter-spamc, this shows up
in the log as 'SPAMD status line failure'

Michael Grant

Re: spamc/spamd failure

[Michael Grant]

On Dec 21, 2007 5:13 PM, Theo Van Dinter <[EMAIL PROTECTED]> wrote:
> On Fri, Dec 21, 2007 at 04:58:44PM +0100, Michael Grant wrote:
> > I'm running 3.2.3.  I'm noticing that spamc/spamd fails when it's
> > presented large messages containing rather large mime attachments
>
> It doesn't fail in as much as it doesn't send it to spamd, as per its design.
>
> > Anyone else seen this?  Anyone have a fix?
>
> man spamd
>
> It's suggested to not send messages to spamd that are > 250K in size.

Actually, it doesn't say this in the spamd man page at all.  However,
spamc does have a limit and it states the maximum message size is
256M, not 256K and defaults to 500K.  I see one needs to set the -s
option to an appropriately larger size to get spamc to work with
larger messages.

But in any case, milter-spamc does not use spamc to submit to spamd,
it talks directly to spamd, thus this limitation in spamc this should
not be an issue.  These messages are well below 256M.  milter-spamc
only sends down the first 64K of the message in fact.

Furthermore, if spamd is rejecting the message because of message
size, it would be really good if it returned an error like "message
greater than 256K" or something like that.  What it's doing is closing
the connection (i.e. no status message at all, just an EOF).  Older
versions of spamd do not do this, I have 3.1.9 running on another box
which does not have this behavior.  Both are using the same version of
perl: 5.8.8.

Michael Grant

over zealous awl

[Michael Grant]

I noticed when I use spamassasin -r that it seems to add virtually
every email address inside the email to the auto-whitelist db with
high values (ie it's blacklisting them), even my own address, even
addresses in received header lines.  This isn't what I expected, I
would have expected this to only look at the from header line.  Am I
missing something?  Am I the only one noticing this?

Michael Grant

Re: How many use CRM114?

[Michael Grant]

http://crm114.sourceforge.net/

CRM114 is highly regarded purportedly extremely accurate in discerning
spam from ham.

It can be used as yet another metric for Spamassassin to judge whether
mail is spam or ham.

The main drawback I see using this at a server level is that crm114 is
best trained by telling it that it got something wrong as opposed to
feeding in a lot of ham and telling it it's ham or spam and telling it
it's spam.  I guess you would say that this child learns best by being
punished!

This could prove to be impractical in a server environment used in
conjunction with spamassasin where users are not going to punish
crm114 when it gets it wrong.  This is because the result of crm114
would be burried inside a spamassassin tag as only one metric of
spaminess.  It seems to me that crm114 is probably more appropriate
standalone or at least used in such a way that the result of spam/ham
is fed back into crm114 by the user if the result of crm114 is
incorrect.

Michael Grant

On Tue, Mar 4, 2008 at 4:19 PM, Marc Perkel <[EMAIL PROTECTED]> wrote:
>
>
>
>
>  Andrew Hearn wrote:
>  Blaine Fleming wrote:
>
>
>  Slightly off-topic, but I'm curious, how many of you are using CRM114?
> How well does it work for you? Was it difficult to train? I've been
> looking at it and haven't found much except the official plugin guide
> and a single page saying that it works better than other learning
> methods. Any info would be appreciated.
>
>  Hello
>
> I've only just started using it on a test server, I'll let you know how
> I find the results!
>
>
>
>  CRM114? What's that? Can't quite figure out what it does. Is it a pony? :)
>
>  --
> Marc Perkel - Sales/Support
> [EMAIL PROTECTED]
> http://www.junkemailfilter.com
> Junk Email Filter dot com
> 415-992-3401
>
>

Re: Ideas

[Michael Grant]

I think I see what you're trying to do.  You want to set up a server
which you can use to see how spamass processes individual mail
messages.  A sort of mirror that you can use to see what your message
looks like after it passes through spamass.

Unfortunatly, the only possible use I can think of for having such a
setup is to check a spam message before it gets sent out for real.

Please could you tell us what legitimate use such a server would
serve?.  Procmail isn't the correct answer here.

Michael Grant

On 10/10/06, Robert Swan <[EMAIL PROTECTED]> wrote:





Hi everyone, I am trying to setup a SPAM server to process incoming email
and then send it back to the original sender.





I have setup Spamassassin and Postfix (latest version), and they are working
great. I am trying to figure out how to get Postfix to automatically send
the "processed" e-mail back to the sender with all of the processed info in
it like below, any ideas??





Thanks in advance




Robert



 Content analysis details:   (1.2 points, -5.0 required)



 pts rule name  description

 --
--

 0.1 FH_MSGID_HUGE_40   FH_MSGID_HUGE_40

-0.0 SPF_PASS   SPF: sender matches SPF record

 0.6 HTML_SHORT_LENGTH  BODY: HTML is extremely short

 0.0 HTML_MESSAGE   BODY: HTML included in message

 0.5 DNS_FROM_RFC_ABUSE RBL: Envelope sender in abuse.rfc-ignorant.org



The original message was not completely plain text.

Re: What to do about Domain Spoofing

[Michael Grant]

On 11/28/06, Paul Hurley <[EMAIL PROTECTED]> wrote:


SM wrote:

> At 11:13 27-11-2006, Paul Hurley wrote:
>
>> I own a domain name, which has a small webpage attached.  It would
>> seem by the flurry of mail delivery reports I'm getting that the
>> domain has been added to one of the current stock pump 'n' dump scams.
>
>
> See BATV
>
> Regards,
> -sm
>
Unfortunatley that's not quite possible.  I'm running Spam Assassin for
Windows 32 as a Pop Proxy (http://sourceforge.net/projects/sawin32/), so
I don't have a mail server to control

I can filter the bad bounce messages, I'm more worried if I should be
doing something to inform anyone, or try to stop this, as obviously
other people are receiving spam that appears to be from my domain ?

What's the current feeling on sending abuse@ emails, and if so, to which
domain (I presume the last domain mentioned as a received from ???)

Paul.

--
Paul Hurley http://www.paulhurley.co.uk/
The knack of flying is learning how to throw yourself at the ground and
miss.
Hitchhikers Guide to the Galaxy



In my oppinion you should add an SPF record to your domain.  It's not a sure
fire fix but it does help mail servers able to know that the mail is bogus.

Michael Grant

two databases

[Michael Grant]

I run spamassassin on my mail server from milter-spam
(http://www.snertsoft.com/sendmail/milter-spamc/).  One mail server is
no longer sufficient for both reliability and load so I'm building 2
identical mail servers (two mx mailers).

Ideally I'd like both of them to share the bayes database.  I'm not
keen on the idea adding something like nfs to the mix.  I was thinking
perhaps I could just set up a cron tab to rsync the databases to the
other machine, hence I'd have a local bayes database and a copy of the
remote one on each machine.

How would one use two databases like that?  Or is it good way to
combine them say hourly?  Or is this a totally idiotic idea and is
there some more appropriate way to solve this problem?

I don't want either mailer to depend on the other one like with nfs
where if one box is down the second one freezes too.  I considered
stuff like drbd and a clustered file system.  I'd like to know if
anyone else has some better solutions to this problem.

Michael Grant

Re: two databases

[Michael Grant]

I did not realize one could store the bayes scores in sql.

So I'd store the bayes scores on a third server and let both mxes use
the same database.

Two equal weighted mx records are a good idea.

Michael Grant

Re: two databases

[Michael Grant]

On Fri, Jun 5, 2009 at 16:08, Micah Anderson  wrote:
> Michael Grant  writes:
>
>> I did not realize one could store the bayes scores in sql.
>>
>> So I'd store the bayes scores on a third server and let both mxes use
>> the same database.
>
> I did this, but my bayes in mysql and pointed two different spamd
> machines at it, but I had severe problems that I could not resolve. I
> posted to the list[0] about the problems.
>
> The basic problem was that as soon as I fired up the second server it
> immediately starts blocking on the bayes work. Average scantimes go from
> 1-2 seconds up to 35+ and the max children get eaten up by blocking on
> the bayes work to the point where its pointless because too many
> processes are blocked. Disabling the bayes_sql stuff on one of the
> machines dropped the scantimes back to their expected average of 1-2
> seconds (but of course none of the BAYES tests will fire and
> autolearning fails).
>
> My mysql server is its own machine, it was local to the first spamd
> (local LAN) and remote to the second (over the net). I eliminated any
> hostname lookup problems, obviously couldn't eliminate network latency,
> but that shouldn't have caused such a severe result. I'm running with
> InnoDB tables, so I shouldn't have any row-level locking issues... in
> any case I might have had some issues because my MySQL database needed
> to be optimized, but I was not able to determine how and now I just run
> one of the spamd's without bayes, which is not too bad because my bayes
> database seems to be totally worthless at the moment. :P
>
> micah
>
> 0. http://permalink.gmane.org/gmane.mail.spam.spamassassin.general/113673
>
>

Wow.  I did not get around to setting this up yet.  But on the MySQL
front, did you try enabling the query cache by adding this to the
mysql command line?

--maximum-query_cache_size=1M

Also, a tool I used a lot to help debug this sort of issue was mytop.

Michael Grant

Re: SORBS bites the dust

[Michael Grant]

Unless I've missed a message... this is the 100th reply to this
thread.  This has to be one of the longest threads I've seen on this
list in years.

I have to say I have issues with your definition of legit mail.  Many
people do send mail to other people out of the blue for legit reasons
other than having some previous relation with that person.

> 4. From gmail, yahoo or hotmail.

These sites do provide an important service for people.  Not everyone
is tech savy to get their own domain name.  If everyone had to use
their ISP's domain name, think of the mess each time you change your
ISP.

But in general, there is definitely a grey area about what is and what
isn't legit email and I have to say that spamassassin does do a pretty
decent job much of the time sorting it out.

Re: constantcontact.com

[Michael Grant]

In defense of Constant Contact, they are in the business of sending
out mailings for people, they are not themselves spammers.  They
perform a service and they do it as best they can given the
circumstances in which they work.

I have used them to send out mail to mailing lists of a non-profit
organization that I help and also used it during the previous
presidential campaign.  All the addresses were collected via people
coming to the website, typing in their address, getting an email from
constant contact and clicking on a "yes, I want to sign up for this
list" link.

All mail was sent out with a return address that went to a real
person, and every message contained a link to get off the mailing.
This is required by Constant Contact.

Secondly, if you unsubscribe using the unsubscribe link, Constant
Contact does not let that address be mailed to again unless it is
re-opted in by signing up again and the person clicking on the opt-in
link.

Constant Contact keeps track of complaints and when it gets above
something like one or two per thousand they cancel the account.

If you are getting spam via them, you should send it to their abuse
department.  They do take the reports seriously.

And by the way, from time to time I receive what surely looks like
spam via Constant Contact.  I save all my mail.  I went back and
searched and sure enough, it *was* something I signed up for but had
completely forgotten.  A simple click of their unsubscribe link and no
more of that.

I would not personally give mail from Constant Contact a higher score
just because it originated from there.  The likelihood is the message
is ham, most likely the user forgot they opted like I did, or perhaps
someone is abusing Constant Comment.

Michael Grant

Re: Amavisd replacement suggestion

[Michael Grant]

Between Mailscanner and Amavisd-new, it seems we need one or the other
of these programs to recursively dig into and possibly uncompress a
message with attachments to be able to virus scan it completely.  Does
Mailscanner do as effective a job as Amavisd in this regard?

When I installed Amavisd a couple years ago now, I seem to remember a
load of dependencies on things like rar, tar, unzip...etc.  I didn't
see a lot of these when I compiled Mailscanner.  (Unzip, yes, but not
some of the other ones).

Michael Grant

Re: Never seen this in headers before

[Michael Grant]

Any header with X- in front of it is a non-standard mail header and
any mailer can stick one of those in if it wants.  This was probably
stuck in by your mailer.  I did a google search for this header and
there are lots and lots of messages out there with this header in it
near or at the top.

Michael Grant

On 3/11/06, Lisa Casey <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I got a couple of those image only spams today but there was something
> different at the top of the headers that I'ld never seen before.
>
> Headers:
>
> X-EMS: wait 10s
> X-EMS: wait 20s
> X-EMS: wait 30s
> Return-Path: <[EMAIL PROTECTED]>
> Received: from p2148-ipbf504marunouchi.tokyo.ocn.ne.jp
> (p2148-ipbf504marunouchi.tokyo.ocn.ne.jp [221.191.114.148])
> etc
>
> What's with the X-EMS wait stuff?
>
> Lisa Casey
>
>

updating spamassassin, inserting blank lines after the X-Spam-Status header

[Michael Grant]

I updated from 3.0.2 to 3.1.1 (I also tried 3.1.2) and I'm now seeing
blank lines (3 of them!) inserted after the X-Spam-Status: header.
Has anyone else seen this?

I can reproduce the problem easily on the command line simply by
piping a message to the spamassassin command.

I have a feeling this is some bad interaction with something old in
one of my config files because it does not appear to be a global
problem.  Google search didn't turn up much.

I tried moving my .spamassassin dir out of the way, that didn't help.
Tried also with /usr/local/etc/mail/spamassassin and
/usr/local/share/spamassassin.  Where else does spamassassin get it's
config from?

Any ideas on how to track this down would be appreciated.

Michael Grant

updating spamassassin, inserting blank lines after the X-Spam-Status header

[Michael Grant]

I updated from 3.0.2 to 3.1.1 (I also tried 3.1.2) and I'm now seeing
blank lines (3 of them!) inserted after the X-Spam-Status: header.
Has anyone else seen this?

I can reproduce the problem easily on the command line simply by
piping a message to the spamassassin command.

I have a feeling this is some bad interaction with something old in
one of my config files because it does not appear to be a global
problem.  Google search didn't turn up much.

I tried moving my .spamassassin dir out of the way, that didn't help.
Tried also with /usr/local/etc/mail/spamassassin and
/usr/local/share/spamassassin.  Where else does spamassassin get it's
config from?

Any ideas on how to track this down would be appreciated.

Michael Grant

spamassassin on a mail relay

[Michael Grant]

Do any of you out there run spamassassin on a mail relay or pop/imap
server  to add the X-Spam headers to all mail that passes through your
gateway?

If you do, how do you let individual users (who don't have accounts on
your relay) tweak their user_prefs file to whitelist things that are
not spam or otherwise tweek the rules?

Do any of you who use spamassassin at the server level (as opposed to
the user level) use it to reject spam (versus just marking it up)?

I had this idea that something could add a url to the bottom of the
message that would let the user click on it and white/black list the
user back on the server.  Maybe something like this exists already?

I must say that in my own experience, I could not blindly reject mail
with Spamassassin because it has too many false positives with my
mail.

Michael Grant

Re: Do we need a new SMTP protocol? (OT)

[Michael Grant]

I do find this topic interesting, perhaps this isn't the most
appropriate place to discuss it, if not here though, where?

I'd like to make an observation.  More and more people are using
"social network" systems like Facebook in place of email.  Also IM
chatting is replacing a lot of person-to-person email.

I actually get precious little spam through these alternatives to smtp
email.  This is primarily because in order to send me a message
through one of these systems, you have to be on my contact list.

Of course, this only moves the problem one layer up.  In order to set
up that reciprocal agreement, some exchange has to take place, either
out of band, or via an invite (which could be spam).

There are many different IM systems out there now.  It annoys me that
I have to get on lots of different systems just to communicate with
everyone.  I'd prefer to see one unified messaging system.

SMTP email is going to be hard to replace.  It would seem like our
best bet is to bolt something on to it like an IM style contact list
manager where you have to be on someone's contact list to be able to
send them mail.  The main problem with this approach is how does
someone send you mail if they're not on your contact list?  I don't
have any magic answers how to solve that beyond what's already out
there as in return messages with captchas in them or things like Blue
Bottle seem to be quite effective.

Michael Grant

Re: also...

[Michael Grant]

Is it possible they are coming from zombie machines?  Machines which
have been infected by a sort of virus which a spammer can take over
and send out mail from remotely.

Michael Grant

On 4/4/07, J. <[EMAIL PROTECTED]> wrote:


--- Matt Kettler <[EMAIL PROTECTED]> wrote:

> J. wrote:
> > I've been doing this sort of thing to block connections which is
> > somewhat more satisfying than just scoring the email higher, but
> these
> > rascals seems to be able to use multiple ip addresses even within a
> > single mailing:
> >
> > 123.156.189.:allow,RBLSMTPD="-Connections refused. domain.com seems
> to
> > ignore bounces."
> > 87.254.321.:allow,RBLSMTPD="-Connections refused due to spam."
>
> Do they have a common reverse DNS?

Good question. They probably do if they're running email lists and want
the messages to get through. They always seems to come through with low
scores so I assume they've got spf and reverse dns set up right.




No need to miss a message. Get email on-the-go
with Yahoo! Mail for Mobile. Get started.
http://mobile.yahoo.com/mail

spamd: spf: lookup failed: addr is not a string at /usr/share/perl5/IO/Socket/IP.pm line 646.

[Michael Grant]

I'm getting this message in my mail.log:

spamd: spf: lookup failed: addr is not a string at
/usr/share/perl5/IO/Socket/IP.pm line 646.

I'm running debian stable with Spamassassin from backports:

SpamAssassin Server version 3.4.0
  running on Perl 5.14.2

and libmail-dkim-perl version 0.39 from stable (wheezy)

I see from this thread on gossamer that this may have something to do with
libmail-dkim-perl:

http://www.gossamer-threads.com/lists/spamassassin/users/188895

Has anyone made any progress on this?  Anyone found a workaround?

Micael

spamass-milter and email addresses starting with --

[Michael Grant]

I'm running Debian, sendmail, spamass-milter, spamc and spamd.

I saw this in my log:

Jun  9 20:30:29 debian sm-mta[15942]: t5A0ULAA015942: to=<--u...@example.com
>

then I saw this:

Jun  9 20:30:29 strange spamc[15947]: invalid usage
Jun  9 20:30:29 strange spamass-milter[1770]: Thrown error: poll says my
write pipe is busted

That seems pretty scary that someone can send to a user which begins with
-- and fake out spamc that it's a command line option.

Re: spamass-milter and email addresses starting with --

[Michael Grant]

I'm using 0.3.2 which seems to be the latest version available for Debian.
It does appear that 0.4.0 hasn't hit Debian yet.

On Wed, Jun 10, 2015 at 10:34 PM, David B Funk  wrote:

> On Wed, 10 Jun 2015, Michael Grant wrote:
>
>  I'm running Debian, sendmail, spamass-milter, spamc and spamd.
>>
>> I saw this in my log:
>> Jun  9 20:30:29 debian sm-mta[15942]: t5A0ULAA015942: to=<--
>> u...@example.com>
>> then I saw this:
>>
>> Jun  9 20:30:29 strange spamc[15947]: invalid usage
>> Jun  9 20:30:29 strange spamass-milter[1770]: Thrown error: poll says my
>> write pipe is busted
>> That seems pretty scary that someone can send to a user which begins with
>> -- and fake out spamc that it's a command
>> line option.
>>
>
> What version of spamass-milter are you using?
> Older versions of spamass-milter used a "system" call to invoke "spamc"
> and feed it messages, thus had a glaring security vulnerability.
>
> That was fixed a while ago, you need to update your spamass-milter.
>
>
> --
> Dave Funk  University of Iowa
> College of Engineering
> 319/335-5751   FAX: 319/384-0549   1256 Seamans Center
> Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
> #include 
> Better is not better, 'standard' is better. B{

FREEMAIL_REPLYTO

[Michael Grant]

We find FREEMAIL_REPLYTO to be quite successful at weeding out spam so we
raised up to 9.1.  i.e. with this in local.cf:

score FREEMAIL_REPLYTO 9.1

However, it causes a false positive with FREEMAIL_REPLYTO and it got me
very curious:

Here's a sanitized minimal example that triggers this (indented by 4
spaces):

Date: Wed, 8 Mar 2017 03:20:05 + (UTC)
From: Winston 
To: Kipper 
Subject: foo
Reply-To: Winston 

> From: Kipper 
> To: Winston , innocentbystan...@ymail.com
> Subject: bar

Reports:

*  9.1 FREEMAIL_REPLYTO Reply-To/From or Reply-To/body contain
different
*  freemails

The problem is caused by innocentbytan...@ymail.com IN THE BODY!

This seems a bit overzealous.  It seems like a bit of an over-reach to look
at headers in the BODY of the message.

This is an excellent rule except for this rude message body cavity search!

I suggest only searching the headers in this rule.

If you really feel it aught to search the body like this, can you please
split it into 2 rules:
  1) the existing rule which searches the body+headers, and
  2) a second that only searches the headers.

MSBL Email Blocklist (EBL) SA usage query

[Michael Grant]

Has anyone tried out the the MSBL Email Blocklist (EBL) HashBL.pm with
Spamassassin from msbl.org and possibly considered packaging this module
(available from this page: http://msbl.org/ebl-implementation.html) with
SpamAssassin (perhaps in a forthcoming release)?  rSpamD already has
internal support for the EBL. So I believe the MSBL folks are for this sort
of thing in general.

This plugin looks through the message (not just headers) for email
addresses which have been identified as email drop boxes for scams like 419
advance fee fraud.  It then looks hashes of these addresses up in a
blocklist.

I'm not affiliated with these folks.  I do however use this module in my
setup though and find it catches a bunch of things we wouldn't have
otherwise caught.

Michael Grant

Re: MSBL Email Blocklist (EBL) SA usage query

[Michael Grant]

Hi Keven

I will tell you in the many months we’ve been using the EBL SA Plug in, we
have yet to see a FP, and thus have raised its initial SA scoring of 1 up
to 5 and then again presently to 9.  The EBL has been wildly successful for
us at catching and now blocking this type of drop box spam at the SMTP
level for which we do with SA  that score at 15 or greater.

This would definitely be a good thing to include in the spamassassin
distribution if possible!

Michael

On 15 October 2017 at 13:22, Kevin A. McGrail 
wrote:

> I use a private plugin that does the same thing and have asked a few times
> for them to consider contributing it.
>
> Perhaps this will encourage them to finally do so. I will take a look at
> msbl and ping the other.
>
> Thanks for pointing this out.
> Regards,
> KAM
>
>
> On October 15, 2017 6:01:31 AM EDT, Michael Grant 
> wrote:
>>
>> Has anyone tried out the the MSBL Email Blocklist (EBL) HashBL.pm with
>> Spamassassin from msbl.org and possibly considered packaging this module
>> (available from this page: http://msbl.org/ebl-implementation.html) with
>> SpamAssassin (perhaps in a forthcoming release)?  rSpamD already has
>> internal support for the EBL. So I believe the MSBL folks are for this sort
>> of thing in general.
>>
>> This plugin looks through the message (not just headers) for email
>> addresses which have been identified as email drop boxes for scams like 419
>> advance fee fraud.  It then looks hashes of these addresses up in a
>> blocklist.
>>
>> I'm not affiliated with these folks.  I do however use this module in my
>> setup though and find it catches a bunch of things we wouldn't have
>> otherwise caught.
>>
>> Michael Grant
>>
>

Re: MSBL Email Blocklist (EBL) SA usage query

[Michael Grant]

N.B. that the HASHBL_EMAIL initial installed -- as distributed --
SCORE is set to a lowly 1 in hashbl.cf, viz.:

loadplugin Mail::SpamAssassin::Plugin::HashBL HashBL.pm

ifplugin Mail::SpamAssassin::Plugin::HashBL
header   HASHBL_EMAIL eval:check_hashbl_emails('ebl.msbl.org')
describe HASHBL_EMAIL Message contains email address found on the EBL
scoreHASHBL_EMAIL 1.0
endif

Highly Recommended you up it to at least 5 (ours is presently set at 9).

In the many months we’ve been using the EBL SA Plugin we have yet to
see a single FP and with the 9 SCORE we are able to blocking this type
of drop box spam at the SMTP level with SA with scores at 15 or
greater!

Re: MSBL Email Blocklist (EBL) SA usage query

[Michael Grant]

On 5 December 2017 18:40:15 GMT-05:00, Benny Pedersen  wrote:
>Michael Grant skrev den 2017-12-05 19:01:
>
>> loadplugin Mail::SpamAssassin::Plugin::HashBL HashBL.pm
>
>this line must not be in cf file but should be in pre file
>
># cat hashbl.pre
>loadplugin Mail::SpamAssassin::Plugin::HashBL 
>/path-to-custom-sa-plugins/HashBL.pm
>
># cat hashbl.cf
>> ifplugin Mail::SpamAssassin::Plugin::HashBL
>> header   HASHBL_EMAIL eval:check_hashbl_emails('ebl.msbl.org')
>> describe HASHBL_EMAIL Message contains email address found on the
>
>> EBL
>> scoreHASHBL_EMAIL 1.0
>> endif

interesting, because "as distributed" & installed on Debian, it's in our .cf 
file (and we've had no "problems"/"issues")!

But thanks for the comments.

Sometimes (rarely) spamass-milter does not add the x-spam-* headers

[Michael Grant]

>From time to time (rarely) I notice that spamass-milter does not for
some reason add the x-spam-* headers to a message, but I clearly see
the "Milter add: header: X-Spam-Status:" in the mail log.

For example, this is in the mail.log but nothing in the message received:

Jan 22 13:47:56 strange sm-mta[22301]: w0MIlSrP022301[1]: Milter add:
header: X-Spam-Report: \n\t* -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed
at http://www.dnswl.org/, low\n\t* trust\n\t* [65.79.165.125 listed in
list.dnswl.org]\n\t* 1.0 GENERIC_IXHASH No description available.\n\t*
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover
relay\n\t* domain\n\t* -0.0 SPF_PASS SPF: sender matches SPF
record\n\t* 0.0 HTML_MESSAGE BODY: HTML included in message\n\t* -4.9
BAYES_00 BODY: Bayes spam probability is 0 to 1%\n\t* [score:
0.]\n\t* 0.7 MIME_HTML_ONLY BODY: Message only has text/html MIME
parts\n\t* 1.1 DCC_CHECK Detected as bulk mail by DCC
(dcc-servers.net)\n\t* 0.1 DKIM_SIGNED Message has a DKIM or DK
signature, not necessarily\n\t* valid\n\t* -0.1 DKIM_VALID_AU Message
has a valid DKIM or DK signature from author's\n\t* domain\n\t* -0.1
DKIM_VALID Message has at least one valid DKIM or DK signature\n\t*
0.5 MISSING_MID Missing Message-Id: ...
Jan 22 13:47:56 strange sm-mta[22301]: w0MIlSrP022301[2]: header\n\t*
1.4 MISSING_DATE Missing Date: header
Jan 22 13:47:56 strange sm-mta[22301]: w0MIlSrP022301: Milter add:
header: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28)
on\n\texample.org

Is it possible that the headers were too long? Why would it not
actually follow through and modify the message?

Could not retrieve sendmail macro "auth_type"!.

[Michael Grant]

I'm running spamassassin on several debian systems using sendmail and using
spamass-milter.

I'm seeing this error in my mail logs on one I updated yesterday:

Sep  1 08:21:01 debian spamass-milter[536]: Could not retrieve sendmail
macro "auth_type"!.  Please add it to confMILTER_MACROS_ENVRCPT for better
spamassassin results

I definitely have this macro in my sendmail.mc file:

define(`confMILTER_MACROS_ENVRCPT',`r, v, Z, {auth_type}, {greylist},
{auth_ssf}')dnl

Furthermore on 2 other nearly identical systems I don't have this warning
message.  I only started seeing this warning message when I ran updates
yesterday.  I only get it on inbound mail.

The main packages are all the same version from one system to the other:

dpkg -l | g 'sendmail|spamass|milter'
ii  libmilter1.0.1:amd64 8.15.2-11
 amd64Sendmail Mail Filter API (Milter)
ii  sa-compile   3.4.1-8   all
Tools for compiling SpamAssassin rules into C
ii  sendmail 8.15.2-11 all
powerful, efficient, and scalable Mail Transport Agent (metapackage)
ii  sendmail-base8.15.2-11 all
powerful, efficient, and scalable Mail Transport Agent (arch
independent files)
ii  sendmail-bin 8.15.2-11
 amd64powerful, efficient, and scalable Mail Transport Agent
ii  sendmail-cf  8.15.2-11 all
powerful, efficient, and scalable Mail Transport Agent (config
macros)
ii  spamass-milter   0.4.0-1+b1
amd64milter for filtering mail through spamassassin
ii  spamassassin 3.4.1-8   all
Perl-based spam filter using text analysis
ii  spamc3.4.1-8
 amd64Client for SpamAssassin spam filtering daemon

The sendmail.mc is also the same (with differences being things like
hostnames).

The only difference I know of is one system was updated via apt yesterday,
other a couple months old.

Anyone else seeing this?  What other change might have caused this?

Michael Grant

using URIBL on other headers

[Michael Grant]

The URIBL plugin looks for URLs in the subject and message body.

Is there some way to coax it to look in the other headers as well, for
example the From: Reply-to: or the Received headers?

Re: using URIBL on other headers

[Michael Grant]

On Sat, 22 Sep 2018 at 23:55, Kevin A. McGrail  wrote:

> On 9/22/2018 5:55 PM, Michael Grant wrote:
> > The URIBL plugin looks for URLs in the subject and message body.
> >
> > Is there some way to coax it to look in the other headers as well, for
> > example the From: Reply-to: or the Received headers?
> >
> >
> It's fractured.  There are various lookups in various states in various
> plugins.
>
> From, Reply-to, Received, nameservers, rdns, webmail server headers,
> etc. are all enhancements I want to add for RBL lookups.  Some sort of
> generic Header lookup would be best.  I can't remember if I have a
> bugzilla for this but I have a lot of private notes about it.
>
>
Thanks Kevin, good to hear other folks and yourself wants this too, it sees
to make sense!

I tried to read through the plugin.  I'm not a spamassassin plugin
developer, I didn't have much luck trying to figure out how to do it
myself.  I know this plugin only does subject and body but I saw nothing in
the plugin itself that referenced the subject header.  So I am gathering
it's more complex than simply running the output of an arbitrary header
through this like the subject and body.

Is this difficult because you feel you need to parse out domain names from
all these fields?

I am not sure you need to do that.  Why not just run all the headers or
rather the entire message including headers through this plugin just like
the body, in fact, just extend it's scope to look at the entire message
rather than just the body & subject.

Just a thought.  Hopefully if it's really that easy or if you can tell me
how to extend the scope of this to encompass the entire message, we could
do this sooner than later!

Thanks for your excellent plugin by the way!

Michael Grant

AskDNS with a DNAME

[Michael Grant]

I'm trying to use a rule like this:

askdns   RBL_SENDGRID_ID _SENDGRIDID_.sendgrid-id.localhost A 127.0.0.2

where I have this in my db.local running bind9:

sendgrid-id IN DNAMEsendgrid-id.LICENSEKEY.invaluement.com.

where LICENSEKEY is a valid license key.  I can query this on the command line:

% host 16582324.sendgrid-id.localhost
sendgrid-id.localhost has DNAME record sendgrid-id.LICENSEKEY.invaluement.com.
16582324.sendgrid-id.localhost is an alias for 
16582324.sendgrid-id.LICENSEKEY.invaluement.com.
16582324.sendgrid-id.LICENSEKEY.invaluement.com has address 127.0.0.2

But the AskDNS plugin seems to see only the first alias response and
ignores the actual 127.0.0.2 response.  The rule is never hit.  If I
put the license key in the cf file directly, it is (and the license
key gets added to the email headers which is what I am trying to
avoid by doing a DNAME record in my .localhost db.local!)

This technique of using a DNAME record works fine for milters.

Ultimately I want the spamassassin report in the headers but I don't
want the license key in there.

Is there some way to get this to work?

Not sure if this isn't actually a bug in AskDNS to be honest.

Michael Grant


signature.asc
Description: PGP signature

Re: AskDNS with a DNAME

[Michael Grant]

On Sun, Feb 28, 2021 at 02:14:55PM +, Damian wrote:
> I don't know about AskDNS, but this technique works with stock spamhaus rules 
> via spamhaustech. I have a local spamhaus.net zone with a DNAME record as 
> their nameservers block me anyway.
> You could try with an invaluement.com zone at least temporarily as a 
> comparison to AskDNS.

As I said, it does work if I do this:

askdns   RBL_SENDGRID_ID _SENDGRIDID_.sendgrid-id..invaluement.com 
A 127.0.0.2

But then, the LICENSEKEY gets embedded in the spamassassin report which I don't 
want.

I've traced through the AskDNS plugin and it's definitely only looking
at the first response that gets returned in this case.  I also tried a regex 
submatch like:

askdns   RBL_SENDGRID_ID _SENDGRIDID_.sendgrid-id.localhost A /127.0.0.2/

and still not working.  The AskDNS code which loops through the result
only looks at the alias result that's returned.


signature.asc
Description: PGP signature

Re: AskDNS with a DNAME

[Michael Grant]

On Sun, Feb 28, 2021 at 03:53:33PM +0100, Giovanni Bechis wrote:
> On Sun, Feb 28, 2021 at 07:38:22AM -0500, Michael Grant wrote:
> > Ultimately I want the spamassassin report in the headers but I don't
> > want the license key in there.
> > 
> you can set 'tflags net nolog' if you are using trunk.
> Invaluement uri and license key will be printed as *redacted*.
>  Giovanni   
> 

Hi Giovanni, unfortunately, this did not work either.

I just pulled from your repo to make sure I was on master.  I added
nolog, the pertinent lines look like this:

  askdns   RBL_SENDGRID_ID _SENDGRIDID_.sendgrid-id.MYLICENSE.invaluement.com A 
127.0.0.2
  describe RBL_SENDGRID_ID Sendgrid Id blacklist
  tflags   RBL_SENDGRID_ID net nolog

  askdns   RBL_SENDGRID_DOM 
_SENDGRIDDOM_.sendgrid-efd.MYLICENSE.invaluement.com A 127.0.0.2
  describe RBL_SENDGRID_DOM Sendgrid domain blacklist
  tflags   RBL_SENDGRID_DOM net nolog

And this is what I see in the spamassassin report in the header:
*  1.0 RBL_SENDGRID_ID ASKDNS: Sendgrid Id blacklist
*  [16582324.sendgrid-id.MYLICENSE.invaluement.com A:127.0.0.2]

Michael Grant


signature.asc
Description: PGP signature

Re: AskDNS with a DNAME

[Michael Grant]

> >   askdns   RBL_SENDGRID_ID
> > _SENDGRIDID_.sendgrid-id.MYLICENSE.invaluement.com A 127.0.0.2
> > describe RBL_SENDGRID_ID Sendgrid Id blacklist tflags
> > RBL_SENDGRID_ID net nolog
> > 
> >   askdns   RBL_SENDGRID_DOM
> > _SENDGRIDDOM_.sendgrid-efd.MYLICENSE.invaluement.com A 127.0.0.2
> > describe RBL_SENDGRID_DOM Sendgrid domain blacklist tflags
> > RBL_SENDGRID_DOM net nolog
> > 
> > And this is what I see in the spamassassin report in the header:
> > *  1.0 RBL_SENDGRID_ID ASKDNS: Sendgrid Id blacklist
> > *  [16582324.sendgrid-id.MYLICENSE.invaluement.com
> > A:127.0.0.2]
> 
> I think what you need, at least in the short term, is: 
> 
> askdns   __RBL_SENDGRID_ID ...
> 
> meta RBL_SENDGRID_ID __RBL_SENDGRID_ID

Ah hah! Thank you, this works.  And it has an added benefit that the
RBL_SENDGRID_ID rule doesn't add a default 1.0 score to the total, so
this is definitely the right way to do it.

_SENDGRIDID_ is set as a variable in the Esp.pm module.  Is there some
way to log this when the meta rule triggers?

Michael Grant


signature.asc
Description: PGP signature

Using spamassassin modules from a git repo

[Michael Grant]

I'm running debian on my mail server.  I use etckeeper to track
changes in /etc.

Often I run across modules such as spamassassin-esp and maybe I would
consider playing with Jared Hall's CHAOS module.

I'm curious what the recommended best practice is to install such
modules from a git repo.

For spamassassin-esp, I cloned the repo into my /etc/spamassassin/
directory and then added this to my local.cf:

loadplugin Mail::SpamAssassin::Plugin::Esp spamassassin-esp/Esp.pm
include spamassassin-esp/Esp.cf

This allows me to 'git pull' from this repository from time to time to
update it.  But it's not perfect, especially as I have local changes
to Esp.cf.  It's actually worse since I forked it to give back some
changes but I'd say that's perhaps less usual.

Furthermore, as I said, I use etckeeper and when I 'apt upgrade', I get
constant warnings:

modified:   spamassassin/spamassassin-esp (modified content, untracked content)

So clearly it's not ideal to clone a spamassassin module into
/etc/spamassassin!

I'm curious if someone has a clean solution here that allows updating
the module from time to time from git.

I realize this may be more a debian question and I may post it on the
debian-users list if I don't get any decent replies here.

Michael Grant


signature.asc
Description: PGP signature

Re: Using spamassassin modules from a git repo

[Michael Grant]

On Thu, Apr 08, 2021 at 04:11:25PM +0200, Benny Pedersen wrote:
> On 2021-04-08 11:05, Michael Grant wrote:
> 
> > loadplugin Mail::SpamAssassin::Plugin::Esp spamassassin-esp/Esp.pm
> > include spamassassin-esp/Esp.cf
> 
> loadplugin must not be in cf files, it belongs to pre files

This may be a stupid question... but for a spamassassin module, for
example spamassassin-esp, how would one normally "install" this so
that it reads the .pre file?

Putting modules in /usr/local/etc/spamassassin/ as Tom Hendrikx
suggested.

What I have at the moment now is a modified version of
/etc/spamassassin/Esp.pre:

  loadplugin Mail::SpamAssassin::Plugin::Esp 
/usr/local/etc/spamassassin-esp/Esp.pm
  include /usr/local/etc/spamassassin-esp/Esp.cf

Given that there is an Esp.pre in the spamassassin-esp folder, is
there a way I would use that pre file directly?  I mean, is there some
way to add that folder to spamassassin's "path"?


signature.asc
Description: PGP signature

Re: Using spamassassin modules from a git repo

[Michael Grant]

On Thu, Apr 08, 2021 at 07:00:57PM +0200, Benny Pedersen wrote:
> On 2021-04-08 18:54, Michael Grant wrote:
> 
> > This may be a stupid question... but for a spamassassin module, for
> > example spamassassin-esp, how would one normally "install" this so
> > that it reads the .pre file?
> 
> all content should be placed in same dir as local.cf
> 
> and custom plugins should have there own .pre with the
> loadplugin
> 
> i think it works if Esp.cf is in same dir as local.cf
> then there is no need to make include lines

This is what I want to avoid which was the goal of my original post.

1. Many modules are from git repos and need to live in their own
directory to be updated from time to time, 2. the /etc/spamassassin/
directory can get very messy if you just dump things in there.  Hard
to know what's what, it becomes impossible to maintain.

So I don't see any alternative to keeping such modules in separate
directories like this.

Is there really no way to tell spamassassin where to look for such
modules, like some sort of search path?  I'm surprised if not, and if
not, would something like this be a reasonable feature to add in the
future?



signature.asc
Description: PGP signature

Re: Using spamassassin modules from a git repo

[Michael Grant]

> To update SpamAssassin module from time to time from Git I am using 
> Puppet/Ansible that will put the code in the right places.
> On simpler install I am using a Makefile like this one:
> 
> 
> install:
> pod2man Esp.pm > 
> "/usr/share/man/man3p/Mail::SpamAssassin::Plugin::Esp.3p"
> perl -cw Esp.pm && podlint Esp.pm && cp Esp.{cf,pm,pre} 
> /etc/mail/spamassassin/
> 
> 
> Then I can run git pull from the directory and run make install to copy all 
> files to the correct places.

Thanks Giovanni, yes, this is what one would normally do, drop the
files into /etc/spamassassin (linked from /etc/mail/spamassassin on my
system).

This also solves my initial problem of git repo within git repo.  But
it doesn't solve my desire to keep things in one place.  It's true
that with this particular module the file names are the same.  And
it's also true that the man page (if I wanted to be able to read it
with the man command) would need to go in a different place.

I do kind of like Tom Hendrikx idea of putting cloning the folder into
somewhere in /usr/local/etc and putting a modified pre file in
/etc/spamassassin/.  But it's true it's not perfect.

The next step in this I suppose could be to build a deb or rpm file
around these contributed modules.  But I doubt people are going to
want to build and maintain packages for each of the different
unix/linux/other OSs out there.

Maybe just recommending module developers to put in a simple Makefile
with an install and uninstall target?  I don't know if that's the
right answer.  It does feel like this should be a bit more admin
friendly, by that I mean it should be more than lore to know the right
way to install spamassassin modules in a maintainable way with a
system.

Thanks all for the answers here.


signature.asc
Description: PGP signature

Re: Senderscore

[Michael Grant]

On Mon, Apr 19, 2021 at 02:04:55PM +1000, Simon Wilson wrote:
> Spamassassin on my mail server uses a local dedicated caching DNS server,
> and it is only service which uses it (it's specified in local.cf).
> 
> The last 3 days I have logged about 500 failed DNS query errors to
> senderscore.com, e.g.:
> 
> 19-Apr-2021 13:28:01.367 query-errors: info: client @0x7f31c334a9a0
> 127.0.0.1#53689 (214.48.240.54.bl.score.senderscore.com): query failed
> (SERVFAIL) for 214.48.240.54.bl.score.senderscore.com/IN/A at
> ../../../bin/named/query.c:9385
> 
> ...where in the month prior there were about 10 failures logged in total.
> It's failing on what looks like every inbound email.
> 
> From what I can see it's a genuine blocklist lookup by SA...
> (RCVD_IN_RP_RNBL in 20_dnsbl_tests.cf) but the error rate is strange.
> 
> Am I the only one with high volume of lookup errors from that bl? :-) or do
> I need to be looking for an issue locally...

I am also interested in finding some sender reputation list like this.
I also had a similar experience trying to get senderscore to work.  I
picked up the phone and called them a few weeks ago.

I think initially senderscore's goal was as you (and I) thought, to
score senders such that bad senders could be blocked.  But that's not
what they do now.  What they do now is enable marketeers to get into
people's inboxes by telling the marketeers' what *their* score is
relative to the person or group they are targeting.  It's your
sendsender score, not the other way around.

I am still looking for a sender reputation list, if anyone has
recommendations, please share!

Michael Grant


signature.asc
Description: PGP signature

Re: How do you set nomail for the List?

[Michael Grant]

> But, but, but...  SpamAssassin's entire purpose is an anti-spam
> function!  Oh the irony of it!
> 
> > After all, if just anyone, without subscription, can post to a list, then 
> > it's 
> > open to the entire Internet, and then, as we all know, anarchy ensues...

The Debian mailing lists too are open for anyone to post regardless if
they are subscribed.  It's not anarchy but sometimes spam does get
posted and some people go and report it to places like spamcop (I've
been guilty here!).  Debian has a mechanism to flag messages as spam
in the archives so they can be removed.  It's far from anarchy that
imagine but it's definitely not zero labor.

For me the biggest problem with allowing non-subscribers or
subscribers that don't get mail back from the list is that there is no
way for someone to know if you are reading their replies.  I'm never
sure if I should CC the person directly or not on these open lists.

On the Spamassassin list, I know the person has to be subscribed so I
don't have to CC them.  I doubt most mailing lists are smart enough to
CC such non-subscribers on replies.

Multiple people I know join lists and then create a filter rule to put
the list directly in to the Trash folder or some folder that they
automatically delete older messages.  Then, they read the lists in
that folder.  That may be your best option in my opinion.

A hack comes to mind... maybe something could be written using sieve
or procmail to spot which messages you sent to the list and move them
and replies to it back to your inbox automatically.

Michael Grant


signature.asc
Description: PGP signature

using spamassassin to classify spam

[Michael Grant]

I would like to write a rule that checks if a header has a domain name that 
doesn't resolve.

For example this header:

List-Unsubscribe: 

I want to extract the mumble.aidemxwzlwt.bwbibibi.edu and run it
through AskDNS and if I get an NXDOMAIN, I want to score it.

Is it possible to do this within a cf file?

I can easily extract the domain name with a regex.  Is there a way to
save that value in a variable in a cf file such that I can then call
AskDNS?





signature.asc
Description: PGP signature

Re: using spamassassin to classify spam

[Michael Grant]

> On 24.03.22 18:34, Grant Taylor wrote:
> > Remember, there are historic mechanisms for an MX for parent domains to
> > handle child domains even if the child domain in question doesn't have
> > it's own MX record.
> 
> which, besides wildcard DNS?
>
> OP, also remember that mumble.aidemxwzlwt.bwbibibi.edu may have no A/MX
> record while not produce NXDOMAIN
> 

Right, good points!  So for each sub domain there, do an MX and A
record lookup and stop before getting to the tld itself (.edu).  Of
course there got to be some list somewhere what the tld and gtlds are.

Unless there's an existing function in some plugin to do this, I'll
have to write my own.  Little surprising that there isn't, this seems
like an obvious check!

However, my question still has another part regarding doing this in a
cf file like local.cf.  If it were simply 2 lines in a local.cf to do
this, I'd rather do it there than cobble together a plugin which is
another order of magnitude more complicated.

I have seen things like _VARIABLE_ in .cf files and they seem to get
there from the perl side by doing something like this:

  $pms->set_tag('VARIABLE', $value);

I was wondering if there was a way to set such a variable from the
output of something within the cf side.

This does not work:

header LIST_UNSUB_DOM List-Unsubscribe =~ /\@(.+)/ VARIABLE
askdns LIST_UNSUB_DOM _VARIABLE_

Even if askdns did do the correct thing, I coudln't find a way to get
it the domain name to look up.  Is there a syntax to do this in the
context of a cf file such as local.cf?  Sure seems like it'd be
useful!

Otherwise, I'll have to write a plugin but it seems a shame.

Michael Grant

p.s. the subject of my original post really could have been clearer!
I'm classifying spam by putting thigs into a buckets with certain
attributes and this is one of them.  Things with domain names that are
bogus, but it's still just spamassassin.  Sorry about any confusion!



signature.asc
Description: PGP signature

Re: using spamassassin to classify spam

[Michael Grant]

On Fri, Mar 25, 2022 at 02:27:09PM +0200, Henrik K wrote:
> On Fri, Mar 25, 2022 at 06:01:43AM -0400, Michael Grant wrote:
> >
> > Unless there's an existing function in some plugin to do this, I'll
> > have to write my own.  Little surprising that there isn't, this seems
> > like an obvious check!
> 
> There is already very basic HEADER() template support added in trunk/4.0.0,
> this would generally work:
> 
> askdns UNSUB_NXDOMAIN _HEADER(List-Unsubscribe:host)_ MX [NXDOMAIN]
> 
> It just tries to find something resembling a hostname (having valid TLD) in
> the header, preferring to match @(.*) first.  So it doesn't differentiate
> between http, mailto etc.

Fantastic, thank you!

I'm trying to test this with the debian experimental 4.0.0~0.0svn1896439-1
package.

Running an email through this version seems to be working (as in
spamassassin < test.eml).  However when I test just a narrow set of
rules in my own cf file, I get this:

$ spamassassin -t -C test.cf < tests/test1.eml
config: no rules were found!  Do you need to run 'sa-update'? at 
/usr/bin/spamassassin line 417.

this works fine on spamassassin 3.x by the way.

I have tried reducing test.cf to something simple, for example:

full DKIM_SIGNEDeval:check_dkim_signed()
describe DKIM_SIGNEDMessage has a DKIM or DK signature, not 
necessarily valid
scoreDKIM_SIGNED 5.0

To be clear, for this I really don't want to run all the tests.  Only
specific ones which is why I tried using the -C option which works
with 3.x.  Is there a correct way to do this with 4.x?

Michael Grant


signature.asc
Description: PGP signature

Running spamassassin only with specific rules

[Michael Grant]

Is there some way to run spamassassin with only a specific set of rules and 
scores?

I've tried putting the rules in a rules.cf file and running spamassassin like:

spamassassin -t -p rules.cf < test.eml

but it runs all the rules including theones in rules.cf

I've tried changing the config path with -C so it doesn't pick up the
other cf files but this breaks things.

I'm trying to identify specific types of spam.

Michael Grant


signature.asc
Description: PGP signature

bayes in sqlite db

[Michael Grant]

Does anyone have a working example of storing Bayes and user prefs in
SQLite?  I only see mysql and postgres schemas in 
/usr/share/doc/spamassassin/sql/

Michael Grant


signature.asc
Description: PGP signature

Re: subscribe to blacklist for domains

[Michael Grant via users]

> WTF, that has been a terrible idea since the 90s, given most spam is 
> spoofed, the end result of this will be your mail server getting the 
> poor reputation as source of backscatter and going into blacklists :)

If you reject, you should reject on their SMTP connection.  If you
return a DSN later, there's a high chance you are causing back-scatter
spam to the wrong place.

When you reject on the initial connection, if the spammer is abusing
someone else's infrastructure, you may cause errors to go back to the
owner of that infrastructure which will clue them into a problem they
need to clean up.  Not always though.

Some ESPs track DSNs they get back and remove those addresses from
future mailouts.  If the spammer reuses that ESP, your address may not
be used again with that account.  This is really more useful for
fringe spam like things you didn't realize you signed up for or things
that weren't meant for you.

On the other hand, some ESPs let you report the account as spam, but
to do that you'd have had to received the message first to click on
some link in it.  Mailchimp for example lets you click a box to be
removed and tell them you consider it spam and if they get sufficient
complaints, the account is blocked.

In short, I don't think it's bad to reject spam.  Care needs to be
taken blanket blocking mail from ESPs though.



signature.asc
Description: PGP signature

Providing my own body text parts function

[Michael Grant via users]

In a body rule, SA uses the textual body of the message. 

From the docs: "The 'body' in this case is the textual parts of the message 
body; any non-text MIME parts are stripped, and the message decoded from 
Quoted-Printable or Base-64-encoded format if necessary.
The message Subject header is considered part of the body and becomes the first 
paragraph"

Is there a way I could provide my own function (override SA's internal 
function) to produce this textual representation myself?

Michael Grant

URIDNSBL full message checking

[Michael Grant via users]

I’m noticing that check_uridnsbl() seems only to check the message body.  Is 
there some way to make it check the headers as well?

In 25_uribl.cf, I have:

urirhssub   URIBL_BLACK multi.uribl.com.A   2
bodyURIBL_BLACK eval:check_uridnsbl('URIBL_BLACK')
describeURIBL_BLACK Contains an URL listed in the URIBL blacklist
tflags  URIBL_BLACK net
reuse   URIBL_BLACK

First obvious thing I tried was changing ‘body’ to ‘full’ in the above.  It 
continues to check only the body.  In fact, changing it to ‘header’, it 
continues to check the body.  I then read through the man page on URIDNSBL and 
it does clearly state a ‘body’ rule.

Is there some clever way to have a URIDNSBL rule check the header of a message 
as well?  Or is there something else I can use separately that would look up a 
domainname in the header section of an email?

Michael Grant

Re: URIDNSBL full message checking

[Michael Grant via users]

On Mon, Feb 06, 2023 at 04:16:46PM -0500, Bill Cole wrote:
> On 2023-02-06 at 12:50:29 UTC-0500 (Mon, 6 Feb 2023 17:50:29 +)
> Michael Grant via users 
> is rumored to have said:
> 
> > I’m noticing that check_uridnsbl() seems only to check the message body.
> > Is there some way to make it check the headers as well?
> 
> No. Which is fine, because there are usually no URIs in headers, and when
> there are, they are likely to be standard List-* headers, which are unlikely
> to be useful.

It's actually just a domain name.  This uridnsbl keys off domain names
in the body too, I was kinda hoping it would look at the domain names
in the headers like the body, guess not.

> You can obviously use 'full' or the 'all' pseudo-header and look for
> specific domains, but identifying everything in the header that COULD be a
> domain and just testing that against a DNSBL designed for domains found in
> URIs could have very bad failure modes.

How about just say the from or received headers?  Is there something
like check_rbl that would look up a domain name rather than an ip
address that I could look up the domain in that URIBL list?

I played with check_rbl() but this seems only to look up numeric ip
addresses.

Michael Grant


signature.asc
Description: PGP signature

Re: URIDNSBL full message checking

[Michael Grant via users]

> You can test with:
> 
> header SURBL_MULTI_HDR eval:check_hashbl_emails('multi.surbl.org',
> 'raw/max=10/shuffle/host', 'ALLFROM/Reply-To', '^127\.0\.0\.\d+$')
> priority   SURBL_MULTI_HDR   -100
> describe   SURBL_MULTI_HDR   Domain in email headers found in
> surbl multi

Raymond, thank you!  This works.

But I'm having an issue using this with multi.surbl.org and
multi.uribl.org.  The response addr needs to be bit-masked.  The \d+
in 127.0.0.\d+ is in fact a bitmap.

If I want to assign different scores for different entries in their
databases, I'd need to mask the \d+.  Is there any easier way to do
this than this?

header URIBL_BLACK eval:check_hashbl_emails('multi.uribl.com', 
'raw/max=10/shuffle/host', 'ALLFROM/Reply-To', 
'^127\.0\.0\.(2|3|6|7|10|11|14|15|18|19|22|23|26|27|30|31|34|35|38|39|42|43|46|47|50|51|54|55|58|59|62|63|66|67|70|71|74|75|78|79|82|83|86|87|90|91|94|95|98|99|102|103|106|107|110|111|114|115|118|119|122|123|126|127|130|131|134|135|138|139|142|143|146|147|150|151|154|155|158|159|162|163|166|167|170|171|174|175|178|179|182|183|186|187|190|191|194|195|198|199|202|203|206|207|210|211|214|215|218|219|222|223|226|227|230|231|234|235|238|239|242|243|246|247|250|251|254)$')

check_uridnsbl() handles bitmaps with the urirhssub parameter (the "2) below:

urirhssub   URIBL_BLACK multi.uribl.com.A   2

Is there something like the mask arg in urirhssub with check_hashbl?
I did have a look at the source of check_hashbl but I couldn't spot it
right off.  I get the feeling there's got to be a more straight
forward way than above!

Michael Grant


signature.asc
Description: PGP signature

Re: Strange findings debugging bayes results

[Michael Grant via users]

On 20 February 2023 12:28:00 CET, Loren Wilton  wrote:
>
> A cron job that will harvest Spam and Ham mboxes and feed them to sa-learn 
> once a day, then archive the learned messages. Per-user bayes and learning. 
> Mail is hand-moved into the spam and ham learning folders, and for my  
> personal account, I do this rarely, generally only when a message is 
> mis-categorized. Although messages being mis-categorized as spam is often the 
> result of a lot of quite aggressive local rules I have rather than a Bayes 
> mis-classification.

When you "harvest" ham from mboxes, what do you consider ham?

You also, additionally, have a Ham folder for your users then? Interesting. Did 
you manage to train your users to use it easily? Does it grow unbounded or are 
old messages removed from it?  If so, how to know they can be deleted like from 
the Spam folder.

It's an interesting idea, just wondering about the details.  Getting my users 
to train spamassassim has always been impossible for me.

Re: Strange findings debugging bayes results

[Michael Grant via users]

On Mon, Feb 20, 2023 at 01:30:15PM -0800, Loren Wilton wrote:
> This is a home system with only a few users. All users have "Spam" and "Ham"
> folders showing up in their email program of choice, and they just drag
> messages they do or don't like into the appropriate folders. There are 
> "Oldham"
> and "Oldspam" mboxes, and the new spam and ham (respectively) get merged into
> these folders after learning, and removed from the current Spam and Ham
> folders.

I had a similar idea but never implemmented it because I felt it was
too difficult for users to deal with.  I was considering 2 folders:
'Spam Training Set' and 'Ham Training Set' which would always
represent the set of messages that Spamassassin was currently trained
with.  If you changed the contents of these mboxes, a cron job would
delete the old bayes tokens and retrain with the current set.

The difference between these folders and the Spam folder (or Junk or
whatever you call it locally) is that messages older than 30 days get
auto-deleted.  After 30 days, those messages would no longer represent
the training set.

Having 2 spam folders is confusing and not easy to manage.

Neither of these 2 extra folders are folders that users would look for
messages so they really do have to copy messages into them which isn't
just dragging them.  That for me was the main issue I faced.

So I abandoned this line of thinkinking.

You mentioned harvesting ham and spam from mboxes as in from the inbox
directly.  This got me wondering more about this.

Clearly using messages that the user dragged to Spam that
spamassassin did not mark as Spam to train as spam.  Easy.

And use messages that the user left in their mailbox or deleted or
archived as ham.  Could be ok but less sure.

And lastly, messages that were in Spam (since Spamassassin marked them
as spam), that a user moved out of Spam.  Just look through all their
folders (except Spam) for messages that Spamassassin marked as spam
and retrain on those as ham.  Again, maybe a bad assumption, could
work though.

I was really just curious to know if other people had workable ideas
how to get bayes trained with the least amount of friction.


signature.asc
Description: PGP signature

installing spamassassin plugins on debian

[Michael Grant via users]

Is there a recommended way of installing a spamassassin plugin on
debian (or ubuntu) such that the plugin gets updated via say apt?  I'm
guessing no because I don't see many spamassassin plugins when I do an
"apt search".

Up to now, I have been manually putting things in /etc/spamassassin/
but I feel like there has to be a better way to manage these.

What do people do to keep things up to date easily? 


signature.asc
Description: PGP signature

Re: installing spamassassin plugins on debian

[Michael Grant via users]

On Fri, Mar 17, 2023 at 11:26:21AM +0200, Henrik K wrote:
> On Fri, Mar 17, 2023 at 04:52:41AM -0400, Michael Grant via users wrote:
> > Is there a recommended way of installing a spamassassin plugin on
> > debian (or ubuntu) such that the plugin gets updated via say apt?  I'm
> > guessing no because I don't see many spamassassin plugins when I do an
> > "apt search".
> > 
> > Up to now, I have been manually putting things in /etc/spamassassin/
> > but I feel like there has to be a better way to manage these.
> > 
> > What do people do to keep things up to date easily? 
> 
> There is no automated handling of third party plugins.  It's up the
> maintainers to provide or not provide any support.  Which usually just means
> monitoring some github repo etc.

What about CPAN?  Do people use that?  It seems like there's quite a
few modules in CPAN already.  I will admit that if I see a debian
package, I go for that, I rarely if ever install stuff from CPAN but I
could be convinced to use it more if this created some order out of
the chaos.



signature.asc
Description: PGP signature

Re: installing spamassassin plugins on debian

[Michael Grant via users]

On Fri, Mar 17, 2023 at 04:03:03PM +0100, Benny Pedersen wrote:
> Michael Grant via users skrev den 2023-03-17 09:52:
> 
> > What do people do to keep things up to date easily?
> 
> i just use gentoo, or freebsd, not a precompiled problems (hehe)
> 
> but what plugin do you need with spamassassin 4 now ?
> 
> are you willing to apt maintain a custom plugin in debian ?, i see no
> problem if you do this :)

I want to try the ExtractText plugin.

What if I just install this from CPAN?  It installs in
/usr/share/perl5/Mail/SpamAssassin/Plugin/ which looks correct.

It was also recommended to me maybe use cpan2deb and install that, but
then I'm maintaining my own private debian package which I really did
not want to do.  What's wrong with just installing from CPAN in this case?



signature.asc
Description: PGP signature

Re: installing spamassassin plugins on debian

[Michael Grant via users]

> I guess you didn't notice that you are actually installing SpamAssassin
> 4.0.0, since that's what you are looking at from CPAN?  It's part of the
> official SA package starting from 4.0.0, not a standalone plugin.

Thank you!  I did not notice that, now I see its there.  I know why, I
have 2 boxes, one with the older 3.4 and a newer one with 4.0.0.  So
that little problem is now a non-issue!




signature.asc
Description: PGP signature

Re: installing spamassassin plugins on debian

[Michael Grant via users]

> you dont need this

I see, I stand corrected!

> maybe ask how to configure extracttext ?

Sure, I'd be happy to see some examples.  The man page looks pretty
straight forward.

I see it depends on some external tools like tesseract and odt2txt so
I had better install those first.

I have not had good luck with tesseract out of the box, I wonder if
there's some options to tune it to make it work better.  Is there
anything better?

To see how well this is working, I am hoping to be able to see the
output of these tools with -D so I can write some rules.

Similarly, is there a way to see the 'body' text that is fed into the
rules?  I don't see that in the output of -D.  By 'body', I mean the
text with the html cleaned out of it plus the subject line.  I have a
message and I want to write a new body rule, I want to see what
spamassassin is using as the 'body' so I can write the regex.  I don't
see the body text in -D.




signature.asc
Description: PGP signature

check_rbl question

[Michael Grant via users]

I'm using check_rbl with some paid lists for example invaluement.  I
don't want to put my license key into the rule or it ends up in the
spamassassin X-Spam-Report header.  On one server, I've configured
bind9 with DNAME records to hide the key.  But what do others do?  Is
there some easier way to do this?

Michael Grant


signature.asc
Description: PGP signature

Re: check_rbl question

[Michael Grant via users]

On Fri, Jul 07, 2023 at 04:50:18PM +0200, giova...@paclan.it wrote:
> if can(Mail::SpamAssassin::Conf::has_tflags_nolog)
>   tflags URIBL_IVMURI net nolog
> else
>   tflags URIBL_IVMURI net
> endif

and Benny Pedersen's idea of using a rule like:

header __FOO eval:check_rbl('ivmSIP-lastexternal', 'my_key.inv-sip.')
meta INVSIP __FOO
describe IVMSIP listed at dnsbl.invaluement.com/ivmsip,
score IVMSIP 5

Neither of these are ideal.  I really need to see what ip address is
being looked up.  Perhaps yes, I'll need to do a feature request.





signature.asc
Description: PGP signature

spamd with mix of real and virtual users

[Michael Grant via users]

I'm in the process of setting up virtual users on my mail server.  It
looks like I may have a mix of both real and virtual users.

The flow when scanning a message is:

sendmail -> spamass-milter -> spamc -> spamd

spamass-milter looks at the To: header and passes just the user part.
I see a -e option which causes the whole address (user@domainname) to
be passed to spamc.  cool.

spamc then will pass that verbatim to spamd.

and here's where my problem begins...

If the user exists locally, I want spamd to use that, but if not, I
want it to use the virtual-config-dir.

but to use --virtual-config-dir option requires I specify a -u option
(pin spamd to run as a specific user).

but there's a -U option which causes spamd to fall back to a specific
user.  It would seem like I should be able to specify something like
'-U dovecot-virtual', but no, spamd doesn't allow -U and
--virtual-config-dir options.  That seems like an oversight.

I'm wondering if the better solution here is to pull the problem back
a level and have spamass-milter try to look up the local user and fall
back to a fallback user (dovecot-virtual in my case).

Has anyone else tackled this issue?

Michael Grant


signature.asc
Description: PGP signature

Re: Beginner Setting up Spam Assassin

[Michael Grant via users]

Can you ban this user in whatever your equivalent of the access file is so 
instead of putting the messages into a spam folder, you reject messages from 
that address at delivery time (SMTP)?



On 30 December 2023 04:08:17 CET, FalconChristopher 
 wrote:
>ⓘ *No issues found, please report it if otherwise*
>Anyone know how I can check and setup SpamAssassin so that I can 
>eliminate some spam from coming in from a email account ?
>
>
>On 12/28/2023 2:24 AM, Matus UHLAR - fantomas wrote:
>> On 27.12.23 16:53, FalconChristopher wrote:
>>> Hi, I want to setup Spam Assassin so that any email that Spam 
>>> Assassin flags as spam
>>
>> this is spamassassin's job
>>
>>> gets placed into a folder for a specific SMTP or IMAP email account.
>>
>> this is not spamassassin's job.
>> It's job of mail delivery agent - procmail, maildrop, sieve
>>
>>> Then if Spam Assassin flags emails that are not spam I can tell it 
>>> which of those emails to not place into the spam folder for the 
>>> specific email client. Until it gradually learns which emails are 
>>> spam and which are not.
>>
>> dovecot (imap/pop3 server) has plugins that support training of 
>> spam/ham, if you move the mail from/to spam folder.
>>
>> https://doc.dovecot.org/configuration_manual/spam_reporting/
>>
>>> I've done a little research and I have access with my distribution to 
>>> a mail directory as well as the local.cf file for which 
>>> configurations are for Spam Assassin but I don't know how to setup 
>>> what I mentioned above ?
>>
>

Re: Question about forwarding email (not specifically SA, pointers greatly appreciated)

[Michael Grant via users]

Here's what I have done in the past from my server to get around this
situation you are having:

1. In my .procmailrc file

:0c:
!exam...@gmail.com

This sends a copy (the c flag in first line) of the message to the
gmail account and leaves a copy in your inbox.

2. From your exam...@gmail.com acct, go to Settings -> Accounts and
Import.  Under the section 'Check email from other accounts', Add an
email account.  Then add your server's account and use POP to suck
over emails as they arrive.  Have it delete the emails once they are
sucked over.

What this does is it causes messages to be forwarded to gmail, but
some small number of them bounce because of whatever decision gmail
makes.  But those messages are popped in later, so there's no lost
mail.  Gmail de-duplicates the messages so you don't get messages
twice, and it never refuses to pop the messages in.  Popping in
messages is slow, so when the forward works (which seems to be most of
the time), mail comes in quick, unless it bounces, in which case, it's
popped in a few minutes, sometimes 10s of minutes, later.

If you are concerned about the bounce messages going back into your
mailbox (gmail doesn't loop here fortunately), you can write a
procmail rule to siphon those off into another folder or into
/dev/null.  (Left as exercise for the reader...)

3. You *may* need to do one further thing, you may need to go back
into gmail's Account and Import settings and set up 'Send mail as' and
set up to send mail as your email address on your server.  I can't
remember if gmail does this automatically for you in step 2 above or
not.

4. You probably want to then click the radio button "Reply from the
same address to which the message was sent".  Otherwise, when you
reply, it'll come from your gmail address and not your server's email
address. These radio buttons only appear once you have at least one
Send As address set up.

Michael Grant


signature.asc
Description: PGP signature

spamassassin with gmail

[Michael Grant via users]

Do any of you use spamassassin with a gmail account, and if so, how are 
people doing it?  The reason to do this is gmail's spam filtering isn't 
perfect and you don't have the control you have with spamassassin.


We built some plumbing to do this using gmail's API, and also IMAP which 
can work with other services such as yahoo or outlook.  I'm wondering if 
this is of any use to anyone other than myself.


Essentially, it's a daemon that connects to the account and acts as a 
mail client (an MUA).  When messages arrive in a mailbox (could be any 
folder really), sucks out the message, runs it through spamassassin, and 
puts the result either into the Spam folder or Inbox.


I'm just wondering what to do with this plumbing software, if it should 
be open sourced or run as a service.  Running it as a service couldn't 
be free as I don't have access to free servers.  The daemon in it's 
current state is a bit complicated to set up on it's own but it could 
definitely be cleaned up, especially if there was sufficient interest.


I bet this could also be put together using getmail5 instead of this 
special built daemon but that would imply polling instead of push.  
Several ways to do this.


Michael Grant

Re[2]: spamassassin with gmail

[Michael Grant via users]

nd putting them back in is not as easy as it sounds and I can 
honestly say the devil is in the details, but the good news is that part 
now works well.  I am just trying to figure out what to do with it, if 
it's useful beyond family and friends, or if there is a more general 
interest in being able to use spamassassin on other providers such as 
gmail or yahoo.  If there's insufficient interest, that's fine, I'll 
just use it myself.


Michael Grant

Re[2]: spamassassin with gmail

[Michael Grant via users]

Matija

Sorry, you have misunderstood what I posted.  I am not at all advocating 
people use gmail.  Something like 68% of the planet already uses it and 
few people like you and me have the skills to host our own email.  It's 
not crazy for the people who use gmail or yahoo or other providers, they 
use it, they're used to it, and they apparently like it enough not to 
leave.


It's not easy for people to run their gmail acct through spamassassin.  
Maybe some hack with forwarding and adding headers and a check for 
looping might work.  This isn't what I was really talking about.  But it 
doesn't matter.


Michael Grant

Re[2]: spamassassin with gmail

[Michael Grant via users]

https://isbg.gitlab.io/isbg/index.html

support gmail and spamassassin

other then that i tryed to make a gentoo ebuild for it, have to retry now :)


Yes that's kinda similar!  I'll have to try that!  Thanks.