Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Wed, Jun 2, 2010 at 10:20, Tom Hendrikx  wrote:

> Did you even try to set this up in your test environment (you have one,
> don't you?) to explore your own capabilities in finding out the
> behaviour of this edge case? It sounds trivial to do, but you cannot
> expect other list members to test this for you.

I wish.  The mail server was the test environment before the domains
were moved over to it.  The move was originally planned for June 30,
but got expedited to May 25 due to issues with the mail service
provider.  I will be moving this again probably around October.
Hopefully that hardware will be in a couple months before.

This is a startup environment.  It means there isn't a ton of cash up
front to have the kind of environment that makes sense.  But it is
coming in at some pace, now.  I may even get a real desktop computer
in July.


> When you change your attitude a bit more towards this direction, you
> could even add some valuable input to dovecot by suggesting a man page
> enhancement describing the behaviour of deliver when hitting an empty
> string as argument to -m (or at least fill the gap when someone else has
> the same question and tries google for it). Your latest posts to dovecot
> (and postfix) mailing lists seem to be only highly speculative and have
> a lot of lengthy arguing, and not much hands-on.

The attitude here is more about get as much done as possible.  The
mail server is a fraction of what is going on.  I work on it when I
can.  Other things did get dropped the week or so leading up to the
expedited changeover.  Now that mail is off the problem provider, I do
have more breathing time and can actually do some reading.  But even
so, I'd much rather spend less time reading only what I need, than
spend more time reading stuff that includes what I don't need.  The
reason for that is because I have so much other stuff to read on other
projects.  Today was spent diagnosing why time servers were refusing
to start, and why the monitoring programs on the RAID file servers
were not able to access the network.  If those problems weren't
happening, I'd have been working on the lighttpd (new stuff to read)
web servers.


> A small example describing a more pro-active attitude:
> 
>  Hi list! I need to put spam in a specific folder and want to use
> 'deliver -m' for this. However, when I set up deliver like this in
> postfix , I get the following results that I did
> not expect when the argument for '-m' was a null string:  omitted>.
>
>  Hi phil, could you try ?. Also, when you do this
> , you'll most likely get what you want.
>
>  That works great, thanks guys. I'll add you suggestion to the
> dovecot wiki for future reference.

But that's not reflective of reality.  First, I was not committed to
using deliver -m at all.  It was ONE option.  And there was even more
than one way to use it.  Now I did misunderstand Sieve at that point,
because I was addressing things in a different order.  Sieve was on
the "back burner" because of issues related to user Sieve scripts.  I
knew I needed to read more about Sieve to address it.  But because I
didn't connect the concept that Sieve would address this problem, I
didn't expedite considering it.

Remember, the plate is more than just full; it is way overflowing.  It
is prudent to manage time the best you can.  This means avoiding
things that don't seem to be urgent.  That is not a perfect process,
since one might not know about the "perfect solution to everything"
that lurks in the bowels of the documentation.

But I do try to phrase my initial questions in terms of what
documentation to read, or of basic planning directions.

Planning and reading documentation are too often at odds.  It's a hard
juggle.  The best planning is done by knowing everything.  But if you
spend all the time to know everything, the deadline is long past by
the time the decision can be made.  So it's something you have to go
back and forth with.  Read some.  Make some decisions based on that.
>From those decisions now you have narrowed the field of what is to be
read.  That means you now have the advantage of less to read.

And often the details are lacking in the documentation.  Ways to fill
that in do include testing (now, for example, I'll have to get and set
up a whole new testing environment for email ... something I hadn't
expected to need to do for a couple more months) ... or, asking.

For example, someone could argue that I should have chosen Exim
instead of Postfix, and that if all I had done was read everything
about Exim, I'd have know it was "better".  Well, I'm not going to
pass judgment on that.  Since I have never used Exim and knew nothing
about it (besides that it is another MTA), and have used Postfix some
and know some about it, I save time by choosing Postfix.  It might not
have been the best choice.  But making the choice is likely the best
time saver because it omits all that time reading about Exim.

Sometimes some choices are ea

Re: [Dovecot] Dovecot aspects of fighting spam

[Tom Hendrikx]

On 02/06/10 14:59, Phil Howard wrote:
> On Wed, Jun 2, 2010 at 02:39, Rainer Frey  wrote:
>> On Tuesday 01 June 2010 17:01:16 Phil Howard wrote:
> 
>>> I cannot determine how deliver with an empty string give to -m would
>>> behave.
>>
>> This is just what I mean: the behavior of deliver -m has absolutely nothing 
>> to
>> do with fighting spam. It is the same whether you want to use it for spam-
>> tagged mails or any other reason. Therefore you'll not find the description 
>> in
>> thedovecot wiki bysearching for "fighting spam". You'll find it by searching
>> for "deliver" (or LDA).
> 
> I found the man page (like) document for deliver just fine.  I did not
> limit my search to just the "spam" keyword.  I'm guessing that you are
> assuming I never found this document:
> 
> http://wiki.dovecot.org/LDA
> 
> But now that you know I have read it (and read it long before the
> previous post), re-read what I said.  You should conclude that I am
> unable to determine fully what -m does from this.  It does not
> describe the behavior if the value given is a null string.  This is a
> concern because in master.cf for Postfix, there is no means to
> dynamically provide or omit the -m option.  You can give it a value
> from a variable.  But it may be necessary to give an empty string
> value in order to not deliver to a specific folder, and I want to know
> if that will get the mail into the INBOX.
> 

Did you even try to set this up in your test environment (you have one,
don't you?) to explore your own capabilities in finding out the
behaviour of this edge case? It sounds trivial to do, but you cannot
expect other list members to test this for you.

When you change your attitude a bit more towards this direction, you
could even add some valuable input to dovecot by suggesting a man page
enhancement describing the behaviour of deliver when hitting an empty
string as argument to -m (or at least fill the gap when someone else has
the same question and tries google for it). Your latest posts to dovecot
(and postfix) mailing lists seem to be only highly speculative and have
a lot of lengthy arguing, and not much hands-on.

A small example describing a more pro-active attitude:

 Hi list! I need to put spam in a specific folder and want to use
'deliver -m' for this. However, when I set up deliver like this in
postfix , I get the following results that I did
not expect when the argument for '-m' was a null string: .

 Hi phil, could you try ?. Also, when you do this
, you'll most likely get what you want.

 That works great, thanks guys. I'll add you suggestion to the
dovecot wiki for future reference.
---

End of rant mode, I'll switch back to ignore now.


To conclude: I think saw some list posting a while ago that contained an
example of postfix master.cf using a shell-like
${some_arg:-default_string} expansion syntax. I'm sorry to see that I
cannot find it right now. This syntax, or a 3-line wrapper script around
deliver containing it, would solve your issue.

However I still think that you'd be better off checking whether deliver
will already handle your issue nicely (albeit in an undocumented way).


--
Regards,
Tom

Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Wed, Jun 2, 2010 at 02:39, Rainer Frey  wrote:
> On Tuesday 01 June 2010 17:01:16 Phil Howard wrote:

>> I cannot determine how deliver with an empty string give to -m would
>> behave.
>
> This is just what I mean: the behavior of deliver -m has absolutely nothing to
> do with fighting spam. It is the same whether you want to use it for spam-
> tagged mails or any other reason. Therefore you'll not find the description in
> thedovecot wiki bysearching for "fighting spam". You'll find it by searching
> for "deliver" (or LDA).

I found the man page (like) document for deliver just fine.  I did not
limit my search to just the "spam" keyword.  I'm guessing that you are
assuming I never found this document:

http://wiki.dovecot.org/LDA

But now that you know I have read it (and read it long before the
previous post), re-read what I said.  You should conclude that I am
unable to determine fully what -m does from this.  It does not
describe the behavior if the value given is a null string.  This is a
concern because in master.cf for Postfix, there is no means to
dynamically provide or omit the -m option.  You can give it a value
from a variable.  But it may be necessary to give an empty string
value in order to not deliver to a specific folder, and I want to know
if that will get the mail into the INBOX.


> Yes, sieve is mostly (with dovecot: only) used at delivery time (Cyrus IMAP
> might have support for applying sieve scripts with IMAP, but if so, it is an
> exception).

If sieve had been defined in terms of the mail client uploading
scripts via IMAP to a designated special folder, it might have been
easier to have users supply their own scripts.  But at this point I
don't want to have users doing things that will send mail back out
(e.g. vacation scripts) unless and until I can ensure that such
outbound mail only goes to known recipients (e.g. to avoid backscatter
for FN spam with forged sender addresses).


> This is not my experience. Esp. dovecot and postfix are easily managed from
> source. Installing the package 'build-essential' and then 'aptitude build-dep
> dovecot' should provide all you need to compile and install dovecot from
> source (which is documented in the wiki).

My experience with Debian and Ubuntu has several cases of packages
being degraded or corrupted after building them from source code.
Colleagues have reported the same with Fedora.  No such issue has ever
happened with Slackware, either to me, or that I have heard of.  But,
for the time being anyway, I am on Ubuntu without any source builds.
For various reasons, I do need to eventually move mail services off
the current machine and onto a pair of new machines.  At that time,
Slackware will be considered, as will OpenBSD.


> This is why I recommended a combination of different measures. Content
> detection should come in last in that list, but still can be done at SMTP time
> (esp. with latest postfix and amavisd-new).

I'm all for the combination.  But I need to do less rejection and more
tagging for the before-queue tests (after-queue tests would limited to
tagging, only).  That's yet to be explored in Postfix.

Re: [Dovecot] Dovecot aspects of fighting spam

[Rainer Frey]

On Tuesday 01 June 2010 17:01:16 Phil Howard wrote:
> On Tue, Jun 1, 2010 at 10:45, Rainer Frey  wrote:
> > I guess you'll have the mental transfer of "fight spam" into "cause
> > dovecot to perform this and that action" yourself. The possible actions
> > are, in general, documented.
> 
> I cannot determine how deliver with an empty string give to -m would
> behave.  

This is just what I mean: the behavior of deliver -m has absolutely nothing to 
do with fighting spam. It is the same whether you want to use it for spam-
tagged mails or any other reason. Therefore you'll not find the description in 
thedovecot wiki bysearching for "fighting spam". You'll find it by searching 
for "deliver" (or LDA).

> OK, so a delivery-time sieve.

Yes, sieve is mostly (with dovecot: only) used at delivery time (Cyrus IMAP  
might have support for applying sieve scripts with IMAP, but if so, it is an 
exception).
> 
> > See http://wiki.dovecot.org/LDA/Sieve
> > If you use dovecot 1.2, using the new Dovecot Sieve plugin is highly
> > recommended above the  old CMUsieve.
> > See http://wiki.dovecot.org/LDA/Sieve/Dovecot
> 
> Ubuntu has me at Dovecot version 1.1.11.  

Then you need to read http://wiki.dovecot.org/LDA/Sieve/CMU
It is quite sufficient for your current needs - though I'd upgrade in the long 
run. CMUSieve (and ManageSieve) are already integrated in the package.

> Ubuntu has turned out to be
> difficult for managing upgrades when packages it has are instead
> compiled from source.  

This is not my experience. Esp. dovecot and postfix are easily managed from 
source. Installing the package 'build-essential' and then 'aptitude build-dep 
dovecot' should provide all you need to compile and install dovecot from 
source (which is documented in the wiki).
 
Also there is an unsupported archive with current dovecot packages, but AFAIK 
those are build from source releases instead of daily snapshot (and should 
thus be quite stable:

> I've had periodic high FPs with content detection in the past (on
> other mail servers I didn't control).  

This is why I recommended a combination of different measures. Content 
detection should come in last in that list, but still can be done at SMTP time 
(esp. with latest postfix and amavisd-new).

Rainer

Re: [Dovecot] Dovecot aspects of fighting spam

[Frank Cusack]

On 6/1/10 3:56 PM -0400 Phil Howard wrote:

Since sieve looks like it will be a problem right now, until I get a
solution to that, I'm seriously considering this solution.  A shim
program I write in C will be run from Postfix master.cf just as

...

you are just making it harder than it has to be.  a global sieve_before
script is all that's needed.

-frank

Re: [Dovecot] Dovecot aspects of fighting spam

[William Blunn]

On 01/06/2010 20:56, Phil Howard wrote:
Since sieve looks like it will be a problem right now, until I get a 
solution to that, I'm seriously considering this solution. A shim 
program I write in C will be run from Postfix master.cf just as 
Dovecot deliver is now. I'd basically change the executable path to 
the shim program. The shim program will read the message (I assume 
from stdin) up to 1MB or the end of headers. If the body isn't reached 
by 1MB it goes into the spam folder. If the X-Spam: header ispresent 
with a sufficient probability of spam, it goes into the spam folder. 
Else it goes into the INBOX. Set up a command argument list to run 
deliver, and include -m with the folder name if this goes to the spam 
folder. Set up pipes, fork, and child will exec deliver with that 
argument list. Pipe the buffer that was read in to deliver until it is 
empty, then pipe any remaining stdin to deliver all as one stream. 
Wait for deliver to exit and capture its exit status, and exit with 
the same status. Postfix should then know if delivery succeeded or failed.


Procmail will do all the things you say above with a few lines of simple 
configuration, but with the benefit of being already done, tried and tested.


Procmail is a little self-contained program which you can just plain 
run, have it do some matches on the message content, and then use that 
to invoke the LDA one way or another.


People may say that Procmail is a bit old, and it is. But it works.

I pass all of my incoming mail through Procmail.

You can make rules with conditions, such as matching header records with 
regular expressions.


If a condition matches (e.g. we found a spam header), then you can tell 
Procmail to pipe the message to a program (e.g. "deliver") with certain 
arguments.


:0
* ^X-Spam-Flag: yes
| deliver -m spam

If none of the rules match, we can get procmail to do something 
different, e.g. pipe the message to a different program, e.g. "deliver" 
but with different arguments.


:0
| deliver

By default, Procmail will try to deliver the message exactly once. If it 
fails, it returns an error code so that the MTA can know that delivery 
failed, and can take the appropriate action.


If you want, Procmail will even pipe your message through another 
program first, e.g. SpamAssassin, so that the other program can change 
the message as required (e.g. adding header records saying whether or 
not it thinks it is spam).


:0 fw
| spamassassin

If you want to pass data from the MTA to Procmail for use in rules, 
(e.g. the envelope recipient), Procmail provides a couple of ways to do 
this.


Documentation can be found in Procmail's four man pages:

Main procmail documentation - man procmail

Procmail configuration file - man procmailrc

Procmail configuration file examples - man procmailex

Procmail weighted scoring technique - man procmailsc

Bill

Re: [Dovecot] Dovecot aspects of fighting spam

[Charles Marcus]

On 2010-06-01 3:56 PM, Phil Howard wrote:
> Since sieve looks like it will be a problem right now, until I get a
> solution to that, I'm seriously considering this solution.  A shim
> program I write in C will be run from Postfix master.cf just as
> Dovecot deliver is now.

I'm trying to figure out why you don't just use dovecots LDA+sieve.

You do realize you could do it with a global script and just not allow
users to have their own scripts (at least I'm 99% sure you can)?

-- 

Best regards,

Charles

Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Tue, Jun 1, 2010 at 14:56, Frank Cusack
 wrote:

> Man oh man.  You don't have much experience with mail and it sounds
> like you are starting from scratch.  You have your work cut out for
> you. :)

Well, actually I do.  A lot of it is way way back there with sendmail.
 Back then I coded sendmail.cf by hand and avoided m4.  I have used
Postfix previously, too, but in a very different circumstance, and
even then it was several years back.

> The standard way to filter spam is [all of]:
>
> a) stop it at the border, then

For my current situation, this will be a smaller amount than I've done
in the past.

> b) content analysis and tagging, then

I haven't done this in the past for my own mail servers primarily
because of principle (e.g. to me UCE isn't about what it said, but
whether it is unsolicited ... which indirectly means what is said ...
and bulk ... in summary, it's about behaviour, not about the message).
 But in this new situation, I have to rebalance these ideas to achieve
different goals.  Being principled doesn't count in this situation.  A
potential client isn't going to be turned down or lectured just
because the ISP they use is spammer friendly.  I don't really like
that, but that's the way it is.

> c) filtering at delivery time
>
> per-user filters for part (b) on the server side is a dead end IMHO but
> there are some programs that do this and dovecot can work with them
> (by having folders that train the filter).  modern mail clients, however,
> have their own spam filters now, which are by definition personalized.
>
> anyway, part (a) and (b) have nothing to do with dovecot.  the standard
> way to do part (c) is to have an X-Spam: header which a sieve script
> filters on.  the -m flag to deliver is not really for spam filtering.

Since sieve looks like it will be a problem right now, until I get a
solution to that, I'm seriously considering this solution.  A shim
program I write in C will be run from Postfix master.cf just as
Dovecot deliver is now.  I'd basically change the executable path to
the shim program.  The shim program will read the message (I assume
from stdin) up to 1MB or the end of headers.  If the body isn't
reached by 1MB it goes into the spam folder.  If the X-Spam: header is
present with a sufficient probability of spam, it goes into the spam
folder.  Else it goes into the INBOX.  Set up a command argument list
to run deliver, and include -m with the folder name if this goes to
the spam folder.  Set up pipes, fork, and child will exec deliver with
that argument list.  Pipe the buffer that was read in to deliver until
it is empty, then pipe any remaining stdin to deliver all as one
stream.  Wait for deliver to exit and capture its exit status, and
exit with the same status.  Postfix should then know if delivery
succeeded or failed.

I may set up a web mail service later, and will see if I can do Sieve
within that.

Re: [Dovecot] Dovecot aspects of fighting spam

[Steve]

 Original-Nachricht 
> Datum: Tue, 1 Jun 2010 09:42:18 -0400
> Von: Phil Howard 
> An: Dovecot Mailing List 
> Betreff: [Dovecot] Dovecot aspects of fighting spam

> I see no documents in the Dovecot documents wiki on how Dovecot
> features can be used in the fight against spam.  One hit came up in a
> search, and it was about how to fight spam in the MoinMoin Wiki itself
> (e.g. TextCHAs and such).
> 
> I already asked in another thread about removing message files at the
> search from a designated "learn-spam" folder.  My next question would
> be how to automatically place detected spam into a specific folder for
> the user to access or ignore as they deem fit.  That would seem to me
> to be something for the deliver command and its -m option.  But this
> might be more of a Dovecot and MTA coordination issue, too (e.g. how
> to get the spam detection done in the MTA ... Postfix in my case ...
> to become a label of some sort that ends deciding the folder.  The -m
> option might be harder to get it dynamically set since it would be
> hard coded in master.cf for Postfix (so a variable has to be able to
> direct it to INBOX in this case).
> 
> But maybe the "standard" way of doing this is to shim another program
> between the MTA (Postfix for me) and the LDA (Dovecot deliver) to
> dynamically add the -m option on the fly?  Lacking any other clever
> solution, this seems to me to be the way I could make it work.  Maybe
> there is a special header deliver could detect, such as "X-Spam: yes"?
> 
> A wiki document on how Dovecot helps (or how to help Dovecot) in the
> fight against spam, I think, would be helpful.
>
Maybe you are looking for something like this:
https://sourceforge.net/apps/mediawiki/dspam/index.php?title=Filter_results_with_Dovecot_Sieve
-- 
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

Re: [Dovecot] Dovecot aspects of fighting spam

[Frank Cusack]

I hadn't gotten to reading much about Sieve, yet, as what I had read
said it was something IMAP users would run.


Man oh man.  You don't have much experience with mail and it sounds
like you are starting from scratch.  You have your work cut out for
you. :)

The standard way to filter spam is [all of]:

a) stop it at the border, then
b) content analysis and tagging, then
c) filtering at delivery time

per-user filters for part (b) on the server side is a dead end IMHO but
there are some programs that do this and dovecot can work with them
(by having folders that train the filter).  modern mail clients, however,
have their own spam filters now, which are by definition personalized.

anyway, part (a) and (b) have nothing to do with dovecot.  the standard
way to do part (c) is to have an X-Spam: header which a sieve script
filters on.  the -m flag to deliver is not really for spam filtering.

-frank

Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Tue, Jun 1, 2010 at 12:41, Stan Hoeppner  wrote:

> Law firm or heavily FED GOV regulated industry?  Or just HUA (head up ass)
> management with zero understanding of email and spam?

People dealing with a large variety of outside clients and potential
clients where bouncing THEIR mail gives bad impressions in too many
cases (already has with the outside mail provider we were using up
until the move, which was expedited for just such issues).

FYI, you forgot the marketing industry (not counting the bulk email
marketing industry ... although we are not that).  Also, lots of media
and other industries can have this issue, as well as GOV itself.

Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Tue, Jun 1, 2010 at 12:55, William Blunn  wrote:

> "The Dovecot Sieve plugin provides mail filtering facilities at time of
> final message delivery using the Sieve (RFC 5228
> ) language."
>
> "Sieve is implemented as a plugin to deliver ."

I hadn't gotten to reading much about Sieve, yet, as what I had read
said it was something IMAP users would run.  So I didn't realize
(then) that I needed to read about it.  It was in the list for stuff
to read about, but just for later.  It is now bumped up and I will be
reading about it this week (the email server is a fraction of what is
going on here).

Re: [Dovecot] Dovecot aspects of fighting spam

[William Blunn]

On 01/06/2010 17:05, Phil Howard wrote:

On Tue, Jun 1, 2010 at 11:01, William Blunn  wrote:

   

Get your anti-spam software to add header records to the messages, then use
Dovecot LDA Sieve to filter based on those header records.
 

So I can run Sieve as part of the deliver?  Or does it run soon after that?
   


TBH I have never used Dovecot's LDA or either of its two available Sieve 
plug-ins.


But if we read http://wiki.dovecot.org/LDA/Sieve , there are some 
interesting bits:


"The Dovecot Sieve plugin provides mail filtering facilities at time of 
final message delivery using the Sieve (RFC 5228 
) language."


"Sieve is implemented as a plugin to deliver ."

"The supported Sieve features are:

Extension: fileinto
CMU Sieve: Yes
Dovecot Sieve: Yes
Purpose: Allows storing messages in folders other than INBOX
..."

It would appear that if Dovecot 'deliver' is invoked as part of your 
system's mail delivery processing, then 'deliver' can run Sieve over 
messages, and have it file messages into folders depending on rules.


I went looking for an example Sieve configuration to file messages which 
SpamAssassin had marked as spam

http://www.google.co.uk/search?q=sieve+spamassassin

First hit was: "Spamfiltering using Spamassassin and sieve"
http://tomster.org/blog/archive/2004/12/15/spamfiltering-using-spamassassin-and-sieve

Second hit was in our back yard: "Dovecot Sieve plugin: SpamAssassin 
tagged mail filtering"

http://wiki.dovecot.org/LDA/Sieve#SpamAssassin_tagged_mail_filtering

Bill

Re: [Dovecot] Dovecot aspects of fighting spam

[Stan Hoeppner]

Phil Howard put forth on 6/1/2010 11:25 AM:
> On Tue, Jun 1, 2010 at 12:17, Stan Hoeppner  wrote:
>> Phil Howard put forth on 6/1/2010 9:15 AM:
>>> On Tue, Jun 1, 2010 at 10:04, Jerry  wrote:
>>>
 The fight against SPAM is NOT Dovecots responsibility.
>>>
>>> Whose responsibility is it to get detected spam delivered into the
>>> correct folder?  Dovecot deliver is typically used to deliver into the
>>> Inbox(es).  It would be involved.
>>
>> It's the MTA's responsibility to _REJECT_ spam attempts at SMTP time.  It's
>> the mail OP's responsibility to properly configure the MTA to do so.  Spam
>> fighting should be performed pre queue, not post queue.
>>
>> I suggest you bone up on modern spam fighting techniques.  Post queue content
>> analysis is not the proper way to fight spam, especially in 2010.
> 
> For my own personal server, I would agree.  Even for a general email
> or freemail service I would agree.
> 
> However, for certain business environments, even a few false positives
> can be very troubling to management.  Greylisting, and with shades of
> grey, is often more appropriate.  Given that this puts mail where it
> will likely never be read or responded to, certainly doing as much as
> you can at SMTP time for a 5XX rejects is preferred, so that
> legitimate mail that would otherwise not be responded to is known to
> be not delivered where it would be read.  But this can't be for
> everything, as even a FP rate of 0.01% is too much.
> 
> So in the end, there has to be an administrative policy decision as to
> what to do with what is detected as spam.  And if I can find a way,
> I'd like to even set that up so end USER policy can be applied even at
> SMTP time (e.g. user decides if spam is blacklisted or greylisted,
> along with user specified whitelists).  But that's pushing the
> boundaries for now.  When I get time, I will look into that degree of
> control.

Law firm or heavily FED GOV regulated industry?  Or just HUA (head up ass)
management with zero understanding of email and spam?

-- 
Stan

Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Tue, Jun 1, 2010 at 12:17, Stan Hoeppner  wrote:
> Phil Howard put forth on 6/1/2010 9:15 AM:
>> On Tue, Jun 1, 2010 at 10:04, Jerry  wrote:
>>
>>> The fight against SPAM is NOT Dovecots responsibility.
>>
>> Whose responsibility is it to get detected spam delivered into the
>> correct folder?  Dovecot deliver is typically used to deliver into the
>> Inbox(es).  It would be involved.
>
> It's the MTA's responsibility to _REJECT_ spam attempts at SMTP time.  It's
> the mail OP's responsibility to properly configure the MTA to do so.  Spam
> fighting should be performed pre queue, not post queue.
>
> I suggest you bone up on modern spam fighting techniques.  Post queue content
> analysis is not the proper way to fight spam, especially in 2010.

For my own personal server, I would agree.  Even for a general email
or freemail service I would agree.

However, for certain business environments, even a few false positives
can be very troubling to management.  Greylisting, and with shades of
grey, is often more appropriate.  Given that this puts mail where it
will likely never be read or responded to, certainly doing as much as
you can at SMTP time for a 5XX rejects is preferred, so that
legitimate mail that would otherwise not be responded to is known to
be not delivered where it would be read.  But this can't be for
everything, as even a FP rate of 0.01% is too much.

So in the end, there has to be an administrative policy decision as to
what to do with what is detected as spam.  And if I can find a way,
I'd like to even set that up so end USER policy can be applied even at
SMTP time (e.g. user decides if spam is blacklisted or greylisted,
along with user specified whitelists).  But that's pushing the
boundaries for now.  When I get time, I will look into that degree of
control.

Re: [Dovecot] Dovecot aspects of fighting spam

[Stan Hoeppner]

Phil Howard put forth on 6/1/2010 9:15 AM:
> On Tue, Jun 1, 2010 at 10:04, Jerry  wrote:
> 
>> The fight against SPAM is NOT Dovecots responsibility.
> 
> Whose responsibility is it to get detected spam delivered into the
> correct folder?  Dovecot deliver is typically used to deliver into the
> Inbox(es).  It would be involved.

It's the MTA's responsibility to _REJECT_ spam attempts at SMTP time.  It's
the mail OP's responsibility to properly configure the MTA to do so.  Spam
fighting should be performed pre queue, not post queue.

I suggest you bone up on modern spam fighting techniques.  Post queue content
analysis is not the proper way to fight spam, especially in 2010.

-- 
Stan

Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Tue, Jun 1, 2010 at 11:01, William Blunn  wrote:

> Get your anti-spam software to add header records to the messages, then use
> Dovecot LDA Sieve to filter based on those header records.

So I can run Sieve as part of the deliver?  Or does it run soon after that?

Re: [Dovecot] Dovecot aspects of fighting spam

[William Blunn]

On 01/06/2010 14:42, Phil Howard wrote:

I see no documents in the Dovecot documents wiki on how Dovecot
features can be used in the fight against spam.  One hit came up in a
search, and it was about how to fight spam in the MoinMoin Wiki itself
(e.g. TextCHAs and such).

I already asked in another thread about removing message files at the
search from a designated "learn-spam" folder.  My next question would
be how to automatically place detected spam into a specific folder for
the user to access or ignore as they deem fit.  That would seem to me
to be something for the deliver command and its -m option.  But this
might be more of a Dovecot and MTA coordination issue, too (e.g. how
to get the spam detection done in the MTA ... Postfix in my case ...
to become a label of some sort that ends deciding the folder.  The -m
option might be harder to get it dynamically set since it would be
hard coded in master.cf for Postfix (so a variable has to be able to
direct it to INBOX in this case).

But maybe the "standard" way of doing this is to shim another program
between the MTA (Postfix for me) and the LDA (Dovecot deliver) to
dynamically add the -m option on the fly?  Lacking any other clever
solution, this seems to me to be the way I could make it work.  Maybe
there is a special header deliver could detect, such as "X-Spam: yes"?

A wiki document on how Dovecot helps (or how to help Dovecot) in the
fight against spam, I think, would be helpful.


Get your anti-spam software to add header records to the messages, then 
use Dovecot LDA Sieve to filter based on those header records.


Bill

Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Tue, Jun 1, 2010 at 10:45, Rainer Frey  wrote:

> I guess you'll have the mental transfer of "fight spam" into "cause dovecot to
> perform this and that action" yourself. The possible actions are, in general,
> documented.

I cannot determine how deliver with an empty string give to -m would
behave.  OTOH, I don't know that an empty string would what would even
happen, since I have yet to explore how the spam status from one of
the spam detections would trickle through to setting up the transport
variables in Postfix.  I don't even know if -m is considered the
appropriate means to deliver to a specific box in the case of junk
spam.  If it is, obviously someone must be using it.


> You need to configure the MTA to accept and tag spam messages, e.g. with
> Amavis D_PASS policy and appropriate tag levels. Then the most flexible way to
> put the mails into a spam folder is to use the deliver plugin for sieve, which
> can sort and filter mails depending on many different conditions, like
> presence and/or value of any header. You can deploy a single system wide
> default script for all users, which can be complemented or overwritten by
> individual users (depending on configuration).

OK, so a delivery-time sieve.


> See http://wiki.dovecot.org/LDA/Sieve
> If you use dovecot 1.2, using the new Dovecot Sieve plugin is highly
> recommended above the  old CMUsieve.
> See http://wiki.dovecot.org/LDA/Sieve/Dovecot

Ubuntu has me at Dovecot version 1.1.11.  Ubuntu has turned out to be
difficult for managing upgrades when packages it has are instead
compiled from source.  So being "up to date at the source level" is
going to have to wait until I get some Slackware running here
(originally Ubuntu was chosen because of the wider variety of
ready-to-install packages).


> The really recommended way nowadays is different though: *reject* as much spam
> as possible in the MTA in the receiving SMTP dialog, with a combination of
> blacklists, sender validation (e.g.DKIM), potentially greylisting (as you deem
> appropriate) and lastly content scanning with s.th. like amavis+spamassassin.
> I'd not expect a noticable amount of false positives, and in the rare cases
> the sender will be notified. The false positives rate of users scanning
> through a folder of tagged mail will be higher.

I've had periodic high FPs with content detection in the past (on
other mail servers I didn't control).  So greylisting for that is what
I will likely be doing, at least at first.  How much I shift to
blacklisting will depend on the feedback from my own users ("hey, my
Junk-spam folder has too much spam" or "How can I find that one false
positive if there are a million spams in there").

Tagging ... yeah, clearly I would do that for anything not rejected at
SMTP time.  How to get Dovecot deliver to put those ... well, OK, I'll
look into Sieve instead of writing my own shim.


> A howto-like page to describe the usage of some dovecot features in an anti-
> spam context wouldn't hurt, OTOH there is not much anti-spam specific stuff at
> all at the dovecot side.

Unfortunately, a good document would likely have to be specific to the
LDA, MTA, and spam tools.  Given so many combinations, that's a lot
documents, or a rather complex one.  I can certainly use my notes from
my own solutions and make something specific to Dovecot and Postfix
and whatever spam tools I use (could be more than one).

Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Tue, Jun 1, 2010 at 10:21, Frank Elsner  wrote:

> Ok, you obviously have a different environment and requirements.
>
> - I do the delivery into special folder by the MTA (exim) and the LDA is
>  not used at all.
>
> - detection training is not done.
>
>
> Sorry, my world may be to narrow :-)

Each of our worlds is narrow.  Some may be wider than others.
Programs like Dovecot support a LOT of things, and I can't see how
anyone can be using more than a (perhaps significant in some cases)
fraction of it.  Obviously your choice of Exim for an MTA doesn't
relate to my choice of Postfix for MTA.  Dovecot supports both and
more.  So the programs can be a lot wider real world uses.  But put
all the variety of uses together as a set, and it can be seen that
diverse tools are useful.

Since you aren't using Dovecot deliver, then you don't need to be
knowledgeable about its -m option.  I'm curious if that is the only
way.  I'm still thinking that a shim (program) between Postfix and
Dovecot deliver that detects headers, markers, or tags placed on the
message by spam detection, and adds a "-m Spam" option or whatever,
might be what I need to do.  But I guess you won't need it.

Re: [Dovecot] Dovecot aspects of fighting spam

[Rainer Frey]

On Tuesday 01 June 2010 15:42:18 Phil Howard wrote:
> I see no documents in the Dovecot documents wiki on how Dovecot
> features can be used in the fight against spam.  One hit came up in a
> search, and it was about how to fight spam in the MoinMoin Wiki itself
> (e.g. TextCHAs and such).

I guess you'll have the mental transfer of "fight spam" into "cause dovecot to 
perform this and that action" yourself. The possible actions are, in general, 
documented.

> I already asked in another thread about removing message files at the
> search from a designated "learn-spam" folder.  My next question would
> be how to automatically place detected spam into a specific folder for
> the user to access or ignore as they deem fit.  

You need to configure the MTA to accept and tag spam messages, e.g. with 
Amavis D_PASS policy and appropriate tag levels. Then the most flexible way to 
put the mails into a spam folder is to use the deliver plugin for sieve, which 
can sort and filter mails depending on many different conditions, like 
presence and/or value of any header. You can deploy a single system wide 
default script for all users, which can be complemented or overwritten by 
individual users (depending on configuration).

See http://wiki.dovecot.org/LDA/Sieve
If you use dovecot 1.2, using the new Dovecot Sieve plugin is highly 
recommended above the  old CMUsieve.
See http://wiki.dovecot.org/LDA/Sieve/Dovecot

The really recommended way nowadays is different though: *reject* as much spam 
as possible in the MTA in the receiving SMTP dialog, with a combination of 
blacklists, sender validation (e.g.DKIM), potentially greylisting (as you deem 
appropriate) and lastly content scanning with s.th. like amavis+spamassassin. 
I'd not expect a noticable amount of false positives, and in the rare cases 
the sender will be notified. The false positives rate of users scanning 
through a folder of tagged mail will be higher.

> A wiki document on how Dovecot helps (or how to help Dovecot) in the
> fight against spam, I think, would be helpful.

A howto-like page to describe the usage of some dovecot features in an anti-
spam context wouldn't hurt, OTOH there is not much anti-spam specific stuff at 
all at the dovecot side.

Rainer

Re: [Dovecot] Dovecot aspects of fighting spam

[Charles Marcus]

On 2010-06-01 10:34 AM, Charles Marcus wrote:
> There is also a plugin for dovecot that makes it easy for users to
> reclassify stuff as spam or not spam by simply moving the messages to
> the appropriate folder, but I think it is mainly designed for DSPAM...
> anyone using it with spamassassin?

Sorry, meant to include a link:

http://johannes.sipsolutions.net/Projects/dovecot-antispam

-- 

Best regards,

Charles

Re: [Dovecot] Dovecot aspects of fighting spam

[Charles Marcus]

On 2010-06-01 9:42 AM, Phil Howard wrote:
> I see no documents in the Dovecot documents wiki on how Dovecot
> features can be used in the fight against spam.  One hit came up in a
> search, and it was about how to fight spam in the MoinMoin Wiki itself
> (e.g. TextCHAs and such).

1. Set up postfix (or whatever  MTA you are using) to reject as much as
possible at SMTP time.

If done right postfix can reject 90+% (depending on how much /what type
of spam your domains get) with pretty much zero FPs...

Lots of examples down in the UCE/Virus section here:

http://www.postfix.org/docs.html

2. Add amavisd-new+spamassassin to the mix, and tag+deliver anything it
detects as spam into the junk folder using sieve...

http://wiki.dovecot.org/LDA/Sieve#SpamAssassin_tagged_mail_filtering

There is also a plugin for dovecot that makes it easy for users to
reclassify stuff as spam or not spam by simply moving the messages to
the appropriate folder, but I think it is mainly designed for DSPAM...
anyone using it with spamassassin?

-- 

Best regards,

Charles

Re: [Dovecot] Dovecot aspects of fighting spam

[Frank Elsner]

On Tue, 1 Jun 2010 10:02:56 -0400 Phil Howard wrote:
> On Tue, Jun 1, 2010 at 09:54, Frank Elsner  
> wrote:
> > On Tue, 1 Jun 2010 09:42:18 -0400 Phil Howard wrote:
> >> I see no documents in the Dovecot documents wiki on how Dovecot
> >> features can be used in the fight against spam.  One hit came up in a
> >> search, and it was about how to fight spam in the MoinMoin Wiki itself
> >> (e.g. TextCHAs and such).
> >
> > Fighting against SPAM is the MTA's job.
> > Dovecot's job is to provide access to the mailboxes for the users.
> 
> Dovecot would still be involved ... getting detected spam into a
> folder for users that don't want false positives to be deleted, and a
> feedback folder for recycling false negatives into detection training.

Ok, you obviously have a different environment and requirements.

- I do the delivery into special folder by the MTA (exim) and the LDA is 
  not used at all.

- detection training is not done.


Sorry, my world may be to narrow :-)



--Frank Elsner

Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Tue, Jun 1, 2010 at 10:04, Jerry  wrote:

> The fight against SPAM is NOT Dovecots responsibility.

Whose responsibility is it to get detected spam delivered into the
correct folder?  Dovecot deliver is typically used to deliver into the
Inbox(es).  It would be involved.

Note that I am not saying "responsibility".  There are features that
can be used (or there aren't, as the case may be).  Documenting them
would be helpful.  Dovecot has documentation for many other things.
If it had documentation on ... dealing with (I'll use that term
instead of fighting, as maybe that will be clearer to you) ... spam
aspects (like folder selection aspects), that would be helpful.


> The fight against SPAM is NOT Dovecots responsibility. There are
> numerous applications that work with your MTA for that purpose. I
> suggest that you investigate this further. You might want to start here:
>
>        http://www.ijs.si/software/amavisd/
>
> They also have their own mailing list that you can use to answer any
> questions that you might have.

Like "How do I get Dovecot deliver to put the spam in the spam folder?" ?

Re: [Dovecot] Dovecot aspects of fighting spam

[Jerry]

On Tue, 1 Jun 2010 09:42:18 -0400
Phil Howard  articulated:


> I see no documents in the Dovecot documents wiki on how Dovecot
> features can be used in the fight against spam.  One hit came up in a
> search, and it was about how to fight spam in the MoinMoin Wiki itself
> (e.g. TextCHAs and such)

The fight against SPAM is NOT Dovecots responsibility. There are
numerous applications that work with your MTA for that purpose. I
suggest that you investigate this further. You might want to start here:

http://www.ijs.si/software/amavisd/

They also have their own mailing list that you can use to answer any
questions that you might have.

-- 
Jerry
dovecot.u...@seibercom.net

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.
__

Re: [Dovecot] Dovecot aspects of fighting spam

[Phil Howard]

On Tue, Jun 1, 2010 at 09:54, Frank Elsner  wrote:
> On Tue, 1 Jun 2010 09:42:18 -0400 Phil Howard wrote:
>> I see no documents in the Dovecot documents wiki on how Dovecot
>> features can be used in the fight against spam.  One hit came up in a
>> search, and it was about how to fight spam in the MoinMoin Wiki itself
>> (e.g. TextCHAs and such).
>
> Fighting against SPAM is the MTA's job.
> Dovecot's job is to provide access to the mailboxes for the users.

Dovecot would still be involved ... getting detected spam into a
folder for users that don't want false positives to be deleted, and a
feedback folder for recycling false negatives into detection training.

Re: [Dovecot] Dovecot aspects of fighting spam

[Frank Elsner]

On Tue, 1 Jun 2010 09:42:18 -0400 Phil Howard wrote:
> I see no documents in the Dovecot documents wiki on how Dovecot
> features can be used in the fight against spam.  One hit came up in a
> search, and it was about how to fight spam in the MoinMoin Wiki itself
> (e.g. TextCHAs and such).

Fighting against SPAM is the MTA's job. 
Dovecot's job is to provide access to the mailboxes for the users.


--Frank Elsner