RE: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-13 Thread David Schwartz


> I haven't seen any succinct justification for providing a
> 550 message rejection for positively-identified spam versus
> silently dropping the message. Lots of how-to instructions
> but no whys.
>
> matthew black
> california state university, long beach

Because your father may forward a copy of a Nigeria scam from a new 
email
address he just got with his new ISP and ask if you if he should send them
money.

Because a machine you own may be responsible for the spam, and someone 
may
be forwarding you a copy of it along with the tracking information to
demonstrate that you were responsible for it.

Because the spam may include a trademark you own and you may need to 
notify
your legal department about it. The spam may have been helpfully forwarded
to you by someone familiar with your trademarks.

Because if you say you are going to deliver a message, that's what you
should do.

Because being spam is not the same as being unimportant.

All of these things really do happen.

>Agreed, but we're willing to live with an error rate of less
>than one in a million. This isn't a space shuttle. I don't think
>the USPS can claim 99.% delivery accuracy. Nonetheless, to
>allay worries, we are considering spam quarantines to allow
>recipients an opportunity to review spam messages themselves, much
>like Yahoo! Mail.

It is one thing to have an error rate of one in a million, it is quite
another thing to choose to have an error rate of one in a million instead of
one in a billion for no rational reason at all. If the measure is what
fraction of positively-identified spam the recipient would probably rather
receive than not receive, it's probably more like one in 5,000. If the
measure is what fraction of positively-identified spam the recipient would
rather the sender get a reject than silently discard, it's probably more
like one on 20,000.

The argument on the other side is if the positively-identified spam 
comes
from a business-critical mailing list that unsubscribes people if they have
too many bounces. This probably isn't an issue for viruses and malware
because these rarely get past the filters these lists already have. It is a
big issue for spam and one of the few for which there is no good solution I
know of. (Other than having the recipient whitelist the list at the MTA,
which is hard to do.)

DS




Re: Spam filtering bcps

2006-04-13 Thread Andy Smith
On Wed, Apr 12, 2006 at 03:35:51PM -0400, [EMAIL PROTECTED] wrote:
> On Wed, 12 Apr 2006 14:28:59 CDT, Bryan Bradsby said:
> > 
> > > Silently deleting other people's e-mail should never even be considered.
> > 
> > Unless that email is a virus, or a spam with a forged envelope sender.
> 
> No, in that case you 550 the sucker.

Unfortunately there is plenty of mailing list manager software that
will disable your subscription if your mail is rejected enough
times.  Mailman being a good example.  I have been unsubbed from
mailman lists that have allowed viruses through, even with the
default mailman settings for boucne processing.

In a perfect world, no mailing lists distribute spam, viruses and
malware.

At the moment therefore while practicing reject after DATA I do find
it necessary to mark as spam and accept if it has Precedence: bulk
(or list or whatever), because otherwise my users complain and
"don't subscribe to poorly-managed lists then" is not an acceptable
answer for them.

Regards,
Andy


signature.asc
Description: Digital signature


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-13 Thread Matthew Black


On Wed, 12 Apr 2006 18:56:44 -0700 (PDT)
 "Steve Thomas" <[EMAIL PROTECTED]> wrote:



How does one properly report delivery failure to a guerrilla spammer?


If you accept the message, you can presumably deliver it. In this day and
age, anyone accepting mail for a domain without first checking the RCPT TO
- even (especially?) on a backup MX - should have their head examined. In
the event that the RCPT TO is valid but the message truly can't be
delivered for some other reason,


In this day and age it is not always possible to check for valid
addresses at a border SMTP gateway. Sites have multiple legacy
systems which are not very interoperable. Some e-mail gateways
are incapable of scanning messages in-line. How does that make
the gateway junk or the system administrator an idiot or
incompetent?


you should bounce the message and fix the problem.


This is advocating collateral damage because nearly all spam
and viruses have return paths which falsely implicate innocent
victims (i.e., blowback). Users don't want it delivered or dropped
in their junk folder; most wouldn't know what to do with a junk
folder.


E-mail systems require investments in the 100s of thousands of
dollars, not some Windows PC running Linux. The largest part of
the cost equation is personnel and training, not hardware.

Large organizations like our shy away from open source software
in many situations NOT because it's open source. We opt for
commercial solutions so employees, like me, can take vacation
and know that other employees can handle problems and let me
enjoy my vacation without carrying a pager (unless you think
it's cool to be tethered to your job 24/7 with a Blackberry).

Dogmatic adherence to a literal reading of every RFC is
impractical. When my organization decided to drop BrightMail
postively-identified spam, we accepted a FP rate of less than
one in a million as a good thing, fully aware that this violated
RFC 821.

I used to love sendmail but recommended our organization drop it.
Sendmail's queue processing algorithm was (is?) hopelessly broken
and delayed e-mail for hours or just discarded it after five days
because sendmail couldn't properly prioritize the queue.

With our IronPort C60 gateway, almost all e-mail is processed
sub-second, users don't see postiviely-identified spam, and
viruses and phishing attempts are a thing of the past. Should
I no longer be able to perform my duties, for whatever reason,
our e-mail system will continue running and someone else can
take on my responsibilities with a tiny learning curve. No
worries about whether SpamAssassin got it's update. No worries
about whether ClamAV will be running next month. No worries
about system outages during complicated open-source software
upgrades, even for a few minutes. Unless you feel those are OK.

Ask yourself this question: can your organization survive a loss
of its entire technical staff? Would new employees be able to
manage your systems or would chaos result?

matthew black
california state university, long beach


SMTP: run-to-completion, backscatter, et cetera (Re: Spam filtering bcps ...)

2006-04-12 Thread Edward B. DREGER

ST> Date: Wed, 12 Apr 2006 18:56:44 -0700 (PDT)
ST> From: Steve Thomas

ST> If you accept the message, you can presumably deliver it. In this

Possibly.  However, insufficient storage is not the only cause of 4xx
status.


ST> day and age, anyone accepting mail for a domain without first
ST> checking the RCPT TO - even (especially?) on a backup MX - should
ST> have their head examined.

Especially.


ST> IN the event that the RCPT TO is valid but the message truly can't
ST> be delivered for some other reason, you should bounce the message
ST> and fix the problem.

*Iff* the bounce can be sent to the correct location.  That's a big
iff these days.


ST> My point was that when it comes to spam, it should either be rejected
ST> inline or delivered.

That's ideal.  I can think of several realistic conditions where a
message could be queued but not validated until later.  I'm simply
stating that { accepted | pending | refused } is a reasonable set of
responses.  From an end-to-end perspective, SMTP transactions are
asynchronous and not guaranteed, anyway.

You're advocating run-to-completion.  I'm suggesting an asynchronous
realtime system instead.  Polls could be coalesced.

Note also the implications of polling for message status: Eliminate
bounces.  Want to know if a message went through?  Poll.  Receive bounce
inline if appropriate.  That seems better than the current push-based
crapshoot.

Want to confirm that a user has retrieved a message?  Now possible at
the MX level.  Want to confirm receipt by the server without divulging
if the user has retrieved the message?  Return a status code indicating
such.

Frankly, I'd go for pull-based response codes just to be rid of
backscatter.  The rest is gravy.


Eddy
--
Everquick Internet - http://www.everquick.net/
A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/
Bandwidth, consulting, e-commerce, hosting, and network building
Phone: +1 785 865 5885 Lawrence and [inter]national
Phone: +1 316 794 8922 Wichita

DO NOT send mail to the following addresses:
[EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED]
Sending mail to spambait addresses is a great way to get blocked.
Ditto for broken OOO autoresponders and foolish AV software backscatter.


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Steve Thomas

> How does one properly report delivery failure to a guerrilla spammer?

If you accept the message, you can presumably deliver it. In this day and
age, anyone accepting mail for a domain without first checking the RCPT TO
- even (especially?) on a backup MX - should have their head examined. In
the event that the RCPT TO is valid but the message truly can't be
delivered for some other reason, you should bounce the message and fix the
problem.

My point was that when it comes to spam, it should either be rejected
inline or delivered. Unless your spam scanner has 100% accuracy, 100% of
the time, there is no justification for sending anything not addressed to
you to /dev/null.

> "Please automatically delete anything that might be spam.  They'll call
> me if it's important.  I know I'll lose some mail, but that's okay."

If you have an agreement with a customer that specifically allows for such
behaviour, great. We can get into individual cases for any concievable
scenario, but that would be silly. It was pretty clear, to me at least,
that we were discussing this as it would pertain to a system-wide
configuration.

> As for MUST bounce using return-path... perhaps you've never experienced
> blowback from a joe job.  It can be unpleasant.

Yes, I have. And yes, it is. However, I never suggested bouncing spam, as
my last message clearly stated. My position is that if you accept the
message (250 OK), you have an obligation to deliver it. If you can't scan
it during the SMTP transaction, do it after and mark up the headers, drop
it in a junk folder - whatever - but don't delete it.

St-




Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Edward B. DREGER

ST> Date: Wed, 12 Apr 2006 10:16:53 -0700 (PDT)
ST> From: Steve Thomas

ST> RFC 2821?
ST> 
ST>   ...the protocol requires that a server accept responsibility
ST>   for either delivering a message or properly reporting the
ST>   failure to do so.

How does one properly report delivery failure to a guerrilla spammer?


ST> Unless you're the final recipient of the message, you have no business
ST> deleting it. If you've accept a message, you should either deliver or
ST> bounce it, per RFC requirements.

"Please automatically delete anything that might be spam.  They'll call
me if it's important.  I know I'll lose some mail, but that's okay."

Throwing RFC 2821 at that user probably would not have made them happy.

As for MUST bounce using return-path... perhaps you've never experienced
blowback from a joe job.  It can be unpleasant.

RFCs are for maintaining interoperability.  They are not infallible.
When a system is clearly broken, it's time to examine alternatives --
not to say that the RFC was handed down from on high.

Proposal:

MXes can say "2xx message queued with ID blahblahblah".  They also can
return 4xx "try back later codes".  Yes?

How about some return code that says "poll by $deadline if you want to
know whether message ID blahblahblah was accepted or rejected"?  No need
to retransmit the entire message, and the sender can learn whether the
message was actually accepted.


Eddy
--
Everquick Internet - http://www.everquick.net/
A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/
Bandwidth, consulting, e-commerce, hosting, and network building
Phone: +1 785 865 5885 Lawrence and [inter]national
Phone: +1 316 794 8922 Wichita

DO NOT send mail to the following addresses:
[EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED]
Sending mail to spambait addresses is a great way to get blocked.
Ditto for broken OOO autoresponders and foolish AV software backscatter.


Re: Spam filtering bcps

2006-04-12 Thread Matthew Sullivan


Bryan Bradsby wrote:


Silently deleting other people's e-mail should never even be considered.
   



Unless that email is a virus, or a spam with a forged envelope sender.
 

Why?  - You can scan for viruses inline using a variety of products (eg: 
I have patched Postfix to use clamav inline on modest hardware (single 
CPU AMD64 will do it, so will a Dual PIII 866) and it will accept 
messages at 50 messages per second (sustained load) and scan for viruses 
before responding to the end-of-data command, rejecting if a virus is 
detected.).


Spam is a different subject altogether - are you that sure you can 
detect spam without a false positive?  If so then why aren't you doing 
it inline?  If you can't why are you blindly deleting the messages? - My 
BCP comment is if you can't detect inline (eg for performance reasons) 
tag it and deliver it (if you have the capabilities, deliver it to a 
junk folder) - that way you are following the RFC's and no non spam mail 
is deleted by the system.


Regards,

Mat



Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Matthew Sullivan


Steve Thomas wrote:


Earlier today, I said:
 


Unless you're the final recipient of the message, you have no business
deleting it. If you've accept a message, you should either deliver or
bounce it, per RFC requirements.
   



I just want to clarify that I was in no way suggesting that anyone bounce
spam - I was merely pointing out that if you choose to 250 a message, you
have to deliver it. The much better option is to 550 it after DATA if you
don't like what you see. Silently deleting other people's e-mail should
never even be considered.
 



This policy I whole heartedly agree with, and I strive where ever 
possible to enforce this in every place I work, where ever people get 
listed in SORBS for backscatter, I work with them telling them how they 
can do this


With the current technologies available there is no reason a 
small-medium organisation cannot virus and spam scan mail inline at the 
SMTP transaction stage. (Even the barracuda's can spamassassin scan at 
around 8 messages per second - my previous employment were receiving 
around 4 messages per second - which translated to 1-2 million emails 
per day)


It is possible to do inline scanning in larger ISPs (I personally have 
configured a 'system' to handle upto 90 message per second inline 
scanning) - though it requires a lot more planning, thought, and careful 
consideration.


Regards,

Mat


Re: Spam filtering bcps

2006-04-12 Thread Matthew Black


On Wed, 12 Apr 2006 14:28:59 -0500 (CDT)
 Bryan Bradsby <[EMAIL PROTECTED]> wrote:



Silently deleting other people's e-mail should never even be considered.


Unless that email is a virus, or a spam with a forged envelope sender.

-bryan bradsby



Aha, so there are situtations where this is acceptable?
What about deleting viral attachments or altering subject
lines...is that permissible? The sweeping generalizations
I've read leave little room for responding to real-world
situations.

matthew black
california state university, long beach


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Matthew Black


On Wed, 12 Apr 2006 14:18:24 -0400
 [EMAIL PROTECTED] wrote:

On Wed, 12 Apr 2006 10:16:53 PDT, Steve Thomas said:


> I haven't seen any succinct justification for providing a
> 550 message rejection for positively-identified spam versus
> silently dropping the message. Lots of how-to instructions
> but no whys.

RFC 2821?

  ...the protocol requires that a server accept responsibility
  for either delivering a message or properly reporting the
  failure to do so.


Your statement is open to multiple interpretations. I argue that
anytime our system identifies a message as spam that it gets
delivered to the system bit bucket.

RFC-821 and netiquette also "mandate" e-mail be properly addressed.
System manufacturers and administrators make compromises because
strict adherence to the rules is not always possible from an
operational perspective.

 

Elsewhere in 2821 (6.1, to be specific):

  When the receiver-SMTP accepts a piece of mail (by sending a "250 OK"
  message in response to DATA), it is accepting responsibility for
  delivering or relaying the message.  It must take this responsibility
  seriously.  It MUST NOT lose the message for frivolous reasons, such
  as because the host later crashes or because of a predictable
  resource shortage.


Lost me on that part about crashes being frivolous reasons.
This is a political statement not an indisputable matter of fact.


OK? Got that? You '250 OK' it, you got a *serious* responsibility.  Losing 
the
message because the whole damned machine crashes is considered a frivolous 
reason.


And throwing it away because you don't like the way it looks is OK?  Man,

...***

you're in for some severe karmic protocol payback down the road... ;)


I'm not the one throwing them away and never look at them; watch
the finger wagging. And thanks for the karma heads up, Bhudda.

matthew black
california state university, long beach


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Henry Yen

On Wed, Apr 12, 2006 at 12:03:51PM -0400, Joe Maimon wrote:
> Matthew Black wrote:
> 
> > there's no bandwidth savings from silently dropping the message
> > versus providing a 550 rejection. In the best of all worlds,
> > it would be nice to give feedback. No system is perfect and a
> > false-positive rate of less than one in a million "220" accepted
> > messages seems pretty small.
> 
> Let me ask you this simple question:
> 
> If you know at close of DATA whether you are going to actually perform 
> final delivery, what does it cost you to follow standards and issue a 
> 550 instead of a 220 and discard it?
> 
> If you use a 550, a real live person sending an email that somehow gets 
> FP will actually benefit.

In today's world, at least with the spamtorrent I see at my clients,
that's just untrue.  If your filtering is set up well, and you mark
an e-mail as SPAM, it almost certainly is (yes, I'll certainly concede
FP's exist, but again, it almost certainly doesn't matter that much in
that teensy number of occurrences); and 99-plus-percent of spam
is emitted from spambots who don't give a $expletive about return
status one way or another.  If you're worrying about "no-status" in
the context of FP's, then your filtering isn't set up well, which really
means you've got larger problems.

> I am with Suresh on this, just like in the past threads. Search the archive.

Though not contradicting what I just wrote, so am I.  However, header-forged
and multi-chained spam from firehose-like spambots don't play by any of our
rules; all they do is blast away in a largely one-way transaction (guess
which direction!).  A 550 now-a-days has nowhere to "go" (and those
"commercial" akak "legit") spamhouses don't wash their lists even on 550's.

-- 
Henry Yen   Aegis Information Systems, Inc.
Senior Systems Programmer   Hicksville, New York


Re: Spam filtering bcps

2006-04-12 Thread Valdis . Kletnieks
On Wed, 12 Apr 2006 14:28:59 CDT, Bryan Bradsby said:
> 
> > Silently deleting other people's e-mail should never even be considered.
> 
> Unless that email is a virus, or a spam with a forged envelope sender.

No, in that case you 550 the sucker.


pgpOxV2xehNAo.pgp
Description: PGP signature


Re: Spam filtering bcps

2006-04-12 Thread Bryan Bradsby

> Silently deleting other people's e-mail should never even be considered.

Unless that email is a virus, or a spam with a forged envelope sender.

-bryan bradsby


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Steve Thomas

Earlier today, I said:
> Unless you're the final recipient of the message, you have no business
> deleting it. If you've accept a message, you should either deliver or
> bounce it, per RFC requirements.

I just want to clarify that I was in no way suggesting that anyone bounce
spam - I was merely pointing out that if you choose to 250 a message, you
have to deliver it. The much better option is to 550 it after DATA if you
don't like what you see. Silently deleting other people's e-mail should
never even be considered.

Returning to lurk status...

St-




Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Valdis . Kletnieks
On Wed, 12 Apr 2006 10:16:53 PDT, Steve Thomas said:
> 
> > I haven't seen any succinct justification for providing a
> > 550 message rejection for positively-identified spam versus
> > silently dropping the message. Lots of how-to instructions
> > but no whys.
> 
> RFC 2821?
> 
>   ...the protocol requires that a server accept responsibility
>   for either delivering a message or properly reporting the
>   failure to do so.

Elsewhere in 2821 (6.1, to be specific):

   When the receiver-SMTP accepts a piece of mail (by sending a "250 OK"
   message in response to DATA), it is accepting responsibility for
   delivering or relaying the message.  It must take this responsibility
   seriously.  It MUST NOT lose the message for frivolous reasons, such
   as because the host later crashes or because of a predictable
   resource shortage.

OK? Got that? You '250 OK' it, you got a *serious* responsibility.  Losing the
message because the whole damned machine crashes is considered a frivolous 
reason.

And throwing it away because you don't like the way it looks is OK?  Man,
you're in for some severe karmic protocol payback down the road... ;)


pgpmW5ds5R1xP.pgp
Description: PGP signature


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Steve Thomas

> I haven't seen any succinct justification for providing a
> 550 message rejection for positively-identified spam versus
> silently dropping the message. Lots of how-to instructions
> but no whys.

RFC 2821?

  ...the protocol requires that a server accept responsibility
  for either delivering a message or properly reporting the
  failure to do so.

  ...

  If an SMTP server has accepted the task of relaying the mail
  and later finds that the destination is incorrect or that
  the mail cannot be delivered for some other reason, then
  it MUST construct an "undeliverable mail" notification message
  and send it to the originator of the undeliverable mail (as
  indicated by the reverse-path).

Unless you're the final recipient of the message, you have no business
deleting it. If you've accept a message, you should either deliver or
bounce it, per RFC requirements.




Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Joe Maimon




Matthew Black wrote:





there's no bandwidth savings from silently dropping the message
versus providing a 550 rejection. In the best of all worlds,
it would be nice to give feedback. No system is perfect and a
false-positive rate of less than one in a million "220" accepted
messages seems pretty small.


I thought I had already participated in beating this dead horse 
sufficiently in multiple threads in multiple forums on multiple 
occasions. Maybe I am in your killfile or something. If I post again on 
this topic, I certainly will deserve to be.


Let me ask you this simple question:

If you know at close of DATA whether you are going to actually perform 
final delivery, what does it cost you to follow standards and issue a 
550 instead of a 220 and discard it?


If you use a 550, a real live person sending an email that somehow gets 
FP will actually benefit.


I am with Suresh on this, just like in the past threads. Search the archive.



Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Matthew Black


On Wed, 12 Apr 2006 21:12:44 +0530
 "Suresh Ramasubramanian" <[EMAIL PROTECTED]> wrote:

On 4/12/06, Matthew Black <[EMAIL PROTECTED]> wrote:


Where is the bandwidth savings once we've accepted an entire message,
scanned it, determined it was spam, then provided a 550 rejection
versus silently droping?


If you can scan it inline, you can stop, issue a 550 and drop the SMTP
connection any time you want.  Like for example, midstream when you
discover a fake header pattern.

You'd start with whatever can be rejected in session - fake HELOs,
blocklist listed IPs, random faked headers,  dodgy attachment types
that are more likely to be viruses than not

Then apply the heavier and more cpu intensive filters later, on a much
smaller volume of spam


We already do this.

 

Maybe not all that much of a bandwidth / cpu saving, but saving remote
postmasters the hassle of troubleshooting lost email is always a good
idea.


After all said methods have been performed and the message gets
through reputation filtering; blacklists; forged/munged headers,
e-mail addresses, domain names the message comes in and then
there's that final dot. Up to this point, the message hasn't
proven to be spam until it can be scanned using BrightMail,
SpamAssassin, Baysian filters, DCC lists, or other methods.
All I'm saying is that once the full DATA submission has completed,
there's no bandwidth savings from silently dropping the message
versus providing a 550 rejection. In the best of all worlds,
it would be nice to give feedback. No system is perfect and a
false-positive rate of less than one in a million "220" accepted
messages seems pretty small.

matthew black
california state university, long beach


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Suresh Ramasubramanian

On 4/12/06, Matthew Black <[EMAIL PROTECTED]> wrote:

> Agreed, but we're willing to live with an error rate of less
> than one in a million. This isn't a space shuttle. I don't think
> the USPS can claim 99.% delivery accuracy. Nonetheless, to

I'm not even saying five nines.  Spam filtering - even with heuristics
etc - is less than perfect, and per user spam filtering, however idiot
proof, sometimes turns out to be like giving Acme Inc gadgets to Wile
E Coyote.  [users having fun with procmail and .forwards should
already be a familiar story I guess?]

> We already reject most connections with a 550 or TCP REFUSE
> based on reputation filtering and blacklists, et al.

That works just fine.  I dont have any argument with it

> Where is the bandwidth savings once we've accepted an entire message,
> scanned it, determined it was spam, then provided a 550 rejection
> versus silently droping?

If you can scan it inline, you can stop, issue a 550 and drop the SMTP
connection any time you want.  Like for example, midstream when you
discover a fake header pattern.

You'd start with whatever can be rejected in session - fake HELOs,
blocklist listed IPs, random faked headers,  dodgy attachment types
that are more likely to be viruses than not

Then apply the heavier and more cpu intensive filters later, on a much
smaller volume of spam

Maybe not all that much of a bandwidth / cpu saving, but saving remote
postmasters the hassle of troubleshooting lost email is always a good
idea.

--
Suresh Ramasubramanian ([EMAIL PROTECTED])


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Matthew Black


On Wed, 12 Apr 2006 20:30:16 +0530
 "Suresh Ramasubramanian" <[EMAIL PROTECTED]> wrote:


On 4/12/06, Matthew Black <[EMAIL PROTECTED]> wrote:


I haven't seen any succinct justification for providing a
550 message rejection for positively-identified spam versus
silently dropping the message. Lots of how-to instructions
but no whys.



For viruses - fine.  But you are not going to find any spam filter in
the world that doesnt have false positives.  And in such cases its
always a good idea to let the sender know his email didnt get through.


Agreed, but we're willing to live with an error rate of less
than one in a million. This isn't a space shuttle. I don't think
the USPS can claim 99.% delivery accuracy. Nonetheless, to
allay worries, we are considering spam quarantines to allow
recipients an opportunity to review spam messages themselves, much
like Yahoo! Mail.


Complaints about e-mail not getting through won't be solved
with a 550 versus silently dropping spam because most users aren't
willing to sift through e-mail errors to find the specific cause
for delivery failure. Members of this list are a rare exception.



Like for example - you see a large webmail provider whose hosts and
domains keep getting forged into spam, misread the headers and block
that provider.  In such cases, its your users who arent getting a lot
of valid email from their friends and relatives who are using that
provider, and 550'ing instead of trashing email saves the senders, and
their provider,  quite  lot of time that'd otherwise be spent
troubleshooting the issue.

Plus, 5xx smtp rejects tend to save your bandwidth a bit compared to
accepting the entire email (not that it matters on a small university
domain where your userbase is going to be fairly small, and bandwidth
available quite generous ..  but for larger sites, or sites with
bandwidth issues, that's definitely a concern)


We already reject most connections with a 550 or TCP REFUSE
based on reputation filtering and blacklists, et al.

Where is the bandwidth savings once we've accepted an entire message,
scanned it, determined it was spam, then provided a 550 rejection
versus silently droping?

matthew black
california state university, long beach


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Tony Finch

On Wed, 12 Apr 2006, Matthew Black wrote:
>
> I haven't seen any succinct justification for providing a
> 550 message rejection for positively-identified spam versus
> silently dropping the message.

If you are wrong about the message being spam, then the sender gets a
bounce.

Tony.
-- 
f.a.n.finch  <[EMAIL PROTECTED]>  http://dotat.at/
BERWICK ON TWEED TO WHITBY: WEST OR SOUTHWEST 5 OR 6, PERHAPS INCREASING 7
LATER IN NORTH. RAIN AT FIRST. MAINLY GOOD. SLIGHT OR MODERATE.


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Suresh Ramasubramanian

On 4/12/06, Matthew Black <[EMAIL PROTECTED]> wrote:
>
> I haven't seen any succinct justification for providing a
> 550 message rejection for positively-identified spam versus
> silently dropping the message. Lots of how-to instructions
> but no whys.
>

For viruses - fine.  But you are not going to find any spam filter in
the world that doesnt have false positives.  And in such cases its
always a good idea to let the sender know his email didnt get through.

Like for example - you see a large webmail provider whose hosts and
domains keep getting forged into spam, misread the headers and block
that provider.  In such cases, its your users who arent getting a lot
of valid email from their friends and relatives who are using that
provider, and 550'ing instead of trashing email saves the senders, and
their provider,  quite  lot of time that'd otherwise be spent
troubleshooting the issue.

Plus, 5xx smtp rejects tend to save your bandwidth a bit compared to
accepting the entire email (not that it matters on a small university
domain where your userbase is going to be fairly small, and bandwidth
available quite generous ..  but for larger sites, or sites with
bandwidth issues, that's definitely a concern)

  --srs

--
Suresh Ramasubramanian ([EMAIL PROTECTED])


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Matthew Black


Several people kindly contacted me off list with laborious
explanations of how to implement delayed 550 rejections using
sedmail, et al. We gave up sendmail years ago in favor of a
competing solution.

I haven't seen any succinct justification for providing a
550 message rejection for positively-identified spam versus
silently dropping the message. Lots of how-to instructions
but no whys.

matthew black
california state university, long beach


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Joe Maimon




Matthew Sullivan wrote:



Suresh Ramasubramanian wrote:


On 4/11/06, Matthew Black <[EMAIL PROTECTED]> wrote:
 


Are you suggesting that we configure our e-mail servers to notify
people upon automatic deletion of spam? Frequently, spam cannot be
properly identified until closure of the SMTP conversation and that
final 200 mMESSAGE ACCEPTED...or do you think that TCP/IP connection
should be held open until the message can be scanned for spam and
viruses just so we can give a 550 MESSAGE REJECTED error instead of
silently dropping it?

  



You can reject right after DATA, at the . stage, 
before QUIT


That's still an in line smtp reject rather than an accept + bounce DSN.

Exim with the spamassassin patches (sa-exim) does this, for example.

-srs
 

Of course Postfix can be setup (using spampd) with spamassassin to do 
exactly the same.


I believe Sendmail+MimeDefang+Spamassassin will also reject inline if 
set to do so.


Regards,

Mat




As will sendmail+spamass-milter+spamassassin

In fact there are quite a few milters that can be used in between 
sendmail and spamassassin


Joe


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Matthew Sullivan


Suresh Ramasubramanian wrote:


On 4/11/06, Matthew Black <[EMAIL PROTECTED]> wrote:
 


Are you suggesting that we configure our e-mail servers to notify
people upon automatic deletion of spam? Frequently, spam cannot be
properly identified until closure of the SMTP conversation and that
final 200 mMESSAGE ACCEPTED...or do you think that TCP/IP connection
should be held open until the message can be scanned for spam and
viruses just so we can give a 550 MESSAGE REJECTED error instead of
silently dropping it?

   



You can reject right after DATA, at the . stage, before QUIT

That's still an in line smtp reject rather than an accept + bounce DSN.

Exim with the spamassassin patches (sa-exim) does this, for example.

-srs
 

Of course Postfix can be setup (using spampd) with spamassassin to do 
exactly the same.


I believe Sendmail+MimeDefang+Spamassassin will also reject inline if 
set to do so.


Regards,

Mat


Re: Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-12 Thread Tony Finch

On Wed, 12 Apr 2006, Suresh Ramasubramanian wrote:
>
> Exim with the spamassassin patches (sa-exim) does this, for example.

SpamAssassin support is built in to Exim since version 4.50.

Tony.
-- 
f.a.n.finch  <[EMAIL PROTECTED]>  http://dotat.at/
BERWICK ON TWEED TO WHITBY: WEST OR SOUTHWEST 5 OR 6, PERHAPS INCREASING 7
LATER IN NORTH. RAIN AT FIRST. MAINLY GOOD. SLIGHT OR MODERATE.


Spam filtering bcps [was Re: Open Letter to D-Link about their NTP vandalism]

2006-04-11 Thread Suresh Ramasubramanian

On 4/11/06, Matthew Black <[EMAIL PROTECTED]> wrote:
>
> Are you suggesting that we configure our e-mail servers to notify
> people upon automatic deletion of spam? Frequently, spam cannot be
> properly identified until closure of the SMTP conversation and that
> final 200 mMESSAGE ACCEPTED...or do you think that TCP/IP connection
> should be held open until the message can be scanned for spam and
> viruses just so we can give a 550 MESSAGE REJECTED error instead of
> silently dropping it?
>

You can reject right after DATA, at the . stage, before QUIT

That's still an in line smtp reject rather than an accept + bounce DSN.

Exim with the spamassassin patches (sa-exim) does this, for example.

-srs