On 6/7/2018 6:55 PM, Brandon Long wrote:
Note I'm not saying that IP time blocking isn't useful. My issue is,
are most RBL's good for IP time blocking? An RBL is a statement that
everything from that IP is bad, but the truth of that statement varies
greatly based on the RBL in question. But, in the end, what everyone
seems to have is that hammer, and so the little built-in software
support is for RBLs and using them either at connection time or
letting the mail through. If the industry had moved to a reputation
model, it would be easier to discuss "how bad is it" and whether it's
bad enough to block at IP time, or whether you mix it into your spam
score.
Brandon,
Much of what you're recommending... has essentially been done - or dealt
with - once you "peel back the layers"...
First I should mention that, because of the following three factors, the
zero-false-positive blacklist ideal... is no longer possible:
(A) legit websites getting hacked by spammers (who install spam content
pages) is epidemic
(B) legit (generally non-spamming) mail servers getting an account
hijacked by spammers is epidemic
(C) normally good ESPs or hosters getting a "bad apple" spammer customer
is still a constant problem (though some ESPs do a MUCH better job than
others of vetting their new customers!)
So this list above is made up of generally good senders who normally
would never get blacklisted - but due to a security lapse - they then
sometimes they get blacklisted, and for good cause! Not ever
blacklisting those would be too large of a loophole for spammers, and
the hijacked sites and accounts wouldn't get fixed! They would just
build up - and many admins wouldn't have a clue about their security
problem.
And then there are occasional dark-gray-hat ESPs and hosters who start
out with BOTH legit customers and spamming customers -AND- who do little
or no vetting - they don't pursue spammers - they just let the chips
fall where they may without vetting their customers very well - and
these days - those are now ALWAYS going to downward spiral into
blacklist hell. There is no way to avoid that, since spammers are
attracted to that like flies on a picnic lunch - and the collateral
damage is well deserved, but won't last long anyways as legit customers
will flee - and that is a GOOD thing!
So, at the end of the day, you're left with 4 categories of blacklists:
(1) blacklists which do a GREAT JOB of keeping false positives to a
minimum by carefully weighing the potential collateral damage of such a
listing into the listing decision - and do a great job of delisting
those who have fixed their problems - and have good
metrics/algorithms/procedures for delistings, but do NOT allow "too
easy-off", but which do self-correct for situations that got fixed. (and
they don't leave already-fixed-situations blacklisted for many days or
weeks after the problem is fixed). These blacklists will still have some
occasional collateral damage, but it is a tiny fraction of a percent of
the whole - and therefore these are safe to use for outright blocking.
(for sending IP lists - examples in this category would definitely
include Spamhaus and invaluement... MAYBE one or two others?)
(2) same, but they do a GOOD job of this, not a great job (so these
would be better for high scoring, but you might get too many FPs if you
score above threshold)
(3) same, but a mediocre job - so for these, I might add recommend
adding 1 or 2 points in SpamAssassin - but these lists are STILL
extremely valuable because they can often put the spam from a normally
legit source that was hijacked "over the top" - yet without causing
massive FPs from that sender's legit email. (whereas blacklists in the
first category which aim for extremely low FPs - had to pass on such
listings)
(4) all of the above - except the list is too "easy off" - helping
snowshoe and other more deliberate spammers to get themselves delisted
too easily (sometimes just in time for their next spam campaign!) These
can often be scored above threshold - but don't count on these for
blocking the more shrewd spammers!
ALL of these play a role in spam filtering. All of them! To varying
degrees - they are ALL useful. And they ALL help reduce the load on
content filters (though the 1st category does the most "heavy lifting"
for minimizing resource usage)
So to answer your question, a whole echo-system/culture has evolved
around this... which effectively deals with the challenges you mentioned!
NOW - WHAT TO DO ABOUT HOSTERS/ESPs WHO DON'T CARE (or don't care much)?
With invaluement - we've had a situations where an ESP or hoster got so
aggressively bad (often, talking the good talk publicly, but not
matching that talk with actions) - that we've had to yank their IPs or
domains out of our whitelist - and there is one particularly painful
ongoing situation right now that is causing a little more collateral
damage than what I'm comfortable with - but this is (hopefully) putting
pressure on this particular ESP to reform, where their IPs are getting
listed by invaluement occasionally due to spam. Also, in cases where the
listing of the IP absolutely would cause too much collateral damage -
but too much spam spews from that IP - we have a special type of
whitewashing that prevents the IPs from getting listed - but AMPLIFIES
the content scoring of all messages from those IPs - and then this
greatly increases the chances that spammer's domains will get listed.
The effected ISP or hoster or ESP thinks that everything is great - but
their lack of quality control means that their clients are at a higher
risk of getting their domains blacklisted! (hopefully, what I just said
will keep certain ESP abuse admins up at night!) ...and this is just a
"tip of the iceberg" brief summary of some of the things/decisions
involved. "its complicated" - and then the blacklists in categories 2
and 3 above will be MORE aggressive (and rightfully so), hitting on many
of the more gray areas where Spamhaus and Invaluement deferred - so this
is like an entire echo system that has evolved with IPv4 blacklists,
which across-the-board makes spam filtering more effective... AND MORE
EFFICIENT! (and even more efficient for those spams that were not
blocked at connection time, because this RBL scoring, even for the
low-scoring blacklists, STILL reduces the load on a more resource-hungry
content filtering due to "early exiting" strategies!)
It will be DIFFICULT to replicate this on IPv6... PRECISELY BECAUSE
hosters and ESPs (and their spammer customers) will be able to run from
this too easily due to too little scarcity of new IPv6 space (unlike
IPv4). (and they'll deploy one-and-done strategies, and listwashing
strategies - that can't be done with IPv4)
Will SMTP be the last hold-out on IPv4?
It OUGHT to be! And that is OK. There is enough inertia with massive
amounts of software and hardware worldwide needing MUCH more time to
transition to IPv6, and then to transition to ONLY-IPv6 - such that I
don't think we'll need to worry about SMTP-IPv4 being the lone holdout
for decades to come. But if it is the last thing - that means that we
gave our systems ample time to develop new strategies and technologies
for dealing with IPv6 - and/or allowed our budgets and/or processing
power to catch up with the need for more content filtering - and that is
superior to giving the spammers a shorter term massive boost in their
capabilities.
--
Rob McEwen
https://www.invaluement.com
_______________________________________________
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
