On 6/8/2018 5:49 AM, David Hofstee wrote:
> ... score of the sending-IP, which is similar to what you've
described, correct?
Correct.
So you have these mechanisms in place. But your customers, who get
access to the invaluement RBL, do not. Am I correct? If I am,
it still results in the conclusion that blacklists are not sufficient
to have a resulting good spam filter. You would be ok, the list would
not have false positives, but your customers would not be sufficiently
covered once bad guys get smarter.
David,
You've made so many false assumptions to come to these conclusions...
and taken things I've said out of context to get there... I had a hard
time knowing where to begin!
(1) First, I "eat my own dogfood", even for my own mailbox! In our own
spam filtering system, we score ALL invaluement blacklists "above
threshold". However, in VERY RARE situations, a message will get
delivered in our mail system where (a) it had one hit on one invaluement
list, (b) NOTHING else spammy triggered, (c) and some rules kicked in
that lowered the spam score just barely below threshold -BUT GUESS
WHAT?- the vast majority of the time that happens, it ends up being a
FALSE NEGATIVE - then I'm jealous of my own customers whose systems
didn't deliver those spams to their users' inboxes!
(2) A large percentage of invaluement subscribers use SpamAssassin, and
likewise use a multi-tiered scoring system where they score blacklists
higher if that blacklist (a) had fewer FPs, -AND- (b) the FPs it
generates are more likely to result in extremely rare and/or extremely
justified FPs. And they score OTHER DNSBLs lower for those which have a
higher frequency of hits on desired mail. And some have similar scoring
options with other spam filtering systems in addition to SpamAssassin.
(3) I'm certain that some portion of invaluement subscribers have BETTER
filtering than we have for our boutique mail hosting system, but that
circumstance is somewhat rare ONLY because we can afford to put a lot
more resources per mailbox into this, due to our DNSBL effectively
subsidizing our small mail hosting system. (But this helps us to make
the quality of invaluement data even better, which only benefits ALL of
our subscribers - the opposite would be true if we neglected our own
filtering system!) But since our subscribers all use a VARIETY of
technologies and approaches to spam, and since invaluement data is only
tacking a subset of all spam, and isn't trying to be a comprehensive
solution for blocking all spam - every client has a unique situation.
And I'm certain that at least SOME of them have even BETTER spam
filtering than what we have for our mail hosting customers. But they all
share one thing in common - they all have excellent filtering in the
areas of their spam filtering that invaluement attempts to improve! :)
(4) You seem to be very confused about what I mean when I talk about how
there has to be some justified level of "collateral damage" these days,
due to the very high frequency of hijacked accounts, hijacked websites,
and spamming ESP customers (from ESP that are overall good). Keep in
mind that, ALL of these added together at one time - can be STILL be an
astronomically tiny percentage-wise when it comes to how much the
collateral damage impacts the average end user, yet can still be
tremendously harmful to the company with the security problem, since the
problem CONCENTRATES there. To give an extreme example, suppose a small
business with 25 employees, who averages 500 outbound legit emails a day
- had a security lapse and their server starts attempting to send out
200,000+ egregious spams per day - and now their 500 outbound legit
emails are getting blocked in many places. The chances that an ISP with
10,000 mailboxes - or even 1M mailboxes - is going to be impacted by the
collateral damage - and have to deal with user complaints about false
positives - is extremely rare - yet this small business is going to have
a very very bad day that day! In a situation like this, their sending IP
is likely to get blacklisted on Invaluement and/or Spamhaus - but will
likely also get delisted fairly soon after they submit a delist request
and/or fix the problem (but that could take longer if they are doing
stupid stuff - like having a poorly formed PTR record that looks dynamic
and doesn't properly convey identity and reputation... but I digress)
And then other higher-FP blacklists will do a lot of similar listings,
except they'll also include situations where the
spam-to-collateral-damage ratios are NOT so clear cut. These other lists
are better for scoring - AND MANY (OR MOST?) INVALUEMENT SUBSCRIBERS
HAVE THE TECHNOLOGY TO DO THAT SCORING. (did you not know that?)
(5) Also, a large percentage of Invaluement subscribers choose to block
at the perimeter (at connection, without accepting the message) based on
Zen (from Spamhaus), ivmSIP, and ivmSIP/24 - and are extremely pleased
with the extremely low number of FPs - and the way that we score ALL
invaluement DNSBL hits on messages to our own users' mailboxes "above
threshold" - is very similar to that. And in those rare instances where
something in our system caused an invaluement-listed message to get
delivered to the inbox, the vast majority of the time - it ended up
being a false negatives in my system, NOT the avoidance of a false
positive. (as it should be - so that we can have even better telemetry
for spotting and quickly fixing potential invaluement FPs!) But, again,
even this exceptional situation. Again, an invaluement-listed message
getting through our own filter, due to a rare set of circumstances (that
usually results in a False Negative!) is extremely rare when compared to
the number of invaluement-listed spams that our own spam filter
routinely blocks.
Finally, regarding your statement "blacklists are not sufficient to have
a resulting good spam filter"
(6) nobody in this thread ever claimed that blacklists-alone are
sufficient for having good spam filtering. I certainly have NEVER made
such a statement, or even implied such. But I have stated that it is
very difficult to have high quality filtering AND it is difficult to
have efficient filtering that can keep up with high volumes of spams,
without using blacklists. Also, something not being 100% comprehensive
and not being perfect - doesn't mean it isn't extremely beneficial and
often even critically important. For example, occasionally, we get
subscribers who use MS Exchange, don't subscriber to Microsoft's
build-in filtering options, and don't have any kind of anti-spam
exchange add-ons installed. They ONLY try to filter using RBLs entered
into MS Exchange. I ALWAYS try to educate such a person that their
filtering will NEVER have the ability to be very good if the ONLY thing
they are doing is blocking on high-quality low-FP DNSBLs. (I have this
conversation with someone at least a couple of times a year.)
--
Rob McEwen
https://www.invaluement.com
+1 (478) 475-9032
_______________________________________________
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
