Re: WARNING: Microsoft has earned removal from SA default welcomelist
On 13/04/2024 03:20, Bill Cole wrote: In my opinion, this is an indication that the default welcomelist entries in the official I'm good with that, so long as likes of google are not in any whitelist either. I haven't been following all the anti spam stuff as much as I used to (I have people to do that for me so I can enjoy more of life) in past few years, but I've never believed the big providers should ever have been whitelisted. I've used clear uridnsbl skip domain for donkies years (I think that's the option that removes the dnsbl whitelistings going off memory) but perhaps there should also be a similar command (if not already exist?) that clears and disables /all/ whitelisting in rules as well, yes I know in the past the recommended method was writing a gazillion entries in local.cf zeroing out there scores, but isn't that kind of stupid in 2024. Trust must be earned, not implied (or bought), as Joanne points out, "my spam is your ham and vice versa" -- Regards, Noel Butler
Re: Defining what the default welcomelist means
On 20240412 16:14:44, Greg Troxel wrote:
jdow writes:
One pesky detail still exists. There is a very broad fuzzy area where
my spam is your ham and vice versa. You could probably drive yourself
to an early grave trying to get the perfect Bayes training plus
perfect rule set.
spam is bulk and unsolicited. So yes the same message could be either,
but if a sender spams anyone, they are spammer, even if they send mail
that isn't spam.
Ah, no, that way leads to disaster. Some people resign from lists by declaring
the sender spam. That could end up cutting access to all the people who want the
emails. (At various times this list and most 'ix lists were unusually difficult
to resign from. And, yes, I have been around that long. I'm just too politically
incorrect for most lists these days, {^_-}) It is wise to be careful about how
soon you pull the "spammer" trigger. YMMV and YAMV (Attitude).
{^_^}
Re: Defining what the default welcomelist means
jdow writes: > One pesky detail still exists. There is a very broad fuzzy area where > my spam is your ham and vice versa. You could probably drive yourself > to an early grave trying to get the perfect Bayes training plus > perfect rule set. spam is bulk and unsolicited. So yes the same message could be either, but if a sender spams anyone, they are spammer, even if they send mail that isn't spam.
Re: Defining what the default welcomelist means
On 20240412 15:56:15, Greg Troxel wrote:
I see it very slightly differently, but mostly agree
Bill Cole writes:
1. We serve our users: receivers, not senders. Senders claiming FPs
need the support of a corroborating would-be receiver.
Agreed. Or maybe we take requests to add only from receivers.
2. If senders have FPs on objectively legitimate mail, their first and
most important step is to identify WHY SpamAssassin thinks it is
spam. and address that. Do you need the invisible text? Is the message
embedded in a remotely-fetched image? The sea of "" entities in
your messages' HTML serves what purpose exactly? If there's a real FP
problem with some rule that regularly is proved out by RuleQA, open a
bug.
Sure, but if you serve receivers, often people will have misfiling and
the sender is opaque, even if not spam and dkim. So saying the sender
should fix is misaligned with serving receivers. Yes, they *should*,
but people shouldn't send html mail either :-)
I agree that requests from senders should be met with "make your mail
less spammy".
3. This is NOT a general-purpose reputation list. It exists to aid SA
users who have FPs from SpamAssassin default rules for wanted mail,
where we cannot determine any acceptable adjustment to rules which
would avoid the problem. It is a "last resort" form of FP mitigation
when we cannot find an acceptable general solution that isn't
domain-specific to a widely accepted sender domain.
I see all spam classification as probabalistic and there is risk of FP.
If a domain emits *only ham* and is dkim signed, and we believe that
receivers want it, I think it makes sense to have it in.
I think of things like alerts from banks, airline saying your flight
time has changed, etc. where FPs are a real problem.
I am extremely skeptical of anything that smells of email marketing
here. I would expect only places sending transactional mail and alerts
to established customers.
4. We should only add or remove listings based on specific requests
backed by transparent evidence. Subversion commit messages are not
enough, we need a bug report or a mailing list discussion.
sure
5. Existing entries are presumed valid unless and until they cause a
false "ham" classification of spam which can be shared publicly in a
useful form.
I guess, or if someone makes an argument that they aren't right.
6. New entries must pass prolonged RuleQA testing of sender-specific
rules before being added to the default welcomelist.
I don't follow this. Do you mean add 'def_welcomelist_dkim foo@bar' to
a testing ruleset and see if it's ok? That seems fine if so. If not, I
didn't follow you.
It might also make sense for each welcomelist rule to have a score.
Basically to bring the mail down to -2, to give it some headroom. But
that might be too complicated compared to benefit.
One pesky detail still exists. There is a very broad fuzzy area where my spam is
your ham and vice versa. You could probably drive yourself to an early grave
trying to get the perfect Bayes training plus perfect rule set.
{o.o} Joanne
Re: Defining what the default welcomelist means
Also, I'm not sure you said this, but I would say: default whitelist is dkim only This means All existing entries are converted to dkim as well as we can, not worrying if they break. We'll prune ones that don't work as dkim, and add a signing domain as we figure it out, as a lightweight thing. But all non-dkim entries go away. to consider a new entry, it must be dkim or maybe that's already true
Re: Defining what the default welcomelist means
I see it very slightly differently, but mostly agree Bill Cole writes: > 1. We serve our users: receivers, not senders. Senders claiming FPs > need the support of a corroborating would-be receiver. Agreed. Or maybe we take requests to add only from receivers. > 2. If senders have FPs on objectively legitimate mail, their first and > most important step is to identify WHY SpamAssassin thinks it is > spam. and address that. Do you need the invisible text? Is the message > embedded in a remotely-fetched image? The sea of "" entities in > your messages' HTML serves what purpose exactly? If there's a real FP > problem with some rule that regularly is proved out by RuleQA, open a > bug. Sure, but if you serve receivers, often people will have misfiling and the sender is opaque, even if not spam and dkim. So saying the sender should fix is misaligned with serving receivers. Yes, they *should*, but people shouldn't send html mail either :-) I agree that requests from senders should be met with "make your mail less spammy". > 3. This is NOT a general-purpose reputation list. It exists to aid SA > users who have FPs from SpamAssassin default rules for wanted mail, > where we cannot determine any acceptable adjustment to rules which > would avoid the problem. It is a "last resort" form of FP mitigation > when we cannot find an acceptable general solution that isn't > domain-specific to a widely accepted sender domain. I see all spam classification as probabalistic and there is risk of FP. If a domain emits *only ham* and is dkim signed, and we believe that receivers want it, I think it makes sense to have it in. I think of things like alerts from banks, airline saying your flight time has changed, etc. where FPs are a real problem. I am extremely skeptical of anything that smells of email marketing here. I would expect only places sending transactional mail and alerts to established customers. > 4. We should only add or remove listings based on specific requests > backed by transparent evidence. Subversion commit messages are not > enough, we need a bug report or a mailing list discussion. sure > 5. Existing entries are presumed valid unless and until they cause a > false "ham" classification of spam which can be shared publicly in a > useful form. I guess, or if someone makes an argument that they aren't right. > 6. New entries must pass prolonged RuleQA testing of sender-specific > rules before being added to the default welcomelist. I don't follow this. Do you mean add 'def_welcomelist_dkim foo@bar' to a testing ruleset and see if it's ok? That seems fine if so. If not, I didn't follow you. It might also make sense for each welcomelist rule to have a score. Basically to bring the mail down to -2, to give it some headroom. But that might be too complicated compared to benefit.
Re: problems with Plugin::ASN and spam
> On Apr 11, 2024, at 5:51 PM, Darrell Budic wrote: > > On Apr 11, 2024, at 3:30 PM, Bill Cole > wrote: >> >> On 2024-04-10 at 21:19:48 UTC-0400 (Wed, 10 Apr 2024 20:19:48 -0500) >> Darrell Budic mailto:bu...@onholyground.com>> >> is rumored to have said: >> On Apr 10, 2024, at 2:52 PM, Benny Pedersen wrote: Darrell Budic skrev den 2024-04-10 19:48: > Anything I’m missing? using amavisd ? then try this in amavisd.conf: >>> >>> No, I”m using spamass-milter to send it over from postfix. Here’s my >>> spamass-milter config in case I missed something there (systemd running it >>> on alma 8 in this case): >>> >>> EXTRA_FLAGS="-e onholyground.com -u defang -m -r 15 -i 127.0.0.1 -g sa-milt >>> -- --max-size=512 >>> --dest=sa0.int.ohgnetworks.com,sa1.int.ohgnetworks.com —randomize" >> Found it, even with the -m, spamass-milter only replaces a hardcoded set of X-Spam-* headers, not anything that comes back from spamd. With some more work, I was able to confirm that spamc/spamd were indeed including the headers where they were supposed to be. Thanks for the help tracking it down, I’m going to reconsider my preference for milters here ;)
Re: Dynamic blacklist ?
On 2024-04-12 at 02:14:59 UTC-0400 (Fri, 12 Apr 2024 08:14:59 +0200) Pierluigi Frullani is rumored to have said: Hello all, do you know if there is a way to have a blacklist, either for user or eventually for an entire server, that could be feeded via some scripts ? If you enable the AWL (or TxRep, if you are adventurous) Plugin, it provides an automated welcome/blocklist mechanism where the past base score of messages are used to adjust the score of later messages from the same sender and network block tuple. Its power can be adjusted and its usage is described in the documentation. The same database is used for the blocklist and welcomelist options of the spamassassin command-line script, which is documented in the 'spamassassin-run' man page. There is also a useful script named sa-awl with a fine man page. A sort of auto_learn but only for addresses ( to or from ) ? Correct: auto_learn and the sa-learn commandline script feed whole messages to a complex naive Bayesian analysis that feeds the Bayes DB. The 'auto_learn' config, the *list options for spamassassin, and sa-awl all operate on the AWL DB using a very simple algorithm. Unlike the Bayes subsystem, the AWL subsystem has no minimum data threshold. If you feed one message to 'spamassassin -W' then the next message from the same sender+network combination will have its score adjusted according to your auto_welcomelist_factor setting, as documented in 'perldoc Mail::SpamAssassin::Plugin::AWL' along with all the other details of AWL. I'll trying to explain: I maintain a couple of mail servers that have a very very limited e-mails volumes, at least in output, so the bayes it's almost not usefull as it takes ages to be feeded for the HAM part. At the moment I'm taking addresses from the spam directory and feeding to local.cf but it's a slow ( and painfull ) process so if there is a better way it would be fantastic. I guess this is the short version of an answer... If you have AWL enabled and configured so that everyone uses the same AWL DB, you could do this if you have a directory full of fresh spam whose senders you want to shun: cd $spamdirectory spamassassin --add-to-blocklist * And if you have a bunch of mail you value in a directory, use "-W" instead. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire
Re: WARNING: Microsoft has earned removal from SA default welcomelist
On 4/12/2024 1:20 PM, Bill Cole wrote: In my opinion, this is an indication that the default welcomelist entries in the official SpamAssassin rules for '*@*.microsoft.com' are inappropriate. Note that there is an entry for '*@accountprotection.microsoft.com' which is still justified as far as I know. This is entirely unrelated to any domains hosted by Microsoft, it is strictly an email address welcomelisting (see SA docs for details.) +1 This may raise some questions and trigger a debate on the formal meaning of the SA default welcomelist entries. That debate belongs on the SpamAssassin Users List, but may pop up elsewhere. I believe that we have left a gap there in having a quite vague definition of what default welcomelist entries represent. As far as I know, clear criteria for inclusion have never been promulgated and accepted by the PMC or the user community. +1
Defining what the default welcomelist means
The de-welcomelisting of MS marketing raises the question: Why do we maintain a "default" welcomelist? Based on the documentation, the original purpose of the def_welcomelist* (then whitelist) feature set was to give a set of senders of purely legitimate mail from FPs, with a listing having reduced power relative to normal welcomelist entires *because they were widely phished by spammers*. This was before sender authentication (SPF & DKIM) were in broad use and before authentication-empowered welcomelist features existed. Having these senders weakly protected in SA was a preventative measure to keep frustrated admins (or poisoned Bayes DBs) from rashly overprotecting them and all the phishes or blocklisting them. Today there's much less risk in the welcomelist features because we recommend and use the authenticated forms, which means that if you like, you can use SA WL/BL rules to demand only authenticated mail from some senders. In that context, the purpose of the default welcomelist has wandered. This explains to some degree the lack of clear relevance and meaning of listings. It is a remnant feature that has outlived its original justification. The original list included big names of respected companies that "everyone" (i.e. as warped by SA committer and vocal user non-diversity...) got occasional important mail from and would never want to block mail from. It has drifted into being a list of "good guys" who, based on our committers' experiences, get FPs that they do not deserve. We have drifted perilously close to being a maintainer of a low-visibility free reputation service with lax oversight. We also come near that peril in our explicit lists of TLDs which are objectively dominated by spammers, but in that case I think we have that risk contained because we have a methodology for validating TLD inclusion and removal: testing single-TLD rules in QA. The default welcomelist is unconfined, because we don't have a clear explicit standard or even a formal transparent mechanism for inclusion and removal. My understanding of some listings is that they were based on one mid-sized site's FPs from their wanted mailstream dominated by one-to-few "personal" B2B email. That is very hard to validate, and at this point the list is too big to trim it back to just the original concept, especially since that concept no longer has much real-world use. It needs more structure to keep it from becoming just "friends, employers, and extorters of SA committers" or being perceived as such. I believe that part of a way to avoid that is an absolute zero-tolerance policy for spam from listees. We cannot support any standard that gets us bogged down into debates with senders over whether their spam is enough spam to justify risking the FPs they would get without a listing, because we cannot measure that. We cannot be subservient to sender business models that require them to take shortcuts in assuring that they do not ever send spam. We must not be telling our users that they should just eat their spam because some sender doesn't want to spend on confirming users and seems to have a working unsub link. I ultimately want to document how we add and remove listings and what users should expect from the default welcomelist. I think some important elements are: 1. We serve our users: receivers, not senders. Senders claiming FPs need the support of a corroborating would-be receiver. 2. If senders have FPs on objectively legitimate mail, their first and most important step is to identify WHY SpamAssassin thinks it is spam. and address that. Do you need the invisible text? Is the message embedded in a remotely-fetched image? The sea of "" entities in your messages' HTML serves what purpose exactly? If there's a real FP problem with some rule that regularly is proved out by RuleQA, open a bug. 3. This is NOT a general-purpose reputation list. It exists to aid SA users who have FPs from SpamAssassin default rules for wanted mail, where we cannot determine any acceptable adjustment to rules which would avoid the problem. It is a "last resort" form of FP mitigation when we cannot find an acceptable general solution that isn't domain-specific to a widely accepted sender domain. 4. We should only add or remove listings based on specific requests backed by transparent evidence. Subversion commit messages are not enough, we need a bug report or a mailing list discussion. 5. Existing entries are presumed valid unless and until they cause a false "ham" classification of spam which can be shared publicly in a useful form. 6. New entries must pass prolonged RuleQA testing of sender-specific rules before being added to the default welcomelist. As with everything SpamAssassin: input from users and other contributors is eagerly desired..., -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For
WARNING: Microsoft has earned removal from SA default welcomelist
Yesterday I received marketing spam from "Microsoft " advertising something apparently called "Microsoft Build" which is either a website or a marketing event: IDGAF. Spam was sent via Marketo, which I gather is now part of the sewer we call Adobe. It was absolutely authentic. Fully authentic Microsoft spam passing SPF, DKIM, and DMARC. That spam was sent to my oldest and most widely scraped address (b...@scconsult.com) which I've literally never given to anyone for subscribing to or purchasing anything and which I am 100% certain I've never given to Microsoft in any way intentionally. There is no indication in the spam of any associated MS account. My comprehensive 29yr archive of all email ever received by that address has NO prior mail from MS. There was an unsub link, which got a page which revealed that I was somehow subscribed to multiple marketing bullshit lists. That page offered me a link to my "profile"(!?) which seemed to start to want to load up a page with an image and text placeholder blobs pulsing a bit before switching to a generic Microsoft account signup/login page. MS knew what my email address was and had me subscribed to multiple lists in some sort of "profile" without even asking me and without associating it to any actual MS account that I could conceivably access. I do have multiple MS accounts that I need for work purposes, and one I use for testing, but none of those are associated with b...@scconsult.com (except as a correspondent.) In my opinion, this is an indication that the default welcomelist entries in the official SpamAssassin rules for '*@*.microsoft.com' are inappropriate. Note that there is an entry for '*@accountprotection.microsoft.com' which is still justified as far as I know. This is entirely unrelated to any domains hosted by Microsoft, it is strictly an email address welcomelisting (see SA docs for details.) I will be committing the rule change today and it should appear in the default rules distribution channel by Monday. Anyone who is relying on that SA welcomelisting to accept wanted mail from MS should do so locally based on the specific local needs. I will also document this in a bug report, which I will resolve, to have a record of when and why this was done. This may raise some questions and trigger a debate on the formal meaning of the SA default welcomelist entries. That debate belongs on the SpamAssassin Users List, but may pop up elsewhere. I believe that we have left a gap there in having a quite vague definition of what default welcomelist entries represent. As far as I know, clear criteria for inclusion have never been promulgated and accepted by the PMC or the user community. More to follow in a separate thread. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire
Re: another problem in disable in spamassassin
On 12.04.24 10:50, natan wrote:
I have problem with disabled spamhaus.org in spamassassin:
In local.cf I disable check like:
...
dns_query_restriction deny spamhaus.org
dns_query_restriction deny zen.spamhaus.org
dns_query_restriction deny dbl.spamhaus.org
But in mail.log I fund still checking RCVD_IN_PBL, URIBL_CSS_A,
URIBL_DBL_SPAM
mail.log
Apr 12 06:04:48 amavis5 amavis[3060074]: (3060074-10) spam-tag,
-> , Yes, score=26.884
tagged_above=3.6 required=6 tests=[AM.IP_BAD_62.133.61.198=1.8,
BAYES_50=0.8, DCC_CHECK=4, DCC_REPUT_99_100=1.4, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
HTML_MESSAGE=0.001, MIME_HTML_ONLY=0.1, RCVD_IN_MSPIKE_BL=0.001,
RCVD_IN_MSPIKE_ZBI=0.001, RCVD_IN_PBL=3.335, RCVD_IN_SBL_CSS=3.335,
RELAYCOUNTRY_BAD=2, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001,
URIBL_BLOCKED=0.2, URIBL_CSS_A=0.1, URIBL_DBL_SPAM=10] autolearn=no
autolearn_force=no
did you reload amavis after changing local.cf?
do they appear when you feed mail in to "spamassassin" commandline client?
It still can be amavis issue.
in /var/lib/spamassassin/3.004006/updates_spamassassin_org/20_dnsbl_tests.cf
...
# PBL is the Policy Block List: https://www.spamhaus.org/pbl/
header RCVD_IN_PBL eval:check_rbl('zen-lastexternal',
'zen.spamhaus.org.', '^127\.0\.0\.1[01]$')
describe RCVD_IN_PBL Received via a relay in Spamhaus PBL
tflags RCVD_IN_PBL net
reuse RCVD_IN_PBL
...
in /var/lib/spamassassin/3.004006/updates_spamassassin_org/25_uribl.cf
...
25_uribl.cf:urirhssub URIBL_DBL_SPAM dbl.spamhaus.org. A
127.0.1.2
25_uribl.cf:body URIBL_DBL_SPAM
eval:check_uridnsbl('URIBL_DBL_SPAM')
25_uribl.cf:describe URIBL_DBL_SPAM Contains a spam URL
listed in the Spamhaus DBL blocklist
25_uribl.cf:tflags URIBL_DBL_SPAM net domains_only notrim
25_uribl.cf:reuse URIBL_DBL_SPAM
...
And I dont have idea how disable all check in spamhaus.org
--
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Spam = (S)tupid (P)eople's (A)dvertising (M)ethod
another problem in disable in spamassassin
Hi
I have problem with disabled spamhaus.org in spamassassin:
In local.cf I disable check like:
...
dns_query_restriction deny spamhaus.org
dns_query_restriction deny zen.spamhaus.org
dns_query_restriction deny dbl.spamhaus.org
...
But in mail.log I fund still checking RCVD_IN_PBL, URIBL_CSS_A,
URIBL_DBL_SPAM
mail.log
Apr 12 06:04:48 amavis5 amavis[3060074]: (3060074-10) spam-tag,
-> , Yes, score=26.884
tagged_above=3.6 required=6 tests=[AM.IP_BAD_62.133.61.198=1.8,
BAYES_50=0.8, DCC_CHECK=4, DCC_REPUT_99_100=1.4, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
HTML_MESSAGE=0.001, MIME_HTML_ONLY=0.1, RCVD_IN_MSPIKE_BL=0.001,
RCVD_IN_MSPIKE_ZBI=0.001, RCVD_IN_PBL=3.335, RCVD_IN_SBL_CSS=3.335,
RELAYCOUNTRY_BAD=2, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001,
URIBL_BLOCKED=0.2, URIBL_CSS_A=0.1, URIBL_DBL_SPAM=10] autolearn=no
autolearn_force=no
in /var/lib/spamassassin/3.004006/updates_spamassassin_org/20_dnsbl_tests.cf
...
# PBL is the Policy Block List: https://www.spamhaus.org/pbl/
header RCVD_IN_PBL eval:check_rbl('zen-lastexternal',
'zen.spamhaus.org.', '^127\.0\.0\.1[01]$')
describe RCVD_IN_PBL Received via a relay in Spamhaus PBL
tflags RCVD_IN_PBL net
reuse RCVD_IN_PBL
...
in /var/lib/spamassassin/3.004006/updates_spamassassin_org/25_uribl.cf
...
25_uribl.cf:urirhssub URIBL_DBL_SPAM dbl.spamhaus.org. A
127.0.1.2
25_uribl.cf:body URIBL_DBL_SPAM
eval:check_uridnsbl('URIBL_DBL_SPAM')
25_uribl.cf:describe URIBL_DBL_SPAM Contains a spam URL
listed in the Spamhaus DBL blocklist
25_uribl.cf:tflags URIBL_DBL_SPAM net domains_only notrim
25_uribl.cf:reuse URIBL_DBL_SPAM
...
And I dont have idea how disable all check in spamhaus.org
--
RE: Dynamic blacklist ?
> do you know if there is a way to have a blacklist, either for user or > eventually for an entire server, that could be feeded via some scripts ? Yes create your own dns blacklist > A sort of auto_learn but only for addresses ( to or from ) ? No such thing as only for... You have to implement multiple things and try to catch the default crap at an early stage by something that does not consume much resources. I have have my from= and/or to= dnsbl at the end of all checks Currently I am marking spam at server level and individual users can white list messages marked as spam. So next time when a marked message arrives, it is unmarked again. But this is not done via dnsbl but with a local user whitelist db with sieve.
Dynamic blacklist ?
Hello all, do you know if there is a way to have a blacklist, either for user or eventually for an entire server, that could be feeded via some scripts ? A sort of auto_learn but only for addresses ( to or from ) ? I'll trying to explain: I maintain a couple of mail servers that have a very very limited e-mails volumes, at least in output, so the bayes it's almost not usefull as it takes ages to be feeded for the HAM part. At the moment I'm taking addresses from the spam directory and feeding to local.cf but it's a slow ( and painfull ) process so if there is a better way it would be fantastic. Thanks in advance. Pierluigi
