whitelist_from_rcvd not working, WAIDW
Header of test message, massaged for privacy, is here: http://pastebin.com/EV6g15aN I have this in user_prefs: trusted_networks 198.1.2.3/32 [...lots snipped...] whitelist_from_rcvd *@wetransfer.com *.wetransfer.com Why is the whitelist not firing? -- Please *no* private copies of mailing list or newsgroup messages. Rule 420: All persons more than eight miles high to leave the court. Local Variables: mode:claws-external End:
Re: Lots of Polish spam
On 25 Feb 2015, at 17:15, Yves Goergen wrote: Am 25.02.2015 um 20:42 schrieb Bill Cole: On 24 Feb 2015, at 17:06, Yves Goergen wrote: I can't block all archives with executable files in them. Then in all seriousness: why bother filtering email specifically for malware? Email is an inherently untrustworthy transport medium. Any sort of executable received via email that is not cryptographically signed by a trusted sender should be considered unsafe to run. If an executable is signed by a trusted sender, it can just as easily be encrypted to protect it from detection as an executable. If your users believe that you are providing them a valuableservice by allowing transport of executables via email, they are mistaken. You are putting them at unnecessary risk. I fully understand you, but tell that end users. I do. Based on my employer's logs and support requests, the frequency of actual user problems with an absolute omnidirectional ban on readily identified executables attached to email is at least 3 orders of magnitude smaller than the frequency of that ban excluding malware in the past 14 months. It is quite likely that some of our users have adopted mechanisms of evading the blockage and informed their correspondents of those mechanisms, which is a relatively low-risk issue -- a problem not worth trying to solve. They're already happy if they manage to get an e-mail with an attached file sent out. I've more than once thought about shutting down the FTP service due to repeated issues with it, requiring that users manage their files through SFTP. But FTP is still the most-used access protocol and the average webmaster(!) doesn't care or know about it all. Yes, I understand that a solid 50% of the human race consists of people with below-median intelligence. That's always been necessary to take into account and it is a persuasive reason to avoid targeting a "mass" market of users. Put another way: a customer who demands FTP instead of SFTP for anything other than anonymous downloading is too dumb to be worth serving. Your objection also applies to unencrypted HTTP downloads, BTW. Yes and no. No one is sent dozens of unsolicited malicious executables daily via unencrypted HTTP, mixed in with a handful of legitimate and possibly important messages that they are expected to see and respond to. A user seeking out a piece of software and transporting it in an insecure fashion is potentially problematic, but it is ultimately a consensual problem that is mitigated by things like file encryption and/or simple hash "fingerprints" to assure that receivers get the files senders believe they are sending. Whether receivers are good judges of sender integrity is a tougher problem, not readily solved by technical measures.
Re: Bayes expiration with Redis backend
Thanks a lot for the explanation Mark, it was very clear.
It would be a good idea considering to add that to the
perldoc of the BayesStore/Redis.pm module.
Regards,
Matteo
On 27.02.2015 14:55, Mark Martinec wrote:
When redis automatically expires tokens internally based on their
TTL, this operation does not affect nspam and nham counts. These
counts just grow all the time (as there is no explicit expiration
that SpamAssassin would know about), reflecting the count of
(auto)learning operations.
Don't worry about large nspam and/or nham counts when redis is
in use, all that matters is that these counts are above 200
(otherwise bayes is disabled).
You may get the number of tokens that are actually in the redis
database (not expired) by counting the number of lines produced
on stdout by 'sa-learn --backup' or 'sa-learn --dump data'.
The format of fields produced by --dump data is:
probability spam_count ham_count atime token
The --backup format is similar, but does not provide
probabilities, just spam and ham counts.
To get some estimate on the number of hammy vs. spammy tokens
(not messages) currently in a database, try something like:
sa-learn --dump data' | \
awk '$1<0.1 {h++}; $1>0.9 {s++}; END{printf("h=%d, s=%d\n",h,s)}'
(caveat: sa-learn --backup or --dump data may not work on a huge
database, as they need all the tokens (redis keys) to fit into memory)
Mark
Re: Bayes expiration with Redis backend
On 27.02.2015 13:54, Axb wrote: Is it possible you reject so much spam tha SA sees very little spam? I believe it's the case. A combination of Postfix policies, blacklists, ClamAV plus additional signatures, etc. greatly reduces the amount of email sent through the filtering pipeline. If classification makes sense, you're not getting FPs or tons of spam with BAYES_00 you have nothing to worry about. Sometimes you have to trust your gut feeling... don't let wild bayes theories misguide you...watch your logs, listen to your users and build up a feel for it. I will certainly do that. So far the feedback from our users is quite good and so I can say I am happy with our current setup. Regards, Matteo
whitelist_from_spf
I was preparing to ask about an error message I was getting when I started up spamd: spamd[18145]: config: failed to parse line, skipping, in "/usr/local/etc/mail/spamassassin/whitelist.cf": whitelist_from_spf *@covisp.net *@kreme.com But I fixed the problem. It turns out that the SPF plugin was not loaded in init.pre #loadplugin Mail::SpamAssassin::Plugin::SPF So, hopefully this message might help someone in the future with this same issue. -- Once upon a time the plural of 'wizard' was 'war'. --The Last Continent
Re: Bayes expiration with Redis backend
Matteo Dessalvi wrote:
I am using a centralized Redis instance to
host the bayesian data for a bunch of MTAs.
AFAICS the SA filter is working quite well
and the BAYES_* rules are triggered correctly,
no false positive so far.
But I am concerned about the expiration of the
bayesian data. sa-learn reports the following:
0.000 0 3 0 non-token data: bayes db
version
0.000 0 8437 0 non-token data: nspam
0.000 0 495000 0 non-token data: nham
As stated here:
search.cpan.org/dist/Mail-SpamAssassin/lib/Mail/SpamAssassin/BayesStore/Redis.pm
"Expiry is done internally in Redis using *_ttl settings (...)
This is why --force-expire etc does nothing, and token counts
and atime values are shown as zero in statistics."
So, why the nham tokens have grown so much? It looks
like it was never 'pruned'.
When redis automatically expires tokens internally based on their
TTL, this operation does not affect nspam and nham counts. These
counts just grow all the time (as there is no explicit expiration
that SpamAssassin would know about), reflecting the count of
(auto)learning operations.
Don't worry about large nspam and/or nham counts when redis is
in use, all that matters is that these counts are above 200
(otherwise bayes is disabled).
You may get the number of tokens that are actually in the redis
database (not expired) by counting the number of lines produced
on stdout by 'sa-learn --backup' or 'sa-learn --dump data'.
The format of fields produced by --dump data is:
probability spam_count ham_count atime token
The --backup format is similar, but does not provide
probabilities, just spam and ham counts.
To get some estimate on the number of hammy vs. spammy tokens
(not messages) currently in a database, try something like:
sa-learn --dump data' | \
awk '$1<0.1 {h++}; $1>0.9 {s++}; END{printf("h=%d, s=%d\n",h,s)}'
(caveat: sa-learn --backup or --dump data may not work on a huge
database, as they need all the tokens (redis keys) to fit into memory)
Mark
Re: Bayes expiration with Redis backend
On 02/27/2015 01:38 PM, Matteo Dessalvi wrote: Hi all. I am using a centralized Redis instance to host the bayesian data for a bunch of MTAs. AFAICS the SA filter is working quite well and the BAYES_* rules are triggered correctly, no false positive so far. But I am concerned about the expiration of the bayesian data. sa-learn reports the following: 0.000 0 3 0 non-token data: bayes db version 0.000 0 8437 0 non-token data: nspam 0.000 0 495000 0 non-token data: nham As stated here: search.cpan.org/dist/Mail-SpamAssassin/lib/Mail/SpamAssassin/BayesStore/Redis.pm "Expiry is done internally in Redis using *_ttl settings (...) This is why --force-expire etc does nothing, and token counts and atime values are shown as zero in statistics." So, why the nham tokens have grown so much? It looks like it was never 'pruned'. Is it possible you reject so much spam tha SA sees very little spam? I am using the following configuration for the expiration: bayes_token_ttl 21d bayes_seen_ttl 8d bayes_auto_expire 1 I have also left bayes_expiry_max_db_size undefined. You can remove that entry - Redis doesn't use it. My other concern is about the proportion between spam and ham tokens. Should I be worried about it? If classification makes sense, you're not getting FPs or tons of spam with BAYES_00 you have nothing to worry about. Sometimes you have to trust your gut feeling... don't let wild bayes theories misguide you...watch your logs, listen to your users and build up a feel for it. Your traffic is NEVER similar to anybody else's. If you want to manually train some missed spam, do it, there's lots of methods but without knowing what your setup looks like it's guesswork. The list archives are full of valuable infos and howtos h2h Axb
Bayes expiration with Redis backend
Hi all. I am using a centralized Redis instance to host the bayesian data for a bunch of MTAs. AFAICS the SA filter is working quite well and the BAYES_* rules are triggered correctly, no false positive so far. But I am concerned about the expiration of the bayesian data. sa-learn reports the following: 0.000 0 3 0 non-token data: bayes db version 0.000 0 8437 0 non-token data: nspam 0.000 0 495000 0 non-token data: nham As stated here: search.cpan.org/dist/Mail-SpamAssassin/lib/Mail/SpamAssassin/BayesStore/Redis.pm "Expiry is done internally in Redis using *_ttl settings (...) This is why --force-expire etc does nothing, and token counts and atime values are shown as zero in statistics." So, why the nham tokens have grown so much? It looks like it was never 'pruned'. I am using the following configuration for the expiration: bayes_token_ttl 21d bayes_seen_ttl 8d bayes_auto_expire 1 I have also left bayes_expiry_max_db_size undefined. My other concern is about the proportion between spam and ham tokens. Should I be worried about it? Thanks in advance! Regards, Matteo
