Re: Amavisd and Bayes (again...)

[Benny Pedersen]

Zhang Huangbin skrev den 2016-11-25 00:20:


Do you have any idea/suggestion/experience how to share/update/sync SA
bayes across multiple mail servers without SQL? e.g. a mail cluster.


redis, make it clusted

do you really need it ?

redis is not that stable in gentoo

spamassassin redis plugin does not support clustered redis yet imho

so its in vein imho

https://github.com/rqlite/rqlite
http://stackoverflow.com/questions/3554502/1-can-sqlite-use-cluster-2-does-sqlite-have-anything-like-mysql-workbench-3
https://www.npmjs.com/package/sqlite3-cluster

Re: Amavisd and Bayes (again...)

[Benny Pedersen]

Marc Stürmer skrev den 2016-11-24 22:18:

Am 24.11.2016 um 22:04 schrieb Alexander Masidlover:

Thanks, I had been wondering about moving to SQL storage as we get a 
reasonably large volume of email
through; however, I’m a bit concerned about doing all the set-up and 
still getting the same result!


One serious advise: don't. Spamassassin's Bayes storage in SQL is slow
as swimming through molasses, it's no fun at all.


spamassassin is opensource feel free to make a bayes sql patch that 
optimize sql bayes, i use sql bayes here on ssd drive without any issue


i just hate to see rspamd is more stable now then spamassassin in most 
areas, its sadly true, only point where spamassassin still wins is its 
simple to configure to make what one wants, but its does of the cost of 
missing plugins to cover spam runs as 2016 is now


i like to see more fokus on rbl repution local, and url repution local, 
free software should not help datafeeds to get paided for data


think about it

Re: Amavisd and Bayes (again...)

[Zhang Huangbin]

> On Nov 25, 2016, at 5:18 AM, Marc Stürmer  wrote:
> 
> One serious advise: don't. Spamassassin's Bayes storage in SQL is slow as 
> swimming through molasses, it's no fun at all.

Hi Marc,

Do you have any idea/suggestion/experience how to share/update/sync SA bayes 
across multiple mail servers without SQL? e.g. a mail cluster.


Zhang Huangbin, founder of iRedMail project: http://www.iredmail.org/
Time zone: GMT+8 (China/Beijing).

Re: Amavisd and Bayes (again...)

[Marc Stürmer]

Am 24.11.2016 um 22:04 schrieb Alexander Masidlover:


Thanks, I had been wondering about moving to SQL storage as we get a reasonably 
large volume of email through; however, I’m a bit concerned about doing all the 
set-up and still getting the same result!


One serious advise: don't. Spamassassin's Bayes storage in SQL is slow 
as swimming through molasses, it's no fun at all.




smime.p7s
Description: S/MIME Cryptographic Signature

Re: Amavisd and Bayes (again...)

[Alexander Masidlover]

So, in doing the first part of the below (adding use_bayes_rules) I spotted 
that in the debugs there was a tainting error in an eval of Spamassassin Logger 
module shortly after the bayes module initialised.

I removed -T from the beginning of amavisd and now I’m getting BAYES headers 
when amavisd runs spamassassin…

When I have a bit more time I’ll revert my change and try and track down what’s 
going on with the taint error since its less than ideal from a security 
perspective to just disable tainting!

> On 24 Nov 2016, at 16:09, Dino Edwards  wrote:
> 
> Why don't you do the following.:
>  
> Edit your SA local.cf file and make sure the following lines are in it. NOTE 
> the bayes_path, set that to a directory of your choice. Please also note that 
> the last bayes of that path is NOT a directory but it’s simply the prefix of 
> that files in that directory (bayes_journal, bayes_seen, bayes_toks) but it 
> has to be set that way in local.cf in order for this to work. So, if you were 
> to use the path in my example, you would simply create the directory 
> /opt/sa-bayes/
>  
> #bayes
> bayes_path /opt/sa-bayes/bayes
> bayes_file_mode 0777
> use_bayes 1
> use_bayes_rules 1
> bayes_auto_learn 0
>  
>  
> What I would do next, is take the bayes files from your current bayes 
> directory and move them to that new directory you created. The new directory 
> should look like below. NOTE the amavis user is the owner of that directory, 
> you usually do that with chown -R amavis:amavis /opt/sa-bayes/
>  
> pwd
> /opt/sa-bayes
>  
> ---
>  
> ls -l
> total 4664
> -rw-rw-rw- 1 amavis amavis   36216 Nov 24 11:00 bayes_journal
> -rw-rw-rw- 1 amavis amavis  651264 Nov 16 09:15 bayes_seen
> -rw-rw-rw- 1 amavis amavis 5197824 Nov 24 10:33 bayes_toks
> -rw-r--r-- 1 amavis amavis1869 Oct 30  2014 user_prefs
>  
> Hope it helps
>  
> Thanks
>  
>  
> --
>  
> Hermes Secure Email Gateway
> Hermes Secure Email Gateway combines Open Source technologies such as 
> Postfix, Apache SpamAssassin, ClamAV, Amavisd-new, MySQL and CipherMail under 
> one unified web based Web GUI for easy administration and management of your 
> incoming and ougoing email for your organization. Anti-spam, anti-virus and 
> anti-malware protection, encrypted S/MIME, encrypted PDF and SMTP TLS 
> support, built-in email archiving, end-user self-service web gui.
>  
> Download the free open-source appliance at:
> http://www.deeztek.com/hermes-secure-email-gateway/ 
> <http://www.deeztek.com/hermes-secure-email-gateway/>
>  
>  
> > -Original Message-
> > From: amavis-users [mailto:amavis-users-
> > bounces+dino.edwards=mydirectmail@amavis.org 
> > <mailto:bounces+dino.edwards=mydirectmail@amavis.org>] On Behalf Of Alex
> > Masidlover
> > Sent: Thursday, November 24, 2016 4:09 AM
> > To: amavis-users@amavis.org <mailto:amavis-users@amavis.org>
> > Subject: Amavisd and Bayes (again...)
> >
> > Hi,
> >
> > I'm currently being deluged with spam and have been trying to use BAYES
> > filters to try and get rid of some of it. I've made a lot of progress but 
> > am now
> > very stuck.
> >
> > I have go to the point where I have (temporarily) given the amavis user a
> > shell and when I run spamassassin on an email from the command line I
> > get:
> >
> > spamassassin -t  >
> > Content analysis details:   (7.4 points, 5.0 required)
> >
> >  pts rule name  description
> >  -- -
> > -
> >  3.5 BAYES_99   BODY: Bayes spam probability is 99 to 100%
> > [score: 1.]
> >  1.8 REMOVE_BEFORE_LINK BODY: Removal phrase right before a link
> >  0.2 BAYES_999  BODY: Bayes spam probability is 99.9 to 100%
> > [score: 1.]
> >  0.0 HTML_MESSAGE   BODY: HTML included in message
> >  1.1 DCC_CHECK  Detected as bulk mail by DCC (dcc-
> > servers.net)
> >  0.8 RDNS_NONE  Delivered to internal network by a host with no 
> > rDNS
> >
> > The debugs show:
> >
> > [Tue Nov 22 16:12:01] amavis@mta0 ~ $ spamassassin -D -t
> > &1 | grep -i bayes Nov 22 16:12:10.355 [10336] dbg:
> > plugin: loading Mail::SpamAssassin::Plugin::Bayes from @INC Nov 22
> > 16:12:10.603 [10336] dbg: config: fixed relative path:
> > /var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf
> > Nov 22 16:12:10.603 [10336] dbg: config: using
> > "/var/lib/spamassass

Re: Amavisd and Bayes (again...)

[Alexander Masidlover]

> On 24 Nov 2016, at 14:41, Zhang Huangbin  wrote:
> 
> 
>> On Nov 24, 2016, at 10:09 PM, Dominic Raferd  wrote:
>> 
>> Maybe this can help: 
>> https://www.nerd-quickies.net/2015/10/02/spamassassin-bayes_00-1-90-although-sa-learn-runs-daily/
> 
> Also this one:
> http://www.iredmail.org/docs/store.spamassassin.bayes.in.sql.html
> 

Thanks, I had been wondering about moving to SQL storage as we get a reasonably 
large volume of email through; however, I’m a bit concerned about doing all the 
set-up and still getting the same result!

The thing that’s throwing me is that as far as I can tell its actually managing 
to write to the bayes tokens files when it learns spam, but doesn’t seem to be 
using them in spam checking (apart from in standalone mode).

Re: Amavisd and Bayes (again...)

[Alexander Masidlover]

Thanks, but sadly not; the debugs show its definitely opening the same set of 
files.

> On 24 Nov 2016, at 14:09, Dominic Raferd  wrote:
> 
> Maybe this can help: 
> https://www.nerd-quickies.net/2015/10/02/spamassassin-bayes_00-1-90-although-sa-learn-runs-daily/
>  
> 
> 
> 
> On 24 November 2016 at 13:43, @lbutlr  > wrote:
> On Nov 24, 2016, at 2:09 AM, Alex Masidlover  > wrote:
> > Any help would be appreciated before I drown in spam...
> 
> I’ve posted a couple of times about the same exact issue with amavisnot using 
> bases and so far no one has been able to provide any guidance as to why.
> 
> For now, I’ve given up. AMavis still catches a lot os spam without bases, and 
> then I manually run received mail through SA again. It’s ugly, but it works.
> 
> 

RE: Amavisd and Bayes (again...)

[Dino Edwards]

Why don't you do the following.:



Edit your SA local.cf file and make sure the following lines are in it. NOTE 
the bayes_path, set that to a directory of your choice. Please also note that 
the last bayes of that path is NOT a directory but it's simply the prefix of 
that files in that directory (bayes_journal, bayes_seen, bayes_toks) but it has 
to be set that way in local.cf in order for this to work. So, if you were to 
use the path in my example, you would simply create the directory /opt/sa-bayes/



#bayes

bayes_path /opt/sa-bayes/bayes

bayes_file_mode 0777

use_bayes 1

use_bayes_rules 1

bayes_auto_learn 0





What I would do next, is take the bayes files from your current bayes directory 
and move them to that new directory you created. The new directory should look 
like below. NOTE the amavis user is the owner of that directory, you usually do 
that with chown -R amavis:amavis /opt/sa-bayes/



pwd

/opt/sa-bayes



---



ls -l

total 4664

-rw-rw-rw- 1 amavis amavis   36216 Nov 24 11:00 bayes_journal

-rw-rw-rw- 1 amavis amavis  651264 Nov 16 09:15 bayes_seen

-rw-rw-rw- 1 amavis amavis 5197824 Nov 24 10:33 bayes_toks

-rw-r--r-- 1 amavis amavis1869 Oct 30  2014 user_prefs



Hope it helps



Thanks





--



Hermes Secure Email Gateway

Hermes Secure Email Gateway combines Open Source technologies such as Postfix, 
Apache SpamAssassin, ClamAV, Amavisd-new, MySQL and CipherMail under one 
unified web based Web GUI for easy administration and management of your 
incoming and ougoing email for your organization. Anti-spam, anti-virus and 
anti-malware protection, encrypted S/MIME, encrypted PDF and SMTP TLS support, 
built-in email archiving, end-user self-service web gui.



Download the free open-source appliance at:

http://www.deeztek.com/hermes-secure-email-gateway/





> -Original Message-

> From: amavis-users [mailto:amavis-users-

> bounces+dino.edwards=mydirectmail@amavis.org] On Behalf Of Alex

> Masidlover

> Sent: Thursday, November 24, 2016 4:09 AM

> To: amavis-users@amavis.org

> Subject: Amavisd and Bayes (again...)

>

> Hi,

>

> I'm currently being deluged with spam and have been trying to use BAYES

> filters to try and get rid of some of it. I've made a lot of progress but am 
> now

> very stuck.

>

> I have go to the point where I have (temporarily) given the amavis user a

> shell and when I run spamassassin on an email from the command line I

> get:

>

> spamassassin -t 

> Content analysis details:   (7.4 points, 5.0 required)

>

>  pts rule name  description

>  -- -

> -

>  3.5 BAYES_99   BODY: Bayes spam probability is 99 to 100%

> [score: 1.]

>  1.8 REMOVE_BEFORE_LINK BODY: Removal phrase right before a link

>  0.2 BAYES_999  BODY: Bayes spam probability is 99.9 to 100%

> [score: 1.]

>  0.0 HTML_MESSAGE   BODY: HTML included in message

>  1.1 DCC_CHECK  Detected as bulk mail by DCC (dcc-

> servers.net)

>  0.8 RDNS_NONE  Delivered to internal network by a host with no 
> rDNS

>

> The debugs show:

>

> [Tue Nov 22 16:12:01] amavis@mta0 ~ $ spamassassin -D -t

> &1 | grep -i bayes Nov 22 16:12:10.355 [10336] dbg:

> plugin: loading Mail::SpamAssassin::Plugin::Bayes from @INC Nov 22

> 16:12:10.603 [10336] dbg: config: fixed relative path:

> /var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf

> Nov 22 16:12:10.603 [10336] dbg: config: using

> "/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf"

> for included file

> Nov 22 16:12:10.603 [10336] dbg: config: read file

> /var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf

> Nov 22 16:12:11.594 [10336] dbg: plugin:

> Mail::SpamAssassin::Plugin::Bayes=HASH(0x27d2868) implements

> 'learner_new', priority 0 Nov 22 16:12:11.595 [10336] dbg: bayes:

> learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x27d2868),

> bayes_store_module=Mail::SpamAssassin::BayesStore::DBM

> Nov 22 16:12:11.609 [10336] dbg: bayes: learner_new: got

> store=Mail::SpamAssassin::BayesStore::DBM=HASH(0x2fa76c8)

> Nov 22 16:12:11.609 [10336] dbg: plugin:

> Mail::SpamAssassin::Plugin::Bayes=HASH(0x27d2868) implements

> 'learner_is_scan_available', priority 0 Nov 22 16:12:11.613 [10336] dbg: 
> bayes:

> tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_toks

> Nov 22 16:12:11.614 [10336] dbg: bayes: tie-ing to DB file R/O

> /var/amavis/.spamassassin/bayes_seen

> Nov 22 16:12:11.614 [10336] dbg: bayes: found bayes db version 3 Nov 22

> 16:12:13.528 [10336] dbg: bayes: untie-ing

Re: Amavisd and Bayes (again...)

[Zhang Huangbin]

> On Nov 24, 2016, at 10:09 PM, Dominic Raferd  wrote:
> 
> Maybe this can help: 
> https://www.nerd-quickies.net/2015/10/02/spamassassin-bayes_00-1-90-although-sa-learn-runs-daily/

Also this one:
http://www.iredmail.org/docs/store.spamassassin.bayes.in.sql.html


Zhang Huangbin, founder of iRedMail project: http://www.iredmail.org/
Time zone: GMT+8 (China/Beijing).

Re: Amavisd and Bayes (again...)

[@lbutlr]

> On Nov 24, 2016, at 7:09 AM, Dominic Raferd  wrote:
> 
> Maybe this can help: 
> https://www.nerd-quickies.net/2015/10/02/spamassassin-bayes_00-1-90-although-sa-learn-runs-daily/

Spamassasin and babes work perfectly well when I run mail through SA myself. It 
is when amavis does it that bates doesn’t work.

I have the root bayes files linked to mavin’s home directory and have 
permissions set so they are readable. Still doesn’t work.

Re: Amavisd and Bayes (again...)

[Dominic Raferd]

Maybe this can help:
https://www.nerd-quickies.net/2015/10/02/spamassassin-bayes_00-1-90-although-sa-learn-runs-daily/


On 24 November 2016 at 13:43, @lbutlr  wrote:

> On Nov 24, 2016, at 2:09 AM, Alex Masidlover 
> wrote:
> > Any help would be appreciated before I drown in spam...
>
> I’ve posted a couple of times about the same exact issue with amavisnot
> using bases and so far no one has been able to provide any guidance as to
> why.
>
> For now, I’ve given up. AMavis still catches a lot os spam without bases,
> and then I manually run received mail through SA again. It’s ugly, but it
> works.
>
>

Re: Amavisd and Bayes (again...)

[@lbutlr]

On Nov 24, 2016, at 2:09 AM, Alex Masidlover  wrote:
> Any help would be appreciated before I drown in spam...

I’ve posted a couple of times about the same exact issue with amavisnot using 
bases and so far no one has been able to provide any guidance as to why.

For now, I’ve given up. AMavis still catches a lot os spam without bases, and 
then I manually run received mail through SA again. It’s ugly, but it works.

Amavisd and Bayes (again...)

[Alex Masidlover]

Hi,

I'm currently being deluged with spam and have been trying to use BAYES
filters to try and get rid of some of it. I've made a lot of progress
but am now very stuck.

I have go to the point where I have (temporarily) given the amavis user
a shell and when I run spamassassin on an email from the command line I
get:

spamassassin -t &1 | grep -i bayes
Nov 22 16:12:10.355 [10336] dbg: plugin: loading
Mail::SpamAssassin::Plugin::Bayes from @INC
Nov 22 16:12:10.603 [10336] dbg: config: fixed relative path:
/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf
Nov 22 16:12:10.603 [10336] dbg: config: using
"/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf"
for included file
Nov 22 16:12:10.603 [10336] dbg: config: read file
/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf
Nov 22 16:12:11.594 [10336] dbg: plugin:
Mail::SpamAssassin::Plugin::Bayes=HASH(0x27d2868) implements
'learner_new', priority 0
Nov 22 16:12:11.595 [10336] dbg: bayes: learner_new
self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x27d2868),
bayes_store_module=Mail::SpamAssassin::BayesStore::DBM
Nov 22 16:12:11.609 [10336] dbg: bayes: learner_new: got
store=Mail::SpamAssassin::BayesStore::DBM=HASH(0x2fa76c8)
Nov 22 16:12:11.609 [10336] dbg: plugin:
Mail::SpamAssassin::Plugin::Bayes=HASH(0x27d2868) implements
'learner_is_scan_available', priority 0
Nov 22 16:12:11.613 [10336] dbg: bayes: tie-ing to DB file R/O
/var/amavis/.spamassassin/bayes_toks
Nov 22 16:12:11.614 [10336] dbg: bayes: tie-ing to DB file R/O
/var/amavis/.spamassassin/bayes_seen
Nov 22 16:12:11.614 [10336] dbg: bayes: found bayes db version 3
Nov 22 16:12:13.528 [10336] dbg: bayes: untie-ing

However, when the same email was received through amavsid-new it
received the following headers:

X-Virus-Scanned: amavisd-new at zednax.com
X-Spam-Flag: NO
X-Spam-Score: 3.962
X-Spam-Level: ***
X-Spam-Status: No, score=3.962 tagged_above=0 required=4
 tests=[DCC_CHECK=1.1, HTML_MESSAGE=0.001, RDNS_NONE=1.274,
 REMOVE_BEFORE_LINK=1.587] autolearn=no autolearn_force=no

I start amavisd in screen with debugs on as the same user; the debugs
from amavisd show:

Nov 22 15:55:46.359 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
plugin: loading Mail::SpamAssassin::Plugin::Bayes from @INC
Nov 22 15:55:46.569 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
config: fixed relative path:
/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf
Nov 22 15:55:46.569 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
config: using
"/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf"
for included file
Nov 22 15:55:46.569 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
config: read file
/var/lib/spamassassin/3.004000/updates_spamassassin_org/23_bayes.cf
Nov 22 15:55:47.565 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'learner_new', priority 0
Nov 22 15:55:47.566 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: learner_new
self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00),
bayes_store_module=Mail::SpamAssassin::BayesStore::DBM
Nov 22 15:55:47.566 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: learner_new: got
store=Mail::SpamAssassin::BayesStore::DBM=HASH(0x5b102c8)
Nov 22 15:55:47.566 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'learner_is_scan_available', priority 0
Nov 22 15:55:47.566 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_toks
Nov 22 15:55:47.567 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_seen
Nov 22 15:55:47.567 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: found bayes db version 3
Nov 22 15:55:49.089 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'learner_close', priority 0
Nov 22 15:55:49.089 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
bayes: untie-ing
Nov 22 15:55:49.089 mta0.zednax.com /usr/sbin/amavisd[7630]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'prefork_init', priority 0
Nov 22 15:55:49.090 mta0.zednax.com /usr/sbin/amavisd[7630]:
SpamAssassin loaded plugins: AskDNS, AutoLearnThreshold, Bayes,
BodyEval, Check, DCC, DKIM, DNSEval, FreeMail, HTMLEval, HTTPSMismatch,
HeaderEval, ImageInfo, MIMEEval, MIMEHeader, Pyzor, Razor2, RelayEval,
ReplaceTags, SpamCop, URIDetail, URIEval, VBounce, WLBLEval,
WhiteListSubject
Nov 22 15:55:49.104 mta0.zednax.com /usr/sbin/amavisd[7649]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'spamd_child_init', priority 0

at startup, then when processing the message:

Nov 22 16:08:37.091 mta0.zednax.com /usr/sbin/amavisd[9727]: SA dbg:
plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55e7a00) implements
'spamd_child_init', priority 0
Nov 2