Re: FW: Tons of spam getting through
On Tue, 19 Aug 2014, Greg Ledford wrote: What exactly are SA headers supposed to look like? On 19.08.14 13:05, John Hardin wrote: SA headers look like this: X-Spam-Status: No, score=0.138 tagged_above=-100 required=5 tests=[MISSING_MID=0.14, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=no autolearn_force=no This one is actually amavisd header, which means that the MTA uses spamassassin indirectly. Just FYI. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. 10 GOTO 10 : REM (C) Bill Gates 1998, All Rights Reserved!
sa-learn site-wide bayes on Redis
Hi all. I am managing a bunch of Linux MTAs which are placed in front of some Exchange servers. In such a configuration the Bayes filter is deployed site-wide. For a new deployment of these servers I am planning to use Redis as a centralized backend (previously the bayes db were just files saved on the disk). My question is: do I have to use a specific option to tell sa-learn that the bayes db is now hosted on Redis? Or sa-learn will use the info from the bayes_sql_dsn directive in my local.cf? Looking into the wiki: http://wiki.apache.org/spamassassin/SiteWideBayesSetup or into the sa-learn docs: http://spamassassin.apache.org/full/3.4.x/doc/sa-learn.html did not give me any clues. Thanks in advance! Best regards, Matteo
Re: sa-learn site-wide bayes on Redis
On 08/20/2014 02:25 PM, Matteo Dessalvi wrote: Hi all. I am managing a bunch of Linux MTAs which are placed in front of some Exchange servers. In such a configuration the Bayes filter is deployed site-wide. For a new deployment of these servers I am planning to use Redis as a centralized backend (previously the bayes db were just files saved on the disk). My question is: do I have to use a specific option to tell sa-learn that the bayes db is now hosted on Redis? Or sa-learn will use the info from the bayes_sql_dsn directive in my local.cf? Looking into the wiki: http://wiki.apache.org/spamassassin/SiteWideBayesSetup or into the sa-learn docs: http://spamassassin.apache.org/full/3.4.x/doc/sa-learn.html did not give me any clues. see http://svn.apache.org/repos/asf/spamassassin/trunk/contrib/HOWTO.Bayes-Redis/ hope that helps. This is not an official doc, so if you see anything that needs to be added/changed, pls let me know.
Re: sa-learn site-wide bayes on Redis
No, unfortunately it does not help me. I already have a proper config file for SA to access Redis as backend and most of the configurations are done automatically through a Chef cookbook (Redis included). In the docs you pointed me there's nothing about the interaction between sa-learn and Redis. Best regards, Matteo On 20.08.2014 14:42, Axb wrote: see http://svn.apache.org/repos/asf/spamassassin/trunk/contrib/HOWTO.Bayes-Redis/ hope that helps. This is not an official doc, so if you see anything that needs to be added/changed, pls let me know.
Re: sa-learn site-wide bayes on Redis
bayes_store_module Mail::SpamAssassin::BayesStore::Redis tells SA to use the Redis backend. To sa-learn this becomes transparent, as with any other backed (DBD,SDBM,SQL) bayes_redis.cf shows what parameters are mandatory/optional On 08/20/2014 03:02 PM, Matteo Dessalvi wrote: No, unfortunately it does not help me. I already have a proper config file for SA to access Redis as backend and most of the configurations are done automatically through a Chef cookbook (Redis included). In the docs you pointed me there's nothing about the interaction between sa-learn and Redis. Best regards, Matteo On 20.08.2014 14:42, Axb wrote: see http://svn.apache.org/repos/asf/spamassassin/trunk/contrib/HOWTO.Bayes-Redis/ hope that helps. This is not an official doc, so if you see anything that needs to be added/changed, pls let me know.
Delays with Check_Bayes
Hello and good morning. We are running into some delays that we are trying to pin down a root cause for. Below are some examples. Within the examples, you can see that the check_bayes: scan is consuming most of the timing. Does anyone have any suggests on what to look at? We use 3.3.2. We have eight scanners setup to handle the scanning with 5GB RAM and 4 CPUs each. Volume is 250K - 500K per day. Aug 19 5:06:07 amavis[1581]: (01581-07-2)TIMING-SA total 138564 ms - parse: 2 0.00% extract_message_metadata: 37 (0.0%) get_uri_detail_list: 7 (0.0%) tests_pri_-1000: 13 (0.0%) tests_pri_-950: 1.08 (0.0%) tests_pri_-900: 1.13 (0.0%) tests_pri_-400: 137793 (99.4%) check_bayes: 137786 (99.4%) tests_pri_0: 708 (0.5%) check_dkim_adsp: 15 (0.0%) check_spf: 10 (0.0%) poll_dns_idle: 6 (0.0%) tests_pri_500: 3 (0.0%) get_report: 0.88 (0.0%) Aug 19 6:06:08 amavis[1271]: (01271-12-3)TIMING-SA total 118903 ms - parse: 1.750.00% extract_message_metadata: 34 (0.0%) get_uri_detail_list: 8 (0.0%) tests_pri_-1000: 7 (0.0%) tests_pri_-950: 1.11 (0.0%) tests_pri_-900: 1.18 (0.0%) tests_pri_-400: 118273 (99.5%) check_bayes: 118266 (99.5%) tests_pri_0: 419 (0.4%) check_dkim_adsp: 46 (0.0%) check_spf: 5 (0.0%) poll_dns_idle: 152 (0.1%) tests_pri_500: 156 (0.1%) get_report: 5 (0.0%) Aug 19 5:06:21 amavis[6764]: (06764-02-6)TIMING-SA total 99680 ms - parse: 2 0.00% extract_message_metadata: 37 (0.0%) get_uri_detail_list: 7 (0.0%) tests_pri_-1000: 13 (0.0%) tests_pri_-950: 1.08 (0.0%) tests_pri_-900: 1.13 (0.0%) tests_pri_-400: 98881 (99.2%) check_bayes: 98874 (99.2%) tests_pri_0: 736 (0.7%) check_dkim_adsp: 12 (0.0%) check_spf: 5 (0.0%) poll_dns_idle: 1.26 (0.0%) tests_pri_500: 3 (0.0%) get_report: 0.85 (0.0%) Aug 19 5:06:19 amavis[9621]: (09621-13-6)TIMING-SA total 99636 ms - parse: 2 0.00% extract_message_metadata: 38 (0.0%) get_uri_detail_list: 7 (0.0%) tests_pri_-1000: 12 (0.0%) tests_pri_-950: 1.09 (0.0%) tests_pri_-900: 1.14 (0.0%) tests_pri_-400: 98847 (99.2%) check_bayes: 98839 (99.2%) tests_pri_0: 726 (0.7%) check_dkim_adsp: 11 (0.0%) check_spf: 6 (0.0%) poll_dns_idle: 2 (0.0%) tests_pri_500: 3 (0.0%) get_report: 0.87 (0.0%) Aug 19 6:06:07 amavis[16447]: (16447-06-10) TIMING-SA total 90079 ms - parse: 2 0.00% extract_message_metadata: 34 (0.0%) get_uri_detail_list: 5 (0.0%) tests_pri_-1000: 9 (0.0%) tests_pri_-950: 1.24 (0.0%) tests_pri_-900: 1.35 (0.0%) tests_pri_-400: 89698 (99.6%) check_bayes: 89685 (99.6%) tests_pri_0: 323 (0.4%) check_dkim_adsp: 51 (0.1%) check_spf: 20 (0.0%) poll_dns_idle: 16 (0.0%)tests_pri_500: 3 (0.0%) get_report: 1.03 (0.0%) Aug 19 5:07:28 amavis[3901]: (03901-02-7)TIMING-SA total 87855 ms - parse: 4 0.00% extract_message_metadata: 84 (0.1%) get_uri_detail_list: 22 (0.0%) tests_pri_-1000: 32 (0.0%) tests_pri_-950: 1.10 (0.0%) tests_pri_-900: 1.18 (0.0%) tests_pri_-400: 87015 (99.0%) check_bayes: 86980 (99.0%) tests_pri_0: 699 (0.8%) check_dkim_adsp: 32 (0.0%) check_spf: 9 (0.0%) poll_dns_idle: 1.03 (0.0%) tests_pri_500: 4 (0.0%) get_report: 1.03 (0.0%) Aug 19 6:15:03 amavis[7851]: (07851-02-3)TIMING-SA total 81154 ms - parse: 2 0.00% extract_message_metadata: 41 (0.1%) get_uri_detail_list: 11 (0.0%) tests_pri_-1000: 11 (0.0%) tests_pri_-950: 1.05 (0.0%) tests_pri_-900: 1.11 (0.0%) tests_pri_-400: 80789 (99.5%) check_bayes: 80778 (99.5%) tests_pri_0: 299 (0.4%) check_spf: 6 (0.0%) poll_dns_idle: 1.31 (0.0%) tests_pri_500: 3 (0.0%) get_report: 1.14 (0.0%) -- View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
On 08/20/2014 03:15 PM, redtailjason wrote: Hello and good morning. We are running into some delays that we are trying to pin down a root cause for. Below are some examples. Within the examples, you can see that the check_bayes: scan is consuming most of the timing. Does anyone have any suggests on what to look at? We use 3.3.2. We have eight scanners setup to handle the scanning with 5GB RAM and 4 CPUs each. Volume is what type of Bayes backed are you using? do you use auto expiration or via cron? and pls post the output of sa-learn --dump magic
Re: sa-learn site-wide bayes on Redis
Ok, perfect! Thanks a lot! This is what I want to know and I was not so sure about. I may be wrong but it looks to me the fact that tools like sa-learn can access transparently the backends configured for SA is not exactly clear from the docs. It would be great if the wiki maintainers could add a short note somewhere in the pages regarding the SiteWide deployment or related topics. Best regards, Matteo On 20.08.2014 15:08, Axb wrote: bayes_store_module Mail::SpamAssassin::BayesStore::Redis tells SA to use the Redis backend. To sa-learn this becomes transparent, as with any other backed (DBD,SDBM,SQL) bayes_redis.cf shows what parameters are mandatory/optional
Re: sa-learn site-wide bayes on Redis
I so love to posters. On 08/20/2014 03:33 PM, Matteo Dessalvi wrote: Ok, perfect! Thanks a lot! This is what I want to know and I was not so sure about. I may be wrong but it looks to me the fact that tools like sa-learn can access transparently the backends configured for SA is not exactly clear from the docs. It would be great if the wiki maintainers could add a short note somewhere in the pages regarding the SiteWide deployment or related topics. Best regards, Matteo On 20.08.2014 15:08, Axb wrote: bayes_store_module Mail::SpamAssassin::BayesStore::Redis tells SA to use the Redis backend. To sa-learn this becomes transparent, as with any other backed (DBD,SDBM,SQL) bayes_redis.cf shows what parameters are mandatory/optional Watch your memory usage: If you configure Redis to dump data from memory to file, it's safe to *double* the amount of memory you planned for Redis usage as in my case: sa-learn --dump magic 0.000 0 3 0 non-token data: bayes db version 0.000 0 25218483 0 non-token data: nspam 0.000 0 11919587 0 non-token data: nham # Memory used_memory:3637407032 used_memory_human:3.39G used_memory_rss:4068585472 used_memory_peak:3702485960 used_memory_peak_human:3.45G used_memory_lua:205824 mem_fragmentation_ratio:1.12 mem_allocator:jemalloc-3.2.0 I keep at least 5 GB of free memory for the dump to file to avoid ugly swaps or crashes. free total used free sharedbuffers cached Mem:1426264857866648475984 0 162744 1343408 -/+ buffers/cache:42805129982136 Swap: 2046968 02046968
Need help with setting up MySQL storage for SA
Hi, I'm using Spamassassin in a virtual user environment. To store preferences like settings, Bayes and AWL for each user I'm trying to set up a MySQL storage. I created the MySQL tables according the instructions from the files awl_mysql.sql, bayes_mysql.sql, README.awl, README.bayes, README and userpref_mysql that came with my Spamassassin 3.4 installation on Ubuntu 14.04. The connection to the database seem to be working. For me the debug output looks like if Spamassassin would expect to be already some data in the tables. Where shall I get this data from? Do I have to manually create entries for each user? What am I missing? When calling spamc -u t...@michi.su testmail.txt I'm getting the following debug output (shortened): Aug 20 08:14:46.563 [16682] dbg: config: Conf::SQL: executing SQL: select preference, value from userpref where username = 't...@michi.su' or username = '@GLOBAL' order by username asc Aug 20 08:14:46.563 [16682] dbg: config: retrieving prefs for t...@michi.su from SQL server Aug 20 08:14:46.564 [16682] dbg: info: user has changed Aug 20 08:14:46.564 [16682] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x30fdce0), bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL Aug 20 08:14:46.564 [16682] dbg: bayes: using username: t...@michi.su Aug 20 08:14:46.564 [16682] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3d1a768) Aug 20 08:14:46.565 [16682] dbg: bayes: database connection established Aug 20 08:14:46.566 [16682] dbg: bayes: found bayes db version 3 Aug 20 08:14:46.566 [16682] dbg: bayes: unable to initialize database for t...@michi.su user, aborting! The MySQL relevant options that I added are: user_scores_dsn DBI:mysql:spamassassin:localhost user_scores_sql_usernamespamassassin user_scores_sql_passwordpass bayes_store_module Mail::SpamAssassin::BayesStore::MySQL bayes_sql_dsn DBI:mysql:spamassassin:localhost bayes_sql_username spamassassin bayes_sql_password pass auto_whitelist_factory Mail::SpamAssassin::SQLBasedAddrList user_awl_dsnDBI:mysql:spamassassin:localhost user_awl_sql_username spamassassin user_awl_sql_password pass
Help determining what is causing mails being marked as Spam
Hi all, I know this is not necessarily related to SpamAssasin itself, but I'm having a hard time trying to make my sent mails (from an own domain) not being marked as Spam on Gmail, so I was wondering if someone of you could have a look at the headers below and find out a reason why is my mail categorically marked as spam? I must mention that I have never done bulk sends (not even now, I'm refering to one-to-one mails), never an account has been compromised on this domain, obviously I have never sent spam, so I just don't understand why is it all marked as spam. The only one tip I have right now is that the reverse DNS query of the server's resolves to a different host than the one sent in EHLO, but in that case I don't know how to avoid that since the hosting where the dedicated server is located automatically assigns their own host to each IP. Could this have something to do with it? The IP is: 92.222.24.114 The mail server is: mail.devels.es The reverse DNS assigned by the hosting to that IP is: 114.ip-92-222-24.eu Below I'm including the headers, I'd be very grateful for any help to find out why is every mail marked as spam since I've already run of ideas... Thanks! Delivered-To: agmailacco...@gmail.com Received: by 10.64.231.7 with SMTP id tc7csp124091iec; Wed, 20 Aug 2014 06:26:33 -0700 (PDT) X-Received: by 10.60.67.34 with SMTP id k2mr38266595oet.52.1408541193378; Wed, 20 Aug 2014 06:26:33 -0700 (PDT) Return-Path: servi...@devels.es Received: from mail.devels.es (114.ip-92-222-24.eu. [92.222.24.114]) by mx.google.com with ESMTPS id y11si29607456oep.28.2014.08.20.06.26.32 for agmailacco...@gmail.com (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Aug 2014 06:26:33 -0700 (PDT) Received-SPF: pass (google.com: domain of servi...@devels.es designates 92.222.24.114 as permitted sender) client-ip=92.222.24.114; Authentication-Results: mx.google.com; spf=pass (google.com: domain of servi...@devels.es designates 92.222.24.114 as permitted sender) smtp.mail=servi...@devels.es; dkim=pass (test mode) header.i=@devels.es Received: by mail.devels.es (Postfix, from userid 111) id D99F41202940; Wed, 20 Aug 2014 14:26:31 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=devels.es; s=mail; t=1408541191; bh=ykRtuaP0t7bumAv86BRHpkisY8xRTGEWixUOr2eU/RM=; h=From:To:Subject:Date:From; b=EjcZcnd8RjwwfAi/sP98ArGfQrm9ls3MN6jafEi44KFqTvgc0qzd3sEA4BX7vFeY2 b/mYt63balL89g+uGntuI1ZMnbaiwG6DprilQ4mHube8VsjOkte3fOwpt9gmu8K2jf nFc2jixDolFsR1OxQ/c6zxme6usOZoZW6m149MAM= X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on vps81276.ovh.net X-Spam-Level: X-Spam-Status: No, score=0.5 required=3.0 tests=ALL_TRUSTED,MISSING_DATE, MISSING_MID,T_DKIM_INVALID autolearn=no autolearn_force=no version=3.4.0 Received: from rpi.devels.es (unknown [77.231.204.119]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: servi...@devels.es) by mail.devels.es (Postfix) with ESMTPSA id E4011120291F for agmailacco...@gmail.com; Wed, 20 Aug 2014 14:26:30 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=devels.es; s=mail; t=1408541191; bh=ykRtuaP0t7bumAv86BRHpkisY8xRTGEWixUOr2eU/RM=; h=From:To:Subject:From; b=ftYApmVO4tNY0pXbs+dCPmYhmYG3C/FYPl/hSUQWAmQ06zYO3SNptS/mpvAGRaiYg YCKekKipmoS3jNnMujyZltOYQRXfc7XkuO5NUuv0IbJN3h41rhpaiETl8frhEc2yOz IXFw6/z0KC69xDPwJva5or//SAmCR+XsduMa0yKE= From: Test account servi...@devels.es To: test55 agmailacco...@gmail.com Subject: This is a test MIME-Version: 1.0 Content-type: text/plain; charset=UTF-8 Message-Id: 20140820132631.d99f41202...@mail.devels.es Date: Wed, 20 Aug 2014 14:26:31 +0100 (BST) This is a test
Re: Help determining what is causing mails being marked as Spam
On 08/20/2014 04:24 PM, Nicolás wrote: The only one tip I have right now is that the reverse DNS query of the server's resolves to a different host than the one sent in EHLO, but in that case I don't know how to avoid that since the hosting where the dedicated server is located automatically assigns their own host to each IP. Could this have something to do with it? get fcRDNS ...OVH lets you set a correct rdns for you hostname.
Re: Delays with Check_Bayes
Thank you for your response! Our Bayes is MySQL. Currently, the expiry runs via cron job to run during low volume times. Here is the dump from one of the scanners: netset: cannot include 127.0.0.1/32 as it has already been included 0.000 0 3 0 non-token data: bayes db version 0.000 0613 0 non-token data: nspam 0.000 0 0 0 non-token data: nham 0.000 0 50382 0 non-token data: ntokens 0.000 0 1362372138 0 non-token data: oldest atime 0.000 0 1396547409 0 non-token data: newest atime 0.000 0 0 0 non-token data: last journal sync atime 0.000 0 0 0 non-token data: last expiry atime 0.000 0 0 0 non-token data: last expire atime delta 0.000 0 0 0 non-token data: last expire reduction count Please let me know if you need additional information. -- View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111074.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
On 08/20/2014 04:35 PM, redtailjason wrote: Thank you for your response! Our Bayes is MySQL. Currently, the expiry runs via cron job to run during low volume times. Here is the dump from one of the scanners: netset: cannot include 127.0.0.1/32 as it has already been included 0.000 0 3 0 non-token data: bayes db version 0.000 0613 0 non-token data: nspam 0.000 0 0 0 non-token data: nham 0.000 0 50382 0 non-token data: ntokens 0.000 0 1362372138 0 non-token data: oldest atime 0.000 0 1396547409 0 non-token data: newest atime 0.000 0 0 0 non-token data: last journal sync atime 0.000 0 0 0 non-token data: last expiry atime 0.000 0 0 0 non-token data: last expire atime delta 0.000 0 0 0 non-token data: last expire reduction count Please let me know if you need additional information. is that really your production bayes DB? So little data? 0.000 0613 0 non-token data: nspam 0.000 0 0 0 non-token data: nham pls post the output of spamassassin --lint -D bayes and make sure you do this under the same user which is used by SA
Re: Need help with setting up MySQL storage for SA
Hi. I did test a similar configuration a while ago and had the same problem. If you take a look at this thread on the mailing list: http://spamassassin.1065346.n5.nabble.com/Bayes-vars-records-on-MySQL-not-created-automatically-td104615.html you'll see it was a problem of running 'sa-learn --sync' as the user who is running the test. Best regards, Matteo On 20.08.2014 16:07, Michael wrote: Hi, I'm using Spamassassin in a virtual user environment. To store preferences like settings, Bayes and AWL for each user I'm trying to set up a MySQL storage. I created the MySQL tables according the instructions from the files awl_mysql.sql, bayes_mysql.sql, README.awl, README.bayes, README and userpref_mysql that came with my Spamassassin 3.4 installation on Ubuntu 14.04. The connection to the database seem to be working. For me the debug output looks like if Spamassassin would expect to be already some data in the tables. Where shall I get this data from? Do I have to manually create entries for each user? What am I missing? When calling spamc -u t...@michi.su testmail.txt I'm getting the following debug output (shortened): Aug 20 08:14:46.563 [16682] dbg: config: Conf::SQL: executing SQL: select preference, value from userpref where username = 't...@michi.su' or username = '@GLOBAL' order by username asc Aug 20 08:14:46.563 [16682] dbg: config: retrieving prefs for t...@michi.su from SQL server Aug 20 08:14:46.564 [16682] dbg: info: user has changed Aug 20 08:14:46.564 [16682] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x30fdce0), bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL Aug 20 08:14:46.564 [16682] dbg: bayes: using username: t...@michi.su Aug 20 08:14:46.564 [16682] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3d1a768) Aug 20 08:14:46.565 [16682] dbg: bayes: database connection established Aug 20 08:14:46.566 [16682] dbg: bayes: found bayes db version 3 Aug 20 08:14:46.566 [16682] dbg: bayes: unable to initialize database for t...@michi.su user, aborting! The MySQL relevant options that I added are: user_scores_dsn DBI:mysql:spamassassin:localhost user_scores_sql_usernamespamassassin user_scores_sql_passwordpass bayes_store_module Mail::SpamAssassin::BayesStore::MySQL bayes_sql_dsn DBI:mysql:spamassassin:localhost bayes_sql_username spamassassin bayes_sql_password pass auto_whitelist_factory Mail::SpamAssassin::SQLBasedAddrList user_awl_dsnDBI:mysql:spamassassin:localhost user_awl_sql_username spamassassin user_awl_sql_password pass
Re: Delays with Check_Bayes
On Wed, 20 Aug 2014, redtailjason wrote: Thank you for your response! Our Bayes is MySQL. Currently, the expiry runs via cron job to run during low volume times. Does the MySQL log say anything suggestive? Here is the dump from one of the scanners: netset: cannot include 127.0.0.1/32 as it has already been included 0.000 0 3 0 non-token data: bayes db version 0.000 0613 0 non-token data: nspam 0.000 0 0 0 non-token data: nham This doesn't explain the long scan time, but you can't expect Bayes to score at all until you learn some ham too. It needs examples of both to be able to tell the difference. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Efficiency can magnify good, but it magnifies evil just as well. So, we should not be surprised to find that modern electronic communication magnifies stupidity as *efficiently* as it magnifies intelligence. -- Robert A. Matern --- 4 days until the 1935th anniversary of the destruction of Pompeii
Re: Delays with Check_Bayes
Below is the output that you are seeking: $ spamassassin --lint -D bayes Aug 20 07:54:53.816 [6955] warn: netset: cannot include 127.0.0.1/32 as it has already been included Aug 20 07:54:54.415 [6955] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x382ddd0), bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL Aug 20 07:54:54.428 [6955] dbg: bayes: using username: administrator Aug 20 07:54:54.428 [6955] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3b0e068) Aug 20 07:54:54.448 [6955] dbg: bayes: database connection established Aug 20 07:54:54.450 [6955] dbg: bayes: found bayes db version 3 Aug 20 07:54:54.452 [6955] dbg: bayes: Using userid: 3 Aug 20 07:54:54.456 [6955] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB 200 Aug 20 07:54:54.464 [6955] dbg: bayes: database connection established Aug 20 07:54:54.464 [6955] dbg: bayes: found bayes db version 3 Aug 20 07:54:54.467 [6955] dbg: bayes: Using userid: 3 Aug 20 07:54:54.472 [6955] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB 200 Please let me know if you need additional information. -- View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111078.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
Do not have enough HAM to kick on bayes. -- Jeremy McSpadden Flux Labs | http://www.fluxlabs.nethttp://www.fluxlabs.net/ | Endless Solutions Office : 850-250-5590x501tel:850-250-5590;501 | Cell : 850-890-2543tel:850-890-2543 | Fax : 850-254-2955tel:850-254-2955 On Aug 20, 2014, at 10:36 AM, redtailjason ja...@redtailtechnology.commailto:ja...@redtailtechnology.com wrote: Aug 20 07:54:54.456 [6955] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB 200
Re: Delays with Check_Bayes
On 08/20/2014 05:35 PM, redtailjason wrote: Below is the output that you are seeking: $ spamassassin --lint -D bayes Aug 20 07:54:53.816 [6955] warn: netset: cannot include 127.0.0.1/32 as it has already been included Aug 20 07:54:54.415 [6955] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x382ddd0), bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL Aug 20 07:54:54.428 [6955] dbg: bayes: using username: administrator Aug 20 07:54:54.428 [6955] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x3b0e068) Aug 20 07:54:54.448 [6955] dbg: bayes: database connection established Aug 20 07:54:54.450 [6955] dbg: bayes: found bayes db version 3 Aug 20 07:54:54.452 [6955] dbg: bayes: Using userid: 3 Aug 20 07:54:54.456 [6955] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB 200 Aug 20 07:54:54.464 [6955] dbg: bayes: database connection established Aug 20 07:54:54.464 [6955] dbg: bayes: found bayes db version 3 Aug 20 07:54:54.467 [6955] dbg: bayes: Using userid: 3 Aug 20 07:54:54.472 [6955] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB 200 Please let me know if you need additional information. you've run this a administrator from your first msg, you run amavis, does you amavis also run under user administrator? You should do/post debugging under that same user.
Re: Delays with Check_Bayes
AXB, The initial post was data extracted from mail.log on the scanner using cat /var/log/mail.log | grep check_bayes while logged as administrator. -- View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111081.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
On 20. aug. 2014 17.36.39 redtailjason ja...@redtailtechnology.com wrote: $ spamassassin --lint -D bayes Aug 20 07:54:53.816 [6955] warn: netset: cannot include 127.0.0.1/32 as it has already been included rfc 1700 is already in default sa config, tanks for that to developpers :) May miss rfc 1918 defined in lokal.cf, save time in rbl checks perldoc Mail::SpamAssassin:::Conf
Re: Delays with Check_Bayes
On Wed, 2014-08-20 at 07:35 -0700, redtailjason wrote:
Here is the dump from one of the scanners:
netset: cannot include 127.0.0.1/32 as it has already been included
0.000 0 3 0 non-token data: bayes db version
0.000 0613 0 non-token data: nspam
0.000 0 0 0 non-token data: nham
0.000 0 50382 0 non-token data: ntokens
0.000 0 1362372138 0 non-token data: oldest atime
0.000 0 1396547409 0 non-token data: newest atime
That's back in April -- and obviously not a production database.
You need to run sa-update as the user SA uses during scan. In your case
that's the user Amavis uses.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Delays with Check_Bayes
On Wed, 2014-08-20 at 08:51 -0700, redtailjason wrote:
The initial post was data extracted from mail.log on the scanner using cat
/var/log/mail.log | grep check_bayes while logged as administrator.
It doesn't matter what user greps the logs.
It was Amavis generating the logs. Thus, for debugging, all execution of
Amavis or SA commands must be done as the user Amavis runs as.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Delays with Check_Bayes
Thank you for the clarification, sorry about that. Below is the updated information: sa-learn --dump magic 0.000 0 3 0 non-token data: bayes db version 0.000 02687726 0 non-token data: nspam 0.000 0 846578 0 non-token data: nham 0.000 0 241756 0 non-token data: ntokens 0.000 0 1357225630 0 non-token data: oldest atime 0.000 0 1408558344 0 non-token data: newest atime 0.000 0 0 0 non-token data: last journal sync atime 0.000 0 1408541108 0 non-token data: last expiry atime 0.000 0 43200 0 non-token data: last expire atime delta 0.000 0 0 0 non-token data: last expire reduction count $ spamassassin --lint -D bayes Aug 20 11:13:37.432 [16980] warn: netset: cannot include 127.0.0.1/32 as it has already been included Aug 20 11:13:37.978 [16980] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x3e11a90), bayes_store_module=Mail::SpamAssassin::BayesStore::MySQL Aug 20 11:13:37.990 [16980] dbg: bayes: using username: amavis Aug 20 11:13:37.990 [16980] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::MySQL=HASH(0x40f2978) Aug 20 11:13:38.011 [16980] dbg: bayes: database connection established Aug 20 11:13:38.012 [16980] dbg: bayes: found bayes db version 3 Aug 20 11:13:38.015 [16980] dbg: bayes: Using userid: 1 Aug 20 11:13:38.040 [16980] dbg: bayes: corpus size: nspam = 2687726, nham = 846578 Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for *F = U*ignore D*compiling.spamassassin.taint.org D*spamassassin.taint.org D*taint.org D*org Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for *m = 1408558416 lint_rules Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for x-spam-relays-external = Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for x-spam-relays-internal = Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for *RT = Aug 20 11:13:38.042 [16980] dbg: bayes: header tokens for *RU = Aug 20 11:13:38.043 [16980] dbg: bayes: tok_get_all: token count: 20 Aug 20 11:13:38.045 [16980] dbg: bayes: cannot use bayes on this message; not enough usable tokens found Aug 20 11:13:38.045 [16980] dbg: bayes: not scoring message, returning undef Please let me know if you need additional information. -- View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111085.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Help determining what is causing mails being marked as Spam
El 20/08/2014 15:29, Axb escribió: On 08/20/2014 04:24 PM, Nicolás wrote: The only one tip I have right now is that the reverse DNS query of the server's resolves to a different host than the one sent in EHLO, but in that case I don't know how to avoid that since the hosting where the dedicated server is located automatically assigns their own host to each IP. Could this have something to do with it? get fcRDNS ...OVH lets you set a correct rdns for you hostname. Ok, already done that, waited a few hours and now the 'correct' DNS host appears in the header, but is still marked as spam. Any other idea? Thanks
Re: Help determining what is causing mails being marked as Spam
On Wed, 20 Aug 2014, Nicolás wrote: El 20/08/2014 15:29, Axb escribió: get fcRDNS ...OVH lets you set a correct rdns for you hostname. Ok, already done that, waited a few hours and now the 'correct' DNS host appears in the header, but is still marked as spam. Any other idea? Please post headers from the latest test. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- One difference between a liberal and a pickpocket is that if you demand your money back from a pickpocket he will not question your motives.-- William Rusher --- 4 days until the 1935th anniversary of the destruction of Pompeii
Re: Delays with Check_Bayes
On Wed, 2014-08-20 at 06:15 -0700, redtailjason wrote:
Hello and good morning. We are running into some delays that we are trying to
pin down a root cause for.
Below are some examples. Within the examples, you can see that the
check_bayes: scan is consuming most of the timing. Does anyone have any
suggests on what to look at? We use 3.3.2. We have eight scanners setup to
handle the scanning with 5GB RAM and 4 CPUs each. Volume is 250K - 500K per
day.
That volume means throughput of about 350 messages per minute, 5.8 per
second. Sounds reasonable for 8 dedicated scanners.
Your samples are showing overall timings between about 90 seconds and
more than 2 minutes. Which means processing commonly takes less time,
and these are some extreme cases -- unless you really do have 50-100
busy processes per machine.
How many such long-running processes do you see, how frequent are they?
Also, you mentioned you are using the MySQL backend for Bayes. You did
not add any further detail, though.
Do you have dedicated MySQL servers for Bayes? Or does each scanner
machine run a local MySQL server? Do they share / sync databases
somehow?
Please elaborate on your environment, in particular everything
concerning Bayes.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Help determining what is causing mails being marked as Spam
El 20/08/2014 19:55, John Hardin escribió: On Wed, 20 Aug 2014, Nicolás wrote: El 20/08/2014 15:29, Axb escribió: get fcRDNS ...OVH lets you set a correct rdns for you hostname. Ok, already done that, waited a few hours and now the 'correct' DNS host appears in the header, but is still marked as spam. Any other idea? Please post headers from the latest test. This would be the latest test: Delivered-To: agmailacco...@gmail.com Received: by 10.217.170.201 with SMTP id gx51csp651014web; Wed, 20 Aug 2014 11:41:49 -0700 (PDT) X-Received: by 10.60.70.65 with SMTP id k1mr51483241oeu.61.1408560108791; Wed, 20 Aug 2014 11:41:48 -0700 (PDT) Return-Path: servi...@devels.es Received: from mail.devels.es (mail.devels.es. [92.222.24.114]) by mx.google.com with ESMTPS id ow18si30489820oeb.94.2014.08.20.11.41.46 for agmailacco...@gmail.com (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Aug 2014 11:41:47 -0700 (PDT) Received-SPF: pass (google.com: domain of servi...@devels.es designates 92.222.24.114 as permitted sender) client-ip=92.222.24.114; Authentication-Results: mx.google.com; spf=pass (google.com: domain of servi...@devels.es designates 92.222.24.114 as permitted sender) smtp.mail=servi...@devels.es; dkim=pass (test mode) header.i=@devels.es Received: by mail.devels.es (Postfix, from userid 111) id EFDE21202940; Wed, 20 Aug 2014 19:41:44 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=devels.es; s=mail; t=1408560104; bh=a/A6+BYLAO2+vq8y7s8N7Wz+VNNllrhxtir8ONLj5hU=; h=From:To:Subject:Date:From; b=ixZZbRmkSq0le67WlxbCx0rwHVvx7+STI8HCYxBSKpiaIzfdw4OA8vwmblHbiRprD 3iNJ0Mddik3EsB0zP6QI4JsyvVbXdtDPaHmelSCnPMlh62inYFMwNQCuRdUhnhU8+C WwkqBK2UCdap2RXJPhYIy5ASn2xSUJV2eTFUx5XY= X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on vps81276.ovh.net X-Spam-Level: X-Spam-Status: No, score=0.5 required=3.0 tests=ALL_TRUSTED,MISSING_DATE, MISSING_MID,T_DKIM_INVALID autolearn=no autolearn_force=no version=3.4.0 Received: from rpi.devels.es (unknown [77.231.204.119]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: servi...@devels.es) by mail.devels.es (Postfix) with ESMTPSA id ED0DE120293C for agmailacco...@gmail.com; Wed, 20 Aug 2014 19:41:43 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=devels.es; s=mail; t=1408560104; bh=a/A6+BYLAO2+vq8y7s8N7Wz+VNNllrhxtir8ONLj5hU=; h=From:To:Subject:From; b=IFOjy8rFQMkMLqAgJ0HoL/ajpd5Rlj+acd3vPbgTFuHELY+e+sWsRwY2GA7IgOPVz 3qQXalitm4c5nR1Nnkt3MUKinyqx1xEpyo8Ted4Lopzga8meQy6+fL2t6FLe9NuG6A YtpgBG8HKpZfdfThz2RKkArKD1R/kUOYINgbmdFs= From: Test account servi...@devels.es To: Another user agmailacco...@gmail.com Subject: This is a test MIME-Version: 1.0 Content-type: text/plain; charset=UTF-8 Message-Id: 20140820184144.efde21202...@mail.devels.es Date: Wed, 20 Aug 2014 19:41:44 +0100 (BST) This is a test
KAM rules keep me giggling
What poison pills are they ;) Someone suggested using kam and I'm happy now. I figure they are not with SA spirit that no one rule should make that happen but... yeah. One helluva rules! -- jarif.bit signature.asc Description: OpenPGP digital signature
Re: KAM rules keep me giggling
On 8/20/2014 3:34 PM, Jari Fredriksson wrote: What poison pills are they ;) Someone suggested using kam and I'm happy now. I figure they are not with SA spirit that no one rule should make that happen but... yeah. One helluva rules! :) We're always curious how our rules work for other people, so if there are any in there that are particularly effective, let us know!
Re: Delays with Check_Bayes
We are seeing about 4000-7000 delayed messages per day. We do utilize a dedicated MySQL Server for the Bayes and all 8 scanners share it. Please let me know if this does not fully clarify our setup for you. Has anyone heard of a configuration where the transactions are written to a file (ie text file) and then inserted into the database every few minutes? MySQL seems to process imports faster than line by line transactions. -- View this message in context: http://spamassassin.1065346.n5.nabble.com/Delays-with-Check-Bayes-tp111067p111092.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Delays with Check_Bayes
On Wed, 20 Aug 2014, redtailjason wrote: We are seeing about 4000-7000 delayed messages per day. We do utilize a dedicated MySQL Server for the Bayes and all 8 scanners share it. Are you open to the possibility of upgrading to 3.4.0 and using the Redis backend for Bayes? (Just offering an alternative.) -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- If Microsoft made hammers, everyone would whine about how poorly screws were designed and about how they are hard to hammer in, and wonder why it takes so long to paint a wall using the hammer. --- 4 days until the 1935th anniversary of the destruction of Pompeii
Re: Delays with Check_Bayes
On Wed, 2014-08-20 at 13:38 -0700, redtailjason wrote:
We are seeing about 4000-7000 delayed messages per day. We do utilize a
dedicated MySQL Server for the Bayes and all 8 scanners share it. Please let
me know if this does not fully clarify our setup for you.
So we're talking about 1% of the messages.
Does this happen with all scanner machines, or is this isolated to a
single one? If not all scanners are affected, any differences in network
connection?
When did this start? Any relevant changes roughly about that time?
What's your DB server load? Any noticeable load spikes, like 5k times a
day? In particular, while a message is taking 2 minutes wall-clock time
for Bayes, does either the scanner or database server have an unusual
high load? Do you have MySQL logs which might show issues?
Can you reproduce the Bayes lags? That is, can you identify a sample
message, and re-process manually?
When replying, please include the relevant quoted parts you're directly
referring to. With some context it is easier to follow the thread.
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Delays with Check_Bayes
On 21/08/14 09:00, John Hardin wrote: Are you open to the possibility of upgrading to 3.4.0 and using the Redis backend for Bayes? (Just offering an alternative.) We just last week moved over to 3.4.0 with a central Redis backend with 6 spamd servers spread over USA and Europe. Bit of a stretch in terms of WAN latency but it seems to be working really well. I love doing a spamc -L spam against one SA server and then immediately re-scanning the same message by a different one and seeing the BAYES_99 light up :-) So far, so good! -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
OT -postfix config
Sorry to be OT. I thought this would be simple, but I am getting muddled, at this time of day. Have setup a postfix host to accept email from various (local) hosts and forward. The initial idea was to simply create a list of people for whom to forward email to another domain (translating the domain portion of the address) . Works a treat. However, now have a situation where more mail is being sent, mainly alerts from various process on the allowed hosts, some to our native domain (off box tho) and some to be sent to the wilds. I have a relay host for each but have gotten muddled on how to tell posftix to forward mydomain to a local host and how to forward every other domain to a different local host. Clue by four anyone? Tutorial page?
Re: OT -postfix config
On 8/20/2014 5:56 PM, Joe Acquisto-j4 wrote: Sorry to be OT. I thought this would be simple, but I am getting muddled, at this time of day. Have setup a postfix host to accept email from various (local) hosts and forward. The initial idea was to simply create a list of people for whom to forward email to another domain (translating the domain portion of the address) . Works a treat. However, now have a situation where more mail is being sent, mainly alerts from various process on the allowed hosts, some to our native domain (off box tho) and some to be sent to the wilds. I have a relay host for each but have gotten muddled on how to tell posftix to forward mydomain to a local host and how to forward every other domain to a different local host. Clue by four anyone? Tutorial page? Maybe you're looking for the transport_maps feature. If you have more questions about postfix, please ask on the postfix-users list. If you ask questions on postfix-users, please include more details about your setup. Helpful pages: http://www.postfix.org/documentation.html http://www.postfix.org/lists.html -- Noel Jones
Re: OT -postfix config
On Wed, 20 Aug 2014 18:56:48 -0400 Joe Acquisto-j4 j...@j4computers.com wrote: Sorry to be OT. Clue by four anyone? Tutorial page? Your best chance of getting help is on the postfix list. You can subscribe at http://www.postfix.org/lists.html they will also gladly help if clue-by-four are needed. jd
