Re: URI-DNSBL problem with spamassassin 3.2.5
Message original Sujet : Re: URI-DNSBL problem with spamassassin 3.2.5 De : Dan Schaefer Dan Schaefer wrote: Please, can someone feed http://pastebin.ca/1495707 into spamassassin 3.3.0 and see how it works ? Hi! pts rule name description -- -- 0.0 HTML_MESSAGE BODY: HTML included in message -4.0 BAYES_00 BODY: Bayesian spam probability is 0 to 1% Sorry that's 3.2.5 Hi! Dan, Many thanks for your reply. I'm also having 3.2.5 but spamassassin freeze when processing this email, it just freeze during 20 minutes and then give the same rule as you mentionned. I also try to use the plugin HitFreqsRuleTiming (see this thread) without any success with 3.2.5 I'm trying to find which rule is the culprit one ! Any hint ? Thanks, Eddy
Re: URI-DNSBL problem with spamassassin 3.2.5
Dan Schaefer wrote: Hi! Please, can someone feed http://pastebin.ca/1495707 into spamassassin 3.3.0 and see how it works ? Many thanks for your help Eddy pts rule name description -- -- 0.0 HTML_MESSAGE BODY: HTML included in message -4.0 BAYES_00 BODY: Bayesian spam probability is 0 to 1% Sorry that's 3.2.5 -- Dan Schaefer Web Developer/Systems Analyst Performance Administration Corp.
Re: URI-DNSBL problem with spamassassin 3.2.5
Hi! Please, can someone feed http://pastebin.ca/1495707 into spamassassin 3.3.0 and see how it works ? Many thanks for your help Eddy pts rule name description -- -- 0.0 HTML_MESSAGE BODY: HTML included in message -4.0 BAYES_00 BODY: Bayesian spam probability is 0 to 1% -- Dan Schaefer Web Developer/Systems Analyst Performance Administration Corp.
Re: URI-DNSBL problem with spamassassin 3.2.5
Message original Sujet : Re: URI-DNSBL problem with spamassassin 3.2.5 Date : 2009-07-14 11:07 but Ido not find any timing.log file on my current directory or anywhere on my system!! Did I missed something ? I doubt all the necessary hooks are in place for that plugin to work in 3.2.5, you'd need to run 3.3 to make use of that plugin. Michael On Jul 9, 2009, at 1:40 PM, Eddy Beliveau wrote: Hi! Michael, Many thanks for the hint. The current devel version is 3.3.0-alpha1 (dated 2 weeks ago) Do you know when the production release will be available ? I do not want to put non-production version on my academic server. Maybe I can send you the culprit email if you have 3.3 installed and see how it reacts on your location ! Is there a web page where I can inject the email to have it analysed by some SA version ? I tried my 250KB message with http://flashmarketing.com/spam-check.htm but it said that my message is too big Thanks, Eddy Hi! Please, can someone feed http://pastebin.ca/1495707 into spamassassin 3.3.0 and see how it works ? Many thanks for your help Eddy -- Eddy Beliveau HEC Montreal Montreal (Quebec) Canada
Re: URI-DNSBL problem with spamassassin 3.2.5
Message original Sujet : Re: URI-DNSBL problem with spamassassin 3.2.5 De : Michael Parker Pour : Eddy Beliveau Copie à : users@spamassassin.apache.org, Mark Martinec Date : 2009-07-09 19:37 On Jul 9, 2009, at 1:40 PM, Eddy Beliveau wrote: but Ido not find any timing.log file on my current directory or anywhere on my system!! Did I missed something ? I doubt all the necessary hooks are in place for that plugin to work in 3.2.5, you'd need to run 3.3 to make use of that plugin. Michael Hi! Michael, Many thanks for the hint. The current devel version is 3.3.0-alpha1 (dated 2 weeks ago) Do you know when the production release will be available ? I do not want to put non-production version on my academic server. Maybe I can send you the culprit email if you have 3.3 installed and see how it reacts on your location ! Is there a web page where I can inject the email to have it analysed by some SA version ? I tried my 250KB message with http://flashmarketing.com/spam-check.htm but it said that my message is too big Thanks, Eddy
Re: URI-DNSBL problem with spamassassin 3.2.5
On Jul 9, 2009, at 1:40 PM, Eddy Beliveau wrote: but Ido not find any timing.log file on my current directory or anywhere on my system!! Did I missed something ? I doubt all the necessary hooks are in place for that plugin to work in 3.2.5, you'd need to run 3.3 to make use of that plugin. Michael
Re: URI-DNSBL problem with spamassassin 3.2.5
Is there some way to find the culprit rule ? other that removing all rules and adding them one at the time. Perhaps the best timing tool for rules is the HitFreqsRuleTiming plugin, which can be found in masses/plugins/HitFreqsRuleTiming.pm in the distribution. Should work with 3.2.5 and with 3.3.0. It is quite primitive in that it does not have any configurables, but just dumps its results to a file 'timing.log' in the current working directory (make sure it is writable for the UID under which SA is running, no error is issued if it can not write there). To activate it, copy it to some place, then add a loadplugin command to one of your .pre files, such as a local.pre, providing the path to the .pm file, e.g.: loadplugin HitFreqsRuleTiming /etc/mail/spamassassin/HitFreqsRuleTiming.pm Then run a command line spamassassin giving it a sample message, e.g.: $ spamassassin -t Hi! Mark, Many thanks for your reply. I'm using SpamAssassin version 3.2.5 running on Perl version 5.8.5 I did extract HitFreqsRuleTiming.pm from spamassassin_20090708151200.tar.gz, move it to /etc/mail/spamassassin then create the /etc/mail/spamassassin/local.pre file with the following line loadplugin HitFreqsRuleTiming /etc/mail/spamassassin/HitFreqsRuleTiming.pm Now, on /tmp directory, I execute "spamassassin --lint -t -D" which correctly said: ...cut... [24936] dbg: plugin: loading HitFreqsRuleTiming from /etc/mail/spamassassin/HitFreqsRuleTiming.pm ...cut... [27955] dbg: plugin: HitFreqsRuleTiming=HASH(0x114a8588) implements 'start_rules', priority 0 [27955] dbg: rules: compiled one_line_body tests [27955] dbg: plugin: Mail::SpamAssassin::Plugin::Rule2XSBody=HASH(0x1197b19c) implements 'run_body_fast_scan', priority 0 [27955] dbg: rules: running head tests; score so far=0 [27955] dbg: rules: compiled head tests [27955] dbg: plugin: HitFreqsRuleTiming=HASH(0x114a8588) implements 'ran_rule', priority 0 ...cut... [27955] dbg: check: is spam? score=4.205 required=5 [27955] dbg: check: tests=MISSING_DATE,MISSING_HEADERS,MISSING_SUBJECT,NO_RECEIVED,NO_RELAYS [27955] dbg: check: subtests=__BOTNET_NOTRUST,__HAS_MSGID,__HAVE_BOUNCE_RELAYS,__MISSING_REF,__MSGID_OK_DIGITS,__MSGID_OK_HOST,__MSOE_MID_WRONG_CASE,__NONEMPTY_BODY,__SANE_MSGID,__SARE_WHITELIST_FLAG,__TVD_BODY,__UNUSABLE_MSGID but Ido not find any timing.log file on my current directory or anywhere on my system!! Did I missed something ? Thanks, Eddy
Re: URI-DNSBL problem with spamassassin 3.2.5
Eddy, > So I spin it again with "-L -D" > 09:24:10.109 16.022 0.036 [20476] dbg: rules: ran rawbody rule > __SARE_HAS_FG_COLOR ==> got hit: ""color:" > 09:45:09.826 1275.740 1259.717 [20476] dbg: rules: ran eval rule > __SARE_HTML_HAS_BR ==> got hit (1) > So, after the 20 minutes delay, it says: > 09:45:09.826 1275.740 1259.717 [20476] dbg: rules: ran eval rule > __SARE_HTML_HAS_BR ==> got hit (1) > > Can I assume that the 20 minutes delay is caused by the > __SARE_HTML_HAS_BR rule ? More likely some other rule inbetween the __SARE_HAS_FG_COLOR and the __SARE_HTML_HAS_BR, which didn't produce any hits and was therefore not logged. > Is there some way to find the culprit rule ? > other that removing all rules and adding them one at the time. Perhaps the best timing tool for rules is the HitFreqsRuleTiming plugin, which can be found in masses/plugins/HitFreqsRuleTiming.pm in the distribution. Should work with 3.2.5 and with 3.3.0. It is quite primitive in that it does not have any configurables, but just dumps its results to a file 'timing.log' in the current working directory (make sure it is writable for the UID under which SA is running, no error is issued if it can not write there). To activate it, copy it to some place, then add a loadplugin command to one of your .pre files, such as a local.pre, providing the path to the .pm file, e.g.: loadplugin HitFreqsRuleTiming /etc/mail/spamassassin/HitFreqsRuleTiming.pm Then run a command line spamassassin giving it a sample message, e.g.: $ spamassassin -t
Re: URI-DNSBL problem with spamassassin 3.2.5
Message original
Sujet : Re: URI-DNSBL problem with spamassassin 3.2.5
De : John Hardin
Pour : Eddy Beliveau
Copie à : SpamAssassin Users List
Date : 2009-07-07 16:49
On Tue, 7 Jul 2009, Mark Martinec wrote:
It is not the DNS query that is a problem here.
Eddy:
What happens when you run the test using -L (no network tests)? Does
it still take as long?
Hi!
Mark & John, many thanks for your replies
So I spin it again with "-L -D"
...cut...
09:24:09.030 14.943 0.217 [20476] dbg: rules: running uri tests; score
so far=0
09:24:09.058 14.971 0.028 [20476] dbg: rules: compiled uri tests
09:24:09.078 14.991 0.020 [20476] dbg: rules: ran uri rule
__DOS_HAS_ANY_URI ==> got hit: "h"
09:24:09.099 15.012 0.020 [20476] dbg: rules: ran uri rule
__LOCAL_PP_NONPPURL ==> got hit: "http://www.davekeller.com";
09:24:09.220 15.133 0.121 [20476] dbg: pdfinfo: Identified 0 possible
mime parts that need checked for PDF content
09:24:09.220 15.133 0.000 [20476] dbg: pdfinfo: set_tag called for
PDFCOUNT 0
09:24:09.220 15.133 0.000 [20476] dbg: pdfinfo: set_tag called for
PDFIMGCOUNT 0
09:24:09.378 15.291 0.158 [20476] dbg: eval: stock info total: 0
09:24:09.379 15.293 0.002 [20476] dbg: rules: ran eval rule
__SARE_BODY_BLANKS_5_100 ==> got hit (1)
09:24:09.380 15.294 0.001 [20476] dbg: rules: ran eval rule
__TAG_EXISTS_BODY ==> got hit (1)
09:24:09.431 15.344 0.051 [20476] dbg: eval: text words: 2280, html
words: 2257
09:24:09.438 15.351 0.007 [20476] dbg: eval: madiff: left: 22, orig:
2257, max-difference: 0.97%
09:24:09.446 15.359 0.008 [20476] dbg: rules: ran eval rule __MIME_HTML
==> got hit (1)
09:24:09.529 15.443 0.084 [20476] dbg: rules: ran eval rule HTML_MESSAGE
==> got hit (1)
09:24:09.532 15.445 0.002 [20476] dbg: rules: ran eval rule
__TAG_EXISTS_HTML ==> got hit (1)
09:24:09.546 15.460 0.015 [20476] dbg: rules: ran eval rule
__TVD_MIME_ATT_TP ==> got hit (1)
09:24:09.561 15.474 0.014 [20476] dbg: rules: ran eval rule
__HAVE_BOUNCE_RELAYS ==> got hit (1)
09:24:09.563 15.476 0.002 [20476] dbg: rules: running rawbody tests;
score so far=0.001
09:24:09.602 15.515 0.039 [20476] dbg: rules: compiled rawbody tests
09:24:09.778 15.691 0.175 [20476] dbg: rules: ran rawbody rule
__SARE_HTML_SINGLET2 ==> got hit: ">o<"
09:24:09.817 15.730 0.040 [20476] dbg: rules: ran rawbody rule
__SARE_BLACK_FG_COLOR ==> got hit: ""color: black"
09:24:10.073 15.986 0.256 [20476] dbg: rules: ran rawbody rule
__TVD_BODY ==> got hit: "vers"
09:24:10.109 16.022 0.036 [20476] dbg: rules: ran rawbody rule
__SARE_HAS_FG_COLOR ==> got hit: ""color:"
09:45:09.826 1275.740 1259.717 [20476] dbg: rules: ran eval rule
__SARE_HTML_HAS_BR ==> got hit (1)
09:45:09.827 1275.741 0.001 [20476] dbg: rules: ran eval rule
__SARE_HTML_HAS_DIV ==> got hit (1)
09:45:09.828 1275.741 0.000 [20476] dbg: rules: ran eval rule __MIME_QP
==> got hit (2)
09:45:09.828 1275.741 0.000 [20476] dbg: rules: ran eval rule
__SARE_HTML_HAS_P ==> got hit (1)
09:45:09.829 1275.742 0.000 [20476] dbg: rules: ran eval rule
__SARE_HTML_HAS_A ==> got hit (1)
09:45:09.829 1275.742 0.001 [20476] dbg: rules: running full tests;
score so far=0.001
09:45:09.838 1275.751 0.009 [20476] dbg: rules: compiled full tests
09:45:10.002 1275.915 0.164 [20476] dbg: rules: running meta tests;
score so far=0.001
09:45:10.003 1275.916 0.001 [20476] dbg: rules: compiled meta tests
09:45:10.003 1275.916 0.000 [20476] dbg: check: running tests for
priority: 500
09:45:10.003 1275.916 0.000 [20476] dbg: dns: harvest_dnsbl_queries
...cut...
So, after the 20 minutes delay, it says:
09:45:09.826 1275.740 1259.717 [20476] dbg: rules: ran eval rule
__SARE_HTML_HAS_BR ==> got hit (1)
Can I assume that the 20 minutes delay is caused by the
__SARE_HTML_HAS_BR rule ?
If so, it is used by one of those 2 rules:
/var/lib/spamassassin/3.002005/70_sare_html0_cf_sare_sa-update_dostech_net/200606040500.cf:
rawbody __SARE_HTML_HAS_BR eval:html_tag_exists('br')
/var/lib/spamassassin/3.002005/70_sare_html1_cf_sare_sa-update_dostech_net/200606040500.cf:
rawbody __SARE_HTML_HAS_BR eval:html_tag_exists('br')
I then just add the following line to my local.cf file
score __SARE_HTML_HAS_BR 0
and re-test it with "-L -D" but I'm having the same result !!
Is there some way to find the culprit rule ?
other that removing all rules and adding them one at the time.
For testing purposes, can I reduce the 20 minutes delay variable to 1
minute ?
Any help will be appreciated.
Many thanks,
Eddy
--
Eddy Beliveau
HEC Montreal
Montreal (Quebec)
Canada
Re: URI-DNSBL problem with spamassassin 3.2.5
On Tue, 7 Jul 2009, Mark Martinec wrote: It is not the DNS query that is a problem here. Eddy: What happens when you run the test using -L (no network tests)? Does it still take as long? -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- End users want eye candy and the "ooo's and hhh's" experience when reading mail. To them email isn't a tool, but an entertainment form. -- Steve Lake --- Today: Robert Heinlein's 102nd birthday
Re: URI-DNSBL problem with spamassassin 3.2.5
Eddy, > I'm using spamassassin 3.2.5 on my academic RHEL server and it works > well :-) > I'm also using Net::DNS version: 0.65, amavisd-new 2.6.4, postfix 2.5.6 > > I'm receiving an email which take too much time to process. > I have to remove it from my postfix's mail queue > > here is part of the spamassassin debug run: > ...cut... > 13:31:25.577 13.621 0.002 [28208] dbg: rules: running rawbody tests; > score so far=0.001 > 13:31:25.617 13.661 0.039 [28208] dbg: rules: compiled rawbody tests > 13:31:25.794 13.839 0.178 [28208] dbg: rules: ran rawbody rule > __SARE_HTML_SINGLET2 ==> got hit: ">o<" > 13:31:25.836 13.880 0.042 [28208] dbg: rules: ran rawbody rule > __SARE_BLACK_FG_COLOR ==> got hit: ""color: black" > 13:31:26.093 14.137 0.257 [28208] dbg: rules: ran rawbody rule > __TVD_BODY ==> got hit: "vers" > 13:31:26.129 14.173 0.036 [28208] dbg: rules: ran rawbody rule > __SARE_HAS_FG_COLOR ==> got hit: ""color:" > 13:51:46.568 1234.612 *1220.439* [28208] dbg: async: select found 17 > responses ready (t.o.=0.0) > 13:51:46.568 1234.612 0.000 [28208] dbg: async: completed in 1221.532 s: > URI-A, A:dns3.nettica.com. > The async step takes 1220 seconds to complete !! > > It happend everytime I'm processing manually that specific message [...] > I do not understand how it can take 1220 seconds to complete when it > said timeout=15s It is not the DNS query that is a problem here. Most likely some regexp rules are taking that long. Async DNS queries are fired off at the beginning, then most other rules are evaluated, giving DNS resolver opportunity to do its job during that time. Only after most rules have been evaluated are the DNS responses collected. There is no point in enforcing a timeout on DNS responses when we have to wait for other computations anyway. Mark
URI-DNSBL problem with spamassassin 3.2.5
Hi! Netfriends, I'm using spamassassin 3.2.5 on my academic RHEL server and it works well :-) I'm also using Net::DNS version: 0.65, amavisd-new 2.6.4, postfix 2.5.6 I'm receiving an email which take too much time to process. I have to remove it from my postfix's mail queue here is part of the spamassassin debug run: ...cut... 13:31:25.577 13.621 0.002 [28208] dbg: rules: running rawbody tests; score so far=0.001 13:31:25.617 13.661 0.039 [28208] dbg: rules: compiled rawbody tests 13:31:25.794 13.839 0.178 [28208] dbg: rules: ran rawbody rule __SARE_HTML_SINGLET2 ==> got hit: ">o<" 13:31:25.836 13.880 0.042 [28208] dbg: rules: ran rawbody rule __SARE_BLACK_FG_COLOR ==> got hit: ""color: black" 13:31:26.093 14.137 0.257 [28208] dbg: rules: ran rawbody rule __TVD_BODY ==> got hit: "vers" 13:31:26.129 14.173 0.036 [28208] dbg: rules: ran rawbody rule __SARE_HAS_FG_COLOR ==> got hit: ""color:" 13:51:46.568 1234.612 *1220.439* [28208] dbg: async: select found 17 responses ready (t.o.=0.0) 13:51:46.568 1234.612 0.000 [28208] dbg: async: completed in 1221.532 s: URI-A, A:dns3.nettica.com. 13:51:46.569 1234.614 0.001 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:13.136.94.64 (timeout 15.0s, min 3.0s) 13:51:46.570 1234.614 0.001 [28208] dbg: async: completed in 1221.538 s: URI-A, A:ns2.planetmind.net. 13:51:46.571 1234.615 0.001 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:122.236.168.205 (timeout 15.0s, min 3.0s) 13:51:46.572 1234.616 0.001 [28208] dbg: async: completed in 1221.538 s: URI-A, A:ns2.accountsupport.com. 13:51:46.573 1234.617 0.002 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:105.254.254.65 (timeout 15.0s, min 3.0s) 13:51:46.574 1234.618 0.000 [28208] dbg: async: completed in 1221.539 s: URI-A, A:ns1.planetmind.net. 13:51:46.575 1234.619 0.002 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:69.236.168.205 (timeout 15.0s, min 3.0s) 13:51:46.576 1234.620 0.000 [28208] dbg: async: completed in 1221.532 s: URI-A, A:dns2.nettica.com. 13:51:46.577 1234.621 0.001 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:34.45.237.64 (timeout 15.0s, min 3.0s) 13:51:46.577 1234.621 0.000 [28208] dbg: async: completed in 1221.524 s: URI-DNSBL, DNSBL:sbl.spamhaus.org.:130.240.6.64 13:51:46.578 1234.622 0.000 [28208] dbg: async: completed in 1221.556 s: URI-DNSBL, DNSBL:sbl.spamhaus.org.:178.161.72.69 13:51:46.578 1234.622 0.000 [28208] dbg: async: completed in 1221.526 s: URI-DNSBL, DNSBL:sbl.spamhaus.org.:1.5.84.66 13:51:46.578 1234.622 0.000 [28208] dbg: async: completed in 1221.530 s: URI-A, A:dns1.nettica.com. 13:51:46.579 1234.624 0.001 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:11.136.94.64 (timeout 15.0s, min 3.0s) 13:51:46.580 1234.624 0.000 [28208] dbg: async: completed in 1221.554 s: URI-DNSBL, DNSBL:sbl.spamhaus.org.:197.68.20.69 13:51:46.580 1234.624 0.000 [28208] dbg: async: completed in 1221.553 s: URI-DNSBL, DNSBL:sbl.spamhaus.org.:242.210.124.74 13:51:46.580 1234.624 0.000 [28208] dbg: async: completed in 1221.528 s: URI-A, A:ns2.musearts.com. 13:51:46.582 1234.626 0.001 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:231.42.34.72 (timeout 15.0s, min 3.0s) 13:51:46.582 1234.626 0.000 [28208] dbg: async: completed in 1221.527 s: URI-A, A:ns3.musearts.com. 13:51:46.583 1234.627 0.001 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:232.42.34.72 (timeout 15.0s, min 3.0s) 13:51:46.584 1234.628 0.000 [28208] dbg: async: completed in 1221.530 s: URI-A, A:dns5.nettica.com. 13:51:46.585 1234.629 0.001 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:15.247.100.212 (timeout 15.0s, min 3.0s) 13:51:46.585 1234.629 0.000 [28208] dbg: async: completed in 1221.531 s: URI-A, A:dns4.nettica.com. 13:51:46.586 1234.630 0.001 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:223.170.41.69 (timeout 15.0s, min 3.0s) 13:51:46.587 1234.631 0.000 [28208] dbg: async: completed in 1221.542 s: URI-DNSBL, DNSBL:sbl.spamhaus.org.:112.7.117.66 13:51:46.587 1234.631 0.000 [28208] dbg: async: completed in 1221.539 s: URI-A, A:ns1.accountsupport.com. 13:51:46.588 1234.632 0.001 [28208] dbg: async: starting: URI-DNSBL, DNSBL:sbl.spamhaus.org.:104.254.254.65 (timeout 15.0s, min 3.0s) 13:51:46.589 1234.633 0.001 [28208] dbg: async: queries completed: 17, started: 11 13:51:46.589 1234.633 0.000 [28208] dbg: async: queries active: URI-DNSBL=11 at Tue Jul 7 13:51:46 2009 13:51:46.589 1234.633 0.000 [28208] dbg: dns: harvested completed queries The async step takes 1220 seconds to complete !! It happend everytime I'm processing manually that specific message I tried to add the following lines in my /etc/mail/spamassassin/local.cf file, with no success uridnsbl_skip_domain nettica.com uridnsbl_skip_domain sbl.spamhaus.org I do not understand how it can take 1220 seconds to complete when it said timeout
