Re: Slow Scan Time

[Sean Kennedy]

Turns out I sent this email too soon.  Even though I have use_pyzor 0
in my local.cf, having these lines in v310.pre uncommented was causing
the 5 seconds extra scan time:

loadplugin Mail::SpamAssassin::Plugin::Pyzor
loadplugin Mail::SpamAssassin::Plugin::Razor2



On Tue, Aug 31, 2010 at 10:51 AM, Sean Kennedy  wrote:
> I'm running SpamAssassin v3.3.1.  A message with 500 lines of text
> (22k) is taking 6-7 seconds.
>
> This is using spamc.  I am using a Bayes DB (no auto expire/learn)
> that's stored in SQL.  User preferences are also stored in SQL.  I
> have skip_rbl_checks on.  I am using only the official rules from
> updates.spamassassin.org and I am using sa-compile.
>
> I've also disabled all of these rules:
>
> score URIBL_SBL                 0
> score URIBL_SC_SURBL      0
> score URIBL_PH_SURBL      0
> score URIBL_AB_SURBL      0
> score URIBL_JP_SURBL       0
> score URIBL_OB_SURBL      0
> score URIBL_WS_SURBL     0
> score URIBL_BLACK            0
> score URIBL_RED                0
> score URIBL_GREY              0
>
> I tried disabling Bayes checks as well.  The message still took
> 5.5-6.5 seconds to finish scanning.  I'm testing scan time using:
>
> time spamc -E -u skenn...@office.vcn.com < ./testmail.txt
>
> The server I am testing this on is a AMD Athlon(tm) 64 X2 Dual Core
> Processor 3800+ w/ 2GB of memory.  I am testing this while the server
> is otherwise idle.
>
>
> Does anyone have any other ideas what could be causing this to go that
> slow?  I would expect 1s or less for a plain-text 22k message.
>
> Thanks,
> Sean
>

Slow Scan Time

[Sean Kennedy]

I'm running SpamAssassin v3.3.1.  A message with 500 lines of text
(22k) is taking 6-7 seconds.

This is using spamc.  I am using a Bayes DB (no auto expire/learn)
that's stored in SQL.  User preferences are also stored in SQL.  I
have skip_rbl_checks on.  I am using only the official rules from
updates.spamassassin.org and I am using sa-compile.

I've also disabled all of these rules:

score URIBL_SBL 0
score URIBL_SC_SURBL  0
score URIBL_PH_SURBL  0
score URIBL_AB_SURBL  0
score URIBL_JP_SURBL   0
score URIBL_OB_SURBL  0
score URIBL_WS_SURBL 0
score URIBL_BLACK0
score URIBL_RED0
score URIBL_GREY  0

I tried disabling Bayes checks as well.  The message still took
5.5-6.5 seconds to finish scanning.  I'm testing scan time using:

time spamc -E -u skenn...@office.vcn.com < ./testmail.txt

The server I am testing this on is a AMD Athlon(tm) 64 X2 Dual Core
Processor 3800+ w/ 2GB of memory.  I am testing this while the server
is otherwise idle.


Does anyone have any other ideas what could be causing this to go that
slow?  I would expect 1s or less for a plain-text 22k message.

Thanks,
Sean

Re: Increase in scan time from 3.3 to 3.3.1

[RW]

On Fri, 11 Jun 2010 17:32:05 -0400
Chris Conn  wrote:

> In a followup to 
> http://www.gossamer-threads.com/lists/spamassassin/users/151470;
> 
> Is it possible to set the priority on RBL rules to run after rules,
> or not at all if shortcircuited?

RBL test are done in parallel, and they are initiated early so SA can
get on with local tests during the DNS lookup. I don't know if what
you're asking for is possible, but it doesn't sound like a good idea.

Re: Increase in scan time from 3.3 to 3.3.1

[Chris Conn]

In a followup to 
http://www.gossamer-threads.com/lists/spamassassin/users/151470;


Is it possible to set the priority on RBL rules to run after rules, or 
not at all if shortcircuited?


I tried:

priority RCVD_IN_BL_SPAMCOP_NET -300
priority RCVD_IN_XBL -300
priority RCVD_IN_PSBL -300
priority RCVD_IN_UCEPROT1 -300
priority RCVD_IN_BRBL_LASTEXT -300
priority MYCUSTOM -900

yet running it through SA;

 pts rule name  description
 -- 
--

 3.0 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in bl.spamcop.net
  [Blocked - see 
]

 2.7 RCVD_IN_PSBL   RBL: Received via a relay in PSBL
[174.121.191.18 listed in psbl.surriel.com]
 1.5 RCVD_IN_UCEPROT1   RBL: Appears in uceprotect-1
[174.121.191.18 listed in 
dnsbl-1.uceprotect.net]

 1.6 RCVD_IN_BRBL_LASTEXT   RBL: RCVD_IN_BRBL_LASTEXT
[174.121.191.18 listed in 
bb.barracudacentral.org]

 8.0 RCVD_IN_XBLRBL: Received via a relay in Spamhaus XBL
[174.121.191.18 listed in zen.spamhaus.org]
 0.0 SHORTCIRCUIT   Not all rules were run, due to a 
shortcircuited rule

[score: 100]
 100 MYCUSTOMevaluation of message via MYCUSTOM



I don't want to run tons of RBLS on an email that I can already identify 
and shortcircuit...


What am I missing?  the shortcircuit plugin is obviously loaded since I 
am using it on other rulesets.


Thanks,

Chris

RE: Reducing scan time

[Chris]

On Thu, 2010-04-22 at 08:44 -0400, Kaleb Hosie wrote:

> Our scan time use to be much longer and it was because of clamav. I realized 
> that I was scanning with clamscan and not clamdscan.
> 
> The clamdscan uses the daemon that's already loaded, so it's not loading the 
> virus database everytime.
> 
> I'm not entirely sure whether that will help, but I know it helped us.
> 
> 
> Kaleb

This is on my home pc, no mail server involved. I'm using the clamav
plugin.

-- 
KeyID 0xE372A7DA98E6705C



signature.asc
Description: This is a digitally signed message part

RE: Reducing scan time

[Kaleb Hosie]

> On Thu, 2010-04-22 at 02:05 +0100, Martin Hepworth wrote:
> >
> >
> > On 22 April 2010 00:44, Chris  wrote:
> > I've posted two files below, one is the time output
> for a spam
> > and one
> > for ham. Seems like over the past few weeks SA scan
> times have
> > become
> > slower and slower. For instance stats from last night below.
> > Anyone with
> > any ideas on how to speed things up?
> >
> > Email:62  Autolearn: 0  AvgScore: -16.94
> AvgScanTime:
> > 21.25 sec
> > Spam: 17  Autolearn: 0  AvgScore:  35.71
> AvgScanTime:
> > 23.12 sec
> > Ham:  45  Autolearn: 0  AvgScore: -36.82
> AvgScanTime:
> > 20.54 sec
> >
> > For instance my scan times about two weeks ago were:
> >
> > Email:58  Autolearn: 0  AvgScore: -12.38
> AvgScanTime:
> > 11.36 sec
> > Spam: 16  Autolearn: 0  AvgScore:  85.06
> >  AvgScanTime:  8.52 sec
> > Ham:  42  Autolearn: 0  AvgScore: -49.50
> AvgScanTime:
> > 12.45 sec
> >
> > Spam
> > http://pastebin.com/fhd3XwHp
> >
> > Ham
> > http://pastebin.com/jbSD894i
> >
> > Any advice would be appreciated.
> >
> > Chris
> >
> > --
> > KeyID 0xE372A7DA98E6705C
> >
> >
> > What version of SA and what rules (extra or otherwise) are you
> > running?
>
> 3.3.0
> Standard rulesets plus:
>
> http://pastebin.com/UK68S0L4
>
> >
> > Anything else on the box in question slower?
>
> Clam was running really slow and sucking a bunch of memory
> however, the problem there was found to be that I had a
> mail.cvd and main.cld db, I removed the .cld file and it
> seemed to speed up somewhat.
> >
> > you run a local caching nameserver on the SA box?
>
> Yes, named is running
>
> > You run sa-update frequently?
>
> Yes, every four hours
> >
> > Does a debug show any obvious place where it's slow?
> >
> dbg: async: timing: 3.375 . dns:A:3.169.13.200.ix.dnsbl.manitu.net.
>
> See http://pastebin.com/7upp4BCp
>
>
>
>
>
>
> --
> KeyID 0xE372A7DA98E6705C
>
>

Our scan time use to be much longer and it was because of clamav. I realized 
that I was scanning with clamscan and not clamdscan.

The clamdscan uses the daemon that's already loaded, so it's not loading the 
virus database everytime.

I'm not entirely sure whether that will help, but I know it helped us.


Kaleb

Re: Reducing scan time

[Benny Pedersen]

On tor 22 apr 2010 04:21:40 CEST, Chris wrote

Clam was running really slow and sucking a bunch of memory however, the
problem there was found to be that I had a mail.cvd and main.cld db, I
removed the .cld file and it seemed to speed up somewhat.


this one is really a design bug in clamav when version update comes  
the tarball have db files put in clamav lib for updateing, wow  
freshclam did not work before :(


2 ways to fix:

make a bug on https://wwws.clamav.net/bugzilla/ on this one so db  
files is not in the source code tarball, seperate them is fine so save  
bandwidth for upgrading


or make the arch maintainers not install the db files when update  
version, just install when its first time install


--
xpoint http://www.unicom.com/pw/reply-to-harmful.html

Re: Reducing scan time

[Chris]

On Thu, 2010-04-22 at 02:05 +0100, Martin Hepworth wrote:
> 
> 
> On 22 April 2010 00:44, Chris  wrote:
> I've posted two files below, one is the time output for a spam
> and one
> for ham. Seems like over the past few weeks SA scan times have
> become
> slower and slower. For instance stats from last night below.
> Anyone with
> any ideas on how to speed things up?
> 
> Email:62  Autolearn: 0  AvgScore: -16.94  AvgScanTime:
> 21.25 sec
> Spam: 17  Autolearn: 0  AvgScore:  35.71  AvgScanTime:
> 23.12 sec
> Ham:  45  Autolearn: 0  AvgScore: -36.82  AvgScanTime:
> 20.54 sec
> 
> For instance my scan times about two weeks ago were:
> 
> Email:58  Autolearn: 0  AvgScore: -12.38  AvgScanTime:
> 11.36 sec
> Spam: 16  Autolearn: 0  AvgScore:  85.06
>  AvgScanTime:  8.52 sec
> Ham:  42  Autolearn: 0  AvgScore: -49.50  AvgScanTime:
> 12.45 sec
> 
> Spam
> http://pastebin.com/fhd3XwHp
> 
> Ham
> http://pastebin.com/jbSD894i
> 
> Any advice would be appreciated.
> 
> Chris
> 
> --
> KeyID 0xE372A7DA98E6705C
> 
> 
> What version of SA and what rules (extra or otherwise) are you
> running? 

3.3.0
Standard rulesets plus:

http://pastebin.com/UK68S0L4

> 
> Anything else on the box in question slower?

Clam was running really slow and sucking a bunch of memory however, the
problem there was found to be that I had a mail.cvd and main.cld db, I
removed the .cld file and it seemed to speed up somewhat.
> 
> you run a local caching nameserver on the SA box? 

Yes, named is running

> You run sa-update frequently?

Yes, every four hours
> 
> Does a debug show any obvious place where it's slow? 
> 
dbg: async: timing: 3.375 . dns:A:3.169.13.200.ix.dnsbl.manitu.net.

See http://pastebin.com/7upp4BCp






-- 
KeyID 0xE372A7DA98E6705C



signature.asc
Description: This is a digitally signed message part

Re: Reducing scan time

[Martin Hepworth]

On 22 April 2010 00:44, Chris  wrote:

> I've posted two files below, one is the time output for a spam and one
> for ham. Seems like over the past few weeks SA scan times have become
> slower and slower. For instance stats from last night below. Anyone with
> any ideas on how to speed things up?
>
> Email:62  Autolearn: 0  AvgScore: -16.94  AvgScanTime: 21.25 sec
> Spam: 17  Autolearn: 0  AvgScore:  35.71  AvgScanTime: 23.12 sec
> Ham:  45  Autolearn: 0  AvgScore: -36.82  AvgScanTime: 20.54 sec
>
> For instance my scan times about two weeks ago were:
>
> Email:58  Autolearn: 0  AvgScore: -12.38  AvgScanTime: 11.36 sec
> Spam: 16  Autolearn: 0  AvgScore:  85.06  AvgScanTime:  8.52 sec
> Ham:  42  Autolearn: 0  AvgScore: -49.50  AvgScanTime: 12.45 sec
>
> Spam
> http://pastebin.com/fhd3XwHp
>
> Ham
> http://pastebin.com/jbSD894i
>
> Any advice would be appreciated.
>
> Chris
>
> --
> KeyID 0xE372A7DA98E6705C
>
>
What version of SA and what rules (extra or otherwise) are you running?

Anything else on the box in question slower?

you run a local caching nameserver on the SA box?

You run sa-update frequently?

Does a debug show any obvious place where it's slow?


-- 
Martin Hepworth
Oxford, UK

Re: Reducing scan time

[Lee Dilkie]

Chris,

Do you use sa-compile? I found that made a tremendous difference for me.

-lee

Chris wrote:
> I've posted two files below, one is the time output for a spam and one
> for ham. Seems like over the past few weeks SA scan times have become
> slower and slower. For instance stats from last night below. Anyone with
> any ideas on how to speed things up?
>
> Email:62  Autolearn: 0  AvgScore: -16.94  AvgScanTime: 21.25 sec
> Spam: 17  Autolearn: 0  AvgScore:  35.71  AvgScanTime: 23.12 sec
> Ham:  45  Autolearn: 0  AvgScore: -36.82  AvgScanTime: 20.54 sec
>
> For instance my scan times about two weeks ago were:
>
> Email:58  Autolearn: 0  AvgScore: -12.38  AvgScanTime: 11.36 sec
> Spam: 16  Autolearn: 0  AvgScore:  85.06  AvgScanTime:  8.52 sec
> Ham:  42  Autolearn: 0  AvgScore: -49.50  AvgScanTime: 12.45 sec
>
> Spam
> http://pastebin.com/fhd3XwHp
>
> Ham
> http://pastebin.com/jbSD894i
>
> Any advice would be appreciated.
>
> Chris
>
>   

Reducing scan time

[Chris]

I've posted two files below, one is the time output for a spam and one
for ham. Seems like over the past few weeks SA scan times have become
slower and slower. For instance stats from last night below. Anyone with
any ideas on how to speed things up?

Email:62  Autolearn: 0  AvgScore: -16.94  AvgScanTime: 21.25 sec
Spam: 17  Autolearn: 0  AvgScore:  35.71  AvgScanTime: 23.12 sec
Ham:  45  Autolearn: 0  AvgScore: -36.82  AvgScanTime: 20.54 sec

For instance my scan times about two weeks ago were:

Email:58  Autolearn: 0  AvgScore: -12.38  AvgScanTime: 11.36 sec
Spam: 16  Autolearn: 0  AvgScore:  85.06  AvgScanTime:  8.52 sec
Ham:  42  Autolearn: 0  AvgScore: -49.50  AvgScanTime: 12.45 sec

Spam
http://pastebin.com/fhd3XwHp

Ham
http://pastebin.com/jbSD894i

Any advice would be appreciated.

Chris

-- 
KeyID 0xE372A7DA98E6705C



signature.asc
Description: This is a digitally signed message part

Re: Increase in scan time from 3.3 to 3.3.1

[Alex]

Hi,

>> What would be involved with making the PSBL DNSBL work with v3.2.5?
>
> Alex, I'm pretty sure that you are already using PSBL through my
> khop-bl channel, which adds PSBL, BRBL, Spam-eating Monkey (SEMBLACK),
> HostKarma/JunkEmailFilter, and, more recently, MSPIKE (as per a
> request from João) to pre-3.3.0 spamassassin versions.

Yes, indeed. Thanks very much for the follow-up. I've disabled it now
in my local config.

Thanks again,
Alex

Re: Increase in scan time from 3.3 to 3.3.1

[Adam Katz]

Alex wrote:
> What would be involved with making the PSBL DNSBL work with v3.2.5?

Alex, I'm pretty sure that you are already using PSBL through my
khop-bl channel, which adds PSBL, BRBL, Spam-eating Monkey (SEMBLACK),
HostKarma/JunkEmailFilter, and, more recently, MSPIKE (as per a
request from João) to pre-3.3.0 spamassassin versions.

It supports 3.3.0+ and avoids re-defining rules that were incorporated.

khop-bl directions:  http://khopesh.com/Anti-spam#sa-update_channels

Re: Increase in scan time from 3.3 to 3.3.1

[João Gouveia]

Hi,

- "Alex"  wrote:

> Hi,
> 
> > Anyway, it'll soon be deprecated in favor of 20_aux_tlds.cf, which
> is
> > part of the stock rule-set since 3.3.1. Bug 6361. As mentioned in
> the
> > release announcement.
> 
> Is the 20_aux_tlds.cf stable and available for use to replace it now?
> 
> Will the new RBLs in v3.3.1 ever be available/compatible with v3.2.5?
> What would be involved with making the PSBL DNSBL work with v3.2.5?

Ours is not included yet, but feel free to use it if it fits your environment.
Implementation details here: http://mailspike.org/

Last statistics from SA rule QA:  
http://ruleqa.spamassassin.org/20100320-r925566-n/%2FRCVD
(rule name T_RCVD_IN_ANBREP_BL)

> Thanks,
> Alex

-- 
João Gouveia

Re: Increase in scan time from 3.3 to 3.3.1

[Ned Slider]

Alex wrote:


Will the new RBLs in v3.3.1 ever be available/compatible with v3.2.5?
What would be involved with making the PSBL DNSBL work with v3.2.5?



You can certainly add additional RBLs to 3.2.5. For example:

# PSBL easy-on, easy-off blacklist: http://psbl.surriel.com
header   RCVD_IN_PSBL   eval:check_rbl('psbl', 'psbl.surriel.com.')
describe RCVD_IN_PSBL   Received via a relay in PSBL
tflags   RCVD_IN_PSBL   net


However, you can't use the new DBL list from Spamhaus, as that 
*requires* SA 3.3.1:


http://www.spamhaus.org/dbl/
http://www.spamhaus.org/faq/answers.lasso?section=Spamhaus%20DBL#287

Re: Increase in scan time from 3.3 to 3.3.1

[Alex]

Hi,

> Anyway, it'll soon be deprecated in favor of 20_aux_tlds.cf, which is
> part of the stock rule-set since 3.3.1. Bug 6361. As mentioned in the
> release announcement.

Is the 20_aux_tlds.cf stable and available for use to replace it now?

Will the new RBLs in v3.3.1 ever be available/compatible with v3.2.5?
What would be involved with making the PSBL DNSBL work with v3.2.5?

Thanks,
Alex

Re: Increase in scan time from 3.3 to 3.3.1

[Karsten Bräckelmann]

On Wed, 2010-03-24 at 14:44 -0400, Kris Deugau wrote:
> [...] that should save a little CPU time because I dropped the SARE
> 90_2tld channel for 3.3.x

The CPU time saved by dropping that file is negligible, hardly
measurable.

Anyway, it'll soon be deprecated in favor of 20_aux_tlds.cf, which is
part of the stock rule-set since 3.3.1. Bug 6361. As mentioned in the
release announcement.

  guenther


-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: Increase in scan time from 3.3 to 3.3.1

[Rick Macdougall]

On 24/03/2010 4:09 PM, Kris Deugau wrote:

Michael Scheidell wrote:

several more RBL's,
check your dns performance?


Looks like the new PSBL DNSBL is a bit slow. I wonder if the new load
from SA 3.3 is the cause? 

A quick walk through the SA log shows it isn't helping much here, so
I've disabled it locally.

I checked out their rsync file, and it's a 27M one-per-line list of IP
addresses. It could probably be trivially adjusted to load into rbldnsd.

-kgd


That seems to be the culprit.  Every thing looks the same as the older 
3.3 servers now.


Thanks for the investigation.

Regards,

Rick

Re: Increase in scan time from 3.3 to 3.3.1

[Kris Deugau]

Michael Scheidell wrote:

several more RBL's,
check your dns performance?


Looks like the new PSBL DNSBL is a bit slow.  I wonder if the new load 
from SA 3.3 is the cause?  


A quick walk through the SA log shows it isn't helping much here, so 
I've disabled it locally.


I checked out their rsync file, and it's a 27M one-per-line list of IP 
addresses.  It could probably be trivially adjusted to load into rbldnsd.


-kgd

Re: Increase in scan time from 3.3 to 3.3.1

[Rick Macdougall]

On 24/03/2010 2:40 PM, Michael Scheidell wrote:



On 3/24/10 2:23 PM, Rick Macdougall wrote:

Hi,

Any one have any idea what might cause an increase of scan times when
going from 3.3 to 3.3.1.

I've upgraded one server and the average scan time is now 4.3 seconds.

The 3 other servers still running 3.3 average 1.38


several more RBL's,
check your dns performance?



Yup, seems to be DNS related I guess.  One message that took 5.6 seconds 
to scan now scans in under 0.5 seconds.


Rick

Re: Increase in scan time from 3.3 to 3.3.1

[Rick Macdougall]

On 24/03/2010 2:40 PM, Michael Scheidell wrote:



On 3/24/10 2:23 PM, Rick Macdougall wrote:

Hi,

Any one have any idea what might cause an increase of scan times when
going from 3.3 to 3.3.1.

I've upgraded one server and the average scan time is now 4.3 seconds.

The 3 other servers still running 3.3 average 1.38


several more RBL's,
check your dns performance?



Hi,

We're running DJB's dnscache and I set the score to 0 for the 3 new 
spamhaus rules.


score URIBL_SBL 0
score URIBL_DBL_SPAM0
score URIBL_DBL_ERROR   0


What other new one's were added as I don't see much listed in the 
Changes file.



Regards,

Rick

Re: Increase in scan time from 3.3 to 3.3.1

[Kris Deugau]

Rick Macdougall wrote:
Any one have any idea what might cause an increase of scan times when 
going from 3.3 to 3.3.1.


I've upgraded one server and the average scan time is now 4.3 seconds.

The 3 other servers still running 3.3 average 1.38

All running Centos on exactly the same hardware.


(I thought I sent a message on the same subject;  looks like I forgot to 
send it from the right account.)


I've seen something similar, however I'm seeing the difference from 
3.2.5 to 3.3.x (both 3.3.0 and 3.3.1 show the same slowdown).  Average 
scan times have jumped from just under a second to just under four 
seconds on the live mail stream, and about the same on a dev machine 
with a small set of test messages.  Different hardware and OS release 
doesn't make much more than ~1% difference in scan times.


SA is installed via script, an initial configuration is set via script, 
and a set of symlinks are used to manage which version is live.  Another 
script updates all of the rules channels we use (and if anything, that 
should save a little CPU time because I dropped the SARE 90_2tld channel 
for 3.3.x).  Supporting Perl modules are all packaged (either Debian 
stock, or local packages).


-kgd

Re: Increase in scan time from 3.3 to 3.3.1

[Michael Scheidell]

On 3/24/10 2:23 PM, Rick Macdougall wrote:

Hi,

Any one have any idea what might cause an increase of scan times when 
going from 3.3 to 3.3.1.


I've upgraded one server and the average scan time is now 4.3 seconds.

The 3 other servers still running 3.3 average 1.38


several more RBL's,
check your dns performance?

--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation

   * Certified SNORT Integrator
   * 2008-9 Hot Company Award Winner, World Executive Alliance
   * Five-Star Partner Program 2009, VARBusiness
   * Best Anti-Spam Product 2008, Network Products Guide
   * King of Spam Filters, SC Magazine 2008

__
This email has been scanned and certified safe by SpammerTrap(r). 
For Information please see http://www.secnap.com/products/spammertrap/
__  

Increase in scan time from 3.3 to 3.3.1

[Rick Macdougall]

Hi,

Any one have any idea what might cause an increase of scan times when 
going from 3.3 to 3.3.1.


I've upgraded one server and the average scan time is now 4.3 seconds.

The 3 other servers still running 3.3 average 1.38

All running Centos on exactly the same hardware.

Thanks in advance.

Rick

Re: excessive scan time

[LuKreme]

On 22-Jan-2009, at 13:57, Brian J. Murrell wrote:

Now users need to know how to edit SQL records, or I
need to install a web interface for that.  The ROI here for that is  
just

not high enough.



Really?  A webface to edit user configuration options in an SQL  
database is trivial.  I know its trivial because *I* can do it.


--
"Whose motorcycle is this?" "It's  chopper, baby." "Whose chopper
is this?" "It's Zed's." "Who's Zed?" "Zed' dead, baby. Zed's
dead."

Re: excessive scan time

[Jonas Eckerman]

Brian J. Murrell wrote:


I'd also suggest using SQL for user preferences.


The user interface (i.e. editing a file) for user preferences is a 
different story.  Now users need to know how to edit SQL records, or I 
need to install a web interface for that.


Or you use a small script that reads the users preferences from 
file (when the file has been modified) and updates the SQL database.


Regards
/Jonas
--
Jonas Eckerman, FSDB & Fruktträdet
http://whatever.frukt.org/
http://www.fsdb.org/
http://www.frukt.org/

Re: excessive scan time

[Brian J. Murrell]

On Thu, 22 Jan 2009 12:37:09 +, Justin Mason wrote:

> you should definitely investigate ways to avoid doing NFS reads/writes
> of the bayes files -- that is extremely I/O intensive, and NFS deals
> with it very badly.

OK.  Noted.  Maybe I will push the bayes database into MySQL as 
previously suggested.

Thanx!

b.

Re: excessive scan time

[Brian J. Murrell]

On Thu, 22 Jan 2009 13:27:57 +0100, Jonas Eckerman wrote:
> 
> If you're not allready using a SQL database for bayes and AWL I'd
> suggest you do that.

Those two I might be willing to consider, however...
 
> I'd also suggest using SQL for user preferences.

The user interface (i.e. editing a file) for user preferences is a 
different story.  Now users need to know how to edit SQL records, or I 
need to install a web interface for that.  The ROI here for that is just 
not high enough.

> With bayes, AWL and user prefs in a SQL database that problem ought to
> be avoided. (Maybe there's more than those that should be moved from
> ~/.spamassassin though).

Yeah.  I tend to doubt those are the real culprits.  I think I have 
identified a backup process on the same server that does the NFS and mail 
as being quite expensive in both disk an memory and it's probably what is 
contending with spamd processes for resources.

b.

Re: excessive scan time

[Justin Mason]

you should definitely investigate ways to avoid doing NFS reads/writes
of the bayes files -- that is extremely I/O intensive, and NFS deals
with it very badly.

--j.

On Thu, Jan 22, 2009 at 12:27, Jonas Eckerman  wrote:
> Brian J. Murrell wrote:
>
>> One thing worth noting is that I have spamassassin using ~/.spamassassin
>> here and people's home dirs can be (i.e. NFS) mounted from remote machines
>> (i.e. their primary workstations), which do occasionally get shut down.
>
> If you're not allready using a SQL database for bayes and AWL I'd suggest
> you do that.
>
> I'd also suggest using SQL for user preferences.
>
>> I wonder what happens in the MTA->SA->local delivery process chain when
>> ~/.spamassassin is unavailable, or worse, on a stale mount.
>
> With bayes, AWL and user prefs in a SQL database that problem ought to be
> avoided. (Maybe there's more than those that should be moved from
> ~/.spamassassin though).
>
> /Jonas
> --
> Jonas Eckerman, FSDB & Fruktträdet
> http://whatever.frukt.org/
> http://www.fsdb.org/
> http://www.frukt.org/
>
>

Re: excessive scan time

[Jonas Eckerman]

Brian J. Murrell wrote:

One thing worth noting is that I have spamassassin using ~/.spamassassin 
here and people's home dirs can be (i.e. NFS) mounted from remote 
machines (i.e. their primary workstations), which do occasionally get 
shut down.


If you're not allready using a SQL database for bayes and AWL I'd 
suggest you do that.


I'd also suggest using SQL for user preferences.

I wonder what happens in the MTA->SA->local delivery process 
chain when ~/.spamassassin is unavailable, or worse, on a stale mount.


With bayes, AWL and user prefs in a SQL database that problem 
ought to be avoided. (Maybe there's more than those that should 
be moved from ~/.spamassassin though).


/Jonas
--
Jonas Eckerman, FSDB & Fruktträdet
http://whatever.frukt.org/
http://www.fsdb.org/
http://www.frukt.org/

Re: excessive scan time

[Brian J. Murrell]

On Mon, 19 Jan 2009 16:47:24 +0100, Matus UHLAR - fantomas wrote:
>  
> When did you sa-update for last time?

Ubuntu appears to install a cron.daily cron job which does this amongst 
other things.

> How many processes are you running
> in parallel?

I have a pretty low volume system but I did just up it from 5 to 8 
yesterday.

> Aren't you running out of memory?

No.
 
>> a) determine why the scan time is so long, after the fact (i.e. I could
>>try to run the same spam through a "spamassassin -D [-t]" but there
>>is no guarantee that whatever took so long the first time through
>>will again take so long)?
> 
> try running spamasssin with -L option

How will -L (local tests only) help me determine which remote tests are 
taking so long?

>> b) reduce some timeouts of some particular tests so that the total test
>>time does not exceed a reasonable threshold?
> 
> razor,pyzor,dcc,spf,dkim,rbl have their timeouts (*_timeout), see their
> (or SpamAssassin) docs.

Indeed.  "dpkg -L spamassassin | xargs grep _timeout" shows some very 
interesting results.

Now that I think about it, I wonder if I am barking up the wrong tree.  
One thing worth noting is that I have spamassassin using ~/.spamassassin 
here and people's home dirs can be (i.e. NFS) mounted from remote 
machines (i.e. their primary workstations), which do occasionally get 
shut down.  I wonder what happens in the MTA->SA->local delivery process 
chain when ~/.spamassassin is unavailable, or worse, on a stale mount.

Is there a reasonable timeout built in to trying to read from that dir?

Thots?

b.

Re: excessive scan time

[Matus UHLAR - fantomas]

On 19.01.09 15:33, Brian J. Murrell wrote:
> I'm running 3.2.4(-1ubuntu1) of spamassassin here and have been noticing 
> some excessive scan times.  i.e.:
> 
> Jan 18 19:07:28 linux spamd[30216]: spamd: result: Y 14 -
> AWL,BAYES_99,DCC_CHECK,DIGEST_MULTIPLE,HTML_IMAGE_ONLY_20,HTML_IMAGE_RATIO_06,HTML_MESSAGE,HTML_SHORT_LINK_IMG_3,MIME_HTML_ONLY,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E8_51_100,RAZOR2_CHECK,RDNS_NONE,TVD_APPROVED,URIBL_BLACK
> scantime=604.3,size=3325,user=brian,uid=1001,required_score=5.5,rhost=localhost,raddr=127.0.0.1,rport=49135,mid=<20090118234025.2fa951cc7...@66v.uwp30.udelmarva.com>,bayes=1.00,autolearn=spam
> 
> The result of this (604 second) scan time is that the MTA ends up giving 
> up waiting after 600 seconds and the scan result is essentially wasted.  
> No doubt some kind of "remote" test is taking an excessive amount of time.

When did you sa-update for last time?
How many processes are you running in parallel? Aren't you running out of
memory?

> a) determine why the scan time is so long, after the fact (i.e. I could
>try to run the same spam through a "spamassassin -D [-t]" but there is
>no guarantee that whatever took so long the first time through will
>again take so long)?

try running spamasssin with -L option

> b) reduce some timeouts of some particular tests so that the total test
>time does not exceed a reasonable threshold?

razor,pyzor,dcc,spf,dkim,rbl have their timeouts (*_timeout), see their
(or SpamAssassin) docs.

-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Microsoft dick is soft to do no harm

excessive scan time

[Brian J. Murrell]

I'm running 3.2.4(-1ubuntu1) of spamassassin here and have been noticing 
some excessive scan times.  i.e.:

Jan 18 19:07:28 linux spamd[30216]: spamd: result: Y 14 - 
AWL,BAYES_99,DCC_CHECK,DIGEST_MULTIPLE,HTML_IMAGE_ONLY_20,HTML_IMAGE_RATIO_06,HTML_MESSAGE,HTML_SHORT_LINK_IMG_3,MIME_HTML_ONLY,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E8_51_100,RAZOR2_CHECK,RDNS_NONE,TVD_APPROVED,URIBL_BLACK
 
scantime=604.3,size=3325,user=brian,uid=1001,required_score=5.5,rhost=localhost,raddr=127.0.0.1,rport=49135,mid=<20090118234025.2fa951cc7...@66v.uwp30.udelmarva.com>,bayes=1.00,autolearn=spam

The result of this (604 second) scan time is that the MTA ends up giving 
up waiting after 600 seconds and the scan result is essentially wasted.  
No doubt some kind of "remote" test is taking an excessive amount of time.

How can I:

a) determine why the scan time is so long, after the fact (i.e. I could
   try to run the same spam through a "spamassassin -D [-t]" but there is
   no guarantee that whatever took so long the first time through will
   again take so long)?
b) reduce some timeouts of some particular tests so that the total test
   time does not exceed a reasonable threshold?

Thanx,
b.

RE: How to analyze scan time

[ram]

On Thu, 2007-09-13 at 07:33 -0500, Skip wrote:

> This is probably going to be a stupid question, but how do I go about
> implementing patches like this?  Should this file be copied in place of the
> file located here?:
> 
> /usr/lib/perl5/site_perl/5.8.0/Mail/SpamAssassin/



No problems. You could have used "man patch", but man pages arent that
too friendly 

The way to do it is 

a) you copy the patch to any directory 
wget -O -
http://issues.apache.org/SpamAssassin/attachment.cgi?id=4081&action=view
> /tmp/sa.patch 


b)  go to your site_perl directory 
That is in your case , most probably 
cd /usr/lib/perl5/site_perl/5.8.0/

c) Run the patch
patch -b -p0 < /tmp/sa.patch 

restart your spam scanning daemon ( spamd,Mailscanner,milter etc ) 

Thanks
Ram








> 
> - Skip
> 

Re: How to analyze scan time

[Luis Hernán Otegui]

2007/9/13, Skip <[EMAIL PROTECTED]>:
> This is probably going to be a stupid question, but how do I go about
> implementing patches like this?  Should this file be copied in place of the
> file located here?:
>
> /usr/lib/perl5/site_perl/5.8.0/Mail/SpamAssassin/
>
> - Skip
>
>
ASQ (another stupid question): is there any chance for Mark's patch to
become a stable part of SA in 3.2.4?

Thanks,


Luis

-- 
-
GNU-GPL: "May The Source Be With You...
Linux Registered User #448382.
When I grow up, I wanna be like Theo...
-

RE: How to analyze scan time

[Skip]

This is probably going to be a stupid question, but how do I go about
implementing patches like this?  Should this file be copied in place of the
file located here?:

/usr/lib/perl5/site_perl/5.8.0/Mail/SpamAssassin/

- Skip

Re: How to analyze scan time

[ram]

On Thu, 2007-09-13 at 00:26 -0400, François Rousseau wrote:
> Hello,
> 
> I have recently change my SA server for another really similar server
> but many software version have change between the 2 servers (include
> SA 3.1.7 --> 3.2.3)
> 
> My old server scan the messages much faster (around 3-4 seconds vs
> 7.5-10 seconds).
> 
well that issue has been there for everyone who upgraded 

This is now patched 
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5589 

I think  you should also patch your install 


Then run spamassassin -D -t < someemail.eml
And see all the query scan times.  Disable all the DNSes that take more
than 3-4 seconds 

That worked for me , as usual YMMV  :-) 


Thanks
Ram

How to analyze scan time

[François Rousseau]

Hello,

I have recently change my SA server for another really similar server
but many software version have change between the 2 servers (include
SA 3.1.7 --> 3.2.3)

My old server scan the messages much faster (around 3-4 seconds vs
7.5-10 seconds).

This is not a critical issue for me because I'm still under the limit
of my server but I'm curious to know why it take longer to scan and
what part of my scan take longer.  Of course, I also want to find a
way to optimize my scan process.

What I search it's a way to know, for exemple, that my clamav scan
have take 2 seconds, the rules processing have take X seconds, the X
module have take X seconds, ...

Any idea?

---
SA 3.2.3 from source
Debian Etch

Thanks,
François Rousseau

Scan Time Problem After Upgrade

[gascione]

We began upgrading to 3.2.3 from 3.2.1. There are 5 machines. On the first
machine prior to the upgrade the average scan time for a message was 2 to 4
seconds, fairly consistent at 2.5 or so seconds. After upgrading the same
systems now have a scan time of 7 or more seconds. Not sure if this was a
message related issue so we upgraded the second machine that was running the
same way. Again, after the upgrade the scan times doubled our more. We then
downgraded back to 3.2.1 on the first machine and the scan times went back
to the lower time.

Everything looks like it's configured the same way. Anyone know what we may
be experiencing?

GA

-- 
View this message in context: 
http://www.nabble.com/Scan-Time-Problem-After-Upgrade-tf4429657.html#a12636664
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.

Re: Slow scan time

[Loren Wilton]

So...  They fabricated a mail with a fake mailer, added some fake 
double-bounce headers, and then sent it to your postmaster account?  That 
seems exceptionally unclever of some idiot.


I just ran that twice here on an unloaded system and it took about 6.5 
seconds.  Only net rule it hit was spamcop.  I would guess that the net 
tests are slowing it down for you, but I'm not real sure why.


BTW, that hit 12 points here.  Of course about 8 of those points are local 
rules.


Still, I can see several potential rules that could be written for those 
puppies, if the signs remain consistent.


   Loren

Slow scan time

[Craig Morrison]

http://www3.2cah.com/spam/sa_slowhtml.txt

I got inundated with messages similar to this today. The average scan 
time here for these is 25+ seconds when the box is under _low_ load.


My guess is that it has to do with the number of URLs.

Any thoughts on this?

--
Craig

Re: Scan time?

[Loren Wilton]

Depends on your hardware, generally available memory for the most part.
Also depends on network lookup time if you have net tests enabled.

If you have less than 512M, I'd consider 6 seconds fine for some messages.
Likewise if you have net tests enabled.

Loren

Scan time?

[MIKE YRABEDRA]

What is an acceptable scan time for SA?

Mine on average runs between 1-6 seconds, shouldn't it be more like 1-3
seconds?