Mark Martinec ha scritto:
> Giuseppe,
>
>
>> Hi Mark, while I was trying to figure out and measure the performance
>> of a amavisd+postfix+spamassassin system, to see whether the
>> TimeElapsedSpamCheck was reliable, I was coming out with this patch
>> (see attach) to know which plugin is effectively loaded by amavisd. In
>> fact I haven't found such information in logs, and for testing purpose
>> one have to put at least in the same conditions as of the amavisd
>> user, and need to know all the plugins effectively loaded
>> (often the ones from the amavis user are not the same as the
>> system-wide SA). Can you include (or maybe a better one) it in the
>> next release?
>>
>
> Ok, I'll add the:
>
> do_log(2, "SpamAssassin loaded plugins: %s", join(', ', sort
> map {my($n)=ref $_; $n=~s/^Mail::SpamAssassin::Plugin:://; $n} @plugins));
>
great, a lot better than mine... :-)
>
>> Apart this I found that on a typical Dual Quad Xeon 5420 2.5Ghz
>> (indeed the extra 7 cores aren't used very much unless you increase
>> the concurrency level), a typical run with:
>>
>> "spamassassin -t -D < mailspam.msg"
>>
>> takes around 10s, so the 7-8s shown for TimeElapsedSpamCheck by
>> amavisd-agent should be reliable, considering also that it should run
>> in daemon mode so the loading time shouldn't be counted.
>>
>
> Right.
>
>
>> Of these 10s, 7-8 seconds are usually taken by DNSBL SA checking;
>>
>
> That's about reasonable, perhaps a bit on the high side.
> I use the settings:
> rbl_timeout 5.5 2.2
> rbl_timeout 7.7 3.3 open-whois.org
>
will try these timeouts too.
> Note that DNS-based checks only add to latency and have
> little effect on throughput. Extra latency can easily be
>
I didn't understand this. Why should have a little effect on troughput?
If a mail
is processed in 3s instead of 15s (apart concurrent sending), means that
I can deliver 20 mail/min, instead of 4 mail/min. If I have 8 processes|core
then I can deliver 20*8 mail/min instead of just 4*8.
> compensated for by running more concurrent processes
> (at the expense of more memory consumed).
>
>
>> disabling all the network checking (i.e. using spamassassin -t -D -L
>> <mailspam.msg), and redoing the tests, this 10s usually drops around
>> 3.9s, while further using the compile (Rule2XS) feature, these 3.9s
>> drops to 2.6s (40% better).
>>
>
> Btw, using SpamAssassin 3.3 with amavisd adds a SA timing report
> to the amavisd log (at level 2 or above), e.g.:
>
> (80089-08) TIMING-SA total 1348 ms - parse: 2 (0.2%),
> extract_message_metadata: 8 (0.6%), get_uri_detail_list: 0.62 (0.0%),
> tests_pri_-1000: 10 (0.7%), tests_pri_-950: 0.90 (0.1%),
> tests_pri_-900: 1.08 (0.1%), tests_pri_-400: 20 (1.5%),
> check_bayes: 18 (1.3%), tests_pri_0: 1097 (81.3%),
> check_dkim_adsp: 62 (4.6%), check_spf: 217 (16.1%),
> poll_dns_idle: 188 (14.0%), check_razor2: 541 (40.1%),
> check_dcc: 166 (12.3%), check_pyzor: 0.03 (0.0%),
> tests_pri_500: 10 (0.7%), tests_pri_899: 98 (7.3%),
> check_crm114: 97 (7.2%), tests_pri_1000: 34 (2.5%),
> total_awl: 33 (2.4%), check_awl: 17 (1.3%), update_awl: 10 (0.7%),
> learn: 30 (2.2%), get_report: 2 (0.2%)
>
>
Thanks. Will give a try to the SA 3.3 SVN.
>> Now I'd like to do the same evaluation for
>> amavisd, i.e. with all the network testing disabled. Is there a way to
>> disable for instance the DNSBL and all the remote testing in one shot
>> from within amavisd.conf (more or less like the -L option does for the
>> spamassassin standalone command)?
>>
>
> $sa_local_tests_only = 1;
>
>
Ah, right!
>> I tried to add sort of
>> "skip_rbl_checks 1" to user_prefs but isn't effective.
>>
>
> It'd better be in local.cf, but it should be effective.
>
>
well, local.cf will be local to every user, while using
/var/lib/amavis/.spamassassin/user_prefs should affect only amavisd
which is running under the amavis username.
>> In this way
>> should be possible to evaluate the theoretical peak limit (in
>> msgs/day) that a server could sustain per CPU/core and tune up the
>> performance accordingly under the best conditions. BTW, what other
>> values people reach under latest spamassassin (3.2.5) and amavisd-new
>> (2.6.1) in these subject?
>>
>
> Your figures look about right. It depends much on what additional
> rules you have, and what additional plugins are in use.
>
>
>> At this point I would like to ask you whether the following feature
>> could be added and if it's worthwhile: it seems that most of the spam
>> is sent during night and the weekend.
>>
>
> I'd rather say that an absolute rate of spam doesn't change much,
> but as there is less regular mail traffic at night, the ratio
> of ham/spam does change.
>
>
>> Would be possible to add
>> amavisd-new a feature to have it's behaviour selective according to
>> the time: e.g. since 75% of SA time is spent over DNS|RBL checks,
>>
>
> DNS/RBL checks are insignificant as far as aggregate mail throughput
> is concerned.
>
what you mean as "aggregate mail troughput"?
>
>> and often a server need to be much more responsive in sending and
>> receiving mails during daylight, would be fine to have a sort of two
>> profiles in amavisd-new: one for day (with maybe all network tests
>> disabled) and one for night and weekends under which the network
>> testing are disabled. In this way, when during daylight much server
>> power and responsiveness is needed for delivering mails it won't spent
>> much time in DNS|RBLs (ok, it will pass more spam), while during night
>> it could use all the tests available.
>>
>
> It would be possible, but I don't think it is worth it.
> If a server can't cope with mail rate (with some reserve),
> one needs a bigger machine, and not sacrifice quality of
> filtering.
>
Hmmm, the problem is rather about the checks for local delivery which should
be boosted (i.e. a mail originating from LAN and directed to the same
host), for which I'm pretty confused.
I could add some whitelist, but I'm not sure where the check is
performed, i.e. which
field are checked? The From (in that case it would fail as spammers spam
recently from yourself)?
The "Received" chain (in this case what if the mails are feeded trough a
local fetchmail server?)
Theoretically I could have a 2nd content filter in postfix master.cd for
the LAN IPs only, but in that
case I think that amavisd is bypassed and I don't won't to totally
disable it (it could still
be useful for clamav, etc.). And what if for some reason, the DNS works
only locally
(because remotely have a link down), i.e. how to have the system be
DNS-fail-over for
local mail delivery (originating and destination) without having long
timeouts due the DNS timeouts
in resolving addresses (e.g. you can't resolve spamhaus.org)? Or should
this be handled trough custom
policy banks + XFORWARD (and can $sa_local_tests_only = 1; be local to a
single policy bank)?
I think that a similar default configuration could be added in the
amavisd RPM that I usually contribute too,
as to me seems a pretty common scenario.
>> Such approach theoretically could be made with some sed|perl "search &
>> replace" scripts for user_prefs, commenting or uncommenting on the fly
>> some test, and then restarts the amavisd process, but this sound a
>> sort of ugly and awkward, and would be much more elegant inside amavisd.
>>
>
> Changing the $sa_local_tests_only is a bit tricky, as it is given
> to SpamAssassin during its initialization. Some other settings
> may be easier to customize through policy banks, loading a policy
> bank could be done by a custom huuk based on hour-of-the-day
> if needed.
>
custom huuk based on timestamp?
Thanks.
Bye
Giuseppe.
------------------------------------------------------------------------------
Stay on top of everything new and different, both inside and
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today.
Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
AMaViS-user mailing list
AMaViS-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
