Re: limiting resources

[Frederik Dannemare]

On Tuesday 21 September 2004 11:51, martin f krafft wrote:
> also sprach Dominique Dumont <[EMAIL PROTECTED]> 
[2004.09.21.1141 +0200]:
> > > I forgot to mention that spamc is of course being used. However,
> > > spamc does not provide the means to strip markup from email
> > > messages, nor can it report to razor. My users don't need
> > > spamassassin for spam checking, but for spam processing.
> >
> > Have you tried the '-m' switch on spamd ?
>
> Sure. As I said, spamd is not the problem. It's users executing
>  multiple times, with the last time being
> spamassassin that almost killed the system.
>
> > You may need to use perl5.8.x to avoid signal handling bugs
> > mentionned in spamd man page. See "PERL_SIGNALS" in perlrun and
> > "Deferred Signals (Safe Signals)" in perlipc for details.
>
> All being done here.

Back in 2002 I asked on lkml a question similar to yours 
. Rik 
Van Riel mentioned his fairsched , but it 
looks like it's not been ported to recent kernels.

Marc-Christian Petersen pointed me to another patch 
. But again, it's 
not been ported to recent kernels, it seems.

B/R,
-- 
Frederik Dannemare | mailto:[EMAIL PROTECTED]
http://qa.debian.org/developer.php?login=Frederik+Dannemare
http://frederik.dannemare.net | http://www.linuxworlddomination.dk
Key fingerprint: BB7B 078A 0DBF 7663 180A  F84A 2D25 FAD5 9C4E B5A8


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: limiting resources

[martin f krafft]

also sprach Dominique Dumont <[EMAIL PROTECTED]> [2004.09.21.1141 +0200]:
> > I forgot to mention that spamc is of course being used. However,
> > spamc does not provide the means to strip markup from email
> > messages, nor can it report to razor. My users don't need
> > spamassassin for spam checking, but for spam processing.
> 
> Have you tried the '-m' switch on spamd ?  

Sure. As I said, spamd is not the problem. It's users executing
 multiple times, with the last time being
spamassassin that almost killed the system.

> You may need to use perl5.8.x to avoid signal handling bugs
> mentionned in spamd man page. See "PERL_SIGNALS" in perlrun and
> "Deferred Signals (Safe Signals)" in perlipc for details.

All being done here.

-- 
Please do not CC me when replying to lists; I read them!
 
 .''`. martin f. krafft <[EMAIL PROTECTED]>
: :'  :proud Debian developer, admin, and user
`. `'`
  `-  Debian - when you have better things to do than fixing a system
 
Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver!


signature.asc
Description: Digital signature

Re: limiting resources

[Dominique Dumont]

martin f krafft <[EMAIL PROTECTED]> writes:

> I forgot to mention that spamc is of course being used. However,
> spamc does not provide the means to strip markup from email
> messages, nor can it report to razor. My users don't need
> spamassassin for spam checking, but for spam processing.

Have you tried the '-m' switch on spamd ?  

You may need to use perl5.8.x to avoid signal handling bugs mentionned
in spamd man page. See "PERL_SIGNALS" in perlrun and "Deferred Signals
(Safe Signals)" in perlipc for details.

HTH

-- 
Dominique Dumont 
"Delivering successful solutions requires giving people what they
need, not what they want." Kurt Bittner


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: limiting resources

[martin f krafft]

also sprach Adam Aube <[EMAIL PROTECTED]> [2004.09.21.0222 +0200]:
> What if you wrote a shell wrapper around SpamAssassin that lowered
> its priority before running it, then had your users run this
> wrapper instead?

... then this may solve the problem temporarily, until the next
comes along. apart, the problems are not with CPU but rather with
locking of the spamassassin database and access to the harddisk. Try
running 40 processes wanting to do the latter at the same time! in
this light, i have been highly unimpressed by the 2.6 scheduler.

-- 
Please do not CC me when replying to lists; I read them!
 
 .''`. martin f. krafft <[EMAIL PROTECTED]>
: :'  :proud Debian developer, admin, and user
`. `'`
  `-  Debian - when you have better things to do than fixing a system
 
Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver!


signature.asc
Description: Digital signature

Re: limiting resources

[martin f krafft]

also sprach Stefan O'Rear <[EMAIL PROTECTED]> [2004.09.21.0246 +0200]:
> Now, if you make the script NOPASSWD sudo -u sa_learn able, and
> make sa_learn only executable by user sa_learn, only one will be
> running at a time.

yeah, and it would force everyone to use the same database, which
is not an option and defeats the purpose.

-- 
Please do not CC me when replying to lists; I read them!
 
 .''`. martin f. krafft <[EMAIL PROTECTED]>
: :'  :proud Debian developer, admin, and user
`. `'`
  `-  Debian - when you have better things to do than fixing a system
 
Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver!


signature.asc
Description: Digital signature

Re: limiting resources

[Adam Aube]

Stefan O'Rear wrote:

> How about a serializing wrapper?

> Now, if you make the script NOPASSWD sudo -u sa_learn able, and make
> sa_learn only executable by user sa_learn, only one will be running at a
> time.

Or you could make the script setGID, then remove world read and execute
rights on the binary. Create a special group just for running this wrapper,
and make it the group of the script and binary.

Adam


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: limiting resources

[Stefan O'Rear]

On Mon, Sep 20, 2004 at 08:22:13PM -0400, Adam Aube wrote:
> martin f krafft wrote:
> > Every now and then, coincidence will have it that 10 or 20 users
> > invoke spamassassin at the same time. Spamassassin is a resource
> > hog and that will cause the machine to basically become unusable,
> > with the load going to 30 and higher.
>  
> > Obviously, I have put limits on local deliveries so that postfix
> > itself does not ever screw up the machine. Now I need to limit the
> > local users. Other than PAM limits, which seems to only work on
> > number of processes, is there a way to dynamically limit the load
> > a shell-user can cause? I am talking load-balancing ... give each
> > user 100% unless others want slices too.
> 
> What if you wrote a shell wrapper around SpamAssassin that lowered its
> priority before running it, then had your users run this wrapper instead?
> 
> Adam

How about a serializing wrapper?

e.g:

#! /bin/sh
set -e
trap 'rm -f /var/tmp/spam.lock' 1 2 15 ERR
lockfile /var/tmp/spam.lock
sa_learn "$@"
rm -f /var/tmp/spam.lock

(Remember to fix errors)

Now, if you make the script NOPASSWD sudo -u sa_learn able, and make
sa_learn only executable by user sa_learn, only one will be running at a
time.

-- 
The world's most effective spam filter:
ln -sf /dev/full /var/mail/$USER


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: limiting resources

[Adam Aube]

martin f krafft wrote:
> Every now and then, coincidence will have it that 10 or 20 users
> invoke spamassassin at the same time. Spamassassin is a resource
> hog and that will cause the machine to basically become unusable,
> with the load going to 30 and higher.
 
> Obviously, I have put limits on local deliveries so that postfix
> itself does not ever screw up the machine. Now I need to limit the
> local users. Other than PAM limits, which seems to only work on
> number of processes, is there a way to dynamically limit the load
> a shell-user can cause? I am talking load-balancing ... give each
> user 100% unless others want slices too.

What if you wrote a shell wrapper around SpamAssassin that lowered its
priority before running it, then had your users run this wrapper instead?

Adam


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: limiting resources

[martin f krafft]

also sprach Brett Carrington <[EMAIL PROTECTED]> [2004.09.20.1806 +0200]:
> I had a similar problem. My solution was to disallow the use of
> spamassassin and force users to use spamc. This requires your
> server to run spamd as well. The spamassassin docs have info on
> the benefits and risks of this method.

I forgot to mention that spamc is of course being used. However,
spamc does not provide the means to strip markup from email
messages, nor can it report to razor. My users don't need
spamassassin for spam checking, but for spam processing.

-- 
Please do not CC me when replying to lists; I read them!
 
 .''`. martin f. krafft <[EMAIL PROTECTED]>
: :'  :proud Debian developer, admin, and user
`. `'`
  `-  Debian - when you have better things to do than fixing a system
 
Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver!


signature.asc
Description: Digital signature

Re: limiting resources

[Brett Carrington]

On Mon, 20 Sep 2004 16:57:21 +0200, martin f krafft <[EMAIL PROTECTED]> wrote:
> I operate a shell server with about 200 users. Spamassassin powers
> the machine, and users use sa-learn to teach it about spam and ham.
> The machine receives about 5-20 mails per second and the hardware
> keeps the load between 0.15 and 0.30.
> 
> Every now and then, coincidence will have it that 10 or 20 users
> invoke spamassassin at the same time. Spamassassin is a resource
> hog and that will cause the machine to basically become unusable,
> with the load going to 30 and higher. It usually takes it about half
> an hour to get back to a usable state (after all backup MX delivered
> the queued mail, causing it to receive up to 30 mails per second.
> 
> Obviously, I have put limits on local deliveries so that postfix
> itself does not ever screw up the machine. Now I need to limit the
> local users. Other than PAM limits, which seems to only work on
> number of processes, is there a way to dynamically limit the load
> a shell-user can cause? I am talking load-balancing ... give each
> user 100% unless others want slices too.
> 
> Do you know of a solution I could employ?
I had a similar problem. My solution was to disallow the use of
spamassassin and force users to use spamc. This requires your server
to run spamd as well. The spamassassin docs have info on the benefits
and risks of this method.


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]