Re: Limiting process startup

2003-09-05 Thread muppet 
On Friday, September 5, 2003, at 10:32  AM, Dean Wilson wrote:

Randal L. Schwartz wrote:

Is it just me or does anybody else have trouble reading the dark 
purple on
almost as dark purple in the top right corner of that page?
it may be just you.  i see black on medium-dark cornflower-ish blue.  
have you calibrated your monitor lately?  ;-)

Re: Limiting process startup

2003-09-05 Thread Shevek 
On Fri, 5 Sep 2003, Paul Crowley wrote:

> Shevek <[EMAIL PROTECTED]> writes:
> 
> > Isn't this the kind of thing grid engine and its ilk have been designed to 
> > solve? There are many such architectures already written, I'm suprised 
> > noone has mentioned it. There's not much point growing your own using IPC.
> 
> Do you mean http://gridengine.sunsource.net/ - if so doesn't it seem
> overkill for such a small application?

I don't know, I tend to work with slightly larger things. However, before 
grid engine, there were smaller and lesser equivalents. I'm out of touch.

I'll go away now.

S.

-- 
Shevekhttp://www.anarres.org/
I am the Borg. http://www.gothnicity.org/

Re: Limiting process startup

2003-09-05 Thread Dean Wilson 
Randal L. Schwartz wrote:
> Dean> Is it just me

Yes :)

> The text is black on #9797FF.  If you can't read text with that
> contrast, maybe you oughta check the gamma on your screen. :)

Just tried it on a laptop and i could read it. I've got the brightness
cranked up so it might be time for a new monitor, i've been having problems
playing Half-life (Well i'm going to blame the monitor for it!)

  Dean
-- 
Profanity is the one language all programmers understand
--- Anon

Re: Limiting process startup

2003-09-05 Thread Randal L. Schwartz 
> "Dean" == Dean Wilson <[EMAIL PROTECTED]> writes:

Dean> Randal L. Schwartz wrote:
>> Damn.  You made it, too.  I also did a similar thing with POE as well,
>> which might make more sense here...
>> 


Dean> Is it just me or does anybody else have trouble reading the dark
Dean> purple on almost as dark purple in the top right corner of that
Dean> page?

The text is black on #9797FF.  If you can't read text with that
contrast, maybe you oughta check the gamma on your screen. :)

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[EMAIL PROTECTED]> http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: Limiting process startup

2003-09-05 Thread Dean Wilson 
Randal L. Schwartz wrote:
> Damn.  You made it, too.  I also did a similar thing with POE as well,
> which might make more sense here...
> 


Is it just me or does anybody else have trouble reading the dark purple on
almost as dark purple in the top right corner of that page?

  Dean
-- 
Profanity is the one language all programmers understand
--- Anon

Re: Limiting process startup

2003-09-05 Thread Randal L. Schwartz 
> "Chris" == Chris Benson <[EMAIL PROTECTED]> writes:

Chris> Randal did a column on that that I've just pointed out to someone trying to
Chris> do something similar ...

Chris> http://www.stonehenge.com/merlyn/LinuxMag/col15.html

Chris> (Trying to answer before Randal wakes up :-)

Damn.  You made it, too.  I also did a similar thing with POE as well,
which might make more sense here...



Note the "KIDSMAX => 10", which sets the upper limit on things
being done in parallel.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[EMAIL PROTECTED]> http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: Limiting process startup

2003-09-05 Thread Paul Crowley 
Nick Cleaton <[EMAIL PROTECTED]> writes:

> On Fri, Sep 05, 2003 at 11:28:02AM +0100, Paul Crowley wrote:
> > 
> > What's the cleanest way to make sure at most N processes are doing X
> > at once, and anyone else wishing to do X blocks until one of those N
> > are finished?
> 
> IPC::Semaphore

This appears to be the Right Way.  Thanks!
-- 
  __  Paul Crowley
\/ o\ [EMAIL PROTECTED]
/\__/ http://www.ciphergoth.org/

Re: Limiting process startup

2003-09-05 Thread Paul Crowley 
Shevek <[EMAIL PROTECTED]> writes:

> Isn't this the kind of thing grid engine and its ilk have been designed to 
> solve? There are many such architectures already written, I'm suprised 
> noone has mentioned it. There's not much point growing your own using IPC.

Do you mean http://gridengine.sunsource.net/ - if so doesn't it seem
overkill for such a small application?
-- 
  __  Paul Crowley
\/ o\ [EMAIL PROTECTED]
/\__/ http://www.ciphergoth.org/

Re: Limiting process startup

2003-09-05 Thread Shevek 
On Fri, 5 Sep 2003, Paul Crowley wrote:

> It was nice to meet people at the pub yesterday and play Set!
> 
> This isn't really Perl-specific, it's Unix specific, but I plan to
> implement the solution in Perl and you seem like a good crowd to ask.
> 
> What's the cleanest way to make sure at most N processes are doing X
> at once, and anyone else wishing to do X blocks until one of those N
> are finished?

Isn't this the kind of thing grid engine and its ilk have been designed to 
solve? There are many such architectures already written, I'm suprised 
noone has mentioned it. There's not much point growing your own using IPC.

S.

-- 
Shevekhttp://www.anarres.org/
I am the Borg. http://www.gothnicity.org/

Re: Limiting process startup

2003-09-05 Thread Paul Crowley 
David Cantrell <[EMAIL PROTECTED]> writes:

> On Fri, Sep 05, 2003 at 11:53:13AM +0100, Nick Cleaton wrote:
> > On Fri, Sep 05, 2003 at 11:28:02AM +0100, Paul Crowley wrote:
> > > What's the cleanest way to make sure at most N processes are doing X
> > > at once, and anyone else wishing to do X blocks until one of those N
> > > are finished?
> > IPC::Semaphore
> 
> That's what I was going to say ;-)  However, seeing that the aim of this
> is to not overload the machine and that only spawning N concurrent
> processes is merely a suggested way of doing this, would it not be
> better to look at the load on the machine and only spawn your process if
> it is sufficiently low?  This is one of the ways in which exim does
> rate-limiting, and I find it works very nicely.

That's what I tried first, but loadavg is averaged over a minute or
so.  So you'd get a quiet minute, and the loadavg would drop below the
threshhold, the gate would be opened, a million users would arrive and
hammer the machine to death.  Eventually the loadavg would catch up
and climb to something sky-high, locking everyone out, then slowly
drop while the machine did nothing.  Repeat until dead...
-- 
  __  Paul Crowley
\/ o\ [EMAIL PROTECTED]
/\__/ http://www.ciphergoth.org/

Re: Limiting process startup

2003-09-05 Thread David Cantrell 
On Fri, Sep 05, 2003 at 11:53:13AM +0100, Nick Cleaton wrote:
> On Fri, Sep 05, 2003 at 11:28:02AM +0100, Paul Crowley wrote:
> > What's the cleanest way to make sure at most N processes are doing X
> > at once, and anyone else wishing to do X blocks until one of those N
> > are finished?
> IPC::Semaphore

That's what I was going to say ;-)  However, seeing that the aim of this
is to not overload the machine and that only spawning N concurrent
processes is merely a suggested way of doing this, would it not be
better to look at the load on the machine and only spawn your process if
it is sufficiently low?  This is one of the ways in which exim does
rate-limiting, and I find it works very nicely.

-- 
David Cantrell | Benevolent Dictator | http://www.cantrell.org.uk/david

Re: Limiting process startup

2003-09-05 Thread Chris Benson 
On Fri, Sep 05, 2003 at 11:28:02AM +0100, Paul Crowley wrote:
> 
> What's the cleanest way to make sure at most N processes are doing X
> at once, and anyone else wishing to do X blocks until one of those N
> are finished?

Randal did a column on that that I've just pointed out to someone trying to
do something similar ...

http://www.stonehenge.com/merlyn/LinuxMag/col15.html

(Trying to answer before Randal wakes up :-)

HTH
-- 
Chris Benson

Re: Limiting process startup

2003-09-05 Thread Jon Nangle 
> "Paul" == Paul Crowley <[EMAIL PROTECTED]> writes:

Paul> What's the cleanest way to make sure at most N processes are
Paul> doing X at once, and anyone else wishing to do X blocks until one
Paul> of those N are finished?

Some sort of scoreboard in shared mem maybe? I don't doubt there's an IPC::*
module that would help here.

BTW, trustflow is pretty cool :)

Jon

Re: Limiting process startup

2003-09-05 Thread Mark Fowler 
On Fri, 5 Sep 2003, Paul Crowley wrote:

> The existing way of doing this is a hack: I have a directory with five
> lock files, and it tries to get a lock on each of those five in turn
> before proceeding.

Why not have one file with a counter in it saying how many processes are
currently running.   Obviously processes will need to lock the file
whenever they write to it, but they will only maintain their lock long
enough to do that.  (also once you've checked the file is low enough
you'll need to lock it, recheck it then, then write to it to avoid a race
condition)

> if there are already N sub-processes, I don't want to abort as before, I
> want to block until one of the sub-processes finishes and I can start
> another.

Since the machine is really busy in this case you might simply sleep for
thirty seconds and then see if the counter is free then.

If you're doing this with a webbrowser and someone is waiting for the
results you'll have to keep sending null chars down the connection every
few seconds to avoid things timing out.

In a similar situatation, I have a POE program CGI that's job is simply to
run another process and while it's waiting for the output and to keep
printing data out (normally I print "Still processing, been processing for
".duration(time-$starttime)."\n".)  When my child process is done I
print javascript saying "document.location='$otherprocessesoutputlocation'"

HTH.

Mark.

-- 
#!/usr/bin/perl -T
use strict;
use warnings;
print q{Mark Fowler, [EMAIL PROTECTED], http://twoshortplanks.com/};

Re: Limiting process startup

2003-09-05 Thread Nick Cleaton 
On Fri, Sep 05, 2003 at 11:28:02AM +0100, Paul Crowley wrote:
> 
> What's the cleanest way to make sure at most N processes are doing X
> at once, and anyone else wishing to do X blocks until one of those N
> are finished?

IPC::Semaphore

--
Nick

Limiting process startup

2003-09-05 Thread Paul Crowley 
It was nice to meet people at the pub yesterday and play Set!

This isn't really Perl-specific, it's Unix specific, but I plan to
implement the solution in Perl and you seem like a good crowd to ask.

What's the cleanest way to make sure at most N processes are doing X
at once, and anyone else wishing to do X blocks until one of those N
are finished?

Context: I'm writing a new version of the TrustFlow trust metric for
LiveJournal ( http://www.gothboffs.co.uk/trustflow/trustflow.pl ).
The CGI behind it all forks off a sub-process to do the actual work of
calculating your list.  However, to prevent the machine getting
overloaded, it refuses to do that if 5 such are already running.

The existing way of doing this is a hack: I have a directory with five
lock files, and it tries to get a lock on each of those five in turn
before proceeding.  If it can't lock any of them, it returns an error
to the user ("this machine is overloaded, sorry!").

This is a bit hacky but basically works.  However, I now want to break
part of the work of these calculation processes into sub-processes
that can run in parallel, and I want to do something similar with
these sub-processes, with one change - if there are already N
sub-processes, I don't want to abort as before, I want to block until
one of the sub-processes finishes and I can start another.

This is hard.  I could choose a lock at random and wait for that, but
this is suboptimal - I want to wait until *any* process finishes, not
until one particular one finishes.

Any ideas?
-- 
  __  Paul Crowley
\/ o\ [EMAIL PROTECTED]
/\__/ http://www.ciphergoth.org/