Re: cron question / process queue
On Fri, March 7, 2008 9:57 am, Steven W. Orr said: > > Still sounds like a job for "at", no? > If a job is scheduled with cron, and something unexpected causes it to miss running or the script to crash before it completes, then it still gets run at the next scheduled time. If the script uses "at" to dynamically schedule its next run, then the same types of problem mean the script no longer gets scheduled until someone notices that it isn't being run. In my opinion, "at" is fine for scheduling one-off tasks, but regularly scheduled tasks really need something like cron. -- John Abreau / Executive Director, Boston Linux & Unix IM: [EMAIL PROTECTED] / [EMAIL PROTECTED] / [EMAIL PROTECTED] / [EMAIL PROTECTED] Email [EMAIL PROTECTED] / WWW http://www.abreau.net / PGP-Key-ID 0xD5C7B5D9 PGP-Key-Fingerprint 72 FB 39 4F 3C 3B D6 5B E0 C8 5A 6E F1 2C BE 99 -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: cron question / process queue
Michael ODonnell writes:
> Of course, synchronization-wise things get interesting as soon
> as any job takes longer than the hourly interval because the next
> hour's queue-runner has to know not to interfere with the one that's
> still active. A lock-file should be enough to handle that, though.
>
> It's a SMOP - here's the whole thing in Perl:
>
>$_()[EMAIL PROTECTED];time~[HiMom]%^$!!__chomp|\+{&};
Interesting Perl-fu. I probably would have done it this way though:
perl -e 'use LockFile::Simple qw(lock unlock);
# we require >=2 args
# 1st arg is lockfile
# 2nd...Nth are executed when we have the lock
die "usage: lock lockfile arg1 arg2\n" if (scalar(@ARGV) < 2);
$lockfile=shift;
lock($lockfile) && system(@ARGV);
unlock($lockfile)' \
/tmp/my-lockfile \
/bin/sh -c 'sleep 10 ; echo elvis has left the building'
Kind regards,
--kevin
--
GnuPG ID: B280F24EDon't you know there ain't no devil,
alumni.unh.edu!kdcthere's just God when he's drunk?
http://kdc-blog.blogspot.com/ -- Tom Waits
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: cron question / process queue
> I don't see it. I have monthly, weekly and daily jobs that I want
> to run sequentially, in that order, starting at 8pm. How would
> I do that using 'at'? Preferably without making the actual jobs
> know about each other, e.g. not teaching the weekly job that it
> should be followed by the daily job.
That's why I like the approach I suggested - the only thing that
your daily/weekly/monthly processes have to do when they decide
it's time for a run is something like:
cp appropriateScriptFile.sh /yourQueueDir/`date
'+%Y%m%d%H%M%S'`-$$+duration.sh
...and then exit. Those scripts would end up having names like
20080307104617-23987+2h.sh which would cause them to sort lexically
in time order, resulting in FIFO behavior. Then your hourly
job would only have to decide whether it was appropriate to run
anything it found there, based on the current time and maybe also
the estimated duration as encoded in the filename. Most hours,
it would just exit because there'd be nothing to do or because
you don't want to run something during business hours. As each
script completed your hourly job could then determine whether it
was appropriate to start another one immediately (if present) or to
instead simply exit, deferring any remaining scripts until later.
No task would know any of the others even exist and your queue-runner
only has to consider one script at a time, independent of the others.
Of course, synchronization-wise things get interesting as soon
as any job takes longer than the hourly interval because the next
hour's queue-runner has to know not to interfere with the one that's
still active. A lock-file should be enough to handle that, though.
It's a SMOP - here's the whole thing in Perl:
$_()[EMAIL PROTECTED];time~[HiMom]%^$!!__chomp|\+{&};
___
gnhlug-discuss mailing list
gnhlug-discuss@mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: cron question / process queue
Kent Johnson wrote: > I made a combined job that basically does this: > if it is the first of the month: >run monthly job and wait for completion > if it is Saturday: >run the weekly job and wait for completion > run the daily job > > The combined job is scheduled for 8pm execution with cron. Seems to work > so far... I'd use a short wrapper in your scripting language of choice for the cron job: while lockfile exists, wait some amount of time and check again... write a lockfile. do the actual job/command remove the lockfile. This will work if the script takes less than two cycles to complete, but may run out of order if it takes longer. HTH, Brian ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: cron question / process queue
You can also use dotlockfile, lockfile, or dotlock to create a lock file for each task that is running. Then have the 'daily' script kick off the right process for whichever lockfile is not there. Of course, the task has to remove the lock file when its done. --Bruce Kent Johnson wrote: > Steven W. Orr wrote: > >> Still sounds like a job for "at", no? >> > > I don't see it. I have monthly, weekly and daily jobs that I want to run > sequentially, in that order, starting at 8pm. How would I do that using > 'at'? Preferably without making the actual jobs know about each other, > e.g. not teaching the weekly job that it should be followed by the daily > job. > > I made a combined job that basically does this: > if it is the first of the month: >run monthly job and wait for completion > if it is Saturday: >run the weekly job and wait for completion > run the daily job > > The combined job is scheduled for 8pm execution with cron. Seems to work > so far... > > Kent > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: cron question / process queue
Steven W. Orr wrote: > Still sounds like a job for "at", no? I don't see it. I have monthly, weekly and daily jobs that I want to run sequentially, in that order, starting at 8pm. How would I do that using 'at'? Preferably without making the actual jobs know about each other, e.g. not teaching the weekly job that it should be followed by the daily job. I made a combined job that basically does this: if it is the first of the month: run monthly job and wait for completion if it is Saturday: run the weekly job and wait for completion run the daily job The combined job is scheduled for 8pm execution with cron. Seems to work so far... Kent ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: cron question / process queue
On Friday, Mar 7th 2008 at 07:49 -, quoth Kent Johnson: =>Steven W. Orr wrote: =>> This is a classic question: How to run something on a periodic basis that =>> may take longer to execute than the interval between the next occurance. =>> Think of it not as a task that needs to be run at an interval so much as a =>> task that needs to be rescheduled after it finally completes. =>> =>> If that description is appropriate for what you're doing then you might want =>> to consider not using cron in the first place. Look at using at(1) instead. =>> Your script that gets run at a certain time would then requeue itself based =>> on when it finished. => =>Thanks for the pointer. That is not quite my situation - I need to queue a =>different job when the first one finishes, and the job to queue (if any) =>depends on the date. => =>I ended up making a program that combines all three jobs and runs the correct =>ones in sequence based on the date. => =>Kent => Still sounds like a job for "at", no? -- steveo at syslang dot net TMMP1 http://frambors.syslang.net/ Do you have neighbors who are not frambors? Steven W. Orr ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: cron question / process queue
Steven W. Orr wrote: > This is a classic question: How to run something on a periodic basis that > may take longer to execute than the interval between the next occurance. > Think of it not as a task that needs to be run at an interval so much as a > task that needs to be rescheduled after it finally completes. > > If that description is appropriate for what you're doing then you might > want to consider not using cron in the first place. Look at using at(1) > instead. Your script that gets run at a certain time would then requeue > itself based on when it finished. Thanks for the pointer. That is not quite my situation - I need to queue a different job when the first one finishes, and the job to queue (if any) depends on the date. I ended up making a program that combines all three jobs and runs the correct ones in sequence based on the date. Kent ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: cron question / process queue
On Tuesday, Mar 4th 2008 at 12:53 -, quoth Kent Johnson: =>Hi, => =>I have a server that runs regular daily, weekly and monthly updates, all =>scheduled with cron. => =>The updates are lengthy - the weekly update now takes about 8 hours - =>and I would like for them not to overlap. I also want them to run at =>night, for some reasonable value of night, so I can't just schedule them =>far apart. => =>I can use the historical run times to schedule the jobs to prevent them =>overlapping, but I was wondering if there is a more robust and flexible =>way to do this; some way to say => daily - queue the daily update => weekly - queue the weekly update => monthly - queue the monthly update =>and have the updates run sequentially. => =>I guess it would not be hard to write a Python program to do determine =>which updates to run and start the correct processes, but I'm wondering =>if there is some system facility that would help. This is a classic question: How to run something on a periodic basis that may take longer to execute than the interval between the next occurance. Think of it not as a task that needs to be run at an interval so much as a task that needs to be rescheduled after it finally completes. If that description is appropriate for what you're doing then you might want to consider not using cron in the first place. Look at using at(1) instead. Your script that gets run at a certain time would then requeue itself based on when it finished. BTW, the at command has some really sexy timespec abilities. You can say things like "TOMORROW at 4PM" or "NEXT Friday". -- Time flies like the wind. Fruit flies like a banana. Stranger things have .0. happened but none stranger than this. Does your driver's license say Organ ..0 Donor?Black holes are where God divided by zero. Listen to me! We are all- 000 individuals! What if this weren't a hypothetical question? steveo at syslang.net ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: cron question / process queue
Just thinking out loud here, but one extremely low-tech way to do it might be to have your daily/weekly/monthly cron jobs create a work queue by depositing disposable copies of the desired scripts into a special directory with names guaranteed to be sequential and unique (like, say, /yourQueueDir/`date '+%Y%m%d%H%M%S'`-$$+duration.sh) and then have an hourly cron job process that queue at the times you deem suitable. If you encode rough guesses about durations into the script names (or maybe have multiple queue dirs?) your queue-processor might then be able to make decisions about whether it was practical to start a given job. Each script would be deleted upon completion. ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
cron question / process queue
Hi, I have a server that runs regular daily, weekly and monthly updates, all scheduled with cron. The updates are lengthy - the weekly update now takes about 8 hours - and I would like for them not to overlap. I also want them to run at night, for some reasonable value of night, so I can't just schedule them far apart. I can use the historical run times to schedule the jobs to prevent them overlapping, but I was wondering if there is a more robust and flexible way to do this; some way to say daily - queue the daily update weekly - queue the weekly update monthly - queue the monthly update and have the updates run sequentially. I guess it would not be hard to write a Python program to do determine which updates to run and start the correct processes, but I'm wondering if there is some system facility that would help. Thanks, Kent ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
