Hi. I have a package R.batch in R.classes
[http://www.maths.lth.se/help/R/R.classes/] to simplify running multiple
batch jobs, which might interest you.
The idea is as follows. You setup a directory structure defining a
'JobBatch';
<path-to>/jobs/
src/
input/
output/
erroneous/
failed/
finished/
interrupted/
running/
todo/
job01/
job02/
job03/
A 'Job' is simply a directory (above job01/, job02/, job03/). Put code
shared by all Job:s in src/. Put code unique to each Job in its job
directory, e.g. job01/setupParameters.R. All *.R files in src/ and then
in job directory are source():ed before a Job is started. When a Job is
run, onRun(job) is called. Thus, you have to define onRun() in src/ with
the option to override it in each job directory.
As soon as the Job is being processed it is moved to running/. When a
Job is successful and completed, onFinally(job) is called and it is
moved to finished/. If a Job is interrupted, say by Ctrl+C or by sending
SIG-INT for elsewhere, onInterrupt() is called and the job is moved to
interrupted/. Similarly, if an error occurs, say, by calling stop(),
onError(job) is called and it is moved to failed/. (If an error occurs
while source():ing the *.R files before starting, the job is moved to
erroneous/). If you call sourceHotCode(job) once in a while in your
onRun(job) code, code in src/hot/ or job01/hot/ will be source():ed *and
*removed. This allows you to fix problems (redefine objective functions
etc) *while running*, say a 10-hour job.
When a Job runs, its working directory is set to "itself", e.g.
running/job01/. Thus, written result files and created images etc will
naturally be save in each job directory. You can also write to
getOutputPath(), which is the output/. Log files are written to
getLogPath(), which defaults to output/.
Each Job can access common data from the input/ path by getInputPath().
Note that in Unix input/ can be a soft link to another directory. To
provide the same functionality under Windows, Windows shortcut files,
say, input.lnk, are recognized and followed if getInputPath() is used.
[This actually holds for other directories too; if multiple batches
share same source code, you can link src/].
Given the above structure, you run all Jobs one by one, by
library(R.batch)
batch <- JobBatch("<path-to>/jobs/")
run(batch)
# wait until all jobs are processed
# Try Ctrl+C, rerun by run(batch).
print(batch) # Gives a summary of the status of all jobs
Logging and everything else is taken care of automatically.
The code is written such it should be possible for several R sessions to
operate on the same batch set simultaneously. Lock files are used to
control for this. I used this last summer to run batch jobs from 30+
computers sharing the same file system.
Want to try it? Try this
> install.packages("R.classes",
contriburl="http://www.maths.lth.se/help/R")
> library(R.batch)
> example(JobBatch)
and a batch of Mandelbrot sets (from Martin Maechler's rhelp example)
will be generated together with images.
Warning: The package works, but the API is not fixed, meaning it may
change in future releases. However, the general idea should remain.
Currently I feel that the names of some methods and directories are a
little bit confusing. Feedback on this is appreciated.
Future: Recently, I have been working on adding dependency control
between jobs so certain jobs are processed before others. This is not
included in the current version. Some kind of mechanism to restarting
interrupted jobs where they where interrupted would also be very nice,
but this is very tricky and will propably require modification of the R
engine, which is beyond my skills.
Cheers
Henrik Bengtsson
Lapointe, Pierre wrote:
Hello,
Let's say I have 50 R scripts to run. What would be the most efficient way
to run them?
I thought I could do multiple Rterms in a DOS batch file:
Ex:
Rterm <1.R> 1.txt
Rterm <2.R> 2.txt
...
Rterm <50.R> 50.txt
However, I'm afraid they will all open at the same time. I know I could
pause the batch file with something like:
PING 1.1.1.1 -n 1 -w 60000 >NUL (to delay 60 seconds)
But that would require that I know how long each of my scripts take.
Is there an easier way? Something like calling R from R and specifying that
the script has to be finished before continuing.
Thanks
Pierre Lapointe
***********************************************************************************
AVIS DE NON-RESPONSABILITE:\ Ce document transmis par courri...{{dropped}}
______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html