Hi GNU Parallel mailing list.
I'm looking for a way to run a large number of jobs (in parallel) on
several computers. The job consists of running an analysis on several
thousand input files. They way I have this set up right now is to split
the number of input files into chunks and move them to the various
computers, and then run the analysis using GNU parallel on each machine.
This has the down side that I have to keep track which computers are
doing what.
I could perhaps set this up using the ssh functionality of parallel, but
I would need to be able to on the fly stop some machines from running
jobs, since the computers belong to co-workers who sometimes need their
computers for their own work.
My idea was that perhaps I can have a file containing the paths to all
the files I want to analyze, which is accesible to all computers on the
network. I would then like to set up separate parallel jobs on the
various computers that would continuously pull paths from this input
file, and then run analysis on it.
Of course the issue here is that I need to be able to keep track of
which paths have been used. Having several computers access it at the
same time seem like it would lead to I/O issues.
Am I missing some more obvious solution?
Any help very much appreciated. And I hope I am not abusing the purpose
of this mailing list.
