Hello, Is it possible to use GNU Parallel to implement a persistent queue system with remote execution?
My setup is: two compute servers, identical configuration, no shared network drive. The job in question downloads its input data over FTP, and --return is used for the processed data. Using 'parallel -S 4/server1 -S 4/server2 command' works fine as long as I only have one set of tasks at a time. However I'd like to run both hourly jobs and daily jobs. If I just run them as two separate simultaneous Parallel jobs the compute servers will run out of memory and swap themselves to a snail's pace. So I'd like to run both sets of jobs in a shared queue. Will a named semaphore do what I want, if I use the same semaphore name for both ses of jobs? Will it use the correct number of CPUs (4) on each server if I specify '-S 4/server1 -S 4/server2', or do I also need to specify --jobs? The hourly job takes around 15-20 minutes to complete. Ideally I'd want the hourly job to have higher priority so that the daily jobs are only running when there are no hourly jobs queued, but it seems this is not possible with Parallel. Åsmund Steen Skjæveland, PhD Senior Engineer Norwegian Mapping Authority, Geodetic Insititute
