Suppose i have a program called prog1, another called prog2, then finally a program called prog3. I want a processing chain so that the output of prog1 is input to prog2 whose output goes into prog3. Consider that each of these programs might have commandline options. So i could set up a processing chain on BAsh as follows:
cat joblist | prog1 -i -f file | prog2 -1 -E | prog3 -h -g But, because each of these programs has different commandline options, i could have in another instance, an execution chain that looks as follows cat joblist | prog1 -u -t -f file | prog2 -1 -lhk | prog3 Now, i know the conditions underwhich i want to run each of these two cases. However, once one of these two cases starts running, i want to be able to issue interactively a command to stop feeding certain types of files from the joblist. I also want to be able to prioritise the jobs in joblist so that those are run first. I would also like to be able to insert new jobs into the joblist with a certain priority level, so that if the inserted is a high priority job for example, it is run next as soon as any of the currently running jobs has finished. I would like to be able to say skip a certain job, or repeat a certain job, take a certain job out of joblist, etc. All this i want to be able to do when one of those two cases has already started running. I need this because the stuff in joblist takes a long time to run, so sometimes our needs change while we still waiting for jobs to finish. For example the first three items in joblist could be very important now when we run the command, but after a day or two of running, the bottom four become of higher priority and we want to run those before any of the jobs in-between. If one of those programs in the pipe finishes with non-zero exit code, i would like all processing to pause to allow investigation, or i might want another process to be made available for computing the other files. I might want to pause any items from joblist that take longer that a certain time. I also might want prog1 to first produce a certain amount of output before prog2 may be started, etc. Regards, Prince On Sat, Feb 11, 2017 at 3:20 PM, Hans Schou <[email protected]> wrote: > > > 2017-02-10 6:46 GMT+01:00 Prince Sibanda <[email protected]>: > >> So i have several programs to pipe together, where each program in the >> pipe could optionally run in parallel. >> > > It sounds like you will have that in shell script and then call parallel. > > >> I would like an extension where GNU Parallel is able to read a user >> supplied file in which one can specify a list of programs to run at each >> pipe stage, and qualify the data that moves through the pipe. >> > > You have to describe it more detailed about what input should give what > output. Give some examples. > > I would also like to have GNU Parallel be interactive so that once it has >> started running, i could issue commands to leave out certain files or to >> bypass certain programs in the pipe. This is the most basic version of the >> extension >> > > You can get feedback with --eta and --joblog. That could be used for input > to your joblist. And you are always free to implement it yourself. > > ./hans >
