I have implemented an instant --load calculator. With this it should be possible to tell GNU Parallel to spawn a process for each CPU that is currently sitting idle.
On a 64 core machine this: seq 10000 | parallel --load 20% burnP6 & sleep 2 seq 10000 | parallel --load 100% burnMMX & sleep 4 seq 10000 | parallel --load 100% burnBX & will start 14 burnP6, 52 burnMMX, and 0 burnBX. If the first parallel is killed, the last remaining parallels will spawn around 14 processes it total. It should also count processes that are waiting for local disks as non-idle. So this: ls *iso | parallel --load 1 -j0 -v cat will only spawn 3 cat and not one for every iso file. The biggest problem I have seen so far is that it is not very accurate. So it may shoot a little over or under the target (2-4). But it seems way better than the previous --load. http://alpha.gnu.org/gnu/parallel/parallel-20130307.tar.bz2 Please give it a spin and provide feedback. /Ole
