Andrew McGill <[EMAIL PROTECTED]> wrote: > Greetings coreutils folks, > > There are a number of interesting filesystems (glusterfs, lustre? ... NFS) > which could benefit from userspace utilities doing certain operatings in > parallel. (I have a very slow glusterfs installation that makes me think > that some things can be done better.) > > For example, copying a number of files is currently done in series ... > cp a b c d e f g h dest/ > but, on certain filesystems, it would be roughly twice as efficient if > implemented in two parallel threads, something like: > cp a c e g dest/ & > cp b d f h dest/ > since the source and destination files can be stored on multiple physical > volumes.
How about parallelizing it via xargs, e.g., $ echo a b c d e f g h | xargs -t -n4 --no-run-if-empty \ --max-procs=2 -- cp --target-directory=dest cp --target-directory=dest a b c d cp --target-directory=dest e f g h Obviously the above is tailored (-L4) to your 8-input example. In practice, you'd use a larger number, unless latency is so high as to dwarf the cost of extra "fork/exec" syscalls, in which case even -L1 might make sense. mv and ln also accept the --target-directory=dest option. > Simlarly, ls -l . will readdir(), and then stat() each file in the directory. > On a filesystem with high latency, it would be faster to issue the stat() > calls asynchronously, and in parallel, and then collect the results for If you can demonstrate a large performance gain on systems that many people use, then maybe... There is more than a little value in keeping programs like those in the coreutils package relatively simple, but if the cost(maintenance+portability burden)/benefit ratio is low enough, then anything is possible. For example, a well-encapsulated, optionally-threaded "stat_all_dir_entries" API might be useful in some situations. If getting any eventual patch into upstream coreutils is important to you, be sure there is some consensus on this list before doing a lot of work on it. > display. (This could improve performance for NFS, in proportion to the > latency and the number of threads.) > > > Question: Is there already a set of "improved" utilities that implement this > kind of technique? Not that I know of. > If not, would this kind of performance enhancements be > considered useful? It's impossible to say without knowing more. _______________________________________________ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils