On 20/08/2013 08:32, krad wrote:
When i migrated a large mailspool in maildir format from the old nfs server
to the new one in a previous job, I 1st generated a list of the top level
maildirs. I then generated the rsync commands + plus a few other bits and
pieces for each maildir to make a single transaction like function. I then
pumped all this auto generated scripts into xjobs and ran them in parallel.
This vastly speeded up the process as sequentially running the tree was far
to slow. THis was for about 15 million maildirs in a hashed structure btw
so a fair amount of files.
eg
find /maildir -type d -maxdepth 4 | while read d
do
r=$(($RANDOM*$RANDOM))
echo rsync -a $d/ /newpath/$d/ > /tmp/scripts/$r
echo some other stuff >> /tmp/scripts/$r
done
ls /tmp/scripts/| while read f
echo /tmp/scripts/$f
done | xjobs -j 20
This isn't what I'd have expected, as running operations in parallel on
mechanical drives would normally result in superfluous head movements
and thus exacerbate the I/O bottleneck. The system must be optimising
the requests from 20 parallel jobs better than I thought it would to
climb out from that hole far enough to get a net benefit. Did you
remember how any other approaches performed?
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
