Reed,
I'd like to add to Sebastian's comments - the problem is probably rsync.
I inherited a smaller setup than you when I assumed my current
responsibilities - an XFS file system on a RAID and exported over NFS. The
backup process is based on RSnapshot, which is based on rsync over SSH, but
the
Providing an overdue update to wrap this thread up.
It turns out I wasn't seeing the forest for the trees.
Parallelizing the copy did in fact yield much larger results than the single
threaded copies.
In the end we used a home-brewed python script to parallelize the copy using
cp, rather than r