> Caching is definitely worth doing but you don't always have > the opportunity to do it. If you are copying a lot of files > across, it would help quite a bit if you can just pipeline > requests (or send fewer bundled requests). If you are copying > very large files, streaming would help. When copying large > amounts of data from various sources to a local file server, > caching is not very relevant.
i'm not sure why the focus on copying a bunch of files. i wouldn't go to the effort to better stand-alone ftp/scp/http. the application is a file server and you choose to operate synchronously, between amdahl's law and the speed of light, you're going to be in quite a box. as a contrived example, imagine two nodes of a distributed fs 80ms apart. now imagine both nodes writing to /sys/log/timesync. naively copying and locking is going to be so slow that one imagines that timesync will drift off course. :-) - erik
