Out of curiosity, do you have a library in mind that would make it easy to setup a bit torrent network and distribute files in an rsync (i.e., apply a diff to a tree, ideally) fashion? I'm not familiar with this space, but we do want to minimize the complexity of our standard ec2 launch scripts to reduce the chance of something breaking.
On Sun, May 18, 2014 at 9:22 PM, Daniel Mahler <dmah...@gmail.com> wrote: > I am launching a rather large cluster on ec2. > It seems like the launch is taking forever on > .... > Setting up spark > RSYNC'ing /root/spark to slaves... > ... > > It seems that bittorrent might be a faster way to replicate > the sizeable spark directory to the slaves > particularly if there is a lot of not very powerful slaves. > > Just a thought ... > > cheers > Daniel > >
