This is a bit off topic for the slurm-dev group, but using HPN-SSH will likely speed up your data movement: http://www.psc.edu/index.php/hpn-ssh
Turn off encryption, use less CPU, go faster :) On Tue, Jun 2, 2015 at 12:07 PM, Jared David Baker <[email protected]> wrote: > Bipin, > > > > Perhaps I’m missing something, but I don’t necessarily see why you’d > really need Slurm for this task. Seems like a fairly straight forward > backup task. Are you trying to limit the amount of networking being used or > something similar? > > > > -Jared > > > > *From:* Bipin Shetty [mailto:[email protected]] > *Sent:* Tuesday, June 02, 2015 10:15 AM > *To:* slurm-dev > *Subject:* [slurm-dev] Request information for our current tasks - Need > inputs - IMPORTANT > *Importance:* High > > > > Hi Slurm Team, > > > > As part of our research for workload management we found SLURM to be a > very effective solution. I would like to understand more about this > solution if it really fits into our scheme of things and would like to take > it forward with your valuable suggestions. > > > > Following is our scenario we have today: > > > > 1. We pull approx 3 TB of data every day from our production linux > servers to a backup linux machine which takes almost 2-3 hours everyday. > > 2. Currently we are using simple Unix scripts to pull the data from > the production servers to the backup linux machine. We have the script > pulling data from all servers using SCP protocol. > > 3. Post that we perform operations on the pulled data and then > spilt that into multiple chunks and delete the original 3 TB data. > > 4. This whole process eats up our network big time due to the heavy > workload involved. > > > > Need your inputs or help to let me know if Slurm would be the right > solution for us to take this forward. Also, do let me know if you need any > additional inputs from me. > > > > We are running against time to get a solution implemented in production. > Would be great if you can respond at the earliest with your suggestions. > > > > Appreciate your time. Have a wonderful day ahead!!. > > > > Warm Regards > > *BIPIN* > > — > > *Sapient**Nitro* > > *mobile *5712512175 > > > > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of, or > taking of any action in reliance upon, this information by persons or > entities other than the intended recipient is prohibited. If you received > this in error, please contact the sender and delete the material from any > computer. > > > -- John Lockman III CTO / HPC Software Specialist [email protected] - 512.903.0951
