Re: Parallelizing rsync through multiple ssh connections

2021-12-16 Thread Robin H. Johnson via rsync
On Thu, Nov 04, 2021 at 04:58:03PM +0100, SERVANT Cyril via rsync wrote: > Hi, I want to increase the speed of rsync transfers over ssh. Thanks for your great email here. Having had similar issues in the past in trying to rsync single large files, I wanted to share some of the ideas I'd found to

Parallelizing rsync through multiple ssh connections

2021-11-04 Thread SERVANT Cyril via rsync
Hi, I want to increase the speed of rsync transfers over ssh. 1. The need TL;DR: we need to transfer one huge file quickly (100Gb/s) through ssh. I'm working at CEA (Alternative Energies and Atomic Energy Commission) in France. We have a compute cluster complex, and our customers regularly

parallelizing rsync

2021-10-27 Thread Steve French via rsync
I noticed a few external threads on problems with rsync performance due to lack of parallelization. For example modern Linux tools are supposed to use "io_uring" for parallel i/o and some apparently have done some experiments with this, and this is something we were looking at as well. For