Hi, You are right that it doesn't seem like rdiff-backup alone is the issue; but to me it sounds like part of the issue is librsync.
>From my original email, I'm getting a good 5 MB/s using raw scp. But I'm only seeing 1-2MB/s using rsync and/or rdiff-backup. So there is clearly overhead with these two protocols over raw ssh/scp. Of course scp is still being a bottleneck (compared to 20MB/s capability on the back-end). But even if I could get rsync/rdiff-backup up to the scp-capability of 5MB/s that would help. I'll take a look at hpn-ssh, but building that for all targets might be a major PITA. I might also look at alternate ciphers. Thanks, -derek On Wed, March 30, 2016 3:29 am, Dominic Raferd wrote: > It doesn't sound like rdiff-backup is the culprit here. You could try > hpn-ssh https://sourceforge.net/projects/hpnssh/ ? > > On 29 March 2016 at 21:03, Derek Atkins <de...@ihtfp.com> wrote: > >> Hi, >> >> On Mon, March 28, 2016 10:56 am, Derek Atkins wrote: >> > Hi, >> > >> > Thank you for taking the time to look at this.. >> > >> > On Mon, March 28, 2016 10:41 am, Dominic Raferd wrote: >> >> Is this really your first rdiff-backup to this location? If you have >> any >> >> previous rdiff-backup runs to this repository then the situation is >> >> complicated by rdiff-backup's need to create a new set of reverse >> diff >> >> files to be able to regress to previous file contents. >> > >> > Yes, this s really the first rdiff-backup to this location. >> > >> > A second backup run shortly after the first one completed finished in >> 55 >> > minutes. >> > >> >> What is your /tmp location? rdiff-backup uses this location for some >> >> operations though not AFAIK for standard backup runs. Still, if /tmp >> is >> >> on >> >> encfs maybe it could be a culprit; you can override rdiff-backup's >> >> temporary file location with --tempdir and --remote-tempdir. >> > >> > If it is truly /tmp then no; /tmp is a ramdisk on the backup server >> and >> is >> > on the root disk on the target server. Neither are being run through >> > encfs. >> > >> > If, however, you mean the rdiff-backup-data.tmp files, those ARE being >> run >> > through encfs. >> > >> >> Might also be worth trying --ssh-no-compression. >> > >> > I already have "Compression no" set in ~/.ssh/config so I'm not sure >> what >> > this would add? >> > >> >> Dominic >> >> http://www.timedicer.co.uk >> > >> > -derek >> > >> > PS: I'm running a raw rsync command now just to see how it behaves -- >> so >> > far I'm only seeing about 2MB/s, but it's only been running for 10 >> minutes >> > or so. >> >> My rsync backup just finished. It copied 202688912 KB in 1599m55.255s >> so >> about 2.1MB/s. Still significantly slower than SCP, but faster than >> rdiff-backup. >> >> The command I ran was: >> >> rsync -art -e "ssh -l root -i /dev/null -o Compression=no" >> root@server:/var/www/ /backups/server >> >> :-( >> >> -derek >> >> > >> >> >> >> On 28 March 2016 at 14:37, Derek Atkins <de...@ihtfp.com> wrote: >> >> >> >>> Just a quick update: I tried making these changes on both sides and >> it >> >>> really didn't make a difference. Full backup of 202852072 Kbytes >> >>> required 2267m25.913s (previously it took 2346m57.800s, so it only >> sped >> >>> up a factor of 3%. >> >>> >> >>> Only thing I have not yet tried is running a raw rsync to see how >> fast >> >>> that runs. I'll do that next. >> >>> >> >>> So, back to my orignal question: anyone have any idea how to get >> >>> initial >> >>> transfers to run faster (or indeed any significant data transfers)? >> >>> >> >>> Thanks, >> >>> >> >>> -derek >> >>> >> >>> "Derek Atkins" <de...@ihtfp.com> writes: >> >>> >> >>> > Hi, >> >>> > >> >>> > I'm trying to use rdiff-backup to backup a bunch of servers. One >> >>> > particular server contains about 160GB of data, but when I try to >> >>> perform >> >>> > the rdiff-backup it's saving the data at a measly 1MB/s. >> >>> > >> >>> > Here's my configuration: >> >>> > >> >>> > [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas] >> >>> > >> >>> > I ran a bunch of tests to try to figure out my bottlenecks. >> >>> > >> >>> > I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) >> on >> >>> the >> >>> > backup server. Going directly to FreeNAS via NFS (bybassing >> encfs) I >> >>> get >> >>> > 50.2MB/s. If I run dd directly on the backup server (through >> encfs) >> >>> I >> >>> get >> >>> > 20.1MB/s. If I go over SSH from the backup server to the target >> >>> server >> >>> > and run the dd on the target server, then write to FreeNas through >> >>> encfs >> >>> > declines to 7.6MB/s. >> >>> > >> >>> > Note that in those SSH tests, however, I forgot to turn off >> >>> compression. >> >>> > When I do that, the throughput for the dd test reduced to 6.6BM/s. >> >>> (Note >> >>> > that this is running simultaneously with a running rdiff-backup, >> so >> >>> it's >> >>> > possible that they are reducing performance). >> >>> > >> >>> > Then I ran an scp test to the same target server; copying about >> 1.4GB >> >>> of >> >>> > photos. Files ranged in size from 10KB to 5MB. When run in >> standard >> >>> mode >> >>> > (displaying each file status) I got 4.4MB/s. Running in quiet >> mode I >> >>> get >> >>> > 5.1MB/s. >> >>> > >> >>> > So clearly the bottleneck is in rdiff-backup -- performance (IMHO) >> >>> should >> >>> > not be significantly slower than the last dd-over-ssh test. It >> >>> appears >> >>> > rdiff-backup is slowing me down by a factor of 5x throughput >> versus >> >>> scp. >> >>> > >> >>> > I found a message from Ben from 2005 where he suggests increasing >> the >> >>> > blocksize and conn_bufsiz settings in Globals.py: >> >>> > >> >>> >> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html >> >>> > >> >>> > What he didn't say was whether this needed to be changed on the >> >>> target >> >>> > server, the backup server, or both. Nor do I know if that would >> >>> actually >> >>> > help this situation. >> >>> > >> >>> > Do you have any ideas? >> >>> > >> >>> > Thanks, >> >>> > >> >>> > -derek >> >>> > >> >>> > PS: According to rpm, both systems are running version 1.2.8. >> >>> >> >>> -- >> >>> Derek Atkins 617-623-3745 >> >>> de...@ihtfp.com www.ihtfp.com >> >>> Computer and Internet Security Consultant >> >>> >> >>> _______________________________________________ >> >>> rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org >> >>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users >> >>> Wiki URL: >> >>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki >> >>> >> >> >> > >> > >> > -- >> > Derek Atkins 617-623-3745 >> > de...@ihtfp.com www.ihtfp.com >> > Computer and Internet Security Consultant >> > >> > >> > _______________________________________________ >> > rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org >> > https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users >> > Wiki URL: >> > http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki >> > >> >> >> -- >> Derek Atkins 617-623-3745 >> de...@ihtfp.com www.ihtfp.com >> Computer and Internet Security Consultant >> >> > -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant _______________________________________________ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki