For what it's worth, an incremental backup of my laptop (~3TB data) to a USB-C HDD took 2 weeks. The initial backup was MUCH faster.
-derek On Fri, May 30, 2025 1:24 pm, [email protected] wrote: > Hello, > rdiff-backup has many advantages but speed isn't one of those. I'm sure we > can improve the situation but this takes time. > Kr, Eric > > On 30 May 2025 16:20:29 CEST, Duane Abrames <[email protected]> wrote: >>(I sent this yesterday, but have not seen it come back to me, nor in the >>online archives. Not sure if there is some additional approval process, >> or >>if I was just too quick to send this after joining. Apologies if it >> makes >>a duplicate.) >> >>Hello, new to the list and rdiff-backup. >> >>First let me describe my situation. I have just started using >> rdiff-backup >>to back up my home server, which contains various music, videos, ebooks, >>comics, and audio books ( 34.0 TB in 29,244 folders and 460,277 files). >>The source and target servers are both Unraid boxes, so I am running >>rdiff-backup in a Docker container. I had initially set up just one >>container on the target server, and just mounted the source data via NFS. >>My initial backup finished last night, coming in at about 6.5 days. There >>seemed to be a lot of latency involved in copying the files; I was >> watching >>files get created, and I watched a directory with 40 or so files of less >>than 10mb each take 5 minutes or more to copy, and this was the 'initial' >>backup, so there would be no comparison, just copying. >>I suspected that performance would be better over SSH than NFS, so I >> added >>a container to the source server and reconfigured things so that the >> paths >>over ssh would match the former NFS paths. After setting all that up, I >>kicked off an increment. That increment has been running for 10 hours >> now, >>and using rdiffweb I can see that it is going through checking the >> folders, >>because the version times change from 10 hours ago (before checking) to 7 >>days ago (after). It looks like even an increment will take days. I was >>hoping to get at least one a day, but if this is the best I can do, I >> might >>have to fall back to weekly backups, or evaluate another option. I wish >> I >>had started this increment with additional verbosity so I might be able >> to >>tell what it's actually doing. I feel like at this speed, it must be >> doing >>checksums or full comparisons, since I can get the 'metadata' (file time, >>size, owner, etc) for the whole share in about 3 minutes with ls -lR. >> Does >>my experience seem typical, and if not, how should I begin nailing down >> the >>source of the issue? I kind of assumed that since there were probably >> less >>than 200 new/changed files, the increment would be measured in minutes >> (or >>at worst hours) and not days. >> >>Some system specs, just so everyone knows I'm not trying to run this on a >>Dorito chip: >>Source: AMD 5600G / 32gb RAM / 7.2k spinning SATA disk x 9 (with parity) >>on IBM SAS controller >>Target: AMD 5500 / 64gb RAM / 7.2k spinning SATA disk x6 (without >> parity) >>on mobo / add-in SATA. >>Connectivity between the two is gigabit ethernet using a dumb switch. >>Servers are less than 10 feet apart, so cable length might be 20 or 30 >> feet >>since cables run under the house. >> >>I don't have a lot of tools inside the containers to troubleshoot with, >> but >>they are being pulled from my personal github container repo, so I can >>always add things and rebuild as needed. The base OS in the container is >>ubuntu/latest. I went into this project excited (as excited as one can >> be >>about backups anyway,) as having increments to go back to, and still >> having >>the filesystem be usable as-is seemed like eating the cake and keeping it >>too (Is the cake a lie?) I am less enthused now, and wondering if I will >>end up with a plain rsync (which is basically what I had a few months ago >>before some hardware deaths) or using a backup product that packages >>backups in its own native format so that any use or restore from the >> system >>requires the use of the backup software. >> >>Thanks for your time and attention. >>-Duane > -- Derek Atkins 617-623-3745 [email protected] www.ihtfp.com Computer and Internet Security Consultant
