Hello, new to the list and rdiff-backup. First let me describe my situation. I have just started using rdiff-backup to back up my home server, which contains various music, videos, ebooks, comics, and audio books ( 34.0 TB in 29,244 folders and 460,277 files). The source and target servers are both Unraid boxes, so I am running rdiff-backup in a Docker container. I had initially set up just one container on the target server, and just mounted the source data via NFS. My initial backup finished last night, coming in at about 6.5 days. There seemed to be a lot of latency involved in copying the files; I was watching files get created, and I watched a directory with 40 or so files of less than 10mb each take 5 minutes or more to copy, and this was the 'initial' backup, so there would be no comparison, just copying. I suspected that performance would be better over SSH than NFS, so I added a container to the source server and reconfigured things so that the paths over ssh would match the former NFS paths. After setting all that up, I kicked off an increment. That increment has been running for 10 hours now, and using rdiffweb I can see that it is going through checking the folders, because the version times change from 10 hours ago (before checking) to 7 days ago (after). It looks like even an increment will take days. I was hoping to get at least one a day, but if this is the best I can do, I might have to fall back to weekly backups, or evaluate another option. I wish I had started this increment with additional verbosity so I might be able to tell what it's actually doing. I feel like at this speed, it must be doing checksums or full comparisons, since I can get the 'metadata' (file time, size, owner, etc) for the whole share in about 3 minutes with ls -lR. Does my experience seem typical, and if not, how should I begin nailing down the source of the issue? I kind of assumed that since there were probably less than 200 new/changed files, the increment would be measured in minutes (or at worst hours) and not days.
Some system specs, just so everyone knows I'm not trying to run this on a Dorito chip: Source: AMD 5600G / 32gb RAM / 7.2k spinning SATA disk x 9 (with parity) on IBM SAS controller Target: AMD 5500 / 64gb RAM / 7.2k spinning SATA disk x6 (without parity) on mobo / add-in SATA. Connectivity between the two is gigabit ethernet using a dumb switch. Servers are less than 10 feet apart, so cable length might be 20 or 30 feet since cables run under the house. I don't have a lot of tools inside the containers to troubleshoot with, but they are being pulled from my personal github container repo, so I can always add things and rebuild as needed. The base OS in the container is ubuntu/latest. I went into this project excited (as excited as one can be about backups anyway,) as having increments to go back to, and still having the filesystem be usable as-is seemed like eating the cake and keeping it too (Is the cake a lie?) I am less enthused now, and wondering if I will end up with a plain rsync (which is basically what I had a few months ago before some hardware deaths) or using a backup product that packages backups in its own native format so that any use or restore from the system requires the use of the backup software. Thanks for your time and attention. -Duane
