Re: [BackupPC-users] RsyncIncrArgsExtra
Hi Alex, Ok, thanks for that suggestion, I’d thought of it, but wasn’t sure if rsync would complain if the arg appeared twice, but apparently it doesn’t. I am NOT sure whether bandwidth limitation is what I want. I am actually trying to throttle down not only the network bandwidth used but also the I/O load. This is a shared file system with hundreds of users accessing it. I’m only backing up our lab’s small portion of the data, and I’m only backing up files less than 1 MB in size. The full backups are done separately by someone else in a different manner. For my <1 MB files, I am doing a full backup once a year and an incremental backup once an hour. I want to have essentially 0 impact on the network bandwidth and on the I/O load between the server that talks to BackupPC and the network storage device. Since I’m just starting, I’m doing the first full backups, and they are taking forever. I have a bandwidth limit of 1 MB/s, very low. I need to explore how high I can go without impacting other’s access, and how high I need to go to finish the full backups and incremental backups in a timely fashion. I’m thinking a higher bandwidth limit for the full backups would get them done quicker with still little impact. For the incrementals, I haven’t done one yet so I don’t know how long it will take, but I may discover I have to increase that bandwidth also, and/or decrease the frequency of the incrementals. Based on that, do you think I should be using ionice too? And by the way, I do not have root access to the server. Ted ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Identical files
>> Would replacing "same permissions, ownership or modification time" by >> "same filename, path, permissions, ownership, modification time, or >> other metadata" make the crowd more happy? Yes, that is all that is needed, just adding “filename, path” to that list. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] RsyncIncrArgsExtra
Hi again, On 02.08.19 11:50, Alexander Kobel wrote: > Hi Ted, > > On 01.08.19 18:31, Ted Toal wrote: >> There is a BackupPC config parameter named RsyncFullArgsExtra, but none >> named RsyncIncrArgsExtra (to provide extra rsync args for an incremental >> backup). I’d like to see such a parameter. My immediate use is that I’d >> like to restrict rsync bandwidth to different amounts depending on whether >> it is a full or incremental backup. > > [...] > > Apart from that, are you sure that a bandwidth limit actually is what > you're after? The (network) *bandwidth* used for incrementals and fulls > does not differ a lot; it's the *I/O load* on the client that makes the > real difference: that being said: if you want to use ionice or similar tools to adjust the I/O load on the client, I suggest that you set RsyncClientPath to a simple wrapper script that calls rsync via ionice. In this script, just check whether --checksum is in the argument list; if it is, you're running a full backup, otherwise an incremental. Again: before solving the problem, make sure that it actually exists. ;-) I wouldn't be surprised if you end up at the *same* ionice arguments for fulls and incrementals... Cheers, Alex smime.p7s Description: S/MIME Cryptographic Signature ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] RsyncIncrArgsExtra
Hi Ted, On 01.08.19 18:31, Ted Toal wrote: > There is a BackupPC config parameter named RsyncFullArgsExtra, but none named > RsyncIncrArgsExtra (to provide extra rsync args for an incremental backup). > I’d like to see such a parameter. My immediate use is that I’d like to > restrict rsync bandwidth to different amounts depending on whether it is a > full or incremental backup. assuming that you want to use --bwlimit, can't you just add --bwlimit= in RsyncArgs, and an additional --bwlimit= in RsyncFullArgsExtra? According to my tests, the second will override the first for the full limits. Slightly unelegant workaround, but effective. Note that the arguments are appended in the order RsyncArgs RsyncFullArgsExtra RsyncArgsExtra (see lib/BackupPC/Xfer/Rsync.pm, lines 307-334 or so), so you have to add the "default" (incremental) limit to RsyncArgs, not RsyncArgsExtra. Apart from that, are you sure that a bandwidth limit actually is what you're after? The (network) *bandwidth* used for incrementals and fulls does not differ a lot; it's the *I/O load* on the client that makes the real difference: IIUC, no matter what backup type, rsync needs to compare all file paths and some metadata. For incrementals, by default it compares path size modification-time; for fulls (with --checksum), it skips the latter two and compares path checksum instead. For the computation of the checksums, the client will read each file in its entirety. That makes for a lot of *I/O bandwidth* on the client. But regarding the *network bandwidth*: The checksum is an MD5 hash, i.e., 128 bit = 16 bytes long. Without digging in the source, size and modtime are probably integers of 4 or 8 bytes each. So my guess is that the bandwidth difference is *at most* 8 bytes per file, but more likely 0... HTH, Alex smime.p7s Description: S/MIME Cryptographic Signature ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Identical files
Hi, On 01.08.19 18:03, G.W. Haywood via BackupPC-users wrote: > Hi there, > > On Fri, 26 Jul 2019, Ted Toal wrote: > >> Is it easy to make updates to the documentation? > > I haven't seen a reply to this so I'll take a stab at it, although I > don't really know the proper procedure. Mr. Barratt will know. no offense meant to Ted, but I dare to guess a reason for the silence: folks simply think that the explanation in the documentation is sufficiently precise. And, to be honest, I tend to agree (with a very minor exception, see the last paragraph of this mail). In particular since I cannot remember that particular question popping up in the last couple of years I follow this list. Anyway, not trying to keep you from proposing something to improve. > It's not clear to me where you saw the "BackupPC documentation" which > you mentioned in your OP. I'd think it's the most official source I can imagine: the 4.3.1 docs, directly linked from the home of http://backuppc.sourceforge.net/, at https://backuppc.github.io/backuppc/BackupPC.html#Backup-basics The questionable passage is: BackupPC pools identical files. By "identical files" we mean files with identical contents, not necessary the same permissions, ownership or modification time. Two files might have different permissions, ownership, or modification time but will still be pooled whenever the contents are identical. This is possible since BackupPC stores the file metadata (permissions, ownership, and modification time) separately from the file contents. I have no brilliant idea how one could spell out "identical contents" more precise. But as a suggestion: Would replacing "same permissions, ownership or modification time" by "same filename, path, permissions, ownership, modification time, or other metadata" make the crowd more happy? IMHO, the only *serious* non-trivial case regarding the "sameness" of files are alternate data streams on NTFS. A caveat regarding those might be in order, if anyone already wants to touch the docs. (AFAIK, ADSs are not part of the backup.) Cheers, Alex smime.p7s Description: S/MIME Cryptographic Signature ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
