Thanks for confirming, Craig. There are lots of approaches that I could
use to reduce the duplication, but they would all add complexity
needlessly if BackupPC was already storing just deltas at a high
granularity (and I completely understand the decision not to do so).
I'm currently looking at MySQL Incremental Backup (which in turn uses
AutoMysqlBackup) - I haven't quite got it working yet but it should be
better than rolling my own.
SteveR.
On 20/11/2018 17:39, Craig Barratt via BackupPC-users wrote:
Steve,
You are exactly right - BackupPC's storage granularity is whole
files. So, in the worst case, a single byte change to a file that is
a unique will result in a new file in the pool. Rsync will only
transfer the deltas, but the full file gets rebuilt on the server.
Before I did the 4.x rewrite, I did some benchmarking on block-level
or more granular deltas, but the typical performance improvement was
modest and the effort to implement it was large. However, there are
two cases where block-level or byte-level deltas would be very helpful
- database files (as you mentioned) and VM images.
Perhaps you could use $Conf{DumpPreUserCmd} to run a script that
generates byte-level deltas, and exclude the original database files?
You could have a weekly schedule where you copy the full database file
on, eg, Sunday, and generate deltas every other day of the week. Then
BackupPC will backup the full file once, and also grab each of the
deltas. That way you'll have a complete database file once per week,
and all the daily (cumulative) deltas.
Craig
On Tue, Nov 20, 2018 at 9:28 AM Steve Richards <b...@boxersoft.com
<mailto:b...@boxersoft.com>> wrote:
Thanks. Yes, I had seen that in the docs but I got the impression
that the deltas referred to there were at the granularity of whole
files. For example, let's say backup 1 contains files A, B and C.
If B is then modified then, during the next backup rsync might
only /transfer/ the deltas needed to change B to B1 e.g. "replace
line 5 with [new content]". I got the impression that those deltas
would be used to create B1 though, and that both complete files (B
and B1) would be stored in the pool as whole files. The deltas
referred to in the docs would then be how to get from one /backup/
to another e.g. "Delete file B, insert file B1" (or vice versa,
depending on whether it's BackupPC V3 or V4).
So that's the way I interpreted it, but I'm very new to this so I
may have got the wrong end of the stick completely. If anyone
could confirm or correct my understanding, I'd appreciate it
either way.
Thanks for the comments on mysqldump, I'll take a look at those
options.
SteveR.
On 20/11/2018 14:05, Mike Hughes wrote:
Hi Steve,
It looks like they are stored using reverse deltas. Maybe you’ve
already seen this from the V4.0 documentation:
* Backups are stored as "reverse deltas" - the most recent
backup is always filled and older backups are reconstituted
by merging all the deltas starting with the nearest future
filled backup and working backwards.
This is the opposite of V3 where incrementals are stored as
"forward deltas" to a prior backup (typically the last full
backup or prior lower-level incremental backup, or the last full
in the case of rsync).
* Since the most recent backup is filled, viewing/restoring
that backup (which is the most common backup used) doesn't
require merging any deltas from other backups.
* The concepts of incr/full backups and unfilled/filled storage
are decoupled. The most recent backup is always filled. By
default, for the remaining backups, full backups are filled
and incremental backups are unfilled, but that is configurable.
Additionally these tips might help apply deltas to the files and
reduce transfer bandwidth:
MySQL dump has an option ‘--order-by-primary’ which sorts
before/while dumping the database. Useful if you’re trying to
limit the amount to be rsync’ed. You’ll need to evaluate the
usefulness of this based on db design.
If you’re compressing your database look into the “--rsyncable”
option available in the package pigz.
*From:* Steve Richards <b...@boxersoft.com>
<mailto:b...@boxersoft.com>
*Sent:* Tuesday, November 20, 2018 04:34
*To:* backuppc-users@lists.sourceforge.net
<mailto:backuppc-users@lists.sourceforge.net>
*Subject:* [BackupPC-users] Large files with small changes
I think some backup programs are able to store just the changes
("deltas") in a file when making incrementals. Am I right in
thinking that BackupPC doesn't do this, and would instead store
the whole of each changed file as separate entries in the pool?
Reason for asking is that I want to implement a backup strategy
for databases, which is likely to involve multi-megabyte SQL
files that differ only slightly from day to day. I'm trying to
decide how best to handle them.
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
<mailto:BackupPC-users@lists.sourceforge.net>
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project:http://backuppc.sourceforge.net/
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
<mailto:BackupPC-users@lists.sourceforge.net>
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
