On 2/8/21 5:55 am, Frank Steinmetzger wrote: > Am Sun, Aug 01, 2021 at 11:36:36AM +0800 schrieb William Kenworthy: > >>>> Its not raid, just a btrfs single on disk (no partition). Contains a >>>> single borgbackup repo for an offline backup of all the online >>>> borgbackup repo's I have for a 3 times a day backup rota of individual >>>> machines/data stores >>> So you are borg’ing a repo into a repo? I am planning on simply rsync’ing >>> the borg directory from one external HDD to another. Hopefully SMR can cope >>> with this adequatly. >>> >>> And you are storing several machines into a single repo? The docs say this >>> is not supported officially. But I have one repo each for /, /home and data >>> for both my PC and laptop. Using a wrapper script, I create snapshots that >>> are named $HOSTNAME_$DATE in each repo. >> Basicly yes: I use a once per hour snapshot of approximately 500Gib of >> data on moosefs, plus borgbackups 3 times a day to individual repos on >> moosefs for each host. > So you have: > Host A ──[hourly]──> Online-Borg A ─┐ > ├──[3/day]──> Offline-Borg > Host B ──[hourly]──> Online-Borg B ─┘ > ? > >> 3 times a day, the latest snapshot is stuffed into a borg repo on moosefs >> and the old snapshots are deleted. > How do you stuff just the latest snapshot of a repo into another repo? > >> 3. borgbackup often to keep changes between updates small - time to >> backup will stay short. > Apparently you are dealing with some kind of productive, high-availability > or high-throughput system. I only have my personal stuff, so I don’t mind it > taking a few minutes. That allows me to watch progress bars and numbers > going up. :) > >> 4. borg'ing a repo into a repo works extreemly well - however there are >> catches based around backup set names and the file change tests used. >> (ping me if you want the details) > But what is the point of that? Why not simply keep the last x hourly/ > daily/weekly snapshots? The only thing I can think of is to have a > small(ish) short-term repo and keep the larger archival repo separate. > Hi Frank,
Not quite - I see I could have been clearer. I "experiment" a lot - which means things break so I need to get back running quickly. So the purpose of the online repos and snapshots is just for that - quick recovery. Longer term I want to have an offline copy for a real disaster (yes, I have needed to restore almost 20TiB including the online backups on the moosefs system three months ago - self caused :) as well as often needing to step back a few days/weeks for a historical copy of a file, or less often a system restore on a host. By the way, this discussion has been useful as I found some failed backups due to a missing dependency in the borgbackup ebuild when looking closer at it - has a bug. Need to do some watcher scripts to detect that. stage 1: online, immediately available Hosts (those with actual attached storage - a mixture of intel, arm32 and arm64 devices are backed up to their own borg repo 3 times a day via push. One repo per machine on moosefs. A separate script does an hourly backup of VM, LXC images, and various data stores via a moosefs snapshot. stage 2: resource management for the snapshots 3 times a day, a script does a borg create on the latest snapshop at the time, and when complete deletes all previous snapshots (-1) so at that point I have two older snapshots available + a couple created during the borg run - note that large multi GiB snapshots can quickly use up all memory (32GiB) on the moosefs master unless culled regularly. stage 3: offline because disasters happen :) All borg repos are on moosefs with a single root directory (/mnt/mfs/backups) so once every day or so I manually mount the offline disk and do a borg create on the backup directory. I was doing this once a week, but operationally its easier to do 5-10 minutes every day than an hour once a week due to the scale of changes over the longer time period. So, its looks like: Host A ──[3/day]──> Online-Borg A ─┐ ├─────────────────────────[common directory]---[manual, 1/day]──> Offline-Borg Host ... ──[3/day]──> Online-Borg ... ─┘ | ? | Snapshots ──[hourly]──> Online-Borg "snaps" ─[3/day] ─────────────────┘ BillK