On 3/8/21 6:03 am, Frank Steinmetzger wrote: > Am Mon, Aug 02, 2021 at 02:12:24PM +0800 schrieb William Kenworthy: >>>>> And you are storing several machines into a single repo? The docs say this >>>>> is not supported officially. But I have one repo each for /, /home and >>>>> data >>>>> for both my PC and laptop. Using a wrapper script, I create snapshots that >>>>> are named $HOSTNAME_$DATE in each repo. >>>> Basicly yes: I use a once per hour snapshot of approximately 500Gib of >>>> data on moosefs, plus borgbackups 3 times a day to individual repos on >>>> moosefs for each host. >>> So you have: >>> Host A ──[hourly]──> Online-Borg A ─┐ >>> ├──[3/day]──> Offline-Borg >>> Host B ──[hourly]──> Online-Borg B ─┘ >>> ? >>> […] >> Hi Frank, >> >> Not quite - I see I could have been clearer. I "experiment" a lot - >> which means things break so I need to get back running quickly. So the >> purpose of the online repos and snapshots is just for that - quick >> recovery. > Whenever you say snapshot, you meen moosefs snapshots, right? Up until this > thread I’ve never heard of that FS. > > I would love to play more with storage systems, moving stuff around, backing > it up, assigning space and so on (basically play admin for a few people), > but apart from my ZFS-based NAS, I have nothing that would need this. I run > a nextcloud instance on my shared internet host and one on my raspi. That’s > as far as it gets. :D > >> stage 1: online, immediately available >> >> Hosts (those with actual attached storage - a mixture of intel, arm32 >> and arm64 devices are backed up to their own borg repo 3 times a day via >> push. One repo per machine on moosefs. >> >> A separate script does an hourly backup of VM, LXC images, and various >> data stores via a moosefs snapshot. >> >> stage 2: resource management for the snapshots >> >> 3 times a day, a script does a borg create on the latest snapshop at the >> time > So you mount the latest snapshot or access it in some other way and borg > *its* content, not the live data, right? Yes, though a moosefs snapshot is really a lazy copy of the data to a new location - issue the mfsmakesnapshot command and a few seconds later you have an identical copy of possibly terabytes of data in a new location with almost no extra disk space needed - though moosefs needs memory allocated to track the contents. i.e. think of it like symlink/hardlink to the original data until a file is changed whereupon its links are broken and its new data - its a little more complex than that but that's the gist of it. If you need data from a snapshot - you just copy it out or use it in place which breaks the link if written to. > >> and when complete deletes all previous snapshots (-1) so at that >> point I have two older snapshots available + a couple created during the >> borg run - note that large multi GiB snapshots can quickly use up all >> memory (32GiB) on the moosefs master unless culled regularly. > Sounds a bit delicate to me. If one link fails for some reason undetectedly, > you risk clog-up. Problems so far relate to borg failing to run for some reason - notice it and fix it, no problems overall as the rest keeps working > >> stage 3: offline because disasters happen :) >> >> All borg repos are on moosefs with a single root directory >> (/mnt/mfs/backups) so once every day or so I manually mount the offline >> disk and do a borg create on the backup directory. > What happens if that daily borg runs while the repos are being written to?
To avoid this I mostly use fcron which has serialisation features so as long as I coordinate start and take into account run times its good. The manual copy is of course a bit tricky but again, its a timing issue. If I make a mistake, I would expect to include a repo that might need a check/repair before use. To borg, its just backing up files - it doesn't care that its another borg repo, in use or not. It still treats open files the same way as any other files - try and read them, but skip if unable to. Two borg instance cant work in the same repo, but one can and one can back that repo up at the same time because it just sees it as files if that makes sense. In reality, I have not needed to deal with it yet. The great thing about borg and the way this rota is structured is that I have a history I can go back to if necessary. In my experimenting in getting this right, I pay attention to the warnings borg issues and often delete the cache and security directories to make sure everything stays sane. Getting way off-topic here so we can take this off list if you are interested, or maybe others are interested here? BillK BillK
