On 2023-06-05 6:45 PM, Alvin Starr via talk wrote:
I can snapshot the volume and then backup the snapshot but that is a
40TB image.
Veeam tries to take a look at the file systems and zero unused space,
like Borg appears to do, but that feature had to be disabled because it
was causing random system crashes.
I will take a closer look at Borg.
At Canonical we used an in-house tool called Turku to handle sharded
backups for many thousands of systems:
https://canonical.com/blog/introducing-turku-cloud-friendly-backups-for-your-infrastructure
I think in 2018 when I was there, we had 4-5 storage nodes with 12-16TB
backup storage in each. Any VM that wanted a backup just had to run the
agent (python app to invoke rsync), and have a copy of the storage
system's public key.
The original lives here:
https://bazaar.launchpad.net/~turku/turku/turku-storage/files and my
past colleague who wrote it has a fork of all three components:
https://github.com/rfinnie/turku-storage and so on.
For object storage with deduplication and B2 (backblaze) support, I use
restic. I've got ~500k files in about 100GB of deduped space stored for
less than $1USD/month. Restic is fast (standalone Go binary), encrypted
in transit and at rest, supports compression and deduplication, and also
handles many different storage backends.
I can't vouch for it scaling beyond 1-10TB though, but I would be
looking at some kind of incremental+sharding solution for anything
larger than that anyways.
Cheers, Jamon
---
Post to this mailing list talk@gtalug.org
Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
