Hi Arjen, Andreas and specially Fabian for your elaborate reply,
Thanks! It all makes much more sense now.
As said before, PBS is a great addition to the proxmox line of products.
Thanks!
MJ
On 3/2/21 11:48 AM, Fabian Grünbichler wrote:
On March 2, 2021 11:22 am, mj wrote:
Hi,
Testing PBS backups taken from PVE VMs on ceph rbd now. Very nice, very
quick, very cool. :-)
We have a question. Something we wonder about.
In our current backup software, we make weekly full_system backups, and
daily incremental_system backups, each incremental based on the same
full_system backup. So: each daily incremental backup becomes bigger,
until the weekend. Then we make a new full_system backup to base the
next set of incrementals on.
In PBS I cannot specify if a backup is full or incremental, we assume
this means that automatically the first backup is a full_system backup,
and subsequent backups are incremental. The PBS backup logs confirm this
assumption, saying: "scsi0: dirty-bitmap status: created new" vs "scsi0:
dirty-bitmap status: OK (7.3 GiB of 501.0 GiB dirty)"
And now the question: At what point in time is a new full_system backup
created, to rebase incremental backups on?
Or is each incremental backup based on the previous incremental? And if
that is the case, how will we ever be able to delete one of the
in-between incrementals, because that would then break to whole chain of
incremental_backup-based-on-incremental_backup...?
We have read the page
https://qemu.readthedocs.io/en/latest/interop/bitmaps.html but it does
not seem to answer this.
Anyone care to share some insight on this logic and how PBS works?
you might want to take a look at
https://pbs.proxmox.com/docs/technical-overview.html
but, the short summary:
PBS does not do full or incremental backups in the classical sense, it
uses a chunk-based deduplicated approach. the backup content is split
into chunks, those chunks are then hashed to get a chunk ID. the
incremental part happens on different levels:
- if the backup is of a VM that has been backed up before, and that
previous backup still exists on the server, and the VM has not been
stopped in the meantime, only chunks which contain changed blocks
(tracked by Qemu with a dirty bitmap) are read, hashed and uploaded,
the rest is re-used. (FAST incremental)
- for all backups, if a previous backup exists, it's index is
downloaded, all local data is read and hashed, but only chunks which
are missing on the server are actually uploaded (incremental)
- if no previous backups exists, all local data is read and hashed and
uploaded ("full" backup)
additionally, an uploaded chunk might still exist on the server (e.g.,
from backups in another backup group), in which case the server will
still re-use the existing chunk.
so, fast incremental does the least work, incremental does full reading
and hashing but less uploading, and we try to avoid unnecessary writes
on the server side in all cases. all of the above is also true when you
add in encryption, although obviously changing encryption mode or keys
will invalidate previous backups for purposes of reusing chunks (and
thus lead to a single full backup even if previous ones exist).
on the server side, a backup snapshots does not consist of a base and a
series of incremental diffs, it always references all the chunks that
represent this full snapshot. the magic is in the chunking and
deduplication, which allows us to store all those snapshots efficiently.
ALL snapshots are equivalent, whether it was the first one or not has no
bearing on how the snapshot or its referenced data is stored, just on
how much work it was to create and transfer it in the first place.
_______________________________________________
pve-user mailing list
[email protected]
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
[email protected]
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
