Re: [Users] Ploop Incremental backups strategy
Hello Steffan. To quickly answer your questions: 1) every successful backup is considered as a full backup, even if it is an "incremental/differential/full" backup under the hood (I know, this is counter intuitive) In short, you do not have to care about choosing which backup you want to restore because there is no computation to "rebuild" a backup. 2) "openvz-diff-backups --help" will show you all options You are right, the doc is missing: I have spent 99% of my time working on the code - and testing it - but only 1% to "build" a simple single website page ... 3) there is no intrinsic performance differences between push or pull modes: it only depends of the hardware that your are using But your question was about "external" servers having access to your backup server => use "pull" mode. This means that every client should save all backups on their own servers and, once it is done, you can download them on your backup server. If this is not possible, create a container to accept their backups (push) and another container to "pull" these backups. *WARNING AGAIN*: never, ever, store OVZDB backups in a container using Ploop layout: please, use bindfs to mount a directory from the host into the container Have a good night! Le 11/09/2020 à 16:05, tranxene50 a écrit : Hello Steffan. I am writing a "survival" guide in English: please let me some time to finish it. ;-) Have a nice day! Le 11/09/2020 à 15:53, mailingl...@tikklik.nl a écrit : Thanx, im looking in to it, it looks very nice Im missing some documentation so i have some quastions... 1) So if you take live backups Then the seccond backup is incremental any you can in case of emergancy take the last incremental backup to restore the whole CT? 2) in your cron you use this, " -l 6 -q" can you explain what that does? 3)Push and pull I miss documentation how to setup. Have you any experiance in performance on push vs pull On my own nodes i can install the script on the live node But i also have clients with nodes i backup, but i dont want them to give access directly tot he backup server. 4) at night i want to backup, from tha main backup server to a remote sitelocation In case of a big desaster 😊 what do i need tor sync? Everything with a date file/map name? and include the data dir? Mayby for the future the option to pull the last update to a remote server 😊 Thanxs for the time. Steffan -Oorspronkelijk bericht- Van: users-boun...@openvz.org Namens tranxene50 Verzonden: woensdag 9 september 2020 01:02 Aan: users@openvz.org Onderwerp: Re: [Users] Ploop Incremental backups strategy Hello! Please forgive my bad English, I live in France. A few years ago, as a hobby in the beginning, I created a file based "incremental" backup tool (that heavily relies on rsync) : https://www.openvz-diff-backups.fr It works flawlessly with OpenVZ 6 but only have been rapidly tested with OpenVZ 7. The main difference between OpenVZ 6 and 7 is that memory dump (checkpoint) is now done using CRIU instead of OpenVZ "Legacy" kernel. But, globally, the process is still the same: create a ploop snapshot, mount it, sync it (with rsync) and then create a "diff" backup (using rsync again with --link-dest). As far as I know, this is one of the rarest GPL tools (any hint appreciated!) able to backup/restore CT files and, most importantly, full memory state. Restoring a "live" backup is like resuming an OS (Windows, Linux, MacOS, etc) after it had been put to sleep: the container will resume and works again, just like nothing had happen. So you can cheat and pretend 100% uptime even if a container was down most of the time... (evidently this a joke and, please, do not play at this game: /var/log/* will betray you) Note: to migrate CT between OpenVZ 6 and 7, you must use "cold" backups because memory dumps are incompatible (OpenVZ Kernel vs CRIU). To answer your questions: 1) you can restore any previous backup because each one is considered as a full backup (no diff/incremental computing: it is just files/directories/other - and hard links) 2) because backups are just a bunch of files (and mostly hard links), you can easily browse any backup of a CT and copy any files/directories needed At the moment, development of openvz-diff-backups is on pause - because it fulfills all my needs with OpenVZ 6 - but I am in the process of moving to OpenVZ 7 in a few months. So, if you encounter a bug or an issue, please leave me a message: the tool has a very conservative approach and is designed to cleanly stop if anything unexpected/unknown/abnormal happens. Have a nice day! Le 08/09/2020 à 14:24, mailingl...@tikklik.nl a écrit : Hello, Using openvz7 Im looking ad my backup strategy I allways used rsync on openvz6 And looks like on openvz7 this can also be done on /vz/root/VEID But i dont know if i can do a full restore from that... Now im looking at snapshots https://gi
Re: [Users] openvz-diff-backups - survival guide - part one
Thanks for this, it's better than what we had before.
Jake
On Fri, Sep 11, 2020 at 5:24 PM tranxene50 <
tranxen...@openvz-diff-backups.fr> wrote:
> Hello!
>
> Here is the first part of a quick "survival" guide in order to start off
> on the right foot with openvz-diff-backups (OVZDB for short).
>
> Please, be aware that English is not my native language. So, if you see
> something weird, please quote the sentence and correct it.
>
> Equally, if something is not clear, quote and ask: I will try to answer
> the best as I can.
>
> # -
>
> Firstly, you need to be aware that OVZDB use three
> "hosts/locations/storages" and "navigate" through them:
>
> # -
>
> - SOURCE : "host" where OVZDB is installed
>
> Most of the time, this is the server on which OpenVZ is running the
> containers you want to backup.
>
> But it can be any *nix system (with Bash/OpenSSH/rsync) in order to
> replicate (upload or download) backups between REMOTE and MASTER.
>
> Everything works over SSH as follow: SOURCE -> SSH key 1 -> MASTER ->
> SSH key 2 -> REMOTE
>
> # -
>
> - MASTER : *mandatory* "host" where backups are stored (copy A)
>
> Ideally, MASTER is a dedicated server/VPS/other because OVZDB relies on
> IOPS and, the more RAM you will have to cache dentries and inodes, the
> faster OVZDB will be.
>
> However, by default, backups are stored on the the same server
> (MASTER_SSH_PATH="root@localhost:/home/backup/openvz-diff-backups").
>
> This is useful if you want to test ASAP or if you have a secondary drive
> where backups can be stored (ex: sda for OpenVZ, sdb for backups).
>
> In this case, SOURCE will communicate with MASTER (both being on the
> same server) using SSH through localhost: as soon as "ssh -p 22
> root@127.0.0.1" gives you a shell without asking for a password, you are
> done.
>
> On the contrary, if MASTER is a distant host (recommended), you need to
> adjust MASTER_SSH_PATH parameter.
>
> Ex:
> MASTER_SSH_PATH="r...@backup.my-server.net:/any-absolute-path-you-want"(trailing
>
> slash is not needed and "backup.my-server.net" will always be resolved
> to its IPV4 or IPV6 address)
>
> If you need to use a SSH port different from 22, please see
> MASTER_SSH_OPTIONS parameter in config file (openvz-diff-backups.conf).
>
> # -
>
> - REMOTE : *optional* host where backups are replicated (copy B)
>
> In order to secure backups, you may want to replicate them, if possible,
> in a different geographical location.
>
> MASTER/REMOTE "hosts" can be anything as long as a *nix system is
> present with a shell, OpenSSH (other SSH servers have not been tested
> yet) and, the most important, rsync.
>
> This can be a big fat dedicated server, a large VPS, a medium instance
> in the Cloud, a NAS at home or even - if someone is willing to test (I
> didn't because mine is too old) - an Android smartphone...
>
> SOURCE "host" always requires a Bash shell but MASTER/REMOTE "hosts"
> only need a shell (sh/dash/ash/etc) and OVZDB can also deal with
> "Busybox" instead of using standard Unix tools.
>
> In short, OVZDB does not care and will run as long as the "host" can
> handle it (which can take hours/days on very low-end hardware).
>
> # -
>
> From SOURCE, you can launch any task (more details in part 2):
>
> - backup task will "convert" containers present on SOURCE into backups
> on MASTER
>
> - restore task will "convert" backups present on MASTER into containers
> on SOURCE
>
> - upload task will replicate backups present on MASTER to REMOTE (push)
>
> - download task will replicate backups present on REMOTE to MASTER (pull)
>
> - delete task will remove backups present on MASTER and/or REMOTE(you
> choose)
>
> - destroy task will wipe "cache" present on MASTER and/or REMOTE (more
> in part 2 because it is not intuitive)
>
> - update task will check and/or update OVZDB to its latest version
> ("one-click" upgrade)
>
> # -
>
> Before going into details about each command, here are some use case
> scenarios about backups:
>
> (to be shorter, I will not talk about migrating IP addresses, adjusting
> firewalls, replacing a dedicated server and other things)
>
> - 1 server
>
> Your only choice is to store backups on the same server, if possible on
> a secondary hard drive or, better, on an external hard drive.
>
> Long story short, if you are a believer, pray! ^^
>
> - 2 servers(one for prod, one for backup)
>
> If you have enough space, store backups on prod server (copy A) and
> replicate them (push) on backup server (copy B).
>
> (or, better, on backup server, replicate backups using "pull" mode: this
> is safer because it would require that both server are compromised to
> loose all your backups)
>
> Then, use OVZDB on backup server and restore every container on a daily
> basis to speed things in the event of an emergency "switch".
>
> This way, if prod server crash, you can restore containers on backup
> ser
[Users] openvz-diff-backups - survival guide - part one
Hello!
Here is the first part of a quick "survival" guide in order to start off
on the right foot with openvz-diff-backups (OVZDB for short).
Please, be aware that English is not my native language. So, if you see
something weird, please quote the sentence and correct it.
Equally, if something is not clear, quote and ask: I will try to answer
the best as I can.
# -
Firstly, you need to be aware that OVZDB use three
"hosts/locations/storages" and "navigate" through them:
# -
- SOURCE : "host" where OVZDB is installed
Most of the time, this is the server on which OpenVZ is running the
containers you want to backup.
But it can be any *nix system (with Bash/OpenSSH/rsync) in order to
replicate (upload or download) backups between REMOTE and MASTER.
Everything works over SSH as follow: SOURCE -> SSH key 1 -> MASTER ->
SSH key 2 -> REMOTE
# -
- MASTER : *mandatory* "host" where backups are stored (copy A)
Ideally, MASTER is a dedicated server/VPS/other because OVZDB relies on
IOPS and, the more RAM you will have to cache dentries and inodes, the
faster OVZDB will be.
However, by default, backups are stored on the the same server
(MASTER_SSH_PATH="root@localhost:/home/backup/openvz-diff-backups").
This is useful if you want to test ASAP or if you have a secondary drive
where backups can be stored (ex: sda for OpenVZ, sdb for backups).
In this case, SOURCE will communicate with MASTER (both being on the
same server) using SSH through localhost: as soon as "ssh -p 22
root@127.0.0.1" gives you a shell without asking for a password, you are
done.
On the contrary, if MASTER is a distant host (recommended), you need to
adjust MASTER_SSH_PATH parameter.
Ex:
MASTER_SSH_PATH="r...@backup.my-server.net:/any-absolute-path-you-want"(trailing
slash is not needed and "backup.my-server.net" will always be resolved
to its IPV4 or IPV6 address)
If you need to use a SSH port different from 22, please see
MASTER_SSH_OPTIONS parameter in config file (openvz-diff-backups.conf).
# -
- REMOTE : *optional* host where backups are replicated (copy B)
In order to secure backups, you may want to replicate them, if possible,
in a different geographical location.
MASTER/REMOTE "hosts" can be anything as long as a *nix system is
present with a shell, OpenSSH (other SSH servers have not been tested
yet) and, the most important, rsync.
This can be a big fat dedicated server, a large VPS, a medium instance
in the Cloud, a NAS at home or even - if someone is willing to test (I
didn't because mine is too old) - an Android smartphone...
SOURCE "host" always requires a Bash shell but MASTER/REMOTE "hosts"
only need a shell (sh/dash/ash/etc) and OVZDB can also deal with
"Busybox" instead of using standard Unix tools.
In short, OVZDB does not care and will run as long as the "host" can
handle it (which can take hours/days on very low-end hardware).
# -
From SOURCE, you can launch any task (more details in part 2):
- backup task will "convert" containers present on SOURCE into backups
on MASTER
- restore task will "convert" backups present on MASTER into containers
on SOURCE
- upload task will replicate backups present on MASTER to REMOTE (push)
- download task will replicate backups present on REMOTE to MASTER (pull)
- delete task will remove backups present on MASTER and/or REMOTE(you
choose)
- destroy task will wipe "cache" present on MASTER and/or REMOTE (more
in part 2 because it is not intuitive)
- update task will check and/or update OVZDB to its latest version
("one-click" upgrade)
# -
Before going into details about each command, here are some use case
scenarios about backups:
(to be shorter, I will not talk about migrating IP addresses, adjusting
firewalls, replacing a dedicated server and other things)
- 1 server
Your only choice is to store backups on the same server, if possible on
a secondary hard drive or, better, on an external hard drive.
Long story short, if you are a believer, pray! ^^
- 2 servers(one for prod, one for backup)
If you have enough space, store backups on prod server (copy A) and
replicate them (push) on backup server (copy B).
(or, better, on backup server, replicate backups using "pull" mode: this
is safer because it would require that both server are compromised to
loose all your backups)
Then, use OVZDB on backup server and restore every container on a daily
basis to speed things in the event of an emergency "switch".
This way, if prod server crash, you can restore containers on backup
server and, because most files are already synced, you will be online
again quickly.
- 2 servers(both prod)
If you have enough space (bis), store backups - of containers of each
prod server - locally (copy A) and replicate them on the other prod
server (copy B).
(since both servers have root acc
Re: [Users] Ploop Incremental backups strategy
Hello Steffan. I am writing a "survival" guide in English: please let me some time to finish it. ;-) Have a nice day! Le 11/09/2020 à 15:53, mailingl...@tikklik.nl a écrit : Thanx, im looking in to it, it looks very nice Im missing some documentation so i have some quastions... 1) So if you take live backups Then the seccond backup is incremental any you can in case of emergancy take the last incremental backup to restore the whole CT? 2) in your cron you use this, " -l 6 -q" can you explain what that does? 3)Push and pull I miss documentation how to setup. Have you any experiance in performance on push vs pull On my own nodes i can install the script on the live node But i also have clients with nodes i backup, but i dont want them to give access directly tot he backup server. 4) at night i want to backup, from tha main backup server to a remote sitelocation In case of a big desaster 😊 what do i need tor sync? Everything with a date file/map name? and include the data dir? Mayby for the future the option to pull the last update to a remote server 😊 Thanxs for the time. Steffan -Oorspronkelijk bericht- Van: users-boun...@openvz.org Namens tranxene50 Verzonden: woensdag 9 september 2020 01:02 Aan: users@openvz.org Onderwerp: Re: [Users] Ploop Incremental backups strategy Hello! Please forgive my bad English, I live in France. A few years ago, as a hobby in the beginning, I created a file based "incremental" backup tool (that heavily relies on rsync) : https://www.openvz-diff-backups.fr It works flawlessly with OpenVZ 6 but only have been rapidly tested with OpenVZ 7. The main difference between OpenVZ 6 and 7 is that memory dump (checkpoint) is now done using CRIU instead of OpenVZ "Legacy" kernel. But, globally, the process is still the same: create a ploop snapshot, mount it, sync it (with rsync) and then create a "diff" backup (using rsync again with --link-dest). As far as I know, this is one of the rarest GPL tools (any hint appreciated!) able to backup/restore CT files and, most importantly, full memory state. Restoring a "live" backup is like resuming an OS (Windows, Linux, MacOS, etc) after it had been put to sleep: the container will resume and works again, just like nothing had happen. So you can cheat and pretend 100% uptime even if a container was down most of the time... (evidently this a joke and, please, do not play at this game: /var/log/* will betray you) Note: to migrate CT between OpenVZ 6 and 7, you must use "cold" backups because memory dumps are incompatible (OpenVZ Kernel vs CRIU). To answer your questions: 1) you can restore any previous backup because each one is considered as a full backup (no diff/incremental computing: it is just files/directories/other - and hard links) 2) because backups are just a bunch of files (and mostly hard links), you can easily browse any backup of a CT and copy any files/directories needed At the moment, development of openvz-diff-backups is on pause - because it fulfills all my needs with OpenVZ 6 - but I am in the process of moving to OpenVZ 7 in a few months. So, if you encounter a bug or an issue, please leave me a message: the tool has a very conservative approach and is designed to cleanly stop if anything unexpected/unknown/abnormal happens. Have a nice day! Le 08/09/2020 à 14:24, mailingl...@tikklik.nl a écrit : Hello, Using openvz7 Im looking ad my backup strategy I allways used rsync on openvz6 And looks like on openvz7 this can also be done on /vz/root/VEID But i dont know if i can do a full restore from that... Now im looking at snapshots https://github.com/TamCore/vzpbackup it can make full backups and incremental backups Is someone using this script? i have some question hope someone can help the incremental backups are nice to save space and time on a remote backupserver But how does a restore works. For what i know you can only restore a full snapshot, the incremental backups have only the changed files Question 2. Is it possible to extract a file from the backup for a single file restore? And if so can someone tell me how? Thanxs Steffan ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users -- tranxene50 tranxen...@openvz-diff-backups.fr ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users -- tranxene50 tranxen...@openvz-diff-backups.fr ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] Ploop Incremental backups strategy
Thanx, im looking in to it, it looks very nice Im missing some documentation so i have some quastions... 1) So if you take live backups Then the seccond backup is incremental any you can in case of emergancy take the last incremental backup to restore the whole CT? 2) in your cron you use this, " -l 6 -q" can you explain what that does? 3)Push and pull I miss documentation how to setup. Have you any experiance in performance on push vs pull On my own nodes i can install the script on the live node But i also have clients with nodes i backup, but i dont want them to give access directly tot he backup server. 4) at night i want to backup, from tha main backup server to a remote sitelocation In case of a big desaster 😊 what do i need tor sync? Everything with a date file/map name? and include the data dir? Mayby for the future the option to pull the last update to a remote server 😊 Thanxs for the time. Steffan -Oorspronkelijk bericht- Van: users-boun...@openvz.org Namens tranxene50 Verzonden: woensdag 9 september 2020 01:02 Aan: users@openvz.org Onderwerp: Re: [Users] Ploop Incremental backups strategy Hello! Please forgive my bad English, I live in France. A few years ago, as a hobby in the beginning, I created a file based "incremental" backup tool (that heavily relies on rsync) : https://www.openvz-diff-backups.fr It works flawlessly with OpenVZ 6 but only have been rapidly tested with OpenVZ 7. The main difference between OpenVZ 6 and 7 is that memory dump (checkpoint) is now done using CRIU instead of OpenVZ "Legacy" kernel. But, globally, the process is still the same: create a ploop snapshot, mount it, sync it (with rsync) and then create a "diff" backup (using rsync again with --link-dest). As far as I know, this is one of the rarest GPL tools (any hint appreciated!) able to backup/restore CT files and, most importantly, full memory state. Restoring a "live" backup is like resuming an OS (Windows, Linux, MacOS, etc) after it had been put to sleep: the container will resume and works again, just like nothing had happen. So you can cheat and pretend 100% uptime even if a container was down most of the time... (evidently this a joke and, please, do not play at this game: /var/log/* will betray you) Note: to migrate CT between OpenVZ 6 and 7, you must use "cold" backups because memory dumps are incompatible (OpenVZ Kernel vs CRIU). To answer your questions: 1) you can restore any previous backup because each one is considered as a full backup (no diff/incremental computing: it is just files/directories/other - and hard links) 2) because backups are just a bunch of files (and mostly hard links), you can easily browse any backup of a CT and copy any files/directories needed At the moment, development of openvz-diff-backups is on pause - because it fulfills all my needs with OpenVZ 6 - but I am in the process of moving to OpenVZ 7 in a few months. So, if you encounter a bug or an issue, please leave me a message: the tool has a very conservative approach and is designed to cleanly stop if anything unexpected/unknown/abnormal happens. Have a nice day! Le 08/09/2020 à 14:24, mailingl...@tikklik.nl a écrit : > Hello, > > Using openvz7 > Im looking ad my backup strategy > I allways used rsync on openvz6 > And looks like on openvz7 this can also be done on /vz/root/VEID But i > dont know if i can do a full restore from that... > > > Now im looking at snapshots > https://github.com/TamCore/vzpbackup > it can make full backups and incremental backups > > Is someone using this script? > i have some question hope someone can help > > the incremental backups are nice to save space and time on a remote > backupserver But how does a restore works. > For what i know you can only restore a full snapshot, the incremental > backups have only the changed files > > Question 2. > Is it possible to extract a file from the backup for a single file restore? > And if so can someone tell me how? > > > Thanxs > Steffan > > > > > > > > > > ___ > Users mailing list > Users@openvz.org > https://lists.openvz.org/mailman/listinfo/users -- tranxene50 tranxen...@openvz-diff-backups.fr ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
