Thanks for clarifying Nicolas. I just wanted to make sure that there wasn't any confusion in this conversation as a 'snapshot' has different meanings to different people/vendors, so I wanted to make sure we were all on the same page.
BTW, for what it is worth, I love this initiative. This is an important feature integration which I know a LOT of organizations will appreciate. Cheers, *Will Stevens* Chief Technology Officer c 514.826.0190 <https://goo.gl/NYZ8KK> On Thu, Jun 7, 2018 at 12:57 PM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > First of all, the framework is agnostic of any backup vendor/provider. It > just provides the way to communicate with the provider to allow users to > backup their VMs. > > > Will is right, backups are not the same as snapshots, I got confused with > the Veeam backup which starts from taking a snapshot to generate the > backup. The way they are generated are up to each vendor, and it is not > relevant to the framework. Sorry for the confusion, we are also introducing > the Veeam plugin in top of this framework. > > > So, in terms of this framework, it does not matter what a backup is, we > just keep a reference to an existing backup on each vendor. > > > I would also like to state that this framework is not using a snapshot > mechanism for backups in CloudStack. It is just providing interfaces and > mechanisms for vendors/providers plugins in CloudStack. Implementation of > each plugin should be relevant to the vendor and any backup handling will > be external to CloudStack. > > > I think that the diagram in the FS could explain what the task of the > framework is: > https://cwiki.apache.org/confluence/download/attachments/85476500/Backup%20and%20Recovery%20UML.png?version=1&modificationDate=1528367734183&api=v2. > The framework simply allows management server to communicate with backup > and recovery vendors/providers. To allow CloudStack communication with an > specific vendor, a plugin should be implemented for it (as we are doing > with Veeam for example). > > > Please let me know if it is still confusing and I'll try to explain better > > ________________________________ > From: Todd Pigram <t...@toddpigram.com> > Sent: Thursday, June 7, 2018 12:11:18 PM > To: dev@cloudstack.apache.org > Subject: Re: [PROPOSE] Backup and Recovery Framework > > Now outside of our ACS, I use Veeam for vCloud and vSphere and use Alike > for our XS pools. Inside ACS, the VMs use MaxBackup > > > nicolas.vazq...@shapeblue.com > www.shapeblue.com > , > @shapeblue > > > > On Thu, Jun 7, 2018 at 11:09 AM, Todd Pigram <t...@toddpigram.com> wrote: > > > I agree completely. That is why I use inside VM backup products > currently. > > > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens <wstev...@cloudops.com> > > wrote: > > > >> I have not been following this thread entirely, but I would not count on > >> our snapshot mechanism for backups. If something like Veeam is used, > then > >> an agent is in play and that agent will quiesce the running processes, > >> such > >> as MySQL. This is important in order to do backup and DR as you will > lose > >> transactions in memory and can potentially have a corrupted backup. > >> > >> Snapshots are not backups!!! > >> > >> *Will Stevens* > >> Chief Technology Officer > >> c 514.826.0190 > >> > >> <https://goo.gl/NYZ8KK> > >> > >> > >> On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner < > >> rafaelweingart...@gmail.com> wrote: > >> > >> > Ok, now I start to understand. So, we will be handling snapshots and > >> they > >> > will be sent to a third-party system. > >> > > >> > Now let’s proceed. If I configure this “feature” with external=false. > >> What > >> > does it do? Nothing? I mean, what is the purpose of this “external” > >> > parameter? > >> > > >> > > >> > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < > >> > nicolas.vazq...@shapeblue.com> wrote: > >> > > >> > > Its format may depend on the provider, but under the hood it is a > >> > snapshot > >> > > which can be full or incremental and it is copied and stored in a > >> backup > >> > > repository in one of the backup servers. For example, if you use > >> Veeam, > >> > you > >> > > will find your backups stored as .vbk (for full backups) and .vib > (for > >> > > incremental backups). It is the task of the backup provider to > create > >> and > >> > > move these files to the backup repository. > >> > > > >> > > ________________________________ > >> > > From: Rafael Weingärtner <rafaelweingart...@gmail.com> > >> > > Sent: Thursday, June 7, 2018 10:39:35 AM > >> > > To: dev > >> > > Subject: Re: [PROPOSE] Backup and Recovery Framework > >> > > > >> > > Ok, there are a lot of things, which are kind of blurry to me. So, > >> let’s > >> > > try to go step by step. > >> > > > >> > > Let me ask again. What is a backup of the VM? You mention > >> restore-point. > >> > > What is it? Is it a snapshot in the hypervisor (complete/partial > >> > snapshot)? > >> > > If it is not a snapshot, what is it then? Are you installing some > >> agent > >> > in > >> > > the VM and then copying/mirroring the files in the VM’s file system? > >> > > > >> > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < > >> > > nicolas.vazq...@shapeblue.com> wrote: > >> > > > >> > > > Hi Daniel and Rafael, > >> > > > > >> > > > > >> > > > I have updated the diagram picture on the FS, please let me know > if > >> the > >> > > > problem persists. > >> > > > > >> > > > > >> > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right > >> about > >> > > > restoring VMs, please let me update the FS. Regarding your > >> question, we > >> > > > have considered the backups as a first class citizens, meaning > that > >> a > >> > > > backup would continue existing unless its originating VM has been > >> > > > removed/expunged. This way, listBackups API method would list them > >> in > >> > > order > >> > > > to allow a user to restore a deleted VM from a backup. > >> > > > > >> > > > > >> > > > @Rafael the idea of the Dummy provider is simply returning the > >> expected > >> > > > entities from a backup provider. For example, the BackupManager > >> could > >> > ask > >> > > > the BackupProvider to list its backups and would expect a list of > >> > > backups. > >> > > > As the Dummy Provider implements the BackupProvider interface, it > >> has > >> > its > >> > > > own implementation of listBackups() method and simply returns some > >> mock > >> > > > entities to the Backup Manager. It does not store any backup, it > >> simply > >> > > > creates them in execution time and returns them. > >> > > > > >> > > > > >> > > > From the CloudStack point of view, Backups are stored externally, > on > >> > the > >> > > > Backup and Recovery provider servers and are handled by them. We > >> keep > >> > > track > >> > > > of the existing backups in the Backup provider. Implementation may > >> > depend > >> > > > on the Backup and Recovery provider, would consist on snapshots, > but > >> > are > >> > > > handled by the provider and not by CloudStack. > >> > > > > >> > > > > >> > > > Backup policies are created on the Backup and Recovery provider > and > >> > > > provide retention options and periods as well as backup times. For > >> > > example, > >> > > > the provider may expose Golden, Silver and Bronze policies and > have > >> > > > different options for each of them. > >> > > > > >> > > > > >> > > > The 'external' parameter on the listBackupPolicies API method > would > >> > only > >> > > > ask the Backup and Recovery provider to list their policies if it > is > >> > set > >> > > to > >> > > > true. If it is not set, or set to false, it just simply returns > the > >> > > > existing mapped policies in CloudStack. Initially, CloudStack does > >> not > >> > > > store any backup policy, those should be imported/mapped by the > >> > > > 'importBackupPolicy' API method. > >> > > > > >> > > > > >> > > > Finally, we consider backups as restore points. As mentioned > above, > >> > > > backups are stored on the Backup and Recovery provider servers, so > >> we > >> > > keep > >> > > > a mapping between the CloudStack entity and the backup, which can > be > >> > used > >> > > > to restore VMs. > >> > > > > >> > > > > >> > > > Please let me know if anything. > >> > > > > >> > > > > >> > > > Regards, > >> > > > > >> > > > Nicolas Vazquez > >> > > > > >> > > > ________________________________ > >> > > > From: Rafael Weingärtner <rafaelweingart...@gmail.com> > >> > > > Sent: Thursday, June 7, 2018 6:55:57 AM > >> > > > To: dev > >> > > > Subject: Re: [PROPOSE] Backup and Recovery Framework > >> > > > > >> > > > Hi Nicolas, > >> > > > > >> > > > The idea is great. I have some questions though. > >> > > > > >> > > > Before getting to the questions, your pictures are broken, or at > >> least > >> > > they > >> > > > seem broken to me. > >> > > > > >> > > > How would the “dummy” backup provider work? I mean, what does it > do? > >> > > where > >> > > > does it storage the "backup"? > >> > > > What is the difference from this Backup to a Snapshot? Is it > simply > >> > > storing > >> > > > a snapshot outside the cloud structure? > >> > > > How would these “backup policies” look like? I mean, what can I > >> > > configure? > >> > > > Regarding that parameter “external” of your backup policy, how > >> would it > >> > > > work if we set it to false? > >> > > > What are you defining as a backup (what do you mean by backup in > the > >> > > > context of this feature?)? I mean, is it simply a snapshot file > that > >> > you > >> > > > store somewhere else? > >> > > > > >> > > > > >> > > > nicolas.vazq...@shapeblue.com > >> > > > www.shapeblue.com<http://www.shapeblue.com> > >> > > > , > >> > > > @shapeblue > >> > > > > >> > > > > >> > > > > >> > > > >> > > nicolas.vazq...@shapeblue.com > >> > > www.shapeblue.com<http://www.shapeblue.com> > >> > > , > >> > > @shapeblue > >> > > > >> > > > >> > > > >> > > > On Thu, Jun 7, 2018 at 6:05 AM, <daniel.herrm...@zv.fraunhofer.de > > > >> > > wrote: > >> > > > > >> > > > > Hi all, > >> > > > > > >> > > > > +1 for this feature request, thank you for bringing this to > life! > >> > > Volume > >> > > > > snapshots are what our users currently use as backup solution, > >> which > >> > > has > >> > > > > some inherent problems such as handling large volumes and the > >> > handling > >> > > of > >> > > > > multiple volumes (consistency). > >> > > > > > >> > > > > One thought: when restoring a VM, I think there are two cases to > >> > > > consider: > >> > > > > > >> > > > > - The VM is still in CS and the user just wants to restore an > old > >> > > version > >> > > > > of the VM > >> > > > > - The VM has already been deleted and the user wants to restore > >> the > >> > > > > deleted VM > >> > > > > > >> > > > > As listBackups is described as " listBackups API method. List > >> > existing > >> > > > > backups for a VM". Is the second use case supported at all? > >> > > > > > >> > > > > I like the approach not to define the backup policies in CS but > in > >> > > > > whatever tool is used in the end, as otherwise a lot of features > >> > might > >> > > be > >> > > > > lost. We as services providers need to create them in the first > >> > place, > >> > > > but > >> > > > > I think this is the better solution. > >> > > > > > >> > > > > Regards > >> > > > > Daniel > >> > > > > > >> > > > > On 06.06.18, 14:43, "Nicolas Vazquez" < > >> > nicolas.vazq...@shapeblue.com> > >> > > > > wrote: > >> > > > > > >> > > > > Hi all, > >> > > > > > >> > > > > > >> > > > > We would like to introduce a new framework into CloudStack > >> that > >> > > will > >> > > > > allow adding Backup and Recovery providers as plugins. This > >> framework > >> > > > goal > >> > > > > is allowing users to backup their guest VMs for recovery > purposes. > >> > > > > > >> > > > > > >> > > > > Please find the FS on this link: https://cwiki.apache.org/ > >> > > > > confluence/display/CLOUDSTACK/Backup+and+Recovery+Framework > >> > > > > > >> > > > > > >> > > > > Regards, > >> > > > > > >> > > > > Nicolas Vazquez > >> > > > > > >> > > > > nicolas.vazq...@shapeblue.com > >> > > > > www.shapeblue.com<http://www.shapeblue.com> > >> > > > > , > >> > > > > @shapeblue > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > >> > > > -- > >> > > > Rafael Weingärtner > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > Rafael Weingärtner > >> > > > >> > > >> > > >> > > >> > -- > >> > Rafael Weingärtner > >> > > >> > > > > > > > > -- > > > > > > Todd Pigram > > http://about.me/ToddPigram > > www.linkedin.com/in/toddpigram/<http://www.linkedin.com/in/toddpigram/> > > @pigram86 on twitter > > https://plus.google.com/+ToddPigram86 > > Mobile - 216-224-5769 > > > > PGP Public Key > > <http://pgp.mit.edu/pks/lookup?op=get&search=0x96B7B0F0C55933BB> > > > > > > > -- > > > Todd Pigram > http://about.me/ToddPigram > www.linkedin.com/in/toddpigram/<http://www.linkedin.com/in/toddpigram/> > @pigram86 on twitter > https://plus.google.com/+ToddPigram86 > Mobile - 216-224-5769 > > PGP Public Key > <http://pgp.mit.edu/pks/lookup?op=get&search=0x96B7B0F0C55933BB> >
