Personally, I am +1 on the coarse grain interface, and totally agree with your points. As long as we separate vmsnasphotmanager and vmsnapshotstrategy, and provide enough helper functions(such as quiesce / un-quiesce vm) for vendors, then write a new vmsnapshotStrategy should be easy.
> -----Original Message----- > From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com] > Sent: Wednesday, October 09, 2013 9:13 PM > To: dev@cloudstack.apache.org > Subject: Re: [DISCUSS] Pluggable VM snapshot related operations? > > Edison, > > I would lean toward doing the coarse grain interface only. I'm having a hard > time seeing how the whole flow is generic and makes sense for everyone. > With starting with the coarse grain you have the advantage in that you avoid > possible upfront over engineering/over design that could wreak havoc down > the line. If you implement the VMSnapshotStrategy and find that it really is > useful to other implementations, you can then implement the fine grain > interface later to allow others to benefit from it. > > Darren > > On Wed, Oct 9, 2013 at 8:54 PM, Mike Tutkowski > <mike.tutkow...@solidfire.com> wrote: > > Hey guys, > > > > I haven't been giving this thread much attention, but am reviewing it > > somewhat now. > > > > I'm not really clear how this would work if, say, a VM has two data > > disks and they are not being provided by the same vendor. > > > > Can someone clarify that for me? > > > > My understanding for how this works today is that it doesn't matter. > > For XenServer, a VDI is on an SR, which could be supported by storage > vendor X. > > Another VDI could be on another SR, supported by storage vendor Y. > > > > In this case, a new VDI appears on each SR after a hypervisor snapshot. > > > > Same idea for VMware. > > > > I don't really know how (or if) this works for KVM. > > > > I'm not clear how this multi-vendor situation would play out in this > > pluggable approach. > > > > Thanks! > > > > > > On Tue, Oct 8, 2013 at 4:43 PM, Edison Su <edison...@citrix.com> wrote: > > > >> > >> > >> > -----Original Message----- > >> > From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com] > >> > Sent: Tuesday, October 08, 2013 2:54 PM > >> > To: dev@cloudstack.apache.org > >> > Subject: Re: [DISCUSS] Pluggable VM snapshot related operations? > >> > > >> > A hypervisor snapshot will snapshot memory also. So determining > >> > whether > >> The memory is optional for hypervisor vm snapshot, a.k.a, the > >> "Disk-only > >> snapshots": > >> http://support.citrix.com/proddocs/topic/xencenter-61/xs-xc-vms-snaps > >> hots-about.html It's supported by both xenserver/kvm/vmware. > >> > >> > do to the hypervisor snapshot from the quiesce option does not seem > >> > proper. > >> > > >> > Sorry, for all the questions, I'm trying to get to the point of > >> understand if this > >> > functionality makes sense at this point of code or if maybe their > >> > is a > >> different > >> > approach. This is what I'm seeing, what if we state it this way > >> > > >> > 1) VM snapshot, AFAIK, are not backed up today and exist solely on > >> primary. > >> > What if we added a backup phase to VM snapshots that can be > >> > optionally supported by the storage providers to possibly backup > >> > the VM snapshot volumes. > >> It's not about backup vm snapshot, it's about how to take vm snapshot. > >> Usually, take/revert vm snapshot is handled by hypervisor itself, but > >> in NetApp(or other storage vendor) case, They want to change the > >> default behavior of hypervisor-base vm snapshot. > >> > >> Some examples: > >> 1. take hypervisor based vm snapshots, on primary storage, hypervisor > >> will maintain the snapshot chain. > >> 2. take vm snapshot through NetApp: > >> a. first, quiesce VM if user specified. There is no separate API > >> to quiesce VM on the hypervisor, so here we will take a VM snapshot > >> through hypervisor API call, hypervisor will take volume snapshot on > >> each volume of the VM. Let's say, on the primary storage, the disk > >> chain looks like: > >> base-image > >> | > >> V > >> Parent disk > >> / \ > >> V V > >> Current disk snapshot-a > >> b. from snapshot-a, find out its parent disk, then take snapshot > >> through NetApp > >> c. un- quiesce VM, here, go to hypervisor, delete snapshot > >> "snapshot-a", hypervisor should be able to consolidate current disk > >> and "parent disk" into one disk, thus from hypervisor point of view , > >> there is always, at most, only one snapshot for the VM. > >> For revert VM snapshot, as long as the VM is stopped, NetApp can > >> revert the snapshot created on NetApp storage easily, and efficiently. > >> The benefit of this whole process, as Chris pointed out, if the > >> snapshot chain is quite long, hypervisor based VM snapshot will get > >> performance hit. > >> > >> > > >> > 2) Additionally you want to be able to backup multiple disks at > >> > once, regardless of VM snapshot. Why don't we add the ability to > >> > put > >> volumeIds in > >> > snapshot cmd that if the storage provider supports it will get a > >> > batch of volumeIds. > >> > > >> > Now I know we talked about 2 and there was some concerns about it > >> > (mostly from me), but I think we could work through those concerns > >> > (forgot what they were...). Right now I just get the feeling we > >> > are shoehorning some functionality into VM snapshot that isn't > >> > quite the right fit. The "no > >> quiesce" > >> > flow just doesn't seem to make sense to me. > >> > >> > >> Not sure above NetApp proposed work flow makes sense to you or to > >> other body or not. If this work flow is only specific to NetApp, then > >> we don't need to enforce the whole process for everybody. > >> > >> > > >> > Darren > >> > > >> > On Tue, Oct 8, 2013 at 2:05 PM, SuichII, Christopher > >> > <chris.su...@netapp.com> wrote: > >> > > Whether the hypervisor snapshot happens depends on whether the > >> > 'quiesce' option is specified with the snapshot request. If a user > >> doesn't care > >> > about the consistency of their backup, then the hypervisor > >> snapshot/quiesce > >> > step can be skipped altogether. This of course is not the case if > >> > the > >> default > >> > provider is being used, in which case a hypervisor snapshot is the > >> > only > >> way of > >> > creating a backup since it can't be offloaded to the storage driver. > >> > > > >> > > -- > >> > > Chris Suich > >> > > chris.su...@netapp.com > >> > > NetApp Software Engineer > >> > > Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat > >> > > > >> > > On Oct 8, 2013, at 4:57 PM, Darren Shepherd > >> > > <darren.s.sheph...@gmail.com> > >> > > wrote: > >> > > > >> > >> Who is going to decide whether the hypervisor snapshot should > >> > >> actually happen or not? Or how? > >> > >> > >> > >> Darren > >> > >> > >> > >> On Tue, Oct 8, 2013 at 12:38 PM, SuichII, Christopher > >> > >> <chris.su...@netapp.com> wrote: > >> > >>> > >> > >>> -- > >> > >>> Chris Suich > >> > >>> chris.su...@netapp.com > >> > >>> NetApp Software Engineer > >> > >>> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat > >> > >>> > >> > >>> On Oct 8, 2013, at 2:24 PM, Darren Shepherd > >> > <darren.s.sheph...@gmail.com> wrote: > >> > >>> > >> > >>>> So in the implementation, when we say "quiesce" is that > >> > >>>> actually being implemented as a VM snapshot (memory and disk). > >> > >>>> And then when you say "unquiesce" you are talking about > >> > >>>> deleting the VM > >> > snapshot? > >> > >>> > >> > >>> If the VM snapshot is not going to the hypervisor, then yes, it > >> > >>> will > >> > actually be a hypervisor snapshot. Just to be clear, the unquiesce > >> > is > >> not quite > >> > a delete - it is a collapse of the VM snapshot and the active VM > >> > back > >> into one > >> > file. > >> > >>> > >> > >>>> > >> > >>>> In NetApp, what are you snapshotting? The whole netapp volume > >> > >>>> (I don't know the correct term), a file on NFS, an iscsi > >> > >>>> volume? I don't know a whole heck of a lot about the netapp > >> > >>>> snapshot > >> > capabilities. > >> > >>> > >> > >>> Essentially we are using internal APIs to create file level > >> > >>> backups > >> - don't > >> > worry too much about the terminology. > >> > >>> > >> > >>>> > >> > >>>> I know storage solutions can snapshot better and faster than > >> > >>>> hypervisors can with COW files. I've personally just been > >> > >>>> always perplexed on whats the best way to implement it. For > >> > >>>> storage solutions that are block based, its really easy to > >> > >>>> have the storage doing the snapshot. For shared file systems, > >> > >>>> like NFS, its seems way more complicated as you don't want to > >> > >>>> snapshot the entire filesystem in order to snapshot one file. > >> > >>> > >> > >>> With filesystems like NFS, things are certainly more > >> > >>> complicated, > >> but that > >> > is taken care of by our controller's operating system, Data ONTAP, > >> > and we simply use APIs to communicate with it. > >> > >>> > >> > >>>> > >> > >>>> Darren > >> > >>>> > >> > >>>> On Tue, Oct 8, 2013 at 11:10 AM, SuichII, Christopher > >> > >>>> <chris.su...@netapp.com> wrote: > >> > >>>>> I can comment on the second half. > >> > >>>>> > >> > >>>>> Through storage operations, storage providers can create > >> > >>>>> backups > >> > much faster than hypervisors and over time, their snapshots are > >> > more efficient than the snapshot chains that hypervisors create. It > >> > is true > >> that a VM > >> > snapshot taken at the storage level is slightly different as it > >> > would be > >> psuedo- > >> > quiesced, not have it's memory snapshotted. This is accomplished > >> > through hypervisor snapshots: > >> > >>>>> > >> > >>>>> 1) VM snapshot request (lets say VM 'A' > >> > >>>>> 2) Create hypervisor snapshot (optional) -VM 'A' is > >> > >>>>> snapshotted, creating active VM 'A*' > >> > >>>>> -All disk traffic now goes to VM 'A*' and A is a snapshot of 'A*' > >> > >>>>> 3) Storage driver(s) take snapshots of each volume > >> > >>>>> 4) Undo hypervisor snapshot (optional) -VM snapshot 'A' is > >> > >>>>> rolled back into VM 'A*' so the hypervisor snapshot no longer > >> > >>>>> exists > >> > >>>>> > >> > >>>>> Now, a couple notes: > >> > >>>>> -The reason this is optional is that not all users > >> > >>>>> necessarily > >> care about > >> > the memory or disk consistency of their VMs and would prefer faster > >> > snapshots to consistency. > >> > >>>>> -Preemptively, yes, we are actually taking hypervisor > >> > >>>>> snapshots > >> which > >> > means there isn't actually a performance of taking storage > >> > snapshots when quiescing the VM. However, the performance gain will > >> > come both during restoring the VM and during normal operations as > described above. > >> > >>>>> > >> > >>>>> Although you can think of it as a poor man's VM snapshot, I > >> > >>>>> would > >> > think of it more as a consistent multi-volume snapshot. Again, the > >> difference > >> > being that this snapshot was not truly quiesced like a hypervisor > >> snapshot > >> > would be. > >> > >>>>> > >> > >>>>> -- > >> > >>>>> Chris Suich > >> > >>>>> chris.su...@netapp.com > >> > >>>>> NetApp Software Engineer > >> > >>>>> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red > >> > >>>>> Hat > >> > >>>>> > >> > >>>>> On Oct 8, 2013, at 1:47 PM, Darren Shepherd > >> > <darren.s.sheph...@gmail.com> wrote: > >> > >>>>> > >> > >>>>>> My only comment is that having the return type as boolean > >> > >>>>>> and using to that indicate quiesce behaviour seems obscure > >> > >>>>>> and will probably lead to a problem later. Your basically > >> > >>>>>> saying the result of the takeVMSnapshot will only ever need > >> > >>>>>> to communicate back whether unquiesce needs to happen. > >> > >>>>>> Maybe some result > >> > object > >> > >>>>>> would be more extensible. > >> > >>>>>> > >> > >>>>>> Actually, I think I have more comments. This seems a bit > >> > >>>>>> odd to > >> me. > >> > >>>>>> Why would a storage driver in ACS implement a VM snapshot > >> > >>>>>> functionality? VM snapshot is a really a hypervisor > >> > >>>>>> orchestrated operation. So it seems like were trying to > >> > >>>>>> implement a poor mans VM snapshot. Maybe if I understood > >> > >>>>>> what NetApp was trying to do it would make more sense, but > >> > >>>>>> its all odd. To do a proper VM snapshot you need to > >> > >>>>>> snapshot memory and disk at the exact same time. How are we > >> > >>>>>> going to do that if ACS is orchestrating the VM snapshot and > >> > >>>>>> delegating to storage providers. Its not like you are going to > pause the VM.... or are you? > >> > >>>>>> > >> > >>>>>> Darren > >> > >>>>>> > >> > >>>>>> On Mon, Oct 7, 2013 at 11:59 AM, Edison Su > >> > >>>>>> <edison...@citrix.com> > >> > wrote: > >> > >>>>>>> I created a design document page at > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Pluggable+VM > >> > +s > >> > napshot+related+operations, feel free to add items on it. > >> > >>>>>>> And a new branch "pluggable_vm_snapshot" is created. > >> > >>>>>>> > >> > >>>>>>>> -----Original Message----- > >> > >>>>>>>> From: SuichII, Christopher [mailto:chris.su...@netapp.com] > >> > >>>>>>>> Sent: Monday, October 07, 2013 10:02 AM > >> > >>>>>>>> To: <dev@cloudstack.apache.org> > >> > >>>>>>>> Subject: Re: [DISCUSS] Pluggable VM snapshot related > operations? > >> > >>>>>>>> > >> > >>>>>>>> I'm a fan of option 2 - this gives us the most flexibility > >> > >>>>>>>> (as you stated). The option is given to completely > >> > >>>>>>>> override the way VM snapshots work AND storage providers > >> > >>>>>>>> are given to opportunity to work within the default VM > snapshot workflow. > >> > >>>>>>>> > >> > >>>>>>>> I believe this option should satisfy your concern, Mike. > >> > >>>>>>>> The snapshot and quiesce strategy would be in charge of > >> > communicating with the hypervisor. > >> > >>>>>>>> Storage providers should be able to leverage the default > >> > >>>>>>>> strategies and simply perform the storage operations. > >> > >>>>>>>> > >> > >>>>>>>> I don't think it should be much of an issue that new > >> > >>>>>>>> method to the storage driver interface may not apply to > >> > >>>>>>>> everyone. In fact, > >> > that is already the case. > >> > >>>>>>>> Some methods such as un/maintain(), attachToXXX() and > >> > >>>>>>>> takeSnapshot() are already not implemented by every driver > >> > >>>>>>>> - they just return false when asked if they can handle the > >> operation. > >> > >>>>>>>> > >> > >>>>>>>> -- > >> > >>>>>>>> Chris Suich > >> > >>>>>>>> chris.su...@netapp.com > >> > >>>>>>>> NetApp Software Engineer > >> > >>>>>>>> Data Center Platforms - Cloud Solutions Citrix, Cisco & > >> > >>>>>>>> Red Hat > >> > >>>>>>>> > >> > >>>>>>>> On Oct 5, 2013, at 12:11 AM, Mike Tutkowski > >> > >>>>>>>> <mike.tutkow...@solidfire.com> > >> > >>>>>>>> wrote: > >> > >>>>>>>> > >> > >>>>>>>>> Well, my first thought on this is that the storage driver > >> > >>>>>>>>> should not be telling the hypervisor to do anything. It > >> > >>>>>>>>> should be responsible for creating/deleting volumes, > snapshots, etc. > >> on > >> > its storage system only. > >> > >>>>>>>>> > >> > >>>>>>>>> > >> > >>>>>>>>> On Fri, Oct 4, 2013 at 5:57 PM, Edison Su < > >> edison...@citrix.com> > >> > wrote: > >> > >>>>>>>>> > >> > >>>>>>>>>> In 4.2, we added VM snapshot for Vmware/Xenserver. > The > >> > >>>>>>>>>> current workflow will be like the following: > >> > >>>>>>>>>> createVMSnapshot api -> VMSnapshotManagerImpl: > >> > >>>>>>>>>> creatVMSnapshot -> send CreateVMSnapshotCommand > to > >> > hypervisor to create vm snapshot. > >> > >>>>>>>>>> > >> > >>>>>>>>>> If anybody wants to change the workflow, then need to > >> > >>>>>>>>>> either change VMSnapshotManagerImpl directly or > subclass > >> > VMSnapshotManagerImpl. > >> > >>>>>>>>>> Both are not the ideal choice, as > VMSnapshotManagerImpl > >> > >>>>>>>>>> should be able to handle different ways to take vm > >> > >>>>>>>>>> snapshot, > >> > instead of hard code. > >> > >>>>>>>>>> > >> > >>>>>>>>>> The requirements for the pluggable VM snapshot coming > from: > >> > >>>>>>>>>> Storage vendor may have their optimization, such as > NetApp. > >> > >>>>>>>>>> VM snapshot can be implemented in a totally different > >> > >>>>>>>>>> way(For example, I could just send a command to guest > >> > >>>>>>>>>> VM, to tell my application to flush disk and hold disk > >> > >>>>>>>>>> write, then come to hypervisor to > >> > >>>>>>>> take a volume snapshot). > >> > >>>>>>>>>> > >> > >>>>>>>>>> If we agree on enable pluggable VM snapshot, then we > can > >> > move > >> > >>>>>>>>>> on discuss how to implement it. > >> > >>>>>>>>>> > >> > >>>>>>>>>> The possible options: > >> > >>>>>>>>>> 1. coarse grained interface. Add a VMSnapshotStrategy > >> > >>>>>>>>>> interface, which has the following interfaces: > >> > >>>>>>>>>> VMSnapshot takeVMSnapshot(VMSnapshot vmSnapshot); > >> > Boolean > >> > >>>>>>>>>> revertVMSnapshot(VMSnapshot vmSnapshot); Boolean > >> > >>>>>>>>>> DeleteVMSnapshot(VMSnapshot vmSnapshot); > >> > >>>>>>>>>> > >> > >>>>>>>>>> The work flow will be: createVMSnapshot api -> > >> > >>>>>>>> VMSnapshotManagerImpl: > >> > >>>>>>>>>> creatVMSnapshot -> VMSnapshotStrategy: > takeVMSnapshot > >> > >>>>>>>>>> VMSnapshotManagerImpl will manage VM state, do the > >> > >>>>>>>>>> sanity check, then will handle over to > VMSnapshotStrategy. > >> > >>>>>>>>>> In VMSnapshotStrategy implementation, it may just send > a > >> > >>>>>>>>>> Create/revert/delete VMSnapshotCommand to > hypervisor > >> > host, or > >> > >>>>>>>>>> do anything special operations. > >> > >>>>>>>>>> > >> > >>>>>>>>>> 2. fine-grained interface. Not only add a > >> > >>>>>>>>>> VMSnapshotStrategy interface, but also add certain > methods on the storage driver. > >> > >>>>>>>>>> The VMSnapshotStrategy interface will be the same as > option 1. > >> > >>>>>>>>>> Will add the following methods on storage driver: > >> > >>>>>>>>>> /* volumesBelongToVM is the list of volumes of the VM > >> > >>>>>>>>>> that created on this storage, storage vendor can either > >> > >>>>>>>>>> take one snapshot for this volumes in one shot, or take > >> > >>>>>>>>>> snapshot for > >> > each volume separately > >> > >>>>>>>>>> The pre-condition: vm is unquiesced. > >> > >>>>>>>>>> It will return a Boolean to indicate, do need > >> > >>>>>>>>>> unquiesce vm > >> or > >> > not. > >> > >>>>>>>>>> In the default storage driver, it will return false. > >> > >>>>>>>>>> */ > >> > >>>>>>>>>> boolean takeVMSnapshot(List<VolumeInfo> > >> > volumesBelongToVM, > >> > >>>>>>>>>> VMSnapshot vmSnapshot); Boolean > >> > >>>>>>>>>> revertVMSnapshot(List<VolumeInfo> > volumesBelongToVM, > >> > >>>>>>>>>> VMSnapshot vmSnapshot); Boolean > >> > >>>>>>>>>> deleteVMSnapshot(List<VolumeInfo> > volumesBelongToVM, > >> > >>>>>>>>>> VMSnapshot vmSNapshot); > >> > >>>>>>>>>> > >> > >>>>>>>>>> The work flow will be: createVMSnapshot api -> > >> > >>>>>>>> VMSnapshotManagerImpl: > >> > >>>>>>>>>> creatVMSnapshot -> VMSnapshotStrategy: > takeVMSnapshot -> > >> > >>>>>>>>>> storage driver:takeVMSnapshot In the implementation of > >> > >>>>>>>>>> VMSnapshotStrategy's takeVMSnapshot, the pseudo > code > >> > looks like: > >> > >>>>>>>>>> HypervisorHelper.quiesceVM(vm); > >> > >>>>>>>>>> val volumes = vm.getVolumes(); > >> > >>>>>>>>>> val maps = new Map[driver, list[VolumeInfo]](); > >> > >>>>>>>>>> Volumes.foreach(volume => maps.put(volume.getDriver, > >> > volume :: > >> > >>>>>>>>>> maps.get(volume.getdriver()))) > >> > >>>>>>>>>> val needUnquiesce = true; > >> > >>>>>>>>>> maps.foreach((driver, volumes) => needUnquiesce = > >> > >>>>>>>>>> needUnquiesce && driver.takeVMSnapshot(volumes)) > >> > >>>>>>>>>> if (needUnquiesce ) { > >> > >>>>>>>>>> HypervisorHelper.unquiesce(vm); } > >> > >>>>>>>>>> > >> > >>>>>>>>>> By default, the quiesceVM in HypervisorHelper will > >> > >>>>>>>>>> actually take vm snapshot through hypervisor. > >> > >>>>>>>>>> Does above logic makes senesce? > >> > >>>>>>>>>> > >> > >>>>>>>>>> The pros of option 1 is that: it's simple, no need to > >> > >>>>>>>>>> change storage driver interfaces. The cons is that each > >> > >>>>>>>>>> storage vendor need to implement a strategy, maybe they > >> > >>>>>>>>>> will do the > >> > same thing. > >> > >>>>>>>>>> The pros of option 2 is that, storage driver won't need > >> > >>>>>>>>>> to worry about how to quiesce/unquiesce vm. The cons is > >> > >>>>>>>>>> that, it will add these methods on each storage drivers, > >> > >>>>>>>>>> so it assumes that this work flow will work for everybody. > >> > >>>>>>>>>> > >> > >>>>>>>>>> So which option we should take? Or if you have other > >> > >>>>>>>>>> options, please let's know. > >> > >>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>> > >> > >>>>>>>>> > >> > >>>>>>>>> -- > >> > >>>>>>>>> *Mike Tutkowski* > >> > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >> > >>>>>>>>> e: mike.tutkow...@solidfire.com > >> > >>>>>>>>> o: 303.746.7302 > >> > >>>>>>>>> Advancing the way the world uses the > >> > >>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> > >> > >>>>>>>>> *(tm)* > >> > >>>>>>> > >> > >>>>> > >> > >>> > >> > > > >> > > > > > > > > -- > > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > > cloud<http://solidfire.com/solution/overview/?video=play> > > *(tm)*
