"The work flow will be: createVMSnapshot api -> VMSnapshotManagerImpl: creatVMSnapshot -> VMSnapshotStrategy: takeVMSnapshot -> storage driver:takeVMSnapshot"
I also think it's a bit weird for the storage driver to have any knowledge of VM snapshots. I would think another part of the system would quiesce (or not) the VM in question and then the takeSnapshot method would be called on the driver. I might have missed something...why does the driver "care" if the snapshot to be taken is going to be in a consistent state or not (I understand why the user care, but not the storage driver)? Why is that not a problem for some other part of the system that is aware of hypervisor snapshots? Shouldn't the driver just take a snapshot (or snapshots) as it is instructed to do (regardless of whether or not a VM is quiesced)? Basically I'm wondering why we need two "take snapshot" methods on the driver. On Wed, Oct 9, 2013 at 11:24 PM, Mike Tutkowski < mike.tutkow...@solidfire.com> wrote: > Yeah, I'm not really clear how the snapshot strategy works if you have > multiple vendors that implement that interface either. > > > On Wed, Oct 9, 2013 at 10:12 PM, Darren Shepherd < > darren.s.sheph...@gmail.com> wrote: > >> Edison, >> >> I would lean toward doing the coarse grain interface only. I'm having >> a hard time seeing how the whole flow is generic and makes sense for >> everyone. With starting with the coarse grain you have the advantage >> in that you avoid possible upfront over engineering/over design that >> could wreak havoc down the line. If you implement the >> VMSnapshotStrategy and find that it really is useful to other >> implementations, you can then implement the fine grain interface later >> to allow others to benefit from it. >> >> Darren >> >> On Wed, Oct 9, 2013 at 8:54 PM, Mike Tutkowski >> <mike.tutkow...@solidfire.com> wrote: >> > Hey guys, >> > >> > I haven't been giving this thread much attention, but am reviewing it >> > somewhat now. >> > >> > I'm not really clear how this would work if, say, a VM has two data >> disks >> > and they are not being provided by the same vendor. >> > >> > Can someone clarify that for me? >> > >> > My understanding for how this works today is that it doesn't matter. For >> > XenServer, a VDI is on an SR, which could be supported by storage >> vendor X. >> > Another VDI could be on another SR, supported by storage vendor Y. >> > >> > In this case, a new VDI appears on each SR after a hypervisor snapshot. >> > >> > Same idea for VMware. >> > >> > I don't really know how (or if) this works for KVM. >> > >> > I'm not clear how this multi-vendor situation would play out in this >> > pluggable approach. >> > >> > Thanks! >> > >> > >> > On Tue, Oct 8, 2013 at 4:43 PM, Edison Su <edison...@citrix.com> wrote: >> > >> >> >> >> >> >> > -----Original Message----- >> >> > From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com] >> >> > Sent: Tuesday, October 08, 2013 2:54 PM >> >> > To: dev@cloudstack.apache.org >> >> > Subject: Re: [DISCUSS] Pluggable VM snapshot related operations? >> >> > >> >> > A hypervisor snapshot will snapshot memory also. So determining >> whether >> >> The memory is optional for hypervisor vm snapshot, a.k.a, the >> "Disk-only >> >> snapshots": >> >> >> http://support.citrix.com/proddocs/topic/xencenter-61/xs-xc-vms-snapshots-about.html >> >> It's supported by both xenserver/kvm/vmware. >> >> >> >> > do to the hypervisor snapshot from the quiesce option does not seem >> >> > proper. >> >> > >> >> > Sorry, for all the questions, I'm trying to get to the point of >> >> understand if this >> >> > functionality makes sense at this point of code or if maybe their is >> a >> >> different >> >> > approach. This is what I'm seeing, what if we state it this way >> >> > >> >> > 1) VM snapshot, AFAIK, are not backed up today and exist solely on >> >> primary. >> >> > What if we added a backup phase to VM snapshots that can be >> optionally >> >> > supported by the storage providers to possibly backup the VM snapshot >> >> > volumes. >> >> It's not about backup vm snapshot, it's about how to take vm snapshot. >> >> Usually, take/revert vm snapshot is handled by hypervisor itself, but >> in >> >> NetApp(or other storage vendor) case, >> >> They want to change the default behavior of hypervisor-base vm >> snapshot. >> >> >> >> Some examples: >> >> 1. take hypervisor based vm snapshots, on primary storage, hypervisor >> will >> >> maintain the snapshot chain. >> >> 2. take vm snapshot through NetApp: >> >> a. first, quiesce VM if user specified. There is no separate API >> to >> >> quiesce VM on the hypervisor, so here we will >> >> take a VM snapshot through hypervisor API call, hypervisor will take >> >> volume snapshot on each volume of the VM. Let's say, on the primary >> >> storage, the disk chain looks like: >> >> base-image >> >> | >> >> V >> >> Parent disk >> >> / \ >> >> V V >> >> Current disk snapshot-a >> >> b. from snapshot-a, find out its parent disk, then take snapshot >> >> through NetApp >> >> c. un- quiesce VM, here, go to hypervisor, delete snapshot >> >> "snapshot-a", hypervisor should be able to consolidate current disk and >> >> "parent disk" into one disk, thus from hypervisor point of view >> >> , there is always, at most, only one snapshot for the VM. >> >> For revert VM snapshot, as long as the VM is stopped, NetApp can >> >> revert the snapshot created on NetApp storage easily, and efficiently. >> >> The benefit of this whole process, as Chris pointed out, if the >> >> snapshot chain is quite long, hypervisor based VM snapshot will get >> >> performance hit. >> >> >> >> > >> >> > 2) Additionally you want to be able to backup multiple disks at once, >> >> > regardless of VM snapshot. Why don't we add the ability to put >> >> volumeIds in >> >> > snapshot cmd that if the storage provider supports it will get a >> batch of >> >> > volumeIds. >> >> > >> >> > Now I know we talked about 2 and there was some concerns about it >> (mostly >> >> > from me), but I think we could work through those concerns (forgot >> what >> >> > they were...). Right now I just get the feeling we are shoehorning >> some >> >> > functionality into VM snapshot that isn't quite the right fit. The >> "no >> >> quiesce" >> >> > flow just doesn't seem to make sense to me. >> >> >> >> >> >> Not sure above NetApp proposed work flow makes sense to you or to other >> >> body or not. If this work flow is only specific to NetApp, then we >> don't >> >> need to enforce the whole process for everybody. >> >> >> >> > >> >> > Darren >> >> > >> >> > On Tue, Oct 8, 2013 at 2:05 PM, SuichII, Christopher >> >> > <chris.su...@netapp.com> wrote: >> >> > > Whether the hypervisor snapshot happens depends on whether the >> >> > 'quiesce' option is specified with the snapshot request. If a user >> >> doesn't care >> >> > about the consistency of their backup, then the hypervisor >> >> snapshot/quiesce >> >> > step can be skipped altogether. This of course is not the case if the >> >> default >> >> > provider is being used, in which case a hypervisor snapshot is the >> only >> >> way of >> >> > creating a backup since it can't be offloaded to the storage driver. >> >> > > >> >> > > -- >> >> > > Chris Suich >> >> > > chris.su...@netapp.com >> >> > > NetApp Software Engineer >> >> > > Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat >> >> > > >> >> > > On Oct 8, 2013, at 4:57 PM, Darren Shepherd >> >> > > <darren.s.sheph...@gmail.com> >> >> > > wrote: >> >> > > >> >> > >> Who is going to decide whether the hypervisor snapshot should >> >> > >> actually happen or not? Or how? >> >> > >> >> >> > >> Darren >> >> > >> >> >> > >> On Tue, Oct 8, 2013 at 12:38 PM, SuichII, Christopher >> >> > >> <chris.su...@netapp.com> wrote: >> >> > >>> >> >> > >>> -- >> >> > >>> Chris Suich >> >> > >>> chris.su...@netapp.com >> >> > >>> NetApp Software Engineer >> >> > >>> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat >> >> > >>> >> >> > >>> On Oct 8, 2013, at 2:24 PM, Darren Shepherd >> >> > <darren.s.sheph...@gmail.com> wrote: >> >> > >>> >> >> > >>>> So in the implementation, when we say "quiesce" is that actually >> >> > >>>> being implemented as a VM snapshot (memory and disk). And then >> >> > >>>> when you say "unquiesce" you are talking about deleting the VM >> >> > snapshot? >> >> > >>> >> >> > >>> If the VM snapshot is not going to the hypervisor, then yes, it >> will >> >> > actually be a hypervisor snapshot. Just to be clear, the unquiesce is >> >> not quite >> >> > a delete - it is a collapse of the VM snapshot and the active VM back >> >> into one >> >> > file. >> >> > >>> >> >> > >>>> >> >> > >>>> In NetApp, what are you snapshotting? The whole netapp volume >> (I >> >> > >>>> don't know the correct term), a file on NFS, an iscsi volume? I >> >> > >>>> don't know a whole heck of a lot about the netapp snapshot >> >> > capabilities. >> >> > >>> >> >> > >>> Essentially we are using internal APIs to create file level >> backups >> >> - don't >> >> > worry too much about the terminology. >> >> > >>> >> >> > >>>> >> >> > >>>> I know storage solutions can snapshot better and faster than >> >> > >>>> hypervisors can with COW files. I've personally just been >> always >> >> > >>>> perplexed on whats the best way to implement it. For storage >> >> > >>>> solutions that are block based, its really easy to have the >> storage >> >> > >>>> doing the snapshot. For shared file systems, like NFS, its >> seems >> >> > >>>> way more complicated as you don't want to snapshot the entire >> >> > >>>> filesystem in order to snapshot one file. >> >> > >>> >> >> > >>> With filesystems like NFS, things are certainly more complicated, >> >> but that >> >> > is taken care of by our controller's operating system, Data ONTAP, >> and we >> >> > simply use APIs to communicate with it. >> >> > >>> >> >> > >>>> >> >> > >>>> Darren >> >> > >>>> >> >> > >>>> On Tue, Oct 8, 2013 at 11:10 AM, SuichII, Christopher >> >> > >>>> <chris.su...@netapp.com> wrote: >> >> > >>>>> I can comment on the second half. >> >> > >>>>> >> >> > >>>>> Through storage operations, storage providers can create >> backups >> >> > much faster than hypervisors and over time, their snapshots are more >> >> > efficient than the snapshot chains that hypervisors create. It is >> true >> >> that a VM >> >> > snapshot taken at the storage level is slightly different as it >> would be >> >> psuedo- >> >> > quiesced, not have it's memory snapshotted. This is accomplished >> through >> >> > hypervisor snapshots: >> >> > >>>>> >> >> > >>>>> 1) VM snapshot request (lets say VM 'A' >> >> > >>>>> 2) Create hypervisor snapshot (optional) -VM 'A' is >> snapshotted, >> >> > >>>>> creating active VM 'A*' >> >> > >>>>> -All disk traffic now goes to VM 'A*' and A is a snapshot of >> 'A*' >> >> > >>>>> 3) Storage driver(s) take snapshots of each volume >> >> > >>>>> 4) Undo hypervisor snapshot (optional) -VM snapshot 'A' is >> rolled >> >> > >>>>> back into VM 'A*' so the hypervisor snapshot no longer exists >> >> > >>>>> >> >> > >>>>> Now, a couple notes: >> >> > >>>>> -The reason this is optional is that not all users necessarily >> >> care about >> >> > the memory or disk consistency of their VMs and would prefer faster >> >> > snapshots to consistency. >> >> > >>>>> -Preemptively, yes, we are actually taking hypervisor snapshots >> >> which >> >> > means there isn't actually a performance of taking storage snapshots >> when >> >> > quiescing the VM. However, the performance gain will come both during >> >> > restoring the VM and during normal operations as described above. >> >> > >>>>> >> >> > >>>>> Although you can think of it as a poor man's VM snapshot, I >> would >> >> > think of it more as a consistent multi-volume snapshot. Again, the >> >> difference >> >> > being that this snapshot was not truly quiesced like a hypervisor >> >> snapshot >> >> > would be. >> >> > >>>>> >> >> > >>>>> -- >> >> > >>>>> Chris Suich >> >> > >>>>> chris.su...@netapp.com >> >> > >>>>> NetApp Software Engineer >> >> > >>>>> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat >> >> > >>>>> >> >> > >>>>> On Oct 8, 2013, at 1:47 PM, Darren Shepherd >> >> > <darren.s.sheph...@gmail.com> wrote: >> >> > >>>>> >> >> > >>>>>> My only comment is that having the return type as boolean and >> >> > >>>>>> using to that indicate quiesce behaviour seems obscure and >> will >> >> > >>>>>> probably lead to a problem later. Your basically saying the >> >> > >>>>>> result of the takeVMSnapshot will only ever need to >> communicate >> >> > >>>>>> back whether unquiesce needs to happen. Maybe some result >> >> > object >> >> > >>>>>> would be more extensible. >> >> > >>>>>> >> >> > >>>>>> Actually, I think I have more comments. This seems a bit odd >> to >> >> me. >> >> > >>>>>> Why would a storage driver in ACS implement a VM snapshot >> >> > >>>>>> functionality? VM snapshot is a really a hypervisor >> orchestrated >> >> > >>>>>> operation. So it seems like were trying to implement a poor >> mans >> >> > >>>>>> VM snapshot. Maybe if I understood what NetApp was trying to >> do >> >> > >>>>>> it would make more sense, but its all odd. To do a proper VM >> >> > >>>>>> snapshot you need to snapshot memory and disk at the exact >> same >> >> > >>>>>> time. How are we going to do that if ACS is orchestrating >> the VM >> >> > >>>>>> snapshot and delegating to storage providers. Its not like >> you >> >> > >>>>>> are going to pause the VM.... or are you? >> >> > >>>>>> >> >> > >>>>>> Darren >> >> > >>>>>> >> >> > >>>>>> On Mon, Oct 7, 2013 at 11:59 AM, Edison Su < >> edison...@citrix.com> >> >> > wrote: >> >> > >>>>>>> I created a design document page at >> >> > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Pluggable+VM+s >> >> > napshot+related+operations, feel free to add items on it. >> >> > >>>>>>> And a new branch "pluggable_vm_snapshot" is created. >> >> > >>>>>>> >> >> > >>>>>>>> -----Original Message----- >> >> > >>>>>>>> From: SuichII, Christopher [mailto:chris.su...@netapp.com] >> >> > >>>>>>>> Sent: Monday, October 07, 2013 10:02 AM >> >> > >>>>>>>> To: <dev@cloudstack.apache.org> >> >> > >>>>>>>> Subject: Re: [DISCUSS] Pluggable VM snapshot related >> operations? >> >> > >>>>>>>> >> >> > >>>>>>>> I'm a fan of option 2 - this gives us the most flexibility >> (as >> >> > >>>>>>>> you stated). The option is given to completely override the >> way >> >> > >>>>>>>> VM snapshots work AND storage providers are given to >> >> > >>>>>>>> opportunity to work within the default VM snapshot workflow. >> >> > >>>>>>>> >> >> > >>>>>>>> I believe this option should satisfy your concern, Mike. The >> >> > >>>>>>>> snapshot and quiesce strategy would be in charge of >> >> > communicating with the hypervisor. >> >> > >>>>>>>> Storage providers should be able to leverage the default >> >> > >>>>>>>> strategies and simply perform the storage operations. >> >> > >>>>>>>> >> >> > >>>>>>>> I don't think it should be much of an issue that new method >> to >> >> > >>>>>>>> the storage driver interface may not apply to everyone. In >> fact, >> >> > that is already the case. >> >> > >>>>>>>> Some methods such as un/maintain(), attachToXXX() and >> >> > >>>>>>>> takeSnapshot() are already not implemented by every driver - >> >> > >>>>>>>> they just return false when asked if they can handle the >> >> operation. >> >> > >>>>>>>> >> >> > >>>>>>>> -- >> >> > >>>>>>>> Chris Suich >> >> > >>>>>>>> chris.su...@netapp.com >> >> > >>>>>>>> NetApp Software Engineer >> >> > >>>>>>>> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red >> Hat >> >> > >>>>>>>> >> >> > >>>>>>>> On Oct 5, 2013, at 12:11 AM, Mike Tutkowski >> >> > >>>>>>>> <mike.tutkow...@solidfire.com> >> >> > >>>>>>>> wrote: >> >> > >>>>>>>> >> >> > >>>>>>>>> Well, my first thought on this is that the storage driver >> >> > >>>>>>>>> should not be telling the hypervisor to do anything. It >> should >> >> > >>>>>>>>> be responsible for creating/deleting volumes, snapshots, >> etc. >> >> on >> >> > its storage system only. >> >> > >>>>>>>>> >> >> > >>>>>>>>> >> >> > >>>>>>>>> On Fri, Oct 4, 2013 at 5:57 PM, Edison Su < >> >> edison...@citrix.com> >> >> > wrote: >> >> > >>>>>>>>> >> >> > >>>>>>>>>> In 4.2, we added VM snapshot for Vmware/Xenserver. The >> >> > >>>>>>>>>> current workflow will be like the following: >> >> > >>>>>>>>>> createVMSnapshot api -> VMSnapshotManagerImpl: >> >> > >>>>>>>>>> creatVMSnapshot -> send CreateVMSnapshotCommand to >> >> > hypervisor to create vm snapshot. >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> If anybody wants to change the workflow, then need to >> either >> >> > >>>>>>>>>> change VMSnapshotManagerImpl directly or subclass >> >> > VMSnapshotManagerImpl. >> >> > >>>>>>>>>> Both are not the ideal choice, as VMSnapshotManagerImpl >> >> > >>>>>>>>>> should be able to handle different ways to take vm >> snapshot, >> >> > instead of hard code. >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> The requirements for the pluggable VM snapshot coming >> from: >> >> > >>>>>>>>>> Storage vendor may have their optimization, such as >> NetApp. >> >> > >>>>>>>>>> VM snapshot can be implemented in a totally different >> way(For >> >> > >>>>>>>>>> example, I could just send a command to guest VM, to tell >> my >> >> > >>>>>>>>>> application to flush disk and hold disk write, then come >> to >> >> > >>>>>>>>>> hypervisor to >> >> > >>>>>>>> take a volume snapshot). >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> If we agree on enable pluggable VM snapshot, then we can >> >> > move >> >> > >>>>>>>>>> on discuss how to implement it. >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> The possible options: >> >> > >>>>>>>>>> 1. coarse grained interface. Add a VMSnapshotStrategy >> >> > >>>>>>>>>> interface, which has the following interfaces: >> >> > >>>>>>>>>> VMSnapshot takeVMSnapshot(VMSnapshot vmSnapshot); >> >> > Boolean >> >> > >>>>>>>>>> revertVMSnapshot(VMSnapshot vmSnapshot); Boolean >> >> > >>>>>>>>>> DeleteVMSnapshot(VMSnapshot vmSnapshot); >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> The work flow will be: createVMSnapshot api -> >> >> > >>>>>>>> VMSnapshotManagerImpl: >> >> > >>>>>>>>>> creatVMSnapshot -> VMSnapshotStrategy: takeVMSnapshot >> >> > >>>>>>>>>> VMSnapshotManagerImpl will manage VM state, do the sanity >> >> > >>>>>>>>>> check, then will handle over to VMSnapshotStrategy. >> >> > >>>>>>>>>> In VMSnapshotStrategy implementation, it may just send a >> >> > >>>>>>>>>> Create/revert/delete VMSnapshotCommand to hypervisor >> >> > host, or >> >> > >>>>>>>>>> do anything special operations. >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> 2. fine-grained interface. Not only add a >> VMSnapshotStrategy >> >> > >>>>>>>>>> interface, but also add certain methods on the storage >> driver. >> >> > >>>>>>>>>> The VMSnapshotStrategy interface will be the same as >> option 1. >> >> > >>>>>>>>>> Will add the following methods on storage driver: >> >> > >>>>>>>>>> /* volumesBelongToVM is the list of volumes of the VM >> that >> >> > >>>>>>>>>> created on this storage, storage vendor can either take >> one >> >> > >>>>>>>>>> snapshot for this volumes in one shot, or take snapshot >> for >> >> > each volume separately >> >> > >>>>>>>>>> The pre-condition: vm is unquiesced. >> >> > >>>>>>>>>> It will return a Boolean to indicate, do need >> unquiesce vm >> >> or >> >> > not. >> >> > >>>>>>>>>> In the default storage driver, it will return false. >> >> > >>>>>>>>>> */ >> >> > >>>>>>>>>> boolean takeVMSnapshot(List<VolumeInfo> >> >> > volumesBelongToVM, >> >> > >>>>>>>>>> VMSnapshot vmSnapshot); Boolean >> >> > >>>>>>>>>> revertVMSnapshot(List<VolumeInfo> volumesBelongToVM, >> >> > >>>>>>>>>> VMSnapshot vmSnapshot); Boolean >> >> > >>>>>>>>>> deleteVMSnapshot(List<VolumeInfo> volumesBelongToVM, >> >> > >>>>>>>>>> VMSnapshot vmSNapshot); >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> The work flow will be: createVMSnapshot api -> >> >> > >>>>>>>> VMSnapshotManagerImpl: >> >> > >>>>>>>>>> creatVMSnapshot -> VMSnapshotStrategy: takeVMSnapshot -> >> >> > >>>>>>>>>> storage driver:takeVMSnapshot In the implementation of >> >> > >>>>>>>>>> VMSnapshotStrategy's takeVMSnapshot, the pseudo code >> >> > looks like: >> >> > >>>>>>>>>> HypervisorHelper.quiesceVM(vm); >> >> > >>>>>>>>>> val volumes = vm.getVolumes(); >> >> > >>>>>>>>>> val maps = new Map[driver, list[VolumeInfo]](); >> >> > >>>>>>>>>> Volumes.foreach(volume => maps.put(volume.getDriver, >> >> > volume :: >> >> > >>>>>>>>>> maps.get(volume.getdriver()))) >> >> > >>>>>>>>>> val needUnquiesce = true; >> >> > >>>>>>>>>> maps.foreach((driver, volumes) => needUnquiesce = >> >> > >>>>>>>>>> needUnquiesce && driver.takeVMSnapshot(volumes)) >> >> > >>>>>>>>>> if (needUnquiesce ) { >> >> > >>>>>>>>>> HypervisorHelper.unquiesce(vm); } >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> By default, the quiesceVM in HypervisorHelper will >> actually >> >> > >>>>>>>>>> take vm snapshot through hypervisor. >> >> > >>>>>>>>>> Does above logic makes senesce? >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> The pros of option 1 is that: it's simple, no need to >> change >> >> > >>>>>>>>>> storage driver interfaces. The cons is that each storage >> >> > >>>>>>>>>> vendor need to implement a strategy, maybe they will do >> the >> >> > same thing. >> >> > >>>>>>>>>> The pros of option 2 is that, storage driver won't need to >> >> > >>>>>>>>>> worry about how to quiesce/unquiesce vm. The cons is >> that, it >> >> > >>>>>>>>>> will add these methods on each storage drivers, so it >> assumes >> >> > >>>>>>>>>> that this work flow will work for everybody. >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> So which option we should take? Or if you have other >> options, >> >> > >>>>>>>>>> please let's know. >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> >> >> > >>>>>>>>>> >> >> > >>>>>>>>> >> >> > >>>>>>>>> >> >> > >>>>>>>>> -- >> >> > >>>>>>>>> *Mike Tutkowski* >> >> > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >> >> > >>>>>>>>> e: mike.tutkow...@solidfire.com >> >> > >>>>>>>>> o: 303.746.7302 >> >> > >>>>>>>>> Advancing the way the world uses the >> >> > >>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play> >> >> > >>>>>>>>> *(tm)* >> >> > >>>>>>> >> >> > >>>>> >> >> > >>> >> >> > > >> >> >> > >> > >> > >> > -- >> > *Mike Tutkowski* >> > *Senior CloudStack Developer, SolidFire Inc.* >> > e: mike.tutkow...@solidfire.com >> > o: 303.746.7302 >> > Advancing the way the world uses the >> > cloud<http://solidfire.com/solution/overview/?video=play> >> > *™* >> > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play> *™*
