Mike, I apologize for being MIA on my due out regarding storage drivers and design. I have gotten pulled onto another $dayjob project that taking all of my time.
+1 to notion that resource tagging of any type is a mess. Ideally, all device drivers should be completely isolated from each other. The tagging model we employ today requires that drivers not only be aware of each other, but all of the possible combinations in which they could be conceivably employed. As a simple example, see the logical mess we encountered dealing with provisioned IOPS for storage devices and hypervisors in 4.1.0. In my opinion, the goal should be for the user to describe the service levels they need from infrastructure, and for orchestration layer to allocate the resources that meet that SLA. We are a long way from such a model, but hopefully, we can start evolving that direction for the next release. In terms of extra specs, we have discussed the notion of generic properties which function in a manner similar to what you describe in OpenStack. Each driver could optionally expose a set of property metadata descriptors (e.g. name, display name, type, required flag, description) which would drive UI rendering and validation to avoid the garbage in garbage out problem. The values of these properties would then be passed into each driver operation. The benefit of this approach is two fold. First, it providers vendors with a simple, but powerful extension mechanism. Second, it helps to prevent the pollution of the orchestration layer's domain model with vendor specific concepts. Thanks, -John On Sep 19, 2013, at 12:32 AM, Mike Tutkowski <mike.tutkow...@solidfire.com> wrote: > We might want to bring John Burwell into this discussion as he has documented > ideas on how storage drivers might be able to advertise to the outside > CloudStack world their capabilities. > > For example, my SolidFire driver could advertise that the volumes it creates > support min, max, and burst IOPS. This info could be leveraged by input > dialogs in the GUI to display certain applicable fields. > > Right now my feeling is that storage tagging is a bit of a mess. > > When you encounter dialogs like Add Disk Offering, the user can enter one or > more storage tags. > > That's great, but it doesn't let the dialog know (easily) what features > primary storage that is tagged with those storage tags support. > > That being the case, the dialog ends up displaying all sorts of options that > users can fill in that may or may not be supported by the storage that > CloudStack ends up placing the newly created volume on. It's really up to the > admin who is creating the Disk Offering (for example) to make sure if he > fills in options that they can be supported by the storage that is tagged. > > OpenStack has a completely different paradigm here. > > It has the concept of a Volume Type. A Volume Type is backed by one and only > one storage driver. You can fill in what they call Extra Specs to pass in > driver-specific info (key/value pairs, ex: min_iops=300). > > If we had something like this, these dialogs would know what data is > applicable to collect. > > > On Wed, Sep 18, 2013 at 6:29 PM, SuichII, Christopher > <chris.su...@netapp.com> wrote: > That's a good point and I'm not sure. Maybe we can have the drivers indicate > whether they support batched/multiple volume snapshotting. If not, then we > can spin up a thread per volume to snapshot. I completely agree that simply > calling them in sequence could end up badly. > > You're right though, the examples I provided are only partial examples as > they really do just edit something in the db. > > Edison, Alex or anyone else knowledgable of the storage system - Do you have > any input. Was my clarification helpful? > > For what it's worth, after some further investigation, it sounds like we > won't actually be performing NetApp volume level snapshots to backup vm > volumes - either when requested or as a batch. We believe we have come up > with a solution that lets us create individual backups while avoiding our 255 > snapshot limit. However, it would still be beneficial to let users request > backups of multiple volumes at once for the other reasons I explained earlier. > > -Chris > -- > Chris Suich > chris.su...@netapp.com > NetApp Software Engineer > Data Center Platforms – Cloud Solutions > Citrix, Cisco & Red Hat > > On Sep 18, 2013, at 6:56 PM, Darren Shepherd <darren.s.sheph...@gmail.com> > wrote: > > > Nah, I guess its not so bad to have a volumeIds param. Just as long as its > > really clear that means nothing about consistency. > > > > I would be a little concerned about how this will be implemented by other > > storage providers though. Currently if you do 5 snapshot API calls that > > will launch 5 threads and they will happen in parallel. If they get > > batched and sent in one thread, how is the framework/driver going to handle > > the snapshots for drivers that don't supporting batching? Sequential would > > be bad as sometimes it takes awhile to snapshot. > > > > The APIs you mentioned that takes lists really just manipulate data in the > > DB, so they can easily batch and transactionally do a bunch at once. > > > > Maybe someone who's more familiar with the storage implementation can > > comment? > > > > Darren > > > > > > On Wed, Sep 18, 2013 at 12:46 PM, SuichII, Christopher < > > chris.su...@netapp.com> wrote: > > > >> That certainly would work, but I don't see how it is a better design. Can > >> you elaborate on how sending multiple volumeIds is hackish? Look at the > >> existing API framework. We have several APIs that accept lists as > >> parameters. Normally, they're used for things like querying or deleting. > >> Take a look at some of these commands: > >> -ArchiveEventsCmd > >> -DeleteEventsCmd > >> -DeleteSnapshotPoliciesCmd > >> > >> This kind of API is simply a shorthand for invoking another API many times. > >> > >> I think it is only an NetApp optimization in the sense that we're the only > >> ones who need it right now. What we're asking for has nothing specific to > >> do to NetApp. We would just like the shorthand ability to do things all at > >> once rather than one at a time. I think other vendors could utilize this > >> just as easily. > >> > >> -Chris > >> -- > >> Chris Suich > >> chris.su...@netapp.com > >> NetApp Software Engineer > >> Data Center Platforms – Cloud Solutions > >> Citrix, Cisco & Red Hat > >> > >> On Sep 18, 2013, at 2:32 PM, Darren Shepherd <darren.s.sheph...@gmail.com> > >> wrote: > >> > >>> Given this explanation. Would the following not work? > >>> > >>> 1) Enhance UI to allow multi select. There is no API change, UI will > >> just > >>> call snapshot a bunch of time > >>> 2) Enhance storage framework or driver to detect that 20 requests just > >> came > >>> in within a window of X seconds and send them to the driver all at once. > >>> > >>> I know you said queuing on the backend is hackish, but having the user > >> send > >>> multiple volumeIds in the API is just as hackish to me. We can only > >>> guarentee to the user that the multiple snapshots taken are as consistent > >>> as if they called snapshot API individually. The user won't really know > >>> exactly what NetApp volume they exist on and really neither will the > >>> storage framework, as export != volume. Only the driver knows if > >> batching > >>> is really possible. So I'm not exactly saying queue, its short batches. > >>> > >>> In short, I'm seeing this as a bit more of a NetApp optimization than a > >>> general thing. I'm all for using storage device level snapshotting but > >> it > >>> seems like its going to be implementation specific. Its interesting, if > >>> you look at digital ocean they have snapshots and backups as two > >> different > >>> concepts. You can see that they ran into this specific issue. Full > >>> storage volume snapshots are really difficult to expose to the user. So > >>> digital ocean does "backups" which are live but on a schedule and seem to > >>> be a full volume backup. And then there are snapshots which are on > >> demand, > >>> but require you to stop your VM (so they can essentially copy the qcow or > >>> lv somewhere). > >>> > >>> Darren > >>> > >>> > >>> > >>> > >>> On Wed, Sep 18, 2013 at 11:12 AM, SuichII, Christopher < > >>> chris.su...@netapp.com> wrote: > >>> > >>>> First, let me apologize for the confusing terms, because some words here > >>>> are overloaded: > >>>> A volume… > >>>> In CloudStack terms is a disk attached to a VM. > >>>> In NetApp terms is an NFS volume, analogous to CloudStack primary > >> storage, > >>>> where all the CloudStack volumes are stored. > >>>> > >>>> A snapshot… > >>>> In CloudStack terms is a backup of a VM. > >>>> In NetApp terms is a copy of all the contents of a NetApp volume, taken > >> at > >>>> a point in time to create an analogous CloudStack snapshot for (up to) > >>>> every CloudStack volume on that primary storage. > >>>> > >>>> There are several reasons that an API for snapshotting multiple volumes > >> is > >>>> more attractive to us than calling a single volume API over and over. A > >> lot > >>>> of it has to do with how we actually create the snapshots. Unlike a > >>>> hypervisor snapshot, when we create a vm snapshot, the entire primary > >>>> storage is backed up (but only the requested volume has an entry added > >> to > >>>> the db). To add on to this, our hardware has a hard limit of 255 storage > >>>> volume level snapshots. So, if there were 255 vms on a single primary > >>>> storage and each one of them performed a backup, no more backups could > >> be > >>>> taken before we start removing the oldest backup (without some trickery > >>>> that we are currently working on). Some might say a solution to this > >> would > >>>> be queueing the requests and waiting till they're all finished, but that > >>>> seems much more error prone and like hackish design compared to simply > >>>> allowing multiple VM volumes to be specified. > >>>> > >>>> This is both a request for optimizing the backend and optimizing the > >>>> experience for users. What happens when a user says they want to backup > >> 30 > >>>> vm volumes at the same time? Is it not a cleaner experience to simply > >>>> select all the volumes they want to back up, then click backup once? > >> This > >>>> way, the storage provider is given all the volumes at once and if they > >> have > >>>> some way of optimizing the request based on their hardware or software, > >>>> they can take advantage of that. It can even be designed in such a way > >> that > >>>> if storage providers don't want to be given all the volumes at once, > >> they > >>>> can be called with each one individually, as to remain backwards > >> compatible. > >>>> > >>>> Now, I'm also not saying that these two solutions can't co-exist. Even > >> if > >>>> we have the ability to backup multiple volumes at once, nothing is > >> stopping > >>>> users from backing them up one by one, so queueing is still something we > >>>> may have to implement. However, I think extending the subsystem API to > >>>> grant storage providers the ability to leverage any optimization they > >> can > >>>> without having to queue is a cleaner solution. If the concern is how > >> users > >>>> interpret what is going on in the backend, I think we can find some way > >> to > >>>> make that clear to them. > >>>> > >>>> -Chris > >>>> -- > >>>> Chris Suich > >>>> chris.su...@netapp.com > >>>> NetApp Software Engineer > >>>> Data Center Platforms – Cloud Solutions > >>>> Citrix, Cisco & Red Hat > >>>> > >>>> On Sep 18, 2013, at 12:26 PM, Alex Huang <alex.hu...@citrix.com> wrote: > >>>> > >>>>> That's my read on the proposal also but, Chris, please clarify. I > >> don't > >>>> think the end user will see the change. It's an optimization for > >>>> interfacing with the storage backend. > >>>>> > >>>>> --Alex > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >>>>>> Sent: Wednesday, September 18, 2013 9:22 AM > >>>>>> To: dev@cloudstack.apache.org > >>>>>> Subject: Re: [PROPOSAL] Storage Subsystem API Interface Additions > >>>>>> > >>>>>> Perhaps he needs to elaborate on the use case and what he means by > >> more > >>>>>> efficient. He may be referring to multiple volumes in the sense of > >>>>>> snapshotting the ROOT disks for 10 different VMs. > >>>>>> > >>>>>> On Wed, Sep 18, 2013 at 10:10 AM, Darren Shepherd > >>>>>> <darren.s.sheph...@gmail.com> wrote: > >>>>>>> Here's my general concern about multiple volume snapshots at once. > >>>>>>> Giving such a feature leads the user to believe that snapshotting > >>>>>>> multiple volumes at once will give them consistency across the > >> volumes > >>>> in > >>>>>> the snapshot. > >>>>>>> This is not true, and difficult to do with many hypervisors, and > >>>>>>> typically requires an agent in the VM. A single snapshot, as exists > >>>>>>> today, is really crash consistent, meaning that there is may exist > >>>>>>> unsync'd data. To do a true multi volume snapshot requires a > >> "quiesce" > >>>>>> functionality in the VM. > >>>>>>> So you do pause I/O queues, fsync, fsync, snapshot, snapshot, unpause > >>>> I/O. > >>>>>>> > >>>>>>> I'm might be fine with the option of allowing multiple volumeId's to > >>>>>>> be specified in the snapshot API, but it needs to be clear that those > >>>>>>> snapshots may be taken sequentially and they are all independently > >>>>>>> crash consistent. But, if you make that clear, then why even have > >> the > >>>> API. > >>>>>>> Essentially it is the same as doing multiple snapshot API commands. > >>>>>>> > >>>>>>> So really I would lean towards having the multiple snapshotting > >>>>>>> supported in the driver or storage subsystem, but not exposed to the > >>>>>>> user. You can easy accomplish it by having a timed window on > >>>>>>> snapshotting. So every 10 seconds you do snapshots, if 5 requests > >>>>>>> have queued in the last 10 seconds, you do them all at once. This > >>>> could be > >>>>>> implemented as a framework thing. > >>>>>>> If your provider implements "SnapshotBatching" interface and that has > >>>>>>> a getBatchWindowTime(), then the framework can detect that it should > >>>>>>> try to queue up some snapshot requests and send them to the driver in > >>>>>>> a batch. Or that could be implemented in the driver itself. I would > >>>>>>> lean toward doing it in the driver and if that goes well, we look at > >>>>>>> pulling the functionality into core ACS. > >>>>>>> > >>>>>>> Darren > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Sep 18, 2013 at 5:22 AM, SuichII, Christopher < > >>>>>>> chris.su...@netapp.com> wrote: > >>>>>>> > >>>>>>>> I would like to raise for discussion the idea of adding a couple > >>>>>>>> methods to the Storage Subsystem API interface. Currently, > >>>>>>>> takeSnapshot() and > >>>>>>>> revertSnapshot() only support single VM volumes. We have a use case > >>>>>>>> for snapshotting multiple VM volumes at the same time. For us, it is > >>>>>>>> more efficient to snapshot them all at once rather than snapshot VM > >>>>>>>> Volumes individually and this seems like a more elegant solution > >> than > >>>>>>>> queueing the requests within our plugin. > >>>>>>>> > >>>>>>>> Base on my investigation, this should require: > >>>>>>>> -Two additional API to be invoked from the UI -Two additional > >> methods > >>>>>>>> added to the Storage Subsystem API interface -Changes in between the > >>>>>>>> API level and invoking the Storage Subsystem API implementations (I > >>>>>>>> know this is broad and vague), mainly around the SnapshotManger/Impl > >>>>>>>> > >>>>>>>> There are a couple topics we would like discussion on: > >>>>>>>> -Would this be beneficial/detrimental/neutral to other storage > >>>> providers? > >>>>>>>> -How should we handle the addition of new methods to the Storage > >>>>>>>> Subsystem API interface? Default them to throw an > >>>>>> UnsupportedOperationException? > >>>>>>>> Default to calling the single VM volume version multiple times? > >>>>>>>> -Does anyone see any issues with allowing multiple snapshots to be > >>>>>>>> taken at the same time or letting storage providers have a list of > >>>>>>>> all the requested volumes to backup? > >>>>>>>> > >>>>>>>> Please let me know if I've missed any major topics for discussion or > >>>>>>>> if anything needs clarification. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Chris > >>>>>>>> -- > >>>>>>>> Chris Suich > >>>>>>>> chris.su...@netapp.com > >>>>>>>> NetApp Software Engineer > >>>>>>>> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat > >>>>>>>> > >>>>>>>> > >>>> > >>>> > >> > >> > > > > > -- > Mike Tutkowski > Senior CloudStack Developer, SolidFire Inc. > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the cloud™
signature.asc
Description: Message signed with OpenPGP using GPGMail
