Hey guys, I'm thinking through what Vladimir wrote below. My comments are inline.
Thanks! On Thu, Mar 28, 2013 at 5:38 PM, Vladimir Popovski < vladi...@zadarastorage.com> wrote: > If I read what Edison mentioned properly, the idea was to provide as much > freedom for the plugin as possible. > > The generic code of datastore creation and manipulations will be > available, but will be optional. The plugin may decide to use it or perform > something else. > > > > We can consider two main scenarios for plugin operations - shared > pools/datastore vs dedicated. It looks like plugin behavior might be > completely different there. > > > > In case of the shared pool/datastore: > > 1. On Initialize (or attachCluster/attachZone) level, the plugin > will call target to create a LUN & attach it to the datastore > > 2. On createAsync/deleteAsync levels, the plugin will operate with > volumes from this pool only (not LUNs). Optionally, if there is no enough > space in the pool, it may allocate another LUN (or expand the previous one). > > > > In case of the dedicated pool: > [Mike T.] This is where my main interest lies, but I am certainly interested in understanding more about the top scenario, as well. > 1. On Initialize level (or same attach Cluster/Zone), the plugin > will establish a connection between source(-s) and destination, but will > not create any LUNs/datastores > [Mike T.] Let's say "attachCluster" is called on the plug-in. I assume if we are dealing with a VMware cluster that the IP address of vCenter is passed in? At this point, would we run our code to add our iSCSI target to the HBAs of each host in the cluster? If it is a XenServer cluster, I don't think this pre-configuration of HBAs is necessary, so we could do nothing in that case? > 2. On createAsync level, it will create a LUN, create corresponding > datastore & carve a volume from this datastore (size == size of the > datastore) > [Mike T.] This makes good sense to me. This is where we are told what kind of a LUN to create (size, for example) on our SAN. After creating said LUN, we can hook it up to the VMware or XenServer (etc.) cluster (creating a datastore or storage repository (etc.) that is based off of the LUN's iSCSI target). The datastore/storage repository would take up all of the space that the LUN provides. > > > It will really help if all datastore functions will be provided by the CS, > but it will be up to the plugin to decide when to invoke them. > [Mike T.] I agree...there is a lot of logic that would need to be duplicated among storage plug-ins if we don't put it in a library and make it accessible to them. > > > Does it make sense? > [Mike T.] I think this is making more sense. I'm still a bit unclear on how the plug-in behaves in a shared LUN environment (where more than one VM or data disk lives on the LUN), but my main use case is dedicated service to VMs and data disks (one LUN per VM or data disk). Do either of you know of any diagrams that convey when each of the methods we implement in are plug-in are invoked? > > > Regards, > > -Vladimir > > > > > > *From:* Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > *Sent:* Thursday, March 28, 2013 3:07 PM > *To:* Edison Su > *Cc:* Vladimir Popovski; dev@cloudstack.apache.org > *Subject:* Re: Storage Subsystem 2.0 plugin docs > > > > Hi Edison, > > > > I think I'm seeing a bit more where you're coming from. > > > > I guess I was under the impression that when a plug-in was invoked to > create storage that the idea was always for that storage to be for a single > VM or a single data disk. > > > > It sounds like the plug-in architecture, however, is being designed with > more than that in mind? > > > > I'm not sure how this plug-in model would be used, though, if more than > one VM can be assigned to a storage volume. Here's what I'm thinking: > > > > * User executes a compute offering. > > > > * Storage framework gets a volume from storage plug-in. > > > > * VM is deployed to use the entire volume. > > > > By the way, when I say "volume" up there, I mean that the same as LUN. > > > > How could this plug-in framework be used again to deploy another VM to the > same volume? I don't understand that part (not that I plan on doing that, > but am curious). > > > > Thanks! > > > > On Thu, Mar 28, 2013 at 3:50 PM, Edison Su <edison...@citrix.com> wrote: > > > > > > *From:* Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > *Sent:* Thursday, March 28, 2013 2:26 PM > *To:* Edison Su > *Cc:* Vladimir Popovski; dev@cloudstack.apache.org > > > *Subject:* Re: Storage Subsystem 2.0 plugin docs > > > > Hi Edison, > > > > Can you clarify what you mean here? > > > > [Edison] If there are HV-dependent storage code there(I assume it should > have some code can be shared between different storage provider at > hypervisor side), we can generalize them and expose them. > > > > I think Vladimir and I are proposing that the storage framework be > modified to only expect its plug-ins to write code that deals with their > array (not hypervisor-related code). For example, the storage framework > could call into my plug-in for a new volume, I would create it using the > SolidFire API, return an IQN, and the storage framework would run the logic > it needs to in order to, say, create a Datastore for VMware hosts based on > that IQN. > > > > [Edison] If we do that way, then we will enforce per datastore per IQN > model, which seems conflict with what Vladimir talking about. > > CloudStack mgt server will not enforce any kind of policy about how the > volume is created and how the volume will be used by hypervisor. We can > share code through library, instead of through framework. For example, We > can put your create datastore code into hypervisor resource code, add a new > command called, createdatastorecommand, then the provider’s driver code at > mgt serer can send above command to hypervisor resource, inside that > createdatastorecommand, which can call the creating datastore code to > create a datastore. > > > > What do you think about that Edison? Either way, I'm working on writing > code that creates a VMware Datastore and we can decide where to place it > (either in a utility shared by the storage plug-ins or in the CS storage > framework). > > > > On Thu, Mar 28, 2013 at 2:12 PM, Edison Su <edison...@citrix.com> wrote: > > Comments embedded in below. > > > > *From:* Vladimir Popovski [mailto:vladi...@zadarastorage.com] > *Sent:* Wednesday, March 27, 2013 12:22 PM > *To:* Mike Tutkowski; Edison Su > *Cc:* cloudstack-...@incubator.apache.org > > > *Subject:* RE: Storage Subsystem 2.0 plugin docs > > > > Hi All, > > > > I was away for couple of days, so sorry for the delay. > > > > I’m completely with Mike & John (& others) on separating storage plugins > from hypervisor-related functions. If every plugin will need to implement > hypervisor-related code, it will be a big mess. > > > > Our use-case is very similar to Mike’s – we would like to be able to > provision volumes with different QoS characteristics directly to VMs, > rather than adding them into “shared” datastores. It might be achieved in > two ways: > > - either to create separate data stores per each volume (storage > LUN), and from there to create volumes & assign to instances. > > - or to assign devices recognized by iSCSI Initiators directly > to instances (I’m not sure if it will be possible in VMware without > datastores) > > > > It looks like Mike started to work on the 1st approach. In this case, for > every volume there will be a separate datastore. If this is the preferred > direction for all storage plugins, then the hypervisor-specific logic of > datastore creation and creating/assigning volumes from the datastore will > be fairly common for all storage plug-ins. At the same time, the storage > plugin should have the control over the datastore management. It will > depend on the plugin, if dedicated or shared datastores should be created. > > > > For the 2nd option (skipping the datastore layer) there might be plenty > of common code as well. > > > > So, my questions are: > > - do you think it is beneficial to support both options for the > CS (or are we good with potentially huge number of datastores)? > > > > [Edison] CloudStack will not enforce any of these options, it’s all up to > provider’s implementation. Either way is OK to me. Do you think, from > architecture point of view, Is the current storage API enough for both > options? If no, we can come up some new APIs. > > > > - are we all agree that HV-dependent storage code should be > generic and appropriate interfaces to be exposed? > > > > [Edison] If there are HV-dependent storage code there(I assume it should > have some code can be shared between different storage provider at > hypervisor side), we can generalize them and expose them. > > As Mike said, the code dealing with storage pool at the hypervisor side, > can be shared. > > > > Regards, > > -Vladimir > > > > > > > > *From:* Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > *Sent:* Wednesday, March 27, 2013 10:21 AM > *To:* Edison Su > *Cc:* cloudstack-...@incubator.apache.org; Vladimir Popovski > *Subject:* Re: Storage Subsystem 2.0 plugin docs > > > > Sounds good, Edison > > > > Last night I finished up code that uses the VI Java API to create a VMware > Datastore. > > > > I want to test it a bit more before I have you look at it. > > > > Today there is a Citrix CloudPlatform demo I'm participating in to handle > part of the SolidFire section of the demo, so I might not have time to do > my Datastore testing, but I should be done with it tomorrow. > > > > Talk to you later! > > > > On Wed, Mar 27, 2013 at 11:15 AM, Edison Su <edison...@citrix.com> wrote: > > For vmware, current cloudstack doesn’t create a vmware datastore through > vmware’s API, admin needs to create the datastore in Vcenter at first, then > add it back into cloudstack. I am not familiar with how to create a VMware > datastore through Vmware’s API, but regarding to add a new host into a > cluster, the current framework lets storage provider adding a listener > which can listen on adding host event. > > > > *From:* Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > *Sent:* Tuesday, March 26, 2013 6:45 PM > > > *To:* Edison Su > *Cc:* cloudstack-...@incubator.apache.org; Vladimir Popovski > *Subject:* Re: Storage Subsystem 2.0 plugin docs > > > > Great - thanks, Edison! > > > > I can take a look at that code. > > > > I've almost gotten the VMware code written. > > > > It's a little more involved than the XenServer code because you have to > add static IQNs for discovery to each host in a VMware cluster (this is > somehow handled behind the scenes, I suppose, with XenServer) before you > can create a Datastore based on your iSCSI target. > > > > One thing I was wondering, though, is when you add a new host to this > VMware cluster. It will need to "inherit" the list of IQNs to discover. I > image this is the case today. Do you know anything about that? I might > just try it out and see if that works today. > > > > On Tue, Mar 26, 2013 at 5:18 PM, Edison Su <edison...@citrix.com> wrote: > > Thanks! > > FYI, there are some code at both xen and kvm hypervisor resource code to > deal with storage pool creation. > > For example, in CitrixResourceBase-> getNfsSR or getIscsiSR to create a > nfs SR or ISCSI SR. In LibvirtStorageAdaptor, which can create storage pool > in libvirt. > > > > > > *From:* Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > *Sent:* Tuesday, March 26, 2013 1:52 PM > *To:* Edison Su > *Cc:* cloudstack-...@incubator.apache.org; Vladimir Popovski > > > *Subject:* Re: Storage Subsystem 2.0 plugin docs > > > > Hi Edison, > > > > Sounds good. > > > > I already have code to create a XenServer Storage Repository (and > optionally use CHAP credentials) and I'm working right now on creating a > vSphere Datastore. > > > > When I have this working and in a nicer state, I can check in with you to > review it and provide comments. > > > > Once those two hypervisors are handled, I'll move on to KVM and OVM. > > > > Thanks! > > > > On Tue, Mar 26, 2013 at 2:33 PM, Edison Su <edison...@citrix.com> wrote: > > Yes, code is welcome!!! Currently Only the interface at the management > server side is defined. At the hypervisor resource side, we may need some > kind of utility library or another plugin framework, as John proposed few > months ago. > > > > *From:* Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > *Sent:* Monday, March 25, 2013 2:37 PM > *To:* Edison Su; cloudstack-...@incubator.apache.org; Vladimir Popovski > > > *Subject:* Re: Storage Subsystem 2.0 plugin docs > > > > Hey Edison, > > > > So...if you think my understanding is correct (please check out the e-mail > below), then I have a question. > > > > Do we really want to have the storage plug-ins taking on the > responsibility of talking to the hypervisors to hook up the storage that > they just created? > > > > I'm a bit familiar with how OpenStack does this and it seems that it only > has its storage plug-ins create a volume (LUN, whatever) and then the > framework handles the process of dealing with the hypervisor in question to > hook up the storage. > > > > It seems like otherwise we'd need to create a utility for all storage > plug-ins to share otherwise they'd be duplicating efforts in talking to > hypervisors. > > > > What do you think? > > > > On Thu, Mar 21, 2013 at 7:52 PM, Mike Tutkowski < > mike.tutkow...@solidfire.com> wrote: > > Hi Edison, > > > > I believe I understand the requirements for the plug-in better now. > > > > It sounds like the flow will be as such: > > > > * The user executes a Compute or Disk Offering that is tied via a storage > tag to a Primary Storage that is associated with a plug-in. > > > > * The storage framework will ask the plug-in to create a volume. The > plug-in will create a volume and hook the volume up to the appropriate > hypervisor. For VMware, this means the plug-in will create a Datastore. > For XenServer, this means the plug-in will create a Storage Repository. > (So on and so forth for other hypervisors.) > > > > * The VM or data disk is then deployed to the hypervisor. > > > > Does that sound correct, Edison? > > > > Thanks! > > > > On Thu, Mar 21, 2013 at 5:44 PM, Edison Su <edison...@citrix.com> wrote: > > > > > > *From:* Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > *Sent:* Thursday, March 21, 2013 4:18 PM > *To:* Edison Su > *Subject:* Re: Storage Subsystem 2.0 plugin docs > > > > Hi Edison, > > > > I wanted to dive into these comments a bit more: > > > > [Edison] plugin’s driver->createasync will be called when mgt server want > to create a volume on the storage. In the driver’s implementation, it can > directly call storage box’s api, or send a command to hypervisor host, then > call storage box’s api to create an iscsi. > > Then create a datastore(for vmware), SR(for xenserver), or storage > pool(for KVM) on hypervisor host, based on the iscsi iqn. > > If the volume is created from a template(for root disk), need to find a > way to import that template(which is nfs based currently, it will be just a > plain http url the future) into the root disk. > > The part about creating a datastore or a storage repository...is that > something the plug-in will be responsible for doing or will the storage > framework cover that piece? I'm thinking the storage framework will since > all sorts of plug-ins would seem to need that ability (to have their > storage hooked up to a datastore or a storage repository). > > > > [Edison] It’s a specific requirement for per volume per LUN case, and > specific for certain hypervisors(seems KVM doesn’t need to create a storage > pool when using iscsi LUN), so the storage framework will not deal with it > right now. > > > > > > Thanks for your time, Edison! :) > > > > On Thu, Mar 21, 2013 at 4:45 PM, Edison Su <edison...@citrix.com> wrote: > > Feedback/comments are appreciated, need to know your input from storage > vendor point of view. > > > > *From:* Vladimir Popovski [mailto:vladi...@zadarastorage.com] > *Sent:* Thursday, March 21, 2013 11:52 AM > *To:* Edison Su; cloudstack > > > *Cc:* mike.tutkow...@solidfire.com > *Subject:* RE: Storage Subsystem 2.0 plugin docs > > > > Hi Edison, > > > > Thank you for the reply. We will check it out. > > > > Regards, > > -Vladimir > > > > > > *From:* Edison Su [mailto:edison...@citrix.com] > *Sent:* Thursday, March 21, 2013 11:36 AM > *To:* 'Vladimir Popovski'; cloudstack > *Cc:* mike.tutkow...@solidfire.com > *Subject:* RE: Storage Subsystem 2.0 plugin docs > > > > > > > > *From:* Vladimir Popovski > [mailto:vladi...@zadarastorage.com<vladi...@zadarastorage.com>] > > *Sent:* Wednesday, March 20, 2013 9:05 AM > *To:* cloudstack > *Cc:* mike.tutkow...@solidfire.com; Edison Su > *Subject:* Storage Subsystem 2.0 plugin docs > > > > Hi All, > > > > Thank you for a great work on CloudStack! We are interested in integrating > CS with our storage system and started to look at your documentation and > storage-related code. I see that Mike from SolidFire started working on > something similar some time ago and Edison even created an empty plugin for > it (in Nov’12?). > > > > We have couple of questions related to that: > > - Is there any documentation about plugins (except of > https://cwiki.apache.org/CLOUDSTACK/storage-subsystem-20.html) > > [Edison] There are not much docs about the plugins other than the above > link. See below. > > - Are there any exemplary plugins for primary & secondary > datastores? Was the SolidFire plugin ever finished? > > [Edison] yesterday, I checked in some code to separate existing cloudstack > storage code into a standalone maven project, called: > cloud-plugin-storage-volume-default, which can give you an example how a > storage plugin will look like. > > - How to activate a new plugin and use it (at least through > CLIs/APIs) > > [Edison] First, put a bean configuration in client/tomcatconf/ > componentContext.xml.in for your plugin provider class, like: > > <bean id="ClassicalPrimaryDataStoreProvider" > class="org.apache.cloudstack.storage.datastore.provider.CloudStackPrimaryDataStoreProviderImpl"> > > </bean> > > Second, when adding a data store into cloudstack, with an extra parameter > in createstoragepoolcmd: provider=your-provider-name, > liststorageproviderscmd can list all the registered providers in mgt server. > > > > > > - How to integrate it with the UI > > There is no UI part of example code for storage yet, the idea is to use > pluggable UI( > https://cwiki.apache.org/confluence/display/CLOUDSTACK/UI+Plugin+Tutorial), > for each storage provider may need a separate UI to add a storage. For > example, in adding primary storage ui, there will be a drop down list, show > all the registered providers, if user selects one of the drop down list, > then UI will pop up a diagram, based on providers’ pluggable ui, then user > can type whatever information needed for a storage(e.g. nfs server, nfs > path, if its nfs). At the end, UI will call createstoragepoolcmd to > register a storage into cloudstack. > > > > Thanks, > > -Vladimir > > > > > > ------- > > Vladimir Popovski > > VP, Cloud Operations > > Zadara Storage > (949) 677-2095 > > vladi...@zadarastorage.com > > www.zadarastorage.com > > > > > > > > > > -- > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > > > > > > -- > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > > > > > > -- > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > > > > > > -- > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > > > > > > -- > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > > > > > > -- > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > > > > > > -- > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > > > > > > -- > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play> *™*
