Hi All, I didn't follow this thread as much so maybe it's already been discussed. If so, let me know.
I don't see why these limits should be used as a way to do allocation. CS already supports storage tiering through tags. Why not reuse that concept? A use case would be. An admin wants to use SolidFire storage for example and enable the rate limiting feature. SolidFire's plugin add tags to the storage pools it added to cloudstack. When the admin creates disk offering, they can either add the tag to the disk offering themselves or call an api command on the solid fire plugin to let solid fire plugin add that tag to the disk offering and set the values for the rate limits. CloudStack's allocator will use tags to match the storage pools. No changes there so big plus. Rate limits are carried to the storage provider and hypervisor by CS. You can add integration in the resources to implement the rate limits. There are several benefits to doing this. - Utilizes something that already works. - Less code to maintain (don't need to modify all the allocators and planners) I think even the values themselves should just be name-value pairs rather than fields in the offering. CS should be responsible for transporting these fields to the interested parties (plugins, and resources) but it itself doesn't understand them. --Alex > -----Original Message----- > From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > Sent: Wednesday, June 12, 2013 3:31 PM > To: John Burwell > Cc: dev@cloudstack.apache.org; Wei Zhou > Subject: Re: [MERGE] disk_io_throttling to MASTER > > We'd also have to recognize that if the Min value is filled in and Storage QoS > is not available that Hypervisor QoS (rate limiting here) cannot satisfy that > constraint. > > > On Wed, Jun 12, 2013 at 4:26 PM, Mike Tutkowski < > mike.tutkow...@solidfire.com> wrote: > > > Yeah, this is interesting. > > > > We'll have to wait and see what Wei's thoughts are on this. > > > > > > On Wed, Jun 12, 2013 at 4:17 PM, John Burwell <jburw...@basho.com> > wrote: > > > >> Mike, > >> > >> Yes. To your point, the appropriate logic would be to check the > >> Volume allocated by the StorageAllocator. If it doesn't support a > >> QoS, then the VM allocator would attempt to fulfill the QoS through the > hypervisor. > >> Another question would be -- what would be in the behavior in the > >> event that no resources are available to fulfill the QoS? We could > >> outright fail or proceed with some kind of warning -- sounds like a > >> good place for a global setting to configure the policy. > >> > >> The other question we haven't answered are usage records. By pushing > >> the QoS concept out to allocation and making it a more general > >> concept, it could make usage record capture more straightforward. > >> > >> Thanks, > >> -John > >> > >> On Jun 12, 2013, at 6:11 PM, Mike Tutkowski > >> <mike.tutkow...@solidfire.com> > >> wrote: > >> > >> I see, John. > >> > >> That is an interesting idea. > >> > >> We'd also have to change the storage allocator(s) to favor > >> QoS-supported storage if those fields are filled in. > >> > >> > >> On Wed, Jun 12, 2013 at 4:09 PM, John Burwell <jburw...@basho.com> > wrote: > >> > >>> Mike, > >>> > >>> My thought is that we present the min/max IOPS fields for read/write > >>> operations for all compute and disk offerings. When the VM is > >>> allocated, we determine the best way to fulfill that QoS. It sounds > >>> like storage level guarantees would always be preferred. If no > >>> storage is available to guarantee the QoS, the allocator falls back > >>> to using hypervisor. This approach only works if summing the > >>> discrete read/write min/max values to get to min/max total IOPS would > be acceptable. > >>> > >>> Thanks, > >>> -John > >>> > >>> > >>> On Jun 12, 2013, at 6:03 PM, Mike Tutkowski < > >>> mike.tutkow...@solidfire.com> wrote: > >>> > >>> I hate to say it, but I believe Storage QoS with a Min and Max will > >>> always be optimal over hypervisor rate limiting. > >>> > >>> The only time you'd want to use hypervisor rate limiting is if > >>> storage QoS is not available. > >>> > >>> We currently have no way to know what storage the root or data disk > >>> will be deployed to, so we have to present the fields in all situation. > >>> > >>> This is because of - in my opinion - the somewhat flawed way > >>> CloudStack handles storage tags. You are free to enter in any text > >>> there and we don't know what it maps to when the Compute or Disk > Offering dialog is up. > >>> > >>> > >>> On Wed, Jun 12, 2013 at 3:58 PM, John Burwell > <jburw...@basho.com>wrote: > >>> > >>>> Mike, > >>>> > >>>> Please see my comments/questions in-line below. > >>>> > >>>> Thanks, > >>>> -John > >>>> > >>>> On Jun 12, 2013, at 5:37 PM, Mike Tutkowski < > >>>> mike.tutkow...@solidfire.com> wrote: > >>>> > >>>> Hi Wei, > >>>> > >>>> So, not entirely sure I follow. > >>>> > >>>> Will what I wrote earlier work? Here is a copy of what I wrote: > >>>> > >>>> Let's just called these radio buttons 1) Hypervisor QoS and 2) > >>>> Storage QoS for the purpose of this e-mail. > >>>> > >>>> By default, neither radio button is selected and no QoS fields are > >>>> visible. > >>>> > >>>> > >>>> I prefer a None option selected by default. I find the no button > >>>> selected in a Radio button group to be confusing. > >>>> > >>>> Also, do we have a mechanism to determine whether or not this radio > >>>> button group should be displayed at all? If there are no devices > >>>> or hypervisors configured to fulfill the QoS, we shouldn't mislead the > user ... > >>>> > >>>> > >>>> If the user selects the Storage QoS radio button, then the Custom > >>>> IOPS checkbox and the Min and Max text fields are made visible. > >>>> > >>>> If the user changes his mind and selects the Hypervisor QoS radio > >>>> button, then the Custom IOPS checkbox and the Min and Max text > >>>> fields disappear and are replaced by the two Hypervisor QoS text fields. > >>>> > >>>> This way, the user can choose neither QoS option or one of them or > >>>> the other, but never both. > >>>> > >>>> On the API side, I was thinking of having logic in place when a > >>>> request comes in to create a Disk Offering to confirm these fields > >>>> are the way we want them. > >>>> > >>>> Once we know the Disk Offering is in order, a user can create a > >>>> data disk from it. Since we checked the validity of the Disk > >>>> Offering when it was created, the VM should never be asked to use > >>>> Hypervisor QoS when Storage QoS in being utilized. > >>>> > >>>> > >>>> Overall, this model sounds very reasonable. Wondering aloud: Would > >>>> be possible to have the UI simply ask for a storage QoS, and let > >>>> the system pick the optional implementation. For example, if a VM > >>>> is attached to KVM and SolidFire is available, we chose to fulfill > >>>> the QoS using the storage device. This approach would also allow > >>>> us to fallback in the event that we can't allocate storage for the > >>>> QoS, but we have hypervisor capacity. Could we always > >>>> deterministically determine the optimal implementation of the QoS at > time of VM deployment? > >>>> > >>>> > >>>> Thanks! > >>>> > >>>> > >>>> On Wed, Jun 12, 2013 at 3:33 PM, Wei ZHOU > <ustcweiz...@gmail.com>wrote: > >>>> > >>>>> Mike, John > >>>>> > >>>>> The Disk I/O Throttling works like: > >>>>> > >>>>> (1) For User VM, > >>>>> (1.1) root disks: service offering -> default value in global > >>>>> configuration > >>>>> (1.2) data disks: disk offering -> default value in global > >>>>> configuration > >>>>> > >>>>> (2) For System VM > >>>>> root disks: service offering > >>>>> > >>>>> -Wei > >>>>> > >>>>> > >>>>> 2013/6/12 John Burwell <jburw...@basho.com> > >>>>> > >>>>> > Mike, > >>>>> > > >>>>> > That is one possibility. The other possibility is that > >>>>> > hypervisor is going to throttle I/O on all disks attached. > >>>>> > Therefore, we need > >>>>> answers to > >>>>> > the following questions: > >>>>> > > >>>>> > 1. Is I/O throttling applied to the root disk or all > >>>>> > disks attached to the VM? > >>>>> > 2. If I/O throttling is applied to all disks, how is the > >>>>> > throttling distributed amongst the disks if only one read/write > >>>>> value is > >>>>> > defined? > >>>>> > > >>>>> > Thanks, > >>>>> > -John > >>>>> > > >>>>> > On Jun 12, 2013, at 5:13 PM, Mike Tutkowski < > >>>>> mike.tutkow...@solidfire.com> > >>>>> > wrote: > >>>>> > > >>>>> > > Hey John, > >>>>> > > > >>>>> > > Perhaps I don't fully understand how Wei's feature works. > >>>>> > > > >>>>> > > I guess I thought if you choose Hypervisor QoS, you do so on > >>>>> Compute > >>>>> > > Offerings (root disks) and/or Disk Offerings (data disks). > >>>>> > > > >>>>> > > From my thinking, you're root disk could be under Hypervisor > >>>>> > > QoS, > >>>>> but > >>>>> > your > >>>>> > > data disk could be under Storage QoS. > >>>>> > > > >>>>> > > Is that incorrect? > >>>>> > > > >>>>> > > Thanks > >>>>> > > > >>>>> > > > >>>>> > > On Wed, Jun 12, 2013 at 3:10 PM, John Burwell > >>>>> > > <jburw...@basho.com> > >>>>> > wrote: > >>>>> > > > >>>>> > >> Mike, > >>>>> > >> > >>>>> > >> As I understand these two patches, the throttled I?O settings > >>>>> > >> are > >>>>> > applied > >>>>> > >> from the hypervisor side, and possibly defined in a compute > >>>>> offering > >>>>> > where > >>>>> > >> provisioned IOPS are defined on the storage side through disk > >>>>> > offerings. I > >>>>> > >> don't see how the management server could enforce this mutual > >>>>> exclusion > >>>>> > >> between provisioned IOPS and throttled I/O until a compute > >>>>> > >> and > >>>>> disk > >>>>> > >> offering were selected. These selections come together when > >>>>> > >> we > >>>>> deploy a > >>>>> > >> VM. Based on these assumptions, I would expect to see > >>>>> enforcement of > >>>>> > this > >>>>> > >> rule in UI at the time of VM creation/definition and on the > >>>>> > >> server > >>>>> > side, as > >>>>> > >> part of the VM creation. It feels like any attempt to > >>>>> > >> enforce > >>>>> this rule > >>>>> > >> when defining offering would be premature, and unnecessarily > >>>>> > >> limit flexibility. Are my assumptions and understanding correct? > >>>>> > >> > >>>>> > >> Thanks, > >>>>> > >> -John > >>>>> > >> > >>>>> > >> On Jun 12, 2013, at 5:04 PM, Mike Tutkowski < > >>>>> > mike.tutkow...@solidfire.com> > >>>>> > >> wrote: > >>>>> > >> > >>>>> > >>> Hi John, > >>>>> > >>> > >>>>> > >>> So, maybe I'm wrong about this, but what I was thinking is > >>>>> > >>> that > >>>>> we'd > >>>>> > >> build > >>>>> > >>> two radio buttons into the Add Disk Offering dialog (we can > >>>>> ignore > >>>>> > >> Compute > >>>>> > >>> Offerings for 4.2 since my feature doesn't yet support them). > >>>>> > >>> > >>>>> > >>> Let's just called these radio buttons 1) Hypervisor QoS and > >>>>> > >>> 2) > >>>>> Storage > >>>>> > >> QoS > >>>>> > >>> for the purpose of this e-mail. > >>>>> > >>> > >>>>> > >>> By default, neither radio button is selected and no QoS > >>>>> > >>> fields > >>>>> are > >>>>> > >> visible. > >>>>> > >>> > >>>>> > >>> If the user selects the Storage QoS radio button, then the > >>>>> Custom IOPS > >>>>> > >>> checkbox and the Min and Max text fields are made visible. > >>>>> > >>> > >>>>> > >>> If the user changes his mind and selects the Hypervisor QoS > >>>>> > >>> radio > >>>>> > button, > >>>>> > >>> then the Custom IOPS checkbox and the Min and Max text > >>>>> > >>> fields > >>>>> disappear > >>>>> > >> and > >>>>> > >>> are replaced by the two Hypervisor QoS text fields. > >>>>> > >>> > >>>>> > >>> This way, the user can choose neither QoS option or one of > >>>>> > >>> them > >>>>> or the > >>>>> > >>> other, but never both. > >>>>> > >>> > >>>>> > >>> On the API side, I was thinking of having logic in place > >>>>> > >>> when a > >>>>> request > >>>>> > >>> comes in to create a Disk Offering to confirm these fields > >>>>> > >>> are > >>>>> the way > >>>>> > we > >>>>> > >>> want them. > >>>>> > >>> > >>>>> > >>> Once we know the Disk Offering is in order, a user can > >>>>> > >>> create a > >>>>> data > >>>>> > disk > >>>>> > >>> from it. Since we checked the validity of the Disk Offering > >>>>> > >>> when > >>>>> it was > >>>>> > >>> created, the VM should never be asked to use Hypervisor QoS > >>>>> > >>> when > >>>>> > Storage > >>>>> > >>> QoS in being utilized. > >>>>> > >>> > >>>>> > >>> Does that make sense or did I miss something? > >>>>> > >>> > >>>>> > >>> Thanks > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> On Wed, Jun 12, 2013 at 2:54 PM, John Burwell < > >>>>> jburw...@basho.com> > >>>>> > >> wrote: > >>>>> > >>> > >>>>> > >>>> Mike, > >>>>> > >>>> > >>>>> > >>>> Looking through the code, I am trying to understand how > >>>>> > >>>> CreateDiskOfferingCmd would have the context to identify > >>>>> > >>>> the > >>>>> conflict. > >>>>> > >>>> Naively, it seems to me that this rule would need to be > >>>>> enforced when > >>>>> > a > >>>>> > >>>> virtual machine is being deployed. Looking through the > >>>>> > >>>> code, > >>>>> it seems > >>>>> > >> like > >>>>> > >>>> we should add a private validateStorageQoS method to > >>>>> > >>>> com.cloud.vm.UserVmManagerImpl to check this condition > and > >>>>> throws a > >>>>> > >>>> ResourceAllocationException when the QoS definitions are > >>>>> inconsistent. > >>>>> > >> We > >>>>> > >>>> would then add calls to it from each of the VM creation > >>>>> > >>>> methods > >>>>> in the > >>>>> > >>>> service. Do this type of approach sound reasonable? > >>>>> > >>>> > >>>>> > >>>> Thanks, > >>>>> > >>>> -John > >>>>> > >>>> > >>>>> > >>>> On Jun 12, 2013, at 4:30 PM, Mike Tutkowski < > >>>>> > >> mike.tutkow...@solidfire.com> > >>>>> > >>>> wrote: > >>>>> > >>>> > >>>>> > >>>>> Hi John, > >>>>> > >>>>> > >>>>> > >>>>> So, here's what I was planning to do. Of course feel free > >>>>> > >>>>> to > >>>>> correct > >>>>> > me > >>>>> > >>>> on > >>>>> > >>>>> this approach. > >>>>> > >>>>> > >>>>> > >>>>> I think it's OK if Wei merges his code into master and > >>>>> > >>>>> then I > >>>>> can > >>>>> > draw > >>>>> > >>>> from > >>>>> > >>>>> the main repo and merge master into mine locally. > >>>>> > >>>>> > >>>>> > >>>>> 1) Once I get Wei's code and merge, I plan to add a little > >>>>> > >>>>> GUI > >>>>> code > >>>>> > to > >>>>> > >>>> make > >>>>> > >>>>> it user friendly (toggle between these features on the Add > >>>>> > >>>>> Disk > >>>>> > >> Offering > >>>>> > >>>>> window). > >>>>> > >>>>> > >>>>> > >>>>> 2) I plan to write validation logic for the > >>>>> create-disk-offering API > >>>>> > >>>>> command which throws an exception if the rules are not > >>>>> followed (this > >>>>> > >>>>> should never be triggered from the GUI since the GUI will > >>>>> > >>>>> have > >>>>> > controls > >>>>> > >>>> in > >>>>> > >>>>> place to toggle between the one feature and the other). > >>>>> > >>>>> > >>>>> > >>>>> I'm not sure about documentation. I haven't had much > >>>>> experience with > >>>>> > it > >>>>> > >>>> on > >>>>> > >>>>> CloudStack projects yet. > >>>>> > >>>>> > >>>>> > >>>>> Thanks! > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Wed, Jun 12, 2013 at 2:21 PM, John Burwell < > >>>>> jburw...@basho.com> > >>>>> > >>>> wrote: > >>>>> > >>>>> > >>>>> > >>>>>> Mike, > >>>>> > >>>>>> > >>>>> > >>>>>> Yes, these server-side rails need to be defined and > >>>>> implemented > >>>>> > before > >>>>> > >>>>>> either patch can be merged. From my perspective, I would > >>>>> like to > >>>>> > see > >>>>> > >>>> the > >>>>> > >>>>>> rule implemented in the hypervisor as part of the > >>>>> > >>>>>> validation > >>>>> of the > >>>>> > >>>> virtual > >>>>> > >>>>>> machine definition. We also need to make sure that this > >>>>> mutual > >>>>> > >>>> exclusion > >>>>> > >>>>>> is documented. Do we usually include this type of > >>>>> documentation > >>>>> > with > >>>>> > >>>>>> patches of this nature? > >>>>> > >>>>>> > >>>>> > >>>>>> Thanks, > >>>>> > >>>>>> -John > >>>>> > >>>>>> > >>>>> > >>>>>> On Jun 12, 2013, at 2:18 PM, Mike Tutkowski < > >>>>> > >>>> mike.tutkow...@solidfire.com> > >>>>> > >>>>>> wrote: > >>>>> > >>>>>> > >>>>> > >>>>>>> Currently they are not yet implemented. > >>>>> > >>>>>>> > >>>>> > >>>>>>> We have to make sure they are implemented in the GUI > >>>>> > >>>>>>> from a > >>>>> > usability > >>>>> > >>>>>>> standpoint, but the API must check for consistency and > >>>>> > >>>>>>> throw > >>>>> an > >>>>> > >>>> exception > >>>>> > >>>>>>> if necessary. > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> On Wed, Jun 12, 2013 at 11:03 AM, John Burwell < > >>>>> jburw...@basho.com > >>>>> > > > >>>>> > >>>>>> wrote: > >>>>> > >>>>>>> > >>>>> > >>>>>>>> Mike, > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> Are the checks only implemented in the UI? > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> Thanks, > >>>>> > >>>>>>>> -John > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> On Jun 12, 2013, at 1:02 PM, Mike Tutkowski > >>>>> > >>>>>>>> <mike.tutkow...@solidfire.com> wrote: > >>>>> > >>>>>>>> > >>>>> > >>>>>>>>> Hi John, > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> Wei and I have discussed making the two features > >>>>> > >>>>>>>>> mutually > >>>>> > >> exclusive. > >>>>> > >>>> We > >>>>> > >>>>>>>>> agree with you that only one should be active at a time. > >>>>> We plan > >>>>> > to > >>>>> > >>>>>>>>> implement in the GUI a mechanism (maybe radio > buttons) > >>>>> > >>>>>>>>> to > >>>>> turn > >>>>> > his > >>>>> > >>>>>>>> feature > >>>>> > >>>>>>>>> on and mine off and vice versa. > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> I was thinking if I wait until he checks in his code, > >>>>> > >>>>>>>>> then > >>>>> I > >>>>> > update > >>>>> > >>>> and > >>>>> > >>>>>>>>> merge that I will be the person resolving merge > >>>>> > >>>>>>>>> conflicts > >>>>> in the > >>>>> > >>>>>>>> JavaScript > >>>>> > >>>>>>>>> code (there shouldn't be a problem in the Java code) > >>>>> > >>>>>>>>> as > >>>>> opposed > >>>>> > to > >>>>> > >>>>>>>> putting > >>>>> > >>>>>>>>> that work on someone else. > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> Let me know what you think. > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> Oh, was going to ask you what "FS" stands for here. > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> Thanks! > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> On Wed, Jun 12, 2013 at 10:56 AM, John Burwell < > >>>>> > jburw...@basho.com > >>>>> > >>> > >>>>> > >>>>>>>> wrote: > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>>> Mike, > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> How have Wei and you resolved the issue of > >>>>> > >>>>>>>>>> conflicting QoS > >>>>> > >>>> mechanisms > >>>>> > >>>>>>>>>> between the Hypervisor and Storage layers? Have the > >>>>> affected > >>>>> > FSs > >>>>> > >>>> been > >>>>> > >>>>>>>>>> updated with that decision? > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> In terms of merge timing, can you describe the > >>>>> dependencies > >>>>> > >> between > >>>>> > >>>>>> the > >>>>> > >>>>>>>>>> patches? > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> Thanks, > >>>>> > >>>>>>>>>> -John > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> On Jun 12, 2013, at 12:43 PM, Mike Tutkowski > >>>>> > >>>>>>>>>> <mike.tutkow...@solidfire.com> wrote: > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>>> No problem, John. > >>>>> > >>>>>>>>>>> > >>>>> > >>>>>>>>>>> I still want to re-review it by myself before coming > >>>>> > >>>>>>>>>>> up > >>>>> with a > >>>>> > >> new > >>>>> > >>>>>>>> patch > >>>>> > >>>>>>>>>>> file. > >>>>> > >>>>>>>>>>> > >>>>> > >>>>>>>>>>> Also, maybe I should first wait for Wei's changes to > >>>>> > >>>>>>>>>>> be > >>>>> checked > >>>>> > >> in > >>>>> > >>>>>> and > >>>>> > >>>>>>>>>>> merge those into mine before generating a new > patch file? > >>>>> > >>>>>>>>>>> > >>>>> > >>>>>>>>>>> > >>>>> > >>>>>>>>>>> On Wed, Jun 12, 2013 at 10:40 AM, John Burwell < > >>>>> > >> jburw...@basho.com > >>>>> > >>>>> > >>>>> > >>>>>>>>>> wrote: > >>>>> > >>>>>>>>>>> > >>>>> > >>>>>>>>>>>> Mike, > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> I just realized that I forgot to publish my review. > >>>>> > >>>>>>>>>>>> I > >>>>> am > >>>>> > >> offline > >>>>> > >>>>>> ATM, > >>>>> > >>>>>>>>>>>> but I will publish it in the next couple of hours. > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> Do you plan to update your the patch in Review > Board? > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> Sorry for the oversight, -John > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> On Jun 12, 2013, at 2:26 AM, Mike Tutkowski > >>>>> > >>>>>>>>>>>> <mike.tutkow...@solidfire.com> wrote: > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>> Hi Edison, John, and Wei (and whoever else is > reading > >>>>> this :) > >>>>> > >> ), > >>>>> > >>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>> Just an FYI that I believe I have implemented all > the > >>>>> areas > >>>>> > we > >>>>> > >>>>>> wanted > >>>>> > >>>>>>>>>>>>> addressed. > >>>>> > >>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>> I plan to review the code again tomorrow morning > or > >>>>> > afternoon, > >>>>> > >>>> then > >>>>> > >>>>>>>>>> send > >>>>> > >>>>>>>>>>>> in > >>>>> > >>>>>>>>>>>>> another patch. > >>>>> > >>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>> Thanks for all the work on this everyone! > >>>>> > >>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>> On Tue, Jun 11, 2013 at 12:29 PM, Mike Tutkowski > < > >>>>> > >>>>>>>>>>>>> mike.tutkow...@solidfire.com> wrote: > >>>>> > >>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>> Sure, that sounds good. > >>>>> > >>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>> On Tue, Jun 11, 2013 at 12:11 PM, Wei ZHOU < > >>>>> > >>>> ustcweiz...@gmail.com > >>>>> > >>>>>>> > >>>>> > >>>>>>>>>>>> wrote: > >>>>> > >>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> Hi Mike, > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> It looks the two feature do not have many > conflicts > >>>>> in Java > >>>>> > >>>> code, > >>>>> > >>>>>>>>>>>> except > >>>>> > >>>>>>>>>>>>>>> the cloudstack UI. > >>>>> > >>>>>>>>>>>>>>> If you do not mind, I will merge > disk_io_throttling > >>>>> branch > >>>>> > >> into > >>>>> > >>>>>>>>>> master > >>>>> > >>>>>>>>>>>>>>> this > >>>>> > >>>>>>>>>>>>>>> week, so that you can develop based on it. > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> -Wei > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> 2013/6/11 Mike Tutkowski < > >>>>> mike.tutkow...@solidfire.com> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>> Hey John, > >>>>> > >>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>> The SolidFire patch does not depend on the > >>>>> object_store > >>>>> > >>>> branch, > >>>>> > >>>>>>>> but > >>>>> > >>>>>>>>>> - > >>>>> > >>>>>>>>>>>> as > >>>>> > >>>>>>>>>>>>>>>> Edison mentioned - it might be easier if we > merge > >>>>> the > >>>>> > >>>> SolidFire > >>>>> > >>>>>>>>>> branch > >>>>> > >>>>>>>>>>>>>>> into > >>>>> > >>>>>>>>>>>>>>>> the object_store branch before object_store > goes > >>>>> into > >>>>> > >> master. > >>>>> > >>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>> I'm not sure how the disk_io_throttling fits > into > >>>>> this > >>>>> > merge > >>>>> > >>>>>>>>>> strategy. > >>>>> > >>>>>>>>>>>>>>>> Perhaps Wei can chime in on that. > >>>>> > >>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>> On Tue, Jun 11, 2013 at 11:07 AM, John Burwell > < > >>>>> > >>>>>>>> jburw...@basho.com> > >>>>> > >>>>>>>>>>>>>>> wrote: > >>>>> > >>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>> Mike, > >>>>> > >>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>> We have a delicate merge dance to perform. > The > >>>>> > >>>>>>>> disk_io_throttling, > >>>>> > >>>>>>>>>>>>>>>>> solidfire, and object_store appear to have a > >>>>> number of > >>>>> > >>>>>>>> overlapping > >>>>> > >>>>>>>>>>>>>>>>> elements. I understand the dependencies > between > >>>>> the > >>>>> > >> patches > >>>>> > >>>> to > >>>>> > >>>>>>>> be > >>>>> > >>>>>>>>>> as > >>>>> > >>>>>>>>>>>>>>>>> follows: > >>>>> > >>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>> object_store <- solidfire -> disk_io_throttling > >>>>> > >>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>> Am I correct that the device management > aspects of > >>>>> > >> SolidFire > >>>>> > >>>>>> are > >>>>> > >>>>>>>>>>>>>>> additive > >>>>> > >>>>>>>>>>>>>>>>> to the object_store branch or there are > circular > >>>>> > dependency > >>>>> > >>>>>>>> between > >>>>> > >>>>>>>>>>>>>>> the > >>>>> > >>>>>>>>>>>>>>>>> branches? Once we understand the > dependency > >>>>> graph, we > >>>>> > can > >>>>> > >>>>>>>>>> determine > >>>>> > >>>>>>>>>>>>>>> the > >>>>> > >>>>>>>>>>>>>>>>> best approach to land the changes in master. > >>>>> > >>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>> Thanks, > >>>>> > >>>>>>>>>>>>>>>>> -John > >>>>> > >>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>> On Jun 10, 2013, at 11:10 PM, Mike Tutkowski > < > >>>>> > >>>>>>>>>>>>>>>> mike.tutkow...@solidfire.com> > >>>>> > >>>>>>>>>>>>>>>>> wrote: > >>>>> > >>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>> Also, if we are good with Edison merging my > code > >>>>> into > >>>>> > his > >>>>> > >>>>>> branch > >>>>> > >>>>>>>>>>>>>>> before > >>>>> > >>>>>>>>>>>>>>>>>> going into master, I am good with that. > >>>>> > >>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>> We can remove the > StoragePoolType.Dynamic code > >>>>> after his > >>>>> > >>>> merge > >>>>> > >>>>>>>> and > >>>>> > >>>>>>>>>>>>>>> we > >>>>> > >>>>>>>>>>>>>>>> can > >>>>> > >>>>>>>>>>>>>>>>>> deal with Burst IOPS then, as well. > >>>>> > >>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>> On Mon, Jun 10, 2013 at 9:08 PM, Mike > Tutkowski < > >>>>> > >>>>>>>>>>>>>>>>>> mike.tutkow...@solidfire.com> wrote: > >>>>> > >>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> Let me make sure I follow where we're > going here: > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> 1) There should be NO references to > hypervisor > >>>>> code in > >>>>> > >> the > >>>>> > >>>>>>>>>> storage > >>>>> > >>>>>>>>>>>>>>>>>>> plug-ins code (this includes the default > storage > >>>>> > plug-in, > >>>>> > >>>>>> which > >>>>> > >>>>>>>>>>>>>>>>> currently > >>>>> > >>>>>>>>>>>>>>>>>>> sends several commands to the hypervisor > in use > >>>>> > (although > >>>>> > >>>> it > >>>>> > >>>>>>>> does > >>>>> > >>>>>>>>>>>>>>> not > >>>>> > >>>>>>>>>>>>>>>>> know > >>>>> > >>>>>>>>>>>>>>>>>>> which hypervisor (XenServer, ESX(i), etc.) > is > >>>>> actually > >>>>> > in > >>>>> > >>>>>> use)) > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> 2) managed=true or managed=false can > be placed > >>>>> in the > >>>>> > url > >>>>> > >>>>>> field > >>>>> > >>>>>>>>>> (if > >>>>> > >>>>>>>>>>>>>>>> not > >>>>> > >>>>>>>>>>>>>>>>>>> present, we default to false). This info is > >>>>> stored in > >>>>> > the > >>>>> > >>>>>>>>>>>>>>>>>>> storage_pool_details table. > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> 3) When the "attach" command is sent to > the > >>>>> hypervisor > >>>>> > in > >>>>> > >>>>>>>>>>>>>>> question, we > >>>>> > >>>>>>>>>>>>>>>>>>> pass the managed property along (this > takes the > >>>>> place > >>>>> > of > >>>>> > >>>> the > >>>>> > >>>>>>>>>>>>>>>>>>> StoragePoolType.Dynamic check). > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> 4) execute(AttachVolumeCommand) in > the hypervisor > >>>>> > checks > >>>>> > >>>> for > >>>>> > >>>>>>>> the > >>>>> > >>>>>>>>>>>>>>>> managed > >>>>> > >>>>>>>>>>>>>>>>>>> property. If true for an attach, the > necessary > >>>>> > hypervisor > >>>>> > >>>>>> data > >>>>> > >>>>>>>>>>>>>>>>> structure is > >>>>> > >>>>>>>>>>>>>>>>>>> created and the rest of the attach > command > >>>>> executes to > >>>>> > >>>> attach > >>>>> > >>>>>>>> the > >>>>> > >>>>>>>>>>>>>>>>> volume. > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> 5) When execute(AttachVolumeCommand) > is invoked > >>>>> to > >>>>> > >> detach a > >>>>> > >>>>>>>>>> volume, > >>>>> > >>>>>>>>>>>>>>>> the > >>>>> > >>>>>>>>>>>>>>>>>>> same check is made. If managed, the > hypervisor > >>>>> data > >>>>> > >>>> structure > >>>>> > >>>>>>>> is > >>>>> > >>>>>>>>>>>>>>>>> removed. > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> 6) I do not see an clear way to support > Burst > >>>>> IOPS in > >>>>> > 4.2 > >>>>> > >>>>>>>> unless > >>>>> > >>>>>>>>>>>>>>> it is > >>>>> > >>>>>>>>>>>>>>>>>>> stored in the volumes and disk_offerings > table. > >>>>> If we > >>>>> > >> have > >>>>> > >>>>>> some > >>>>> > >>>>>>>>>>>>>>> idea, > >>>>> > >>>>>>>>>>>>>>>>>>> that'd be cool. > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> Thanks! > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> On Mon, Jun 10, 2013 at 8:58 PM, Mike > Tutkowski < > >>>>> > >>>>>>>>>>>>>>>>>>> mike.tutkow...@solidfire.com> wrote: > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>>> "+1 -- Burst IOPS can be implemented > while > >>>>> avoiding > >>>>> > >>>>>>>>>> implementation > >>>>> > >>>>>>>>>>>>>>>>>>>> attributes. I always wondered about the > details > >>>>> > field. > >>>>> > >> I > >>>>> > >>>>>>>> think > >>>>> > >>>>>>>>>>>>>>> we > >>>>> > >>>>>>>>>>>>>>>>> should > >>>>> > >>>>>>>>>>>>>>>>>>>> beef up the description in the > documentation > >>>>> regarding > >>>>> > >> the > >>>>> > >>>>>>>>>>>>>>> expected > >>>>> > >>>>>>>>>>>>>>>>> format > >>>>> > >>>>>>>>>>>>>>>>>>>> of the field. In 4.1, I noticed that the > >>>>> details are > >>>>> > >> not > >>>>> > >>>>>>>>>>>>>>> returned on > >>>>> > >>>>>>>>>>>>>>>>> the > >>>>> > >>>>>>>>>>>>>>>>>>>> createStoratePool updateStoragePool, or > >>>>> > listStoragePool > >>>>> > >>>>>>>>>> response. > >>>>> > >>>>>>>>>>>>>>>> Why > >>>>> > >>>>>>>>>>>>>>>>>>>> don't we return it? It seems like it would > be > >>>>> useful > >>>>> > >> for > >>>>> > >>>>>>>>>> clients > >>>>> > >>>>>>>>>>>>>>> to > >>>>> > >>>>>>>>>>>>>>>> be > >>>>> > >>>>>>>>>>>>>>>>>>>> able to inspect the contents of the details > >>>>> field." > >>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>>> Not sure how this would work storing > Burst IOPS > >>>>> here. > >>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>>> Burst IOPS need to be variable on a Disk > >>>>> > >> Offering-by-Disk > >>>>> > >>>>>>>>>> Offering > >>>>> > >>>>>>>>>>>>>>>>>>>> basis. For each Disk Offering created, you > have > >>>>> to be > >>>>> > >> able > >>>>> > >>>>>> to > >>>>> > >>>>>>>>>>>>>>>> associate > >>>>> > >>>>>>>>>>>>>>>>>>>> unique Burst IOPS. There is a > >>>>> disk_offering_details > >>>>> > >> table. > >>>>> > >>>>>>>> Maybe > >>>>> > >>>>>>>>>>>>>>> it > >>>>> > >>>>>>>>>>>>>>>>> could > >>>>> > >>>>>>>>>>>>>>>>>>>> go there? > >>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>>> I'm also not sure how you would accept > the > >>>>> Burst IOPS > >>>>> > in > >>>>> > >>>> the > >>>>> > >>>>>>>> GUI > >>>>> > >>>>>>>>>>>>>>> if > >>>>> > >>>>>>>>>>>>>>>>> it's > >>>>> > >>>>>>>>>>>>>>>>>>>> not stored like the Min and Max fields > are in > >>>>> the DB. > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> -- > >>>>> > >>>>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>> > >>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire > Inc.* > >>>>> > >>>>>>>>>>>>>>>>>>> e: mike.tutkow...@solidfire.com > >>>>> > >>>>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>> > >>>>>>>>>>>>>>>>>>> Advancing the way the world uses the > cloud< > >>>>> > >>>>>>>>>>>>>>>>> > http://solidfire.com/solution/overview/?video=play > >>>>> > > >>>>> > >>>>>>>>>>>>>>>>>>> *(tm)* > >>>>> > >>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>>>> -- > >>>>> > >>>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>> > >>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire > Inc.* > >>>>> > >>>>>>>>>>>>>>>>>> e: mike.tutkow...@solidfire.com > >>>>> > >>>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>> > >>>>>>>>>>>>>>>>>> Advancing the way the world uses the > >>>>> > >>>>>>>>>>>>>>>>>> cloud< > >>>>> > http://solidfire.com/solution/overview/?video=play> > >>>>> > >>>>>>>>>>>>>>>>>> *(tm)* > >>>>> > >>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>>> -- > >>>>> > >>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>> > >>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>> > >>>>>>>>>>>>>>>> e: mike.tutkow...@solidfire.com > >>>>> > >>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>> > >>>>>>>>>>>>>>>> Advancing the way the world uses the > >>>>> > >>>>>>>>>>>>>>>> cloud< > >>>>> http://solidfire.com/solution/overview/?video=play> > >>>>> > >>>>>>>>>>>>>>>> *(tm)* > >>>>> > >>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>> -- > >>>>> > >>>>>>>>>>>>>> *Mike Tutkowski* > >>>>> > >>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>> > >>>>>>>>>>>>>> e: mike.tutkow...@solidfire.com > >>>>> > >>>>>>>>>>>>>> o: 303.746.7302 > >>>>> > >>>>>>>>>>>>>> Advancing the way the world uses the cloud< > >>>>> > >>>>>>>>>>>> > http://solidfire.com/solution/overview/?video=play> > >>>>> > >>>>>>>>>>>>>> *(tm)* > >>>>> > >>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>> -- > >>>>> > >>>>>>>>>>>>> *Mike Tutkowski* > >>>>> > >>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>> > >>>>>>>>>>>>> e: mike.tutkow...@solidfire.com > >>>>> > >>>>>>>>>>>>> o: 303.746.7302 > >>>>> > >>>>>>>>>>>>> Advancing the way the world uses the > >>>>> > >>>>>>>>>>>>> cloud< > >>>>> http://solidfire.com/solution/overview/?video=play> > >>>>> > >>>>>>>>>>>>> *(tm)* > >>>>> > >>>>>>>>>>> > >>>>> > >>>>>>>>>>> > >>>>> > >>>>>>>>>>> > >>>>> > >>>>>>>>>>> -- > >>>>> > >>>>>>>>>>> *Mike Tutkowski* > >>>>> > >>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>> > >>>>>>>>>>> e: mike.tutkow...@solidfire.com > >>>>> > >>>>>>>>>>> o: 303.746.7302 > >>>>> > >>>>>>>>>>> Advancing the way the world uses the > >>>>> > >>>>>>>>>>> > cloud<http://solidfire.com/solution/overview/?video=play > >>>>> > > >>>>> > >>>>>>>>>>> *(tm)* > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> -- > >>>>> > >>>>>>>>> *Mike Tutkowski* > >>>>> > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>> > >>>>>>>>> e: mike.tutkow...@solidfire.com > >>>>> > >>>>>>>>> o: 303.746.7302 > >>>>> > >>>>>>>>> Advancing the way the world uses the > >>>>> > >>>>>>>>> > cloud<http://solidfire.com/solution/overview/?video=play> > >>>>> > >>>>>>>>> *(tm)* > >>>>> > >>>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> -- > >>>>> > >>>>>>> *Mike Tutkowski* > >>>>> > >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>> > >>>>>>> e: mike.tutkow...@solidfire.com > >>>>> > >>>>>>> o: 303.746.7302 > >>>>> > >>>>>>> Advancing the way the world uses the > >>>>> > >>>>>>> > cloud<http://solidfire.com/solution/overview/?video=play> > >>>>> > >>>>>>> *(tm)* > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> > >>>>> *Mike Tutkowski* > >>>>> > >>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>> > >>>>> e: mike.tutkow...@solidfire.com > >>>>> > >>>>> o: 303.746.7302 > >>>>> > >>>>> Advancing the way the world uses the > >>>>> > >>>>> cloud<http://solidfire.com/solution/overview/?video=play> > >>>>> > >>>>> *(tm)* > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> -- > >>>>> > >>> *Mike Tutkowski* > >>>>> > >>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>> > >>> e: mike.tutkow...@solidfire.com > >>>>> > >>> o: 303.746.7302 > >>>>> > >>> Advancing the way the world uses the > >>>>> > >>> cloud<http://solidfire.com/solution/overview/?video=play> > >>>>> > >>> *(tm)* > >>>>> > >> > >>>>> > >> > >>>>> > > > >>>>> > > > >>>>> > > -- > >>>>> > > *Mike Tutkowski* > >>>>> > > *Senior CloudStack Developer, SolidFire Inc.* > >>>>> > > e: mike.tutkow...@solidfire.com > >>>>> > > o: 303.746.7302 > >>>>> > > Advancing the way the world uses the > >>>>> > > cloud<http://solidfire.com/solution/overview/?video=play> > >>>>> > > *(tm)* > >>>>> > > >>>>> > > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> *Mike Tutkowski* > >>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>> e: mike.tutkow...@solidfire.com > >>>> o: 303.746.7302 > >>>> Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > >>>> *(tm)* > >>>> > >>>> > >>>> > >>> > >>> > >>> -- > >>> *Mike Tutkowski* > >>> *Senior CloudStack Developer, SolidFire Inc.* > >>> e: mike.tutkow...@solidfire.com > >>> o: 303.746.7302 > >>> Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > >>> *(tm)* > >>> > >>> > >>> > >> > >> > >> -- > >> *Mike Tutkowski* > >> *Senior CloudStack Developer, SolidFire Inc.* > >> e: mike.tutkow...@solidfire.com > >> o: 303.746.7302 > >> Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > >> *(tm)* > >> > >> > >> > > > > > > -- > > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > > *(tm)* > > > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *(tm)*
