Mike,
The current implementation of the Dynamic type attach behavior works in terms
of Xen ISCSI which why I ask about the difference. Another way to ask the
question -- what is the definition of a Dynamic storage pool type?
Thanks,
-John
On Jun 3, 2013, at 5:10 PM, Mike Tutkowski <mike.tutkow...@solidfire.com> wrote:
> As far as I know, the iSCSI type is uniquely used by XenServer when you
> want to set up Primary Storage that is directly based on an iSCSI target.
> This allows you to skip the step of going to the hypervisor and creating a
> storage repository based on that iSCSI target as CloudStack does that part
> for you. I think this is only supported for XenServer. For all other
> hypervisors, you must first go to the hypervisor and perform this step
> manually.
>
> I don't really know what RBD is.
>
>
> On Mon, Jun 3, 2013 at 2:13 PM, John Burwell <jburw...@basho.com> wrote:
>
>> Mike,
>>
>> Reading through the code, what is the difference between the ISCSI and
>> Dynamic types? Why isn't RBD considered Dynamic?
>>
>> Thanks,
>> -John
>>
>> On Jun 3, 2013, at 3:46 PM, Mike Tutkowski <mike.tutkow...@solidfire.com>
>> wrote:
>>
>>> This new type of storage is defined in the Storage.StoragePoolType class
>>> (called Dynamic):
>>>
>>> public static enum StoragePoolType {
>>>
>>> Filesystem(false), // local directory
>>>
>>> NetworkFilesystem(true), // NFS or CIFS
>>>
>>> IscsiLUN(true), // shared LUN, with a clusterfs overlay
>>>
>>> Iscsi(true), // for e.g., ZFS Comstar
>>>
>>> ISO(false), // for iso image
>>>
>>> LVM(false), // XenServer local LVM SR
>>>
>>> CLVM(true),
>>>
>>> RBD(true),
>>>
>>> SharedMountPoint(true),
>>>
>>> VMFS(true), // VMware VMFS storage
>>>
>>> PreSetup(true), // for XenServer, Storage Pool is set up by
>>> customers.
>>>
>>> EXT(false), // XenServer local EXT SR
>>>
>>> OCFS2(true),
>>>
>>> Dynamic(true); // dynamic, zone-wide storage (ex. SolidFire)
>>>
>>>
>>> boolean shared;
>>>
>>>
>>> StoragePoolType(boolean shared) {
>>>
>>> this.shared = shared;
>>>
>>> }
>>>
>>>
>>> public boolean isShared() {
>>>
>>> return shared;
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> On Mon, Jun 3, 2013 at 1:41 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com
>>>> wrote:
>>>
>>>> For example, let's say another storage company wants to implement a
>>>> plug-in to leverage its Quality of Service feature. It would be dynamic,
>>>> zone-wide storage, as well. They would need only implement a storage
>>>> plug-in as I've made the necessary changes to the hypervisor-attach
>> logic
>>>> to support their plug-in.
>>>>
>>>>
>>>> On Mon, Jun 3, 2013 at 1:39 PM, Mike Tutkowski <
>>>> mike.tutkow...@solidfire.com> wrote:
>>>>
>>>>> Oh, sorry to imply the XenServer code is SolidFire specific. It is not.
>>>>>
>>>>> The XenServer attach logic is now aware of dynamic, zone-wide storage
>>>>> (and SolidFire is an implementation of this kind of storage). This
>> kind of
>>>>> storage is new to 4.2 with Edison's storage framework changes.
>>>>>
>>>>> Edison created a new framework that supported the creation and deletion
>>>>> of volumes dynamically. However, when I visited with him in Portland
>> back
>>>>> in April, we realized that it was not complete. We realized there was
>>>>> nothing CloudStack could do with these volumes unless the attach logic
>> was
>>>>> changed to recognize this new type of storage and create the
>> appropriate
>>>>> hypervisor data structure.
>>>>>
>>>>>
>>>>> On Mon, Jun 3, 2013 at 1:28 PM, John Burwell <jburw...@basho.com>
>> wrote:
>>>>>
>>>>>> Mike,
>>>>>>
>>>>>> It is generally odd to me that any operation in the Storage layer
>> would
>>>>>> understand or care about details. I expect to see the Storage
>> services
>>>>>> expose a set of operations that can be composed/driven by the
>> Hypervisor
>>>>>> implementations to allocate space/create structures per their needs.
>> If
>>>>>> we
>>>>>> don't invert this dependency, we are going to end with a massive
>> n-to-n
>>>>>> problem that will make the system increasingly difficult to maintain
>> and
>>>>>> enhance. Am I understanding that the Xen specific SolidFire code is
>>>>>> located in the CitrixResourceBase class?
>>>>>>
>>>>>> Thanks,
>>>>>> -John
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 3, 2013 at 3:12 PM, Mike Tutkowski <
>>>>>> mike.tutkow...@solidfire.com
>>>>>>> wrote:
>>>>>>
>>>>>>> To delve into this in a bit more detail:
>>>>>>>
>>>>>>> Prior to 4.2 and aside from one setup method for XenServer, the admin
>>>>>> had
>>>>>>> to first create a volume on the storage system, then go into the
>>>>>> hypervisor
>>>>>>> to set up a data structure to make use of the volume (ex. a storage
>>>>>>> repository on XenServer or a datastore on ESX(i)). VMs and data disks
>>>>>> then
>>>>>>> shared this storage system's volume.
>>>>>>>
>>>>>>> With Edison's new storage framework, storage need no longer be so
>>>>>> static
>>>>>>> and you can easily create a 1:1 relationship between a storage
>> system's
>>>>>>> volume and the VM's data disk (necessary for storage Quality of
>>>>>> Service).
>>>>>>>
>>>>>>> You can now write a plug-in that is called to dynamically create and
>>>>>> delete
>>>>>>> volumes as needed.
>>>>>>>
>>>>>>> The problem that the storage framework did not address is in creating
>>>>>> and
>>>>>>> deleting the hypervisor-specific data structure when performing an
>>>>>>> attach/detach.
>>>>>>>
>>>>>>> That being the case, I've been enhancing it to do so. I've got
>>>>>> XenServer
>>>>>>> worked out and submitted. I've got ESX(i) in my sandbox and can
>> submit
>>>>>> this
>>>>>>> if we extend the 4.2 freeze date.
>>>>>>>
>>>>>>> Does that help a bit? :)
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 3, 2013 at 1:03 PM, Mike Tutkowski <
>>>>>>> mike.tutkow...@solidfire.com
>>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi John,
>>>>>>>>
>>>>>>>> The storage plug-in - by itself - is hypervisor agnostic.
>>>>>>>>
>>>>>>>> The issue is with the volume-attach logic (in the agent code). The
>>>>>>> storage
>>>>>>>> framework calls into the plug-in to have it create a volume as
>>>>>> needed,
>>>>>>> but
>>>>>>>> when the time comes to attach the volume to a hypervisor, the attach
>>>>>>> logic
>>>>>>>> has to be smart enough to recognize it's being invoked on zone-wide
>>>>>>> storage
>>>>>>>> (where the volume has just been created) and create, say, a storage
>>>>>>>> repository (for XenServer) or a datastore (for VMware) to make use
>>>>>> of the
>>>>>>>> volume that was just created.
>>>>>>>>
>>>>>>>> I've been spending most of my time recently making the attach logic
>>>>>> work
>>>>>>>> in the agent code.
>>>>>>>>
>>>>>>>> Does that clear it up?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jun 3, 2013 at 12:48 PM, John Burwell <jburw...@basho.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Mike,
>>>>>>>>>
>>>>>>>>> Can you explain why the the storage driver is hypervisor specific?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> -John
>>>>>>>>>
>>>>>>>>> On Jun 3, 2013, at 1:21 PM, Mike Tutkowski <
>>>>>>> mike.tutkow...@solidfire.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, ultimately I would like to support all hypervisors that
>>>>>>> CloudStack
>>>>>>>>>> supports. I think I'm just out of time for 4.2 to get KVM in.
>>>>>>>>>>
>>>>>>>>>> Right now this plug-in supports XenServer. Depending on what we do
>>>>>>> with
>>>>>>>>>> regards to 4.2 feature freeze, I have it working for VMware in my
>>>>>>>>> sandbox,
>>>>>>>>>> as well.
>>>>>>>>>>
>>>>>>>>>> Also, just to be clear, this is all in regards to Disk Offerings.
>>>>>> I
>>>>>>>>> plan to
>>>>>>>>>> support Compute Offerings post 4.2.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jun 3, 2013 at 11:14 AM, Kelcey Jamison Damage <
>>>>>>> kel...@bbits.ca
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Is there any plan on supporting KVM in the patch cycle post 4.2?
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Mike Tutkowski" <mike.tutkow...@solidfire.com>
>>>>>>>>>>> To: dev@cloudstack.apache.org
>>>>>>>>>>> Sent: Monday, June 3, 2013 10:12:32 AM
>>>>>>>>>>> Subject: Re: [MERGE] disk_io_throttling to MASTER
>>>>>>>>>>>
>>>>>>>>>>> I agree on merging Wei's feature first, then mine.
>>>>>>>>>>>
>>>>>>>>>>> If his feature is for KVM only, then it is a non issue as I don't
>>>>>>>>> support
>>>>>>>>>>> KVM in 4.2.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 3, 2013 at 8:55 AM, Wei ZHOU <ustcweiz...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> John,
>>>>>>>>>>>>
>>>>>>>>>>>> For the billing, as no one works on billing now, users need to
>>>>>>>>> calculate
>>>>>>>>>>>> the billing by themselves. They can get the service_offering and
>>>>>>>>>>>> disk_offering of a VMs and volumes for calculation. Of course
>>>>>> it is
>>>>>>>>>>> better
>>>>>>>>>>>> to tell user the exact limitation value of individual volume,
>>>>>> and
>>>>>>>>> network
>>>>>>>>>>>> rate limitation for nics as well. I can work on it later. Do you
>>>>>>>>> think it
>>>>>>>>>>>> is a part of I/O throttling?
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry my misunstand the second the question.
>>>>>>>>>>>>
>>>>>>>>>>>> Agree with what you said about the two features.
>>>>>>>>>>>>
>>>>>>>>>>>> -Wei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/6/3 John Burwell <jburw...@basho.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Wei,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jun 3, 2013, at 2:13 AM, Wei ZHOU <ustcweiz...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi John, Mike
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I hope Mike's aswer helps you. I am trying to adding more.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (1) I think billing should depend on IO statistics rather than
>>>>>>> IOPS
>>>>>>>>>>>>>> limitation. Please review disk_io_stat if you have time.
>>>>>>>>>>> disk_io_stat
>>>>>>>>>>>>> can
>>>>>>>>>>>>>> get the IO statistics including bytes/iops read/write for an
>>>>>>>>>>> individual
>>>>>>>>>>>>>> virtual machine.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Going by the AWS model, customers are billed more for volumes
>>>>>> with
>>>>>>>>>>>>> provisioned IOPS, as well as, for those operations (
>>>>>>>>>>>>> http://aws.amazon.com/ebs/). I would imagine our users would
>>>>>> like
>>>>>>>>> the
>>>>>>>>>>>>> option to employ similar cost models. Could an operator
>>>>>> implement
>>>>>>>>>>> such a
>>>>>>>>>>>>> billing model in the current patch?
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (2) Do you mean IOPS runtime change? KVM supports setting
>>>>>> IOPS/BPS
>>>>>>>>>>>>>> limitation for a running virtual machine through command line.
>>>>>>>>>>> However,
>>>>>>>>>>>>>> CloudStack does not support changing the parameters of a
>>>>>> created
>>>>>>>>>>>> offering
>>>>>>>>>>>>>> (computer offering or disk offering).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I meant at the Java interface level. I apologize for being
>>>>>>> unclear.
>>>>>>>>>>> Can
>>>>>>>>>>>>> we more generalize allocation algorithms with a set of
>>>>>> interfaces
>>>>>>>>> that
>>>>>>>>>>>>> describe the service guarantees provided by a resource?
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (3) It is a good question. Maybe it is better to commit Mike's
>>>>>>> patch
>>>>>>>>>>>>> after
>>>>>>>>>>>>>> disk_io_throttling as Mike needs to consider the limitation in
>>>>>>>>>>>> hypervisor
>>>>>>>>>>>>>> type, I think.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I will expand on my thoughts in a later response to Mike
>>>>>> regarding
>>>>>>>>> the
>>>>>>>>>>>>> touch points between these two features. I think that
>>>>>>>>>>> disk_io_throttling
>>>>>>>>>>>>> will need to be merged before SolidFire, but I think we need
>>>>>> closer
>>>>>>>>>>>>> coordination between the branches (possibly have have solidfire
>>>>>>> track
>>>>>>>>>>>>> disk_io_throttling) to coordinate on this issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Wei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/6/3 John Burwell <jburw...@basho.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Mike,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The things I want to understand are the following:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) Is there value in capturing IOPS policies be captured in
>>>>>> a
>>>>>>>>>>> common
>>>>>>>>>>>>>>> data model (e.g. for billing/usage purposes, expressing
>>>>>>> offerings).
>>>>>>>>>>>>>>> 2) Should there be a common interface model for reasoning
>>>>>> about
>>>>>>>>>>> IOP
>>>>>>>>>>>>>>> provisioning at runtime?
>>>>>>>>>>>>>>> 3) How are conflicting provisioned IOPS configurations
>>>>>> between
>>>>>>> a
>>>>>>>>>>>>>>> hypervisor and storage device reconciled? In particular, a
>>>>>>>>> scenario
>>>>>>>>>>>>> where
>>>>>>>>>>>>>>> is lead to believe (and billed) for more IOPS configured for
>>>>>> a VM
>>>>>>>>>>>> than a
>>>>>>>>>>>>>>> storage device has been configured to deliver. Another
>>>>>> scenario
>>>>>>>>>>>> could a
>>>>>>>>>>>>>>> consistent configuration between a VM and a storage device at
>>>>>>>>>>> creation
>>>>>>>>>>>>>>> time, but a later modification to storage device introduces
>>>>>>> logical
>>>>>>>>>>>>>>> inconsistency.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Jun 2, 2013, at 8:38 PM, Mike Tutkowski <
>>>>>>>>>>>>> mike.tutkow...@solidfire.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi John,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I believe Wei's feature deals with controlling the max
>>>>>> number of
>>>>>>>>>>> IOPS
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>> the hypervisor side.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My feature is focused on controlling IOPS from the storage
>>>>>> system
>>>>>>>>>>>> side.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I hope that helps. :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Jun 2, 2013 at 6:35 PM, John Burwell <
>>>>>> jburw...@basho.com
>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Wei,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My opinion is that no features should be merged until all
>>>>>>>>>>> functional
>>>>>>>>>>>>>>>> issues have been resolved and it is ready to turn over to
>>>>>> test.
>>>>>>>>>>>> Until
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> total Ops vs discrete read/write ops issue is addressed and
>>>>>>>>>>>> re-reviewed
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> Wido, I don't think this criteria has been satisfied.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, how does this work intersect/compliment the SolidFire
>>>>>>> patch
>>>>>>>>> (
>>>>>>>>>>>>>>>> https://reviews.apache.org/r/11479/)? As I understand it
>>>>>> that
>>>>>>>>>>> work
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> also involves provisioned IOPS. I would like to ensure we
>>>>>> don't
>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>> scenario where provisioned IOPS in KVM and SolidFire are
>>>>>>>>>>>> unnecessarily
>>>>>>>>>>>>>>>> incompatible.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Jun 1, 2013, at 6:47 AM, Wei ZHOU <ustcweiz...@gmail.com
>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Wido,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sure. I will change it next week.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Wei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/6/1 Wido den Hollander <w...@widodh.nl>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Wei,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 06/01/2013 08:24 AM, Wei ZHOU wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Wido,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Exactly. I have pushed the features into master.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If anyone object thems for technical reason till Monday, I
>>>>>> will
>>>>>>>>>>>> revert
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For the sake of clarity I just want to mention again that we
>>>>>>>>> should
>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the total IOps to R/W IOps asap so that we never release a
>>>>>>> version
>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> only total IOps.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You laid the groundwork for the I/O throttling and that's
>>>>>> great!
>>>>>>>>> We
>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> however prevent that we create legacy from day #1.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Wido
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Wei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/5/31 Wido den Hollander <w...@widodh.nl>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 05/31/2013 03:59 PM, John Burwell wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Wido,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +1 -- this enhancement must to discretely support read and
>>>>>> write
>>>>>>>>>>>> IOPS.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> don't see how it could be fixed later because I don't see
>>>>>> how we
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> correctly
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> split total IOPS into read and write. Therefore, we would
>>>>>> be
>>>>>>>>> stuck
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> with a
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> total unless/until we decided to break backwards
>>>>>> compatibility.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What Wei meant was merging it into master now so that it
>>>>>> will go
>>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 4.2 branch and add Read / Write IOps before the 4.2 release
>>>>>> so
>>>>>>>>> that
>>>>>>>>>>>> 4.2
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> will be released with Read and Write instead of Total IOps.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is to make the May 31st feature freeze date. But if the
>>>>>>>>> window
>>>>>>>>>>>>> moves
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (see other threads) then it won't be necessary to do that.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Wido
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I also completely agree that there is no association between
>>>>>>>>>>> network
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> disk I/O.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On May 31, 2013, at 9:51 AM, Wido den Hollander <
>>>>>> w...@widodh.nl
>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Wei,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 05/31/2013 03:13 PM, Wei ZHOU wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Wido,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks. Good question.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I thought about at the beginning. Finally I decided to
>>>>>> ignore
>>>>>>> the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> difference of read and write mainly because the network
>>>>>>> throttling
>>>>>>>>>>>> did
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> care the difference of sent and received bytes as well.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That reasoning seems odd. Networking and disk I/O completely
>>>>>>>>>>>> different.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Disk I/O is much more expensive in most situations then
>>>>>> network
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> bandwith.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Implementing it will be some copy-paste work. It could be
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> implemented in
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> few days. For the deadline of feature freeze, I will
>>>>>> implement
>>>>>>> it
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> that , if needed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It think it's a feature we can't miss. But if it goes into
>>>>>> the
>>>>>>> 4.2
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> window we have to make sure we don't release with only total
>>>>>>> IOps
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> it in 4.3, that will confuse users.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Wido
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Wei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/5/31 Wido den Hollander <w...@widodh.nl>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Wei,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 05/30/2013 06:03 PM, Wei ZHOU wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would like to merge disk_io_throttling branch into master.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If nobody object, I will merge into master in 48 hours.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The purpose is :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Virtual machines are running on the same storage device
>>>>>> (local
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> storage or
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> share strage). Because of the rate limitation of device
>>>>>> (such as
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> iops), if
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> one VM has large disk operation, it may affect the disk
>>>>>>>>> performance
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> other VMs running on the same storage device.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It is neccesary to set the maximum rate and limit the disk
>>>>>> I/O
>>>>>>> of
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> VMs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Looking at the code I see you make no difference between
>>>>>> Read
>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Write
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IOps.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Qemu and libvirt support setting both a different rate for
>>>>>> Read
>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Write
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IOps which could benefit a lot of users.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's also strange, in the polling side you collect both the
>>>>>> Read
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Write
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IOps, but on the throttling side you only go for a global
>>>>>> value.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Write IOps are usually much more expensive then Read IOps,
>>>>>> so it
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> like a valid use-case where that an admin would set a lower
>>>>>>> value
>>>>>>>>>>> for
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IOps vs Read IOps.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Since this only supports KVM at this point I think it would
>>>>>> be
>>>>>>> of
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> great
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> value to at least have the mechanism in place to support
>>>>>> both,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> implementing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> this later would be a lot of work.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If a hypervisor doesn't support setting different values for
>>>>>>> read
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> write you can always sum both up and set that as the total
>>>>>>> limit.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you explain why you implemented it this way?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Wido
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The feature includes:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (1) set the maximum rate of VMs (in disk_offering, and
>>>>>> global
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> configuration)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (2) change the maximum rate of VMs
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (3) limit the disk rate (total bps and iops)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> JIRA ticket: https://issues.apache.org/****
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> jira/browse/CLOUDSTACK-1192<ht**tps://
>>>>>> issues.apache.org/****
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> jira/browse/CLOUDSTACK-1192<
>>>>>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <ht**tps://
>>>>>> issues.apache.org/**jira/**browse/CLOUDSTACK-1192<
>>>>>>>>>>>>>>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1192>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <**
>>>>>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192<
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-1192>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> FS (I will update later) :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******<
>>>>>>>>>>>>>>>
>>>>>> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>
>>>>>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**
>>>>>>> <
>>>>>>>>>>>>>>> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> VM+Disk+IO+Throttling<https://****
>>>>>>>>> cwiki.apache.org/confluence/****
>>>>>>>>>>> <
>>>>>>>>>>>>>>>> http://cwiki.apache.org/confluence/**>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<https://cwiki.
>>>>>> **
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling
>>>>>>>>>>> <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Merge check list :-
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * Did you check the branch's RAT execution success?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * Are there new dependencies introduced?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> No
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * What automated testing (unit and integration) is included
>>>>>> in
>>>>>>> the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> feature?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Unit tests are added.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * What testing has been done to check for potential
>>>>>> regressions?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (1) set the bytes rate and IOPS rate on CloudStack UI.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (2) VM operations, including
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> deploy, stop, start, reboot, destroy, expunge. migrate,
>>>>>> restore
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (3) Volume operations, including
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Attach, Detach
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To review the code, you can try
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> git diff c30057635d04a2396f84c588127d7e******be42e503a7
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> f2e5591b710d04cc86815044f5823e******73a4a58944
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Wei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******<
>>>>>>>>>>>>>>>
>>>>>> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>
>>>>>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**
>>>>>>> <
>>>>>>>>>>>>>>> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> VM+Disk+IO+Throttling<https://****
>>>>>>>>> cwiki.apache.org/confluence/****
>>>>>>>>>>> <
>>>>>>>>>>>>>>>> http://cwiki.apache.org/confluence/**>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<https://cwiki.
>>>>>> **
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling
>>>>>>>>>>> <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [2] refs/heads/disk_io_throttling
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>>>>> https://issues.apache.org/******jira/browse/CLOUDSTACK-1301
>>>>>> <
>>>>>>>>>>>>>>> https://issues.apache.org/****jira/browse/CLOUDSTACK-1301>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <ht**tps://
>>>>>> issues.apache.org/****jira/browse/CLOUDSTACK-1301<
>>>>>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <ht**tps://
>>>>>> issues.apache.org/**jira/**browse/CLOUDSTACK-1301<
>>>>>>>>>>>>>>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1301>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <**
>>>>>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301<
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-1301>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <ht**tps://
>>>>>> issues.apache.org/****jira/**browse/CLOUDSTACK-2071<
>>>>>>>>>>>>>>>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> **<
>>>>>>>>>>>>>>>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071<
>>>>>>>>>>>>>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-2071>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <**
>>>>>>>>>>>>>>>> https://issues.apache.org/****jira/browse/CLOUDSTACK-2071<
>>>>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-2071>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <h**ttps://issues.apache.org/jira/**browse/CLOUDSTACK-2071<
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-2071>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (**CLOUDSTACK-1301
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - VM Disk I/O Throttling)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>>>>>>>> e: mike.tutkow...@solidfire.com
>>>>>>>>>>>>>>> o: 303.746.7302
>>>>>>>>>>>>>>> Advancing the way the world uses the
>>>>>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>>>>>>>>> *™*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>>>> e: mike.tutkow...@solidfire.com
>>>>>>>>>>> o: 303.746.7302
>>>>>>>>>>> Advancing the way the world uses the
>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>>>>> *™*
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>>> e: mike.tutkow...@solidfire.com
>>>>>>>>>> o: 303.746.7302
>>>>>>>>>> Advancing the way the world uses the
>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>>>> *™*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Mike Tutkowski*
>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>> e: mike.tutkow...@solidfire.com
>>>>>>>> o: 303.746.7302
>>>>>>>> Advancing the way the world uses the cloud<
>>>>>>> http://solidfire.com/solution/overview/?video=play>
>>>>>>>> *™*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Mike Tutkowski*
>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>> e: mike.tutkow...@solidfire.com
>>>>>>> o: 303.746.7302
>>>>>>> Advancing the way the world uses the
>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>> *™*
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Mike Tutkowski*
>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>> e: mike.tutkow...@solidfire.com
>>>>> o: 303.746.7302
>>>>> Advancing the way the world uses the cloud<
>> http://solidfire.com/solution/overview/?video=play>
>>>>> *™*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Mike Tutkowski*
>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>> e: mike.tutkow...@solidfire.com
>>>> o: 303.746.7302
>>>> Advancing the way the world uses the cloud<
>> http://solidfire.com/solution/overview/?video=play>
>>>> *™*
>>>>
>>>
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkow...@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the
>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>> *™*
>>
>>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *™*
