Hi Hieu, Thanks for sending a link to your proposal.
Some items we should consider: 1) We need to make sure that CloudStack does not delete your golden template in the background. As it stands today with XenServer, if a template resides on a primary storage and no VDI is referencing it, the template will eventually get deleted. We would need to make sure that - even though another VDI on another SR is referencing your golden template - it does not get deleted (i.e. that CloudStack understands not to delete the template due to this new use case). Also, the reverse should still work: if no VDI on any SR is referencing this template, the template should get deleted in a similar fashion to how this works today. 2) Is it true that you are proposing that a given primary storage be dedicated to hosting only golden templates? In other words, it cannot also be used for traditional template/root disks? 3) I recommend you diagram how VM migration would work in this new model. 4) I recommend you diagram how a VM snapshot and backup/restore would work in this new model. Thanks! Mike On Thu, Jun 5, 2014 at 6:11 AM, Punith S <punit...@cloudbyte.com> wrote: > hi Hieu, > > after going through your "Golden Primary Storage" proposal , from my > understanding you are creating a SSD golden PS for holding parent > VDH(nothing but the template which go copied from secondary storage) and a > normal primary storage for ROOT volumes(child VHD) for the corresponding > vm's. > > from the following flowchart , i have the following questions, > > 1. since you are having problem with slow boot time of the vm's, will the > booting of the vm's happen in golden PS, ie while cloning ? > if so, the spawning of the vm's will be always fast . > > but i see you are starting the vm after moving the cloned vhd to the > ROOT PS and pointing the child vhd to its parent vhd on the GOLDEN PS, > hence , there will be a network traffic between these two > primary storages, which will obviously slow down the vm's performance > forever. > > 2. what if someone removes the golden primary storage containing the the > parent VHD(template) where all the child VDH's in the root primary storage > are been pointed to ? > if so, all vm's running will be crashed immediately. since its child > vhd's parent is removed. > > thanks > > > On Thu, Jun 5, 2014 at 8:59 AM, Hieu LE <hieul...@gmail.com> wrote: > >> Mike, Punith, >> >> Please review "Golden Primary Storage" proposal. [1] >> >> Thank you. >> >> [1]: >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage >> >> >> On Wed, Jun 4, 2014 at 10:32 PM, Mike Tutkowski < >> mike.tutkow...@solidfire.com> wrote: >> >>> Daan helped out with this. You should be good to go now. >>> >>> >>> On Tue, Jun 3, 2014 at 8:50 PM, Hieu LE <hieul...@gmail.com> wrote: >>> >>> > Hi Mike, >>> > >>> > Could you please give edit/create permission on ASF Jira/Wiki >>> confluence ? >>> > I can not add a new Wiki page. >>> > >>> > My Jira ID: hieulq >>> > Wiki: hieulq89 >>> > Review Board: hieulq >>> > >>> > Thanks ! >>> > >>> > >>> > On Wed, Jun 4, 2014 at 9:17 AM, Mike Tutkowski < >>> > mike.tutkow...@solidfire.com >>> > > wrote: >>> > >>> > > Hi, >>> > > >>> > > Yes, please feel free to add a new Wiki page for your design. >>> > > >>> > > Here is a link to applicable design info: >>> > > >>> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Design >>> > > >>> > > Also, feel free to ask more questions and have me review your design. >>> > > >>> > > Thanks! >>> > > Mike >>> > > >>> > > >>> > > On Tue, Jun 3, 2014 at 7:29 PM, Hieu LE <hieul...@gmail.com> wrote: >>> > > >>> > > > Hi Mike, >>> > > > >>> > > > You are right, performance will be decreased over time because >>> writes >>> > > IOPS >>> > > > will always end up on slower storage pool. >>> > > > >>> > > > In our case, we are using CloudStack integrated in VDI solution to >>> > > provived >>> > > > pooled VM type[1]. So may be my approach can bring better UX for >>> user >>> > > with >>> > > > lower bootime ... >>> > > > >>> > > > A short change in design are followings >>> > > > - VM will be deployed with golden primary storage if primary >>> storage is >>> > > > marked golden and this VM template is also marked as golden. >>> > > > - Choosing the best deploy destionation for both golden primary >>> storage >>> > > and >>> > > > normal root volume primary storage. Chosen host can also access >>> both >>> > > > storage pools. >>> > > > - New Xen Server plug-in for modifying VHD parent id. >>> > > > >>> > > > Is there some place for me to submit my design and code. Can I >>> write a >>> > > new >>> > > > proposal in CS wiki ? >>> > > > >>> > > > [1]: >>> > > > >>> > > > >>> > > >>> > >>> http://support.citrix.com/proddocs/topic/xendesktop-rho/cds-choose-scheme-type-rho.html >>> > > > >>> > > > >>> > > > On Mon, Jun 2, 2014 at 9:04 PM, Mike Tutkowski < >>> > > > mike.tutkow...@solidfire.com >>> > > > > wrote: >>> > > > >>> > > > > It is an interesting idea. If the constraints you face at your >>> > company >>> > > > can >>> > > > > be corrected somewhat by implementing this, then you should go >>> for >>> > it. >>> > > > > >>> > > > > It sounds like writes will be placed on the slower storage pool. >>> This >>> > > > means >>> > > > > as you update OS components, those updates will be placed on the >>> > slower >>> > > > > storage pool. As such, your performance is likely to somewhat >>> > decrease >>> > > > over >>> > > > > time (as more and more writes end up on the slower storage pool). >>> > > > > >>> > > > > That may be OK for your use case(s), though. >>> > > > > >>> > > > > You'll have to update the storage-pool orchestration logic to >>> take >>> > this >>> > > > new >>> > > > > scheme into account. >>> > > > > >>> > > > > Also, we'll have to figure out how this ties into storage >>> tagging (if >>> > > at >>> > > > > all). >>> > > > > >>> > > > > I'd be happy to review your design and code. >>> > > > > >>> > > > > >>> > > > > On Mon, Jun 2, 2014 at 1:54 AM, Hieu LE <hieul...@gmail.com> >>> wrote: >>> > > > > >>> > > > > > Thanks Mike and Punith for quick reply. >>> > > > > > >>> > > > > > Both solutions you gave here are absolutely correct. But as I >>> > > mentioned >>> > > > > in >>> > > > > > the first email, I want another better solution for current >>> > > > > infrastructure >>> > > > > > at my company. >>> > > > > > >>> > > > > > Creating a high IOPS primary storage using storage tags is >>> good but >>> > > it >>> > > > > will >>> > > > > > be very waste of disk capacity. For example, if I only have >>> 1TB SSD >>> > > and >>> > > > > > deploy 100 VM from a 100GB template. >>> > > > > > >>> > > > > > So I think about a solution where a high IOPS primary storage >>> can >>> > > only >>> > > > > > store golden image (master image), and a child image of this VM >>> > will >>> > > be >>> > > > > > stored in another normal (NFS, ISCSI...) storage. In this case, >>> > with >>> > > > 1TB >>> > > > > > SSD Primary Storage I can store as much golden image as I need. >>> > > > > > >>> > > > > > I have also tested it with 256 GB SSD mounted on Xen Server >>> 6.2.0 >>> > > with >>> > > > > 2TB >>> > > > > > local storage 10000RPM, 6TB NFS share storage with 1GB >>> network. The >>> > > > IOPS >>> > > > > of >>> > > > > > VMs which have golden image (master image) in SSD and child >>> image >>> > in >>> > > > NFS >>> > > > > > increate more than 30-40% compare with VMs which have both >>> golden >>> > > image >>> > > > > and >>> > > > > > child image in NFS. The boot time of each VM is also decrease. >>> > > ('cause >>> > > > > > golden image in SSD only reduced READ IOPS). >>> > > > > > >>> > > > > > Do you think this approach OK ? >>> > > > > > >>> > > > > > >>> > > > > > On Mon, Jun 2, 2014 at 12:50 PM, Mike Tutkowski < >>> > > > > > mike.tutkow...@solidfire.com> wrote: >>> > > > > > >>> > > > > > > Thanks, Punith - this is similar to what I was going to say. >>> > > > > > > >>> > > > > > > Any time a set of CloudStack volumes share IOPS from a common >>> > pool, >>> > > > you >>> > > > > > > cannot guarantee IOPS to a given CloudStack volume at a given >>> > time. >>> > > > > > > >>> > > > > > > Your choices at present are: >>> > > > > > > >>> > > > > > > 1) Use managed storage (where you can create a 1:1 mapping >>> > between >>> > > a >>> > > > > > > CloudStack volume and a volume on a storage system that has >>> QoS). >>> > > As >>> > > > > > Punith >>> > > > > > > mentioned, this requires that you purchase storage from a >>> vendor >>> > > who >>> > > > > > > provides guaranteed QoS on a volume-by-volume bases AND has >>> this >>> > > > > > integrated >>> > > > > > > into CloudStack. >>> > > > > > > >>> > > > > > > 2) Create primary storage in CloudStack that is not managed, >>> but >>> > > has >>> > > > a >>> > > > > > high >>> > > > > > > number of IOPS (ex. using SSDs). You can then storage tag >>> this >>> > > > primary >>> > > > > > > storage and create Compute and Disk Offerings that use this >>> > storage >>> > > > tag >>> > > > > > to >>> > > > > > > make sure their volumes end up on this storage pool (primary >>> > > > storage). >>> > > > > > This >>> > > > > > > will still not guarantee IOPS on a CloudStack >>> volume-by-volume >>> > > basis, >>> > > > > but >>> > > > > > > it will at least place the CloudStack volumes that need a >>> better >>> > > > chance >>> > > > > > of >>> > > > > > > getting higher IOPS on a storage pool that could provide the >>> > > > necessary >>> > > > > > > IOPS. A big downside here is that you want to watch how many >>> > > > CloudStack >>> > > > > > > volumes get deployed on this primary storage because you'll >>> need >>> > to >>> > > > > > > essentially over-provision IOPS in this primary storage to >>> > increase >>> > > > the >>> > > > > > > probability that each and every CloudStack volume that uses >>> this >>> > > > > primary >>> > > > > > > storage gets the necessary IOPS (and isn't as likely to >>> suffer >>> > from >>> > > > the >>> > > > > > > Noisy Neighbor Effect). You should be able to tell >>> CloudStack to >>> > > only >>> > > > > > use, >>> > > > > > > say, 80% (or whatever) of the storage you're providing to it >>> (so >>> > as >>> > > > to >>> > > > > > > increase your effective IOPS per GB ratio). This >>> > over-provisioning >>> > > of >>> > > > > > IOPS >>> > > > > > > to control Noisy Neighbors is avoided in option 1. In that >>> > > situation, >>> > > > > you >>> > > > > > > only provision the IOPS and capacity you actually need. It >>> is a >>> > > much >>> > > > > more >>> > > > > > > sophisticated approach. >>> > > > > > > >>> > > > > > > Thanks, >>> > > > > > > Mike >>> > > > > > > >>> > > > > > > >>> > > > > > > On Sun, Jun 1, 2014 at 11:36 PM, Punith S < >>> > punit...@cloudbyte.com> >>> > > > > > wrote: >>> > > > > > > >>> > > > > > > > hi hieu, >>> > > > > > > > >>> > > > > > > > your problem is the bottle neck we see as a storage >>> vendors in >>> > > the >>> > > > > > cloud, >>> > > > > > > > meaning all the vms in the cloud have not been guaranteed >>> iops >>> > > from >>> > > > > the >>> > > > > > > > primary storage, because in your case i'm assuming you are >>> > > running >>> > > > > > > 1000vms >>> > > > > > > > on a xen cluster whose all vm's disks are lying on a same >>> > primary >>> > > > nfs >>> > > > > > > > storage mounted to the cluster, >>> > > > > > > > hence you won't get the dedicated iops for each vm since >>> every >>> > vm >>> > > > is >>> > > > > > > > sharing the same storage. to solve this issue in >>> cloudstack we >>> > > the >>> > > > > > third >>> > > > > > > > party vendors have implemented the plugin(namely cloudbyte >>> , >>> > > > > solidfire >>> > > > > > > etc) >>> > > > > > > > to support managed storage(dedicated volumes with >>> guaranteed >>> > qos >>> > > > for >>> > > > > > each >>> > > > > > > > vms) , where we are mapping each root disk(vdi) or data >>> disk >>> > of a >>> > > > vm >>> > > > > > with >>> > > > > > > > one nfs or iscsi share coming out of a pool, also we are >>> > > proposing >>> > > > > the >>> > > > > > > new >>> > > > > > > > feature to change volume iops on fly in 4.5, where you can >>> > > increase >>> > > > > or >>> > > > > > > > decrease your root disk iops while booting or at peak >>> times. >>> > but >>> > > to >>> > > > > use >>> > > > > > > > this plugin you have to buy our storage solution. >>> > > > > > > > >>> > > > > > > > if not , you can try creating a nfs share out of ssd pool >>> > storage >>> > > > and >>> > > > > > > > create a primary storage in cloudstack out of it named as >>> > golden >>> > > > > > primary >>> > > > > > > > storage with specific tag like gold, and create a compute >>> > > offering >>> > > > > for >>> > > > > > > your >>> > > > > > > > template with the storage tag as gold, hence all the vm's >>> you >>> > > > create >>> > > > > > will >>> > > > > > > > sit on this gold primary storage with high iops. and other >>> data >>> > > > disks >>> > > > > > on >>> > > > > > > > other primary storage but still here you cannot guarantee >>> the >>> > qos >>> > > > at >>> > > > > vm >>> > > > > > > > level. >>> > > > > > > > >>> > > > > > > > thanks >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > On Mon, Jun 2, 2014 at 10:12 AM, Hieu LE < >>> hieul...@gmail.com> >>> > > > wrote: >>> > > > > > > > >>> > > > > > > >> Hi all, >>> > > > > > > >> >>> > > > > > > >> There are some problems while deploying a large amount of >>> VMs >>> > in >>> > > > my >>> > > > > > > >> company >>> > > > > > > >> with CloudStack. All VMs are deployed from same template >>> (e.g: >>> > > > > Windows >>> > > > > > > 7) >>> > > > > > > >> and the quantity is approximately ~1000VMs. The problems >>> here >>> > is >>> > > > low >>> > > > > > > IOPS, >>> > > > > > > >> low performance of VM (about ~10-11 IOPS, boot time is >>> very >>> > > high). >>> > > > > The >>> > > > > > > >> storage of my company is SAN/NAS with NFS and Xen Server >>> > 6.2.0. >>> > > > All >>> > > > > > Xen >>> > > > > > > >> Server nodes have standard server HDD disk raid. >>> > > > > > > >> >>> > > > > > > >> I have found some solutions for this such as: >>> > > > > > > >> >>> > > > > > > >> - Enable Xen Server Intellicache and some tweaks in >>> > > CloudStack >>> > > > > > codes >>> > > > > > > to >>> > > > > > > >> deploy and start VM in Intellicache mode. But this >>> solution >>> > > > will >>> > > > > > > >> transfer >>> > > > > > > >> all IOPS from shared storage to all local storage, >>> hence >>> > > affect >>> > > > > and >>> > > > > > > >> limit >>> > > > > > > >> some CloudStack features. >>> > > > > > > >> - Buying some expensive storage solutions and network >>> to >>> > > > increase >>> > > > > > > IOPS. >>> > > > > > > >> Nah.. >>> > > > > > > >> >>> > > > > > > >> So, I am thinking about a new feature that (may be) >>> increasing >>> > > > IOPS >>> > > > > > and >>> > > > > > > >> performance of VMs: >>> > > > > > > >> >>> > > > > > > >> 1. Separate golden image in high IOPS partition: >>> buying new >>> > > > SSD, >>> > > > > > plug >>> > > > > > > >> in >>> > > > > > > >> Xen Server and deployed a new VM in NFS storage WITH >>> golden >>> > > > image >>> > > > > > in >>> > > > > > > >> this >>> > > > > > > >> new SSD partition. This can reduce READ IOPS in shared >>> > > storage >>> > > > > and >>> > > > > > > >> decrease >>> > > > > > > >> boot time of VM. (Currenty, VM deployed in Xen Server >>> > always >>> > > > > have a >>> > > > > > > >> master >>> > > > > > > >> image (golden image - in VMWare) always in the same >>> storage >>> > > > > > > repository >>> > > > > > > >> with >>> > > > > > > >> different image (child image)). We can do this trick by >>> > > > tweaking >>> > > > > in >>> > > > > > > VHD >>> > > > > > > >> header file with new Xen Server plug-in. >>> > > > > > > >> 2. Create golden primary storage and VM template that >>> > enable >>> > > > this >>> > > > > > > >> feature. >>> > > > > > > >> 3. So, all VMs deployed from template that had enabled >>> this >>> > > > > feature >>> > > > > > > >> will >>> > > > > > > >> have a golden image stored in golden primary storage >>> (SSD >>> > or >>> > > > some >>> > > > > > > high >>> > > > > > > >> IOPS >>> > > > > > > >> partition), and different image (child image) stored in >>> > other >>> > > > > > normal >>> > > > > > > >> primary storage. >>> > > > > > > >> >>> > > > > > > >> This new feature will not transfer all IOPS from shared >>> > storage >>> > > to >>> > > > > > local >>> > > > > > > >> storage (because high IOPS partition can be another high >>> IOPS >>> > > > shared >>> > > > > > > >> storage) and require less money than buying new storage >>> > > solution. >>> > > > > > > >> >>> > > > > > > >> What do you think ? If possible, may I write a proposal in >>> > > > > CloudStack >>> > > > > > > >> wiki ? >>> > > > > > > >> >>> > > > > > > >> BRs. >>> > > > > > > >> >>> > > > > > > >> Hieu Lee >>> > > > > > > >> >>> > > > > > > >> -- >>> > > > > > > >> -----BEGIN GEEK CODE BLOCK----- >>> > > > > > > >> Version: 3.1 >>> > > > > > > >> GCS/CM/IT/M/MU d-@? s+(++):+(++) !a C++++(++++)$ >>> > ULC++++(++)$ P >>> > > > > > > >> L++(+++)$ E >>> > > > > > > >> !W N* o+ K w O- M V- PS+ PE++ Y+ PGP+ t 5 X R tv+ >>> b+(++)>+++ >>> > DI- >>> > > > D+ >>> > > > > G >>> > > > > > > >> e++(+++) h-- r(++)>+++ y- >>> > > > > > > >> ------END GEEK CODE BLOCK------ >>> > > > > > > >> >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > -- >>> > > > > > > > regards, >>> > > > > > > > >>> > > > > > > > punith s >>> > > > > > > > cloudbyte.com >>> > > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > -- >>> > > > > > > *Mike Tutkowski* >>> > > > > > > *Senior CloudStack Developer, SolidFire Inc.* >>> > > > > > > e: mike.tutkow...@solidfire.com >>> > > > > > > o: 303.746.7302 >>> > > > > > > Advancing the way the world uses the cloud >>> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™* >>> > > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > -- >>> > > > > > -----BEGIN GEEK CODE BLOCK----- >>> > > > > > Version: 3.1 >>> > > > > > GCS/CM/IT/M/MU d-@? s+(++):+(++) !a C++++(++++)$ ULC++++(++)$ >>> P >>> > > > > L++(+++)$ >>> > > > > > E >>> > > > > > !W N* o+ K w O- M V- PS+ PE++ Y+ PGP+ t 5 X R tv+ b+(++)>+++ >>> DI- >>> > D+ G >>> > > > > > e++(+++) h-- r(++)>+++ y- >>> > > > > > ------END GEEK CODE BLOCK------ >>> > > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > *Mike Tutkowski* >>> > > > > *Senior CloudStack Developer, SolidFire Inc.* >>> > > > > e: mike.tutkow...@solidfire.com >>> > > > > o: 303.746.7302 >>> > > > > Advancing the way the world uses the cloud >>> > > > > <http://solidfire.com/solution/overview/?video=play>*™* >>> > > > > >>> > > > >>> > > > >>> > > > >>> > > > -- >>> > > > -----BEGIN GEEK CODE BLOCK----- >>> > > > Version: 3.1 >>> > > > GCS/CM/IT/M/MU d-@? s+(++):+(++) !a C++++(++++)$ ULC++++(++)$ P >>> > > L++(+++)$ >>> > > > E >>> > > > !W N* o+ K w O- M V- PS+ PE++ Y+ PGP+ t 5 X R tv+ b+(++)>+++ DI- >>> D+ G >>> > > > e++(+++) h-- r(++)>+++ y- >>> > > > ------END GEEK CODE BLOCK------ >>> > > > >>> > > >>> > > >>> > > >>> > > -- >>> > > *Mike Tutkowski* >>> > > *Senior CloudStack Developer, SolidFire Inc.* >>> > > e: mike.tutkow...@solidfire.com >>> > > o: 303.746.7302 >>> > > Advancing the way the world uses the cloud >>> > > <http://solidfire.com/solution/overview/?video=play>*™* >>> > > >>> > >>> > >>> > >>> > -- >>> > -----BEGIN GEEK CODE BLOCK----- >>> > Version: 3.1 >>> > GCS/CM/IT/M/MU d-@? s+(++):+(++) !a C++++(++++)$ ULC++++(++)$ P >>> L++(+++)$ >>> > E >>> > !W N* o+ K w O- M V- PS+ PE++ Y+ PGP+ t 5 X R tv+ b+(++)>+++ DI- D+ G >>> > e++(+++) h-- r(++)>+++ y- >>> > ------END GEEK CODE BLOCK------ >>> > >>> >>> >>> >>> -- >>> *Mike Tutkowski* >>> *Senior CloudStack Developer, SolidFire Inc.* >>> e: mike.tutkow...@solidfire.com >>> o: 303.746.7302 >>> Advancing the way the world uses the cloud >>> <http://solidfire.com/solution/overview/?video=play>*™* >>> >> >> >> >> -- >> -----BEGIN GEEK CODE BLOCK----- >> Version: 3.1 >> GCS/CM/IT/M/MU d-@? s+(++):+(++) !a C++++(++++)$ ULC++++(++)$ P >> L++(+++)$ E !W N* o+ K w O- M V- PS+ PE++ Y+ PGP+ t 5 X R tv+ b+(++)>+++ >> DI- D+ G e++(+++) h-- r(++)>+++ y- >> ------END GEEK CODE BLOCK------ >> > > > > -- > regards, > > punith s > cloudbyte.com > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud <http://solidfire.com/solution/overview/?video=play>*™*
