Oh ya. Gotta love NFS syncing to servers halfway around the world. :P *Will STEVENS* Lead Developer
*CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Tue, Jul 19, 2016 at 4:06 PM, Syed Ahmed <sah...@cloudops.com> wrote: > Right, but is that an artificial limitation imposed by Cloudstack of is > there something that will break if we make NFS region wide? If not, we can > look at moving the NFS to a region wide storage. > > -Syed > > On Tue, Jul 19, 2016 at 3:54 PM, Will Stevens <wstev...@cloudops.com> > wrote: > > > Probably because ACS does not treat it as region wide, unlike the object > > storage implementations. > > > > *Will STEVENS* > > Lead Developer > > > > *CloudOps* *| *Cloud Solutions Experts > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > w cloudops.com *|* tw @CloudOps_ > > > > On Tue, Jul 19, 2016 at 2:52 PM, Syed Ahmed <sah...@cloudops.com> wrote: > > > > > Is there a reason why NFS cannot be region-wide? > > > > > > -Syed > > > > > > On Tue, Jul 19, 2016 at 2:08 PM, Nux! <n...@li.nux.ro> wrote: > > > > > > > Thanks for your input guys. > > > > > > > > I think I'll stay with NFS for now after your replies, but it'd be > > pretty > > > > sweet the day when we'll be having region wide secondary storage. > > > > > > > > I'll need to think of another way to provide DR, perhaps some > automatic > > > > snapshot+download of sorts. > > > > > > > > Lucian > > > > > > > > -- > > > > Sent from the Delta quadrant using Borg technology! > > > > > > > > Nux! > > > > www.nux.ro > > > > > > > > ----- Original Message ----- > > > > > From: "Syed Ahmed" <sah...@cloudops.com> > > > > > To: d...@cloudstack.apache.org > > > > > Cc: "Nux!" <n...@li.nux.ro>, "Cloudstack Users List" < > > > > users@cloudstack.apache.org> > > > > > Sent: Tuesday, 19 July, 2016 17:06:52 > > > > > Subject: Re: State of the S3 secondary storage > > > > > > > > > As Will put it, the implementation for using Object store as > > secondary > > > > > storage relies on using Staging NFS. You would also loose the > > > > functionality > > > > > to do differential snapshots. All your snapshots will basically be > > full > > > > > volume snapshots. So, if you have a lot of snapshots being taken at > > > > > scheduled intervals, you might end up with wasting a lot of space > on > > > your > > > > > S3. Not to mention all of this has to go through the NFS which can > > (an > > > in > > > > > our experience has) fill up preventing you from creating new VMs > > > (because > > > > > templates cannot be downloaded). However you do benefit from a > region > > > > wide > > > > > Image store. We've worked around the NFS problem by basically > having > > > > > multiple staging NFS stores. > > > > > > > > > > I have plans to make Object stores better integrated in Cloudstack. > > The > > > > > holy grail would be to avoid staging NFS completely (or atleast in > > the > > > > > upload path). But for now we have to live with the limitations. > > > > > > > > > > -Syed > > > > > > > > > > > > > > > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens < > > wstev...@cloudops.com> > > > > > wrote: > > > > > > > > > >> We are using Swift as secondary storage. It also offers > inter-zone > > > > >> secondary storage. We have had to do fixes to make it work > > correctly, > > > > but > > > > >> we have pushed those fixes upstream so they are available in 4.9 > > (and > > > > some > > > > >> fixes earlier than that). > > > > >> > > > > >> Like Wido said, the staging NFS is still needed which is a real > > > problem > > > > >> with these implementations. You end up copying the data many > times > > > and > > > > you > > > > >> will run into problems if you have lots of customers doing > scheduled > > > > >> snapshots at the same time because you can fill your NFS staging > > area > > > > >> unless you give it a lot of space to work with. > > > > >> > > > > >> The object storage integrations are working, but they are not > > designed > > > > very > > > > >> well at present. You have code for both S3 and Swift basically > just > > > > >> separated with conditionals and much of the logic is shared. This > > is > > > > very > > > > >> brittle because making changes in one integration can (and > > > historically > > > > >> has) broken the other implementation. This is a relatively big > > > problem > > > > >> because most people working with either Swift or S3 will only have > > the > > > > >> ability to test one of them, so they don't really have the ability > > to > > > > test > > > > >> if they have broken the other. It does not help that there is > Zero > > CI > > > > >> coverage run for either the Swift or S3 integrations due to the > > > > difficulty > > > > >> building and tearing down object storage environments on the fly. > > > > >> > > > > >> All in all, yes they should be working, but expect a bit of a > bumpy > > > > road. > > > > >> > > > > >> Hope that helps, > > > > >> > > > > >> *Will STEVENS* > > > > >> Lead Developer > > > > >> > > > > >> *CloudOps* *| *Cloud Solutions Experts > > > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > > > >> w cloudops.com *|* tw @CloudOps_ > > > > >> > > > > >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander < > > w...@widodh.nl> > > > > >> wrote: > > > > >> > > > > >> > > > > > >> > > Op 19 juli 2016 om 14:38 schreef Nux! <n...@li.nux.ro>: > > > > >> > > > > > > >> > > > > > > >> > > Hello, > > > > >> > > > > > > >> > > Could anyone summarise the state of S3 (and clones) for > > secondary > > > > >> > storage? > > > > >> > > I read at some point that it's the only secondary storage type > > > > >> supported > > > > >> > inter-zone, so it would appeal from a DR pov, but not sure how > > > mature > > > > it > > > > >> is > > > > >> > or if there are any serious gotchas involved. > > > > >> > > > > > > >> > > > > > >> > We have it running for our largest region (Amsterdam) with > Ceph's > > > > RADOS > > > > >> > Gateway as backend. We had some issues, but they seem to be > fixed > > > > after > > > > >> our > > > > >> > latest patches. > > > > >> > > > > > >> > You still need the staging NFS though, so that doesn't solve > > > anything. > > > > >> > > > > > >> > Wido > > > > >> > > > > > >> > > Thanks > > > > >> > > Lucian > > > > >> > > > > > > >> > > -- > > > > >> > > Sent from the Delta quadrant using Borg technology! > > > > >> > > > > > > >> > > Nux! > > > > >> > > www.nux.ro > > > > >> > > > > > > > > > > >
