I am building a short term solution right now. I hopefully will have the building blocks in place by the weekend so we can start working with Infra to get it in place.
We will have a web server which we can point something like downloads.cloudstack.org (or whatever url) at. Then we will have an ASF repo which tracks a mirror list and exposes a static site to be able to query the mirror list. The web server will receive a request and will do a 302 redirect to the appropriate resource on one of the mirrors. This gets us started. Legacy environments will have to do a DB change to change from 'download.cloud.com' to 'downloads.cloudstack.org' (or whatever), but otherwise it should be pretty simple. We can improve how we deliver templates going forward, but this obviously requires some discussion still. I will try to buy us some time with an implementation which solves our problems today... *Will STEVENS* Lead Developer <https://goo.gl/NYZ8KK> On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek < abhinandan.prat...@shapeblue.com> wrote: > > Initial seeding is a manual step and that is not going to change in near > future. A handy list of official places from where these templates can be > downloaded won’t harm this part of the installation. Just allow anyone > credible who follows democratic process to publish their templates with > apache’s blessings. > > > Coming to urls hard coded in db, the popular suggestion is to point them > to mirrors. This looks good but this is going to take some effort that has > not materialised yet. Once this change is made it will be going in some > future release like 4.11; the problem still remains for the releases that > are already out there. In case access to download.cloud.com is dropped at > some point we will have to document the procedure such that people know how > to make these stable releases work. That is where the procedure documented > here https://shankerbalan.net/blog/seed-cloudstack-templates-offline/ > should be officially adapted with a marketplace to pick up templates of > choosing. Maybe we can have a pre-setup script to make choices instead of > written wiki. Note that this will be anyway needed by the older releases. > > > > > On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote: > > >-1 > > > >We are trying to get MORE people to use CloudStack. And raising the > barrier to entry does nothing to help that. > > > > > >Kind regards, > > > >Paul Angus > > > >paul.an...@shapeblue.com > >www.shapeblue.com > >53 Chandos Place, Covent Garden, London WC2N 4HSUK > >@shapeblue > > > > > > > > > >-----Original Message----- > >From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com] > >Sent: 01 March 2017 09:39 > >To: dev@cloudstack.apache.org > >Subject: Re: Modern template hosting > > > >+1 > > > >After seeing this discussion running in circles several times, I think we > should at least get started with the simplest option. > > > > > > > > > >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote: > > > >>The alternative foolproof way is to simply not provide the automatic > >>download. > >> > >>Just document the mirror list URL where the templates can be found. I > >>don't know why people reject this option. > >> > >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/ > >> > >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens > >><williamstev...@gmail.com> > >>wrote: > >> > >>> OK. Thanks for the heads up. > >>> > >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chirade...@gmail.com> > wrote: > >>> > >>> > Sounds workable. The downloader code in the SSVM won't follow > >>> > redirects I think. > >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444 > >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/ > >>> > HttpTemplateDownloader.java#L93 > >>> > https://goo.gl/dSi0r5 > >>> > Might need to add > >>> > client.setRedirectStrategy(new LaxRedirectStrategy()); > >>> > > >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens > >>> > <wstev...@cloudops.com> > >>> > wrote: > >>> > > >>> > > We haven't opened a ticket yet because we don't have a strategy > yet. > >>> > > > >>> > > What do you guys think of this: > >>> > > - We setup a new github repo in the 'apache' org which consists > >>> > > of a > >>> > single > >>> > > file with a list of active/supported mirrors. > >>> > > - I write a small web server, distributed as a binary, which can > >>> > > be > >>> > hosted > >>> > > by ASF Infra. This web server will query the current list of > >>> > > mirrors > >>> and > >>> > > will select one and then do a 302 redirect to that mirror. > >>> > > > >>> > > The act of 'choosing' a mirror could be done in a number of ways. > >>> > > - If we want to define an order, then it could just try from the > >>> > > top of > >>> > the > >>> > > list and work its way down. It would curl the target to make > >>> > > sure it > >>> > gets > >>> > > a 200 and if it does, it would do a 302 redirect. > >>> > > - Or, if we want to distribute the load across the mirrors, we > >>> > > could > >>> pick > >>> > > from the list randomly. Again, doing a curl to verify the mirror > >>> > > is up > >>> > and > >>> > > then doing a redirect. > >>> > > - If we want to get fancy, we could do a reverse IP lookup and > >>> > > try to > >>> > match > >>> > > the requester with their closest geographical mirror. > >>> > > > >>> > > Thoughts? > >>> > > > >>> > > *Will STEVENS* > >>> > > Lead Developer > >>> > > > >>> > > <https://goo.gl/NYZ8KK> > >>> > > > >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal < > >>> chirade...@gmail.com > >>> > > > >>> > > wrote: > >>> > > > >>> > > > What steps are needed to set up a mirror? What does Infra need > to do? > >>> > Has > >>> > > > anybody filed a ticket with Infra? > >>> > > > > >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela < > >>> > > > raja.pull...@accelerite.com> > >>> > > > wrote: > >>> > > > > >>> > > > > Hi will, > >>> > > > > > >>> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’ > >>> > because > >>> > > we > >>> > > > > needed someone on the Apache Infra side to close this. BTW, > >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has > >>> > > > > all/most of > >>> > the > >>> > > > > content. Once we can close on the Apache mirror for hosting > >>> > > > > the > >>> > > > content, I > >>> > > > > can help assist getting the content there. > >>> > > > > > >>> > > > > For now, we have replicated the download.cloud.com content to > >>> > > > > ‘ s3.download.accelerite.com’. > >>> > > > > Also, we are working on a set of steps/procedure to help with > >>> > > > > this change. I will update everyone in about a week’s time > >>> > > > > on the > >>> > details. > >>> > > > > > >>> > > > > Best, > >>> > > > > Raja Pullela > >>> > > > > Engineering Team, > >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054 > >>> > > > > > >>> > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf of > >>> > > > > Will Stevens" <williamstev...@gmail.com on behalf of > >>> > wstev...@cloudops.com> > >>> > > > > wrote: > >>> > > > > > >>> > > > > unfortunately the template mirror conversation got caught up > >>> > > > > in > >>> > details > >>> > > > and > >>> > > > > nobody took the lead on implementing a solution. > >>> > > > > > >>> > > > > citrix has been pinging me every couple months to say 'dude, > >>> > > > > we > >>> need > >>> > to > >>> > > > > remove the dependency on download.citrix.com', but i have not > >>> > > > > had > >>> > the > >>> > > > > cycles to get in and solve the problem. the shutdown of that > >>> > > > > is > >>> > > imminent > >>> > > > > right now, so we need to solve it asap. > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > *Will STEVENS* > >>> > > > > Lead Developer > >>> > > > > > >>> > > > > <https://goo.gl/NYZ8KK> > >>> > > > > > >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus < > >>> > paul.an...@shapeblue.com > >>> > > > > >>> > > > > wrote: > >>> > > > > > >>> > > > > > Hi Nathan, > >>> > > > > > > >>> > > > > > Ideally, if you put the template location in (or use a > >>> > > > > > template > >>> > > defined > >>> > > > > > in) test_data.py then the actual location can be > >>> > > > > > overridden by > >>> > > anyone > >>> > > > > > testing. > >>> > > > > > > >>> > > > > > For Trillian, we've copied all of the templates that people > >>> > > > > > have > >>> > > define > >>> > > > > to > >>> > > > > > a local repo and then replace the URLs in test_data.py to > >>> > > > > > reduce > >>> > > > > bandwidth > >>> > > > > > use and download times. > >>> > > > > > > >>> > > > > > Ie: > >>> > > > > > > >>> > > > > > "bootableIso": > >>> > > > > > { > >>> > > > > > "displaytext": "Test Bootable ISO", > >>> > > > > > "name": "testISO", > >>> > > > > > "bootable": True, > >>> > > > > > "ispublic": False, > >>> > > > > > "url": "{{ marvin_images_location > >>> > > > > > }}/TinyCore-current.iso", > >>> > > > > > "ostype": 'Other Linux (64-bit)', > >>> > > > > > "mode": 'HTTP_DOWNLOAD' > >>> > > > > > }, > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > I thought that we had come up with a solution for > >>> > download.cloud.com > >>> > > , > >>> > > > by > >>> > > > > > having a mirrorlist hosted in Community Apache 'space' with > >>> anyone > >>> > > able > >>> > > > > to > >>> > > > > > out themselves forward as a mirror. > >>> > > > > > But I must admit I lost track of whether anyone made the > >>> requisite > >>> > > > > changes > >>> > > > > > in code.... > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > Kind regards, > >>> > > > > > > >>> > > > > > Paul Angus > >>> > > > > > > >>> > > > > > paul.an...@shapeblue.com > >>> > > > > > www.shapeblue.com > >>> > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > >>> > > > > > @shapeblue > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > -----Original Message----- > >>> > > > > > From: williamstev...@gmail.com > >>> > > > > > [mailto:williamstev...@gmail.com] > >>> > On > >>> > > > > > Behalf Of Will Stevens > >>> > > > > > Sent: 24 February 2017 16:30 > >>> > > > > > To: dev@cloudstack.apache.org > >>> > > > > > Subject: Re: Modern template hosting > >>> > > > > > > >>> > > > > > this is a hard questions. in general, we should be setting > >>> > > > > > up a > >>> > > mirror > >>> > > > > on > >>> > > > > > some cloudstack/apache domain and then mirror to other > >>> > > > > > provided > >>> > > > > templates. > >>> > > > > > > >>> > > > > > we MUST come up with a solution to deprecate > >>> > > > > > 'download.cloud.com > >>> ', > >>> > > > that > >>> > > > > > is going to be going away any day now. > >>> > > > > > > >>> > > > > > i don't know the right way to solve this to be honest, but > >>> > > > > > if you > >>> > > have > >>> > > > > > ideas, i am willing to help. > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > *Will STEVENS* > >>> > > > > > Lead Developer > >>> > > > > > > >>> > > > > > <https://goo.gl/NYZ8KK> > >>> > > > > > > >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson < > >>> njohn...@ena.com > >>> > > > >>> > > > > wrote: > >>> > > > > > > >>> > > > > > > So not to re-open a can of worms, but I’m in a situation > >>> > > > > > > where > >>> I > >>> > > need > >>> > > > > > > to come up with a Marvin component test that depends on a > >>> > template > >>> > > > > > > based on a kernel that’s relatively new, i.e., newer than > >>> Centos > >>> > > 5.3 > >>> > > > / > >>> > > > > > Ubuntu 10.04 . > >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 > >>> > > > > > > for > >>> KVM), > >>> > > but > >>> > > > > > > from looking at the thread "Migrating CloudStack content > >>> > > > > > > from download.cloud.com” it looks like there is > >>> > > > > > > resistance to using > >>> > > this > >>> > > > at > >>> > > > > > > least for hosting system vm templates over concerns of > >>> > neutrality. > >>> > > > > > > Would this be suitable for a component test? If not, > >>> > > > > > > what is a > >>> > > > > “blessed” > >>> > > > > > > template location? > >>> > > > > > > > >>> > > > > > > Thanks in advance! > >>> > > > > > > > >>> > > > > > > Nathan Johnson > >>> > > > > > > R&D Engineer > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN 37211 > >>> > > > > > > General Office: 615-312-6000 > >>> > > > > > > > >>> > > > > > > website | blog | support > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > DISCLAIMER > >>> > > > > ========== > >>> > > > > This e-mail may contain privileged and confidential > >>> > > > > information > >>> which > >>> > > is > >>> > > > > the property of Accelerite, a Persistent Systems business. It > >>> > > > > is > >>> > > intended > >>> > > > > only for the use of the individual or entity to which it is > >>> > addressed. > >>> > > If > >>> > > > > you are not the intended recipient, you are not authorized to > >>> > > > > read, > >>> > > > retain, > >>> > > > > copy, print, distribute or use this message. If you have > >>> > > > > received > >>> > this > >>> > > > > communication in error, please notify the sender and delete > >>> > > > > all > >>> > copies > >>> > > of > >>> > > > > this message. Accelerite, a Persistent Systems business does > >>> > > > > not > >>> > accept > >>> > > > any > >>> > > > > liability for virus infected mails. > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > > > >abhinandan.prat...@shapeblue.com > >www.shapeblue.com > >53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > > > abhinandan.prat...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > >
