Hey All, Please review this repo and tell me what you think: https://github.com/swill/mirror302
Let me know if you have questions or you would like me to make changes before I bring this topic up with ASF Infra and open a ticket to get this implemented. Cheers, *Will Stevens* On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <wstev...@cloudops.com> wrote: > I am building a short term solution right now. I hopefully will have the > building blocks in place by the weekend so we can start working with Infra > to get it in place. > > We will have a web server which we can point something like > downloads.cloudstack.org (or whatever url) at. Then we will have an ASF > repo which tracks a mirror list and exposes a static site to be able to > query the mirror list. > > The web server will receive a request and will do a 302 redirect to the > appropriate resource on one of the mirrors. > > This gets us started. Legacy environments will have to do a DB change to > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or > whatever), but otherwise it should be pretty simple. > > We can improve how we deliver templates going forward, but this obviously > requires some discussion still. I will try to buy us some time with an > implementation which solves our problems today... > > *Will STEVENS* > Lead Developer > > <https://goo.gl/NYZ8KK> > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek < > abhinandan.prat...@shapeblue.com> wrote: > >> >> Initial seeding is a manual step and that is not going to change in near >> future. A handy list of official places from where these templates can be >> downloaded won’t harm this part of the installation. Just allow anyone >> credible who follows democratic process to publish their templates with >> apache’s blessings. >> >> >> Coming to urls hard coded in db, the popular suggestion is to point them >> to mirrors. This looks good but this is going to take some effort that has >> not materialised yet. Once this change is made it will be going in some >> future release like 4.11; the problem still remains for the releases that >> are already out there. In case access to download.cloud.com is dropped >> at some point we will have to document the procedure such that people know >> how to make these stable releases work. That is where the procedure >> documented here https://shankerbalan.net/blog/ >> seed-cloudstack-templates-offline/ should be officially adapted with a >> marketplace to pick up templates of choosing. Maybe we can have a pre-setup >> script to make choices instead of written wiki. Note that this will be >> anyway needed by the older releases. >> >> >> >> >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote: >> >> >-1 >> > >> >We are trying to get MORE people to use CloudStack. And raising the >> barrier to entry does nothing to help that. >> > >> > >> >Kind regards, >> > >> >Paul Angus >> > >> >paul.an...@shapeblue.com >> >www.shapeblue.com >> >53 Chandos Place, Covent Garden, London WC2N 4HSUK >> >@shapeblue >> > >> > >> > >> > >> >-----Original Message----- >> >From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com] >> >Sent: 01 March 2017 09:39 >> >To: dev@cloudstack.apache.org >> >Subject: Re: Modern template hosting >> > >> >+1 >> > >> >After seeing this discussion running in circles several times, I think >> we should at least get started with the simplest option. >> > >> > >> > >> > >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote: >> > >> >>The alternative foolproof way is to simply not provide the automatic >> >>download. >> >> >> >>Just document the mirror list URL where the templates can be found. I >> >>don't know why people reject this option. >> >> >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/ >> >> >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens >> >><williamstev...@gmail.com> >> >>wrote: >> >> >> >>> OK. Thanks for the heads up. >> >>> >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chirade...@gmail.com> >> wrote: >> >>> >> >>> > Sounds workable. The downloader code in the SSVM won't follow >> >>> > redirects I think. >> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444 >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/ >> >>> > HttpTemplateDownloader.java#L93 >> >>> > https://goo.gl/dSi0r5 >> >>> > Might need to add >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy()); >> >>> > >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens >> >>> > <wstev...@cloudops.com> >> >>> > wrote: >> >>> > >> >>> > > We haven't opened a ticket yet because we don't have a strategy >> yet. >> >>> > > >> >>> > > What do you guys think of this: >> >>> > > - We setup a new github repo in the 'apache' org which consists >> >>> > > of a >> >>> > single >> >>> > > file with a list of active/supported mirrors. >> >>> > > - I write a small web server, distributed as a binary, which can >> >>> > > be >> >>> > hosted >> >>> > > by ASF Infra. This web server will query the current list of >> >>> > > mirrors >> >>> and >> >>> > > will select one and then do a 302 redirect to that mirror. >> >>> > > >> >>> > > The act of 'choosing' a mirror could be done in a number of ways. >> >>> > > - If we want to define an order, then it could just try from the >> >>> > > top of >> >>> > the >> >>> > > list and work its way down. It would curl the target to make >> >>> > > sure it >> >>> > gets >> >>> > > a 200 and if it does, it would do a 302 redirect. >> >>> > > - Or, if we want to distribute the load across the mirrors, we >> >>> > > could >> >>> pick >> >>> > > from the list randomly. Again, doing a curl to verify the mirror >> >>> > > is up >> >>> > and >> >>> > > then doing a redirect. >> >>> > > - If we want to get fancy, we could do a reverse IP lookup and >> >>> > > try to >> >>> > match >> >>> > > the requester with their closest geographical mirror. >> >>> > > >> >>> > > Thoughts? >> >>> > > >> >>> > > *Will STEVENS* >> >>> > > Lead Developer >> >>> > > >> >>> > > <https://goo.gl/NYZ8KK> >> >>> > > >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal < >> >>> chirade...@gmail.com >> >>> > > >> >>> > > wrote: >> >>> > > >> >>> > > > What steps are needed to set up a mirror? What does Infra need >> to do? >> >>> > Has >> >>> > > > anybody filed a ticket with Infra? >> >>> > > > >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela < >> >>> > > > raja.pull...@accelerite.com> >> >>> > > > wrote: >> >>> > > > >> >>> > > > > Hi will, >> >>> > > > > >> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’ >> >>> > because >> >>> > > we >> >>> > > > > needed someone on the Apache Infra side to close this. BTW, >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has >> >>> > > > > all/most of >> >>> > the >> >>> > > > > content. Once we can close on the Apache mirror for hosting >> >>> > > > > the >> >>> > > > content, I >> >>> > > > > can help assist getting the content there. >> >>> > > > > >> >>> > > > > For now, we have replicated the download.cloud.com content to >> >>> > > > > ‘ s3.download.accelerite.com’. >> >>> > > > > Also, we are working on a set of steps/procedure to help with >> >>> > > > > this change. I will update everyone in about a week’s time >> >>> > > > > on the >> >>> > details. >> >>> > > > > >> >>> > > > > Best, >> >>> > > > > Raja Pullela >> >>> > > > > Engineering Team, >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054 >> >>> > > > > >> >>> > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf of >> >>> > > > > Will Stevens" <williamstev...@gmail.com on behalf of >> >>> > wstev...@cloudops.com> >> >>> > > > > wrote: >> >>> > > > > >> >>> > > > > unfortunately the template mirror conversation got caught up >> >>> > > > > in >> >>> > details >> >>> > > > and >> >>> > > > > nobody took the lead on implementing a solution. >> >>> > > > > >> >>> > > > > citrix has been pinging me every couple months to say 'dude, >> >>> > > > > we >> >>> need >> >>> > to >> >>> > > > > remove the dependency on download.citrix.com', but i have not >> >>> > > > > had >> >>> > the >> >>> > > > > cycles to get in and solve the problem. the shutdown of that >> >>> > > > > is >> >>> > > imminent >> >>> > > > > right now, so we need to solve it asap. >> >>> > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > *Will STEVENS* >> >>> > > > > Lead Developer >> >>> > > > > >> >>> > > > > <https://goo.gl/NYZ8KK> >> >>> > > > > >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus < >> >>> > paul.an...@shapeblue.com >> >>> > > > >> >>> > > > > wrote: >> >>> > > > > >> >>> > > > > > Hi Nathan, >> >>> > > > > > >> >>> > > > > > Ideally, if you put the template location in (or use a >> >>> > > > > > template >> >>> > > defined >> >>> > > > > > in) test_data.py then the actual location can be >> >>> > > > > > overridden by >> >>> > > anyone >> >>> > > > > > testing. >> >>> > > > > > >> >>> > > > > > For Trillian, we've copied all of the templates that people >> >>> > > > > > have >> >>> > > define >> >>> > > > > to >> >>> > > > > > a local repo and then replace the URLs in test_data.py to >> >>> > > > > > reduce >> >>> > > > > bandwidth >> >>> > > > > > use and download times. >> >>> > > > > > >> >>> > > > > > Ie: >> >>> > > > > > >> >>> > > > > > "bootableIso": >> >>> > > > > > { >> >>> > > > > > "displaytext": "Test Bootable ISO", >> >>> > > > > > "name": "testISO", >> >>> > > > > > "bootable": True, >> >>> > > > > > "ispublic": False, >> >>> > > > > > "url": "{{ marvin_images_location >> >>> > > > > > }}/TinyCore-current.iso", >> >>> > > > > > "ostype": 'Other Linux (64-bit)', >> >>> > > > > > "mode": 'HTTP_DOWNLOAD' >> >>> > > > > > }, >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > I thought that we had come up with a solution for >> >>> > download.cloud.com >> >>> > > , >> >>> > > > by >> >>> > > > > > having a mirrorlist hosted in Community Apache 'space' with >> >>> anyone >> >>> > > able >> >>> > > > > to >> >>> > > > > > out themselves forward as a mirror. >> >>> > > > > > But I must admit I lost track of whether anyone made the >> >>> requisite >> >>> > > > > changes >> >>> > > > > > in code.... >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > Kind regards, >> >>> > > > > > >> >>> > > > > > Paul Angus >> >>> > > > > > >> >>> > > > > > paul.an...@shapeblue.com >> >>> > > > > > www.shapeblue.com >> >>> > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK >> >>> > > > > > @shapeblue >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > -----Original Message----- >> >>> > > > > > From: williamstev...@gmail.com >> >>> > > > > > [mailto:williamstev...@gmail.com] >> >>> > On >> >>> > > > > > Behalf Of Will Stevens >> >>> > > > > > Sent: 24 February 2017 16:30 >> >>> > > > > > To: dev@cloudstack.apache.org >> >>> > > > > > Subject: Re: Modern template hosting >> >>> > > > > > >> >>> > > > > > this is a hard questions. in general, we should be setting >> >>> > > > > > up a >> >>> > > mirror >> >>> > > > > on >> >>> > > > > > some cloudstack/apache domain and then mirror to other >> >>> > > > > > provided >> >>> > > > > templates. >> >>> > > > > > >> >>> > > > > > we MUST come up with a solution to deprecate >> >>> > > > > > 'download.cloud.com >> >>> ', >> >>> > > > that >> >>> > > > > > is going to be going away any day now. >> >>> > > > > > >> >>> > > > > > i don't know the right way to solve this to be honest, but >> >>> > > > > > if you >> >>> > > have >> >>> > > > > > ideas, i am willing to help. >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > *Will STEVENS* >> >>> > > > > > Lead Developer >> >>> > > > > > >> >>> > > > > > <https://goo.gl/NYZ8KK> >> >>> > > > > > >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson < >> >>> njohn...@ena.com >> >>> > > >> >>> > > > > wrote: >> >>> > > > > > >> >>> > > > > > > So not to re-open a can of worms, but I’m in a situation >> >>> > > > > > > where >> >>> I >> >>> > > need >> >>> > > > > > > to come up with a Marvin component test that depends on a >> >>> > template >> >>> > > > > > > based on a kernel that’s relatively new, i.e., newer than >> >>> Centos >> >>> > > 5.3 >> >>> > > > / >> >>> > > > > > Ubuntu 10.04 . >> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 >> >>> > > > > > > for >> >>> KVM), >> >>> > > but >> >>> > > > > > > from looking at the thread "Migrating CloudStack content >> >>> > > > > > > from download.cloud.com” it looks like there is >> >>> > > > > > > resistance to using >> >>> > > this >> >>> > > > at >> >>> > > > > > > least for hosting system vm templates over concerns of >> >>> > neutrality. >> >>> > > > > > > Would this be suitable for a component test? If not, >> >>> > > > > > > what is a >> >>> > > > > “blessed” >> >>> > > > > > > template location? >> >>> > > > > > > >> >>> > > > > > > Thanks in advance! >> >>> > > > > > > >> >>> > > > > > > Nathan Johnson >> >>> > > > > > > R&D Engineer >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN 37211 >> >>> > > > > > > General Office: 615-312-6000 >> >>> > > > > > > >> >>> > > > > > > website | blog | support >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > DISCLAIMER >> >>> > > > > ========== >> >>> > > > > This e-mail may contain privileged and confidential >> >>> > > > > information >> >>> which >> >>> > > is >> >>> > > > > the property of Accelerite, a Persistent Systems business. It >> >>> > > > > is >> >>> > > intended >> >>> > > > > only for the use of the individual or entity to which it is >> >>> > addressed. >> >>> > > If >> >>> > > > > you are not the intended recipient, you are not authorized to >> >>> > > > > read, >> >>> > > > retain, >> >>> > > > > copy, print, distribute or use this message. If you have >> >>> > > > > received >> >>> > this >> >>> > > > > communication in error, please notify the sender and delete >> >>> > > > > all >> >>> > copies >> >>> > > of >> >>> > > > > this message. Accelerite, a Persistent Systems business does >> >>> > > > > not >> >>> > accept >> >>> > > > any >> >>> > > > > liability for virus infected mails. >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > >> >>> >> > >> >abhinandan.prat...@shapeblue.com >> >www.shapeblue.com >> >53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >> > >> > >> > >> >> abhinandan.prat...@shapeblue.com >> www.shapeblue.com >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK >> @shapeblue >> >> >> >> >
