Will great job. I had the same doubt as Daan.
Have you considered using HTTP 300 working mode? Then, we could let the client decide which mirror is the best(closest?). Section “10.3.1” of [1] talks about it; this would put the pressure on deciding upon a mirror on the client side, but I think it would be fair. [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html PS: You really like very short variables names! On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <daan.hoogl...@shapeblue.com> wrote: > Nice little thing Will, > > One question: if I read the code correctly it ‘go’es and tries all mirrors > at once and whichever responses first is redirected to the client. This > might well be the same every time. This might be the one closest to the > mirror302 and not to the client. Is that correct or did I miss a weighing > algorithm hidden in there? > > Good coding, > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote: > > Hey All, > Please review this repo and tell me what you think: > https://github.com/swill/mirror302 > > Let me know if you have questions or you would like me to make changes > before I bring this topic up with ASF Infra and open a ticket to get > this > implemented. > > Cheers, > > *Will Stevens* > > > > On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <wstev...@cloudops.com> > wrote: > > > I am building a short term solution right now. I hopefully will > have the > > building blocks in place by the weekend so we can start working with > Infra > > to get it in place. > > > > We will have a web server which we can point something like > > downloads.cloudstack.org (or whatever url) at. Then we will have > an ASF > > repo which tracks a mirror list and exposes a static site to be able > to > > query the mirror list. > > > > The web server will receive a request and will do a 302 redirect to > the > > appropriate resource on one of the mirrors. > > > > This gets us started. Legacy environments will have to do a DB > change to > > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or > > whatever), but otherwise it should be pretty simple. > > > > We can improve how we deliver templates going forward, but this > obviously > > requires some discussion still. I will try to buy us some time with > an > > implementation which solves our problems today... > > > > *Will STEVENS* > > Lead Developer > > > > <https://goo.gl/NYZ8KK> > > > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek < > > abhinandan.prat...@shapeblue.com> wrote: > > > >> > >> Initial seeding is a manual step and that is not going to change in > near > >> future. A handy list of official places from where these templates > can be > >> downloaded won’t harm this part of the installation. Just allow > anyone > >> credible who follows democratic process to publish their templates > with > >> apache’s blessings. > >> > >> > >> Coming to urls hard coded in db, the popular suggestion is to point > them > >> to mirrors. This looks good but this is going to take some effort > that has > >> not materialised yet. Once this change is made it will be going in > some > >> future release like 4.11; the problem still remains for the > releases that > >> are already out there. In case access to download.cloud.com is > dropped > >> at some point we will have to document the procedure such that > people know > >> how to make these stable releases work. That is where the procedure > >> documented here https://shankerbalan.net/blog/ > >> seed-cloudstack-templates-offline/ should be officially adapted > with a > >> marketplace to pick up templates of choosing. Maybe we can have a > pre-setup > >> script to make choices instead of written wiki. Note that this will > be > >> anyway needed by the older releases. > >> > >> > >> > >> > >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com> > wrote: > >> > >> >-1 > >> > > >> >We are trying to get MORE people to use CloudStack. And raising > the > >> barrier to entry does nothing to help that. > >> > > >> > > >> >Kind regards, > >> > > >> >Paul Angus > >> > > >> >paul.an...@shapeblue.com > >> >www.shapeblue.com > >> >53 Chandos Place, Covent Garden, London WC2N 4HSUK > >> >@shapeblue > >> > > >> > > >> > > >> > > >> >-----Original Message----- > >> >From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com] > >> >Sent: 01 March 2017 09:39 > >> >To: dev@cloudstack.apache.org > >> >Subject: Re: Modern template hosting > >> > > >> >+1 > >> > > >> >After seeing this discussion running in circles several times, I > think > >> we should at least get started with the simplest option. > >> > > >> > > >> > > >> > > >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com> > wrote: > >> > > >> >>The alternative foolproof way is to simply not provide the > automatic > >> >>download. > >> >> > >> >>Just document the mirror list URL where the templates can be > found. I > >> >>don't know why people reject this option. > >> >> > >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/ > >> >> > >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens > >> >><williamstev...@gmail.com> > >> >>wrote: > >> >> > >> >>> OK. Thanks for the heads up. > >> >>> > >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" < > chirade...@gmail.com> > >> wrote: > >> >>> > >> >>> > Sounds workable. The downloader code in the SSVM won't follow > >> >>> > redirects I think. > >> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d > 9cd0444 > >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/ > >> >>> > HttpTemplateDownloader.java#L93 > >> >>> > https://goo.gl/dSi0r5 > >> >>> > Might need to add > >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy()); > >> >>> > > >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens > >> >>> > <wstev...@cloudops.com> > >> >>> > wrote: > >> >>> > > >> >>> > > We haven't opened a ticket yet because we don't have a > strategy > >> yet. > >> >>> > > > >> >>> > > What do you guys think of this: > >> >>> > > - We setup a new github repo in the 'apache' org which > consists > >> >>> > > of a > >> >>> > single > >> >>> > > file with a list of active/supported mirrors. > >> >>> > > - I write a small web server, distributed as a binary, > which can > >> >>> > > be > >> >>> > hosted > >> >>> > > by ASF Infra. This web server will query the current list > of > >> >>> > > mirrors > >> >>> and > >> >>> > > will select one and then do a 302 redirect to that mirror. > >> >>> > > > >> >>> > > The act of 'choosing' a mirror could be done in a number of > ways. > >> >>> > > - If we want to define an order, then it could just try > from the > >> >>> > > top of > >> >>> > the > >> >>> > > list and work its way down. It would curl the target to > make > >> >>> > > sure it > >> >>> > gets > >> >>> > > a 200 and if it does, it would do a 302 redirect. > >> >>> > > - Or, if we want to distribute the load across the mirrors, > we > >> >>> > > could > >> >>> pick > >> >>> > > from the list randomly. Again, doing a curl to verify the > mirror > >> >>> > > is up > >> >>> > and > >> >>> > > then doing a redirect. > >> >>> > > - If we want to get fancy, we could do a reverse IP lookup > and > >> >>> > > try to > >> >>> > match > >> >>> > > the requester with their closest geographical mirror. > >> >>> > > > >> >>> > > Thoughts? > >> >>> > > > >> >>> > > *Will STEVENS* > >> >>> > > Lead Developer > >> >>> > > > >> >>> > > <https://goo.gl/NYZ8KK> > >> >>> > > > >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal < > >> >>> chirade...@gmail.com > >> >>> > > > >> >>> > > wrote: > >> >>> > > > >> >>> > > > What steps are needed to set up a mirror? What does Infra > need > >> to do? > >> >>> > Has > >> >>> > > > anybody filed a ticket with Infra? > >> >>> > > > > >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela < > >> >>> > > > raja.pull...@accelerite.com> > >> >>> > > > wrote: > >> >>> > > > > >> >>> > > > > Hi will, > >> >>> > > > > > >> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on > Apache’ > >> >>> > because > >> >>> > > we > >> >>> > > > > needed someone on the Apache Infra side to close this. > BTW, > >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has > >> >>> > > > > all/most of > >> >>> > the > >> >>> > > > > content. Once we can close on the Apache mirror for > hosting > >> >>> > > > > the > >> >>> > > > content, I > >> >>> > > > > can help assist getting the content there. > >> >>> > > > > > >> >>> > > > > For now, we have replicated the download.cloud.com > content to > >> >>> > > > > ‘ s3.download.accelerite.com’. > >> >>> > > > > Also, we are working on a set of steps/procedure to > help with > >> >>> > > > > this change. I will update everyone in about a week’s > time > >> >>> > > > > on the > >> >>> > details. > >> >>> > > > > > >> >>> > > > > Best, > >> >>> > > > > Raja Pullela > >> >>> > > > > Engineering Team, > >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054 > >> >>> > > > > > >> >>> > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on > behalf of > >> >>> > > > > Will Stevens" <williamstev...@gmail.com on behalf of > >> >>> > wstev...@cloudops.com> > >> >>> > > > > wrote: > >> >>> > > > > > >> >>> > > > > unfortunately the template mirror conversation got > caught up > >> >>> > > > > in > >> >>> > details > >> >>> > > > and > >> >>> > > > > nobody took the lead on implementing a solution. > >> >>> > > > > > >> >>> > > > > citrix has been pinging me every couple months to say > 'dude, > >> >>> > > > > we > >> >>> need > >> >>> > to > >> >>> > > > > remove the dependency on download.citrix.com', but i > have not > >> >>> > > > > had > >> >>> > the > >> >>> > > > > cycles to get in and solve the problem. the shutdown > of that > >> >>> > > > > is > >> >>> > > imminent > >> >>> > > > > right now, so we need to solve it asap. > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > *Will STEVENS* > >> >>> > > > > Lead Developer > >> >>> > > > > > >> >>> > > > > <https://goo.gl/NYZ8KK> > >> >>> > > > > > >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus < > >> >>> > paul.an...@shapeblue.com > >> >>> > > > > >> >>> > > > > wrote: > >> >>> > > > > > >> >>> > > > > > Hi Nathan, > >> >>> > > > > > > >> >>> > > > > > Ideally, if you put the template location in (or use a > >> >>> > > > > > template > >> >>> > > defined > >> >>> > > > > > in) test_data.py then the actual location can be > >> >>> > > > > > overridden by > >> >>> > > anyone > >> >>> > > > > > testing. > >> >>> > > > > > > >> >>> > > > > > For Trillian, we've copied all of the templates that > people > >> >>> > > > > > have > >> >>> > > define > >> >>> > > > > to > >> >>> > > > > > a local repo and then replace the URLs in > test_data.py to > >> >>> > > > > > reduce > >> >>> > > > > bandwidth > >> >>> > > > > > use and download times. > >> >>> > > > > > > >> >>> > > > > > Ie: > >> >>> > > > > > > >> >>> > > > > > "bootableIso": > >> >>> > > > > > { > >> >>> > > > > > "displaytext": "Test Bootable > ISO", > >> >>> > > > > > "name": "testISO", > >> >>> > > > > > "bootable": True, > >> >>> > > > > > "ispublic": False, > >> >>> > > > > > "url": "{{ marvin_images_location > >> >>> > > > > > }}/TinyCore-current.iso", > >> >>> > > > > > "ostype": 'Other Linux (64-bit)', > >> >>> > > > > > "mode": 'HTTP_DOWNLOAD' > >> >>> > > > > > }, > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > I thought that we had come up with a solution for > >> >>> > download.cloud.com > >> >>> > > , > >> >>> > > > by > >> >>> > > > > > having a mirrorlist hosted in Community Apache > 'space' with > >> >>> anyone > >> >>> > > able > >> >>> > > > > to > >> >>> > > > > > out themselves forward as a mirror. > >> >>> > > > > > But I must admit I lost track of whether anyone made > the > >> >>> requisite > >> >>> > > > > changes > >> >>> > > > > > in code.... > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > Kind regards, > >> >>> > > > > > > >> >>> > > > > > Paul Angus > >> >>> > > > > > > >> >>> > > > > > paul.an...@shapeblue.com > >> >>> > > > > > www.shapeblue.com > >> >>> > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > >> >>> > > > > > @shapeblue > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > -----Original Message----- > >> >>> > > > > > From: williamstev...@gmail.com > >> >>> > > > > > [mailto:williamstev...@gmail.com] > >> >>> > On > >> >>> > > > > > Behalf Of Will Stevens > >> >>> > > > > > Sent: 24 February 2017 16:30 > >> >>> > > > > > To: dev@cloudstack.apache.org > >> >>> > > > > > Subject: Re: Modern template hosting > >> >>> > > > > > > >> >>> > > > > > this is a hard questions. in general, we should be > setting > >> >>> > > > > > up a > >> >>> > > mirror > >> >>> > > > > on > >> >>> > > > > > some cloudstack/apache domain and then mirror to other > >> >>> > > > > > provided > >> >>> > > > > templates. > >> >>> > > > > > > >> >>> > > > > > we MUST come up with a solution to deprecate > >> >>> > > > > > 'download.cloud.com > >> >>> ', > >> >>> > > > that > >> >>> > > > > > is going to be going away any day now. > >> >>> > > > > > > >> >>> > > > > > i don't know the right way to solve this to be > honest, but > >> >>> > > > > > if you > >> >>> > > have > >> >>> > > > > > ideas, i am willing to help. > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > *Will STEVENS* > >> >>> > > > > > Lead Developer > >> >>> > > > > > > >> >>> > > > > > <https://goo.gl/NYZ8KK> > >> >>> > > > > > > >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson < > >> >>> njohn...@ena.com > >> >>> > > > >> >>> > > > > wrote: > >> >>> > > > > > > >> >>> > > > > > > So not to re-open a can of worms, but I’m in a > situation > >> >>> > > > > > > where > >> >>> I > >> >>> > > need > >> >>> > > > > > > to come up with a Marvin component test that > depends on a > >> >>> > template > >> >>> > > > > > > based on a kernel that’s relatively new, i.e., > newer than > >> >>> Centos > >> >>> > > 5.3 > >> >>> > > > / > >> >>> > > > > > Ubuntu 10.04 . > >> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu > 16.0.4 > >> >>> > > > > > > for > >> >>> KVM), > >> >>> > > but > >> >>> > > > > > > from looking at the thread "Migrating CloudStack > content > >> >>> > > > > > > from download.cloud.com” it looks like there is > >> >>> > > > > > > resistance to using > >> >>> > > this > >> >>> > > > at > >> >>> > > > > > > least for hosting system vm templates over concerns > of > >> >>> > neutrality. > >> >>> > > > > > > Would this be suitable for a component test? If > not, > >> >>> > > > > > > what is a > >> >>> > > > > “blessed” > >> >>> > > > > > > template location? > >> >>> > > > > > > > >> >>> > > > > > > Thanks in advance! > >> >>> > > > > > > > >> >>> > > > > > > Nathan Johnson > >> >>> > > > > > > R&D Engineer > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN > 37211 > >> >>> > > > > > > General Office: 615-312-6000 > >> >>> > > > > > > > >> >>> > > > > > > website | blog | support > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > DISCLAIMER > >> >>> > > > > ========== > >> >>> > > > > This e-mail may contain privileged and confidential > >> >>> > > > > information > >> >>> which > >> >>> > > is > >> >>> > > > > the property of Accelerite, a Persistent Systems > business. It > >> >>> > > > > is > >> >>> > > intended > >> >>> > > > > only for the use of the individual or entity to which > it is > >> >>> > addressed. > >> >>> > > If > >> >>> > > > > you are not the intended recipient, you are not > authorized to > >> >>> > > > > read, > >> >>> > > > retain, > >> >>> > > > > copy, print, distribute or use this message. If you have > >> >>> > > > > received > >> >>> > this > >> >>> > > > > communication in error, please notify the sender and > delete > >> >>> > > > > all > >> >>> > copies > >> >>> > > of > >> >>> > > > > this message. Accelerite, a Persistent Systems business > does > >> >>> > > > > not > >> >>> > accept > >> >>> > > > any > >> >>> > > > > liability for virus infected mails. > >> >>> > > > > > >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > >> > > >> >abhinandan.prat...@shapeblue.com > >> >www.shapeblue.com > >> >53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > >> > > >> > > >> > > >> > >> abhinandan.prat...@shapeblue.com > >> www.shapeblue.com > >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK > >> @shapeblue > >> > >> > >> > >> > > > > > > daan.hoogl...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > -- Rafael Weingärtner
