You understand the logic correctly. Yes, it will favor mirrors which are closest to the hosting of mirror302. However, if there is a problem with a mirror being overloaded, another mirror will win the race and serve the resource.
I originally started writing it to pick the closest mirrors geographically and then verify with head that the selected mirror has the requested resource. With this approach I was getting the feeling that I was over complicating the problem. How do I verify if the closest mirror is having problems. Let's say it is over loaded or has some other problem and takes twice as long to deliver the resource as a different mirror. How would I be able to verify this and be sure the resource can be served in a timely fashion? The geo way is very deterministic about which mirror is chosen. That is good for most cases, but if a mirror is having a hard time but is still responding to head, there does not seem to be a good way for the impacted client to delivered a different mirror. I released the code with this simplified implementation for a few reasons. 1) it is easier to understand and is more likely to handle the case that a mirror is struggling and not direct clients to it. 2) verify with the community that this approach to solving the problem is acceptable before spending a lot of time to 'perfect' it. 3) get feedback on the issues around geo selection before trying to do a final implementation. 4) have something to show in order to open a ticket with ASF infra to start working through the process of getting this in place. One other note around where the mirror list is hosted. Maybe we can serve the mirror list from cloudstack-www. We already have automatic publishing of the static resources on commit to the ASF. Is there a reason we need anything more complicated than that? Anyway, would love feedback all around. I am happy to adapt this as we see fit, but want to make sure we are generally happy with the approach before I invest a bunch of time into it. Thanks for the review. :) On Mar 3, 2017 1:24 AM, "Daan Hoogland" <daan.hoogl...@shapeblue.com> wrote: > Nice little thing Will, > > One question: if I read the code correctly it ‘go’es and tries all mirrors > at once and whichever responses first is redirected to the client. This > might well be the same every time. This might be the one closest to the > mirror302 and not to the client. Is that correct or did I miss a weighing > algorithm hidden in there? > > Good coding, > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote: > > Hey All, > Please review this repo and tell me what you think: > https://github.com/swill/mirror302 > > Let me know if you have questions or you would like me to make changes > before I bring this topic up with ASF Infra and open a ticket to get > this > implemented. > > Cheers, > > *Will Stevens* > > > > On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <wstev...@cloudops.com> > wrote: > > > I am building a short term solution right now. I hopefully will > have the > > building blocks in place by the weekend so we can start working with > Infra > > to get it in place. > > > > We will have a web server which we can point something like > > downloads.cloudstack.org (or whatever url) at. Then we will have > an ASF > > repo which tracks a mirror list and exposes a static site to be able > to > > query the mirror list. > > > > The web server will receive a request and will do a 302 redirect to > the > > appropriate resource on one of the mirrors. > > > > This gets us started. Legacy environments will have to do a DB > change to > > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or > > whatever), but otherwise it should be pretty simple. > > > > We can improve how we deliver templates going forward, but this > obviously > > requires some discussion still. I will try to buy us some time with > an > > implementation which solves our problems today... > > > > *Will STEVENS* > > Lead Developer > > > > <https://goo.gl/NYZ8KK> > > > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek < > > abhinandan.prat...@shapeblue.com> wrote: > > > >> > >> Initial seeding is a manual step and that is not going to change in > near > >> future. A handy list of official places from where these templates > can be > >> downloaded won’t harm this part of the installation. Just allow > anyone > >> credible who follows democratic process to publish their templates > with > >> apache’s blessings. > >> > >> > >> Coming to urls hard coded in db, the popular suggestion is to point > them > >> to mirrors. This looks good but this is going to take some effort > that has > >> not materialised yet. Once this change is made it will be going in > some > >> future release like 4.11; the problem still remains for the > releases that > >> are already out there. In case access to download.cloud.com is > dropped > >> at some point we will have to document the procedure such that > people know > >> how to make these stable releases work. That is where the procedure > >> documented here https://shankerbalan.net/blog/ > >> seed-cloudstack-templates-offline/ should be officially adapted > with a > >> marketplace to pick up templates of choosing. Maybe we can have a > pre-setup > >> script to make choices instead of written wiki. Note that this will > be > >> anyway needed by the older releases. > >> > >> > >> > >> > >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com> > wrote: > >> > >> >-1 > >> > > >> >We are trying to get MORE people to use CloudStack. And raising > the > >> barrier to entry does nothing to help that. > >> > > >> > > >> >Kind regards, > >> > > >> >Paul Angus > >> > > >> >paul.an...@shapeblue.com > >> >www.shapeblue.com > >> >53 Chandos Place, Covent Garden, London WC2N 4HSUK > >> >@shapeblue > >> > > >> > > >> > > >> > > >> >-----Original Message----- > >> >From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com] > >> >Sent: 01 March 2017 09:39 > >> >To: dev@cloudstack.apache.org > >> >Subject: Re: Modern template hosting > >> > > >> >+1 > >> > > >> >After seeing this discussion running in circles several times, I > think > >> we should at least get started with the simplest option. > >> > > >> > > >> > > >> > > >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com> > wrote: > >> > > >> >>The alternative foolproof way is to simply not provide the > automatic > >> >>download. > >> >> > >> >>Just document the mirror list URL where the templates can be > found. I > >> >>don't know why people reject this option. > >> >> > >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/ > >> >> > >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens > >> >><williamstev...@gmail.com> > >> >>wrote: > >> >> > >> >>> OK. Thanks for the heads up. > >> >>> > >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" < > chirade...@gmail.com> > >> wrote: > >> >>> > >> >>> > Sounds workable. The downloader code in the SSVM won't follow > >> >>> > redirects I think. > >> >>> > https://github.com/apache/cloudstack/blob/ > 5511065fc20787619d9cd0444 > >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/ > >> >>> > HttpTemplateDownloader.java#L93 > >> >>> > https://goo.gl/dSi0r5 > >> >>> > Might need to add > >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy()); > >> >>> > > >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens > >> >>> > <wstev...@cloudops.com> > >> >>> > wrote: > >> >>> > > >> >>> > > We haven't opened a ticket yet because we don't have a > strategy > >> yet. > >> >>> > > > >> >>> > > What do you guys think of this: > >> >>> > > - We setup a new github repo in the 'apache' org which > consists > >> >>> > > of a > >> >>> > single > >> >>> > > file with a list of active/supported mirrors. > >> >>> > > - I write a small web server, distributed as a binary, > which can > >> >>> > > be > >> >>> > hosted > >> >>> > > by ASF Infra. This web server will query the current list > of > >> >>> > > mirrors > >> >>> and > >> >>> > > will select one and then do a 302 redirect to that mirror. > >> >>> > > > >> >>> > > The act of 'choosing' a mirror could be done in a number of > ways. > >> >>> > > - If we want to define an order, then it could just try > from the > >> >>> > > top of > >> >>> > the > >> >>> > > list and work its way down. It would curl the target to > make > >> >>> > > sure it > >> >>> > gets > >> >>> > > a 200 and if it does, it would do a 302 redirect. > >> >>> > > - Or, if we want to distribute the load across the mirrors, > we > >> >>> > > could > >> >>> pick > >> >>> > > from the list randomly. Again, doing a curl to verify the > mirror > >> >>> > > is up > >> >>> > and > >> >>> > > then doing a redirect. > >> >>> > > - If we want to get fancy, we could do a reverse IP lookup > and > >> >>> > > try to > >> >>> > match > >> >>> > > the requester with their closest geographical mirror. > >> >>> > > > >> >>> > > Thoughts? > >> >>> > > > >> >>> > > *Will STEVENS* > >> >>> > > Lead Developer > >> >>> > > > >> >>> > > <https://goo.gl/NYZ8KK> > >> >>> > > > >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal < > >> >>> chirade...@gmail.com > >> >>> > > > >> >>> > > wrote: > >> >>> > > > >> >>> > > > What steps are needed to set up a mirror? What does Infra > need > >> to do? > >> >>> > Has > >> >>> > > > anybody filed a ticket with Infra? > >> >>> > > > > >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela < > >> >>> > > > raja.pull...@accelerite.com> > >> >>> > > > wrote: > >> >>> > > > > >> >>> > > > > Hi will, > >> >>> > > > > > >> >>> > > > > I believe, we didn’t get to close ‘getting a mirror on > Apache’ > >> >>> > because > >> >>> > > we > >> >>> > > > > needed someone on the Apache Infra side to close this. > BTW, > >> >>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has > >> >>> > > > > all/most of > >> >>> > the > >> >>> > > > > content. Once we can close on the Apache mirror for > hosting > >> >>> > > > > the > >> >>> > > > content, I > >> >>> > > > > can help assist getting the content there. > >> >>> > > > > > >> >>> > > > > For now, we have replicated the download.cloud.com > content to > >> >>> > > > > ‘ s3.download.accelerite.com’. > >> >>> > > > > Also, we are working on a set of steps/procedure to > help with > >> >>> > > > > this change. I will update everyone in about a week’s > time > >> >>> > > > > on the > >> >>> > details. > >> >>> > > > > > >> >>> > > > > Best, > >> >>> > > > > Raja Pullela > >> >>> > > > > Engineering Team, > >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa Clara, CA, 95054 > >> >>> > > > > > >> >>> > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on > behalf of > >> >>> > > > > Will Stevens" <williamstev...@gmail.com on behalf of > >> >>> > wstev...@cloudops.com> > >> >>> > > > > wrote: > >> >>> > > > > > >> >>> > > > > unfortunately the template mirror conversation got > caught up > >> >>> > > > > in > >> >>> > details > >> >>> > > > and > >> >>> > > > > nobody took the lead on implementing a solution. > >> >>> > > > > > >> >>> > > > > citrix has been pinging me every couple months to say > 'dude, > >> >>> > > > > we > >> >>> need > >> >>> > to > >> >>> > > > > remove the dependency on download.citrix.com', but i > have not > >> >>> > > > > had > >> >>> > the > >> >>> > > > > cycles to get in and solve the problem. the shutdown > of that > >> >>> > > > > is > >> >>> > > imminent > >> >>> > > > > right now, so we need to solve it asap. > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > *Will STEVENS* > >> >>> > > > > Lead Developer > >> >>> > > > > > >> >>> > > > > <https://goo.gl/NYZ8KK> > >> >>> > > > > > >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus < > >> >>> > paul.an...@shapeblue.com > >> >>> > > > > >> >>> > > > > wrote: > >> >>> > > > > > >> >>> > > > > > Hi Nathan, > >> >>> > > > > > > >> >>> > > > > > Ideally, if you put the template location in (or use a > >> >>> > > > > > template > >> >>> > > defined > >> >>> > > > > > in) test_data.py then the actual location can be > >> >>> > > > > > overridden by > >> >>> > > anyone > >> >>> > > > > > testing. > >> >>> > > > > > > >> >>> > > > > > For Trillian, we've copied all of the templates that > people > >> >>> > > > > > have > >> >>> > > define > >> >>> > > > > to > >> >>> > > > > > a local repo and then replace the URLs in > test_data.py to > >> >>> > > > > > reduce > >> >>> > > > > bandwidth > >> >>> > > > > > use and download times. > >> >>> > > > > > > >> >>> > > > > > Ie: > >> >>> > > > > > > >> >>> > > > > > "bootableIso": > >> >>> > > > > > { > >> >>> > > > > > "displaytext": "Test Bootable > ISO", > >> >>> > > > > > "name": "testISO", > >> >>> > > > > > "bootable": True, > >> >>> > > > > > "ispublic": False, > >> >>> > > > > > "url": "{{ marvin_images_location > >> >>> > > > > > }}/TinyCore-current.iso", > >> >>> > > > > > "ostype": 'Other Linux (64-bit)', > >> >>> > > > > > "mode": 'HTTP_DOWNLOAD' > >> >>> > > > > > }, > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > I thought that we had come up with a solution for > >> >>> > download.cloud.com > >> >>> > > , > >> >>> > > > by > >> >>> > > > > > having a mirrorlist hosted in Community Apache > 'space' with > >> >>> anyone > >> >>> > > able > >> >>> > > > > to > >> >>> > > > > > out themselves forward as a mirror. > >> >>> > > > > > But I must admit I lost track of whether anyone made > the > >> >>> requisite > >> >>> > > > > changes > >> >>> > > > > > in code.... > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > Kind regards, > >> >>> > > > > > > >> >>> > > > > > Paul Angus > >> >>> > > > > > > >> >>> > > > > > paul.an...@shapeblue.com > >> >>> > > > > > www.shapeblue.com > >> >>> > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > >> >>> > > > > > @shapeblue > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > -----Original Message----- > >> >>> > > > > > From: williamstev...@gmail.com > >> >>> > > > > > [mailto:williamstev...@gmail.com] > >> >>> > On > >> >>> > > > > > Behalf Of Will Stevens > >> >>> > > > > > Sent: 24 February 2017 16:30 > >> >>> > > > > > To: dev@cloudstack.apache.org > >> >>> > > > > > Subject: Re: Modern template hosting > >> >>> > > > > > > >> >>> > > > > > this is a hard questions. in general, we should be > setting > >> >>> > > > > > up a > >> >>> > > mirror > >> >>> > > > > on > >> >>> > > > > > some cloudstack/apache domain and then mirror to other > >> >>> > > > > > provided > >> >>> > > > > templates. > >> >>> > > > > > > >> >>> > > > > > we MUST come up with a solution to deprecate > >> >>> > > > > > 'download.cloud.com > >> >>> ', > >> >>> > > > that > >> >>> > > > > > is going to be going away any day now. > >> >>> > > > > > > >> >>> > > > > > i don't know the right way to solve this to be > honest, but > >> >>> > > > > > if you > >> >>> > > have > >> >>> > > > > > ideas, i am willing to help. > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > *Will STEVENS* > >> >>> > > > > > Lead Developer > >> >>> > > > > > > >> >>> > > > > > <https://goo.gl/NYZ8KK> > >> >>> > > > > > > >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson < > >> >>> njohn...@ena.com > >> >>> > > > >> >>> > > > > wrote: > >> >>> > > > > > > >> >>> > > > > > > So not to re-open a can of worms, but I’m in a > situation > >> >>> > > > > > > where > >> >>> I > >> >>> > > need > >> >>> > > > > > > to come up with a Marvin component test that > depends on a > >> >>> > template > >> >>> > > > > > > based on a kernel that’s relatively new, i.e., > newer than > >> >>> Centos > >> >>> > > 5.3 > >> >>> > > > / > >> >>> > > > > > Ubuntu 10.04 . > >> >>> > > > > > > I see openvm.eu has a suitable template (Ubuntu > 16.0.4 > >> >>> > > > > > > for > >> >>> KVM), > >> >>> > > but > >> >>> > > > > > > from looking at the thread "Migrating CloudStack > content > >> >>> > > > > > > from download.cloud.com” it looks like there is > >> >>> > > > > > > resistance to using > >> >>> > > this > >> >>> > > > at > >> >>> > > > > > > least for hosting system vm templates over concerns > of > >> >>> > neutrality. > >> >>> > > > > > > Would this be suitable for a component test? If > not, > >> >>> > > > > > > what is a > >> >>> > > > > “blessed” > >> >>> > > > > > > template location? > >> >>> > > > > > > > >> >>> > > > > > > Thanks in advance! > >> >>> > > > > > > > >> >>> > > > > > > Nathan Johnson > >> >>> > > > > > > R&D Engineer > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 Nashville, TN > 37211 > >> >>> > > > > > > General Office: 615-312-6000 > >> >>> > > > > > > > >> >>> > > > > > > website | blog | support > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > DISCLAIMER > >> >>> > > > > ========== > >> >>> > > > > This e-mail may contain privileged and confidential > >> >>> > > > > information > >> >>> which > >> >>> > > is > >> >>> > > > > the property of Accelerite, a Persistent Systems > business. It > >> >>> > > > > is > >> >>> > > intended > >> >>> > > > > only for the use of the individual or entity to which > it is > >> >>> > addressed. > >> >>> > > If > >> >>> > > > > you are not the intended recipient, you are not > authorized to > >> >>> > > > > read, > >> >>> > > > retain, > >> >>> > > > > copy, print, distribute or use this message. If you have > >> >>> > > > > received > >> >>> > this > >> >>> > > > > communication in error, please notify the sender and > delete > >> >>> > > > > all > >> >>> > copies > >> >>> > > of > >> >>> > > > > this message. Accelerite, a Persistent Systems business > does > >> >>> > > > > not > >> >>> > accept > >> >>> > > > any > >> >>> > > > > liability for virus infected mails. > >> >>> > > > > > >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > >> > > >> >abhinandan.prat...@shapeblue.com > >> >www.shapeblue.com > >> >53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > >> > > >> > > >> > > >> > >> abhinandan.prat...@shapeblue.com > >> www.shapeblue.com > >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK > >> @shapeblue > >> > >> > >> > >> > > > > > > daan.hoogl...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > >
