Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
Hi Ken, Op 18-10-2013 22:05, Ken Snider schreef: The Wikimedia Foundation's Technical Operations team is seeking proposals on the provisioning of a new data-centre facility. After working through the specifics internally, we now have a public RFP posted[1] and ready for proposals. We invite any organization meeting the requirements outlined to submit a proposal for review. You have stated some technical requirements, but not the availability you would like to have. You probably want to include that you're looking for a tier-4 data center (https://en.wikipedia.org/wiki/Tier_4_data_center#Data_center_tiers). Is this one going to replace the Florida data center? Where are you keeping documentation these days? The information on wikitech seems to be very incomplete and outdated. Something related: While travelling in China I noticed the bad performance of our sites. Would it be a good idea to investigate this (lack of) performance and maybe consider a caching site somewhere in Asia? The latency should be much better than getting is all the way from the USA. http://www.glif.is/publications/maps/GLIF_5-11_World_4k.jpg (from http://www.glif.is/publications/maps/) gives a good idea of connectivity by the way. Maarten ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
On Sat, Oct 19, 2013 at 1:17 PM, Maarten Dammers maar...@mdammers.nl wrote: Hi Ken, Op 18-10-2013 22:05, Ken Snider schreef: The Wikimedia Foundation's Technical Operations team is seeking proposals on the provisioning of a new data-centre facility. After working through the specifics internally, we now have a public RFP posted[1] and ready for proposals. We invite any organization meeting the requirements outlined to submit a proposal for review. You have stated some technical requirements, but not the availability you would like to have. You probably want to include that you're looking for a tier-4 data center (https://en.wikipedia.org/wiki/Tier_4_data_center#Data_center_tiers). Is this one going to replace the Florida data center? Where are you keeping documentation these days? The information on wikitech seems to be very incomplete and outdated. Wikitech is our best source of documentation Something related: While travelling in China I noticed the bad performance of our sites. Would it be a good idea to investigate this (lack of) performance and maybe consider a caching site somewhere in Asia? The latency should be much better than getting is all the way from the USA. http://www.glif.is/publications/maps/GLIF_5-11_World_4k.jpg (from http://www.glif.is/publications/maps/) gives a good idea of connectivity by the way. (Employee hat) - we are starting to move some of the Asian traffic to a new caching center on the US west coast, which has helped latency. (Personal hat) - I would love to get an Asian caching center (Hong Kong or Tokyo would be my top 2 choices), but IMHO the biggest barrier to this the time and availability of people resources. Maarten ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
Hoi, Are some of our partners in Wikipedia Zero not caching already ? Thanks, GerardM On 19 October 2013 12:59, Leslie Carr lc...@wikimedia.org wrote: On Sat, Oct 19, 2013 at 1:17 PM, Maarten Dammers maar...@mdammers.nl wrote: Hi Ken, Op 18-10-2013 22:05, Ken Snider schreef: The Wikimedia Foundation's Technical Operations team is seeking proposals on the provisioning of a new data-centre facility. After working through the specifics internally, we now have a public RFP posted[1] and ready for proposals. We invite any organization meeting the requirements outlined to submit a proposal for review. You have stated some technical requirements, but not the availability you would like to have. You probably want to include that you're looking for a tier-4 data center (https://en.wikipedia.org/wiki/Tier_4_data_center#Data_center_tiers). Is this one going to replace the Florida data center? Where are you keeping documentation these days? The information on wikitech seems to be very incomplete and outdated. Wikitech is our best source of documentation Something related: While travelling in China I noticed the bad performance of our sites. Would it be a good idea to investigate this (lack of) performance and maybe consider a caching site somewhere in Asia? The latency should be much better than getting is all the way from the USA. http://www.glif.is/publications/maps/GLIF_5-11_World_4k.jpg (from http://www.glif.is/publications/maps/) gives a good idea of connectivity by the way. (Employee hat) - we are starting to move some of the Asian traffic to a new caching center on the US west coast, which has helped latency. (Personal hat) - I would love to get an Asian caching center (Hong Kong or Tokyo would be my top 2 choices), but IMHO the biggest barrier to this the time and availability of people resources. Maarten ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
On Sat, Oct 19, 2013 at 2:18 PM, Gerard Meijssen gerard.meijs...@gmail.com wrote: Hoi, Are some of our partners in Wikipedia Zero not caching already ? Thanks, GerardM Are you asking if they have varnish caches (they do not) or if they are using some web caching on their environment (which is possible, using transparent proxies , though I do not know of providers who use them... however my mobile web ecosystem knowledge is limited) On 19 October 2013 12:59, Leslie Carr lc...@wikimedia.org wrote: On Sat, Oct 19, 2013 at 1:17 PM, Maarten Dammers maar...@mdammers.nl wrote: Hi Ken, Op 18-10-2013 22:05, Ken Snider schreef: The Wikimedia Foundation's Technical Operations team is seeking proposals on the provisioning of a new data-centre facility. After working through the specifics internally, we now have a public RFP posted[1] and ready for proposals. We invite any organization meeting the requirements outlined to submit a proposal for review. You have stated some technical requirements, but not the availability you would like to have. You probably want to include that you're looking for a tier-4 data center (https://en.wikipedia.org/wiki/Tier_4_data_center#Data_center_tiers). Is this one going to replace the Florida data center? Where are you keeping documentation these days? The information on wikitech seems to be very incomplete and outdated. Wikitech is our best source of documentation Something related: While travelling in China I noticed the bad performance of our sites. Would it be a good idea to investigate this (lack of) performance and maybe consider a caching site somewhere in Asia? The latency should be much better than getting is all the way from the USA. http://www.glif.is/publications/maps/GLIF_5-11_World_4k.jpg (from http://www.glif.is/publications/maps/) gives a good idea of connectivity by the way. (Employee hat) - we are starting to move some of the Asian traffic to a new caching center on the US west coast, which has helped latency. (Personal hat) - I would love to get an Asian caching center (Hong Kong or Tokyo would be my top 2 choices), but IMHO the biggest barrier to this the time and availability of people resources. Maarten ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
Hoi, I am not so much interested in what they use. What I would like to suggest is that we have partners in both Africa and Asia. They are likely to have the expertise to run a caching centre on our behalf. They provide us a service in bringing Wikipedia at no cost to their customers. When we pay them to run a non-discriminatory caching service, it would increase our service and maybe even the cost of transatlantic traffic. Thanks, GerardM On 19 October 2013 13:25, Leslie Carr lc...@wikimedia.org wrote: On Sat, Oct 19, 2013 at 2:18 PM, Gerard Meijssen gerard.meijs...@gmail.com wrote: Hoi, Are some of our partners in Wikipedia Zero not caching already ? Thanks, GerardM Are you asking if they have varnish caches (they do not) or if they are using some web caching on their environment (which is possible, using transparent proxies , though I do not know of providers who use them... however my mobile web ecosystem knowledge is limited) On 19 October 2013 12:59, Leslie Carr lc...@wikimedia.org wrote: On Sat, Oct 19, 2013 at 1:17 PM, Maarten Dammers maar...@mdammers.nl wrote: Hi Ken, Op 18-10-2013 22:05, Ken Snider schreef: The Wikimedia Foundation's Technical Operations team is seeking proposals on the provisioning of a new data-centre facility. After working through the specifics internally, we now have a public RFP posted[1] and ready for proposals. We invite any organization meeting the requirements outlined to submit a proposal for review. You have stated some technical requirements, but not the availability you would like to have. You probably want to include that you're looking for a tier-4 data center (https://en.wikipedia.org/wiki/Tier_4_data_center#Data_center_tiers). Is this one going to replace the Florida data center? Where are you keeping documentation these days? The information on wikitech seems to be very incomplete and outdated. Wikitech is our best source of documentation Something related: While travelling in China I noticed the bad performance of our sites. Would it be a good idea to investigate this (lack of) performance and maybe consider a caching site somewhere in Asia? The latency should be much better than getting is all the way from the USA. http://www.glif.is/publications/maps/GLIF_5-11_World_4k.jpg (from http://www.glif.is/publications/maps/) gives a good idea of connectivity by the way. (Employee hat) - we are starting to move some of the Asian traffic to a new caching center on the US west coast, which has helped latency. (Personal hat) - I would love to get an Asian caching center (Hong Kong or Tokyo would be my top 2 choices), but IMHO the biggest barrier to this the time and availability of people resources. Maarten ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Request to review the Wikimaps Atlas Grant
Dear Wikihackers, User:Yug[1] from the Graphic Labs and I, User:Planemad[2] a cartographer from the Indian community, have put together a comprehensive IEG proposal to improve all the base maps used on WIkimedia projects with updated cartographic conventions and accuracy using the latest tools and data. The output will be a large set of editable vector maps, research documentation and a comprehensive map generation workflow and infrastructure that anyone can reuse. The total grant request is for an amount of USD 10500, which will allow both of us to work on this project full time for a period of 3 months. The grant also includes a budget to hire external consultants where required to accomplish the stated goals. You can view the proposal and provide your valuable feedback/endorsement here before Oct 22: https://meta.wikimedia.org/wiki/Grants:IEG/Wikimaps_Atlas PS: This will be the first step of the much larger Wikimaps[3] project that aims to be the map engine that brings together Wikidata, Commons and Openstreetmap. [1] https://en.wikipedia.org/wiki/User:Yug [2] https://en.wikipedia.org/wiki/User:Planemad [3] https://meta.wikimedia.org/wiki/Wikimaps -- Arun Ganesh (planemad) http://en.wikipedia.org/wiki/User:Planemad http://j.mp/ArunGanesh ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
On Sat, Oct 19, 2013 at 2:39 PM, Gerard Meijssen gerard.meijs...@gmail.com wrote: Hoi, I am not so much interested in what they use. What I would like to suggest is that we have partners in both Africa and Asia. They are likely to have the expertise to run a caching centre on our behalf. They provide us a service in bringing Wikipedia at no cost to their customers. When we pay them to run a non-discriminatory caching service, it would increase our service and maybe even the cost of transatlantic traffic. Thanks, GerardM Running a mobile network is completely different operation from running a CDN. -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
Op 19-10-2013 12:59, Leslie Carr schreef: Wikitech is our best source of documentation Ouch, time for a little documentation sprint? Something related: While travelling in China I noticed the bad performance of our sites. Would it be a good idea to investigate this (lack of) performance and maybe consider a caching site somewhere in Asia? The latency should be much better than getting is all the way from the USA. http://www.glif.is/publications/maps/GLIF_5-11_World_4k.jpg (from http://www.glif.is/publications/maps/) gives a good idea of connectivity by the way. (Employee hat) - we are starting to move some of the Asian traffic to a new caching center on the US west coast, which has helped latency. I guess that's the one in San Francisco? That's already a lot closer than US east coast, but would it be enough? (Personal hat) - I would love to get an Asian caching center (Hong Kong or Tokyo would be my top 2 choices), but IMHO the biggest barrier to this the time and availability of people resources. I agree. Would be nice to connect to http://www.ams-ix.hk/ too :-) This is in the strategic plan: Deploy additional caching centers in key locations to serve growing audiences in Asia, Latin America, and the Middle East. [1] so allocating resources to this should be a valid choice. Maarten [1] http://wikimediafoundation.org/wiki/Wikimedia_Movement_Strategic_Plan_Summary ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
Before it is send to a mobile phone, the data has to be retrieved in the classic way. Do you really think that the companies who run mobile network do not have the expertise to keep a bunch of caching servers in the air ? When you do, do you think that all these mobile operators do not have that skill? Do you think they do not have capacity at the key locations of the Internet infrastructure? Thanks, GerardM On 19 October 2013 13:44, Leslie Carr lc...@wikimedia.org wrote: On Sat, Oct 19, 2013 at 2:39 PM, Gerard Meijssen gerard.meijs...@gmail.com wrote: Hoi, I am not so much interested in what they use. What I would like to suggest is that we have partners in both Africa and Asia. They are likely to have the expertise to run a caching centre on our behalf. They provide us a service in bringing Wikipedia at no cost to their customers. When we pay them to run a non-discriminatory caching service, it would increase our service and maybe even the cost of transatlantic traffic. Thanks, GerardM Running a mobile network is completely different operation from running a CDN. -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
On Sat, Oct 19, 2013 at 2:50 PM, Gerard Meijssen gerard.meijs...@gmail.com wrote: Before it is send to a mobile phone, the data has to be retrieved in the classic way. As an engineer I do know how the internet works ;) Do you really think that the companies who run mobile network do not have the expertise to keep a bunch of caching servers in the air ? When you do, do you think that all these mobile operators do not have that skill? Do you think they do not have capacity at the key locations of the Internet infrastructure? That is exactly what I think. Thanks, GerardM On 19 October 2013 13:44, Leslie Carr lc...@wikimedia.org wrote: On Sat, Oct 19, 2013 at 2:39 PM, Gerard Meijssen gerard.meijs...@gmail.com wrote: Hoi, I am not so much interested in what they use. What I would like to suggest is that we have partners in both Africa and Asia. They are likely to have the expertise to run a caching centre on our behalf. They provide us a service in bringing Wikipedia at no cost to their customers. When we pay them to run a non-discriminatory caching service, it would increase our service and maybe even the cost of transatlantic traffic. Thanks, GerardM Running a mobile network is completely different operation from running a CDN. -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
Ken Snider wrote: The Wikimedia Foundation's Technical Operations team is seeking proposals on the provisioning of a new data-centre facility. After working through the specifics internally, we now have a public RFP posted[1] and ready for proposals. We invite any organization meeting the requirements outlined to submit a proposal for review. Most of the relevant details are in the document itself, but feel free to reach out to myself or the list should anyone have any questions. Hi. I'm pretty confused how this RFP relates to the ulsfo datacenter. As far as I know, ulsfo is not being proposed, it's already been approved. Is this RFP intended for an additional new datacenter somewhere in the U.S.? I looked at https://meta.wikimedia.org/wiki/Wikimedia_servers#Hosting to try to gain clarity. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
On Sat, Oct 19, 2013 at 3:01 PM, MZMcBride z...@mzmcbride.com wrote: Ken Snider wrote: The Wikimedia Foundation's Technical Operations team is seeking proposals on the provisioning of a new data-centre facility. After working through the specifics internally, we now have a public RFP posted[1] and ready for proposals. We invite any organization meeting the requirements outlined to submit a proposal for review. Most of the relevant details are in the document itself, but feel free to reach out to myself or the list should anyone have any questions. Hi. I'm pretty confused how this RFP relates to the ulsfo datacenter. As far as I know, ulsfo is not being proposed, it's already been approved. Is this RFP intended for an additional new datacenter somewhere in the U.S.? I looked at https://meta.wikimedia.org/wiki/Wikimedia_servers#Hosting to try to gain clarity. ulsfo is a caching center (varnish + LVS servers, but no backend infrastructure and very few machines) MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
Leslie Carr wrote: ulsfo is a caching center (varnish + LVS servers, but no backend infrastructure and very few machines) Thank you for clarifying a bit. That helps. Is this new datacenter intended to be more like eqiad, then? And (echoing Maarten's question) will this new datacenter replace pmtpa? I'm just trying to wrap my head around this at a very high level. :-) MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
On Sat, Oct 19, 2013 at 3:29 PM, MZMcBride z...@mzmcbride.com wrote: Leslie Carr wrote: ulsfo is a caching center (varnish + LVS servers, but no backend infrastructure and very few machines) Thank you for clarifying a bit. That helps. Is this new datacenter intended to be more like eqiad, then? And (echoing Maarten's question) will this new datacenter replace pmtpa? Yes to both! I'm just trying to wrap my head around this at a very high level. :-) MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Leslie Carr Wikimedia Foundation AS 14907, 43821 http://as14907.peeringdb.com/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit Patch Uploader beta
On 16 October 2013 05:15, MZMcBride z...@mzmcbride.com wrote: I looked at your original e-mail and gerrit-patch-uploader itself and couldn't find a link to the source code. Could one be added to the user interface? I think it would help sustain the project. As Matt already mentioned: it's at https://github.com/valhallasw/gerrit-patch-uploader . I've added a link there from the front page, and a 'hip' fork me on GitHub ribbon :-) I completely missed that the Committer name gets updated on changes such as https://gerrit.wikimedia.org/r/90002. I looked at the Owner field and got very confused. I'm not sure there's anything to be done about this. :-/ A note in the comments or in the commit message (such as a keyword) might have gotten my attention. This might just be me, though. I can see how it can be confusing, yes. There are three fields in Gerrit: - Owner = the Gerrit username that uploaded the first patchset. There is no way to change this name other than asking an admin. - PS Author = whoever wrote (most of) the patchset. This is a free-form field, and is what you fill in in the 'Author' field of GPU. - PS Committer = whoever uploaded the patchset, but with a free-form user name field. This becomes '[[mw:user:yourusername]] gerritpatchuploa...@gmail.com I have now added a message that is automatically submitted as comment: https://gerrit.wikimedia.org/r/#/c/90698/ If you have any suggestions on how to improve the username mess and/or the comment, please let me know (or send a pull request!) Merlijn ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Optimizing the deployment train schedule
On 2013-10-18 9:40 PM, Rob Lanphier ro...@wikimedia.org wrote: Hi Erik, I'm not a fan of removing one of the stages of our current deployments. More inline: On Fri, Oct 18, 2013 at 3:26 PM, Erik Moeller e...@wikimedia.org wrote: Option B: No Monday deploy. This would mean we'd have to improve our testing process to catch issues affecting the non-Wikipedia wikis before they hit production. I personally think getting rid of the Monday deploy could create some _desirable_ pain that would act as a forcing function to improve pre-release test practices, rather than using production wikis to test. At the same time, we'd have a full week to work out the kinks we find in testing before they hit any production wiki, and could have a more systematic process of backing out changes if needed prior to deployment. The Monday deploy is where we catch load based issues in a way that's not absolutely crushing. The cumulative traffic of the wikis is approximately 10% of our overall traffic, which is large enough to notice load-based problems, but small enough to make the difference between hmm, we seem to have a load issue to oh crap, we just brought down the site. We also generally discover many more issues through getting it in front of more people, but not foisting it on everyone. It's not great that there are bugs that some people have to suffer through, but it's better than making all people suffer through them. We can change the mix of wikis so that it's not always the same set that's part of the pilot group (and maybe one day in the glorious future be able to do mixed versioning on a per-wiki basis so that people could opt-in), but I'd rather not foist everything on everyone at once. Finally, another advantage of staging things this way is that we get some time to focus on non-Wikipedia sister project bugs before we deploy to Wikipedia. There are often project-specific bugs, and our test infrastructure isn't *nearly* built out enough to catch even the majority of them. If we deploy to all projects at once, we get hit with all of the bugs at once. Option C: Shift Monday deploys to Tuesday. This would at least give us an additional work day to fix issues that have occurred in testing before they hit prod. I personally don't think this goes far enough, but might be a useful tweak to make if option B seems too problematic. I like this option. U.S. Holidays (and holidays observed by a significant chunk of key WMF employees) often fall on Monday, which means we often have to reschedule these for Tuesday anyway. Rob ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l Tuesdays are also nice as that gives a day for bugs filed by a user on a weekend to be found/triaged by someone, and the correct person notified before the next stage of deploy. As a user I have vauge memories of the site going down much more often in the past due to performance issues, which doesn't seem to happen anymore with the split up deploy. Our ability to do effective load testing prior to a deploy is essentially zero other than reading code afaik, and I have yet to hear any proposals to change that. I don't think the pain points caused would actually get fixed. (Ok, I guess comparing profiling data of the testwikis before and after deploy carefully can reveal performance issues, but I still think one has to actually test with high load to see the high load issues) -bawolff ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Optimizing the deployment train schedule
Le 19/10/13 00:26, Erik Moeller a écrit : Are there other ways to optimize / issues I'm missing or misrepresenting above? Hello, As a summary we deploy a new release in three stages spanned over a one week window. The last stage of the previous window occurring the same day as the first stage of the next window. The three stages are: 1) test wikis (ie mediawiki) 2) non-wikipedias 3) wikipedias The stages are scheduled as: Thursdaywindow 1 stage 1 Monday window 1 stage 2 Thursday+7 window 1 stage 3, window 2 stage 1 Monday window 2 stage 2 ... What about doing all three stages the same day? We could take advantage of our 18 hours presence from Europe to San Francisco. Hence we could go with something like: 8:00 UTC (1am PST): deploy on test wikis (Europe folks) 16:00 UTC (9am PST): deploy non wikipedias (Europe, East Coast + SF) 20:00 UTC (1pm PST): deploy on wikipedias (East Coast + SF) European folks would catch issues appearing on test wikis, the non wikipedias could be done with Europe+SF and the wikipedias by SF. We also have ops coverage on all that time frame. With such a system, we could keep deploying on Thursdays and Mondays, though we will deploy two releases per weeks. Evil plan: deploy automatically on merge. But we are not ready yet :-] -- Antoine hashar Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] 2013 Datacenter RFP - open for submissions
On Oct 19, 2013, at 3:17 AM, Maarten Dammers maar...@mdammers.nl wrote: You probably want to include that you're looking for a tier-4 data center This is more marketing-foo than realistic. I have had longer-than-expected-max outages in every datacenter I've had systems in save one (which is luck, not extra robustness). Every time I site survey a tier 4 I can find vulnerabilities. -george william herbert george.herb...@gmail.com Sent from Kangphone ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Optimizing the deployment train schedule
Hi there, tldr; I like a modified Option C, but also propose a very different Option D that I think would also be good, either now or as the next next step. quote name=Erik Moeller date=2013-10-18 time=15:26:16 -0700 [snip overview of problem, combined with Robla's and you get a good picture of the issues.] == Some options == Option A: Change nothing. I've not heard from enough folks to see if the problems above are widely perceived to _be_ problems. If the consensus is that current practice, for now, is the best possible approach, obviously we should stick with it. I think this is a non-option, honestly. The current schedule has issues that can be resolved; let's try to resolve them. Option B: No Monday deploy. This would mean we'd have to improve our testing process to catch issues affecting the non-Wikipedia wikis before they hit production. I personally think getting rid of the Monday deploy could create some _desirable_ pain that would act as a forcing function to improve pre-release test practices, rather than using production wikis to test. At the same time, we'd have a full week to work out the kinks we find in testing before they hit any production wiki, and could have a more systematic process of backing out changes if needed prior to deployment. Due to the concerns raised by Robla (and I, when in person), I'm not sure this is the right way to go next. It might be an option later when our cycle is a matter of a day or two, but not now with the week-long cycle. Option C: Shift Monday deploys to Tuesday. This would at least give us an additional work day to fix issues that have occurred in testing before they hit prod. I personally don't think this goes far enough, but might be a useful tweak to make if option B seems too problematic. I like this option as a next step, but with a caveat/suggestion: we mix up the wikis in stage 0, 1, and 2. And, we should be open to changing the mix more frequently and based on community feedback (I know some are actually willing/wanting to join the fun of being earlier in the cycle...). Until we have a way to gradually increase the % of users who are using the new wmf *cross wiki*, then our only option is doing things per wiki, which gives you two conceptual options: a test/production split, and that's it, or a tiered system like the 3-tier one we have now. I have two suggestions; a safe one and a less safe one (where 'safe' being 'easy to sell to people'): 1) the safe one: We move Monday's deploy to Tuesday. Let some wikis move into phase 1 from phase 2, and some move from phase 1 to phase 2 (but probably keep phase 0 the same unless some community is as crazy as mw.org's ;) ). This will give more agency to communities on their placement in the cycle while still giving us a more thorough load test on Tuesday after blatant issues are found on Thur/Fri. 2) the less safe one (Option D): We have a four-tiered system. tier0 on Mon, tier1 on Tue, tier2 on Wed, tier3 on Thurs, on Friday we rest (er, merge into master for Monday). Ideal breakdown of user load (of total cross cluster) would be something like: tier0:5% (5% total) tier1:20% (25% total) tier2:30% (55% total) tier3:45% (100%) This gives us: increasing load, with more measurable moments in time. What I mean by that is: With Ori's awesome new work (and planned work), we'll be able to make more sense of performance/load pre/post a deploy. We already look at 500s and similar logs, but those are lumped in the 'apparent bugs' that are found right after a deploy (along with obvious this button went missing things). With only a 3 tier system, where the first tier is basically so small it is hard to tell signal from noise in pre/post deploy performance data. We still only get one chance to test load (tier1, non-wikipedias now) before going everywhere and potentially having downtime. I argue/theorize, that with 3 deploys before we get to everywhere, we would be better able to spot performance issues. Now, we can't probably do that idealized load distribution I lay out above. See: http://stats.wikimedia.org/EN/TablesPageViewsMonthlyAllProjectsOriginal.htm for the breakdown per project type. Also (for the Wikpedia's breakdown): http://stats.wikimedia.org/EN/TablesPageViewsMonthlyOriginalCombined.htm insert time where Greg goes off to sift through data Ok, I'm going to have to sit down with this data on Monday (this current naptime session won't be long enough) and come back with a proposed distribution. Simply: I'll try to hit the above idealized breakdown, but with these restrictions: A) ENWP in tier3 (which is 44% by itself, using Sept'13 data); B) for tiers 1 and 2, get a mix of project types (ie: include WPs, wikibookos, wiktionaries, etc in both); and C) tier0 being only testwikis (and mw.org). But leave this open for others to join, if desired. Other benefits of Option D: * gets us accustomed to more frequent deploys. * will provide some of that beneficial pain
