[Foundation-l] Technical aspects of forking (was: 86% of german users disagree with the introduction of the personal image filter)
2011/9/17 David Gerard dger...@gmail.com: On 17 September 2011 10:16, John Vandenberg jay...@gmail.com wrote: On Sat, Sep 17, 2011 at 7:11 PM, David Gerard dger...@gmail.com wrote: We need people to try the technical basics of a fork, i.e. taking an en:wp dump, an images dump, .. Is there an images dump? If there isn't, there should be. (I'm now trying to work out how to get the images without using up all my bandwidth allowances ever.) I have no traffic limit on my home connection. Just ship a big enough storage device and I'll be happy to provide you a full dump :P Just kidding, of course. On a more serious note, what do you expect the difficulties to be? I believe the most difficult part would be to replicate the foundation's secret sauce, i.e. the configuration files that are not made public, if such thing exists. Then would come the whole traffic balancing/caching/optimization settings, which would greatly depend on the actual traffic a fork would have. Strainu ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Technical aspects of forking (was: 86% of german users disagree with the introduction of the personal image filter)
the foundation's secret sauce, i.e. the configuration files that are not made public, if such thing exists. Then would come the whole traffic balancing/caching/optimization settings, which would greatly depend on the actual traffic a fork would have. Strainu If you control your own servers, vital for a fully functioning fork, you'll eventually work out the technical details. However, I'm afraid the secret sauce involves interpersonal elements, including respecting the sensitivities of others on a global basis. Fred ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Technical aspects of forking (was: 86% of german users disagree with the introduction of the personal image filter)
On Sat, Sep 17, 2011 at 02:42:08PM +0300, Strainu wrote: I believe the most difficult part would be to replicate the foundation's secret sauce, i.e. the configuration files that are not made public, if such thing exists. Special:Version has always been good enough for me ;-) Then would come the whole traffic balancing/caching/optimization settings, which would greatly depend on the actual traffic a fork would have. My first instincts for de.wikipedia would be to note down de.wikipedia's usage statistics, get a bunch of techies together, and all go have a nice chat with say hetzner.de, to figure out roughly what things will cost. You can always start a bit small and work your way up. sincerely, Kim Bruning ( To help jumpstart maths: Renting a 49 unit 19 rack with 1TB traffic, 1GB/sec costs around Eur 200/month these days. You still have to buy equipment to mount in that rack and set it all up, of course, which might cost you around 50KEur[1] Rent+obsoleting over 3 years gets in the ballpark of about 20K/year amortized. Extra traffic allowance can typically be ordered separately. You may need more, or less, than a single rack, depending on traffic. * http://www.nedworks.org/~mark/reqstats//trafficstats-hourly.png * http://www.nedworks.org/~mark/reqstats//trafficstats-monthly.png For the entire cluster(all projects), it looks like traffic can peak to 6Gbit/s at the moment * http://stats.wikimedia.org/EN/Sitemap.htm enwiki: 7,976,862 views per hour dewiki: 1,054,677 views per hour [1] @~Eur 1000/system. Note that you also need to mount switches, UPS, etc, so you can't use the whole rack just for computation. Also note that things like blade servers or NAS servers can fit more processor power or storage into less rackspace, where required. ) -- ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Technical aspects of forking (was: 86% of german users disagree with the introduction of the personal image filter)
On 17 September 2011 15:50, Kim Bruning k...@bruning.xs4all.nl wrote: My first instincts for de.wikipedia would be to note down de.wikipedia's usage statistics, get a bunch of techies together, and all go have a nice chat with say hetzner.de, to figure out roughly what things will cost. You can always start a bit small and work your way up. Starting small is *most definitely* the way to go. Citizendium burned through enough funding for *20 years* in three years by ridiculously overestimating their server needs. - d. ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
[Foundation-l] Technical aspects of forking (was: 86% of german users disagree with the introduction of the personal image filter)
Sanity in IT terms and practicality in regulatory terms don't always go hand in hand. Transporting an image dump on a hard drive might well be the most practical way to move that much data - though it should be encrypted at least whilst in transit. But forking doesn't sound to me a good reason to disclose deleted edits. Or for that matter account passwords. So that drive would need to be an extract of the material covered in the license. WereSpielChequers -- Message: 9 Date: Sat, 17 Sep 2011 10:06:08 -0400 From: MZMcBride z...@mzmcbride.com Subject: Re: [Foundation-l] 86% of german users disagree with the introduction of the personal image filter To: Wikimedia Foundation Mailing List foundation-l@lists.wikimedia.org Message-ID: ca9a2190.1422...@mzmcbride.com Content-Type: text/plain; charset=US-ASCII David Gerard wrote: On 17 September 2011 10:16, John Vandenberg jay...@gmail.com wrote: On Sat, Sep 17, 2011 at 7:11 PM, David Gerard dger...@gmail.com wrote: We need people to try the technical basics of a fork, i.e. taking an en:wp dump, an images dump, .. Is there an images dump? If there isn't, there should be. (I'm now trying to work out how to get the images without using up all my bandwidth allowances ever.) It's easy enough to get a VPS with unlimited bandwidth. It's a few terabytes of data, though, depending on what you're talking about. Thumbnails, current images, older versions of images, deleted images, math renderings, etc. The sanest solution probably involves mailing a hard drive to someone and then having them mail it back. MZMcBride ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Technical aspects of forking (was: 86% of german users disagree with the introduction of the personal image filter)
On 17 September 2011 15:50, Kim Bruning k...@bruning.xs4all.nl wrote: My first instincts for de.wikipedia would be to note down de.wikipedia's usage statistics, get a bunch of techies together, and all go have a nice chat with say hetzner.de, to figure out roughly what things will cost. You can always start a bit small and work your way up. Starting small is *most definitely* the way to go. Citizendium burned through enough funding for *20 years* in three years by ridiculously overestimating their server needs. - d. May I suggest a Apple Mini Mac with the Lion server, loaded with plenty of memory and storage of course. Fred ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Technical aspects of forking (was: 86% of german users disagree with the introduction of the personal image filter)
On Sat, Sep 17, 2011 at 05:04:33PM +0100, WereSpielChequers wrote: Sanity in IT terms and practicality in regulatory terms don't always go hand in hand. Transporting an image dump on a hard drive might well be the most practical way to move that much data - though it should be encrypted at least whilst in transit. Encrypted? But forking doesn't sound to me a good reason to disclose deleted edits. Or for that matter account passwords. So that drive would need to be an extract of the material covered in the license. Right, if we do this, we don't need to encrypt. And now you know why we always kept warning people why delete actually really *does* mean delete. At some point in time, the deleted data will actually be lost. If all is well, all data needed to replicate en.wikipedia should be online and downloadable. It's definitely a good idea to test that at some point! sincerly, Kim Bruning -- ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Technical aspects of forking (was: 86% of german users disagree with the introduction of the personal image filter)
On Sat, Sep 17, 2011 at 04:52:46PM +0100, David Gerard wrote: On 17 September 2011 15:50, Kim Bruning k...@bruning.xs4all.nl wrote: My first instincts for de.wikipedia would be to note down de.wikipedia's usage statistics, get a bunch of techies together, and all go have a nice chat with say hetzner.de, to figure out roughly what things will cost. You can always start a bit small and work your way up. Starting small is *most definitely* the way to go. Citizendium burned through enough funding for *20 years* in three years by ridiculously overestimating their server needs. Zigzacktly. Rent for a decent single server these days is around Eur 100/month (with a one-off setup fee around the same numbers) It's probably wise to have a plan (and a little spare cash) on hand, should usage skyrocket, of course. sincerely, Kim Bruning -- ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Technical aspects of forking (was: 86% of german users disagree with the introduction of the personal image filter)
On 09/17/11 8:52 AM, David Gerard wrote: On 17 September 2011 15:50, Kim Bruningk...@bruning.xs4all.nl wrote: My first instincts for de.wikipedia would be to note down de.wikipedia's usage statistics, get a bunch of techies together, and all go have a nice chat with say hetzner.de, to figure out roughly what things will cost. You can always start a bit small and work your way up. Starting small is *most definitely* the way to go. Citizendium burned through enough funding for *20 years* in three years by ridiculously overestimating their server needs. Optimism is a key characteristic of those who believe they have a killer site. Ray ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
