Re: [Wikitech-l] Jettisoning our history?
Le 31/05/2014 04:25, Chad a écrit : > When we end up moving MW core to Phabricator I'd like us to jettison our > history. The > repo is large and clunky and not conducive to development. It's only going > to grow in > size unless we do something to cut back on the junk we're carrying around. Hello, My repacked copy of core is 270MB which is not that huge and I really like having the whole history for bisecting and blaming code. What about attempting to slowdown the rate of growth? The i18n messages can probably be split to another repository or at least be updated only once in a while instead of on a daily basis. We might also have some big objects floating around in the repo which we could potentially drop from the repo. IIRC we had a few .jar committed by mistake in SVN, though we dropped them when migrating to git. There is probably some other big objects we could remove. cheers, -- Antoine "hashar" Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] enwiki display issues
Shouldn't we have an incident report[1] about this? Helder [1] https://wikitech.wikimedia.org/wiki/Incident_documentation#Incident_reports ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
I don't like this idea, for the same reasons that other have already given. Grafting histories with git-replace might be viable, but it'd still be clunky and non-intuitive. Why don't we just suggest that people use shallow clones? Git supports pushing from and pulling to them since 1.9, and while Gerrit doesn't accept pushes from them (or at least it didn't when I just tried), I see no reason why Phabricator would have any issues if it only works on diffs anyway, not commits. -- Matma Rex ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] deleting config directory confusion
https://www.mediawiki.org/wiki/Manual:Config_script says to delete the config directory. The instructions displayed within the config script do not, and it the page is protected after install by the random key. So, what is the correct instruction? Delete it? don't delete it? delete it optionally for purpose X? ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] deleting config directory confusion
delete it optionally for purpose X? You can use the config script to perform database upgrades later on if you upgrade MediaWiki versions. Most people just use the command line update.php instead, but for some people this is not an option. -- Tyler Romeo 0xC86B42DF ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
On Sat, May 31, 2014 at 1:05 AM, Antoine Musso wrote: > Le 31/05/2014 04:25, Chad a écrit : > > When we end up moving MW core to Phabricator I'd like us to jettison our > > history. The > > repo is large and clunky and not conducive to development. It's only > going > > to grow in > > size unless we do something to cut back on the junk we're carrying > around. > > Hello, > > My repacked copy of core is 270MB which is not that huge and I really > like having the whole history for bisecting and blaming code. > > 270MB is gigantic for a git repo. > > What about attempting to slowdown the rate of growth? The i18n messages > can probably be split to another repository or at least be updated only > once in a while instead of on a daily basis. > > A solution for containing the growth, yes. I'd +1 this along with Daniel F's idea earlier. > We might also have some big objects floating around in the repo which we > could potentially drop from the repo. IIRC we had a few .jar committed > by mistake in SVN, though we dropped them when migrating to git. There > is probably some other big objects we could remove. > How would we do that without rewriting history? Same problem. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] deleting config directory confusion
On Sat, May 31, 2014 at 6:04 AM, Ian Kelling wrote: > > https://www.mediawiki.org/wiki/Manual:Config_script > says to delete the config directory. The instructions displayed within the > config > script do not, and it the page is protected after install by the random > key. So, what is the correct instruction? Delete it? don't delete it? > delete it optionally for purpose X? > > There's no reason you have to delete the install directory. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
On 31 May 2014 16:08, Chad wrote: > 270MB is gigantic for a git repo. > > But it's not an issue /per se/. The issue is slow clones/slow pulls, not so much the 270MB on your hard drive. The slow clones/pulls can be improved by re-packing the git repository on the server side -- this helped significantly for the pywikibot repositories. I'm not sure if this has been attempted for mw/core yet. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
On Sat, May 31, 2014 at 8:03 AM, Merlijn van Deen wrote: > On 31 May 2014 16:08, Chad wrote: > > > 270MB is gigantic for a git repo. > > > > > But it's not an issue /per se/. The issue is slow clones/slow pulls, not so > much the 270MB on your hard drive. The slow clones/pulls can be improved by > re-packing the git repository on the server side -- this helped > significantly for the pywikibot repositories. I'm not sure if this has been > attempted for mw/core yet. > > I do it weekly for core. It's the only thing keeping it from exploding to many hundreds of MB on the Gerrit box. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] enwiki display issues
On Sat, 2014-05-31 at 07:52 -0300, Helder . wrote: > Shouldn't we have an incident report[1] about this? https://wikitech.wikimedia.org/wiki/Incident_documentation/20140529-appservers (as mentioned in the other thread) andre -- Andre Klapper | Wikimedia Bugwrangler http://blogs.gnome.org/aklapper/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
On Sat, May 31, 2014 at 5:52 AM, Bartosz Dziewoński wrote: > I don't like this idea, for the same reasons that other have already > given. Grafting histories with git-replace might be viable, but it'd still > be clunky and non-intuitive. > > Ok, fair enough. Everyone's made some really good points so let's drop the idea of dropping our history. However I think we should continue to discuss ways to contain the repo size going forward. That, combined with some aggressive repacking and dropping of refs/changes/* (when we move to Phabricator) should help get it under control. > Why don't we just suggest that people use shallow clones? Git supports > pushing from and pulling to them since 1.9, and while Gerrit doesn't accept > pushes from them (or at least it didn't when I just tried), I see no reason > why Phabricator would have any issues if it only works on diffs anyway, not > commits. This is also a good idea. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
On Sat, May 31, 2014 at 9:38 AM, Chad wrote: > On Sat, May 31, 2014 at 5:52 AM, Bartosz Dziewoński > wrote: > > > Why don't we just suggest that people use shallow clones? Git supports > > pushing from and pulling to them since 1.9, and while Gerrit doesn't > accept > > pushes from them (or at least it didn't when I just tried), I see no > reason > > why Phabricator would have any issues if it only works on diffs anyway, > not > > commits. > > > This is also a good idea. > See https://bugzilla.wikimedia.org/show_bug.cgi?id=57430 for discussion on this in the context of mw-vagrant. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
(anonymous) wrote: >> I don't like this idea, for the same reasons that other have already >> given. Grafting histories with git-replace might be viable, but it'd still >> be clunky and non-intuitive. > Ok, fair enough. Everyone's made some really good points so let's drop the > idea of dropping our history. > However I think we should continue to discuss ways to contain the repo size > going forward. That, combined with some aggressive repacking and dropping > of refs/changes/* (when we move to Phabricator) should help get it under > control. > [...] Just to clarify: refs/changes/* = Gerrit patchsets (minus the ones referenced as submitted changes)? If so, sure, they're only scratchpads, but on the other hand they should- n't affect the size of a default clone that just pulls in the parents of master's HEAD? Tim ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
On Sat, May 31, 2014 at 12:30 PM, Tim Landscheidt wrote: > (anonymous) wrote: > > >> I don't like this idea, for the same reasons that other have already > >> given. Grafting histories with git-replace might be viable, but it'd > still > >> be clunky and non-intuitive. > > > Ok, fair enough. Everyone's made some really good points so let's drop > the > > idea of dropping our history. > > > However I think we should continue to discuss ways to contain the repo > size > > going forward. That, combined with some aggressive repacking and dropping > > of refs/changes/* (when we move to Phabricator) should help get it under > > control. > > > [...] > > Just to clarify: refs/changes/* = Gerrit patchsets (minus > the ones referenced as submitted changes)? If so, sure, > they're only scratchpads, but on the other hand they should- > n't affect the size of a default clone that just pulls in > the parents of master's HEAD? > Right. That's less of a cloning problem as it's a problem on the remote and slows down operations on *that* repo. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
Hey, One thing I have noticed is that it is much faster for me to clone core from GitHub then from WMF. Guess that having the thing also hosted in the EU would help. Cheers -- Jeroen De Dauw - http://www.bn2vs.com Software craftsmanship advocate Evil software architect at Wikimedia Germany ~=[,,_,,]:3 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
On 31/05/14 15:03, Merlijn van Deen wrote: On 31 May 2014 16:08, Chad wrote: 270MB is gigantic for a git repo. But it's not an issue /per se/. The issue is slow clones/slow pulls, not so much the 270MB on your hard drive. The slow clones/pulls can be improved by re-packing the git repository on the server side -- this helped significantly for the pywikibot repositories. I'm not sure if this has been attempted for mw/core yet. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l Currently the slow clones can be almost completely avoided by cloning from the github mirror, if you have a reasonably fast connection. But considering how most folks are likely to want older history for blames and whatnot sooner or later anyway, the longer download time for the bulk of the data will still come up when they have to download the rest, but this way it would be with a few extra steps. What /does/ github do, do we know? Would that be useful/applicable? Is phabricator similar? -I ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Jettisoning our history?
On Sat, May 31, 2014 at 5:22 PM, Isarra Yos wrote: > What /does/ github do, do we know? Would that be useful/applicable? Is > phabricator similar? > > They use way more than one server for their cluster, probably have caching. Plus all kinds of proprietary secret sauce including their own in-house implementation of Git. Phabricator uses the normal system git. We won't be using jgit anymore like with Gerrit. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Tech questions for outreach presentation
Hi all, I'm developing a presentation that's intended to introduce Wikimedia's universe to other organizations that may have an interest in partnering with us. I estimate the presentation in its full length to be about 1.5 hours, suitable for a Wikimedia Academy. I will shorten the presentation as needed to accommodate presentation slots at events hosted by other organizations. Wikimedians in my area have been invited to present at a technologist meetup, and there are a number of open source organizations in the area who might be interested in having a Wikimedia guest presentation. This is part of my larger scheme engage more people and organizations in Wikimedia content contributions or other mutually beneficial partnerships. I would like to talk with someone who has broad knowledge on the tech side of the Wikimedia universe so I can ask some questions about hardware, Labs, and MediaWiki. An IRC conversation would be ideal because then I would have a written record that I could refer back to. Is there anyone who would be willing to set up a time for a conversation? I am happy to credit you in the presentation. I intend to publish the presentation in a Wikimedia-compatible license. Thanks! Pine ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
