[Wikitech-l] Wikimedia Conferentie Nederland 2012 and Wikimedia Nederland Hackathon 2012.
Hi, last month I have spent a few days in Utrecht and Amsterdam working with Antoine “hashar” Musso and Timo Tijhof (Krinkle). We also attended Wikimedia Conferentie Nederland 2012 and Wikimedia Nederland Hackathon 2012. Since I am pretty new to both WMF and MediaWiki, most of the time I was just trying to figure out how everything is set up. More specifically, I was working on MediaWiki browser tests[1] or trying to install MediaWiki locally via Vagrant[2]. For more information, please read my blog post[3]. Regards, Željko Filipin -- [1] https://github.com/zeljkofilipin/browsertests [2] https://github.com/wikimedia/wmf-vagrant [3] http://filipin.eu/utrecht-amsterdam-2012/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Complete (basic) analysis of MediaWiki
If last October we got a bunch of MediaWiki developer stats thanks to the aggregation of data by Ohloh [1], now we are getting plenty more stats from Bitergia, including data from bug reporting and mailing lists: http://blog.bitergia.com/2012/12/03/complete-basic-analysis-of-mediawiki/ Bitergia is a company based in Madrid formed by a small team of developers that have been working on FLOSS stats software for a long time. All the tools they develop are free software publicly available and open to contributions. They have been kind enough to contribute some time and work setting up stats for the MediaWiki community. They also welcome feedback about the service and the data collected. I'm CCing Jesús M. González-Barahona, who has been my regular contact for this task in the past weeks. Al good news for http://www.mediawiki.org/wiki/Community_Metrics ! [1] https://www.ohloh.net/orgs/wikimedia -- Quim Gil Technical Contributor Coordinator Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] problems merging changes from master - remote branch
Awesome Roan, thank you. That did the trick. On Sat, Dec 1, 2012 at 3:30 PM, Roan Kattouw roan.katt...@gmail.com wrote: On Fri, Nov 30, 2012 at 2:12 PM, Arthur Richards aricha...@wikimedia.org wrote: ! [remote rejected] HEAD - refs/publish/esisupport/bug/41286 (change 32896 closed) It looks like https://gerrit.wikimedia.org/r/#/c/32896/ is a commit that you submitted to esisupport at some point, then abandoned, and are now trying to push in again. Gerrit doesn't like that very much. To fix it, you can try restoring the change (click Restore Change on the Gerrit page for the change; you may need the author or a Gerrit admin to do this for you, I don't know if we've managed to give everyone permission to restore yet), then trying to run 'git review' again. That should introduce new changes for the things you're submitting, and create a new patchset for the change that was giving you trouble. Roan ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Complete (basic) analysis of MediaWiki
That data is hardly useful, it doesn't explain what it refers to and, even when it does, seems wrong. Compare e.g. https://www.ohloh.net/p/mediawiki/contributors?query=sort=commits Also, https://bugzilla.wikimedia.org/weekly-bug-summary.cgi?tops=10days=10 proves they're not talking of the whole bugzilla but then they don't say which components. Nemo ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Complete (basic) analysis of MediaWiki
On Mon, 2012-12-03 at 19:40 +0100, Federico Leva (Nemo) wrote: Compare e.g. https://www.ohloh.net/p/mediawiki/contributors?query=sort=commits Also, https://bugzilla.wikimedia.org/weekly-bug-summary.cgi?tops=10days=10 proves they're not talking of the whole bugzilla but then they don't say which components. Would be helpful to mention the exact dataset you refer to. Also I'd rather challenge weekly-bug-summary.cgi's results: MediaWiki extensions has 2031 open bugs, and only 1883 have been filed in the last 10 days? = 148 bug reports got opened more than 274 years ago? But maybe I fail to read weekly-bug-summary.cgi correctly. andre -- Andre Klapper | Wikimedia Bugwrangler http://blogs.gnome.org/aklapper/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Complete (basic) analysis of MediaWiki
El 03/12/12 19:40, Federico Leva (Nemo) escribió: That data is hardly useful, it doesn't explain what it refers to I agree a glossary of each term would be useful. It took me a while to realise that committers/closers/senders where the terms used for users of git/bugzilla/mailing list. They should track authors instead of committers, though (preferably skipping merge commits) Also, https://bugzilla.wikimedia.org/weekly-bug-summary.cgi?tops=10days=10 proves they're not talking of the whole bugzilla but then they don't say which components. Nemo Looking at http://bitergia.com/public/previews/2012_11_mediawiki/data/db/acs_bicho_mediawiki.sql.bz2 they seem to have obtained data from bugs 1 to 19775. Not that they skipped bugs based on components. Seems that Jesús did a fine job. It could be polished quite more with some local knowledge, merging users, hiding bots, etc. I would also change the layout of the summary page, making the graphs larger and placing the tables below. Plus some cosmetics empty brackets, missing name... ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] LabeledSectionTransclusion performance problems
Looks like I misspoke here. I said It would be nice to get some feedback from Merlijn about how to solve the problems he's trying to solve in a more efficient way. I meant to say It would be nice to get some feedback FOR Merlijn about how to solve the problems he's trying to solve in a more efficient way. So, I'm bumping this thread for two reasons: 1. Still would like to see some feedback for Merlijn 2. Reminder that we need to either fix or revert Merlijn's changes on master prior to deploying 1.21wmf6, lest we bring down itwikisource again. I'm going to guess that without more feedback, Merlijn won't be able to fix this, so we should plan on reverting on Thursday or so if this thread goes stale. Rob -- Forwarded message -- From: Rob Lanphier ro...@wikimedia.org Date: Fri, Nov 30, 2012 at 5:03 PM Subject: Re: [Wikitech-l] LabeledSectionTransclusion performance problems To: Wikimedia developers wikitech-l@lists.wikimedia.org On Fri, Nov 30, 2012 at 4:09 AM, Guillaume Paumier gpaum...@wikimedia.org wrote: On Fri, Nov 30, 2012 at 10:07 AM, Merlijn van Deen valhall...@arctus.nl wrote: After the new version of LabeledSectionTransclusion (LST) was deployed on itwikisource, performance issues popped up. itwikisource's main page makes heavy use of LST, and the new version is clearly heavier than the old one. As a sidenote: because of the performance issues, the most recent changes to the LST extension will probably be reverted today (Friday, November 30). If you made changes to articles or templates to accommodate the new version or benefit from new features, you may want to revert those changes temporarily. Aaron just reverted this a little earlier: https://gerrit.wikimedia.org/r/#/c/36316/ I think the way that he did this, though, means that if we do nothing, then we'll be redeploying this starting December 10th with the start of the 1.21wmf6 cycle. There's a reasonable amount of time between now and then, so we can leave it like this until next week, depending on the result of this conversation. It would be nice to get some feedback from Merlijn about how to solve the problems he's trying to solve in a more efficient way. For those wanting to see the code in question, Merlijn's main commit is here: https://gerrit.wikimedia.org/r/#/c/31330/ For my part, it seems that even just making sure that there is only one extra parse per referenced page might be enough to make this perform acceptably, even if it means keeping all of the parsed transcluded pages around in memory. I'm not sure the preferred method these days (e.g. global, singleton, context object, or attaching to some other existing object), but that may be worth exploring. This is just a wild guess, though; I have no idea if that would be too heavy on our memory usage, and I'm only suggesting it because it seems relatively easy compared to the alternatives. Rob ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] LabeledSectionTransclusion performance problems
On 3 December 2012 21:16, Rob Lanphier ro...@wikimedia.org wrote: 2. Reminder that we need to either fix or revert Merlijn's changes on master prior to deploying 1.21wmf6, lest we bring down itwikisource again. I'm going to guess that without more feedback, Merlijn won't be able to fix this, so we should plan on reverting on Thursday or so if this thread goes stale. I'll give some more background on my analysis of why this happened. In any case, I'll create a revert patch before thursday, so it shouldn't be a problem to deploy it. The reason itwikisource died is a combination of how LST is used on itwikisource and how the new LST implementation worked. I'll try to give a short description here. On most wiki's, the Main Page consists of several transcluded pages, i.e. 'Main Page/header', 'Main Page/Section Something', 'Main Page/Section Something Else'. This is not the case on itwikisource: instead of using seperate pages, it uses a single page, and LST's the relevant sections - so Pagina Principale transcludes *sections *from Pagina principale/Sezioni - eight of them. In the original LST implementation, this was not a problem: just the transcluded sections were rendered, so overall, 'Pagina principale/Sezioni' was only parsed once - but in a eight seperate pieces. However, in the * current* situation, the whole of 'Pagina principale/Sezioni' is rendered for *every* transclusion, which means that the parsing times goes up by a factor of 8. To make matters worse, this structure is also used further down the template tree, which means the overall parsing time goes up to ~ 30s, the parser's limit. (the current parsing time is ~2s). As such, I think Rob's suggestion of caching the parsing step might be enough to solve the issues. The parsing will still be more resource intensive, but at least it shouldn't reach bring itwikisource to it's knees. However, I don't have the setup to test this currently, nor do I really have the time to set this up this week. So -- I'll first make sure there is a working revert patch submitted. After that, I'll try to work on creating a more resource-friendly implementation. Hopefully there will be some comments on how to do that in an even better way by then. Best, Merlijn ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Complete (basic) analysis of MediaWiki
El 03/12/12 22:45, Jesus M. Gonzalez-Barahona escribió: On Mon, 2012-12-03 at 21:04 +0100, Platonides wrote: El 03/12/12 19:40, Federico Leva (Nemo) escribió: That data is hardly useful, it doesn't explain what it refers to I guess I missed your message, Federico. He forgot to keep you in CC, so it was sent only to the mailing list. I agree a glossary of each term would be useful. It took me a while to realise that committers/closers/senders where the terms used for users of git/bugzilla/mailing list. Well, in fact commiters are committers to the git repository (you also have authors, see below), closers are specifically ticket closers (you also have people opening or changing tickets) and senders are indeed senders of mailing list messages. We'll work to make this much more clear. Again, thanks for the suggestion. They should track authors instead of committers, though (preferably skipping merge commits) We do both. In the summary (main) page you have authors in the summary (orange) chart, since authors seemed more meaningful than committers. Same for the blue chart in that page. In the source code page you have committers (orange chart) and both authors and committers (blue charts), for a more detailed comparison. I was specifically thinking in the table of Top committers. Also, the summary page has an authors graph but http://bitergia.com/public/previews/2012_11_mediawiki/scm.html has a committers one. When the committer is different than the author there are usually two options: - It was a merge and the committer is 'gerrit'. - The patchset was (slightly) changed by the committer from the original by the author. There's also a less common one of committing a patch from a different source, such as a bugzilla patch. Number of commits by gerrit are meaningless, and committers with little changes inflate some numbers but are not too useful. Number of comments / approvals in gerrit would be more appropiate than that. Equally, the author field of merges should IMHO be ignore since that's not a commit which really touches the code (could be measured in a different statistic), so many commits produce two entries. Seems that Jesús did a fine job. It could be polished quite more with some local knowledge, merging users, hiding bots, etc. Thanks a lot. We usually go, after this first stage, with that identification of bots, unification of identities, identification of large commits, classification of different kinds of tickets, etc. In this case, we were mainly testing the automated (first) stage: the second one, as you mention, usually needs some detailed knowledge about the project, and some manual intervention. Sure. I wasn't intending to put pressure on you. A few quirks I noticed: - nore...@sourceforge.net is abusing its second place as sender (2525). - I bet the two brion are the same, with different emails (4561+1285=5846, wow!) l10n-bot is indeed a bot. On svn localisation commits weren't done with a specific account, but you can look for commit messages like «Localisation updates for core and extension messages from translatewiki.net» Commits migrated from svn will have emails of usern...@users.mediawiki.org All commits done on git use a different one. Moreover, some people have used different mails (see other threads on the mailing list about this in ohloh). I would also change the layout of the summary page, making the graphs larger and placing the tables below. Plus some cosmetics empty brackets, missing name... This is a very good point, and something we didn't work too much into. I take note. Thanks a lot for the feedback! You are welcome! ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] LabeledSectionTransclusion performance problems
On 04/12/12 09:01, Merlijn van Deen wrote:
In the original LST implementation, this was not a problem: just the
transcluded sections were rendered, so overall, 'Pagina principale/Sezioni'
was only parsed once - but in a eight seperate pieces. However, in the *
current* situation, the whole of 'Pagina principale/Sezioni' is rendered
for *every* transclusion, which means that the parsing times goes up by a
factor of 8. To make matters worse, this structure is also used further
down the template tree, which means the overall parsing time goes up to ~
30s, the parser's limit. (the current parsing time is ~2s).
Judging by the itwikisource jobs I just killed on most of the job
runners, the parse time went up by a factor of infinity, not a factor
of 8. They had been running for 3.5 days with --maxtime=300.
However, it was not possible to do something like:
page O: ===section start='header'{{{1}}}/section end='header'===
page P: {{O|Some header text}}
page Q: {{#lst:P|header}}
So don't do that. I find it hard to believe that allowing such a
feature will be an amazing win for the wiki's usability.
In the new situation, the #lst mechanism does something like:
1) get expanded wikitext using
$parser-preprocess({{:page_to_be_transcluded}})
2) get the DOM by calling $parser-preprocessToDom() on the expanded
wikitext
3) traverse this DOM, find section tags, and call
$parser-replaceVariables()
on the relevant sections (unchanged)
Double-parsing will break the syntax, {{!}} etc. will not work.
-- Tim Starling
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Advertising changes to MediaWiki messages in core
Hi all, Recently, in order to make a rather small user interface change (see bug #42215), our team needed to replace MediaWiki:Welcomecreation with a new message, MediaWiki:Welcomecreation-msg. The new message contains all of the things MediaWiki normally inserts in that message, and simply changes the header text. Simple enough. My question here is... Is there a best practice for advertising changes to MediaWiki messages like these? This message, which is given to users once after they register, is sometimes customized. We can go look and see which wikis have written custom content, and inform them they need to migrate it to the new message. But I was wondering if others have encountered this problem, and how they dealt with it. -- Steven Walling https://wikimediafoundation.org/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Advertising changes to MediaWiki messages in core
At the least, I think we should post at the closest equivalent to http://en.wikipedia.org/wiki/Wikipedia:MediaWiki_messages for each wiki. Typically, this might be the local version of http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical) or just http://en.wikipedia.org/wiki/Wikipedia:Village_pump (depending how big the wiki is). A bot should be able to deliver this message to all the pages, once the list is in place. Ideally, we'll give them enough time to prepare, and provide a link for central feedback (e.g. a bugzilla). Matt Flaschen On Mon, Dec 3, 2012 at 9:37 PM, Steven Walling swall...@wikimedia.orgwrote: Hi all, Recently, in order to make a rather small user interface change (see bug #42215), our team needed to replace MediaWiki:Welcomecreation with a new message, MediaWiki:Welcomecreation-msg. The new message contains all of the things MediaWiki normally inserts in that message, and simply changes the header text. Simple enough. My question here is... Is there a best practice for advertising changes to MediaWiki messages like these? This message, which is given to users once after they register, is sometimes customized. We can go look and see which wikis have written custom content, and inform them they need to migrate it to the new message. But I was wondering if others have encountered this problem, and how they dealt with it. -- Steven Walling https://wikimediafoundation.org/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Advertising changes to MediaWiki messages in core
On Mon, Dec 3, 2012 at 7:09 PM, Matthew Flaschen mflasc...@wikimedia.orgwrote: Typically, this might be the local version of http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical) or just http://en.wikipedia.org/wiki/Wikipedia:Village_pump (depending how big the wiki is). A bot should be able to deliver this message to all the pages, once the list is in place. Yeah, there is EdwardsBot which will blast all the Village Pumps for us, and I've written/delivered messages myself on a smaller scale. It's just that the only time I've done this in the past was for new features entirely, usually an extension. Using the global message delivery system for a single MediaWiki message change feels like a bit like overkill, which is why asked about precedent. :) Steven ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Advertising changes to MediaWiki messages in core
Good to know. Yeah, if there's a required action item (you *have* to do this) for each wiki, no matter how brief the action, I think it's generally worth the blast. Matt Flaschen On Mon, Dec 3, 2012 at 10:16 PM, Steven Walling steven.wall...@gmail.comwrote: On Mon, Dec 3, 2012 at 7:09 PM, Matthew Flaschen mflasc...@wikimedia.org wrote: Typically, this might be the local version of http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical) or just http://en.wikipedia.org/wiki/Wikipedia:Village_pump (depending how big the wiki is). A bot should be able to deliver this message to all the pages, once the list is in place. Yeah, there is EdwardsBot which will blast all the Village Pumps for us, and I've written/delivered messages myself on a smaller scale. It's just that the only time I've done this in the past was for new features entirely, usually an extension. Using the global message delivery system for a single MediaWiki message change feels like a bit like overkill, which is why asked about precedent. :) Steven ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] TechOps welcomes Mike Wang...
All, Mike Wang is joining us today as our part time Labs Ops Engineer (consultant), based in Tampa, FL. As a member of the Labs team, he will be working closely with Ryan Lane, Andrew Bogott and Faidon Liambotis, to perform day-to-day labs support duties such as systems administration and resolving user issues. Prior to joining us, Mike was a Linux Administrator at Xcira Inc. and Videogroup Distributor Inc., in Tampa. He grew up in China and where he received a B.S. in Mathematics from Shanxi University and a M.S. in Robotics from Beijing Institute of Technology. Please join us to welcome Mike and you can chat with him on our IRC #wikimedia-operations and #wikimedia-labs channels - his nick is 'wangatlargo'. Thanks, CT Woo ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
