[Wikitech-l] LabeledSectionTransclusion performance problems
Hello all,
After the new version of LabeledSectionTransclusion (LST) was deployed on
itwikisource, performance issues popped up. itwikisource's main page makes
heavy use of LST, and the new version is clearly heavier than the old one.
In this mail, I'll try to describe the aims of the new version, how the old
version worked and how the new version works.
Aims
---
In the old situation, it was possible to transclude sections of pages by
marking them with section tags. However, it was impossible
to include those tags from within a template. I.e. given
page P: something before section start='a'something with a/section
end='a' something after
page Q: {{#lst:P|a}}
then Q was rendered as
something with a
However, it was not possible to do something like:
page O: ===section start='header'{{{1}}}/section end='header'===
page P: {{O|Some header text}}
page Q: {{#lst:P|header}}
Changes in the #lst parser
--
This was because in the old situation, the #lst mechanism did something
along these lines:
1) get DOM using $parser-getTemplateDom( $title ); - note that this is a
non-expanded DOM, as in templates are not expanded
2) traverse this DOM, find section tags, and call
$parser-replaceVariables() on the relevant sections
In the new situation, the #lst mechanism does something like:
1) get expanded wikitext using
$parser-preprocess({{:page_to_be_transcluded}})
2) get the DOM by calling $parser-preprocessToDom() on the expanded
wikitext
3) traverse this DOM, find section tags, and call
$parser-replaceVariables()
on the relevant sections (unchanged)
One obvious performance issue is that (1) and (2) are not cached - not
within one response (so if a page {{#lst}}'s the same page twice, that page
is processed twice), and not between responses (no caching).
In general, I think it would be preferrable not to do a full parse, but
just to expand the DOM of the templates. Unfortunately, I have not been
able to find a simple way to do this: PPFrame::Expand expands the templates
to their final form, not to an 'expanded DOM'.
I don't know MediaWiki caching well enough to say something about which
caches are used (or not), and what would be an effective caching strategy.
Any ideas on how to do LST without bluntly doing a full page parse for
every transcluded page, or on caching strategies, would be very welcome.
Best,
Merlijn
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] LabeledSectionTransclusion performance problems
On Fri, Nov 30, 2012 at 10:07 AM, Merlijn van Deen valhall...@arctus.nl wrote: After the new version of LabeledSectionTransclusion (LST) was deployed on itwikisource, performance issues popped up. itwikisource's main page makes heavy use of LST, and the new version is clearly heavier than the old one. As a sidenote: because of the performance issues, the most recent changes to the LST extension will probably be reverted today (Friday, November 30). If you made changes to articles or templates to accommodate the new version or benefit from new features, you may want to revert those changes temporarily. -- Guillaume Paumier ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: MediaWiki Groups
Hi, Summary: this proposal is serious and it is being done within the Wikimedia Foundation goals. If you want to start a MediaWiki Group you can help getting the proposals for groups and reps approved and you could be the first one in the row, perhaps in no more than a couple of weeks. :) On 11/29/2012 10:55 PM, Gerard Meijssen wrote: Hoi, How did this come a about? Are there things that are moving in this way or is this an idea that you are floating ? I also find it to be a good idea :) but it's not mine. Having user groups is a goal of the Engineering team at the Wikimedia Foundation. Being the Technical Contributor Coordinator this falls completely in my plate - and I'm happy for that! Actually 3 real user groups are expected to be created before then end of the year (that is, in the following 4 weeks!): https://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals#Milestones_by_quarter_17 Q2 (October-December) Goals for subcommunities: (...) 3 user groups Q4 (April-June) Goals for subcommunities: (...) 5 user groups I expect these groups to become a main delegation/empowerment tool to organize more, more distributed and more diverse technical activities. It fits perfectly in the Wikimedia Foundation strategy and the way the Wikimedia movement grows. As soon as I figure out how budget is requested at the Engineering team ;) I will ask for some in order to help bootstrapping the first groups and related activities. I also want to discuss with the people close to Wikimedia chapters and the new Funds Dissemination Committee how can we help MediaWiki groups to fund their activities through their processes. PS It is not a bad idea. It is just that much easier to do something like this when it supports things on the ground. Absolutely. I didn't want to blow whistles too loud after a first draft that not even myself had re-read yesterday. :) Today I will forward the proposal to the WMF communications / community / legal teams to make sure that whatever we do fits well in the overall picture. Once we have something we broadly agree upon here, I will forward the proposal to wikimedia-l for wider awareness and polishing. Thanks, Gerard On 30 November 2012 01:50, Quim Gil q...@wikimedia.org wrote: Hi, here you have a first draft about MediaWiki Groups, and implicitly MediaWiki reps: http://www.mediawiki.org/wiki/**User:Qgil/MediaWiki_groupshttp://www.mediawiki.org/wiki/User:Qgil/MediaWiki_groups MediaWiki groups organize open source community activities within the scope of specific topics and geographical areas. They extend the capacity of the Wikimedia Foundation in events, training, promotion and other technical activities benefiting Wikipedia, the Wikimedia movement and the MediaWiki software. Imagine MediaWiki Germany Group, MediaWiki Lua Group... These groups may become a significant source of growth and wider diversity of our community. Please bring your ideas to the discussion page - or here. Thank you! -- Quim Gil Technical Contributor Coordinator Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Proposal: MediaWiki Groups
On 11/29/2012 11:47 PM, rupert THURNER wrote: hi quim, you managed to confuse me :) i thought that it is a great idea to finally implement groups, and access control lists in mediawiki as first class citizen, like e.g. moinmoin has it. one enters one ACL line on top of the wiki wiki page see here for details: http://moinmo.in/HelpOnAccessControlLists rupert. On Fri, Nov 30, 2012 at 1:50 AM, Quim Gil q...@wikimedia.org wrote: Hi, here you have a first draft about MediaWiki Groups, and implicitly MediaWiki reps: http://www.mediawiki.org/wiki/User:Qgil/MediaWiki_groups MediaWiki groups organize open source community activities within the scope of specific topics and geographical areas. They extend the capacity of the Wikimedia Foundation in events, training, promotion and other technical activities benefiting Wikipedia, the Wikimedia movement and the MediaWiki software. :) With this name I expected to confuse a few dozens of people in the entire World while they finish the first paragraph of the email. I reality MediaWiki Groups names will be seen in certain context, making that confusion almost impossible: ... presented by Mary Smith, from the MediaWiki HTML5 Group ... the booth will be run by the MediaWiki Bangalore Group volunteers ... the contest is organized by the MediaWiki Group Brazil ... this series of webinars have been produced by the MediaWiki UX Group etc Imagine MediaWiki Germany Group, MediaWiki Lua Group... These groups may become a significant source of growth and wider diversity of our community. Please bring your ideas to the discussion page - or here. Thank you! -- Quim Gil Technical Contributor Coordinator Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Apply for the FOSS Outreach Program for Women internships
Another little update about the Outreach Program for Women: We have decided that we have a lot more good candidates than we can take. We have started communicating to some applicants that they have better chances among other organizations: https://live.gnome.org/OutreachProgramForWomen#Organizations_Looking_for_Applicants In practice this means that our gate for new submissions is basically closed. On 11/15/2012 12:28 PM, Quim Gil wrote: Hi, the FOSS Outreach Program for Women internships has started! The blog post: http://blog.wikimedia.org/2012/11/15/apply-for-the-foss-outreach-program-for-women-internships/ The program: https://live.gnome.org/OutreachProgramForWomen The Wikimedia specific information: https://www.mediawiki.org/wiki/Outreach_Program_for_Women If you are following Wikimedia in Identica, Twitter, G+ or Facebook, you likes and shares are welcome. Please help out reaching out to women in tech out there! -- Quim Gil Technical Contributor Coordinator Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Standardizing highest priority in Bugzilla: IMMEDIATE priority added
Thanks for all the good and valuable feedback, explaining your workflows, and discussing current flaws potential improvements in the long run. For the short term I have now created a priority called Immediate which should be used to identify issues that need immediate attention. This means that teams don't have to change their use of highest. Please set IMMEDIATE PRIORITY when it is appropriate. The documentation at http://www.mediawiki.org/wiki/Bugzilla/Fields has been updated accordingly: * Immediate priority: Must be fixed immediately (means: Drop any other work). Reports should have an assignee set in the Assigned to field. * Highest priority: Should be fixed next by a team. Teams should only have very few issues (preferably one) with highest priority at the same time. Reports should have an assignee set in the Assigned to field. andre -- Andre Klapper | Wikimedia Bugwrangler http://blogs.gnome.org/aklapper/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] problems merging changes from master - remote branch
I am trying to merge a bunch of changes that have occurred in the master branch of the MobileFrontend extension to a remote branch (esisupport), but git review errors out, failing on a change that had been abandoned in the remote branch. First I merged the changes from master to a local checkout of the remote. After fixing merge conflicts, I ran 'git review esisupport' and this is what happened: You have more than one commit that you are about to submit. The outstanding commits are: 503bd3d (HEAD, esisupport) getArticleUrl returns canonical url (bug 41286) 8c70e8b Fix adding unnecessary callbacks in mf-cleanuptemplates snip Is this really what you meant to do? Type 'yes' to confirm: yes remote: Resolving deltas: 100% (315/315) remote: Processing changes: refs: 1, done To ssh:// awjricha...@gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend.git ! [remote rejected] HEAD - refs/publish/esisupport/bug/41286 (change 32896 closed) error: failed to push some refs to 'ssh:// awjricha...@gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend.git ' What is going on? Am I doing this wrong? Thanks for any help! -- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Going to FOSDEM
Are you going to FOSDEM? If so (or if you are considering going) please add yourself to http://www.mediawiki.org/wiki/Events/FOSDEM I still don't know. Depends on whether we have a MediaWiki EU critical mass. -- Quim Gil Technical Contributor Coordinator Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] LabeledSectionTransclusion performance problems
On Fri, Nov 30, 2012 at 4:09 AM, Guillaume Paumier gpaum...@wikimedia.org wrote: On Fri, Nov 30, 2012 at 10:07 AM, Merlijn van Deen valhall...@arctus.nl wrote: After the new version of LabeledSectionTransclusion (LST) was deployed on itwikisource, performance issues popped up. itwikisource's main page makes heavy use of LST, and the new version is clearly heavier than the old one. As a sidenote: because of the performance issues, the most recent changes to the LST extension will probably be reverted today (Friday, November 30). If you made changes to articles or templates to accommodate the new version or benefit from new features, you may want to revert those changes temporarily. Aaron just reverted this a little earlier: https://gerrit.wikimedia.org/r/#/c/36316/ I think the way that he did this, though, means that if we do nothing, then we'll be redeploying this starting December 10th with the start of the 1.21wmf6 cycle. There's a reasonable amount of time between now and then, so we can leave it like this until next week, depending on the result of this conversation. It would be nice to get some feedback from Merlijn about how to solve the problems he's trying to solve in a more efficient way. For those wanting to see the code in question, Merlijn's main commit is here: https://gerrit.wikimedia.org/r/#/c/31330/ For my part, it seems that even just making sure that there is only one extra parse per referenced page might be enough to make this perform acceptably, even if it means keeping all of the parsed transcluded pages around in memory. I'm not sure the preferred method these days (e.g. global, singleton, context object, or attaching to some other existing object), but that may be worth exploring. This is just a wild guess, though; I have no idea if that would be too heavy on our memory usage, and I'm only suggesting it because it seems relatively easy compared to the alternatives. Rob ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
