[Wikitech-l] Changes in the ULS API and modules
Hallo, Several updates were made recently in the UniversalLanguageSelector extension to improve its performance. This means that if you use any of the ULS facilities in your extensions or gadgets, such as web fonts, IMEs, language data functions like getDir and getAutonym, etc., you need to ensure that the modules you need are loaded before you use their functions. We wrote some documentation that explains how to do the fixes: https://www.mediawiki.org/wiki/Universal_Language_Selector/Developing_with_ULS We are aware of ULS usage in Wikibase and VisualEditor, and we could use assistance from the Wikidata and VE developers in discussing, reviewing and testing the changes. It is also used in the Translate and TwnMainPage extensions, which we have fixed ourselves. Particularly important updates are: * https://gerrit.wikimedia.org/r/#/c/93926 , which makes the extension lazy-load some of its functional ResourceLoader modules to make the initial page loading lighter. * https://gerrit.wikimedia.org/r/#/c/94116 , which lazy-loads the uls.data module - language info database and related functions. Please contact us here or on IRC ( #mediawiki, #mediawiki-i18n ) if you have any questions. Thanks! -- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] MediaWiki to Latex Converter
I have a log of what happens on when the commands: sudo apt-get install mediawiki2latex mediawiki2latex -u https://en.wikipedia.org/wiki/Adam_Ries -o AdamRies.pdf are entered on the command line of ubuntu (13.10) Better than TV... Happy to send it to anyone. Fred ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Changes in the ULS API and modules
Amir E. Aharoni wrote: Several updates were made recently in the UniversalLanguageSelector extension to improve its performance. Fantastic! :-) Thank you for all of the work you and others have done to address this. As the MediaWiki platform continues to grow in complexity, it'll be useful to have concrete ways in which to measure both server-side and client-side performance and changes in performance. I'm hoping we'll see a lot of improvement in this area in 2014. We wrote some documentation that explains how to do the fixes: https://www.mediawiki.org/wiki/ULS/DEV And thank you for this as well. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Wikimedia engineering report, October 2013
Hi, The report covering Wikimedia engineering activities in October 2013 is now available. Wiki version: https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/October Blog version: https://blog.wikimedia.org/2013/11/12/engineering-report-october-2013/ We're also proposing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge: https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/October/summary Below is the HTML text of the report. As always, feedback is appreciated on the usefulness of the report and its summary, and on how to improve them. -- Major news in October include: - A report on the Open Access Media Importerhttps://blog.wikimedia.org/2013/10/21/scientific-multimedia-files-get-a-second-life-on-wikipedia/, a tool that transfers multimedia files from scientific publications to Wikimedia Commons; - A request for proposals for a new datacenter in the continental UShttps://blog.wikimedia.org/2013/10/21/rfp-new-datacenter-continental-us/, published by the Operations team; - The creation of the Autonym Fonthttps://blog.wikimedia.org/2013/10/28/the-autonym-font-for-language-names/, which allows language names to be displayed properly without degrading page loading time. *Note: We're also providing a shorter, simpler and translatable version of this report https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/October/summary that does not assume specialized technical knowledge.* Personnel Work with us https://wikimediafoundation.org/wiki/Work_with_us Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles. - Software Engineer - Fundraisinghttp://hire.jobvite.com/Jobvite/Job.aspx?j=oawpXfwM - Software Engineer - Growthhttp://hire.jobvite.com/Jobvite/Job.aspx?j=o8NJXfwl - Software Engineer - Core Featureshttp://hire.jobvite.com/Jobvite/Job.aspx?j=o6NJXfwj - Software Engineer - Language Engineeringhttp://hire.jobvite.com/Jobvite/Job.aspx?j=oH3gXfwH - Software Engineer http://hire.jobvite.com/Jobvite/Job.aspx?j=o09WXfwM - Senior Software Engineer - Team Leadhttp://hire.jobvite.com/Jobvite/Job.aspx?j=oC9OXfwg - Software Engineer Data Analytics (Back End)http://hire.jobvite.com/Jobvite/Job.aspx?j=oLdOXfwt - Dev-Ops Engineer - SREhttp://hire.jobvite.com/Jobvite/Job.aspx?j=ocLCWfwf - Ops Engineer - Labs Contractorhttp://hire.jobvite.com/Jobvite/Job.aspx?j=oIcUXfwv - User Experience Designerhttp://hire.jobvite.com/Jobvite/Job.aspx?j=oO8OXfwr Announcements - Ori Livneh transitioned to the Platform Engineering group as Senior Performance Engineer (announcementhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-October/072325.html ). - Gergő Tisza joined the Platfom Engineering group as Software Engineer on the Multimedia team (announcementhttp://lists.wikimedia.org/pipermail/multimedia/2013-October/07.html ). - Leslie Carr and Ryan Lane were both promoted to the position of Senior Operations Engineer (announcementhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-October/072475.html ). - Rummana Yasmeen joined the Platfom Engineering group as Software Test Engineer on the QA team, working primarily on VisualEditor (announcementhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-October/072758.html ). - Vibha Bamba was promoted to the position of Senior User Experience Designer (announcementhttp://lists.wikimedia.org/pipermail/design/2013-October/001066.html ) Technical Operations *Site infrastructure* The team continued to heavily refactor the Puppet configuration: manifests for MySQL, nginx, SSH, puppetmaster, and several others are now properly organized as modules. Alexandros Kosiaris has made considerable progress towards supporting multiple puppetmasters on our cluster, which will greatly improve Puppet performance. *Wikimedia Labs https://www.mediawiki.org/wiki/Wikimedia_Labs* Andrew Bogott has been working with Yuvaraj Pandian to get the new proxy system properly deployed; this will greatly reduce our need to hand out public IPs to labs users. We're close to hiring a contractor to help with the upcoming migration of Labs from Tampa to Ashburn. Features Engineeringhttps://www.mediawiki.org/wiki/Wikimedia_Features_engineering Editor retention: Editing tools *VisualEditor https://www.mediawiki.org/wiki/VisualEditor* In October, the VisualEditor team continued to improve the stability and performance of the system, and add new features. The deployed version of the code was updated five times (1.22-wmf20https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf20#VisualEditor, 1.22-wmf21https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf21#VisualEditor, 1.22-wmf22https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf22#VisualEditor, 1.23-wmf1
Re: [Wikitech-l] FOSS OPW, 13 candidates (was 11)
Two candidates that have notified their intentions to apply before the deadline did suubmit their applications with a little delay, and we have accepted them as well. This means that we have currently 13 candidates. https://www.mediawiki.org/wiki/Outreach_Program_for_Women/Round_7#Candidates Note that other candidates might be proposed by the OPW organizers as transfers from other projects. The same may happen to our candidates if e.g. we have two good candidates applying to the same project and we will only select one. If you have questions or comments about OPW you can join the Engineering Community team IRC meeting starting at 17:00 UTC, in 90 minutes. Otherwise you can reply here or find us on #wikimedia-dev IRC. On 11/11/2013 03:30 PM, Quim Gil wrote: Eleven candidates will go through the FOSS Outreach Program selection process, starting today. You can check the table of candidates and projects at https://www.mediawiki.org/wiki/Outreach_Program_for_Women/Round_7 Wikimedia plans to send the list of recommended candidates to the OPW organizers by November 18. OPW will announce the candidates selected by November 25 at 19:00 UTC. All mentors and candidates are encouraged to learn about the selection process and the lessons already leaned in previous editions: https://www.mediawiki.org/wiki/Mentorship_programs/Selection_process https://www.mediawiki.org/wiki/Mentorship_programs/Lessons_learned Mentors and the rest of the community are encouraged to comment on the proposals in their related discussion pages. Thank you! -- Quim Gil Technical Contributor Coordinator @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Engineering Community team updates
On 11/11/2013 05:07 PM, Andre Klapper wrote: On Mon, 2013-11-11 at 16:01 -0800, Quim Gil wrote: On 11/05/2013 01:52 PM, Quim Gil wrote: Next week we will have our monthly IRC meeting, just like every second Tuesday of the month. You can propose topics to be discussed at https://www.mediawiki.org/wiki/Engineering_Community_Team/Meetings#2013-11-12 This is happening tomorrow, Nobember 12, at 15:00 UTC on #wikimedia-meetbot Small typo - Make that a 17:00UTC please This is correct. Thank you Andre! Meeting minutes can be found at http://integration-meetbot.instance-proxy.wmflabs.org/wikimedia-meetbot/2013/wikimedia-meetbot.2013-11-12-17.02.html -- Quim Gil Technical Contributor Coordinator @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Changes in the ULS API and modules
On Tue, Nov 12, 2013 at 5:47 AM, MZMcBride z...@mzmcbride.com wrote: Amir E. Aharoni wrote: Several updates were made recently in the UniversalLanguageSelector extension to improve its performance. Fantastic! :-) Thank you for all of the work you and others have done to address this. Seconded. Thank you very much for making this a priority. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Fate of hierarchical list on Special:Allpages
Following change I189ba71de[0], the hierarchical list in Special:Allpages becomes a simple alphabetic pager if the total number of pages exceeds a safety threshold. The threshold is designed to protect wikis on which the load generated by the process of generating the hierarchical list would be prohibitively expensive (bug 56840[1]). I189ba71de resolved the immediate operational issue, but there is a further question of whether we want to keep the hierarchical list at all, especially given that it cannot be enabled (in its current implementation, at least) on larger installations. From my perspective, the ideal outcome of this discussion would be that we agree that the hierarchical list is a poor fit for the MediaWiki of today, and we resolve to remove it from core. According to stats.grok.se, enwiki's Special:Allpages receives approximately 158 hits a day.[2] [0]: https://gerrit.wikimedia.org/r/#/c/94690/ [1]: https://bugzilla.wikimedia.org/show_bug.cgi?id=56840 [2]: http://stats.grok.se/en/latest90/Special:Allpages --- Ori Livneh o...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] OPW proposal
https://www.mediawiki.org/wiki/OPW_2013_Flow_Edit_Filter_Integration This is the proposal I submitted for this year's OPW. Please see and suggest changes. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Fate of hierarchical list on Special:Allpages
Ori Livneh wrote: From my perspective, the ideal outcome of this discussion would be that we agree that the hierarchical list is a poor fit for the MediaWiki of today, and we resolve to remove it from core. Sounds good to me. For anyone who doesn't know what a(n) hierarchical list is: https://www.mediawiki.org/wiki/Special:AllPages currently outputs one. Without objection, I'd go ahead and file a bug and mark it as easy. It should be fairly trivial to kill, I think. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Architectural leadership in Wikimedia's technical community
On 11/06/2013 05:50 AM, Ori Livneh wrote: What you are proposing is that we determine who is infallible and benevolent, so we can style them dictators for life. This kind of wide-eyed earnestness about the term architecture is very dangerous. It misses the essential irony, and in so doing it risks transforming an institution which satirizes (and thus curbs) inscrutable authority with one that enshrines it. We don't have any benevolent dictators for life in MediaWiki currently. The three architects don't really fit that term. They do have the final call on whether an RFC should be done (which as you say is good, both because they have experience, and because sometimes we just have to choose without endless discussion). However, other important community decisions (most notably +2 rights) are made by the broader community of developers. I don't see anyone saying we need a BDFL currently, or that architects are infallible. As far as what we should do, I think the status quo is okay for now. It's worth thinking about making more people architects (or replacing a slot if one of the current ones leaves), but it's not currently urgent. Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Architectural leadership in Wikimedia's technical community
On 11/06/2013 06:35 AM, Antoine Musso wrote: I would have a look at the way IETF is handling its RFC process. I wrote about it back in July in the thread proposed RFC process: http://lists.wikimedia.org/pipermail/wikitech-l/2013-July/070241.html The IETF does have a long, successful track record. But I'm not sure this is really a good fit for a single software project. The workflow is as bureaucratic as you can imagine given the number of parties involved and all the political / commercial context that goes behind creating internet standards. You can still achieve a RFC pretty quickly. It has to be more structured, since the scope is so big. The IETF's overall scope is basically, Any standard relevant to the Internet, so they can't simply let the whole technical community on every issue on a single mailing list. First, people who are experts in audio codecs (https://tools.ietf.org/html/rfc6716) are a very different set from the QoS experts (https://tools.ietf.org/html/draft-ietf-dime-qos-parameters). MW is much smaller, so it doesn't have the same problem. Such working groups could work here, but we wouldn't want a new group every time there was an RFC. However, having front-end working groups, database working groups, Wikidata working groups, etc. might work. https://www.mediawiki.org/wiki/Requests_for_comment/LESS Working group was Ori Livneh, Jon Robson, Steven Walling. They came with a draft and implementation after a few review cycles. Along the process Brion stepped in to offer technical reviews/guidance. Brion eventually accepted it by marking the RfC complete and merging the implementation. I don't think this example supports the need for formality. In this case, Ori proposed a POC change, the RFC was created to document why it would be useful and discuss the proposal, a lot of people commented on both the code and the RFC, and Brion eventually merged it. I don't think it would have been better with more discussion before coding, or with more formality (e.g. who is in the working group, versus just a participating engineer). Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Architectural leadership in Wikimedia's technical community
On 11/08/2013 12:00 PM, Bryan Davis wrote: I think the second is more consistent with the tenor of the discussion here so far, because in the first case, the coupling between job titles and responsibilities in our community might be too tight to maintain flexibility and openness. It would also recognize that technical leadership doesn't _just_ mean taking on broad architectural responsibilities. So for example development of unique and mission-critical domain expertise might be another way to progress into Sr. II. I personally think this route (separating the role of architectural leadership from the title/pay band of WMF employees) is the most reasonable way forward. +1 on separating WMF job titles with community technical leadership positions. This will work best if it applies to the current architects too. I.E. all three are changed to Principal Platform/Software/Operations Engineers on the WMF side, while remaining architects on the MW side. I like the Principal Software Engineer and Senior Fellow suggestions for the WMF part. Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] $wgRedactedFunctionArguments
On 10/29/2013 11:48 AM, Zack Weinberg wrote: Theoretically speaking, the right way to do this would be to identify the (small, one hopes) number of *sources* of sensitive data and change them to return objects of a special class, which would then automatically print out as [REDACTED] (if so configured) in a stack trace. This would have other benefits; for instance, the special class could arrange to handle the data extra-carefully (scrubbing it from memory when no longer required, doing constant-time comparisons, that sort of thing) and code that needed to treat the datum as anything other than an opaque blob would have to explicitly unwrap it, which would then be a red flag for code review. I don't agree with this. Whitelists are the preferred approach theoretically, and there have been many cases where blacklists have failed in practice. This is all the more true when the set of possibilities is big (all MW functions). Even if we get all the sensitive functions or data now (difficult), it will probably not hold up for code in extensions, hooks, and just future core changes where no one things of $wgRedactedFunctionArguments Ori is right that blacklists are not safe here. I think traces that show just method names (for public wikis) or (for private wikis, if a config is explicitly set true) everything (no redaction) makes sense. Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Re-implementing PDF support
Hi folks, for a long time we've relied on the mwlib libraries by PediaPress to generate PDFs on Wikimedia sites. These have served us well (we generate 200K PDFs/day), but they architecturally pre-date a lot of important developments in MediaWiki, and actually re-implement the MediaWiki parser (!) in Python. The occasion of moving the entire PDF service to a new data-center has given us reason to re-think the architecture and come up with a minimally viable alternative that we can support long term. Most likely, we'll end up using Parsoid's HTML5 output, transform it to add required bits like licensing info and prettify it, and then render it to PDF via phantomjs, but we're still looking at various rendering options. Thanks to Matt Walker, C. Scott Ananian, Max Semenik, Brad Jorsch and Jeff Green for joining the effort, and thanks to the PediaPress folks for giving background as needed. Ideally we'd like to continue to support printed book generation via PediaPress' web service, while completely replacing the rendering tech stack on the WMF side of things (still using the Collection extension to manage books). We may need to deprecate some output formats - more on that as we go. We've got the collection-alt-renderer project set up on Labs (thanks Andrew) and can hopefully get a plan to our ops team soon as to how the new setup could work. If you want to peek - work channel is #mediawiki-pdfhack on FreeNode. Live notes here: http://etherpad.wikimedia.org/p/pdfhack Stuff will be consolidated here: https://www.mediawiki.org/wiki/PDF_rendering Some early experiments with different rendering strategies here: https://github.com/cscott/pdf-research Some improvements to Collection extension underway: https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/extensions/Collection,n,z More soon, Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] migrating hooks doc to doxygen?
On Sat, Jun 8, 2013 at 8:26 AM, Brad Jorsch bjor...@wikimedia.org wrote: On Sat, Jun 8, 2013 at 11:07 AM, Brian Wolff bawo...@gmail.com wrote: Both Manual:Hooks/foo and all the $wgFoo pages can definitely benefit from some automation. I know the reason I usually don't update the Manual pages is because by the time the change gets reviewed and merged, I don't remember anymore that the change actually requires Manual page updates. I have the same problem with remembering to close bugs; the automated comments posted to the bug on merge now often remind me, at least for bugs I'm on the CC list for. A bit late, but I wrote a script[1] to do this today, since I didn't want to manually create a page for the hook I added to core today[2]. It's not totally perfect, [3] doesn't look that nice, and it's not smart enough to figure out things like the type of an object based on reading the text. If someone were to write a bot that would comment on the changeset after it was merged linking to hook and $wgFoo manual pages that probably need updating, that would serve as a useful reminder. If people like this script, we could have a bot automatically create a stub page on mw.o when the change is merged, and then a human can fix it up. -- Legoktm [1] https://github.com/legoktm/mwhooks/blob/master/create_onwiki.py [2] https://www.mediawiki.org/wiki/Manual:Hooks/GetLogTypesOnUser [3] https://www.mediawiki.org/wiki/Manual:Hooks/WikiPageDeletionUpdates ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
