[Wikitech-l] Content Security Policy
The good folks at Mozilla are working on what they call Content Security Policy [1], basically a whitelist for JavaScript cross-domain access. I'm just flagging this up here because of the potential benefits from querying toolserver tools from wiki(p|m)edia sites. (and yes, there's JSON, but it's not supported by most tools) Magnus [1] http://people.mozilla.org/~bsterne/content-security-policy/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Content Security Policy
2009/6/30 Magnus Manske magnusman...@googlemail.com: The good folks at Mozilla are working on what they call Content Security Policy [1], basically a whitelist for JavaScript cross-domain access. I'm just flagging this up here because of the potential benefits from querying toolserver tools from wiki(p|m)edia sites. (and yes, there's JSON, but it's not supported by most tools) There's already a wikitech-l thread about that [1]. Roan Kattouw (Catrope) [1] http://lists.wikimedia.org/pipermail/wikitech-l/2009-June/043678.html ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] On templates and programming languages
As many folks have noted, our current templating system works ok for simple things, but doesn't scale well -- even moderately complex conditionals or text-munging will quickly turn your template source into what appears to be line noise. And we all thought Perl was bad! ;) There's been talk of Lua as an embedded templating language for a while, and there's even an extension implementation. One advantage of Lua over other languages is that its implementation is optimized for use as an embedded language, and it looks kind of pretty. An _inherent_ disadvantage is that it's a fairly rarely-used language, so still requires special learning on potential template programmers' part. An _implementation_ disadvantage is that it currently is dependent on an external Lua binary installation -- something that probably won't be present on third-party installs, meaning Lua templates couldn't be easily copied to non-Wikimedia wikis. There are perhaps three primary alternative contenders that don't involve making up our own scripting language (something I'd dearly like to avoid): * PHP Advantage: Lots of webbish people have some experience with PHP or can easily find references. Advantage: we're pretty much guaranteed to have a PHP interpreter available. :) Disadvantage: PHP is difficult to lock down for secure execution. * JavaScript Advantage: Even more folks have been exposed to JavaScript programming, including Wikipedia power-users. Disadvantage: Server-side interpreter not guaranteed to be present. Like Lua, would either restrict our portability or would require an interpreter reimplementation. :P * Python Advantage: A Python interpreter will be present on most web servers, though not necessarily all. (Windows-based servers especially.) Wash: Python is probably better known than Lua, but not as well as PHP or JS. Disadvantage: Like PHP, Python is difficult to lock down securely. Any thoughts? Does anybody happen to have a PHP implementation of a Lua or JavaScript interpreter? ;) -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 12:16 PM, Brion Vibberbr...@wikimedia.org wrote: As many folks have noted, our current templating system works ok for simple things, but doesn't scale well -- even moderately complex conditionals or text-munging will quickly turn your template source into what appears to be line noise. And we all thought Perl was bad! ;) There's been talk of Lua as an embedded templating language for a while, and there's even an extension implementation. One advantage of Lua over other languages is that its implementation is optimized for use as an embedded language, and it looks kind of pretty. An _inherent_ disadvantage is that it's a fairly rarely-used language, so still requires special learning on potential template programmers' part. An _implementation_ disadvantage is that it currently is dependent on an external Lua binary installation -- something that probably won't be present on third-party installs, meaning Lua templates couldn't be easily copied to non-Wikimedia wikis. There are perhaps three primary alternative contenders that don't involve making up our own scripting language (something I'd dearly like to avoid): * PHP Advantage: Lots of webbish people have some experience with PHP or can easily find references. Advantage: we're pretty much guaranteed to have a PHP interpreter available. :) Disadvantage: PHP is difficult to lock down for secure execution. * JavaScript Advantage: Even more folks have been exposed to JavaScript programming, including Wikipedia power-users. Disadvantage: Server-side interpreter not guaranteed to be present. Like Lua, would either restrict our portability or would require an interpreter reimplementation. :P * Python Advantage: A Python interpreter will be present on most web servers, though not necessarily all. (Windows-based servers especially.) Wash: Python is probably better known than Lua, but not as well as PHP or JS. Disadvantage: Like PHP, Python is difficult to lock down securely. Any thoughts? Does anybody happen to have a PHP implementation of a Lua or JavaScript interpreter? ;) -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l I haven't tried it, but there seems to be a Lua Pecl extension. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 9:27 AM, Chadinnocentkil...@gmail.com wrote: I haven't tried it, but there seems to be a Lua Pecl extension. The Lua Pecl says: We should emphasize that is still under development and is completely experimental. That was nearly two years ago and there doesn't appear to have been any real work on it since. Someone would probably need to look at it carefully to make sure it is adequately functional before considering that path. -Robert Rohde ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
Brion Vibber wrote: As many folks have noted, our current templating system works ok for simple things, but doesn't scale well -- even moderately complex conditionals or text-munging will quickly turn your template source into what appears to be line noise. And we all thought Perl was bad! ;) There's been talk of Lua as an embedded templating language for a while, and there's even an extension implementation. One advantage of Lua over other languages is that its implementation is optimized for use as an embedded language, and it looks kind of pretty. An _inherent_ disadvantage is that it's a fairly rarely-used language, so still requires special learning on potential template programmers' part. An _implementation_ disadvantage is that it currently is dependent on an external Lua binary installation -- something that probably won't be present on third-party installs, meaning Lua templates couldn't be easily copied to non-Wikimedia wikis. There are perhaps three primary alternative contenders that don't involve making up our own scripting language (something I'd dearly like to avoid): * PHP Advantage: Lots of webbish people have some experience with PHP or can easily find references. Advantage: we're pretty much guaranteed to have a PHP interpreter available. :) Disadvantage: PHP is difficult to lock down for secure execution. * JavaScript Advantage: Even more folks have been exposed to JavaScript programming, including Wikipedia power-users. Disadvantage: Server-side interpreter not guaranteed to be present. Like Lua, would either restrict our portability or would require an interpreter reimplementation. :P * Python Advantage: A Python interpreter will be present on most web servers, though not necessarily all. (Windows-based servers especially.) Wash: Python is probably better known than Lua, but not as well as PHP or JS. Disadvantage: Like PHP, Python is difficult to lock down securely. Any thoughts? Does anybody happen to have a PHP implementation of a Lua or JavaScript interpreter? ;) -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l I'm working on rewriting abuse filter parser so it's suitable for embedding in wikitext. It's half-done and will be ready soon. -- ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 9:16 AM, Brion Vibberbr...@wikimedia.org wrote: snip There are perhaps three primary alternative contenders that don't involve making up our own scripting language (something I'd dearly like to avoid): snip In the Lua Bugzilla thread (#19298), there was some extended discussion about using the AbuseFilter parser as the basis for a Mediawiki scripting language. From your comment should I assume we are taking that option off the table? There are advantages to that approach in terms of integration and flexibility, though rolling our own scripting language would obviously be a quite complex (and probably long-term) undertaking. -Robert Rohde ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
Robert Rohde wrote: On Tue, Jun 30, 2009 at 9:27 AM, Chadinnocentkil...@gmail.com wrote: I haven't tried it, but there seems to be a Lua Pecl extension. The Lua Pecl says: We should emphasize that is still under development and is completely experimental. That was nearly two years ago and there doesn't appear to have been any real work on it since. Someone would probably need to look at it carefully to make sure it is adequately functional before considering that path. A PECL extension wouldn't be a compatibility improvement over shelling out to a Lua binary; it still requires compilation and installation on the server. (Though it could be a performance win by having the Lua interpreter available in-process.) -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
Robert Rohde wrote: In the Lua Bugzilla thread (#19298), there was some extended discussion about using the AbuseFilter parser as the basis for a Mediawiki scripting language. From your comment should I assume we are taking that option off the table? There are advantages to that approach in terms of integration and flexibility, though rolling our own scripting language would obviously be a quite complex (and probably long-term) undertaking. Right, that's exactly what I don't want to have to do. I'd honestly rather implement a JS interpreter in PHP than create and maintain our own programming language, if it came to that. :) -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 10:03 AM, Trevor Parscaltpars...@wikimedia.org wrote: On 6/30/09 9:16 AM, Brion Vibber wrote: Any thoughts? Does anybody happen to have a PHP implementation of a Lua or JavaScript interpreter? ;) -- brion GPL, Alpha software, seems to be abandoned in 2005 http://j4p5.sourceforge.net/ Perhaps this could be tested, considered, brought back to life, etc? Google also turns up http://phpjs.berlios.de/ Like J4P5 it also seems to be an abandoned alpha. For a stand-alone JavaScript interpreters there are some well-supported projects, like Jaxer: http://www.aptana.com/jaxer However, this again gets back to separately compiled code, and would not easily be able to interact with PHP. -Robert Rohde ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
Brion Vibber wrote: Victor Vasiliev wrote: I'm working on rewriting abuse filter parser so it's suitable for embedding in wikitext. It's half-done and will be ready soon. Eh, I'd rather replace the AbuseFilter scripting with JS/Lua/Python/whatever too. :) -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l We'll anyway have to remove for() and while() from it and restrict it in other ways. --vvv ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
So far my favorite idea is to use a restricted subset of PHP.
I would like to broach an important topic however: How can we convert all of
the existing ParserFunctions and difficult-to-read template code to this new
language automatically? Are we really talking about the dream of getting rid
of templates entirely? The end of {{||}} ?
How difficult would it be to modify the parser to spit out some of its data
structures in the new language as opposed to HTML etc.. ?
This seems to be the more difficult part of the project.
There is a more practical/pragmatic approach which is to deprecate the
current syntax similar to the way languages sometimes deprecate language
features. I fear that the conversion is a superhuman task, however.
On Tue, Jun 30, 2009 at 10:16 AM, Brion Vibber br...@wikimedia.org wrote:
As many folks have noted, our current templating system works ok for
simple things, but doesn't scale well -- even moderately complex
conditionals or text-munging will quickly turn your template source into
what appears to be line noise.
And we all thought Perl was bad! ;)
There's been talk of Lua as an embedded templating language for a while,
and there's even an extension implementation.
One advantage of Lua over other languages is that its implementation is
optimized for use as an embedded language, and it looks kind of pretty.
An _inherent_ disadvantage is that it's a fairly rarely-used language,
so still requires special learning on potential template programmers' part.
An _implementation_ disadvantage is that it currently is dependent on an
external Lua binary installation -- something that probably won't be
present on third-party installs, meaning Lua templates couldn't be
easily copied to non-Wikimedia wikis.
There are perhaps three primary alternative contenders that don't
involve making up our own scripting language (something I'd dearly like
to avoid):
* PHP
Advantage: Lots of webbish people have some experience with PHP or can
easily find references.
Advantage: we're pretty much guaranteed to have a PHP interpreter
available. :)
Disadvantage: PHP is difficult to lock down for secure execution.
* JavaScript
Advantage: Even more folks have been exposed to JavaScript programming,
including Wikipedia power-users.
Disadvantage: Server-side interpreter not guaranteed to be present. Like
Lua, would either restrict our portability or would require an
interpreter reimplementation. :P
* Python
Advantage: A Python interpreter will be present on most web servers,
though not necessarily all. (Windows-based servers especially.)
Wash: Python is probably better known than Lua, but not as well as PHP
or JS.
Disadvantage: Like PHP, Python is difficult to lock down securely.
Any thoughts? Does anybody happen to have a PHP implementation of a Lua
or JavaScript interpreter? ;)
-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 10:15 AM, Brianbrian.min...@colorado.edu wrote:
So far my favorite idea is to use a restricted subset of PHP.
I would like to broach an important topic however: How can we convert all of
the existing ParserFunctions and difficult-to-read template code to this new
language automatically? Are we really talking about the dream of getting rid
of templates entirely? The end of {{||}} ?
How difficult would it be to modify the parser to spit out some of its data
structures in the new language as opposed to HTML etc.. ?
This seems to be the more difficult part of the project.
There is a more practical/pragmatic approach which is to deprecate the
current syntax similar to the way languages sometimes deprecate language
features. I fear that the conversion is a superhuman task, however.
You couldn't ever turn template syntax off without making old
revisions unrenderable. The best one could likely do is encourage
people to upgrade and provide tools to make that easier. However,
given the nastiness of template syntax, I would expect no end of wiki
authors willing to help convert the commonly used stuff.
-Robert Rohde
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 11:20 AM, Robert Rohde raro...@gmail.com wrote: You couldn't ever turn template syntax off without making old revisions unrenderable. The best one could likely do is encourage people to upgrade and provide tools to make that easier. However, given the nastiness of template syntax, I would expect no end of wiki authors willing to help convert the commonly used stuff. -Robert Rohde The solution (no doubt first developed on this list many years ago) is to have mark revisions that still trigger the template/parser functions parser as using that syntax, and to mark revisions that have moved on as using the new language. Moreover, old revisions already are unrenderable. They may look like they render correctly but in fact they don't. This is because mediawiki has no notion of the fact that a particular revision of an article also uses particular revisions of templates etc... At any rate, I don't see how this nitpick is a difficult problem technically. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 10:45 AM, Amir E. Aharoniamir.ahar...@gmail.com wrote: On Tue, Jun 30, 2009 at 20:42, Robert Rohderaro...@gmail.com wrote: On Tue, Jun 30, 2009 at 10:22 AM, Bryan Tong Also for Python you really will want an editor that supports indenting. Web browsers are not suitable for programming Python. Though indenting is mandatory for Python, the use or reasonable indenting is pretty much necessary to produce readable code in any language. That said, I don't see any reason we couldn't use two or three consecutive spaces to indicate indentations. Four! Four is the default size of a tab in Python, but indents aren't actually required to be that size. If we are required to type consecutive spaces to format code, I'd actually prefer a smaller default size. -Robert Rohde ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, 30 Jun 2009 09:16:41 -0700, Brion Vibber wrote: As many folks have noted, our current templating system works ok for simple things, but doesn't scale well -- even moderately complex conditionals or text-munging will quickly turn your template source into what appears to be line noise. And we all thought Perl was bad! ;) There's been talk of Lua as an embedded templating language for a while, and there's even an extension implementation. One advantage of Lua over other languages is that its implementation is optimized for use as an embedded language, and it looks kind of pretty. An _inherent_ disadvantage is that it's a fairly rarely-used language, so still requires special learning on potential template programmers' part. An _implementation_ disadvantage is that it currently is dependent on an external Lua binary installation -- something that probably won't be present on third-party installs, meaning Lua templates couldn't be easily copied to non-Wikimedia wikis. There are perhaps three primary alternative contenders that don't involve making up our own scripting language (something I'd dearly like to avoid): I was thinking about something similar this weekend, although I'd thought about different languages: 1 - XSLT Since the syntax is XML (like the extensions tags) and XPath (vaguely similar to template syntax, although it's XML that calls XPath, the opposite of what we have) It would be reasonably consistent with current syntax. It also should also already be fairly well locked down, and the interface seems fairly clear - present template parameters as stylesheet parameters, and other magic words as an input document. We may just need a few simplifications to make it easier to use. 2- lisp/scheme Should be easy to write a parser for if needed, since the grammer is so simple, and it should be relatively simple to lock down or extend as needed. Of course, those are both a bit more esoteric than your recommendations. Perl is nice for getting useful results from short code, if we're not bothered by one parser with no grammer specification calling another one. Tcl may be a reasonable compromise; a less esoteric, imperative language which is often used as an extension language. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Support for internal images in wikitext?
Just shooting out a quick thought before I forget again -- it would be helpful in a number of places to be able to reference internal software-provided images from wikitext. A primary example is for help pages and inline explanatory text in parts of the UI which we want to be able to customize. This is something that's coming up a lot in things the Usability crew is working on -- folding the wiki markup cheat sheet into mini-help pages accessible from the editor, having screen shots for skins and features so users can see what things are about before enabling them, etc. One simple implementation could be to slip in a custom FileRepo, so they could be referred to just like other images: [[Image:help-markup-example-thumbnail.jpg]] To be slightly more complex, we could give them a unique sub-namespace of some sort, or strip out hardcoded extensions so that types can be more flexible. Skins might want/need to be able to override certain names, too... and we might consider about needs for localized images -- whether RTL/LTR only or actually making translations available. Any thoughts? -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 10:22 AM, Bryan Tong Minhbryan.tongm...@gmail.com wrote: On Tue, Jun 30, 2009 at 6:16 PM, Brion Vibberbr...@wikimedia.org wrote: * Python Advantage: A Python interpreter will be present on most web servers, though not necessarily all. (Windows-based servers especially.) Wash: Python is probably better known than Lua, but not as well as PHP or JS. Disadvantage: Like PHP, Python is difficult to lock down securely. Also for Python you really will want an editor that supports indenting. Web browsers are not suitable for programming Python. Though indenting is mandatory for Python, the use or reasonable indenting is pretty much necessary to produce readable code in any language. That said, I don't see any reason we couldn't use two or three consecutive spaces to indicate indentations. -Robert Rohde ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 20:42, Robert Rohderaro...@gmail.com wrote: On Tue, Jun 30, 2009 at 10:22 AM, Bryan Tong Also for Python you really will want an editor that supports indenting. Web browsers are not suitable for programming Python. Though indenting is mandatory for Python, the use or reasonable indenting is pretty much necessary to produce readable code in any language. That said, I don't see any reason we couldn't use two or three consecutive spaces to indicate indentations. Four! -- Amir Elisha Aharoni http://aharoni.wordpress.com We're living in pieces, I want to live in peace. - T. Moore ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Support for internal images in wikitext?
On Tue, Jun 30, 2009 at 2:09 PM, Brion Vibberbr...@wikimedia.org wrote: Just shooting out a quick thought before I forget again -- it would be helpful in a number of places to be able to reference internal software-provided images from wikitext. A primary example is for help pages and inline explanatory text in parts of the UI which we want to be able to customize. This is something that's coming up a lot in things the Usability crew is working on -- folding the wiki markup cheat sheet into mini-help pages accessible from the editor, having screen shots for skins and features so users can see what things are about before enabling them, etc. One simple implementation could be to slip in a custom FileRepo, so they could be referred to just like other images: [[Image:help-markup-example-thumbnail.jpg]] To be slightly more complex, we could give them a unique sub-namespace of some sort, or strip out hardcoded extensions so that types can be more flexible. Skins might want/need to be able to override certain names, too... and we might consider about needs for localized images -- whether RTL/LTR only or actually making translations available. Any thoughts? -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l A built-in FileRepo would be nice. Pretty straightforward to implement, wouldn't require any major changes to put in place. We could easily put some standard descriptions into the i18n files and load those for the page's info. Make sure to place it higher in the priority than any custom-defined remote repos, or we'll end up loading the remote repo's images that we already have locally. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Support for internal images in wikitext?
Hoi. At the Open Translation Tools conference last week we talked about pictures and icons in a user interface. I learned that some of the obvious symbols like a magnifying glass are not appropriate in other cultures. So I am completely in favour of integrating images in the UI but PLEASE allow for internationalisation and the corresponding localisation to remove images when there is a need. Thanks, GerardM 2009/6/30 Brion Vibber br...@wikimedia.org Just shooting out a quick thought before I forget again -- it would be helpful in a number of places to be able to reference internal software-provided images from wikitext. A primary example is for help pages and inline explanatory text in parts of the UI which we want to be able to customize. This is something that's coming up a lot in things the Usability crew is working on -- folding the wiki markup cheat sheet into mini-help pages accessible from the editor, having screen shots for skins and features so users can see what things are about before enabling them, etc. One simple implementation could be to slip in a custom FileRepo, so they could be referred to just like other images: [[Image:help-markup-example-thumbnail.jpg]] To be slightly more complex, we could give them a unique sub-namespace of some sort, or strip out hardcoded extensions so that types can be more flexible. Skins might want/need to be able to override certain names, too... and we might consider about needs for localized images -- whether RTL/LTR only or actually making translations available. Any thoughts? -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Support for internal images in wikitext?
Hoi, The operational word is replace not remove. Thanks, Gerard 2009/6/30 Gerard Meijssen gerard.meijs...@gmail.com Hoi. At the Open Translation Tools conference last week we talked about pictures and icons in a user interface. I learned that some of the obvious symbols like a magnifying glass are not appropriate in other cultures. So I am completely in favour of integrating images in the UI but PLEASE allow for internationalisation and the corresponding localisation to remove images when there is a need. Thanks, GerardM 2009/6/30 Brion Vibber br...@wikimedia.org Just shooting out a quick thought before I forget again -- it would be helpful in a number of places to be able to reference internal software-provided images from wikitext. A primary example is for help pages and inline explanatory text in parts of the UI which we want to be able to customize. This is something that's coming up a lot in things the Usability crew is working on -- folding the wiki markup cheat sheet into mini-help pages accessible from the editor, having screen shots for skins and features so users can see what things are about before enabling them, etc. One simple implementation could be to slip in a custom FileRepo, so they could be referred to just like other images: [[Image:help-markup-example-thumbnail.jpg]] To be slightly more complex, we could give them a unique sub-namespace of some sort, or strip out hardcoded extensions so that types can be more flexible. Skins might want/need to be able to override certain names, too... and we might consider about needs for localized images -- whether RTL/LTR only or actually making translations available. Any thoughts? -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Support for internal images in wikitext?
On Tue, Jun 30, 2009 at 2:36 PM, Gerard Meijssengerard.meijs...@gmail.com wrote: Hoi, The operational word is replace not remove. Thanks, Gerard 2009/6/30 Gerard Meijssen gerard.meijs...@gmail.com Hoi. At the Open Translation Tools conference last week we talked about pictures and icons in a user interface. I learned that some of the obvious symbols like a magnifying glass are not appropriate in other cultures. So I am completely in favour of integrating images in the UI but PLEASE allow for internationalisation and the corresponding localisation to remove images when there is a need. Thanks, GerardM 2009/6/30 Brion Vibber br...@wikimedia.org Just shooting out a quick thought before I forget again -- it would be helpful in a number of places to be able to reference internal software-provided images from wikitext. A primary example is for help pages and inline explanatory text in parts of the UI which we want to be able to customize. This is something that's coming up a lot in things the Usability crew is working on -- folding the wiki markup cheat sheet into mini-help pages accessible from the editor, having screen shots for skins and features so users can see what things are about before enabling them, etc. One simple implementation could be to slip in a custom FileRepo, so they could be referred to just like other images: [[Image:help-markup-example-thumbnail.jpg]] To be slightly more complex, we could give them a unique sub-namespace of some sort, or strip out hardcoded extensions so that types can be more flexible. Skins might want/need to be able to override certain names, too... and we might consider about needs for localized images -- whether RTL/LTR only or actually making translations available. Any thoughts? -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l This is very true. I don't see any reason why the file repo designed around this cannot handle such scenarios. Either that, or the Language classes a way to load localized images when needed (which the file repo could easily sit on top of) -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 12:16 PM, Brion Vibberbr...@wikimedia.org wrote: * PHP Advantage: Lots of webbish people have some experience with PHP or can easily find references. Advantage: we're pretty much guaranteed to have a PHP interpreter available. :) Disadvantage: PHP is difficult to lock down for secure execution. I think it would be easy to provide a very simple locked-down version, with most of the features gone. You could, for instance, only permit variable assignment, use of built-in operators, a small whitelist of functions, and conditionals. You could omit loops, function definitions, and abusable functions like str_repeat() (let alone exec(), eval(), etc.) from a first pass. This would still be vastly more powerful, more readable, and faster than ParserFunctions. Hopefully, we could make this secure enough for your average shared-host website to run it by default with no special measures taken and without much risk. Installations with more access and higher security requirements, like Wikimedia, could shell out to a process that's sandboxed on the OS level to be on the safe side. I'd like to hear what Tim thinks about the possibility of securing PHP like this. Of course, PHP is evil, and supporting it sucks. :( But if we *really* *really* need to support users who can't shell out to other programs, I think it's the only real language that's a feasible solution. I'd encourage you to consider requiring exec() support for full use of Wikipedia templates, though. Many really big shared hosts allow it, like 1and1.com. Anyone big enough to include much Wikipedia content will likely be on at least a VPS anyway. And if your host doesn't support exec(), then at *worst* you can still get the articles in a totally usable form -- just run Special:ExpandTemplates on all the article's templates. You can then transclude those on a per-article basis; we could update Special:Export to make this easier. The only problem in this case would be that you can't easily change the formatting of all the templates at once -- but such a small site would likely have few enough articles to do it by hand, if they even want to. I think saying that users without exec() support get to use Wikipedia content in a somewhat less usable form would be just fine, and it would *really* open up our options. We could support basically any programming language in that case. * Python Advantage: A Python interpreter will be present on most web servers, though not necessarily all. (Windows-based servers especially.) Wash: Python is probably better known than Lua, but not as well as PHP or JS. Disadvantage: Like PHP, Python is difficult to lock down securely. It doesn't matter whether it's present, does it? If the user has exec() support, they could download a binary interpreter for *any* language to their webspace and run it from there regardless of whether the language is supported on the host. So Python is on exactly the same level as Lua here. Much though I love Python, Lua looks like the better option. First of all, it's *very* small. sudo apt-get install lua50 on my machine uses up only 180 KB of disk space, and the package is 30 KB gzipped. Our current tarballs are 10 MB; we could easily just chuck in Lua binaries for Linux x86-32 and Windows without even noticing the size increase, and allow users to enable it with one line in LocalSettings.php. By contrast, python2.6 is around 10 MB uncompressed, 2.5 MB compressed. Perl is twice that size. Windows users, or users with exec() allowed but open_basedir preventing access to /usr/bin, would have to obtain Python/Perl/etc. themselves. It looks to me like Lua would be a lot easier to sandbox. It seems pretty simple to deny all I/O within the language itself, so you'd (hopefully) just need memory and CPU limits. Both of those could be implemented on Linux with hard setrlimit() values plus nice. Similar things exist on Windows, hopefully accessible by command line somehow. If we're shipping binaries with MediaWiki, we could even hack the code if necessary, to use whatever sandboxing mechanisms the OS makes available, although hopefully that would be unneeded. I don't think we should fixate too much on how many people know the language. It's not hard to pick up a new language if you already know one, and Lua has the reputation of being simple (although I haven't tried to learn it). I think Lua is the best option here. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
Brion Vibber wrote:
Any thoughts? Does anybody happen to have a PHP implementation of a
Lua or JavaScript interpreter?
Rather than reinventing the wheel, why not look at fixing the existing
template syntax?
The biggest problem that I see is the excessive dependence on the
parentheses { and }. In a moderately complex template, you've got a mix
of double {{...}} and triple {{{...}}} brackets, occasionally nested,
that result in an unreadable mess.
If {{{xxx}}} was replaced with a local-variable-like syntax, say $xxx
(where xxx is whatever name you wish, $1, $2... for unnamed), then the
mess is reduced from something like:
{{blah|{{{xxx}}}|{{{yyy}{{#if: {{{ggg}}}|{{{h}}}|{{{4}{{{5}}}
becomes:
{{blah|$xxx|$yyy}}{{#if: $ggg|$h|$4}}$5
which is somewhat more tolerable. (whether or not the above makes real
sense is not my objective - I'm just trying to show how removing the
blizzard of {{{}}} reduces visual clutter).
Since $ doesn't have a close, that makes things like {{{xxx|default
value}}} slightly problematic, since $xxx|$default_value is slightly
more awkward to parse. But that only shows how templates are also
overly reliant on the pipe (|) symbol - as anyone who has tried to use
tables in templates has discovered.
If parsing templates allows the semi-restricted use of a couple of
symbols (unlike parsing other pages - I know... don't go there), then
both {{{}}} and | could be replaced with $ and
I-don't-care-what-make-a-choice. Then templates become a tad more
readable and we get rid of kludges like {{!}} and other clutter or
confusion in tables, parser functions, etc.
As an aside - obliging template writers to declare variables used in the
template, say, as a definition of the input format at the top of the
template definition, would make parsing the variables out later a tad
easier. If it's declared, it's a variable; if not, it's not a variable
and is treated as plain text. Thus the first line of a template would
be the example of its use:
Template:foobar
--
{{Foobar|$var1|$var2|$andAnotherVar}}
...(implementation)...
--
But what do I know, I've only implemented one OO language compiler in my
life and that was 20 years ago.
Mike
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Support for internal images in wikitext?
2009/6/30 Brion Vibber br...@wikimedia.org: Skins might want/need to be able to override certain names, too... and we might consider about needs for localized images -- whether RTL/LTR only or actually making translations available. Like http://laxstrom.name/blag/2008/07/05/localisation-of-images/ ? Or was the question about how to refer to those alternatives in wiki syntax? -- Niklas Laxström ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
Aryeh Gregor wrote: On Tue, Jun 30, 2009 at 12:16 PM, Brion Vibberbr...@wikimedia.org wrote: * PHP Advantage: Lots of webbish people have some experience with PHP or can easily find references. Advantage: we're pretty much guaranteed to have a PHP interpreter available. :) Disadvantage: PHP is difficult to lock down for secure execution. I think it would be easy to provide a very simple locked-down version, with most of the features gone. You could, for instance, only permit variable assignment, use of built-in operators, a small whitelist of functions, and conditionals. You could omit loops, function definitions, and abusable functions like str_repeat() (let alone exec(), eval(), etc.) from a first pass. This would still be vastly more powerful, more readable, and faster than ParserFunctions. IMO by the time you've implemented your whitelisting parser you might as well just interpret it rather than eval()ing. (And of course, eval() might be disabled on the server. :) Looping constructs are also extremely valuable -- at a minimum in a foreach() kind of way. I'd encourage you to consider requiring exec() support for full use of Wikipedia templates, though. Many really big shared hosts allow it, like 1and1.com. Anyone big enough to include much Wikipedia content will likely be on at least a VPS anyway. It's not about Wikipedia content, but about being able to grab things you see on another wiki and use or adapt them to your own needs. We get lots of questions from people trying to grab some particular template off Wikipedia to use on their own site for their own needs. * Python Advantage: A Python interpreter will be present on most web servers, though not necessarily all. (Windows-based servers especially.) Wash: Python is probably better known than Lua, but not as well as PHP or JS. Disadvantage: Like PHP, Python is difficult to lock down securely. It doesn't matter whether it's present, does it? If the user has exec() support, they could download a binary interpreter for *any* language to their webspace and run it from there regardless of whether the language is supported on the host. Considering the amount of trouble people have getting texvc working, I wouldn't want to force that on people just to use templates. Much though I love Python, Lua looks like the better option. First of all, it's *very* small. sudo apt-get install lua50 on my machine uses up only 180 KB of disk space, and the package is 30 KB gzipped. Python comes with batteries included, which is to say it's got a huge standard library (most of which of course wouldn't be available in a restricted environment). Lua's bare interpreter of course wins in an embedded-shipping contest. :D Our current tarballs are 10 MB; we could easily just chuck in Lua binaries for Linux x86-32 and Windows without even noticing the size increase, and allow users to enable it with one line in LocalSettings.php. Hmm... it might be interesting to experiment with something like this, if it can _really_ be compiled standalone. (Linux binary distribution is a hellhole of incompatible linked library versions!) It looks to me like Lua would be a lot easier to sandbox. It seems pretty simple to deny all I/O within the language itself, so you'd (hopefully) just need memory and CPU limits. *nod* being designed as an embedded language is a win. :D -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 10:25 PM, Brion Vibber br...@wikimedia.org wrote: Aryeh Gregor wrote: Our current tarballs are 10 MB; we could easily just chuck in Lua binaries for Linux x86-32 and Windows without even noticing the size increase, and allow users to enable it with one line in LocalSettings.php. Hmm... it might be interesting to experiment with something like this, if it can _really_ be compiled standalone. (Linux binary distribution is a hellhole of incompatible linked library versions!) Static compiling the stuff? How would this affect the binary size? (And: is static linking working across different libc versions?) BTW, what about Mac OS / FreeBSD hosts? Marco -- VMSoft GbR Nabburger Str. 15 81737 München Geschäftsführer: Marco Schuster, Volker Hemmert http://vmsoft-gbr.de ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
2009/6/30 Michael Daly michael.d...@kayakwiki.org:
Brion Vibber wrote:
Any thoughts? Does anybody happen to have a PHP implementation of a
Lua or JavaScript interpreter?
Rather than reinventing the wheel, why not look at fixing the existing
template syntax?
I would support that. We really don't need a Turing-complete template system.
As an aside - obliging template writers to declare variables used in the
template, say, as a definition of the input format at the top of the
template definition, would make parsing the variables out later a tad
easier. If it's declared, it's a variable; if not, it's not a variable
and is treated as plain text. Thus the first line of a template would
be the example of its use:
Template:foobar
--
{{Foobar|$var1|$var2|$andAnotherVar}}
...(implementation)...
--
How does that work with anonymous variables? Are all $[NUMBER] style
names count as auto-declared?
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 4:25 PM, Brion Vibberbr...@wikimedia.org wrote: IMO by the time you've implemented your whitelisting parser you might as well just interpret it rather than eval()ing. I don't think so. You'd only have to do the whitelisting once, on page save. After that you could just execute with no extra overhead. Even better, you could write it to a file and include() the file; this would be a huge win if you have an opcode cache. Of course, parsing PHP within PHP should be much easier than parsing another language within PHP: just use token_get_all() to do most of the work. (And of course, eval() might be disabled on the server. :) Does anyone actually do this? It would break a lot of major web apps, surely. If anyone does do this, it would still work if you could write to a file and then include it. Looping constructs are also extremely valuable -- at a minimum in a foreach() kind of way. Right, but we could live without them in an initial version. They could be added later. It's not about Wikipedia content, but about being able to grab things you see on another wiki and use or adapt them to your own needs. We get lots of questions from people trying to grab some particular template off Wikipedia to use on their own site for their own needs. Sure. The point still holds. Some third parties would be unable to use Wikipedia templates, yes. But given the tangle of dependencies the major ones have, and how complicated they are, I'm guessing most small third-party wikis don't bother in the end anyway. Requiring exec() for full use of content is viable IMO. Considering the amount of trouble people have getting texvc working, I wouldn't want to force that on people just to use templates. The problem with texvc is installing dependencies and compiling it. A much better analogy is things like diff3 -- which we shell out to out of the box, with zero configuration, if they exist and shelling out works. (We'd probably want scripting off by default, of course, but we could require just a single config line.) Python comes with batteries included, which is to say it's got a huge standard library (most of which of course wouldn't be available in a restricted environment). Lua's bare interpreter of course wins in an embedded-shipping contest. :D Yep, but that's a big advantage. It means Windows users don't have to do any extra work. It also lets us ensure a specific version is reliably available. Imagine Wikimedia using Python 2.6, and someone trying to run that on some shared host running Fedora 8 or God knows what, with Python 2.2 or something. (Someone actually came into #mediawiki a few months ago for help and it turned out their VPS was something like Fedora 7 or 8. And horribly overpriced at that!) Hmm... it might be interesting to experiment with something like this, if it can _really_ be compiled standalone. (Linux binary distribution is a hellhole of incompatible linked library versions!) I hadn't thought of libraries, you're right. It should work pretty reliably on Linux (and hopefully not be too much bigger) if it's statically linked, though, right? ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 4:33 PM, Marco Schusterma...@harddisk.is-a-geek.org wrote: Static compiling the stuff? How would this affect the binary size? Hopefully not too badly if you use the right options. libc is huge, but the linker should be able to throw out most of it if statically linking, since Lua likely doesn't use most libc functions. Alternatively, is the libc ABI stable enough that we could dynamically link libc, and statically link everything else? The other libraries required are very small. (And: is static linking working across different libc versions?) Yes, it should work fine, AFAIK. If you statically link everything you're just using the kernel ABIs, which are supposed to be very stable (especially for reasonably common stuff). BTW, what about Mac OS / FreeBSD hosts? Are there any shared webhosts you know of that run Mac or BSD? At worst, they can fall into the same group as the no-exec() camp, able to use Wikipedia content but not 100%. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
-Original Message-
From: wikitech-l-boun...@lists.wikimedia.org
[mailto:wikitech-l-boun...@lists.wikimedia.org] On Behalf Of
Aryeh Gregor
Sent: 30 June 2009 20:56
To: Wikimedia developers
Subject: Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 12:16 PM, Brion
Vibberbr...@wikimedia.org wrote:
* PHP
Advantage: Lots of webbish people have some experience with
PHP or can
easily find references.
Advantage: we're pretty much guaranteed to have a PHP interpreter
available. :)
Disadvantage: PHP is difficult to lock down for secure execution.
I think it would be easy to provide a very simple locked-down
version, with most of the features gone. You could, for
instance, only permit variable assignment, use of built-in
operators, a small whitelist of functions, and conditionals.
You could omit loops, function definitions, and abusable
functions like str_repeat() (let alone exec(), eval(), etc.)
from a first pass. This would still be vastly more powerful,
more readable, and faster than ParserFunctions.
Pity there is not a method of locking down code execution to a single
namespace, (think ahead with php5.3)
namespace Template
{
function strlen($string) { return \strlen($string) * 2; }
function exec() { throw new \Exception(); }
class Template
{
function paint()
{
// Redirect \ namespace to Template, so \exec() is also
caught.
echo strlen('data');
}
}
}
Jared
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 10:45 PM, Aryeh Gregor simetrical+wikil...@gmail.com simetrical%2bwikil...@gmail.com wrote: Alternatively, is the libc ABI stable enough that we could dynamically link libc, and statically link everything else? The other libraries required are very small. I wouldn't count on this... at least we should provide a dyn-linked version for those wanting less storage/memory/whatever consumption. How do statically compiled programs for x86 platforms behave on x64, btw? And what about more exotic platforms like ARM (which can also be multi-endian, IXP4xx is an example) / SPARC (Toolserver!!!) or PowerPC? Are they actually supported by Lua? BTW, what about Mac OS / FreeBSD hosts? Are there any shared webhosts you know of that run Mac or BSD? At worst, they can fall into the same group as the no-exec() camp, able to use Wikipedia content but not 100%. The webhoster hosting our school's homepage does, for example... They host all schools in Munich, and I think they're a bit security-paranoid. We don't have any issues hosting a MediaWiki there, actually. (OK, we never imported WP content.) Marco -- VMSoft GbR Nabburger Str. 15 81737 München Geschäftsführer: Marco Schuster, Volker Hemmert http://vmsoft-gbr.de ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 5:11 PM, Marco Schusterma...@harddisk.is-a-geek.org wrote: How do statically compiled programs for x86 platforms behave on x64, btw? I'm pretty sure they work fine. Someone with more knowledge of Linux binaries needs to comment on how we could best do this, though. And what about more exotic platforms like ARM (which can also be multi-endian, IXP4xx is an example) / SPARC (Toolserver!!!) or PowerPC? Are they actually supported by Lua? Lua is designed to be extremely portable IIRC, across both architectures and compilers. The webhoster hosting our school's homepage does, for example... They host all schools in Munich, and I think they're a bit security-paranoid. That's not a shared host. They can easily install Lua themselves. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, 30 Jun 2009 21:38:07 +0100, Thomas Dalton wrote:
2009/6/30 Michael Daly michael.d...@kayakwiki.org:
How does that work with anonymous variables? Are all $[NUMBER] style
names count as auto-declared?
They're not anonymous, they're just named sequentially. Most languages
should have some method of accessing/declaring those, i.e.
XSL:
xsl:param name=1default/xsl:param
xsl:value-of select=$1/
perl:
my $p=$ARG{1};
print $p;
etc...
If we do roll our own, it should have similar fucntionality.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
2009/6/30 Steve Sanbeg ssan...@ask.com: On Tue, 30 Jun 2009 21:38:07 +0100, Thomas Dalton wrote: 2009/6/30 Michael Daly michael.d...@kayakwiki.org: How does that work with anonymous variables? Are all $[NUMBER] style names count as auto-declared? They're not anonymous, they're just named sequentially. They are anonymous when you call the template, though. The names are determined by the order in the call rather than written explicitly. They do need to be considered separately. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 12:56 PM, Aryeh Gregorsimetrical+wikil...@gmail.com wrote: On Tue, Jun 30, 2009 at 12:16 PM, Brion Vibberbr...@wikimedia.org wrote: * PHP Advantage: Lots of webbish people have some experience with PHP or can easily find references. Advantage: we're pretty much guaranteed to have a PHP interpreter available. :) Disadvantage: PHP is difficult to lock down for secure execution. I think it would be easy to provide a very simple locked-down version, with most of the features gone. You could, for instance, only permit variable assignment, use of built-in operators, a small whitelist of functions, and conditionals. You could omit loops, function definitions, and abusable functions like str_repeat() (let alone exec(), eval(), etc.) from a first pass. This would still be vastly more powerful, more readable, and faster than ParserFunctions. Hopefully, we could make this secure enough for your average shared-host website to run it by default with no special measures taken and without much risk. Installations with more access and higher security requirements, like Wikimedia, could shell out to a process that's sandboxed on the OS level to be on the safe side. I'd like to hear what Tim thinks about the possibility of securing PHP like this. Of course, PHP is evil, and supporting it sucks. :( But if we *really* *really* need to support users who can't shell out to other programs, I think it's the only real language that's a feasible solution. I'd encourage you to consider requiring exec() support for full use of Wikipedia templates, though. Many really big shared hosts allow it, like 1and1.com. Anyone big enough to include much Wikipedia content will likely be on at least a VPS anyway. And if your host doesn't support exec(), then at *worst* you can still get the articles in a totally usable form -- just run Special:ExpandTemplates on all the article's templates. You can then transclude those on a per-article basis; we could update Special:Export to make this easier. The only problem in this case would be that you can't easily change the formatting of all the templates at once -- but such a small site would likely have few enough articles to do it by hand, if they even want to. I think saying that users without exec() support get to use Wikipedia content in a somewhat less usable form would be just fine, and it would *really* open up our options. We could support basically any programming language in that case. * Python Advantage: A Python interpreter will be present on most web servers, though not necessarily all. (Windows-based servers especially.) Wash: Python is probably better known than Lua, but not as well as PHP or JS. Disadvantage: Like PHP, Python is difficult to lock down securely. It doesn't matter whether it's present, does it? If the user has exec() support, they could download a binary interpreter for *any* language to their webspace and run it from there regardless of whether the language is supported on the host. So Python is on exactly the same level as Lua here. Much though I love Python, Lua looks like the better option. First of all, it's *very* small. sudo apt-get install lua50 on my machine uses up only 180 KB of disk space, and the package is 30 KB gzipped. Our current tarballs are 10 MB; we could easily just chuck in Lua binaries for Linux x86-32 and Windows without even noticing the size increase, and allow users to enable it with one line in LocalSettings.php. By contrast, python2.6 is around 10 MB uncompressed, 2.5 MB compressed. Perl is twice that size. Windows users, or users with exec() allowed but open_basedir preventing access to /usr/bin, would have to obtain Python/Perl/etc. themselves. It looks to me like Lua would be a lot easier to sandbox. It seems pretty simple to deny all I/O within the language itself, so you'd (hopefully) just need memory and CPU limits. Both of those could be implemented on Linux with hard setrlimit() values plus nice. Similar things exist on Windows, hopefully accessible by command line somehow. If we're shipping binaries with MediaWiki, we could even hack the code if necessary, to use whatever sandboxing mechanisms the OS makes available, although hopefully that would be unneeded. I don't think we should fixate too much on how many people know the language. It's not hard to pick up a new language if you already know one, and Lua has the reputation of being simple (although I haven't tried to learn it). I think Lua is the best option here. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l In addition to resource limits, any scheme better make sure what's passed into the programming language and what's passed out makes sense. For example, you shouldn't have it generating raw HTML and probably shouldn't
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 11:20 AM, Robert Rohde raro...@gmail.com wrote:
However,
given the nastiness of template syntax, I would expect no end of wiki
authors willing to help convert the commonly used stuff.
-Robert Rohde
I was curious just how terrible of a task conversion can be expected
to be. This is just a heuristic I came up with..
# Simple English parser functions
$ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
22,211
# Simple English templates
$ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{' | wc -l
416,126 - 22,211 = 393,915
# English parser functions
$ bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
430,980
# English templates
$ bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{' | wc -l
44,928,358 - 430,980 = 44,497,378
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On 30/06/2009, at 9:42 PM, Aryeh Gregor wrote: On Tue, Jun 30, 2009 at 4:25 PM, Brion Vibberbr...@wikimedia.org wrote: IMO by the time you've implemented your whitelisting parser you might as well just interpret it rather than eval()ing. I don't think so. You'd only have to do the whitelisting once, on page save. After that you could just execute with no extra overhead. That's just scary. We'd definitely want to do the validation as close as possible to the actual eval()ing, to minimise backdoors like Special:Import et al. -- Andrew Garrett Contract Developer, Wikimedia Foundation agarr...@wikimedia.org http://werdn.us ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Wikipedia downtime?
Was that planned downtime? ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] zh.wikipedia.org says you are now logged in as $1
I logged in to zh.wikipedia.org and it says I am $1. en.wikipedia.org didn't have that problem. 您現在以「$1」的身分登入維基百科。 如果您對維基百科有任何問題的話，請參看幫助頁。 您已登入其他的維基媒體計劃： ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
I would opt for Javascript. PHP and Python are intended for large and complex applications and come with a huge standard library people probably expect to be available. Security concerns are a problem too, so a subset would probably be necessary So, in essence you get a crippled-down language that isn't really useful for templates. Making our own language, either by 'fixing' the template language or by inventing something new would only mean we introduce a new language that'll be specific to our own platform and nobody knows outside of Mediawiki developers. XSLT is not meant to be written or read by humans. It's a Turing-complete language stuffed into horrendous XML statements. Let's not go down that road. That leaves us to Lua and Javascript, which are both small and efficient languages meant to solve tasks like this. Remember, i'm talking about 'core' Javascript here, not with all DOM methods and stuff. If you strip that all out (take a look at the 1.5. core reference at Mozilla.com: https://developer.mozilla.org/en/Core_JavaScript_1.5_Reference) you get a pretty nice and simple language that isn't very large. Both would require a new parser and/or installed compilers on the server-side. Compared to the disadvantages of other options, that seems like a pretty small loss for a great win. Javascript is a widely understood and implemented language, with lots of efforts to get it even faster in modern browsers. Every Wikipedia user has a copy of it implemented in their browser and can start experimenting without the need for installing a compiler or a web server. Many people program in Javascript, so you have a huge potential number of people who could start programming Mediawiki templates. And it's already closely tied to the web, so you don't have to invent new ways of dealing with web-specific stuff. So, let's choose Javascript as our new template programming language. Regards, -- Hay On Tue, Jun 30, 2009 at 6:16 PM, Brion Vibberbr...@wikimedia.org wrote: As many folks have noted, our current templating system works ok for simple things, but doesn't scale well -- even moderately complex conditionals or text-munging will quickly turn your template source into what appears to be line noise. And we all thought Perl was bad! ;) There's been talk of Lua as an embedded templating language for a while, and there's even an extension implementation. One advantage of Lua over other languages is that its implementation is optimized for use as an embedded language, and it looks kind of pretty. An _inherent_ disadvantage is that it's a fairly rarely-used language, so still requires special learning on potential template programmers' part. An _implementation_ disadvantage is that it currently is dependent on an external Lua binary installation -- something that probably won't be present on third-party installs, meaning Lua templates couldn't be easily copied to non-Wikimedia wikis. There are perhaps three primary alternative contenders that don't involve making up our own scripting language (something I'd dearly like to avoid): * PHP Advantage: Lots of webbish people have some experience with PHP or can easily find references. Advantage: we're pretty much guaranteed to have a PHP interpreter available. :) Disadvantage: PHP is difficult to lock down for secure execution. * JavaScript Advantage: Even more folks have been exposed to JavaScript programming, including Wikipedia power-users. Disadvantage: Server-side interpreter not guaranteed to be present. Like Lua, would either restrict our portability or would require an interpreter reimplementation. :P * Python Advantage: A Python interpreter will be present on most web servers, though not necessarily all. (Windows-based servers especially.) Wash: Python is probably better known than Lua, but not as well as PHP or JS. Disadvantage: Like PHP, Python is difficult to lock down securely. Any thoughts? Does anybody happen to have a PHP implementation of a Lua or JavaScript interpreter? ;) -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Wikipedia downtime?
Dan Collins wrote: Apparently, for whatever reason, the master database server for enwiki got overloaded. This was following a few updates, which may have (I don't think they're sure yet) caused the problem. What actually happened was the database server had a large number of queries stuck in the 'statistics' status, leading to overload, leading to wiki down. Enwiki was set to read only, and the Almighty Tim, Patron Saint of Master Databases, arrived on the scene to heroically run the master-database switch script. The S1 (enwiki) master database was changed from db14 to db16, and db14 was removed from the slave rotation. From what I understand, db14 will need a swift kick to the power button to make it all jolly and happy again. Ah excellent, you did my summary post for me. :) Lots of threads being in the statistics state seems to be MySQL's way of saying I've fallen and I can't get up. It's unclear exactly what set it off, but basically nothing works well until you restart it. At 52 minutes from start of event, this took us a bit longer than I'd like to resolve -- we had to percolate through a couple levels of alert calls. (Sorry to wake you up early Tim!) A similar event in future should be fixable within a few minutes, thanks to Tim's work on making the master-switch system more foolproof. We're fixing up our internal documentation so all our site ops will now know how to run the database master switch script next time! Only en.wikipedia.org was affected, other than a couple of minutes where we threw the whole site to read-only while figuring out what was going on. -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 4:01 PM, Brianbrian.min...@colorado.edu wrote:
On Tue, Jun 30, 2009 at 11:20 AM, Robert Rohde raro...@gmail.com wrote:
However,
given the nastiness of template syntax, I would expect no end of wiki
authors willing to help convert the commonly used stuff.
-Robert Rohde
I was curious just how terrible of a task conversion can be expected
to be. This is just a heuristic I came up with..
# Simple English parser functions
$ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{#' | wc
-l
22,211
# Simple English templates
$ bunzip2 -c simplewiki-20090623-pages-articles.xml.bz2 | grep -o '{{' | wc -l
416,126 - 22,211 = 393,915
# English parser functions
$ bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{#' | wc -l
430,980
# English templates
$ bunzip2 -c enwiki-20090618-pages-articles.xml.bz2 | grep -o '{{' | wc -l
44,928,358 - 430,980 = 44,497,378
I assume we are primarily talking about replacing template code and
not template calls, per se.
In other words, I assume things like {{fact}} and {{msg | foo is
bar }} will be be basically unchanged on the article side but
rewritten on the implementation side in Template: space. If that is
correct, it would be more useful to simply ask how large Template:
space is rather than counting all the template calls.
-Robert Rohde
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 6:09 PM, Robert Rohderaro...@gmail.com wrote:
In other words, I assume things like {{fact}} and {{msg | foo is
bar }} will be be basically unchanged on the article side but
rewritten on the implementation side in Template: space. If that is
correct, it would be more useful to simply ask how large Template:
space is rather than counting all the template calls.
-Robert Rohde
Mixing the new language with existing wikicode? With a new language I
would like to see the old language go out the door. The end of double
braces.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
Aryeh Gregor wrote: On Tue, Jun 30, 2009 at 4:25 PM, Brion Vibberbr...@wikimedia.org wrote: It's not about Wikipedia content, but about being able to grab things you see on another wiki and use or adapt them to your own needs. We get lots of questions from people trying to grab some particular template off Wikipedia to use on their own site for their own needs. Sure. The point still holds. Some third parties would be unable to use Wikipedia templates, yes. But given the tangle of dependencies the major ones have, and how complicated they are, I'm guessing most small third-party wikis don't bother in the end anyway. That's why we want to fix it! :) It *should* be fairly trivial to fetch a template/plugin sort of thing off of one wiki and put it on another. Consider this as one of our goals for next-gen templating. -- brion ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
2009/7/1 Brian brian.min...@colorado.edu:
On Tue, Jun 30, 2009 at 6:09 PM, Robert Rohderaro...@gmail.com wrote:
In other words, I assume things like {{fact}} and {{msg | foo is
bar }} will be be basically unchanged on the article side but
rewritten on the implementation side in Template: space. If that is
correct, it would be more useful to simply ask how large Template:
space is rather than counting all the template calls.
-Robert Rohde
Mixing the new language with existing wikicode? With a new language I
would like to see the old language go out the door. The end of double
braces.
What would you replace them with? The wikitext used by regular editors
should be as simple as possible, we don't want to require PHP or
Javascript to be used by anyone wanting to add an infobox to an
article.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 8:28 PM, Brianbrian.min...@colorado.edu wrote:
On Tue, Jun 30, 2009 at 6:09 PM, Robert Rohderaro...@gmail.com wrote:
In other words, I assume things like {{fact}} and {{msg | foo is
bar }} will be be basically unchanged on the article side but
rewritten on the implementation side in Template: space. If that is
correct, it would be more useful to simply ask how large Template:
space is rather than counting all the template calls.
-Robert Rohde
Mixing the new language with existing wikicode? With a new language I
would like to see the old language go out the door. The end of double
braces.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Unless we plan on trying to mass-convert not only years of old revisions
but change years-old behavior that millions of users have come to expect?
I would expect _any_ change to keep {{sometemplate}} always working,
even if the mechanics behind it change.
-Chad
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 6:34 PM, Thomas Daltonthomas.dal...@gmail.com wrote: What would you replace them with? The wikitext used by regular editors should be as simple as possible, we don't want to require PHP or Javascript to be used by anyone wanting to add an infobox to an article. There is nothing in the OP that indicates that we are keeping the current template code or even that it would be desirable. Whatever facilities the language we choose has for including other files and passing arguments to functions is 100% sufficient. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 6:43 PM, Thomas Daltonthomas.dal...@gmail.com wrote: There is no proposal to replace wikitext with PHP (it wouldn't even work, PHP isn't a markup language, ditto Javascript, Python, etc.), the proposal is to replace the template code, ie. the code on the template pages. The OP does not say it is a recommendation to replace ParserFunctions, it says, our current templating system. In my mind that absolutely includes the use of templates in the article namespace. There are lots of usability improvements that can be made to the templating system. First and foremost the new system should allow advanced wiki users to perform programmatic operations on article data without the requirement that the data in the article be made unreadable. If we only focus our efforts on making the template namespace more complicated by giving it a more advanced programming language and we leave the article namespace as it is then we have not even touched the usability issue. We have just made it worse. I do of course have some specific ideas about how to achieve this goal, but I'm kind of in shock and awe that it's not seen as the main reason for improving the template system! ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
http://jtemplates.tpython.com/ ? :)
Trevor Parscal wrote:
Seems like JSON syntax is pretty simple and could be a big improvement
to how templates are currently invoked.
Bottom line, a well defined syntax like JavaScript is going to be more
user friendly than a syntax which is only defined by the behavior of a
parser with standardization at all.
- Trevor
Sent from my iPod
On Jun 30, 2009, at 5:34 PM, Thomas Dalton thomas.dal...@gmail.com
wrote:
2009/7/1 Brian brian.min...@colorado.edu:
On Tue, Jun 30, 2009 at 6:09 PM, Robert Rohderaro...@gmail.com
wrote:
In other words, I assume things like {{fact}} and {{msg | foo is
bar }} will be be basically unchanged on the article side but
rewritten on the implementation side in Template: space. If that is
correct, it would be more useful to simply ask how large Template:
space is rather than counting all the template calls.
-Robert Rohde
Mixing the new language with existing wikicode? With a new language I
would like to see the old language go out the door. The end of double
braces.
What would you replace them with? The wikitext used by regular editors
should be as simple as possible, we don't want to require PHP or
Javascript to be used by anyone wanting to add an infobox to an
article.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 6:08 PM, Robert Rohderaro...@gmail.com wrote: In addition to resource limits, any scheme better make sure what's passed into the programming language and what's passed out makes sense. For example, you shouldn't have it generating raw HTML and probably shouldn't let it mess with strip markers. Some of this may be automatic depending how it's integrated into the parser. One would probably also want to limit the size of an allowed output (e.g. don't let it send 5 MB to the user). Depending on the integration there may be other control sequences that one needs to catch when it returns as well. I was assuming it would just return wikitext, and that would be integrated into the page and parsed, following all limits on wikitext (including size) -- just as with current parser functions. On a separate point, one of the limitations of stand-alone type sandboxes is that it would make it harder for the code to call other template pages. One of the few virtues of the current template code is that it is relatively modular, with more complex templates being built out of less complex ones. If this programming language is meant to replace that then it would also need to be able to reference the results of other template pages. One solution is to pre-expand those sections (similar to what is done now, I believe), but that can get rather delicate once one has programming constructs like variable assignments, looping, and recursion since the template parameters won't necessarily be fixed at the Preprocessor stage. I'd assume we'd support some kind of includes. One rudimentary way to do it would be to run Lua stuff after or during preprocessing, so you could just include Lua code macro-style using templates. A better way would probably be to support the include features of the language itself (I don't know how they work offhand, for Lua). On Tue, Jun 30, 2009 at 6:12 PM, Jared Williamsjared.willia...@ntlworld.com wrote: Yeah, would also need time mem use restrictions. Which is impossible for in-process use. You'd have to shell out if you do that, which defeats the entire point of using PHP instead of something else to begin with. On Tue, Jun 30, 2009 at 7:16 PM, Andrew Garrettagarr...@wikimedia.org wrote: That's just scary. We'd definitely want to do the validation as close as possible to the actual eval()ing, to minimise backdoors like Special:Import et al. You'd be saving the code to a file on disk somewhere, probably named using a hash of the input. The only thing saving the code would be the code that sanitizes it. There's no way anything could go wrong unless an attacker gains filesystem write access, in which case you're hosed anyway. Parsing PHP on every page view when you could cache it in APC is crazy. On Tue, Jun 30, 2009 at 7:24 PM, Hay (Husky)hus...@gmail.com wrote: That leaves us to Lua and Javascript, which are both small and efficient languages meant to solve tasks like this. Remember, i'm talking about 'core' Javascript here, not with all DOM methods and stuff. If you strip that all out (take a look at the 1.5. core reference at Mozilla.com: https://developer.mozilla.org/en/Core_JavaScript_1.5_Reference) you get a pretty nice and simple language that isn't very large. Both would require a new parser and/or installed compilers on the server-side. Compared to the disadvantages of other options, that seems like a pretty small loss for a great win. Reasonable enough, yeah. Sandboxing might easier too. What are some standalone JavaScript interpreters we could use? Ideally we'd use a heavily-optimized JIT compiler, like V8 or TraceMonkey, but I don't know if those work standalone. On Tue, Jun 30, 2009 at 8:33 PM, Brion Vibberbr...@wikimedia.org wrote: That's why we want to fix it! :) It *should* be fairly trivial to fetch a template/plugin sort of thing off of one wiki and put it on another. Consider this as one of our goals for next-gen templating. Eh. Then that really ties our hands. If we have to have support for shared hosts without exec() support, then I don't see any viable option except sanitized PHP. On Tue, Jun 30, 2009 at 8:37 PM, Brion Vibberbr...@wikimedia.org wrote: Executing PHP from apache-writable files saved on disk is also a security danger. The original implementation of the MonoBook skin used the TAL templating language, which was compiled into executable PHP at runtime and stored in /tmp so it could be cached for the next view. In addition to difficulties with hosts which had misconfigured /tmp directories, we found that people sharing their hosts with poorly-secured WordPress installations would end up finding their wikis hacked -- worms exploiting vulnerabilities in other PHP apps would hop around the system modifying any .php files they could write to... including the cached PHPTAL templates. It could be eval()ed by default, but the performance wins from using APC would surely be huge.
Re: [Wikitech-l] On templates and programming languages
I don't know about scripting languages for the templating, it might be an
overkill.
When I was picking lower language for MediaWiki Widgets extension, I looked
at popular PHP templating systems and ended up picking Smarty (
http://smarty.net/) - it can be security locked, it has a few useful
features.
You can see Widget code here:
http://www.mediawikiwidgets.org/w/index.php?title=Widget:Google_Calendaraction=editand
widget is called using a parser function like this: {{widget:
Name|param=val|param2=val2}}.
Double curlys are far from perfect, but there are not that many good
alternatives - XML is probably the only good alternative because it's
universal and use by many-many tools out there. Can't say that I'm an expert
in templating languages though, especially when we're talking about
power-users and not developers.
Thank you,
Sergey
--
Sergey Chernyshev
http://www.sergeychernyshev.com/
On Tue, Jun 30, 2009 at 12:16 PM, Brion Vibber br...@wikimedia.org wrote:
As many folks have noted, our current templating system works ok for
simple things, but doesn't scale well -- even moderately complex
conditionals or text-munging will quickly turn your template source into
what appears to be line noise.
And we all thought Perl was bad! ;)
There's been talk of Lua as an embedded templating language for a while,
and there's even an extension implementation.
One advantage of Lua over other languages is that its implementation is
optimized for use as an embedded language, and it looks kind of pretty.
An _inherent_ disadvantage is that it's a fairly rarely-used language,
so still requires special learning on potential template programmers' part.
An _implementation_ disadvantage is that it currently is dependent on an
external Lua binary installation -- something that probably won't be
present on third-party installs, meaning Lua templates couldn't be
easily copied to non-Wikimedia wikis.
There are perhaps three primary alternative contenders that don't
involve making up our own scripting language (something I'd dearly like
to avoid):
* PHP
Advantage: Lots of webbish people have some experience with PHP or can
easily find references.
Advantage: we're pretty much guaranteed to have a PHP interpreter
available. :)
Disadvantage: PHP is difficult to lock down for secure execution.
* JavaScript
Advantage: Even more folks have been exposed to JavaScript programming,
including Wikipedia power-users.
Disadvantage: Server-side interpreter not guaranteed to be present. Like
Lua, would either restrict our portability or would require an
interpreter reimplementation. :P
* Python
Advantage: A Python interpreter will be present on most web servers,
though not necessarily all. (Windows-based servers especially.)
Wash: Python is probably better known than Lua, but not as well as PHP
or JS.
Disadvantage: Like PHP, Python is difficult to lock down securely.
Any thoughts? Does anybody happen to have a PHP implementation of a Lua
or JavaScript interpreter? ;)
-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] can't comment on techblog from TW
http://techblog.wikimedia.org/wp-comments-post.php says: Your location has been identified as part of a reported spam network. Comments have been disabled to prevent spam. $ whois 122.127.35.74 ...country: TW What should I do, upgrade my country? ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
On Tue, Jun 30, 2009 at 10:45 PM, Sergey Chernyshevsergey.chernys...@gmail.com wrote: I don't know about scripting languages for the templating, it might be an overkill. People are using ParserFunctions as a scripting language already. That's not feasibly going to be removed at this point. So the only way to go is to replace it with a better scripting language, which is what we're talking about. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
Brion Vibber wrote:
There's been talk of Lua as an embedded templating language for a while,
and there's even an extension implementation.
One advantage of Lua over other languages is that its implementation is
optimized for use as an embedded language, and it looks kind of pretty.
An _inherent_ disadvantage is that it's a fairly rarely-used language,
so still requires special learning on potential template programmers' part.
An _implementation_ disadvantage is that it currently is dependent on an
external Lua binary installation -- something that probably won't be
present on third-party installs, meaning Lua templates couldn't be
easily copied to non-Wikimedia wikis.
There are problems with all the shell-based solutions. MediaWiki
callbacks, like template expansion, {{VARIABLES}} and ifexist, are
commonly used in templates on Wikipedia, and a scripting language
without these would suffer from poor community buy-in. You could
implement them from the shell using IPC, but IPC in PHP is rather
cumbersome. The interface between the parser and the scripting engine
would be performance-sensitive, because users would write templates
that invoked the scripting engine hundreds of times in the course of
rendering an article. So there's a case there for a persistent
scripting engine with a command-based interface over a pipe.
The reason I like Lua is because of the potential to embed it in PHP
as an extension, with fast setup and fast callbacks to MediaWiki. It
does all its memory allocation via a callback to the application,
including VM stack space, which means that it's possible to control
the memory usage without killing the process when the limit is
exceeded. But its standard library is unsuitable for running untrusted
scripts, since it contains all the usual process control and file
read/write functions.
The current PECL extension doesn't have any of the features that make
Lua attractive: it does not have support for callbacks to PHP, or for
replacing the standard library with something more sensible, or for
limiting memory without killing the request when the limit is
exceeded. Obviously the distributed standalone does not have these
features either.
I had imagined the task of embedding Lua in MediaWiki as being
primarily a C project, writing the necessary glue code between the
embedded interpreter and PHP. I had hoped that banging the drum for
Lua might encourage someone to look at these issues and start work on
that project.
* PHP
Advantage: Lots of webbish people have some experience with PHP or can
easily find references.
Advantage: we're pretty much guaranteed to have a PHP interpreter
available. :)
Disadvantage: PHP is difficult to lock down for secure execution.
PHP can be secured against arbitrary execution using token_get_all(),
there's a proof-of-principle validator of this kind in the master
switch script project. But there are problems with attempting a
single-process PHP-in-PHP sandbox:
* The poor support for signals in PHP makes it difficult to limit the
execution time of a script snippet. Ticks only occur at the end of
each statement, so you can defeat them by making a single statement
that runs forever.
* Apart from blacklisting function definition, there is no way to
protect against infinite recursion, which exhausts the process stack
and causes a segfault.
* Memory limits are implemented on a per-request basis, and there's no
way to recover from exceeding the memory limit, the request is just
killed.
* JavaScript
Advantage: Even more folks have been exposed to JavaScript programming,
including Wikipedia power-users.
Disadvantage: Server-side interpreter not guaranteed to be present. Like
Lua, would either restrict our portability or would require an
interpreter reimplementation. :P
* Python
Advantage: A Python interpreter will be present on most web servers,
though not necessarily all. (Windows-based servers especially.)
Wash: Python is probably better known than Lua, but not as well as PHP
or JS.
Disadvantage: Like PHP, Python is difficult to lock down securely.
Any thoughts? Does anybody happen to have a PHP implementation of a Lua
or JavaScript interpreter? ;)
SpiderMonkey and Python both lack control over memory usage. Python
lacks a sandbox mode, the rexec module has been removed. SpiderMonkey
isn't embedded in any useful kind of standalone, so you'd have to
start with a C development project, like you would for Lua.
I think Rhino would be an easier path to JavaScript execution than
SpiderMonkey. You can pass an -Xmx option to the java VM, and it'll
throw an OutOfMemory exception when it hits that limit, allowing you
to implement per-snippet memory limits without killing the
interpreter. You could do wall-clock time limits using
java.util.Timer, or CPU time limits using a JNI hack to poll clock().
You could turn off LiveConnect by making your own ClassShutter,
leaving what (on initial impressions) is
Re: [Wikitech-l] On templates and programming languages
2009/7/1 Michael Daly michael.d...@kayakwiki.org: Why not switch the template syntax for articles to match the syntax for tags (which in turn is based on XML or whatever syntax that comes from ultimately)? What is wrong with the current syntax for calling templates? At least, what is wrong with it that would be improved by that change? ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] On templates and programming languages
1 - XSLT Since the syntax is XML (like the extensions tags) and XPath (vaguely similar to template syntax, although it's XML that calls XPath, the opposite of what we have) It would be reasonably consistent with current syntax. It also should also already be fairly well locked down, and the interface seems fairly clear - present template parameters as stylesheet parameters, and other magic words as an input document. We may just need a few simplifications to make it easier to use. XSLT itself is a way too much locked down - even simple things like substrings manipulation and loops aren't so easy to perform. Well, maybe I am too stupid for XSLT but from my experience bringing tag syntax in programming language make the code poorly readable and bloated. I've used XSLT for just one of my projects. 2- lisp/scheme Should be easy to write a parser for if needed, since the grammer is so simple, and it should be relatively simple to lock down or extend as needed. Deeply nested braces of lisp remind me of current MediaWiki parser. Of course, those are both a bit more esoteric than your recommendations. Perl is nice for getting useful results from short code, if we're not bothered by one parser with no grammer specification calling another one. Tcl may be a reasonable compromise; a less esoteric, imperative language which is often used as an extension language. Lua was highly valued here at computer lab, also Ocaml (not sure of proper spelling). Dmitriy ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
