Christopher Lenz skrev 08. april 2010 11:38:
On 08.04.2010, at 09:23, Eirik Schwenke wrote:
(...)
I'm a bit confused -- why wouldn't we just slap a 0.6 on genshi,
and place it under the trac umbrella ?
Genshi pretty much *is* under the Trac umbrella, and releasing trunk
as 0.6 is the plan.
Ok, that is more or less what I thought, and hence why I was confused ;-)
(...)
Anyway, I think there are three seperate issues, that all warrant
discussion:
a) Is the speed of genshi doomed due to genshis design? And if so,
should we for *speed reasons* give up on genshi ?
The current design is inefficient in a number of ways. The whole
design choice that everything is streamed through the pipeline event
by event (SAX-style) using Python generators has proved to be rather
poor.
Hm, and "everyone" keeps claiming that SAX is pretty efficient,
certainly compared to parsing entire document trees (XLST-style).
This sort of reminds me of this article I recently came across for an
entirely different project:
http://effbot.org/zone/simple-top-down-parsing.htm
(discussing merits of a pure python top-down descent parser,
inspired by the paper by Pratt):
http://portal.acm.org/citation.cfm?doid=512927.512931
Most notably, the parser ends up being *more* efficient than the one in
python, tokenizer part aside.
It would appear that event-based parsing of templates is somewhat
similar to top-down-parsing -- but maybe we're missing something in the
implementation, due to the way functions are called ?
That match templates are processed at render time makes their
use quite expensive, and it certainly doesn't scale to a larger
number of match templates.
It would be possible to move Genshi towards a more efficient
implementation by:
* Dropping support for dynamic XInclude tags, i.e. Genshi would need
to know what gets included at parse time.
> * Moving match template processing to the parsing stage (aka static
> match templates); or alternatively, drop the match template idea
> altogether and move to the more conventional model of template
> inheritance where the master
predefines slots that can be filled.
> * Compiling templates to Python bytecode.
> * Providing a fragment caching system.
Essentially either implement caching (as bytecodes, static templates) or
redesign the templates to another design ? (includes, static/simple
templates)?
It would still not be in the same league as highly optimized text
template engine such as Mako or Jinja2, but I think it would be
totally usable.
Due to the event-based design ?
We should perhaps keep in mind that the latest intel instruction set
includes operators designed for stream processing of unicode strings
(targeting xml, but the opcodes are general text/match instructions):
http://www.strchr.com/strcmp_and_strlen_using_sse_4.2
http://en.wikipedia.org/wiki/SSE4
http://software.intel.com/en-us/articles/xml-parsing-accelerator-with-intel-streaming-simd-extensions-4-intel-sse4
(the last article notes some justifications from Intel on why they
created the new instructions).
Apparently XML is so slow, by design, that Intel found the need to add
new hardware instructions. As of yet AMD hasn't implemented SSE4.2 as
far as I know.
As a point of reference, Genshi trunk is in some
cases actually faster than Django templates in raw throughput when
you don't use things like match templates (last time I tested at
least), and lots of sites use Django templates. I think that doing
the above changes would make Genshi consistently faster than Django
templates (at least the current implementation).
Ok. So it's definitely stacking several match-templates that is slow?
Particularly because there currently is no caching, is that correct?
From looking briefly at the code, it would appear the first step might
be moving towards generating python code (and compiling that to
bytecode), to add caching. Certainly not a trivial task.
Or maybe, if most (current) use cases transform some form of xml to
another form of xml, it would be possible to safely cache intermediate
(and final) versions ? I suppose that would lead to a lot of cached text
for pages with many variables?
That assumes that tokenizing, parsing and building the ASTs (or
equivalent) structures is the slow part?
(...)
c) Does genshi's stream-filters provide enough of an advandage to
push for b) over a) ?
Not sure. Stream filtering is used for a couple things, most
importantly form filling and localization. Removing stream filtering
leaves two alternative ways to achieve these (and similar) tasks: *
post-process the generated output (parse/transform) * take care of
everything right in the template, i.e. manually fill in form values,
gettext() all strings, etc.
Stream filters are nice in that they let you implement "cross-cutting
concerns" outside of the templates, and doesn't require a
post-processing stage. Whether that's worth the cost I don't know.
Having worked with other template system on other projects, I definitely
like the idea of having l10n and i18n *outside* of templates, as far as
that makes sense. Feels like it would lead to easier reuse of other
peoples translations as well.
Thank you for your detailed answers!
Best regards,
--
.---. Eirik Schwenke <[email protected]>
( NSD ) Harald Hårfagresgate 29 Rom 150
'---' N-5007 Bergen tlf: (555) 889 13
GPG-key at pgp.mit.edu Id 0x8AA3392C
--
You received this message because you are subscribed to the Google Groups "Trac
Development" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/trac-dev?hl=en.
