Re: [GSoC] Template Compilation

[Russell Keith-Magee]

On Sun, Mar 27, 2011 at 10:48 AM, G.Boutsioukis  wrote:
> Hi, I'm thinking about submitting a proposal for template compilation
> and I'm posting this as a request for more info.
>
> In particular, I remember this project being discussed last year and I
> pretty much assumed that Alex Gaynor's proposal would have been
> accepted(I see he's listed as a mentor this year BTW). What was the
> rationale behind the decision to reject it? Unless, of course, it was
> made on his part.
>
> In any case, any other comment around compatibility, speed or other
> concerns would also be helpful.

As Karen notes, it's wasn't rejected, Alex changed his project.

However, the level of interest for this proposal is still high (at
least, it is for me). The speed of Django's template language is one
of two major reasons often cited as a reason that it should be
avoided. There is also at least one proposal kicking around at the
moment (template-based form widgets) that would benefit directly from
speed improvements in the template language.

The biggest issue I see with this project -- it is hard, and in order
to be accepted as a GSoC student, we need to have reasonable grounds
to believe that you're capable of delivering.

This wasn't a problem for Alex: he has a history of contributing to
Django itself, including delivering the multi-db implementation for
the GSoC the year before. He is also a committer to PyPy, so even if
he didn't have a history of working with the Django community, it's
clear that he knows how Python's bytecode works at a low level (which
is a relevant skill for this project).

Since the proposal has been largely written and is ready to use in
it's current form, you need to find another way to demonstrate that
you can deliver. For example, you could:

  * Write some proof of concept code for the project
  * Get someone known to the Django community to vouch for your skills
  * Solve some non-trivial bug in Django

Best of luck with your application for GSoC!

Yours
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSoC] Template Compilation

[Karen Tracey]

On Sat, Mar 26, 2011 at 10:48 PM, G.Boutsioukis wrote:

> What was the
> rationale behind the decision to reject it? Unless, of course, it was
> made on his part.
>

It wasn't rejected, Alex withdrew the template compilation proposal in favor
of a non-relational DB project, which he did do for GSOC 2010.

Karen

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSOC] Template Compilation

[Alex Gaynor]

On Thu, Mar 4, 2010 at 6:58 PM, Russell Keith-Magee
 wrote:
> On Fri, Mar 5, 2010 at 12:26 AM, Alex Gaynor  wrote:
>> On Thu, Mar 4, 2010 at 3:41 AM, Russell Keith-Magee
>>  wrote:
>>> On Thu, Mar 4, 2010 at 7:29 AM, Alex Gaynor  wrote:
>
 Templates will be compiled by turning each template into a series of 
 functions,
 one per block (note that the base template from which other templates 
 extend is
 a single function, not one per block).
>>> ...
 After an intermediate representation is created a compiler is invoked which
 translates the IR into Python code. This handles the details of Django 
 template
 scoping, spilling variables in the event of conflicts, and calling 
 template tag
 functions.
>>>
>>> You don't ever say this explicitly, but I presume that the plan is to
>>> do this as an in-memory construction -- that is, the intermediate
>>> representation is handled in-memory, not as a python file on disk.
>>> You're proposing to writing a (2 pass?) compiler, not a code
>>> generator, right?
>>>
>>
>> My plan was an in-memory construction, but there's no reason once we
>> have the infrastructure in place that we couldn't generate templates
>> to disk.  As for a compiler vs. a code generator, a compiler *is* a
>> code generator, so I'm not sure I see the distinction being made.
>
> Ok - sure - a compiler is a code generator. What I'm driving at is
> that you're not proposing that we move to a Cheetah-style model. The
> output of the compiler isn't something the end user should ever need
> to see, and won't ever see under normal circumstances. The compiled
> output isn't something the user will (under normal conditions) have
> the opportunity of modifying and editing, and they won't need to have
> makefiles or some similar construct to make sure their compiled
> templates are up to date. Django template use isn't going to devolve
> into "import mytemplate".
>

That's correct, although as I said there's no reason someone couldn't
utilize the compilation APIs to write them out to disk and do "import
mytemplate".

 To provide an optimal solution we must know which variables a given 
 template
 tag needs, and which variables it can mutate. This can be accomplished by
 attaching a new class attribute to ``Nodes`` and passing only those values 
 to
 the class, (instead of the full context dictionary). Subsequently, we would
 only need to mirror a few given values into the locals, and since these are
 known names, we avoid degrading the local lookups into dictionaries. 
 Old-style
 ``Nodes`` will continue to work, but in a less efficient manner.
>>>
>>> The term "less efficient" needs clarification here -- less efficient
>>> than a compiled node that has been specifically optimized for
>>> compilation, or less efficient than existing template rendering? It's
>>> completely understandable if a node doesn't render as fast as it's
>>> optimized and compiled counterparts, but I would consider it a failure
>>> if a compiled template resulted in a slowdown when compared to
>>> existing non-compiled rendering speed.
>>>
>>
>> less efficient than existing rendering, for that specific node.  This
>> is because the Context dictionary would have to be rematerialized for
>> each and every old-style node in the template.
>
> That's what I expected; I just wanted to make sure we were on the same page.
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>

Alex

-- 
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSOC] Template Compilation

[Russell Keith-Magee]

On Fri, Mar 5, 2010 at 12:26 AM, Alex Gaynor  wrote:
> On Thu, Mar 4, 2010 at 3:41 AM, Russell Keith-Magee
>  wrote:
>> On Thu, Mar 4, 2010 at 7:29 AM, Alex Gaynor  wrote:

>>> Templates will be compiled by turning each template into a series of 
>>> functions,
>>> one per block (note that the base template from which other templates 
>>> extend is
>>> a single function, not one per block).
>> ...
>>> After an intermediate representation is created a compiler is invoked which
>>> translates the IR into Python code. This handles the details of Django 
>>> template
>>> scoping, spilling variables in the event of conflicts, and calling template 
>>> tag
>>> functions.
>>
>> You don't ever say this explicitly, but I presume that the plan is to
>> do this as an in-memory construction -- that is, the intermediate
>> representation is handled in-memory, not as a python file on disk.
>> You're proposing to writing a (2 pass?) compiler, not a code
>> generator, right?
>>
>
> My plan was an in-memory construction, but there's no reason once we
> have the infrastructure in place that we couldn't generate templates
> to disk.  As for a compiler vs. a code generator, a compiler *is* a
> code generator, so I'm not sure I see the distinction being made.

Ok - sure - a compiler is a code generator. What I'm driving at is
that you're not proposing that we move to a Cheetah-style model. The
output of the compiler isn't something the end user should ever need
to see, and won't ever see under normal circumstances. The compiled
output isn't something the user will (under normal conditions) have
the opportunity of modifying and editing, and they won't need to have
makefiles or some similar construct to make sure their compiled
templates are up to date. Django template use isn't going to devolve
into "import mytemplate".

>>> To provide an optimal solution we must know which variables a given template
>>> tag needs, and which variables it can mutate. This can be accomplished by
>>> attaching a new class attribute to ``Nodes`` and passing only those values 
>>> to
>>> the class, (instead of the full context dictionary). Subsequently, we would
>>> only need to mirror a few given values into the locals, and since these are
>>> known names, we avoid degrading the local lookups into dictionaries. 
>>> Old-style
>>> ``Nodes`` will continue to work, but in a less efficient manner.
>>
>> The term "less efficient" needs clarification here -- less efficient
>> than a compiled node that has been specifically optimized for
>> compilation, or less efficient than existing template rendering? It's
>> completely understandable if a node doesn't render as fast as it's
>> optimized and compiled counterparts, but I would consider it a failure
>> if a compiled template resulted in a slowdown when compared to
>> existing non-compiled rendering speed.
>>
>
> less efficient than existing rendering, for that specific node.  This
> is because the Context dictionary would have to be rematerialized for
> each and every old-style node in the template.

That's what I expected; I just wanted to make sure we were on the same page.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSOC] Template Compilation

[Alex Gaynor]

On Thu, Mar 4, 2010 at 3:41 AM, Russell Keith-Magee
 wrote:
> On Thu, Mar 4, 2010 at 7:29 AM, Alex Gaynor  wrote:
>> Django Template Compilation
>> ===
>
> First up, this looks like a solid proposal Alex. Anything that speeds
> up template generation is good, and if your indicative stats turn out
> to be representative of real-world performance, it would be a major
> gain for Django.
>
> I have a couple of quick concerns that pop out. In all likelihood,
> they aren't actually problems, just areas that require clarification
> in your proposal.
>
>> Templates will be compiled by turning each template into a series of 
>> functions,
>> one per block (note that the base template from which other templates extend 
>> is
>> a single function, not one per block).
> ...
>> After an intermediate representation is created a compiler is invoked which
>> translates the IR into Python code. This handles the details of Django 
>> template
>> scoping, spilling variables in the event of conflicts, and calling template 
>> tag
>> functions.
>
> You don't ever say this explicitly, but I presume that the plan is to
> do this as an in-memory construction -- that is, the intermediate
> representation is handled in-memory, not as a python file on disk.
> You're proposing to writing a (2 pass?) compiler, not a code
> generator, right?
>

My plan was an in-memory construction, but there's no reason once we
have the infrastructure in place that we couldn't generate templates
to disk.  As for a compiler vs. a code generator, a compiler *is* a
code generator, so I'm not sure I see the distinction being made.

>> To provide an optimal solution we must know which variables a given template
>> tag needs, and which variables it can mutate. This can be accomplished by
>> attaching a new class attribute to ``Nodes`` and passing only those values to
>> the class, (instead of the full context dictionary). Subsequently, we would
>> only need to mirror a few given values into the locals, and since these are
>> known names, we avoid degrading the local lookups into dictionaries. 
>> Old-style
>> ``Nodes`` will continue to work, but in a less efficient manner.
>
> The term "less efficient" needs clarification here -- less efficient
> than a compiled node that has been specifically optimized for
> compilation, or less efficient than existing template rendering? It's
> completely understandable if a node doesn't render as fast as it's
> optimized and compiled counterparts, but I would consider it a failure
> if a compiled template resulted in a slowdown when compared to
> existing non-compiled rendering speed.
>

less efficient than existing rendering, for that specific node.  This
is because the Context dictionary would have to be rematerialized for
each and every old-style node in the template.

>> Alternate ideas
>> ---
>>
>> James Bennett has suggested making template compilation something that is
>> opt-in by use of a custom template loader. The major advantage of this 
>> approach
>> is that we can require all ``Nodes`` to specify the names they use, and thus
>> not have to deal with mirroring the locals back and forth, which drastically
>> reduces the possibility of performance regressions.  I am in favor of going
>> with this proposal as it gives us a clear transition between the current
>> implementation and the future (see next section).
>
> I concur. The template system is a criticial part of enough Django
> stacks that I'd like to see this as an explicit opt-in feature.
>
>> Long Term
>> -
>>
>> In the short term almost all template tags will be old-style (consider the
>> number of tags using the "as X" idiom to modify the context). However, a long
>> term goal should be to move to exclusively compiled template tags.  This has
>> several benefits:
>>
>>  * Fewer codepaths to maintain
>>  * No chance for differing semantics (this follows from the first)
>>  * More consistant performance.
>>
>> One of the critical goals of this project will therefore be to develop the 
>> API
>> for template tags in a manner where old-style tags can easily migrate to the
>> new form.
>
> I'd like to see a little more elaboration here - exactly what do you
> mean by an "API for template tags"? Do you have any initial thoughts
> on what this would constitute this API? What sort of problems need to
> be addressed or solved? Does this have any crossover with the eternal
> pony to "make template tags easier to write?"
>

Right now the runtime API for template tags is the render() method of
nodes.  This takes the current context, and can mutate it however it
wants, read any data from the context, and then return some string.
Now, every template tag I've ever seen knows at compile time what
values will be read and written at compile now, however there's no way
for the template system to know what these names are.  Therefore we
want to transition to a runtime API that 

Re: [GSOC] Template Compilation

[Russell Keith-Magee]

On Thu, Mar 4, 2010 at 7:29 AM, Alex Gaynor  wrote:
> Django Template Compilation
> ===

First up, this looks like a solid proposal Alex. Anything that speeds
up template generation is good, and if your indicative stats turn out
to be representative of real-world performance, it would be a major
gain for Django.

I have a couple of quick concerns that pop out. In all likelihood,
they aren't actually problems, just areas that require clarification
in your proposal.

> Templates will be compiled by turning each template into a series of 
> functions,
> one per block (note that the base template from which other templates extend 
> is
> a single function, not one per block).
...
> After an intermediate representation is created a compiler is invoked which
> translates the IR into Python code. This handles the details of Django 
> template
> scoping, spilling variables in the event of conflicts, and calling template 
> tag
> functions.

You don't ever say this explicitly, but I presume that the plan is to
do this as an in-memory construction -- that is, the intermediate
representation is handled in-memory, not as a python file on disk.
You're proposing to writing a (2 pass?) compiler, not a code
generator, right?

> To provide an optimal solution we must know which variables a given template
> tag needs, and which variables it can mutate. This can be accomplished by
> attaching a new class attribute to ``Nodes`` and passing only those values to
> the class, (instead of the full context dictionary). Subsequently, we would
> only need to mirror a few given values into the locals, and since these are
> known names, we avoid degrading the local lookups into dictionaries. Old-style
> ``Nodes`` will continue to work, but in a less efficient manner.

The term "less efficient" needs clarification here -- less efficient
than a compiled node that has been specifically optimized for
compilation, or less efficient than existing template rendering? It's
completely understandable if a node doesn't render as fast as it's
optimized and compiled counterparts, but I would consider it a failure
if a compiled template resulted in a slowdown when compared to
existing non-compiled rendering speed.

> Alternate ideas
> ---
>
> James Bennett has suggested making template compilation something that is
> opt-in by use of a custom template loader. The major advantage of this 
> approach
> is that we can require all ``Nodes`` to specify the names they use, and thus
> not have to deal with mirroring the locals back and forth, which drastically
> reduces the possibility of performance regressions.  I am in favor of going
> with this proposal as it gives us a clear transition between the current
> implementation and the future (see next section).

I concur. The template system is a criticial part of enough Django
stacks that I'd like to see this as an explicit opt-in feature.

> Long Term
> -
>
> In the short term almost all template tags will be old-style (consider the
> number of tags using the "as X" idiom to modify the context). However, a long
> term goal should be to move to exclusively compiled template tags.  This has
> several benefits:
>
>  * Fewer codepaths to maintain
>  * No chance for differing semantics (this follows from the first)
>  * More consistant performance.
>
> One of the critical goals of this project will therefore be to develop the API
> for template tags in a manner where old-style tags can easily migrate to the
> new form.

I'd like to see a little more elaboration here - exactly what do you
mean by an "API for template tags"? Do you have any initial thoughts
on what this would constitute this API? What sort of problems need to
be addressed or solved? Does this have any crossover with the eternal
pony to "make template tags easier to write?"

> Goals
> ~
>
> As with any good project we need some criteria by which to measure success:
>
>  * Successfully compile complete (real world) templates.
>  * Speed up templates. For reference purposes, Jinja2 is about 10-20x faster
>   than Django templates.  My goal is to come within a factor of 2-3 of this.
>  * Complete backwards compatibility.
>  * Develop a complete porting guide for old-style template tags to minimize 
> any
>   pain in the transition.

Looks like an interesting proposal, and certainly one with aims that
make it attractive for Django's core (assuming you can deliver on the
performance gains you suggest are possible).

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.