Hello,
I'm Andrey Zubko, student faculty of Computer Science in Zaporozhye
State Engineering Academy. I'm senior chief developer of a local
Internet Service Provider "Telza" which provides Internet and
telephony services. My responsibilities includes enhancing, improving
existent Billing system that is written by myself.
In GSOC program I want to improve template subsystem by integrating
template compilation into python bytecode.
Goals and challenges
Integrating of template compilation has 2 goals. The first goal and
the chief goal is to provide backward-compatibility to templates,
custom tags, custom filters that are already written. The second
goal is to minimize modifications of Django sources to provide more
stability and faster integrating in trunk.
Implementation
Support of template compilation can be achieved by adding method
'compile' to every Node-derived class, and some other classes,
functions, such as django.template.Variable,
django.template.FilterExpression, django.template.NodeList, and others
described further.
1. Node and Node-derived classes modifications
Class Node and derived from it classes should have optional method –
compile, it returns list of python bytecode commands for write in
compiled template. It receives parameter parent_name that will
contains name of uppermost tag/block. Parent_name is a name of
uppermost tag/block that identifies entry in hash blocks_output that
is used for caching all outputs. Only one case can provide parent_name
to be overridden – meet of the '{% block %}' tag in a parent tag, this
tag should override parent_name to its unique name. Approach for
caching all output is used for providing template inheritance ability.
Use of template Inheritance provides following cases :
child template has one sequence of blocks, but parent template has
another sequence of blocks
child template has blocks that haven't specified in parent's template
For solving this cases I provided list blocks_sequence_output that
keeps sequence of blocks outputting. Implementation of 'compile'
method in Node class :
def compile(self,parent_name):
self.generate_name(parent_name)
compiled_code = ["t = Template('%s')" % self.template,
"blocks_output['%s'] = blocks_output['%s'] +
t._render(context)" % (self.parent_name,self.parent_name)]
return compiled_code
As you saw, in purposes of providing backward-compatibility all Node-
derived classes, that haven't implemented 'compile' method, will use
current rendering approach.
Node-derived classes with implemented 'compile' method will be looked
like :
class TextNode(Node):
def compile(self,parent_name):
return ["blocks_output['%s'] = blocks_output['%s'] + \"\"\"%s
\"\"\"" % (self.parent_name,
self.parent_name,
self.s)]
As shown before 'compile' method is not so difficult to implement.
To recursively compilation of Node-derived classes is used method
'compile' in the class NodeList :
class NodeList(list):
def generate_name(self,node):
return id(node)
def compile(self, parent_name=None):
bits = []
for node in self:
if isinstance(node, Node):
if parent_name is None or isinstance(node,
BlockNode) :
if isinstance(node, BlockNode) :
parent_name_new = node.name
else :
parent_name_new = self.generate_name(node)
bits.append("blocks_output['%s']=''" %
parent_name_new)
else :
parent_name_new = parent_name
bits.append(self.compile_node(node, parent_name_new))
else:
bits.append(node)
return mark_safe(''.join([force_unicode(b) for b in bits]))
Also this function is used for initialising blocks_output hashes for
uppermost nodes.
2. Blocks and dynamical inheritance
I have described before Node-derived classes improvements, but
django.template.loader_tags.BlockNode, django.template.loader_tags,
django.template.loader_tages.ExtendsNode and others have big influence
on compiled template architecture design, that's why I have allocated
whole section for them. Internal variables blocks_output,
blocks_output_sequence are used for providing support of dynamic
inheritance. They specify the way how template should be rendered, in
what sequence, what blocks should be outputted to user.
'blocks_output' hash collects blocks or tags outputs, and
blocks_output_sequence saves sequence of block outputting. Because
parent template can contain its own sequence of blocks that is saved
in blocks_output_sequence, we must update the child
blocks_output_sequence without right of overriding it. Because parent
template can contain own blocks that is not present in child template,
we must update child blocks_output hash with not existing items in it.
By solving this requirements class BlockNode will be looked like :
class BlockNode(Node):
def super_compile(self):
compiled_code = ["blocks_output['%s'] = blocks_output['%s'] +
blocks_output_parent['%s']" % (self.name,self.name,self.name)]
return compiled_code
def compile(self,parent_name):
compiled_childs = self.nodelist.compile()
return compiled_childs
blocks_output_parent – is blocks_output hash received from parent
template that contains cached parent output.
For providing support of static and dynamic inheritance class
ExtendsNode should have 'compile' method that will be looked like :
class ExtendsNode(Node):
def get_parent_compiled(self):
if self.parent_name_expr: # dynamical inheritance
parent = self.parent_name_expr.compile()
if not parent :
if self.parent_name_expr:
error_msg = " Got this from the '%s' variable." %
self.parent_name_expr.token
raise TemplateSyntaxError(error_msg)
else : # static inheritance
parent = self.parent_name
if not parent:
error_msg = "Invalid template name in 'extends' tag:
%r." % parent
raise TemplateSyntaxError(error_msg)
if hasattr(parent, 'compile'):
return parent.compile() # parent is a Template object
code = ["blocks_output, blocks_sequence_output_parent =
get_compiled_template(%s,no_output=True)" % self.parent_name,
"blocks_output_parent = blocks_output"]
return code
def compile(self,parent_name):
compiled_parent = self.get_compiled_parent()
return compiled_parent
3. Variables and filters modifications
Support of variables and filters are implemented by
django.template.FilterExpression class. It is wrapped by VariableNode
class – Node-derived class that implements simple call of
FilterExpression instance. For supporting compilation into python
bytecode VariableNode class should contain 'compile' function that
calls 'compile' method of FilterExpression instance :
class VariableNode(Node):
def __compile__(self,parent_name):
return self.filter_expression.compile()
And FilterExpression should have implemented 'compile' method in which
it calls filters compiled functions and variable resolving :
class FilterExpression(object):
def compile(self, ignore_failures=False):
if isinstance(self.var, Variable):
obj =
self.var.compile(ignore_failures,settings.COMPILED_TEMPLATE_STRING_IF_INVALID)
else:
obj = self.var
for func, args in self.filters:
arg_vals = ""
i = 0
for lookup, arg in args:
if i > 0 :
arg_vals = arg_vals + ","
if not lookup:
arg_vals = arg_vals + "\"%s\"" % mark_safe(arg)
else:
arg_vals = arg_vals + "%s" % arg.compile(context)
if getattr(func, 'needs_autoescape', False):
new_obj = "%s(%s,autoescape=context.autoescape,%s)" %
(func.__name__,obj,arg_vals)
else:
new_obj = "%s(%s,%s)" % (func.__name__,obj,arg_vals)
if getattr(func, 'is_safe', False) and isinstance(obj,
SafeData):
obj = "mark_safe(%s)" % new_obj
elif isinstance(obj, EscapeData):
obj = "mark_for_escaping(%s)" % new_obj
else:
obj = new_obj
return [obj]
By the way, 'compile' method FilterExpression has no parent_name param
– it is omitted because FilterExpression can't produce a block – it
can be called only from a block – VariableNode.
And django.template.Variable receives parameters ignore_failures, and
default value for variable :
class Variable(object):
def compile(self,ignore_failures,default_value):
if self.lookups is not None:
value = "resolve(context,'%s',%s,'%s')" %
(self.lookups,ignore_failures,default_value)
else:
# We're dealing with a literal, so it's already been
"resolved"
value = "'%s'" % self.literal
if self.translate:
value = "_(%s)" % value
Using the Variable class introduced us to inner compiled template
function named 'resolve'. 'resolve' function implements finding entry
in context with looking name, value of entry can be simple value,
class, function. 'resolve' receives params :
1.context – context variable that is passed to compiled template
'render' method
2.key – name of looking entry
3.ignore_failures – if ignore_failures is true and finding entry is
failed, then it will return default_value or if it is omitted – None,
otherwise it will raise VariableDoesNotExist exception
4.default_value – default value
Compiled Template Architecture
Compiled template contains two sections :
1.Dependencies section
2.Compiled code section
Dependencies section is section that has default imports, and external
imports of user-written custom tags, filters. External imports are
determined by calling {% load %} tag.
Compiled code section is a section that contains template's python
byte code.
Structure of the compiled code :
# -- DEPENDENCIES SECTION
# ----- Default imports
from django.template import Template, get_compiled_template
from django.template.defaultfilters import *
# ----- User custom imports
# from ... import ...
# -- END OF DEPENDENCIES SECTION
def resolve(context,ignore_failures,default_value):
""" should be looked like method _resolve_lookup in
django.template.Variable """
blocks_output = {}
blocks_output_sequence = []
blocks_sequence_output_parent = None
def output():
for block_name in blocks_output_sequence :
print blocks_output.get(block_name,'')
def render(context,no_output=False):
# START OF COMPILED TEMPLATE PYTHON BYTECODE
# ...
# END OF COMPILED TEMPLATE PYTHON BYTECODE
if blocks_sequence_output_parent is not None :
blocks_sequence_output = blocks_sequence_output_parent
if no_output :
return blocks_output, blocks_output_sequence
else :
output()
Every compiled template has 3 functions :
resolve – discussed earlier
output – task of this function is to output blocks output cache in the
determined sequence
render – entrypoint of compiled template, that contains template's
compiled python bytecode. It supports one optional parameter –
'no_output' – it is used only when called parent template – not to
output, to return cached data.
Examples
Examples based on already written code :
compiling variables and filters :
from django.template import Parser, FilterExpression
token = 'variable|default:"Default value"|date:"Y-m-d"'
p = Parser('')
fe = FilterExpression(token, p)
fe.compile()
=>
'date(default(resolve(context,variable,None,''),"Default value"),"Y-m-
d")'
template compilation sample :
child.html :
{% extends parent %}
{% block first_block %}
{% if True %}
some_info
{% endif %}
{% endblock %}
{% block second_block %}
{% endblock %}
parent.html :
TextNode
{% block first_block %}
{% endblock %}
=>
Compiled child template :
# -- DEPENDENCIES SECTION
# ----- Default imports
from django.template import Template, get_compiled_template
from django.template.defaultfilters import *
# ----- User custom imports
# from ... import ...
# -- END OF DEPENDENCIES SECTION
def resolve(context,key,ignore_failures,default_value):
""" should be looked like method _resolve_lookup in
django.template.Variable """
blocks_output = {}
blocks_output_sequence = []
blocks_sequence_output_parent = None
def output():
for block_name in blocks_output_sequence :
print blocks_output.get(block_name,'')
def render(context,no_output=False):
blocks_output, blocks_sequence_output_parent =
get_compiled_template(resolve(context,'parent',None,''),no_output=True)
blocks_output_parent = blocks_output
blocks_output['first_block'] = '' # entering block tag in NodeList
compile
blocks_sequence_output.append('first_block')
if True :
blocks_output['first_block'] = blocks_output['first_block'] +
"some_info"
blocks_output['second_block'] = '' # entering block tag in
NodeList compile
if blocks_sequence_output_parent is not None :
blocks_sequence_output = blocks_sequence_output_parent
if no_output :
return blocks_output, blocks_output_sequence
else :
output()
Compiled parent template :
# -- DEPENDENCIES SECTION
# ----- Default imports
from django.template import Template, get_compiled_template
from django.template.defaultfilters import *
# ----- User custom imports
# from ... import ...
# -- END OF DEPENDENCIES SECTION
def resolve(context,key,ignore_failures,default_value):
""" should be looked like method _resolve_lookup in
django.template.Variable """
blocks_output = {}
blocks_output_sequence = []
blocks_sequence_output_parent = None
def output():
for block_name in blocks_output_sequence :
print blocks_output.get(block_name,'')
def render(context,no_output=False):
blocks_output['first_block'] = '' # entering block tag in NodeList
compile
blocks_sequence_output.append('first_block')
if blocks_sequence_output_parent is not None :
blocks_sequence_output = blocks_sequence_output_parent
if no_output :
return blocks_output, blocks_output_sequence
else :
output()
Milestones
1. Designing concept - current step
2. Coding "template compilation" subsystem
3. Writing unit-tests
4. Optimization of written code
The code examples here are not final, but intermediates, to show the
basic algorithms of creating and execution of compiled templates.
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
