Re: Templates, filesystem, caching?
On Sat, 2007-12-22 at 23:00 -0800, Rob Hudson wrote: [...] > I wonder if the template system has become a bit more complex since > then. I also wonder if whatever tests they used included things like > includes in for loops. I tend to think that the filesystem is slow > and anything to remove FS calls and shove things in memory is a good > thing If only there was some way to test all these wonderings. Hmm ... perhaps some kind of timing test over real world applications... we could call it "profiling". Work out how much the template parsing contributew to the overall request time (use some real world apps so that the not insignificant database portion doesn get omitted) and comapre before and afterwards. Then we'd have hard data, rather than hypotheses. :-) People tend to think I'm against these types of things on principle, which couldn't be further from the truth. However, I trust in science. Prove that it's a *significant* improvement for the level of added complexity (and that the simpler solution of wrapping exist template loaders in a caching test doesn't get even close to the same performance) and then we've got something to work with. Malcolm -- How many of you believe in telekinesis? Raise my hand... http://www.pointy-stick.com/blog/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
Template loader returns a source code for template, not parsed Template object. So no big gain here :( > In the early days of Django, Adrian, Simon et al looked at that. It > wasn't worth it, since, in the grand scheme of things, template caching > and checking the cache wasn't that much faster than loading and parsing, > particularly in the overall response time of a request (of which > template parsing is a relatively small component). Adding complexity for > minimal gain isn't usually a good idea. Unless this is a universal win, > it would be better to write it as a third-party template loader. It's > fairly easy to write a template loader that takes another template > loader as a parameter and just wraps caching around it and that keeps > the core code cleaner. > > Regards, > Malcolm > > -- > The cost of feathers has risen; even down is > up!http://www.pointy-stick.com/blog/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
Hi. In production environment IMHO it really doesn't matter because you have to restart server anytime you change the python code. If you treat templates as code that really doesn't matter. We have about 50 production servers and we restart a farm every time we change the code. We do it progressively so users don't feel the difference. We do it 2-3 times a day. So I think this is not a huge issue when you have to restart server every time you change templates. On Dec 23, 4:55 pm, Eratothene <[EMAIL PROTECTED]> wrote: > Hi forgems! > > Your snippet requires to restart django each time the templates have > changed. Did you try to add checking of template file modification > date in order automatically invalidate cache? What it is performance > of such implementation? Adding this kind of mechanism will increase > performace without any loss of functionality. Such implemention will > definitely be included into django base. Please, try it out. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
On 12/23/07, Eratothene <[EMAIL PROTECTED]> wrote: > Your snippet requires to restart django each time the templates have > changed. Did you try to add checking of template file modification > date in order automatically invalidate cache? What it is performance > of such implementation? Adding this kind of mechanism will increase > performace without any loss of functionality. Such implemention will > definitely be included into django base. Please, try it out. I can't speak for the author of the snippet but I think checking the mtime of a file would defeat the purpose. Plus, if DEBUG=True that snippet bypasses the cache anyway. -Rob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
Hi forgems! Your snippet requires to restart django each time the templates have changed. Did you try to add checking of template file modification date in order automatically invalidate cache? What it is performance of such implementation? Adding this kind of mechanism will increase performace without any loss of functionality. Such implemention will definitely be included into django base. Please, try it out. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
Hi, i'm author of the snippet http://www.djangosnippets.org/snippets/507/. I have used simple dictionary because pickling and unpickling values from the cache are IMHO to expensive and don't give a big speed boost over parsing. Also if you use memcache or filesystem backend for caching (pretty standard thing in production environment) you don't really have a speed boost because you have to use IO to get template (+pickle). In my tests with pretty simple templates, lighttpd and fastcgi django process, with my setup i've got from ~400 rps to almost 900 rps. You could just create a blank app in your project and paste my snippet in __init__.py file of that app, and put that app into your INSTALLED_APPS tuple. Now you can enable and disable template caching with just one line in your settings. On Dec 23, 8:00 am, "Rob Hudson" <[EMAIL PROTECTED]> wrote: > On 12/21/07, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote: > > > In the early days of Django, Adrian, Simon et al looked at that. It > > wasn't worth it, since, in the grand scheme of things, template caching > > and checking the cache wasn't that much faster than loading and parsing, > > particularly in the overall response time of a request (of which > > template parsing is a relatively small component). Adding complexity for > > minimal gain isn't usually a good idea. Unless this is a universal win, > > it would be better to write it as a third-party template loader. It's > > fairly easy to write a template loader that takes another template > > loader as a parameter and just wraps caching around it and that keeps > > the core code cleaner. > > I wonder if the template system has become a bit more complex since > then. I also wonder if whatever tests they used included things like > includes in for loops. I tend to think that the filesystem is slow > and anything to remove FS calls and shove things in memory is a good > thing -- especially something that could potentially be in a for loop. > Obviously there are trade-offs as you mention but the patch didn't > look that complex to me -- actually it was surprisingly straight > forward. > > I've considered applying this patch and testing against a project I'm > working on. Maybe that would help prove to either Django or me that > this is or isn't worth it. > > Thanks, > -Rob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
On 12/21/07, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote: > In the early days of Django, Adrian, Simon et al looked at that. It > wasn't worth it, since, in the grand scheme of things, template caching > and checking the cache wasn't that much faster than loading and parsing, > particularly in the overall response time of a request (of which > template parsing is a relatively small component). Adding complexity for > minimal gain isn't usually a good idea. Unless this is a universal win, > it would be better to write it as a third-party template loader. It's > fairly easy to write a template loader that takes another template > loader as a parameter and just wraps caching around it and that keeps > the core code cleaner. I wonder if the template system has become a bit more complex since then. I also wonder if whatever tests they used included things like includes in for loops. I tend to think that the filesystem is slow and anything to remove FS calls and shove things in memory is a good thing -- especially something that could potentially be in a for loop. Obviously there are trade-offs as you mention but the patch didn't look that complex to me -- actually it was surprisingly straight forward. I've considered applying this patch and testing against a project I'm working on. Maybe that would help prove to either Django or me that this is or isn't worth it. Thanks, -Rob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
On Tue, 2007-12-18 at 15:47 +0100, Michael Elsdörfer wrote: > James Bennett schrieb: > > Manually call get_template() or select_template(), and stuff the > > resulting Template object into a module-global variable somewhere. > > Then just re-use it, calling render() with different contexts, each > > time you need it. > > Is there anything speaking against building that behaviour directly into > Django? In the early days of Django, Adrian, Simon et al looked at that. It wasn't worth it, since, in the grand scheme of things, template caching and checking the cache wasn't that much faster than loading and parsing, particularly in the overall response time of a request (of which template parsing is a relatively small component). Adding complexity for minimal gain isn't usually a good idea. Unless this is a universal win, it would be better to write it as a third-party template loader. It's fairly easy to write a template loader that takes another template loader as a parameter and just wraps caching around it and that keeps the core code cleaner. Regards, Malcolm -- The cost of feathers has risen; even down is up! http://www.pointy-stick.com/blog/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
On Dec 21, 2007 4:50 PM, SmileyChris <[EMAIL PROTECTED]> wrote: > > After reading this thread the other day, I decided to write up a patch > [1]. > > [1] http://code.djangoproject.com/ticket/6262 Of course you did. Always with the patches. :) P.S. Thank you Michael Trier blog.michaeltrier.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
Awesome! What happens if you don't have caching enabled? On 12/21/07, SmileyChris <[EMAIL PROTECTED]> wrote: > > After reading this thread the other day, I decided to write up a patch > [1]. > > [1] http://code.djangoproject.com/ticket/6262 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
After reading this thread the other day, I decided to write up a patch [1]. [1] http://code.djangoproject.com/ticket/6262 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
On 12/18/07, Michael Elsdörfer <[EMAIL PROTECTED]> wrote: > James Bennett schrieb: > > Manually call get_template() or select_template(), and stuff the > > resulting Template object into a module-global variable somewhere. > > Then just re-use it, calling render() with different contexts, each > > time you need it. > > Is there anything speaking against building that behaviour directly into > Django? I was curious of the same thing. The only downside I see is that templates wouldn't "reload" automatically if you changed them. But you could disable this when DEBUG=True. And most people are used to the idea of reloading Apache if there is a change in a .py file. -Rob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
James Bennett schrieb: > Manually call get_template() or select_template(), and stuff the > resulting Template object into a module-global variable somewhere. > Then just re-use it, calling render() with different contexts, each > time you need it. Is there anything speaking against building that behaviour directly into Django? Michael --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
Manually call get_template() or select_template(), and stuff the >resulting Template object into a module-global variable somewhere. >Then just re-use it, calling render() with different contexts, each >time you need it. Along those lines, I recently noticed this bit of code that swaps the get_template fn out with one that does some caching: http://www.djangosnippets.org/snippets/507/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
On Dec 17, 2007 4:03 PM, Rob Hudson <[EMAIL PROTECTED]> wrote: > If I understand correctly, these cache the templates after they have > been rendered. What I'm curious about is if there is a way to cache > templates before they are rendered so you can provide different > contexts to them. It seems like there would still be some gains > involved -- a filesystem read, parsing the template into nodes, etc. Manually call get_template() or select_template(), and stuff the resulting Template object into a module-global variable somewhere. Then just re-use it, calling render() with different contexts, each time you need it. -- "Bureaucrat Conrad, you are technically correct -- the best kind of correct." --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
Rob Hudson schrieb:
> If I understand correctly, these cache the templates after they have
> been rendered. What I'm curious about is if there is a way to cache
> templates before they are rendered so you can provide different
> contexts to them. It seems like there would still be some gains
> involved -- a filesystem read, parsing the template into nodes, etc.
I've been wondernig the same thing. If I do {% include "item.html" %}
for 200 items, will item.html be re-read each time? Glancing at the
code, it appears so.
Michael
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
On 12/17/07, Alex Koshelev <[EMAIL PROTECTED]> wrote: > > http://www.djangoproject.com/documentation/cache/ If I understand correctly, these cache the templates after they have been rendered. What I'm curious about is if there is a way to cache templates before they are rendered so you can provide different contexts to them. It seems like there would still be some gains involved -- a filesystem read, parsing the template into nodes, etc. Thanks, Rob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Templates, filesystem, caching?
http://www.djangoproject.com/documentation/cache/ On 17 дек, 23:49, "Rob Hudson" <[EMAIL PROTECTED]> wrote: > Howdy, > > A thought occurred to me today and I'm not 100% sure what Django does > by default... > > Similar to the idea that PHP parses scripts for each request and > having an opcode cache can increase performance, I'm wondering if > Django reads templates from the file system, parses them, and then > does the context replacement on them for each and every request. And > if so, would there be some optimization to be gleaned by caching the > parsed templates? (Maybe optionally tied to DEBUG=False?) > > Thanks, > Rob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
