On 26 July 2013 14:12, Roger Barnes <ro...@mindsocket.com.au> wrote: > If you do want to go down the streaming path despite the caching, static > and other advice, I can see a couple of options: > 1) Write your own sitemap view that doesn't use the sitemap.xml template > and instead streams back results using a StreamingHttpResponse and > generator based get_url methods in sitemap classes. If that works out well, > it might be an appropriate dropin replacement for the current view, > although that would remove the availability of a customisable template. >
Well, I was figuring that since sitemap is a XML based format, a generator based rendering should be quite easy to achieve, even rendering a template-per-record. > 2) Ticket 13910 has a pull request to enable streaming templates, which > would be a stepping stone to allowing refactoring of the existing sitemap > view to stream a response using the template. The same caveat applies about > whether this would improve memory usage or response-time performance if the > sitemap urls are pre-calculated. > Ah, yes, I saw mindsocket working on this at PyConAU sprints, but wasn't sure how far through he'd got. The issues I've had with generator responses in the past is when there's an error in rendering, they tend to just stop. However, in the case of generating a stream of XML snippets, it's fairly reliable. -- Curtis > - Roger > > [1] https://code.djangoproject.com/ticket/13910 > > On Friday, 26 July 2013 10:20:06 UTC+10, Curtis Maloney wrote: > >> We ended up splitting out sitemaps up... and caching them heavily. >> Pre-generating periodically would be equally effective. >> >> I can, however, see the merit in a generator style SiteMap engine with a >> StreamingHttpResponse. It would allow the client to consume the data as >> it's being consumed, instead of potentially timing out waiting for the >> whole document to be generated. >> >> However, this may hide the growing problem for people whose sitemaps >> gradually become large enough to warrant caching, but won't notice because >> nothing is timing out, per se. >> >> Then again, micro-caching at the nginx/apache layer would also mitigate >> the problem. >> >> -- >> Curtis Maloney >> >> >> >> On 25 July 2013 23:51, Benjamin Wohlwend <piqu...@gmail.com> wrote: >> >>> Hi Julian, >>> >>> after having a site almost taken down by an over-eager spider >>> downloading the sitemaps dozens of times in parallel, I learned the lesson >>> that large sitemaps shouldn't be rendered on demand. Since that day a few >>> years ago, I always generate the sitemap periodically and let apache/nginx >>> serve it. Recently, I even found a package that does all the heavy lifting >>> for you: >>> https://github.com/**xaralis/django-static-sitemaps<https://github.com/xaralis/django-static-sitemaps> >>> >>> Kind regards, >>> Benjamin >>> >>> >>> On Thu, Jul 25, 2013 at 10:06 AM, julianb <julia...@gmail.com> wrote: >>> >>>> Since 1.5 we have streaming responses. What is the state of >>>> contrib.sitemaps in this regard? I have some very large sitemaps and >>>> experimented with making them faster a few years ago. >>>> >>>> If the do not yet stream, I think this would be a good idea to get >>>> memory usage down. Is there anything to keep an eye on? Would it be >>>> valuable to Django if this is looked into? >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Django developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to django-develop...@**googlegroups.com. >>>> To post to this group, send email to django-d...@**googlegroups.com. >>>> >>>> Visit this group at >>>> http://groups.google.com/**group/django-developers<http://groups.google.com/group/django-developers> >>>> . >>>> For more options, visit >>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>>> >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-develop...@**googlegroups.com. >>> To post to this group, send email to django-d...@**googlegroups.com. >>> >>> Visit this group at >>> http://groups.google.com/**group/django-developers<http://groups.google.com/group/django-developers> >>> . >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >>> >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
