If you do want to go down the streaming path despite the caching, static and other advice, I can see a couple of options: 1) Write your own sitemap view that doesn't use the sitemap.xml template and instead streams back results using a StreamingHttpResponse and generator based get_url methods in sitemap classes. If that works out well, it might be an appropriate dropin replacement for the current view, although that would remove the availability of a customisable template. 2) Ticket 13910 has a pull request to enable streaming templates, which would be a stepping stone to allowing refactoring of the existing sitemap view to stream a response using the template. The same caveat applies about whether this would improve memory usage or response-time performance if the sitemap urls are pre-calculated.
- Roger [1] https://code.djangoproject.com/ticket/13910 On Friday, 26 July 2013 10:20:06 UTC+10, Curtis Maloney wrote: > > We ended up splitting out sitemaps up... and caching them heavily. > Pre-generating periodically would be equally effective. > > I can, however, see the merit in a generator style SiteMap engine with a > StreamingHttpResponse. It would allow the client to consume the data as > it's being consumed, instead of potentially timing out waiting for the > whole document to be generated. > > However, this may hide the growing problem for people whose sitemaps > gradually become large enough to warrant caching, but won't notice because > nothing is timing out, per se. > > Then again, micro-caching at the nginx/apache layer would also mitigate > the problem. > > -- > Curtis Maloney > > > > On 25 July 2013 23:51, Benjamin Wohlwend <piqu...@gmail.com > <javascript:>>wrote: > >> Hi Julian, >> >> after having a site almost taken down by an over-eager spider downloading >> the sitemaps dozens of times in parallel, I learned the lesson that large >> sitemaps shouldn't be rendered on demand. Since that day a few years ago, I >> always generate the sitemap periodically and let apache/nginx serve it. >> Recently, I even found a package that does all the heavy lifting for you: >> https://github.com/xaralis/django-static-sitemaps >> >> Kind regards, >> Benjamin >> >> >> On Thu, Jul 25, 2013 at 10:06 AM, julianb <julia...@gmail.com<javascript:> >> > wrote: >> >>> Since 1.5 we have streaming responses. What is the state of >>> contrib.sitemaps in this regard? I have some very large sitemaps and >>> experimented with making them faster a few years ago. >>> >>> If the do not yet stream, I think this would be a good idea to get >>> memory usage down. Is there anything to keep an eye on? Would it be >>> valuable to Django if this is looked into? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-develop...@googlegroups.com <javascript:>. >>> To post to this group, send email to >>> django-d...@googlegroups.com<javascript:> >>> . >>> Visit this group at http://groups.google.com/group/django-developers. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-develop...@googlegroups.com <javascript:>. >> To post to this group, send email to >> django-d...@googlegroups.com<javascript:> >> . >> Visit this group at http://groups.google.com/group/django-developers. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
