We ended up splitting out sitemaps up... and caching them heavily. Pre-generating periodically would be equally effective.
I can, however, see the merit in a generator style SiteMap engine with a StreamingHttpResponse. It would allow the client to consume the data as it's being consumed, instead of potentially timing out waiting for the whole document to be generated. However, this may hide the growing problem for people whose sitemaps gradually become large enough to warrant caching, but won't notice because nothing is timing out, per se. Then again, micro-caching at the nginx/apache layer would also mitigate the problem. -- Curtis Maloney On 25 July 2013 23:51, Benjamin Wohlwend <piquad...@gmail.com> wrote: > Hi Julian, > > after having a site almost taken down by an over-eager spider downloading > the sitemaps dozens of times in parallel, I learned the lesson that large > sitemaps shouldn't be rendered on demand. Since that day a few years ago, I > always generate the sitemap periodically and let apache/nginx serve it. > Recently, I even found a package that does all the heavy lifting for you: > https://github.com/xaralis/django-static-sitemaps > > Kind regards, > Benjamin > > > On Thu, Jul 25, 2013 at 10:06 AM, julianb <julian....@gmail.com> wrote: > >> Since 1.5 we have streaming responses. What is the state of >> contrib.sitemaps in this regard? I have some very large sitemaps and >> experimented with making them faster a few years ago. >> >> If the do not yet stream, I think this would be a good idea to get memory >> usage down. Is there anything to keep an eye on? Would it be valuable to >> Django if this is looked into? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-developers+unsubscr...@googlegroups.com. >> To post to this group, send email to django-developers@googlegroups.com. >> Visit this group at http://groups.google.com/group/django-developers. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
