On Mon, Oct 10, 2016 at 10:01 AM, Constantine Covtushenko < constantine.covtushe...@gmail.com> wrote:
> Hi Dmitry, > > Please check that documentation page > <https://docs.djangoproject.com/en/1.10/ref/models/querysets/#iterator>. > As you can see iterator() loads all in the memory and after that returns > an iterator. > > Try to fetch records say by 50 - 100 items. > > Hope that helps. > > That's correct. The Queryset is evaluated (which means the entire result is loaded into memory). The iterator() call only optimizes the access to the result, not the result itself. 50 - 100 items may be a bit conservative with the size of machine that you have, but it all depends on the size of the objects and how utilized the server is. There is likely a point of diminishing returns somewhere higher (probably in the thousands), but only experimentation is going to help you find it. I also wanted to draw attention to the note relating to DB caching, which may be adding extra taxation on your server for large queryset results (although I can't imagine a DB would try to cache a result that large without specifically being tuned to do so). Is there a reason this is run in a batch, rather than being performed during the initial save() or an update() of the data? Seems like the operation you are performing is pretty static and should be able to be calculated as the data is created and/or updated. Might save you from having to deal with this type of operation at all. -James -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2Be%2BciWECBQxbxZwe60rNLoD0EMfM82Q4omu3w4KAj7wFzDEqw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
