On Tue, 2009-02-03 at 15:06 -0800, Casey Deccio wrote:
> Hi,
>
> I'm using django 1.0.2. I have an model with a large number of
> records. I was hoping that the built-in iteration on queryset objects
> would reduce the memory requirement, but it is using up all my memory
> (and swap), and I'm forced to break up the chunks manually using
> queryset slices. Iterating over slices of 10000 entries at a time my
> memory usage is only around 40MB of memory, but without breaking it up
> manually (i.e., iterating over the entire, unsliced queryset) was
> using over 1GB before I killed it.
"Default" iteration over a queryset (using __iter__) also caches the
instances in memory so that they can be reused without requiring extra
database queries. Normally, the memory overhead is negligible, since the
queryset is being used to put results on, say, a web page, when a
billion rows isn't the right number to send back.
If you want to iterate over the results without that caching behaviour,
using the iterator() method on the Queryset, rather than the __iter__()
method. Thus:
for i in qs.iterator():
...
There will still be a lot of memory used here (proportional to the
number of results returned, which in your case is "a lot"), since the
database wrapper (psycopg or whatever) will pull all the data back from
the database. That will still be a multiple less than also keeping
copies of the Python objects, but it won't be zero. Using a pagination
equivalent, as your doing, could well be the best solution if you want
to keep memory at an absolute minimum.
Regards,
Malcolm
>
> Is there anything I might be missing?
>
> Here are the examples:
>
> # without manual slicing
> qs = MyModel.objects.all()
> for obj in qs:
> # do something
> # ...
> pass
>
> # with manual slicing
> index = 0
> qs = MyModel.objects.all()[0:10000]
> while qs:
> for obj in qs:
> # do something
> # ...
> pass
> index += 10000
> qs = MyModel.objects.all()[index:index+10000]
>
> Regards,
> Casey
>
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
