On 1/30/08, James Bennett <[EMAIL PROTECTED]> wrote: > On Jan 30, 2008 10:21 AM, Jarek Zgoda <[EMAIL PROTECTED]> wrote: > > Can you share any hints on how to reduce the memory usage in such > > situation? The underlying database structure is rather complicated and I > > would like to not do all queries manually. > > At this level -- hundreds of thousands of objects per query -- I doubt > that any ORM solution is going to deliver decent performance.
That may be true, but in this case it's actually a bug in Django: under some (many?) circumstances QuerySets load the entire result set into memory even when used with an iterator. The good news is that this has been fixed; the bad news is that the fix is on the queryset-refactor branch, which likely won't merge to trunk for at least a few more weeks, if not longer. Also note that values() won't really help you, either: the overhead for a model instance isn't all that big; the problem you're running into is simply the shear number of results. In your situation, I'd do one of two things: either switch to the qsrf branch if you're a living-on-the-edge kind of guy, or else look into using an ObjectPaginator to churn through results in chunks. Jacob --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
