On 08/05/2009 04:09 PM, drakkan wrote:
> The code is something like this
>
> obj=DBModels.objects.filter(somefilter)
> results=[]
> for o in obj:
> results.append({'field1':o.field1,'field2':o.field2})
>
> return JsonResponse(results)
>
> where JsonResponse is a class derived from HttpResponse that add a
> simplejson.dumps
>
> I have a really large database and in some edge case I need to return
> a large number of results (about 150.000 when I get the error). I make
> only one query with select_related and I tried also with DEBUG=False
> but nothing changed only the detailed stack trace isn't printed,
>
> thanks
> drakkan
>
Hi drakkan,
Your code:
results=[]
for o in obj:
results.append({'field1':o.field1,'field2':o.field2})
is building a list in memory. Chances are, with 150k+ objects, that's
where you are running into the memory issue. That's a big list!
A couple of possibilities come to mind. First, have you looked into
using a values query? You may be able to avoid building the list
entirely. The relevant documentation is here:
http://docs.djangoproject.com/en/dev/ref/models/querysets/#values-fields
and the syntax would be something like this:
results = DBModels.objects.filter(somefilter).values('field1', 'field2')
At which point, results is basically this:
[{'field1':o1.field1, 'field2':o1.field2}, {'field1':o2.field1,
'field2':o2.field1}, ...]
but it's a lazy iterator rather than an in-memory list.
Another thing that comes to mind is using Django's built-in serializers,
which will take your filtered QuerySet, a tuple of fields, and serialize
to JSON for you. Docs are here:
http://docs.djangoproject.com/en/dev/topics/serialization/#topics-serialization
Hope that helps,
---Peter
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
