On 11/02/2011 01:36 PM, Marco Paolini wrote:
maybe we could implement something like:
for obj in qs.all().chunked(100):
pass
.chunked() will automatically issue LIMITed SELECTs
that should work with all backends
I don't think that will be a performance improvement - this will get rid
of the memory overhead in Django, but would lead to a lot of overhead in
the DB. Assuming you are fetching 10000 objects from the DB, you would
issue these commands to the DB (I hope I got the idea correctly):
SELECT ID, ...
ORDER BY order
LIMIT 0 OFFSET 100
SELECT ID, ...
ORDER BY order
LIMIT 100 OFFSET 100
...
SELECT ID, ...
ORDER BY order
LIMIT 100 OFFSET 9900
For each query the DB will need to do:
- query parse & plan
- If the order is not indexed, a top N sort.
- Fetch the items (even in indexed case you will need to travel the
index for the OFFSET which is not free at all).
So, for the last fetch the DB would need to travel the first 9900 items
in the index (or worse, do a top 10000 sort) and then return the 100
items wanted. This is going to be expensive in the DB. The trade-off of
saving some memory Django side at the expense of doing a lot more work
at the DB side is not a good one. DB resources are in general much more
harder to scale than the Django resources.
You are going to do in total 10000 + 9900 + 9800 + ... + 100 index
travels in the DB, which equals to somewhere around 0.5 million items
traveled in the index. In addition, you will do 100 parse + plan stages.
You really don't want to do that. In addition if there are concurrent
updates to the items, it might be you will miss some objects and see
some objects as duplicates.
- Anssi
--
You received this message because you are subscribed to the Google Groups "Django
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
