#18702: Remove chunked reads from iter(qs)
-------------------------------------+-------------------------------------
Reporter: akaariai | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: 1.4
Component: Database layer | Resolution:
(models, ORM) | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by akaariai):
Yes, `__bool__` seems to be another case hit. Minor correction is that you
do still convert 100 objects currently, not just one.
It is true that this patch is dealing with a tradeoff. On one hand we have
full resultset iteration performance and code clarity, on the other doing
bool(qs), contains or iteration with break. If the resultset is small it
doesn't matter what will be done (less than 100 objects and the results
are converted in full anyways). If the resultset is large, the user who
cares about performance should be doing something else anyways (unless
using Oracle).
It turns out that it is somewhat easy to support bool() and contains in
"one object at time" manner, but still do full conversion when accessing
full resultset. This way the only regression case is iteration with break.
But then, just use .iterator(). Also, those wanting to use server side
cursors will likely want to use .iterator().
The code is somewhat more complex than in the "always fetch all objects"
but not much.
1000 objects with pk + one field from db with .all() is 10% faster using
the patched code. .iterator() speed isn't changed.
Patch available from
https://github.com/akaariai/django/tree/non_chunked_iter_squashed.
Sidenote about chunked reads: Without better support from Postgres it will
be impossible to automatically use chunked reads. If you don't use WITH
HOLD, the cursor will be unusable outside transactions. Add in autocommit
mode, and it means this is a total no-go (not that it was a good idea even
before). If you use WITH HOLD, then the cursor must be closed explicitly.
So, if you have "if obj in qs" but don't iterate the whole qs it means you
just leaked a cursor (which is even worse than having idle-in-tx
connections). So, a dedicated API is needed in any case.
--
Ticket URL: <https://code.djangoproject.com/ticket/18702#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
