Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  Anssi Kääriäinen |Owner:  Alex
 Type:   |  Aktsipetrov
  Cleanup/optimization   |   Status:  closed
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Mariusz Felisiak ):

 In [changeset:"036f160733fcceb10975bf03c2fc9ffd9c114612" 036f160]:
 {{{
 #!CommitTicketReference repository=""
 revision="036f160733fcceb10975bf03c2fc9ffd9c114612"
 Refs #20577 -- Deferred filtering of prefetched related querysets by
 reverse m2o relation.
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.4ef66289f9170b6b50b032fc0e536032%40djangoproject.com.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  Anssi Kääriäinen |Owner:  Alex
 Type:   |  Aktsipetrov
  Cleanup/optimization   |   Status:  closed
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak ):

 * status:  assigned => closed
 * resolution:   => fixed


Comment:

 In [changeset:"681f7e2b13846def9eff3d26ce1371d05fa90c4c" 681f7e2]:
 {{{
 #!CommitTicketReference repository=""
 revision="681f7e2b13846def9eff3d26ce1371d05fa90c4c"
 Fixed #20577 -- Deferred filtering of prefetched related querysets.

 Added internal interface to QuerySet that allows to defer next filter
 call till .query is accessed. Used it to optimize prefetch_related().

 Thanks Simon Charette for the review.
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.dd2ce1bfec4d7cb20e64d5c0568ea026%40djangoproject.com.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  Anssi Kääriäinen |Owner:  Alex
 Type:   |  Aktsipetrov
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by felixxm):

 * stage:  Accepted => Ready for checkin


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.b26c452e9daebca7b0976132f0e97a6e%40djangoproject.com.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  Anssi Kääriäinen |Owner:  Alex
 Type:   |  Aktsipetrov
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by felixxm):

 * owner:  nobody => Alex Aktsipetrov
 * status:  new => assigned


Comment:

 [https://github.com/django/django/pull/11916 PR]

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.21b94b1709d5223a6cc263c1accb5595%40djangoproject.com.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  Anssi Kääriäinen |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Simon Charette):

 * needs_better_patch:  1 => 0


Comment:

 The patch should be RFC pending some minimal adjustments.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.34daf38d02cbc87689ce76411a267406%40djangoproject.com.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  akaariai |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-

Comment (by tomlinford):

 Echoing karyon here -- We're seeing pretty significant performance hits
 for our django rest api on endpoints that return objects that need to have
 access to their child objects.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.960807144bed25c2f23e908ff159d61e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  akaariai |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by karyon):

 * cc: karyon (added)


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.72faac60c1aeab01367edbdeae2176a3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  akaariai |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-

Comment (by karyon):

 fyi, i can still observe this behavior. the profile looks pretty similar.
 using the Prefetch object didn't change anything.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.148e3644eb60e8c30c9a83b20c50e1ad%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  akaariai |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |  Version:  master
Component:  Database layer   |   Resolution:
  (models, ORM)  | Triage Stage:  Accepted
 Severity:  Normal   |  Needs documentation:  0
 Keywords:   |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-

Comment (by akaariai):

 The approach in #17001 (prefetch custom querysets) will provide the
 speedup promised in this ticket. Unfortunately the speedup is only
 available to custom prefetches, not ordinary
 .prefetch_related('some_related_set').

 I am going to do a wait-and-see on this ticket, changes required for fast
 prefetches directly into querysets require some hacks. The hacks needed:
 create QuerySet.query lazily, and have a "on_query_creation_ops" QuerySet
 attribute, that is things that are ran when the inner query is created.

 I am going to leave this in accepted status - if a suitably clean patch is
 written I think speeding up prefetch_related is a good idea.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.2576a8b4a96b054d3beb0817d2aa4465%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  akaariai |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |  Version:  master
Component:  Database layer   |   Resolution:
  (models, ORM)  | Triage Stage:  Accepted
 Severity:  Normal   |  Needs documentation:  0
 Keywords:   |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-

Comment (by akaariai):

 #19264 was a duplicate. In addition #20880 (splitted clone()) tracks
 splitting clone() to pre_next_op() and clone() parts. That seems to be a
 pre-requisite for the chain_ops() approach.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.2a11c0f7f376ad8698362118e6063fa3%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  akaariai |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |  Version:  master
Component:  Database layer   |   Resolution:
  (models, ORM)  | Triage Stage:  Accepted
 Severity:  Normal   |  Needs documentation:  0
 Keywords:   |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-
Changes (by akaariai):

 * needs_better_patch:  0 => 1
 * has_patch:  0 => 1


Comment:

 The _sticky_filter is related, but I don't see any immediate way to
 implement that with chain_ops. The _sticky_filter tells self.query to not
 reset reusable aliases when cloning so that two .filter() calls in row
 target the same multijoin alias (or, this is how I recall). There are a
 couple of other cases where clone() resets some state to start next
 operation. The chain_ops() will need a way to tell self.query to reset
 state after each operation without doing a clone, while _sticky_filter
 needs a way to not reset state while doing a clone.

 The Query.clone() is responsible for even changing the class of the Query
 sometimes. It might be better to separate the responsibilities a bit, so
 that .clone() does a full clone. Then there is a .start_op() or something
 like that that does the state cleanup & class copying for starting a new
 operation. I am not sure how this interacts with all the Query subclasses,
 or if this will have noticeable performance impact.

 I am not sure if chain_ops() is needed API after all. For user API the
 wanted behaviour is to first do a clone, then run all the operations
 without cloning between and finally return the result. chain_ops() doesn't
 clone at all, and delays the operations until next queryset operation. As
 is, chain_ops() is very useful for prefetch_related but completely wrong
 as user API. So, maybe there needs to be some internal setup for
 prefetch_related delayed execution. The same setup can then be used for
 chain_ops() too if need be.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.80cfee27a7e5d3bf001eed565820631e%40djangoproject.com?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  akaariai |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |  Version:  master
Component:  Database layer   |   Resolution:
  (models, ORM)  | Triage Stage:  Accepted
 Severity:  Normal   |  Needs documentation:  0
 Keywords:   |  Patch needs improvement:  0
Has patch:  0|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-

Comment (by lukeplant):

 Nice work!

 I'm wondering if there is some overlap with `_sticky_filter` (or if that
 could be implemented using `chain_ops`).

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.5a00cc1d6a2b653de5a9eb38ceaf420e%40djangoproject.com?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #20577: Make prefetch_related faster by lazily creating related querysets

[Django]

#20577: Make prefetch_related faster by lazily creating related querysets
-+-
 Reporter:  akaariai |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |  Version:  master
Component:  Database layer   |   Resolution:
  (models, ORM)  | Triage Stage:  Accepted
 Severity:  Normal   |  Needs documentation:  0
 Keywords:   |  Patch needs improvement:  0
Has patch:  0|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-

Comment (by akaariai):

 I implemented a branch where it is possible to chain operations. The
 operations are executed on next queryset access. See
 https://github.com/akaariai/django/tree/chain_ops. The branch isn't meant
 to be committable, and it fails a couple of tests (one reason might be
 that cloning does a bit more than just clone(), so when chaining ops
 without clone things will break).

 The basic idea is to add a new QuerySet method chain_ops(). The usage is
 simple: qs.chain_ops(lambda qs:
 qs.filter(foo=bar).using(baz).order_by(xyzzy)). The chain_ops() method
 will *not* clone the queryset. The operations in the lambda function are
 executed when needed, that is on queryset evaluation, accessing qs.query,
 or cloning the queryset. When the operations in the lambda function are
 executed, the queryset isn't cloned in between.

 The result of this is 2.3x speedup in
 
[https://github.com/django/djangobench/blob/master/djangobench/benchmarks/query_prefetch_related/benchmark.py
 query_prefetch_related djangobench benchmark].

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/066.3322610c288096baa3af62ad86c29820%40djangoproject.com?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.