#29215: Document potential change to behaviour of QuerySet methods when
upgrading
to Python 3.6
-------------------------------------+-------------------------------------
Reporter: Matt | Owner: nobody
Fisher |
Type: | Status: new
Cleanup/optimization |
Component: | Version: 2.0
Documentation |
Severity: Normal | Keywords: documentation
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Python 3.6 introduces
[https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-pep468 a change]
that preserves the ordering of keyword arguments -
[https://www.python.org/dev/peps/pep-0468/ PEP 468: Preserving Keyword
Argument Order]
Since this is not a difference between Python 2 and 3 per se, it is not
mentioned in the [https://docs.djangoproject.com/en/1.11/topics/python3/
“Porting to Python 3”] Django 1.11 topic page.
Preserving keyword argument ordering can change the SQL produced by
QuerySet methods that accept `**kwargs`, specifically .get, .filter, and
.exclude, and potentially also .values, .annotate, .aggregate, .create,
.get_or_create, .update, and .update_or_create.
As an example,
`.filter(b__in=some_qs, a=1)`
could produce
`... WHERE (a=1 AND b IN (SELECT …))` in Python < 3.6 and
`... WHERE (b IN (SELECT …) AND a=1)` in Python >= 3.6.
The query is semantically equivalent but the postgres query planner can
produce dramatically different query plans and resultant performance if
the query has a significant number of joins.
We had one particular frequently-run query that ran in 2-3 seconds in
Python 2, and 80+ seconds in Python 3.6. The problem was only apparent in
production, and presented as severely degraded responsiveness from the
postgres RDS database. The additional load on the database made most other
queries take longer as well, which made it difficult to track down the
cause as it was not obvious that any queries could have changed. To
compound the problem, the different query execution caused many more temp
block writes on the RDS instance, which burned through our IOPS burst
balance and further degraded site performance. Because the query didn’t
change semantically, all our automated tests passed and we didn’t see the
issue until it was in production under full load. Googling the problem
produced no good results, which suggests it is not common, but in need of
a useful resource.
People may be affected by this change when upgrading from Python 2.7 to
Python 3.6+ using Django 1.x, in which case it would be helpful to have a
warning in the [https://docs.djangoproject.com/en/1.11/topics/python3/
1.11 porting topic], but it will also occur when people are upgrading from
Python 3.[0-5] to 3.6+ while potentially using Django 2, in which case the
porting topic is no longer in the current docs. Not sure what would be an
appropriate place for a warning in this case.
--
Ticket URL: <https://code.djangoproject.com/ticket/29215>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/053.a276fea7be60a828a7d9241ef92aa39c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
