Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
--+
 Reporter:  manfre|Owner:  nobody
 Type:  Cleanup/optimization  |   Status:  new
Component:  Testing framework |  Version:  1.7
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  1 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  1
Easy pickings:  0 |UI/UX:  0
--+

Comment (by akaariai):

 A yet new alternate approach is here:
 https://github.com/django/django/pull/5905. The idea is to skip disabling
 constraints when no fixture files are found in loaddata. The 1.8 test
 suite runs in 2500 seconds on my laptop after the patch (plus one line fix
 in django-mssql, see the pull request).

 The previously proposed alternate approach takes off another 10 minutes
 from the runtime of the test suite on mssql, yielding a runtime of around
 30 minutes. The changes needed there are a lot more invasive, as we will
 need to pre-walk the objects in the fixture files. This could lead to
 performance problems. The change might be worth it for master branch, but
 backpatching seems too risky.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.b178f4f6366d272893e8b5bc9c4e9027%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
--+
 Reporter:  manfre|Owner:  nobody
 Type:  Cleanup/optimization  |   Status:  new
Component:  Testing framework |  Version:  1.7
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  1 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  1
Easy pickings:  0 |UI/UX:  0
--+
Changes (by timgraham):

 * needs_better_patch:  0 => 1


Comment:

 Anssi has proposed [https://github.com/django/django/pull/5801 an
 alternate approach].

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.7a393a9f08fb82a1d80c6576e4e32c68%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
--+
 Reporter:  manfre|Owner:  nobody
 Type:  Cleanup/optimization  |   Status:  new
Component:  Testing framework |  Version:  1.7
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  1 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+
Changes (by timgraham):

 * status:  closed => new
 * has_patch:  0 => 1
 * resolution:  needsinfo =>


Comment:

 The [https://github.com/django/django/pull/5438 proposed PR] for Django
 1.8 reduces the time of the entire suite on MSSQL from about 7.75 hours to
 5.25 hours (benchmarks from @manfre). I'm not particularly proud of that
 hack, but it seems worthwhile unless someone can spot a problem with it.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.a4584d79f0149aa403b0342f612a8ff2%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
-+-
 Reporter:  manfre   |Owner:  nobody
 Type:   |   Status:  closed
  Cleanup/optimization   |
Component:  Testing framework|  Version:  1.7
 Severity:  Normal   |   Resolution:  needsinfo
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by timgraham):

 I wonder if this might be better on master (1.9) after commits like
 67235fd4ef1b006fc9cdb2fa20e7bb93b0edff4b and
 4aa089a9a9504c4a833eee8161be013206da5d15 -- the `migrate` command should
 be more "lightweight" after them.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.8d4b7cff9b3fe0790c44e4560802%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
-+-
 Reporter:  manfre   |Owner:  nobody
 Type:   |   Status:  closed
  Cleanup/optimization   |
Component:  Testing framework|  Version:  1.7
 Severity:  Normal   |   Resolution:  needsinfo
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by MarkusH):

 * status:  new => closed
 * resolution:   => needsinfo


Comment:

 If this ''is'' related to the performance of the migration operations -
 which doesn't seem to be completely clear at this point - the issue should
 be fixed or at leased reduced by an order of magnitude by the
 optimizations the migrate process received for 1.8.

 I'm closing this with ''needsinfo'' for the reasons posted in [comment:10
 comment 10]

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.ae02a5d34a7809a495988e662d51315e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
--+
 Reporter:  manfre|Owner:  nobody
 Type:  Cleanup/optimization  |   Status:  new
Component:  Testing framework |  Version:  1.7
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+
Changes (by coagulant):

 * cc: baryshev@… (added)


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.d25896b963496405dbe5d78a8f4418a7%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
--+
 Reporter:  manfre|Owner:  nobody
 Type:  Cleanup/optimization  |   Status:  new
Component:  Testing framework |  Version:  1.7
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+

Comment (by timgraham):

 Robert, when you run your tests in 1.7+ all migrations are applied, which
 takes several seconds per migration on 1.7 as reported in #23745. The
 performance of that will be improved in 1.8 as described in that ticket.
 You can check if you can use the
 [https://docs.djangoproject.com/en/dev/ref/django-admin/#django-admin-
 option---keepdb --keepdb] option in 1.8. Also, we might add an option to
 disable migrations when testing in the future. But ultimately, no, I don't
 think your issue is the one tracked by this ticket.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.fa0de1df579fedeafc2de4bbae26e927%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
--+
 Reporter:  manfre|Owner:  nobody
 Type:  Cleanup/optimization  |   Status:  new
Component:  Testing framework |  Version:  1.7
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+

Comment (by RobertPeters):

 I have an application in 1.7 that I have been building for a while. I have
 about 75 migrations. When I run
 {{{
 python manage.py test
 }}}
 with SQLite, it takes about 30 seconds to create the test database. In 1.6
 this was almost instant (in fact, I just confirmed this. I ran my tests in
 an old environment at 1.6 and the test database was created instantly. I
 ran "pip install django --upgrade" and re-ran the tests. It took 30
 seconds to create the test database).

 I don't know if my issue is the same as this issue (being as I'm not using
 MSSQL Server), but I'd just like to add that there appears to be a
 significant slow down when going from 1.6 to 1.7, regardless of the
 database type.

 I'm using "django.db.backends.sqlite3" on Windows 7 64-bit with a Core i7
 processor and 8 GB ram.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.b13df32ba28ce8d37b8b28dddf8acbbc%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
--+
 Reporter:  manfre|Owner:  nobody
 Type:  Cleanup/optimization  |   Status:  new
Component:  Testing framework |  Version:  1.7
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+
Changes (by timgraham):

 * version:  1.7-rc-3 => 1.7


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.3a2e25685857de20ade1344ad6d6e89b%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
--+
 Reporter:  manfre|Owner:  nobody
 Type:  Cleanup/optimization  |   Status:  new
Component:  Testing framework |  Version:  1.7-rc-3
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+

Comment (by shaib):

 I've seen `loaddata` take eons on SQL Server, and even in the context of
 migrations (with South). As far as we could figure out, the problem was
 not transaction management per se, but the handling of constraints: For
 `loaddata` to avoid complex and sometimes non-satisfiable ordering
 requirements, it needs FK constraints to be deferred while it works. On
 core databases, each for its own reasons, this is not an issue.

 SQL Server does not support defering in the normal sense -- what you need
 to do is turn the constraints off in the beginning of worlk and back on in
 the end. I don't remember if this is an operation you can do within a
 transaction, but for sure it is not tied to a transaction -- you can turn
 the constraints off and then run several transactions before turning them
 back on. When you do turn them back on, however, you can do it `WITH CHECK
 OPTION` -- which loaddata does, or else it might put constraint-violating
 data into the database.

 SQL Server does not -- perhaps, given the semantics, cannot -- keep track
 of data changed while constraints were off. When you turn them on `WITH
 CHECK OPTION`, it checks every single record in the database -- or, at
 least, takes long enough to do so.

 We had a project with several dozen apps, and relatively simple sets of
 migrations on a full database (for upgrades) would take minutes on Oracle
 and hours on SQL Server. Then we told SQL Server users to use `migrate
 --all --no-initial-data` (we didn't have any initial data anywhere), and
 sanity returned.

 I'm not sure what this implies with regards to a solution. Some of the
 problem should go away if `loaddata` only defered constraints when initial
 data actually existed. Perhaps there's a way to avoid the check in most
 cases (say, most data loaded in the Django test suite can be given a free
 ticket because it is verified by the other backends).

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.d7a5554ef02a44a9397959195b76d488%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
--+
 Reporter:  manfre|Owner:  nobody
 Type:  Cleanup/optimization  |   Status:  new
Component:  Testing framework |  Version:  1.7-rc-3
 Severity:  Normal|   Resolution:
 Keywords:| Triage Stage:  Accepted
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+
Changes (by timgraham):

 * stage:  Unreviewed => Accepted


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.d56760ddae892d2d0b77857de5477c27%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
-+-
 Reporter:  manfre   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |  Version:  1.7-rc-3
Component:  Testing framework|   Resolution:
 Severity:  Normal   | Triage Stage:
 Keywords:   |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by aaugustin):

 Not a blocker for 1.7 indeed, but fairly high priority for 1.7.1.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.0a91112cf146335ae47dd7c91c7e5f47%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
-+-
 Reporter:  manfre   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |  Version:  1.7-rc-3
Component:  Testing framework|   Resolution:
 Severity:  Normal   | Triage Stage:
 Keywords:   |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by andrewgodwin):

 * severity:  Release blocker => Normal


Comment:

 Not a release blocker, I'm afraid; at this point release blocker is
 reserved for bugs that either crash Django or remove features that
 previously existed. I'm not willing to delay the 1.7 release, which is
 literally on the verge of happening, for test slowdown, or we'll literally
 never get round to releasing.

 Notably, the code in question has not been touched by migrations; we do a
 dumpdata as part of #22487, but that doesn't seem to be part of this. I'm
 not sure what the cause of the regression could be.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.35cebb8cbc5c8fdf9cc3a4c65c24556e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
-+-
 Reporter:  manfre   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |  Version:  1.7-rc-3
Component:  Testing framework|   Resolution:
 Severity:  Release blocker  | Triage Stage:
 Keywords:   |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by timgraham):

 MS SQL Server I presume. I'd guess the slowness might be the
 `transaction.atomic()` calls that happen in `loaddata` (one for app app in
 Django's test suite) even if no fixtures exist.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.6f007094d481fb888f3cdc48bda62422%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
-+-
 Reporter:  manfre   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |  Version:  1.7-rc-3
Component:  Testing framework|   Resolution:
 Severity:  Release blocker  | Triage Stage:
 Keywords:   |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by claudep):

 You didn't mention the database on which you encounter this problem,
 default SQLite?
 It would be nice to track a bit further what is so slow inside `loaddata`.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.6bccf1ee26809eb6aac5e58962d69855%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #23372: Test suite initial database construction is too slow

[Django]

#23372: Test suite initial database construction is too slow
-+-
 Reporter:  manfre   |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |  Version:  1.7-rc-3
Component:  Testing framework|   Resolution:
 Severity:  Release blocker  | Triage Stage:
 Keywords:   |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by manfre):

 * needs_better_patch:   => 0
 * needs_tests:   => 0
 * needs_docs:   => 0


Comment:

 Set as release blocker because this is an order of magnitude performance
 regression added by new functionality.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.7983162e05c66137fac2c841809992a2%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.