Re: [Django] #34841: Reverse migrations model state rendering slow with moderate to large migrations

[Django]

#34841: Reverse migrations model state rendering slow with moderate to large
migrations
-+-
 Reporter:  David Sanders|Owner:  David
 Type:   |  Sanders
  Cleanup/optimization   |   Status:  closed
Component:  Migrations   |  Version:  4.2
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak ):

 * resolution:   => fixed
 * status:  assigned => closed

Comment:

 In [changeset:"b6e2b83901d841cf4d2179f650d1e7dca7e90c16" b6e2b83]:
 {{{#!CommitTicketReference repository=""
 revision="b6e2b83901d841cf4d2179f650d1e7dca7e90c16"
 Fixed #34841 -- Avoided rendering apps on state still requiring mutation.
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e6808d63a-4f573758-a600-4022-98e9-01150ee4d6b7-00%40eu-central-1.amazonses.com.

Re: [Django] #34841: Reverse migrations model state rendering slow with moderate to large migrations

[Django]

#34841: Reverse migrations model state rendering slow with moderate to large
migrations
-+-
 Reporter:  David Sanders|Owner:  David
 Type:   |  Sanders
  Cleanup/optimization   |   Status:  assigned
Component:  Migrations   |  Version:  4.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_better_patch:  1 => 0
 * stage:  Accepted => Ready for checkin

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e67c7881d-6b1ba34f-fbb8-4d6f-aaaf-20cd860e42ba-00%40eu-central-1.amazonses.com.

Re: [Django] #34841: Reverse migrations model state rendering slow with moderate to large migrations

[Django]

#34841: Reverse migrations model state rendering slow with moderate to large
migrations
-+-
 Reporter:  David Sanders|Owner:  David
 Type:   |  Sanders
  Cleanup/optimization   |   Status:  assigned
Component:  Migrations   |  Version:  4.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_better_patch:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018b3c898a32-8f56e09a-6e07-45e0-89ac-1b20aa6d1dea-00%40eu-central-1.amazonses.com.

Re: [Django] #34841: Reverse migrations model state rendering slow with moderate to large migrations

[Django]

#34841: Reverse migrations model state rendering slow with moderate to large
migrations
-+-
 Reporter:  David Sanders|Owner:  David
 Type:   |  Sanders
  Cleanup/optimization   |   Status:  assigned
Component:  Migrations   |  Version:  4.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * cc: Markus Holtermann, Shai Berger, Josh Smeaton (added)
 * stage:  Unreviewed => Accepted


Comment:

 Tentatively accepted for the investigation.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018acf97eb38-b12d461e-4d46-4d28-8f58-710bca6c4a13-00%40eu-central-1.amazonses.com.

Re: [Django] #34841: Reverse migrations model state rendering slow with moderate to large migrations

[Django]

#34841: Reverse migrations model state rendering slow with moderate to large
migrations
-+-
 Reporter:  David Sanders|Owner:  David
 Type:   |  Sanders
  Cleanup/optimization   |   Status:  assigned
Component:  Migrations   |  Version:  4.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * cc: Simon Charette (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018a984754a6-3f0c11d8-edd7-4441-8123-2a31d04a110b-00%40eu-central-1.amazonses.com.

Re: [Django] #34841: Reverse migrations model state rendering slow with moderate to large migrations

[Django]

#34841: Reverse migrations model state rendering slow with moderate to large
migrations
-+-
 Reporter:  David Sanders|Owner:  David
 Type:   |  Sanders
  Cleanup/optimization   |   Status:  assigned
Component:  Migrations   |  Version:  4.2
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by David Sanders):

 * owner:  nobody => David Sanders
 * status:  new => assigned
 * has_patch:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018a94dd13b2-eb46ecb2-2561-40ad-9a3d-1f067bc6e3dc-00%40eu-central-1.amazonses.com.

[Django] #34841: Reverse migrations model state rendering slow with moderate to large migrations

[Django]

#34841: Reverse migrations model state rendering slow with moderate to large
migrations
+
   Reporter:  David Sanders |  Owner:  nobody
   Type:  Cleanup/optimization  | Status:  new
  Component:  Migrations|Version:  4.2
   Severity:  Normal|   Keywords:
   Triage Stage:  Unreviewed|  Has patch:  0
Needs documentation:  0 |Needs tests:  0
Patch needs improvement:  0 |  Easy pickings:  0
  UI/UX:  0 |
+
 Relevant discussion: https://forum.djangoproject.com/t/migrating-
 backwards-takes-orders-of-magnitude-longer-than-migrating-forwards/20873/2

 Investigations revealed 2 things.

 1. this block of code in `db/migrations/excecutor.py` was responsible for
 a large portion of processing time:
 
https://github.com/django/django/blob/b8b2f7451201f3ff60891b6ce55f177400700d7a/django/db/migrations/executor.py#L222-L232

 In my small project with only a moderate number of migrations, a simple
 reverse migrate of the last migration on the main app took ~20s. The block
 below was responsible for ~16s.

 {{{
 # Generate the post migration state by starting from the state
 before
 # the last migration is unapplied and mutating it to include all
 the
 # remaining applied migrations.
 last_unapplied_migration = plan[-1][0]
 state = states[last_unapplied_migration]
 for index, (migration, _) in enumerate(full_plan):
 if migration == last_unapplied_migration:
 for migration, _ in full_plan[index:]:
 if migration in applied_migrations:
 migration.mutate_state(state, preserve=False)
 break
 }}}

 2. these 2 lines appeared to be responsible for the slowdown, commenting
 them out made the block above run almost instantly:
 
https://github.com/django/django/blob/b8b2f7451201f3ff60891b6ce55f177400700d7a/django/db/migrations/executor.py#L203-L204

 {{{
 if "apps" not in state.__dict__:
 state.apps  # Render all -- performance critical
 }}}

 I have a small patch to move record the unapplied state before these 2
 lines and clone it, removing the `state.apps` attribute.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018a94dcdbae-91edbd0a-bdbe-4fb9-bb1e-d38a5a281415-00%40eu-central-1.amazonses.com.