Re: Some thoughts about improving migration squashing

2017-02-20 Thread Patryk Zawadzki
W dniu poniedziałek, 20 lutego 2017 11:49:38 UTC+1 użytkownik Markus Holtermann napisał: > > On Thu, Feb 16, 2017 at 03:25:16PM -0800, rap...@makeleaps.com > wrote: > >RemoveField('A', 'foo') also references A and foo, but does it reference > B? > >if it does, then it' s hard to have

Re: Some thoughts about improving migration squashing

2017-02-20 Thread Markus Holtermann
On Thu, Feb 16, 2017 at 03:25:16PM -0800, raph...@makeleaps.com wrote: When you have AddField('A', 'foo', ForeignKey('B')), this operation references A and foo, but also references B. correct. RemoveField('A', 'foo') also references A and foo, but does it reference B? if it does, then it' s

Re: Some thoughts about improving migration squashing

2017-02-17 Thread charettes
Hello Raphael, Thanks for your detailed explanation! You clearly expressed why it's safe to optimize through RemoveField operations and helped me lift any doubt about what was a wrong assumption[1]. I gave your two passes optimization strategy a try and I believe I managed to implement it

Re: Some thoughts about improving migration squashing

2017-02-16 Thread Gaschignard, Raphael
To clarify on my previous post, if we're in the first case, then the +M and -F operations can be optimized in one path to remove the dependency, and then the +M' and -M' operation can be optimized through. In the second case (with an AddField operation), the +F and -F operations will cancel each

Re: Some thoughts about improving migration squashing

2017-02-16 Thread Gaschignard, Raphael
Hi Simon, I think it's a bit more general than that. Why does the `RemoveField` exist? Because somewhere, an `AddField`-esque operation exists before it, right? Let's say we have m, m' as models. Let -F be a RemoveField(m, 'foo', ForeignKey(m') operation. We also have two operations +M',

Re: Some thoughts about improving migration squashing

2017-02-16 Thread charettes
> RemoveField('A', 'foo') also references A and foo, but does it reference B? if it does, then it' s hard to have optimizations that pass through this, because this field could be referencing any model (theoretically). I think we all agree on that. > But if we assert that RemoveField doesn't

Re: Some thoughts about improving migration squashing

2017-02-16 Thread raphael
When you have AddField('A', 'foo', ForeignKey('B')), this operation references A and foo, but also references B. RemoveField('A', 'foo') also references A and foo, but does it reference B? if it does, then it' s hard to have optimizations that pass through this, because this field could be

Re: Some thoughts about improving migration squashing

2017-02-16 Thread Markus Holtermann
I'm not sure if it's related or not wo what you're investigating, RemoveField cannot "just" optimized through, as you might have another AddField operation afterwards adding another field with the same name. /Markus On Thu, Feb 16, 2017 at 08:19:01AM -0800, raph...@makeleaps.com wrote: Hey

Re: Some thoughts about improving migration squashing

2017-02-16 Thread raphael
Hey Simon, I looked through your PR and added a couple comments. The main thing is I think we can actually ignore the field context on "RemoveField", if only because the executor doesn't need it. Even though the field might be pointing to a related model, that doesn't prevent being optimized

Re: Some thoughts about improving migration squashing

2017-02-15 Thread charettes
Hi Raphael, I've been working on making the optimizer a bit smarter recently and came to the same conclusion as you concerning the "left" and "right" optimizations. This should be possible to solve by allowing `Operation.reduce()` to return the full list of operations it replaces by appending

Re: Some thoughts about improving migration squashing

2017-02-15 Thread Andrew Godwin
> > > You're essentially speaking about 2 things here, in my opinion: > > 1. Adding a new feature for interactive squash > 2. Improving the MigrationOptimizer > > I certainly see a point for 2. Not sure how much for 1. Anyway, your > reasoning for 2 sounds great! I'd be more than happy if you want

Re: Some thoughts about improving migration squashing

2017-02-15 Thread raphael
Markus: Now that I've written a command to optimize a single migration file, I think that it's sufficient for the "squash, edit, optimize" workflow that I was doing before. It's more about offering people to get their squashing done well until our optimizer becomes omniscient. Florian: Having

Re: Some thoughts about improving migration squashing

2017-02-15 Thread Markus Holtermann
Thanks Raphael, that's a pretty good write up! You're essentially speaking about 2 things here, in my opinion: 1. Adding a new feature for interactive squash 2. Improving the MigrationOptimizer I certainly see a point for 2. Not sure how much for 1. Anyway, your reasoning for 2 sounds great!

Re: Some thoughts about improving migration squashing

2017-02-15 Thread Markus Holtermann
What might be interesting to look into when squashing all migrations in one app would be to assume no migrations would exist. That could then result in only 2 migrations which could run through the optimizer (as opposed to let's say 20 migrations with many more operations). /Markus On

Re: Some thoughts about improving migration squashing

2017-02-15 Thread Florian Apolloner
Fwiw I think by default it could/should try to optimize all migrations of an app, manually specifying the migration name should be optional. On Wednesday, February 15, 2017 at 2:00:54 PM UTC+1, rap...@makeleaps.com wrote: > > I ended up having some time today, so wrote up a management command

Re: Some thoughts about improving migration squashing

2017-02-15 Thread Tim Graham
Hi Raphael, It looks like a similar idea was proposed in https://groups.google.com/d/topic/django-developers/C1L-NhyQYG4/discussion. I don't think a ticket was ever created, so you can do that. 100% test coverage is required. Why would we accept untested code? ;-) On Wednesday, February 15,

Re: Some thoughts about improving migration squashing

2017-02-15 Thread raphael
I ended up having some time today, so wrote up a management command for the first suggestion! I called it "optizemigration" >>> ./manage.py optimizemigration appname 0001_squashed # snipped django startup noise Optimized from 9 operations to 4 operations Optimized migration

Some thoughts about improving migration squashing

2017-02-15 Thread raphael
Hello everyone, I spent a couple hours last night trying to improve the migration squasher optimizer (migrations were taking almost 15 minutes in CI). I came up with a couple ideas for anyone interested in improvements: 1- Having an interactive mode for squashing would be interested.