Hello Raphael, Thanks for your detailed explanation! You clearly expressed why it's safe to optimize through RemoveField operations and helped me lift any doubt about what was a wrong assumption[1].
I gave your two passes optimization strategy a try and I believe I managed to implement it correctly[2]. The plan is to modify the optimization algorithm to start by trying to perform a "right" (as right of the migration in between) reduction and attempt a "left" one if a subsequent operation can be reduced with. The "left" reduction will only be performed if all the operations in between can optimize through (their reduce() method returns True) the operation we're trying to perform a reduction with. Cheers, Simon [1] https://github.com/django/django/pull/7999/commits/6d0b740cbc9ae038f9eef95dac3057dccb283e6d [2] https://github.com/django/django/pull/7999 Le vendredi 17 février 2017 01:14:58 UTC-5, Gaschignard, Raphael a écrit : > > To clarify on my previous post, if we're in the first case, then the +M > and -F operations can be optimized in one path to remove the dependency, > and then the +M' and -M' operation can be optimized through. > > In the second case (with an AddField operation), the +F and -F operations > will cancel each other out (since they will be "in between" +M' and -M') > and then the optimization can happen. > > A bit more holistically, since M's field depends on M' in the code (which > is where these operations come from, after all), in order to remove M', you > would first need to remove M's field, so you're almost guaranteed that a > RemoveField will be "in between" any of its dependency's creations/removals. > > If we have the field information in the RemoveField, we could check it. > But if we do not make an assumption on (no field information) RemoveFields, > it blocks a lot of possible optimizations. We could just do that (start > generating RemoveField with field info), though it would not allow for > older migrations to get optimized. I'd be good with doing either/both. > > Raphael > > On Fri, Feb 17, 2017 at 3:09 PM, Gaschignard, Raphael < > rap...@makeleaps.com <javascript:>> wrote: > >> Hi Simon, >> >> I think it's a bit more general than that. Why does the `RemoveField` >> exist? Because somewhere, an `AddField`-esque operation exists before it, >> right? >> >> Let's say we have m, m' as models. >> >> Let -F be a RemoveField(m, 'foo', ForeignKey(m') operation. We also >> have two operations +M', a CreateModel(m'), and -M', a RemoveModel(m') >> operation. Because we have a removeField operation, we also have +M, a >> "CreateModel(m), somewhere >> >> Let's assume that we have an operations list like: [ ...(1)..., +M', >> ...(2)..., -F, ...(3)..., -M']. How do we know that we can reduce +M' and >> -M' together "through" -F? >> >> Because we have -F, we have +M somewhere. because m has a field for m', >> either: >> 1 - The field is within the initial +M operation. Because the >> definition depends on M', it must be in group (2). >> 2 - The field is not within the initial +M operation. Because the >> definition of the field depends on M', we need an AddField operation (or >> moral equivalent) +F in group (2) >> >> So, if we have -F in between +M' and -M', we will also have either a +M >> which depends on M' or a +F which depends on M' within (2). >> >> So if the -F is between the two, there will also be another operation >> that will be present between the two that expresses the same dependencies. >> So the -F operation itself can avoid expressing its "RemoveField" indirect >> dependency to m' because another operation will do it for them. >> >> ---- >> >> It's hard to generalize this across everything because there are, after >> all, arbitrary migrations. But considering that RemoveField will only be >> reduced with a "moral equivalent" to AddField, I think we can expand this >> reasoning across everything to say that RemoveField will be sufficiently >> protected by the AddField operation's location (which will have to be after >> +M'). >> >> >> Raphael >> >> >> On Fri, Feb 17, 2017 at 12:28 PM, charettes <chare...@gmail.com >> <javascript:>> wrote: >> >>> > RemoveField('A', 'foo') also references A and foo, but does it >>> reference B? if it does, then it' s hard to have optimizations that pass >>> through this, because this field could be referencing any model >>> (theoretically). >>> >>> I think we all agree on that. >>> >>> > But if we assert that RemoveField doesn't refer to any models >>> referenced to by its field, then our optimizer can take a couple more >>> liberties. >>> >>> Do you have suggestion on how we can assert that it's the case? The only >>> way I could come up with was to make sure RemoveField has a reference to >>> the field it's removing. e.g. It would be generated in the form >>> `RemoveField('A', 'foo', ForeignKey('B'))`. >>> >>> Simon >>> >>> >>> Le jeudi 16 février 2017 18:25:16 UTC-5, rap...@makeleaps.com a écrit : >>>> >>>> When you have AddField('A', 'foo', ForeignKey('B')), this operation >>>> references A and foo, but also references B. >>>> >>>> RemoveField('A', 'foo') also references A and foo, but does it >>>> reference B? if it does, then it' s hard to have optimizations that pass >>>> through this, because this field could be referencing any model >>>> (theoretically). >>>> >>>> But if we assert that RemoveField doesn't refer to any models >>>> referenced to by its field, then our optimizer can take a couple more >>>> liberties. >>>> >>>> Raphael >>>> >>>> On Friday, February 17, 2017 at 2:15:47 AM UTC+9, Markus Holtermann >>>> wrote: >>>>> >>>>> I'm not sure if it's related or not wo what you're investigating, >>>>> RemoveField cannot "just" optimized through, as you might have another >>>>> AddField operation afterwards adding another field with the same name. >>>>> >>>>> /Markus >>>>> >>>>> On Thu, Feb 16, 2017 at 08:19:01AM -0800, rap...@makeleaps.com wrote: >>>>> >Hey Simon, >>>>> > >>>>> > I looked through your PR and added a couple comments. The main thing >>>>> is I >>>>> >think we can actually ignore the field context on "RemoveField", if >>>>> only >>>>> >because the executor doesn't need it. Even though the field might be >>>>> >pointing to a related model, that doesn't prevent being optimized >>>>> through. >>>>> > >>>>> > This is hard to explain, but intuitively, each "RemoveField" is >>>>> paired >>>>> >with an "AddField" or "CreateModel" that *does *depend on the related >>>>> >model. So if we have a potentially dangerous optimization, those >>>>> initial >>>>> >operations will "protect" the causal order, not "RemoveField". >>>>> > >>>>> > Raphael >>>>> > >>>>> >-- >>>>> >You received this message because you are subscribed to the Google >>>>> Groups "Django developers (Contributions to Django itself)" group. >>>>> >To unsubscribe from this group and stop receiving emails from it, >>>>> send an email to django-develop...@googlegroups.com. >>>>> >To post to this group, send email to django-d...@googlegroups.com. >>>>> >Visit this group at https://groups.google.com/group/django-developers. >>>>> >>>>> >To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/django-developers/9dfdcec6-b98c-44f2-86af-99aaa8857cc9%40googlegroups.com. >>>>> >>>>> >>>>> >For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/django-developers/YMbYXiZgrF0/unsubscribe >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> django-develop...@googlegroups.com <javascript:>. >>> To post to this group, send email to django-d...@googlegroups.com >>> <javascript:>. >>> Visit this group at https://groups.google.com/group/django-developers. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/1805b163-4d47-4ace-b9d3-79b6346126cd%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/django-developers/1805b163-4d47-4ace-b9d3-79b6346126cd%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/30f9183f-3576-47b8-bc54-2ae007f304c3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
