Could we allow applications to have fake migration files? I'm thinking of something like operations.AllModelsUpdated(). After this operation is ran, Django will use the present models for that app. You could place this as your last migration, and run direct SQL in previous migrations (or use some completely different mechanism) . Direct SQL without model state changes will be fast. The last migration would supposedly be fast, as it only needs to construct up to date images of the current models.
The problem is that I'm not at all sure how this will mix with migrations from other applications, so this proposal is perhaps technically impossible to do. But if this is feasible, then it should be possible to mix migrations and manual database changes pretty much seamlessly. This one would work much better if we had project level migrations - implementing a migration operation that tells Django that all of the project's models are up to date is almost trivial. I believe a fake migration would work pretty well for the use case where you want to use manual database control for your project, but use 3rd party applications with migrations. Another idea I've been thinking is that maybe we could have snapshots for migrations. The reason I believe this would work is that building the model state is expensive, but in many cases it is completely redundant to build the model state from scratch. The snapshots would essentially do the following operation: after migrations X, Y and Z are applied the model state will be as presented in this migration. The snapshots would only contain model state, not any operations. When you run manage.py migrate, the app state builder will be able to skip to latest snapshot that has all dependencies already applied, and then continue building model state from there. When performance begins to degrade, you can create a new snapshot. The snapshots would probably have to be project level to work properly. This would seriously speed up migrations, except for the case where you run the migrations against fresh database. - Anssi On Fri, Aug 28, 2015 at 3:18 AM, Andrew Godwin <and...@aeracode.org> wrote: > I think I agree with Markus that we do not yet have a good enough > performance story to make migrations mandatory. If you want to ditch them > and do your own database management, we should let you - which we could do > with some kind of MIGRATIONS_MODULES opt-out that basically sets everything > to pseudo-unmanaged, but I see no reason not to have the syncdb-like > solution keep working as it at least runs off of SchemaEditor now. > > Andrew > > On Thu, Aug 27, 2015 at 5:14 PM, Markus Holtermann > <i...@markusholtermann.eu> wrote: >> >> The general benefit of *not* having migrations is the incredible better >> performance to setup the tables. Especially for larger projects this is >> something we should at least keep in mind (see >> https://code.djangoproject.com/ticket/24743). I spent a fair bit of time on >> the issue and have not yet found a satisfying solution. >> >> Supporting apps without migrations would essentially continue to support >> the old `syncdb` behavior, though not as a standalone command, but as a flag >> for `manage.py migrate`. It takes the models from those apps and calls >> `schema_editor.create_model()`, taking the deferred SQL and running it after >> all models of apps w/o migrations have been created (current behavior [in >> tests] I think). This would continue to raise issues if models in apps w/o >> migrations refer to models in apps w/ migrations (missing constraints), but >> that is a limitation we had since 1.7 (silent failure; 1.8 fails loud). >> >> While migrations should provide *the* way to maintain your database I see >> the points that have been made in other recent discussions. Although Django >> doesn't force you to use migrations there are some things (e.g. not create >> the `django_migrations` table) we probably should not do unless you actually >> use migrations. >> >> We might want to consider looking at the `MIGRATION_MODULES` settings >> variable and allowing to set values to `None` in order to disable migrations >> for that app, but that's a different story. >> >> /Markus >> >> >> On Friday, August 28, 2015 at 4:26:19 AM UTC+10, Tim Graham wrote: >>> >>> It would be great if we could decide this by next Friday (September 4) or >>> so, so that we have some time to do the necessary implementations and >>> testing before alpha. (For example, running manage.py test without >>> migrations currently gives "no such table ...". It would be nice to give a >>> friendlier message if we require migrations for running tests >>> (https://github.com/django/django/pull/5100 does this to some extent, but >>> could be improved I think). >>> >>> I know at least Markus said he would like to present an argument for >>> keeping support for apps without migrations. >>> >>> On Thursday, August 13, 2015 at 1:56:05 PM UTC-4, Tim Graham wrote: >>>> >>>> It's a one line change in the code to keep the status quo from 1.8. The >>>> main drawback I see is continued support tickets about dependencies between >>>> apps with and without migrations. I already removed the documentation >>>> describing the problem and "won't fixed" tickets that suggested doing more >>>> to detect and throw errors about these cases. >>>> >>>> I guess I'd like here why migrations could be a hindrance "for quick >>>> prototyping and initial development." I'd argue that starting with >>>> migrations from the start (manage.py makemigrations && manage.py migrate) >>>> will save the need to learn about --fake-initial if you decide to add >>>> migrations at some point later (and of course avoid dependency issues). Is >>>> "manage.py makemigrations && manage.py migrate" more difficult that the >>>> proposed ""manage.py migrate --allow-apps-without-migrations"? Am I missing >>>> a motive behind the request? >>>> >>>> diff --git a/django/core/management/commands/migrate.py >>>> b/django/core/management >>>> index 2f04824..dec86e1 100644 >>>> --- a/django/core/management/commands/migrate.py >>>> +++ b/django/core/management/commands/migrate.py >>>> @@ -133,7 +133,7 @@ class Command(BaseCommand): >>>> targets = executor.loader.graph.leaf_nodes() >>>> >>>> plan = executor.migration_plan(targets) >>>> - run_syncdb = options.get('run_syncdb') and >>>> executor.loader.unmigrated_a >>>> + run_syncdb = bool(executor.loader.unmigrated_apps) >>>> >>>> # Print some useful info >>>> if self.verbosity >= 1: >>>> >>>> On Thursday, August 6, 2015 at 9:50:10 PM UTC-4, Markus Holtermann >>>> wrote: >>>>> >>>>> Instead of rushing to a decision if and of how we continue to support >>>>> "apps w/o migrations" I would argue that, since Django's test suite >>>>> does >>>>> infact create tables for apps w/o migrations, we could add a >>>>> commandline >>>>> flag (e.g. --run-syncdb or whatever) to the migrate command and warn >>>>> the >>>>> user if there are apps w/o migrations an the user doesn't supplied: >>>>> >>>>> https://github.com/django/django/blob/master/django/core/management/commands/migrate.py#L136 >>>>> >>>>> /Markus >>>>> >>>>> On Thu, Aug 06, 2015 at 05:26:50PM -0700, Andrew Godwin wrote: >>>>> >Do we know exactly what "support for apps without migrations" would >>>>> > consist >>>>> >of at this point? I have that half-done code for replicating syncdb >>>>> > with >>>>> >the autodetector stuck onto the migration executor, but it's not >>>>> > especially >>>>> >speedy and would need some work to make it a sensible speed before it >>>>> > goes >>>>> >forward. >>>>> > >>>>> >Alternately, we could modify the autodetector to have a "create only" >>>>> > mode >>>>> >that's more efficient or just write a new generator that is quick >>>>> > because >>>>> >it knows it's only making nonmigrated apps' migrations. >>>>> > >>>>> >Andrew >>>>> > >>>>> >On Thu, Aug 6, 2015 at 4:56 PM, Tim Graham <timog...@gmail.com> wrote: >>>>> > >>>>> >> With the alpha for 1.9 coming up in 6 weeks, we need to decide >>>>> >> whether or >>>>> >> not to continue support for apps without migrations (currently in >>>>> >> master no >>>>> >> tables are created for such apps (as the deprecation timeline says), >>>>> >> but it >>>>> >> might be appropriate to add a warning or error message for this case >>>>> >> if >>>>> >> things don't change). >>>>> >> >>>>> >> Is anyone sufficiently interested in continuing support for apps >>>>> >> without >>>>> >> migrations to commit to completing it by the alpha? If not, do you >>>>> >> consider >>>>> >> it a release blocker that I should spend time on? >>>>> >> >>>>> >> Related tickets: >>>>> >> >>>>> >> https://code.djangoproject.com/ticket/25144 - No way to create >>>>> >> tables for >>>>> >> apps without migrations >>>>> >> https://code.djangoproject.com/ticket/24919 - Add an option not to >>>>> >> run >>>>> >> migrations when running tests >>>>> >> https://code.djangoproject.com/ticket/24481 - Improve sqlmigrate to >>>>> >> be >>>>> >> more flexible and allow bypassing migrations on disk >>>>> >> https://code.djangoproject.com/ticket/24901 - makemigrations should >>>>> >> create empty migrations dir for any installed app without it >>>>> >> https://code.djangoproject.com/ticket/24588 - Improve handling apps >>>>> >> without migrations while running migrate command. >>>>> >> >>>>> >> On Sunday, January 18, 2015 at 5:11:53 PM UTC-5, Markus Holtermann >>>>> >> wrote: >>>>> >>> >>>>> >>> Creating in-memory migrations for all apps that don't have >>>>> >>> migration >>>>> >>> files seems to be an option to solve the dependency problem. This >>>>> >>> would >>>>> >>> even allow apps without migrations to depend on those with >>>>> >>> migrations. >>>>> >>> >>>>> >>> We have to consider though, that there are tens of apps and >>>>> >>> hundreds of >>>>> >>> models in our own test suite, and generating all migrations during >>>>> >>> start >>>>> >>> seems to be quite an expensive task. And I'm not even talking about >>>>> >>> the >>>>> >>> migration optimizer which probably needs to get a lot smarter if we >>>>> >>> take >>>>> >>> this road. >>>>> >>> >>>>> >>> syncdb, which leaves a developer with a database scheme that cannot >>>>> >>> be >>>>> >>> altered automatically, is something we should get rid of as soon as >>>>> >>> possible, especially since Django has a out-of-the-box migration >>>>> >>> system. >>>>> >>> >>>>> >>> /Markus >>>>> >>> >>>>> >>> On Sun, Jan 18, 2015 at 01:50:14PM -0800, Andrew Godwin wrote: >>>>> >>> >My main argument for removing them entirely was the dependency >>>>> >>> > issues >>>>> >>> >between apps with and without migrations. Having syncdb use >>>>> >>> > SchemaEditor >>>>> >>> is >>>>> >>> >a big step and one I'm happy we've got to, but the only advantage >>>>> >>> > of >>>>> >>> >keeping syncdb is for the test suite and I'd rather we approach >>>>> >>> > that >>>>> >>> more >>>>> >>> >as "migrations made and run at runtime as a special case" rather >>>>> >>> > than >>>>> >>> >"syncdb". >>>>> >>> > >>>>> >>> >If nothing else, I'd like to see the end-developer-facing parts, >>>>> >>> > like >>>>> >>> the >>>>> >>> >syncdb command itself, gone. >>>>> >>> > >>>>> >>> >Andrew >>>>> >>> > >>>>> >>> >On Sun, Jan 18, 2015 at 1:43 PM, Claude Paroz <cla...@2xlibre.net> >>>>> >>> wrote: >>>>> >>> > >>>>> >>> >> Tim recently did a fabulous job of removing deprecated code for >>>>> >>> >> the >>>>> >>> >> future 1.9 on master. Thanks for that. >>>>> >>> >> >>>>> >>> >> However, one thing he removed was support for apps without >>>>> >>> >> migrations. >>>>> >>> >> >>>>> >>> >> >>>>> >>> >>>>> >>> https://github.com/django/django/commit/7e8cf74dc74539f40f4cea53c1e8bba82791fcb6 >>>>> >>> >> >>>>> >>> >> Considering that we have to keep internal support for Django's >>>>> >>> >> own >>>>> >>> test >>>>> >>> >> suite anyway, I wonder if we should remove that support at all >>>>> >>> >> for >>>>> >>> >> "normal" projects. I think one of Andrew's motivation was not to >>>>> >>> >> have >>>>> >>> to >>>>> >>> >> keep two schema editing code bases. But now that the old syncdb >>>>> >>> >> also >>>>> >>> >> uses the new schema editor, I think that this argument doesn't >>>>> >>> >> stand. >>>>> >>> >> >>>>> >>> >> So what about keeping support for apps without migrations in the >>>>> >>> longer >>>>> >>> >> term. Of course, the fact that those apps cannot depend on >>>>> >>> >> migrated >>>>> >>> apps >>>>> >>> >> limits its use, but I think that for quick prototyping and >>>>> >>> >> initial >>>>> >>> >> developement, migrations could be more of a hindrance. Would you >>>>> >>> >> see >>>>> >>> >> major drawbacks with keeping this support? >>>>> >>> >> >>>>> >>> >> Opinions welcome. >>>>> >>> >> >>>>> >>> >> Claude >>>>> >>> >> -- >>>>> >>> >> www.2xlibre.net >>>>> >>> >>>>> >> -- >>>>> >> You received this message because you are subscribed to the Google >>>>> >> Groups >>>>> >> "Django developers (Contributions to Django itself)" group. >>>>> >> To unsubscribe from this group and stop receiving emails from it, >>>>> >> send an >>>>> >> email to django-develop...@googlegroups.com. >>>>> >> To post to this group, send email to django-d...@googlegroups.com. >>>>> >> Visit this group at >>>>> >> http://groups.google.com/group/django-developers. >>>>> >> To view this discussion on the web visit >>>>> >> >>>>> >> https://groups.google.com/d/msgid/django-developers/89839628-6b11-4c61-86ed-4d9a076e9e3d%40googlegroups.com >>>>> >> >>>>> >> <https://groups.google.com/d/msgid/django-developers/89839628-6b11-4c61-86ed-4d9a076e9e3d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> >> . >>>>> >> >>>>> >> For more options, visit https://groups.google.com/d/optout. >>>>> >> >>>>> > >>>>> >-- >>>>> >You received this message because you are subscribed to the Google >>>>> > Groups "Django developers (Contributions to Django itself)" group. >>>>> >To unsubscribe from this group and stop receiving emails from it, send >>>>> > an email to django-develop...@googlegroups.com. >>>>> >To post to this group, send email to django-d...@googlegroups.com. >>>>> >Visit this group at http://groups.google.com/group/django-developers. >>>>> >To view this discussion on the web visit >>>>> > https://groups.google.com/d/msgid/django-developers/CAFwN1urojSqXkaDnjVdrPHrs5TKpQC-baz9TZLtvOZ_3UJ%2BM2Q%40mail.gmail.com. >>>>> >For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> -- >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-developers+unsubscr...@googlegroups.com. >> To post to this group, send email to django-developers@googlegroups.com. >> Visit this group at http://groups.google.com/group/django-developers. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/b079f398-92fb-43e4-8ab5-6c1c8dbc9b09%40googlegroups.com. >> >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/CAFwN1ur_yiBmRuj%2BY26bFNJZe7meZpVsQnTuZLRWB9dj6GBqNg%40mail.gmail.com. > > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CALMtK1Fi9Tc7m38Xc62enWcM_YX8UsKZxqeYwM4GeDXapLmkaQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
