Hello, sorry for the delay (I'm moving home), thanks for the detailed explanations, I've learned a lot about the new status of migrations, and I guess every django users might benefit from this "shift of canonical representation", that's why there shouldbe room for a proper introduction in django docs, explaining all that and especially discarding the belief (which I guess is true, for example, in Sqlalchemy) that ORM classes ARE the source of all other schema representations.
I should draft something like that, but before that, we should advance on the points raised previously. Migrations are a very complex subject (to me at least), and there is a risk of incoherence between the SQL DB schema (as altered by migrations), and what the ORM sees via ORM classes. So having public debugging tools, as proposed by Marc and Aymeric, would be very appreciable. It could be: - an SQL dump of the structure of ORM classes (we would just have to explain that it's an INCOMPLETE view of what might be in real DB, due to RUNSQL/RUNPYTHON and the like) ; a separate command would be better I think, so that people understand it's only "hints". - an autocheck, which ensures for example that at least simple selects() on all known columns fields don't crash, and if possible a column-type check too ; I added somethings like this to an SQLAlchemy-based soft, dunno how hard it'd be in Django, but it saved my life on numerous occasions. Do these propositions sound acceptable ? regards, Pascal 2015-04-02 23:03 GMT+02:00 Marc Tamlyn <marc.tam...@gmail.com>: > As far as I'm aware, we have some code paths which still work in 1.9 which > generate tables directly from the models. We use this when running Django's > own test suite, and it is also used now by djangobench. I haven't looked > into exactly how to turn this into logged SQL rather than run SQL but it > should be possible. > > I think we should document such a tool as a debugging tool rather than a > development tool, but it would be useful nonetheless. Something like > `sqlmigrate --from-clean` might be appropriate. > > Marc > On 2 Apr 2015 21:50, "Aymeric Augustin" < > aymeric.augus...@polytechnique.org> wrote: > >> On 30 mars 2015, at 23:10, Carl Meyer <c...@oddbird.net> wrote: >> >> > So it is not true that the Python models are the canonical >> > representation of your schema, and the SQL DDL is just a translation of >> > them. In fact, your migrations are the (only) canonical representation >> > of your schema, which may include things (specialized indices, views, >> > SQL functions, triggers...) that don't have any direct representation in >> > the Python models at all. The Python models just need to be in-sync >> > enough to generate the correct SQL for queries at run-time, but they >> > often will not fully represent your schema. >> >> While I understand this argument in theory, I'm not sure we can stop >> there in >> practice. I know that anecdote != data but here's my story and what I >> think it >> means for Django. >> >> I started a moderately complex Django-based e-commerce project about one >> year >> ago. We switched to Django 1.7 right after it was released and went live >> a few >> days later. We made the most basic use of migrations: add a model here, >> run >> makemigrations, commit; change a field there, run makemigrations, commit. >> >> We did only one complicated thing. We moved our custom user model because >> of a >> circular import problem. (Pro-tip #1: put your custom user model in an app >> that doesn't depend on anything else. Perhaps we should document that.) We >> papered over the resulting mess by squashing migrations and life was good. >> (Pro-tip #2: if you change AUTH_USER_MODEL, throw your migrations history >> away >> and regenerate it. It's easy. We should definitely document how to do >> that.) >> >> A few weeks ago, we ran into issues I can't describe exactly when >> generating >> or applying new migrations. I'm sorry, I don't remember the details >> because >> I'm not monitoring development very closely. Judging by how much time had >> been >> spent on the issue and how messy it looked, I decided to simply scratch >> our >> migrations history, which we should have done earlier. (See pro-tip #2.) >> >> I wanted to check that the resulting schema matched what we had in >> production >> in order to make sure that I wouldn't introduce inconsistencies. On my >> machine, I emptied all migrations folders, ran makemigrations, created a >> fresh >> database, ran migrate, and dumped the schema. Then I grabbed a dump of the >> production schema. The dumps were about 11 000 lines long. The diff >> produced >> by apgdiff -- a great tool -- was 2 000 lines long. >> >> Much of the diff was noise created by non-deterministic index and >> constraint >> names. I'm almost sure we fixed that in 1.8. I hacked a script to >> renormalize >> hashes and was still left with significant differences. Oops. >> >> These differences happen when Django produces a different schema: >> >> - if you simply create a model >> - if you make a series of changes that result in the same model >> >> These differences appear not only when you reset migrations like I did, >> but >> also if you squash migrations, since the squashed migration is pretty >> much a >> fresh migration that declares that it replaces a bunch of previous >> migrations. >> The history of migrations runs a series of alterations. The squashed >> migration >> runs a simple creation. >> >> Here are the differences I found and couldn't explain by problems in our >> code: >> >> - Sequences aren't handled correctly when changing an AutoField into an >> IntegerField. I filed #24533. This could result in different behavior in >> dev/test and production, which I find dangerous. I could probably >> present it >> as a crashing or data loss issue. >> >> - Some unique indexes on OneToOne fields were missing. This may be a >> variant >> of #24163. I haven't investigated it fully. This is about as bad as the >> previous one. >> >> - Many varchar_pattern_ops indexes were missing in production. See #23954. >> This could result in performance issues. >> >> - I had a constraint generated twice locally, once in prod. I'm not sure >> why. >> >> Given that basic use of makemigrations & migrate can result in significant >> errors in the resulting database schema, in the current state of >> migrations, >> we cannot promote them as the single way to manage the database schema. >> The >> risk of not creating the expected schema -- or, worse, not having created >> the >> expected schema in the past -- is simply too high. Every project that ever >> used 1.7 or 1.7.1 suffers from such issues. >> >> Therefore I think we must document how our users with basic needs, like >> myself, can obtain the reference database schema corresponding to their >> models, assuming that they never used RunSQL or that they know what they >> did with it. >> >> I’m arguing that there’s a practical need here and practicality beats >> purity. >> >> -- >> Aymeric. >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-developers+unsubscr...@googlegroups.com. >> To post to this group, send email to django-developers@googlegroups.com. >> Visit this group at http://groups.google.com/group/django-developers. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/D3FF8860-764D-48EB-AAEE-8CE64B9ABED5%40polytechnique.org >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "Django developers (Contributions to Django itself)" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/django-developers/h92CcblbYfk/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/CAMwjO1F8VO04xEG9-PEP7SojvG%2BZmtGZHbhpV12f%3DFEZKEBJ2A%40mail.gmail.com > <https://groups.google.com/d/msgid/django-developers/CAMwjO1F8VO04xEG9-PEP7SojvG%2BZmtGZHbhpV12f%3DFEZKEBJ2A%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CA%2BXNUS2_WASfPW1dkKDSzdrZZ8dvpG7_Op%3D%3Du9pgLicmdKYr0Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
