Ah, but the point I'm trying to make is that they're not two separate notions at all.
The current migrations system was deliberately designed in a way such that it is both - it's a representation of changes through time, sure, but it's also an entire statement of how the database is made. Migrations actually runs by running through the migrations in memory to build up models and using those for operations (it's theoretically possible to throw away models.py files and just run from migrations' end state, but that wouldn't be very useable). This is important as there are some DDL things that Django cannot easily extract from the database. For example, if you made a partial index in a migration with RunSQL, "sqlall" and Django's introspection library didn't know how to get that out of the database, and the place it used to go before migrations (custom SQL files) are gone, so it would miss it out and give you incorrect results. However, running through migrations and using those as a source of truth does give you the right results - the migrations _are_ what the database should be. Django sees them as the exact set of statements needed to make a new database, and so if you want a schema, where better to get one from? The current set of models isn't expressive enough to contain all DDL changes (though some projects are working on improving that, e.g. Marc Tamlyn's Postgres work), and why throw away the data when we already have it? The code inside Django that makes sqlall/sqlclear/etc is now a deprecated codepath, since migrations uses a brand new and improved codepath to do all its changes; the plan was always to introduce migrations and this new DDL codepath, and slowly retire the old one. If we were to keep sqlall it would need rewriting to run on top of the new schema editing code - which is entirely possible, but realise that you're asking us for a lot more work, rather than just simply asking to leave existing code in place. My preferred solution, instead of reimplementing all these sql* commands, is to just improve upon sqlmigrate so it can give you this singular schema dump easily, and take the place of sqlall and sqlclear (which are the forwards and backwards directions respectively). flush/sqlflush are different, as they're not DDL but DML, and so less affected. I'm open to someone reimplementing the old sql commands on the new system, but realise it's not easy work to make it so it doesn't miss out things here and there and does it properly (it's also not too much more than wiring the autodetector and the SQL output system together in a few different ways, kind of like makemigrations + sqlmigrate but without the middle step of saving migrations to disk) Andrew On Mon, Mar 30, 2015 at 1:57 PM, Pascal Chambon <chambon.pas...@gmail.com> wrote: > Thanks for your answers, > > l'm confused nonetheless, because their are two notions mixing up, here, > that we should probably separate: > > - the HISTORY of SQL schemas, aka "django/south migrations" > - the CURRENT STATE of SQL schemas, that I'll call "ORM Models Dump" > > Let's leave the SQL data (and its migrations) out of the equation please, > here i'm only caring about the structure of data, the schemas, and their > corresponding SQL DDL. > > I completely understand and agree that migrations are the way to go to > upgrade between app version, not only because it allows adding/removing > columns (which syncdb didn't do, previously), but also doing all sorts of > on-the-fly custom conversions. > > However, what I am seeking, is not a dump of successive migrations (even > aggregated), it's an "ORM models dump", i.e a "transliteration" of django > ORM classes to their corresponding SQL CREATE statements. Just like a > "mysqldump --no-data", except that it's the logical, theoretical schema > that I want here, not the real, SQL-side one. > > I really do not believe that we need all the history of schema changes to > get this schema dump ; AFAIK, it's the django ORM models which define the > logical schema of data, not the migrations ; if a class MyUser defines a > field "my_adress", and migrations have no track of it, it's the migrations > that are wrong/incomplete, not the python code. Both "current models" and > "the sum of all migrations" are supposed to be in sync at any time, arent > they ? Custom SQL code is thus not a problem, since we don't care about > the path taken, just were we are currently. > > So if django was probably able to dump all the DDL for its ORM models in > the past, why couldn't it continue to do it ? > > As for the question: > *"why do you want it"? manage.py migration *should* be handling all your > table creation and migration functionality, so the need to manually diff a > schema shouldn't exist* > > My answer is : for the same reason that I need all other debug tools > available, from django-debug-toolbar to lowest-level cpython debuggers. ^^ > > I see tons of ways how stuffs could go wrong, from an accidental > interruption of migrations (DDL instructions often having the flaw of being > non-rollbackable), to bugs in migrations' code, without forgetting > unorthodoxes events which may have happend to a project's conf (eg. > switching from a USER_MODEL to another, and thus having to fix stuffs the > hard way, with "--fake" migrations if necessary), or the handling of apps > that have no support for migrations. > > Sure, people encountering troubles might make like I did - start a brand > new project and mysqldump it after whole migration - but if we can expose > that information in a straightforward way, like it was before, it's all > benefits for everyone, isn't it ? > > regards, > Pascal > > > > > > > > 2015-03-30 5:59 GMT+02:00 Andrew Godwin <and...@aeracode.org>: > >> I must also point out that the sqlmigrate command will still get you SQL >> from migrations, though it sadly lacks the ability to take a range of >> migrations, optimise them down, and output the SQL for _that_ - that's >> probably the best thing for us to aim towards, and will almost entirely >> recreate the functionality of sqlall without actually being wrong (sqlall >> doesn't understand things like data migrations or custom SQL sections of >> migrations, and so will sadly be wrong in if you have anything like a >> moderately complex migration set). >> >> The plus side is that it would be possible to do this sort of thing as a >> third-party command/backport if we wanted to have it usable on 1.8 and even >> 1.7 (the 1.8 feature set is frozen now, or I'd suggest quickly squeezing it >> in). >> >> Andrew >> >> On Sun, Mar 29, 2015 at 4:57 PM, Russell Keith-Magee < >> russ...@keith-magee.com> wrote: >> >>> >>> On Mon, Mar 30, 2015 at 1:36 AM, Pkl <chambon.pas...@gmail.com> wrote: >>> >>>> Hello all, >>>> >>>> I've noticed that all sql* commands that were in django core management >>>> have disappeared from the latest sources. >>>> >>>> However, even though Django now has builtin south-style migrations now, >>>> these *sql** commands (especially sql-all) were priceless to debug >>>> differences between the models and the actual DB schema. >>>> Especially as long as some django apps haven't caught up with all these >>>> recent changes - I myself ended up with missing tables after scrupulously >>>> following upgrade instructions, with "manage.py migrate" saying "all is >>>> fine". >>>> >>>> *What is the new way to dump the sql schema of currently installed >>>> django appz ?* It'd maybe be worth that I provide a doc patch to >>>> inform users about it. >>>> *If there is none, is there an agreement to restore at least sqlall and >>>> sqlclear ?* >>>> >>> >>> No, there isn't :-) >>> >>> Also what was the reasoning behind, on the first, place, blocking access >>>> to these utilities in django1.7 (they seemed to work, when I forced them), >>>> and now removing them ? A simple warning about the new "migrations" stuffs >>>> (or a "--force" argument) would most probably have sufficed to prevent >>>> improper usage of these sql* commands, while letting django users getting >>>> the info they need to move forward, wouldn't it ? >>>> >>> >>> Right now, there are two (almost) completely independent implementations >>> of SQL table generation logic: >>> >>> * The legacy version, used by syncdb, sqlall, et al >>> >>> * The new version, used by migrate. >>> >>> In the past, there was a good reason for sqlall to exist - you needed to >>> be able to run syncdb and create tables. The only time you ever called >>> "CREATE TABLE" was when a table was fresh, and after that, you never issued >>> any command related to the table schema. It didn't hurt anyone to allow >>> sqlall to generate "what would the CREATE TABLE be if we had clean >>> databases", and it was at least partially useful as a migration assistant, >>> so the sqlall functionality was available. >>> >>> In the era of migrations, there's still a need to do a CREATE TABLE, but >>> the table creation logic is a lot more complex, because it's not *just* a >>> CREATE TABLE that is needed; all the ALTER TABLE statements are needed to >>> add/remove columns as well. >>> >>> It would probably be *possible* to refactor manage.py sqlall to use the >>> new table creation logic; but the question would then be "why do you want >>> it"? manage.py migration *should* be handling all your table creation and >>> migration functionality, so the need to manually diff a schema shouldn't >>> exist. If you've hit problems with "manage.py migrate" *not* identifying >>> migrations, then that's a much bigger problem, and one that needs to be >>> reported as a bug - because it's a huge bug. Anything you can do to narrow >>> down the situation that led to the problems you've observed would be very >>> helpful. >>> >>> Yours, >>> Russ Magee %-) >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-developers+unsubscr...@googlegroups.com. >>> To post to this group, send email to django-developers@googlegroups.com. >>> Visit this group at http://groups.google.com/group/django-developers. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/CAJxq84_Ww_ZcuhEx8MmfcxwY%3DGpMzuPGh1maGxBDJFCikUO5MA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/django-developers/CAJxq84_Ww_ZcuhEx8MmfcxwY%3DGpMzuPGh1maGxBDJFCikUO5MA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/django-developers/h92CcblbYfk/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> django-developers+unsubscr...@googlegroups.com. >> To post to this group, send email to django-developers@googlegroups.com. >> Visit this group at http://groups.google.com/group/django-developers. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/CAFwN1upDReQpmz%2BVDaHy_SVa8Q_KtEikO-YTyz7RyDcHJAVD%3DQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/django-developers/CAFwN1upDReQpmz%2BVDaHy_SVa8Q_KtEikO-YTyz7RyDcHJAVD%3DQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/CA%2BXNUS2qpByi%3D0n%3D35Hn-HhXg0xAJxY72EeiBrA4P1DEpjS7rQ%40mail.gmail.com > <https://groups.google.com/d/msgid/django-developers/CA%2BXNUS2qpByi%3D0n%3D35Hn-HhXg0xAJxY72EeiBrA4P1DEpjS7rQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uo_VSbR-ZXeBWwiu7sVvC28-2ZRp__C2%2By7h9iLBg-T8w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
