Re: Fate of sql* management commands
On Sun, Mar 29, 2015 at 4:57 PM, Russell Keith-Magee wrote: > It would probably be *possible* to refactor manage.py sqlall to use the new > table creation logic; but the question would then be "why do you want it"? Speaking from experience, I have found these commands to be a extremely helpful when migrating a legacy application to Django. In my situation, the legacy application would do whatever it wanted with the schema and was not very opinionated nor well maintained. When moving to Django having a quick way to dump Django's *ideal* SQL representation was a very useful way to debug what needed to change in the legacy application. There may even be a interim where SQL migrations are handled outside Django but with the goal of moving towards Django's representation. Having a way to diff actual vs expected is very useful for this situation. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CADhq2b5x66TN1mky3PoZoJNd78RtVA0ee951kd7_k4Qy%3DVPHHg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Fate of sql* management commands
Ah, but the point I'm trying to make is that they're not two separate notions at all. The current migrations system was deliberately designed in a way such that it is both - it's a representation of changes through time, sure, but it's also an entire statement of how the database is made. Migrations actually runs by running through the migrations in memory to build up models and using those for operations (it's theoretically possible to throw away models.py files and just run from migrations' end state, but that wouldn't be very useable). This is important as there are some DDL things that Django cannot easily extract from the database. For example, if you made a partial index in a migration with RunSQL, "sqlall" and Django's introspection library didn't know how to get that out of the database, and the place it used to go before migrations (custom SQL files) are gone, so it would miss it out and give you incorrect results. However, running through migrations and using those as a source of truth does give you the right results - the migrations _are_ what the database should be. Django sees them as the exact set of statements needed to make a new database, and so if you want a schema, where better to get one from? The current set of models isn't expressive enough to contain all DDL changes (though some projects are working on improving that, e.g. Marc Tamlyn's Postgres work), and why throw away the data when we already have it? The code inside Django that makes sqlall/sqlclear/etc is now a deprecated codepath, since migrations uses a brand new and improved codepath to do all its changes; the plan was always to introduce migrations and this new DDL codepath, and slowly retire the old one. If we were to keep sqlall it would need rewriting to run on top of the new schema editing code - which is entirely possible, but realise that you're asking us for a lot more work, rather than just simply asking to leave existing code in place. My preferred solution, instead of reimplementing all these sql* commands, is to just improve upon sqlmigrate so it can give you this singular schema dump easily, and take the place of sqlall and sqlclear (which are the forwards and backwards directions respectively). flush/sqlflush are different, as they're not DDL but DML, and so less affected. I'm open to someone reimplementing the old sql commands on the new system, but realise it's not easy work to make it so it doesn't miss out things here and there and does it properly (it's also not too much more than wiring the autodetector and the SQL output system together in a few different ways, kind of like makemigrations + sqlmigrate but without the middle step of saving migrations to disk) Andrew On Mon, Mar 30, 2015 at 1:57 PM, Pascal Chambon wrote: > Thanks for your answers, > > l'm confused nonetheless, because their are two notions mixing up, here, > that we should probably separate: > > - the HISTORY of SQL schemas, aka "django/south migrations" > - the CURRENT STATE of SQL schemas, that I'll call "ORM Models Dump" > > Let's leave the SQL data (and its migrations) out of the equation please, > here i'm only caring about the structure of data, the schemas, and their > corresponding SQL DDL. > > I completely understand and agree that migrations are the way to go to > upgrade between app version, not only because it allows adding/removing > columns (which syncdb didn't do, previously), but also doing all sorts of > on-the-fly custom conversions. > > However, what I am seeking, is not a dump of successive migrations (even > aggregated), it's an "ORM models dump", i.e a "transliteration" of django > ORM classes to their corresponding SQL CREATE statements. Just like a > "mysqldump --no-data", except that it's the logical, theoretical schema > that I want here, not the real, SQL-side one. > > I really do not believe that we need all the history of schema changes to > get this schema dump ; AFAIK, it's the django ORM models which define the > logical schema of data, not the migrations ; if a class MyUser defines a > field "my_adress", and migrations have no track of it, it's the migrations > that are wrong/incomplete, not the python code. Both "current models" and > "the sum of all migrations" are supposed to be in sync at any time, arent > they ? Custom SQL code is thus not a problem, since we don't care about > the path taken, just were we are currently. > > So if django was probably able to dump all the DDL for its ORM models in > the past, why couldn't it continue to do it ? > > As for the question: > *"why do you want it"? manage.py migration *should* be handling all your > table creation and migration functionality, so the need to manually diff a > schema shouldn't exist* > > My answer is : for the same reason that I need all other debug tools > available, from django-debug-toolbar to lowest-level cpython debuggers. ^^ > > I see tons of ways how stuffs could go wrong, from an accidental > interruption of migrations (DDL i
Re: Fate of sql* management commands
Hi Pascal, On 03/30/2015 02:57 PM, Pascal Chambon wrote: > Thanks for your answers, > > l'm confused nonetheless, because their are two notions mixing up, here, > that we should probably separate: > > - the HISTORY of SQL schemas, aka "django/south migrations" > - the CURRENT STATE of SQL schemas, that I'll call "ORM Models Dump" > > Let's leave the SQL data (and its migrations) out of the equation > please, here i'm only caring about the structure of data, the schemas, > and their corresponding SQL DDL. > > I completely understand and agree that migrations are the way to go to > upgrade between app version, not only because it allows adding/removing > columns (which syncdb didn't do, previously), but also doing all sorts > of on-the-fly custom conversions. > > However, what I am seeking, is not a dump of successive migrations (even > aggregated), it's an "ORM models dump", i.e a "transliteration" of > django ORM classes to their corresponding SQL CREATE statements. Just > like a "mysqldump --no-data", except that it's the logical, theoretical > schema that I want here, not the real, SQL-side one. > > I really do not believe that we need all the history of schema changes > to get this schema dump ; AFAIK, it's the django ORM models which define > the logical schema of data, not the migrations ; if a class MyUser > defines a field "my_adress", and migrations have no track of it, it's > the migrations that are wrong/incomplete, not the python code. Both > "current models" and "the sum of all migrations" are supposed to be in > sync at any time, arent they ? Custom SQL code is thus not a problem, > since we don't care about the path taken, just were we are currently. I understand what you are saying, but your understanding is no longer correct. Migrations are (intentionally) powerful, and you are permitted to include arbitrary SQL DDL in your migrations (via the RunSQL migration). This DDL is part of your schema just as much as any DDL auto-generated by Django in a CreateModel operation is, and your code may depend on it. So if we dumped a schema directly from the models, ignoring migrations, it could be missing crucial elements of your actual schema. So it is not true that the Python models are the canonical representation of your schema, and the SQL DDL is just a translation of them. In fact, your migrations are the (only) canonical representation of your schema, which may include things (specialized indices, views, SQL functions, triggers...) that don't have any direct representation in the Python models at all. The Python models just need to be in-sync enough to generate the correct SQL for queries at run-time, but they often will not fully represent your schema. There should be only one canonical representation of your schema. This canonical representation cannot possibly be anything but your migrations, unless we removed the ability to run arbitrary DDL in migrations (which is not an option; that's too useful). Therefore, migrations are the canonical schema. Therefore, we won't introduce any way to "dump SQL schema" unless it is based on the contents of your migrations (the approach Andrew mentioned). > So if django was probably able to dump all the DDL for its ORM models in > the past, why couldn't it continue to do it ? Because in the past we did not have migrations, and we don't want two different (possibly inconsistent) canonical representations of your schema. The old way of generating SQL DDL directly from models is deprecated, and will disappear entirely in a future version of Django. The new way of generating SQL DDL from models _is_ the migrations system. Carl -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5519BBCC.6070901%40oddbird.net. For more options, visit https://groups.google.com/d/optout. signature.asc Description: OpenPGP digital signature
Re: Fate of sql* management commands
Thanks for your answers, l'm confused nonetheless, because their are two notions mixing up, here, that we should probably separate: - the HISTORY of SQL schemas, aka "django/south migrations" - the CURRENT STATE of SQL schemas, that I'll call "ORM Models Dump" Let's leave the SQL data (and its migrations) out of the equation please, here i'm only caring about the structure of data, the schemas, and their corresponding SQL DDL. I completely understand and agree that migrations are the way to go to upgrade between app version, not only because it allows adding/removing columns (which syncdb didn't do, previously), but also doing all sorts of on-the-fly custom conversions. However, what I am seeking, is not a dump of successive migrations (even aggregated), it's an "ORM models dump", i.e a "transliteration" of django ORM classes to their corresponding SQL CREATE statements. Just like a "mysqldump --no-data", except that it's the logical, theoretical schema that I want here, not the real, SQL-side one. I really do not believe that we need all the history of schema changes to get this schema dump ; AFAIK, it's the django ORM models which define the logical schema of data, not the migrations ; if a class MyUser defines a field "my_adress", and migrations have no track of it, it's the migrations that are wrong/incomplete, not the python code. Both "current models" and "the sum of all migrations" are supposed to be in sync at any time, arent they ? Custom SQL code is thus not a problem, since we don't care about the path taken, just were we are currently. So if django was probably able to dump all the DDL for its ORM models in the past, why couldn't it continue to do it ? As for the question: *"why do you want it"? manage.py migration *should* be handling all your table creation and migration functionality, so the need to manually diff a schema shouldn't exist* My answer is : for the same reason that I need all other debug tools available, from django-debug-toolbar to lowest-level cpython debuggers. ^^ I see tons of ways how stuffs could go wrong, from an accidental interruption of migrations (DDL instructions often having the flaw of being non-rollbackable), to bugs in migrations' code, without forgetting unorthodoxes events which may have happend to a project's conf (eg. switching from a USER_MODEL to another, and thus having to fix stuffs the hard way, with "--fake" migrations if necessary), or the handling of apps that have no support for migrations. Sure, people encountering troubles might make like I did - start a brand new project and mysqldump it after whole migration - but if we can expose that information in a straightforward way, like it was before, it's all benefits for everyone, isn't it ? regards, Pascal 2015-03-30 5:59 GMT+02:00 Andrew Godwin : > I must also point out that the sqlmigrate command will still get you SQL > from migrations, though it sadly lacks the ability to take a range of > migrations, optimise them down, and output the SQL for _that_ - that's > probably the best thing for us to aim towards, and will almost entirely > recreate the functionality of sqlall without actually being wrong (sqlall > doesn't understand things like data migrations or custom SQL sections of > migrations, and so will sadly be wrong in if you have anything like a > moderately complex migration set). > > The plus side is that it would be possible to do this sort of thing as a > third-party command/backport if we wanted to have it usable on 1.8 and even > 1.7 (the 1.8 feature set is frozen now, or I'd suggest quickly squeezing it > in). > > Andrew > > On Sun, Mar 29, 2015 at 4:57 PM, Russell Keith-Magee < > russ...@keith-magee.com> wrote: > >> >> On Mon, Mar 30, 2015 at 1:36 AM, Pkl wrote: >> >>> Hello all, >>> >>> I've noticed that all sql* commands that were in django core management >>> have disappeared from the latest sources. >>> >>> However, even though Django now has builtin south-style migrations now, >>> these *sql** commands (especially sql-all) were priceless to debug >>> differences between the models and the actual DB schema. >>> Especially as long as some django apps haven't caught up with all these >>> recent changes - I myself ended up with missing tables after scrupulously >>> following upgrade instructions, with "manage.py migrate" saying "all is >>> fine". >>> >>> *What is the new way to dump the sql schema of currently installed >>> django appz ?* It'd maybe be worth that I provide a doc patch to inform >>> users about it. >>> *If there is none, is there an agreement to restore at least sqlall and >>> sqlclear ?* >>> >> >> No, there isn't :-) >> >> Also what was the reasoning behind, on the first, place, blocking access >>> to these utilities in django1.7 (they seemed to work, when I forced them), >>> and now removing them ? A simple warning about the new "migrations" stuffs >>> (or a "--force" argument) would most probably have sufficed to prevent >>> improper usage of t
Re: Help needed with Oracle GIS backend
On Thu, 26 Mar 2015 22:29:00 +0200 Shai Berger wrote: > Hi Jani. > > On Wednesday 25 March 2015 09:58:00 Jani Tiainen wrote: > > > > We're still running Oracle and GIS. Though we do have built custom > > backend based on Django GIS backend - mainly to have support for > > Oracle XE, 3D functionality and in general to make it faster. > > > > It's currently closed source but we're working on to open source it > > in hope that it could interest more wider audience (mainly because it > > would enable GIS use with free Oracle XE). > > > That is good news. I would like to draw your attention to > https://code.djangoproject.com/ticket/21273 where a partial attempt was > already made to add GIS/Oracle-XE support. > > Thanks, > Shai. Well, last comment is mine :) describing exactly how our backend does work in general level and explains rationales why we chose a slightly different approach. -- Jani Tiainen -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20150330163219.4b0fe487%40jns42-l.w2k.keypro.fi. For more options, visit https://groups.google.com/d/optout.
