Hi Russell, On the first issue: Good point. I have not the opportunity to work with such a huge database.
On the second issue: Yes, what I am doing now is really about writing conversion code to fit the old json to match the new schema. I find this to be quite straight forward for my use cases, it's only modifying a list of hash objects. But I probably haven't encounter the more complicated use cases out there. I feel the fact that Django ORM works, dumpdata+loaddata works, ... makes it compelling as the foundation for a migration tool. Developers can work with 100% Python code, not needing to learn migration tools specific APIs, and maybe easier to provide some logic in the migration process (for example, calculating default values for new fields, based on some existing database data). Also, independent Django app (as in INSTALLED_APP) developers can all rely on the serialized JSON format, and provide their own migration code, when they release new version of their app. (How is this currently done anyway? Lets say if django-tagging changes the model class in the next release.) Thank you for the reply. PS: As for this three points ... >>> * If you add a new non-null field, the fixture won't load because it won't >>> provide data for that field. >>> * If you change the name of a field, the fixture will contain data for the >>> old name, but no data for the new name. Any existing data in that field >>> will be lost. >>> * If you change the type of a field, there is no guarantee that the old >>> data will be format-compatible with the new field. ... the old fixtures are loaded with old model class. The serialized json is modified to fit the new model class, and then loaded using the new model class with "manage loaddata". This is done on a per app basis (only for the app that needs the migration). On Jun 3, 2:57 pm, Russell Keith-Magee <freakboy3...@gmail.com> wrote: > On Wed, Jun 3, 2009 at 12:06 PM, Kegan <ke...@kegan.info> wrote: > > > Hi, > > > About Django database migration. I know there's a couple of tools > > available (South, evolution, dmigration, etc), but I am pondering an > > alternative here. Hope good discussion entails. > > > This is what I am doing manually now for my database migration needs. > > > 1. Use Django's management command "dumpdata" to get the JSON > > representative of an app. Save the JSON into a file (oldmodel.json). > > 2. Git pull the latest code. And do a reset to the app. So the > > database will have the new model schema now. > > 3. Use the python shell to migrate the JSON file to match the new > > model representation. Save the new JSON into a file (newmodel.json). > > 4. Use management command "loaddata" to populate the new model with > > the newmodel.json. > > > This has work well so far for my use with PostgreSQL, albeit very > > manual. It's all Python code. > > > So I guess what I want to ask Django experts/developers here is that: > > > 1. Is there any reason this shouldn't be done? > > 2. Any technical challenge that prevent this method from being > > realized into a more automate tool? > > The answer to these two questions is essentially the same - there are > 2 technical challenges that provides reasons why you might not want to > commit to this path. > > Firstly, there is a simple matter of time and space. If you have a > small database, deserializing then reserializing won't take too long, > and won't result in huge files. However, I routinely work with a > database that contains 5GB of data (and growing) - and that's when > it's neatly packed into a database binary format. Spooled out into a > relatively space-inefficient serialized form will consume much more > disk space. On top of that, the time required to produce and reload a > fixture in this format would be prohibitive. > > Secondly, there is the limitations of the approach itself. The > fixture-based migration tool will really only help with simple > migrations where the schema isn't really changing that much - for > example, increasing the size of a CharField. However, for any > non-trivial change, you will hit problems. For example: > > * If you add a new non-null field, the fixture won't load because it > won't provide data for that field. > > * If you change the name of a field, the fixture will contain data > for the old name, but no data for the new name. Any existing data in > that field will be lost. > > * If you change the type of a field, there is no guarantee that the > old data will be format-compatible with the new field. > > So while the dump+load approach will work for simple cases, in > practice, it doesn't work at all. You might be able to work around the > second problem by writing a conversion tool for fixtures that modifies > the old fixture to suit the new format, but you will still be left > with the first problem, now amplified by the need to create and store > an updated fixture. > > > (I read somewhere that dumpdata > > doesn't always work?) > > I've heard many make this claim, but very few have been able to back > it up. I would be the last person to claim that the serializers are > bug free, but I would claim that they are very stable, and there is an > extensive test suite to back up my claim. There are a small number of > known problems, but they tend to be in fairly esoteric areas of > serialization, and shouldn't be encountered by most users. > > If anyone finds a bug in the serializers, I strongly encourage them to > report the problem. Problems only get fixed when they are reported. > > > 3. What do you think about having migration code be more part of the > > deployment tool, rather than couple into the source code ? > > I would heartily agree. However, I would also claim that the tools you > mention (Evolution, South etc) are deployment tools and are in no way > coupled to source code. Migrations don't interact with the day-to-day > usage of your code - they are only ever invoked by the deployment tool > as part of a syncdb/migrate command. > > Yours > Russ Magee %-) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
