Re: An idea on DB migration
On 7/6/07, SmileyChris <[EMAIL PROTECTED]> wrote: > > If you haven't already, check out http://www.aswmc.com/dbmigration/ > > It's quite a good solution for migration. > I've been writing a wrapper on top of that to handle auto-migration, > which I'm mostly happy with. Just have to tie some loose ends and I'll > put it up somewhere to share. Thanks for the link! My main concern about this method: I have to use SQL. I don't really understand SQL, and I'm very happy to let Django write it up for me. When I write it, I'm never sure if it will exactly fit what Django expects, or if it will be efficient (Should I define some columns as indices? As relations?) If my method was available, I wouldn't have to write a single line of SQL, and I would still be able to do whatever I want. Have a good day, Noam --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: An idea on DB migration
If you haven't already, check out http://www.aswmc.com/dbmigration/ It's quite a good solution for migration. I've been writing a wrapper on top of that to handle auto-migration, which I'm mostly happy with. Just have to tie some loose ends and I'll put it up somewhere to share. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: An idea on DB migration
On 7/4/07, Carl Karsten <[EMAIL PROTECTED]> wrote: > > The problem I see is maintaining the generated file. This process will often > start before the target system is stable, so as soon as you start making > changes > to the target model, the converter will need to be updated too. If the > transformation formulas are part of one of the models, then there isn't > anything > to keep in sync. > You don't have to maintain the generated file. After you've updated the database, there's no need to keep it. When you want another update to the database, you create a new file and change there what you want. > > > > >> Also, along the lines of slow... you are going to love this one: how about > >> making it use the web server as the source which would allow migrating > >> between 2 > >> servers where http is the only exposed protocol? it would also take care > >> of > >> migrating between systems where I don't have both db stacks installed on > >> one box > >> - consider win/mssql -> linux/SqlLite. yeah yeah, django/mssql doesn't > >> exist > >> yet... but it makes a good example :) > > > > I think that for this you can use "./manage.py dumpdata" and > > "./manage.py loaddata". See > > http://groups.google.com/group/django-users/msg/02f5447f41207a65 > > But then you don't get any of the nifty transformations which is kinda the > point > of your idea. > Well, you can separate nifty transformations (which will be done on one computer) and system migration. Noam --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: An idea on DB migration
Noam Raphael wrote: > On 7/4/07, Carl Karsten <[EMAIL PROTECTED]> wrote: >> I don't like the "edit the file" part. I can think of 2 alternatives: >> >> 1. add an attribute to one/both of the models. something like >> >> class Book(models.Model): >> name = models.CharField(blank=True, "migrate_to"="book_name" ) >> >> 2. a 3rd file that defines those kind of translations, which might be about >> the >> same by just subclassing the existing models. >> >> class Book(models.Book): >> book_name = Book.name() >> >> (I am totally winging it here, so forgiveme if my brain storm has some >> turbulence.) >> > > What I like about my idea is that you don't need to learn new syntax > for migration. And syntax like this will never be enough - for > example, how would you convert a field with a full name into two > fields, the first name and last name? Using my idea, it will simply > be: > > n.first_name, n.last_name = o.name.split() > > (Well, you would have to add some error handling, but it won't be a > lot more complicated.) The problem I see is maintaining the generated file. This process will often start before the target system is stable, so as soon as you start making changes to the target model, the converter will need to be updated too. If the transformation formulas are part of one of the models, then there isn't anything to keep in sync. maybe put the transformation in the target model: def getfirst(name): return name.split()[0] def getfirst(name): return name.split()[1] first_name = models.CharField(blank=True, "migrate_from"="getfirst(name)" ) last_name = models.CharField(blank=True, "migrate_from"="getlast(name)" ) but I am not thrilled about jamming a bunch of conversion stuff in the new model either. conversion stuff sucks. > >> Also, along the lines of slow... you are going to love this one: how about >> making it use the web server as the source which would allow migrating >> between 2 >> servers where http is the only exposed protocol? it would also take care of >> migrating between systems where I don't have both db stacks installed on one >> box >> - consider win/mssql -> linux/SqlLite. yeah yeah, django/mssql doesn't exist >> yet... but it makes a good example :) > > I think that for this you can use "./manage.py dumpdata" and > "./manage.py loaddata". See > http://groups.google.com/group/django-users/msg/02f5447f41207a65 But then you don't get any of the nifty transformations which is kinda the point of your idea. Carl K --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: An idea on DB migration
On 7/4/07, nick feng <[EMAIL PROTECTED]> wrote: > The django has already provided the way to use SQL, I don't think this is > needed. Well, I'm not a SQL expert, and I don't feel comfortable with adding fields manually. I'm never sure whether my SQL statement is what Django would have issued when creating the table - not to mention relations and indices. If I don't need to know SQL to use Django, why should I need to learn SQL to modify my models? Noam --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: An idea on DB migration
On 7/4/07, Carl Karsten <[EMAIL PROTECTED]> wrote: > > I don't like the "edit the file" part. I can think of 2 alternatives: > > 1. add an attribute to one/both of the models. something like > > class Book(models.Model): > name = models.CharField(blank=True, "migrate_to"="book_name" ) > > 2. a 3rd file that defines those kind of translations, which might be about > the > same by just subclassing the existing models. > > class Book(models.Book): > book_name = Book.name() > > (I am totally winging it here, so forgiveme if my brain storm has some > turbulence.) > What I like about my idea is that you don't need to learn new syntax for migration. And syntax like this will never be enough - for example, how would you convert a field with a full name into two fields, the first name and last name? Using my idea, it will simply be: n.first_name, n.last_name = o.name.split() (Well, you would have to add some error handling, but it won't be a lot more complicated.) > Also, along the lines of slow... you are going to love this one: how about > making it use the web server as the source which would allow migrating > between 2 > servers where http is the only exposed protocol? it would also take care of > migrating between systems where I don't have both db stacks installed on one > box > - consider win/mssql -> linux/SqlLite. yeah yeah, django/mssql doesn't exist > yet... but it makes a good example :) I think that for this you can use "./manage.py dumpdata" and "./manage.py loaddata". See http://groups.google.com/group/django-users/msg/02f5447f41207a65 Noam --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: An idea on DB migration
The django has already provided the way to use SQL, I don't think this is needed. - Original Message - From: "Noam" <[EMAIL PROTECTED]> To: "Django users" <django-users@googlegroups.com> Sent: Thursday, July 05, 2007 2:40 AM Subject: An idea on DB migration > > > Hello, > > I have an idea on how DB migration can work in Django. It's very > simple, it will allow the programmer to do anything he wants in the > migration, and he will have to learn almost nothing in order to > migrate the DB. The drawback is that it's not very efficient, which > means that it will probably work only for small to medium databases - > but I think that most Django sites are like this. If you have a bigger > site, you will have to use SQL. > > My idea is this: create a tool which will create a script which will > migrate the DB. The user will then edit the script as he likes, and > run it to create a new DB with the migrated data. If the result is not > what he wanted, he will edit the script again and run it again. > > For example, say that I have a directory with the old site (and old > models) and a directory with the new site. The only difference between > them is that I renamed the field store.Book.name to > store.Book.book_name. I will run "django-admin.py migrate oldsitedir > newsitedir". This will create a file which will look like this: > > = > #!/usr/bin/env python > > from django.db import ... > > class old_settings: >DATABASE_ENGINE = 'mysql' >DATABASE_NAME = 'store_3' >... > > class new_settings: >DATABASE_ENGINE = 'mysql' >DATABASE_NAME = 'store_4' >... > > old_connection = make_connection_from_settings(old_settings) > new_connection = make_connection_from_settings(new_settings) > > # Old models > > class old: >class auth: >class User(models.Model): >_connection = old_connection # or whatever will be needed > >username = models.CharField(unique=True, maxlength=30) >... > >class store: >class Book(models.Model): >_connection = old_connection > >name = models.CharField() >... > > # New models > > class new: >... >class store: >class Book(models.Model): >_connection = old_connection > >book_name = models.CharField() >... > > # Migration function > > def migrate(): >drop_tables_in_new_db_and_sync_it() > ># Migrate auth.User >for o in old.auth.User.objects.all(): >n = new.auth.User() >n.id = o.id >n.username = o.username >... >n.save() > ># Migrate store.Book >for o in old.store.Book.objects.all(): >n = new.store.Book() >n.id = o.id ># TODO: ># Removed fields: name ># Added fields: book_name >n.save() >... > > if __name__ == '__main__': >migrate() > > = > > Then, to migrate the DB, I will edit the file, search for "TODO" > comments, and replace the comment in the file with "n.book_name = > o.name". I will create a new empty DB with the name "store_4". > > Then I will run "./migrate.py", and the database will be migrated. I > will run the new version of my site, and if everything works I can > delete the DB "store_3". > > Another thing: the migrate command will work also with only one given > site directory. In that case the old db definition and the new db > definition will be identical, and the user will be able to edit them > manually in the generated file. This feature will also help in > migrating between different types of databases - in that case, the > user will only change the settings on the top of migrate.py. > > To summarize: the programmer won't have to learn commands which add > new fields, delete fields, rename fields, modify fields, etc. The > programmer will just have to slightly modify a generated file, to > create the new version of the DB as he likes. > > I hope I managed to explain my idea. What do you think about it? > > Have a good day, > Noam > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: An idea on DB migration
Noam wrote: > For example, say that I have a directory with the old site (and old > models) and a directory with the new site. The only difference between > them is that I renamed the field store.Book.name to > store.Book.book_name. I will run "django-admin.py migrate oldsitedir > newsitedir". This will create a file which will look like this: > [snip] > > Then, to migrate the DB, I will edit the file, search for "TODO" > comments, and replace the comment in the file with "n.book_name = > o.name". I will create a new empty DB with the name "store_4". I don't like the "edit the file" part. I can think of 2 alternatives: 1. add an attribute to one/both of the models. something like class Book(models.Model): name = models.CharField(blank=True, "migrate_to"="book_name" ) 2. a 3rd file that defines those kind of translations, which might be about the same by just subclassing the existing models. class Book(models.Book): book_name = Book.name() (I am totally winging it here, so forgiveme if my brain storm has some turbulence.) Also, along the lines of slow... you are going to love this one: how about making it use the web server as the source which would allow migrating between 2 servers where http is the only exposed protocol? it would also take care of migrating between systems where I don't have both db stacks installed on one box - consider win/mssql -> linux/SqlLite. yeah yeah, django/mssql doesn't exist yet... but it makes a good example :) Carl K --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
An idea on DB migration
Hello, I have an idea on how DB migration can work in Django. It's very simple, it will allow the programmer to do anything he wants in the migration, and he will have to learn almost nothing in order to migrate the DB. The drawback is that it's not very efficient, which means that it will probably work only for small to medium databases - but I think that most Django sites are like this. If you have a bigger site, you will have to use SQL. My idea is this: create a tool which will create a script which will migrate the DB. The user will then edit the script as he likes, and run it to create a new DB with the migrated data. If the result is not what he wanted, he will edit the script again and run it again. For example, say that I have a directory with the old site (and old models) and a directory with the new site. The only difference between them is that I renamed the field store.Book.name to store.Book.book_name. I will run "django-admin.py migrate oldsitedir newsitedir". This will create a file which will look like this: = #!/usr/bin/env python from django.db import ... class old_settings: DATABASE_ENGINE = 'mysql' DATABASE_NAME = 'store_3' ... class new_settings: DATABASE_ENGINE = 'mysql' DATABASE_NAME = 'store_4' ... old_connection = make_connection_from_settings(old_settings) new_connection = make_connection_from_settings(new_settings) # Old models class old: class auth: class User(models.Model): _connection = old_connection # or whatever will be needed username = models.CharField(unique=True, maxlength=30) ... class store: class Book(models.Model): _connection = old_connection name = models.CharField() ... # New models class new: ... class store: class Book(models.Model): _connection = old_connection book_name = models.CharField() ... # Migration function def migrate(): drop_tables_in_new_db_and_sync_it() # Migrate auth.User for o in old.auth.User.objects.all(): n = new.auth.User() n.id = o.id n.username = o.username ... n.save() # Migrate store.Book for o in old.store.Book.objects.all(): n = new.store.Book() n.id = o.id # TODO: # Removed fields: name # Added fields: book_name n.save() ... if __name__ == '__main__': migrate() = Then, to migrate the DB, I will edit the file, search for "TODO" comments, and replace the comment in the file with "n.book_name = o.name". I will create a new empty DB with the name "store_4". Then I will run "./migrate.py", and the database will be migrated. I will run the new version of my site, and if everything works I can delete the DB "store_3". Another thing: the migrate command will work also with only one given site directory. In that case the old db definition and the new db definition will be identical, and the user will be able to edit them manually in the generated file. This feature will also help in migrating between different types of databases - in that case, the user will only change the settings on the top of migrate.py. To summarize: the programmer won't have to learn commands which add new fields, delete fields, rename fields, modify fields, etc. The programmer will just have to slightly modify a generated file, to create the new version of the DB as he likes. I hope I managed to explain my idea. What do you think about it? Have a good day, Noam --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
