Hi Jeremy,
I see your point, and yes, migrations can be run out of sequence -
although in practice I've found that it is rarely an issue and if it is,
it can be resolved fairly simply. Let's take a couple of examples. I'm
not trying to be condescending, just thinking it through outloud ;)
1) Mike wants to add " favourite_colour" attribute to a Profile object,
and Jeremy wants to add "shoe_size". There are two migrations that CAN
be applied out of order - they do not depend on each other at all. When
Mike updates, he will get the code and migration for "shoe_size". When
Jeremy updates, he will get the code and migration for
"favourite_colour". No problems there.
2) Jeremy wants to abstract some information (let's say shoe_size) from
the Profile object into a separate 'Clothing' class and add a reference
in Profile to the new class. At the same time, Mike alters the shoe_size
attribute to be a different type. Now we get a situation where the
migrations depend on each other but are not available to both developers.
So, if we used a migration number instead of just the name, the commit
would flag the conflict as the two migration files would conflict.
Great! We've seen the problem immediately!
If we just used the name, our pre-commit update and test would flag the
problem.
Lets say that Jeremy has committed his code first.
Before he commits, Mike updates runs his tests. As part of the test run,
the database is dropped and rebuilt - using the newly merged migrations.
Depending on how SVN merged the migration list, one of them is going to
fail. Mike can see the problem and go and fix it.
3) Five developers are working on a project. Each of them produce a
migration. None of the migrations depend on each other.
If we just use names, none of the migrations will conflict and none of
them depend on each other so all will run without problems.
If we just use numbers, 4 developers will see a conflict after the first
developer commits, even though their code is not broken. They've just
taken the next migration number without being aware that other people
needed to commit migrations too. So they increment their migration
number and commit. Now the first one to commit will cause a conflict for
the rest, and so on until they all contact each other and agree what
their migration numbers will be.
On the other hand, it's a much more efficient way to see if a schema has
all migrations run on it in the right order if we use a number. Although
with a descriptive name, it's much easier to track down what each one is
doing and find an offending migration if there is a problem with the
code that depends on it.
Perhaps a combination of both would be good?
By the way, my dbmigration should have no problems with older versions
of django, I just haven't tested it with them. The patch for
management.py would need changing though.
Cheers,
Mike
Jeremy Dunck wrote:
> On 4/20/07, Mike H <[EMAIL PROTECTED]> wrote:
>
>> It seems to be working well for the developers here, so here's hoping
>> it's useful for some other people too :)
>>
>
> This is interesting-- migrating django apps using django. Are there
> cases where Django or the DB could become so broken that the migrator
> couldn't run? I wasn't sure, so I took the coward's way.
>
> I'm attaching my db_upgrade.py for comparison. (I've attached it
> on-list a time or two before, but there are many migration threads.
> :)) It uses numbers instead of names for migrations, and has a custom
> db_version table (no django model for it). It supports only python
> upgrade steps, but those steps are handed a connection to work with if
> they want to use SQL directly. Finally, it runs against Ye Olde
> Django (0.91) and probably needs some tweaks to run against 0.96.
>
> If I'm reading your code right, the migrations could be applied out of
> order, depending on which ones you have already run. Since schema is
> stateful, I think strict order is better.
>
> Consider developers Mike and Jeremy, starting with the same DB. Mike
> creates a migration, "HalfPrice" and runs it locally. Jeremy creates
> a migration, "FullPrice" and runs it locally. Each then update their
> checkouts and run syncdb.
>
> The migrator conflict is update like this, say:
> migrations = [
> "HalfPrice",
> "FullPrice"
> ]
>
> The changes of FullPrice will be run for Mike (since that's not in the
> applied list in his DB), and the changes of HalfPrice will be run for
> Jeremy (for the same reason).
>
> Now, this probably isn't an issue outside of development, but it was
> this situation that led me to use migration numbers rather than names
> and have my migration script halt if the DB rev was at or higher than
> the highest migration step.
>
> Cheers,
> Jeremy
>
> >
>
> ------------------------------------------------------------------------
>
> #!/usr/bin/python
> import sys, os
> import psycopg
> from django.core.db import db
> from django.conf.settings import DATABASE_USER, DATABASE_NAME, DATABASE_HOST,
> DATABASE_PORT, DATABASE_PASSWORD, DEBUG, TIME_ZONE, UPGRADE_STEPS_PATH, DEBUG
>
> def usage():
> sys.stderr.write("%s [--nobackup] \n Expects DJANGO_SETTINGS_MODULE to be
> set.\n")
>
> if len(sys.argv) == 1:
> do_backup = True
> else:
> if sys.argv[1] == "--nobackup":
> do_backup = False
> else:
> usage()
> sys.exit()
>
>
> try:
> os.environ['DJANGO_SETTINGS_MODULE']
> except KeyError:
> usage()
> sys.exit()
>
> def get_conn_string():
> if DATABASE_NAME == '':
> from django.core.exceptions import ImproperlyConfigured
> raise ImproperlyConfigured, "You need to specify DATABASE_NAME in
> your Django settings file."
> conn_string = "dbname=%s" % DATABASE_NAME
> if DATABASE_USER:
> conn_string = "user=%s %s" % (DATABASE_USER, conn_string)
> if DATABASE_PASSWORD:
> conn_string += " password='%s'" % DATABASE_PASSWORD
> if DATABASE_HOST:
> conn_string += " host=%s" % DATABASE_HOST
> if DATABASE_PORT:
> conn_string += " port=%s" % DATABASE_PORT
> return conn_string
>
> def get_db():
> log("Opening DB for upgrade")
> cur = db.cursor()
> conn = db.connection
> return conn, cur
>
> def get_current_version(cur):
> """
> Assumes:
> create table db_version (branch varchar(50), version integer);
> insert into db_version values ('trunk', 1);
>
> """
> try:
> cur.execute('select branch, version from db_version')
> except psycopg.ProgrammingError, e:
> logerr("Couldn't find db_version \n====%s\n====:\n perhaps you
> forgot to restore a DB dump or should create the table with ('trunk', 1)?" %
> e)
> return cur.fetchone()
>
> def log(s):
> sys.stderr.write('info: %s\n' % s)
>
> def logerr(s):
> sys.stdout.write('error: %s\n' % s)
> sys.stdout.write('error: You probably want to restore the
> DB!\nQuitting.\n')
> sys.exit()
>
> def backup_db(init, final) :
> import subprocess
> from time import gmtime, strftime
> dumpfile = "db_preupgrade_%s_from_%d_to_%d_at_%s_dump" % \
> (DATABASE_NAME, init, final, strftime("%Y%m%dT%H%M%S",
> gmtime()))
> log("If you're asked for the DB password, it's: %s" % DATABASE_PASSWORD)
> #FIXME: use subprocess.Popen and tie pg_dump output to gzip input.
> ret = subprocess.call(["pg_dump", "-f", dumpfile, "-U", DATABASE_USER,
> "-h", DATABASE_HOST, DATABASE_NAME])
>
> if ret != 0:
> raise RuntimeError, "Failed to create pg_dump %d" % ret
> return dumpfile
>
> if __name__ == '__main__':
> sys.path.insert(0, UPGRADE_STEPS_PATH)
> import upgrade_steps
>
> conn, cur = get_db()
>
> label, initial_version = get_current_version(cur)
>
> log("Starting at version %d of db branch %s." % (initial_version, label))
>
> steps = []
>
> version = initial_version
> while True:
> try:
> version += 1
> func_name = '%s%d' % (label, version)
> steps.append(getattr(upgrade_steps, func_name))
> if not callable(steps[-1]):
> logerr("%s is not callable, quitting." % func_name)
> sys.exit()
> except SystemExit:
> raise
> except:
> break
> final_version = version-1
>
>
> if initial_version == final_version:
> log("No DB upgrade to do. Already at %s version %d." % (label,
> initial_version))
> log("Quitting.")
> sys.exit()
>
> log("Upgrading from %d to %d on %s" % (initial_version, final_version,
> label))
>
> if do_backup:
> try:
> log("Backing up existing db.")
> dumpfile = backup_db(initial_version, final_version)
> except RuntimeError, e:
> logerr(e)
>
> log("Backed up to %s" % dumpfile)
> else:
> log("skipping backup")
>
> version = initial_version
> for step in steps:
> try:
> version += 1
> # FIXME: a successful Django ORM .save() causes a commit in the
> transaction
> # thereby committing something we really don't want to
> step(conn, cur)
> cur.execute("update db_version set version = %d", [version])
> if DEBUG: #commit changes from each step to aid in debugging.
> conn.commit()
> log("Done with version %s" % version)
> except psycopg.ProgrammingError, e:
> logerr("SQL failed in db upgrade to version %d:%s" % (version, e))
> except Exception, e:
> logerr("Unknown error updating on to version %d: %s" % (version,
> e))
> conn.commit() #final commit with all steps completed.
>
> log("Successfully completed upgrade from %d to %d" % (initial_version,
> final_version))
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
