On 04/19/2012 07:43 PM, Michael Bayer wrote:
If you've seen my recent talks you saw that I'm a little skeptical of what you're terming "non-monolithic" databases. Let's say this means, a database with a set of tables maintained by entirely different packages, but with the possibility of dependencies between those tables. If I understand correctly, if we were dealing with sets of tables that didn't have any dependency, you wouldn't need a distributed migration tool, each package would handle migrations for its own set of tables independently, is that right ?
That suggests that every package would have its own migration tool, which is not very practical from a sysadmin point of view. I am an upgrading an application I want to be able to run all necessary migrations for all components of an application in one run. I do not want to be required to figure out which packages an application was running and then migrate them all separately. So I definitely see a need for an upgrade framework that can deal with multiple packages.
I think what I need to see here are, what exactly are these packages, outside of the Django community, that actually create their own tables yet encourage dependencies between those tables and your app's own tables ? I know people are working on them since I see people asking questions about those use cases, but what are they ? What's the openid and user/groups package you're thinking of here ?
s4u.image is such an example: https://github.com/2style4you/s4u.image . That package implements an image store which supports on-demand scaling of images. Metadata is stored in a SQL database and commonly you add references to images to other tables. Every site we build uses s4u.image to manage image handling. This happens to be in-house developed by us, but for all intents and purposes it is a third-party package to our front-end developers.
In the development world I've always lived in, we just don't have third party libraries that bring in their own sub-schemas. Up until now the thinking has been, if it's significant enough that it is part of your datamodel, it's part of what you should own yourself, though certainly drawing upon past recipes.
I suspect a difference is that we are often building different sites that build on shared common functionality. Our main business is building sites that deal with online fashion, so everything we build has to deal with things like images and clothing articles. The code to handle those has been split out to separate packages (s4u.image is one of those) that define core datamodels and some logic, and our sites build on those. Sometimes we extend the base models, for example when for a particular site we need to track extra data for clothing, and sometimes we use the base models as-is and reference them directly via relationships and foreign keys. That results in an ecosystem of many different packages and sites that each have their own evolve in their own way and require their own migrations.
When we upgrade a site our process is pretty simple: upgrade version pins for buildout, rerun buildout, run upgrade-script, tell mod_wsgi to reload. The upgrade-script walks through all migrations from all packages a site uses so we have a single interface for administrators to upgrade everything. The upgrade framework itself is extremely minimal (see https://github.com/2style4you/s4u.upgrade ), but works well enough for us. Note that we deviate from stucco_evolution in three important ways: we do not use versioning but require upgrade steps to test if an upgrade is necessary, our upgrade framework is not tied to SQLAlchemy but has a more generic requirements-system so you can use it for other things (we use it for filesystem changes and SOLR configuration as well for example), and it does not support dependencies. Personally I consider the first two to be desirable qualities for an upgrade framework. Dependencies are something that we will probably need to add at some point.
Wichert. -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
