Hi Monty, >>> I think if you're working on a non-alembic plan and boris is working on >>> an alembic plan, then something is going to be unhappy in the >>> not-too-distant future. Can we get alignment on this?
As I said before, we are preparing our DB code to move from sqlalchemy-migrate to something another. There will be a tons of work before we will be able to rewrite or migration scripts to alembic or something else. And we are not sure that we would like to use alembic=) Best regards, Boris Pavlovic On Wed, Jul 3, 2013 at 9:30 PM, Monty Taylor <mord...@inaugust.com> wrote: > > > On 07/02/2013 10:50 AM, Boris Pavlovic wrote: > > > ############################################################################### > > Goal > > > ############################################################################### > > > > We should fix work with DB, unify it in all projects and use oslo code > > for all common things. > > Just wanted to say a quick word that isn't about migrations... > > Thank you. This is all great, and I'm thrilled someone is taking on the > task of fixing what is probably one of OpenStack's biggest nightmares. > > > In more words: > > > > DB API > > > > *) Fully cover by tests. > > > > *) Run tests against all backends (now they are runed only against > > sqlite). > > > > *) Unique constraints (instead of select + insert) > > a) Provide unique constraints. > > b) Add missing unique constraints. > > > > *) DB Archiving > > a) create shadow tables > > b) add tests that checks that shadow and main table are synced. > > c) add code that work with shadow tables. > > > > *) DB API performance optimization > > a) Remove unused joins.. > > b) 1 query instead of N (where it is possible). > > c) Add methods that could improve performance. > > d) Drop unused methods. > > > > *) DB reconnect > > a) Don’t break huge task if we lost connection for a moment.. just > > retry DB query. > > > > *) DB Session cleanup > > a) do not use session parameter in public DB API methods. > > b) fix places where we are doing N queries in N transactions instead > > of 1. > > c) get only data that is used (e.g. len(query.all()) => > query.count()). > > > > ---- > > > > DB Migrations > > > > *) Test DB Migrations against all backends and real data. > > > > *) Fix: DB schemas after Migrations should be same in different > backends > > > > *) Fix: hidden bugs, that are caused by wrong migrations: > > a) fix indexes. e.g. 152 migration in Nova drop all Indexes that > > has deleted column > > b) fix wrong types > > c) drop unused tables > > > > *) Switch from sqlalchemy-migrate to something that is not death (e.g. > > alembic). > > > > ---- > > > > DB Models > > > > *) Fix: Schema that is created by Models should be the same as after > > migrations. > > > > *) Fix: Unit tests should be runed on DB that was created by Models > > not migrations. > > > > *) Add test that checks that Models are synced with migrations. > > > > ---- > > > > Oslo Code > > > > *) Base Sqlalchemy Models. > > > > *) Work around engine and session. > > > > *) SqlAlchemy Utils - that helps us with migrations and tests. > > > > *) Test migrations Base. > > > > *) Use common test wrapper that allows us to run tests on different > > backends. > > > > > > > ############################################################################### > > Implementation > > > ############################################################################### > > > > This is really really huge task. And we are almost done with Nova=). > > > > In OpenStack for such work there is only one approach (“baby steps” > > development deriven). So we are making tons of patches that could be > > easy reviewed. But there is also minuses in such approach. It is pretty > > hard to track work on high level. And sometimes there are misunderstand. > > > > For example with oslo code. In few words at this moment we would like > > to add (for some time) in oslo monkey patching for sqlalchemy-migrate. > > And I got reasonable question from Doug Hellmann. Why? I answer because > > of our “baby steps”. But if you don’t have a list of baby steps it is > > pretty hard to understand why our baby steps need this thing. And why we > > don’t switch to alembic firstly. So I would like to describe our Road > > Map and write list of "baby steps". > > > > > > ------- > > > > OSLO > > > > *) (Merged) Base code for Models and sqlalchemy engine (session) > > > > *) (On review) Sqlalchemy utils that are used to: > > 1. Fix bugs in sqlalchemy-migrate > > 2. Base code for migrations that provides Unique Constraints. > > 3. Utils for db.archiving helps us to create and check shadow > tables. > > > > *) (On review) Testtools wrapper > > We should have only one testtool wrapper in all projects. And > > this is the one of base steps in task of running tests against all > backends. > > > > *) (On review) Test migrations base > > Base classes that provides us to test our migrations against all > > backends on real data > > > > *) (On review, not finished yet) DB Reconnect. > > > > *) (Not finished) Test that checks that schemas and models are synced > > > > ------- > > > > ${PROJECT_NAME} > > > > > > In different projects we could work absolutely simultaneously, and first > > candidates are Glance and Cinder. But inside project we could also work > > simultaneously. Here is the workflow: > > > > > > 1) (SYNC) Use base code for Models and sqlalchemy engines (from oslo) > > > > 2) (SYNC) Use test migrations base (from oslo) > > > > 3) (SYNC) Use SqlAlchemy utils (from oslo) > > > > 4) (1 patch) Switch to OSLO DB code > > > > 5) (1 patch) Remove ported test migrations > > > > 6) (1 Migration) Provide unique constraints (change type of “deleted” > > column) > > > > 7) (1 Migration) Add shadow tables > > a) Create shadow tables > > b) Add test that checks that they are synced always > > > > 8) (N Migrations) UniqueConstraint/Session/Optimization workflow: > > a) (1 patch) Add/Improve/Refactor tests for part of api (that is > > connected with model) > > b) (1 patch) Fix session > > c) (1 patch) Optimize method > > d) if required (1 Migration) Add missing Unique Constraints. > > > > 9) (M Migrations) Sync Models with DB schemas (almost independent from > 8) > > a) (K pathces) Sync Models with MySQL Migrations > > (because they are different in different backneds) > > b) (Fix, L migrations) Fix migrations, that have bugs (e.g. 152 > > migraiton in Nova, drops all > > Indexes that have ‘deleted’ column). > > c) Add test for MySql > > d) (Fix, 2 migrations) Sync other backends schemas with MySql schema > > e) Add test for SQLite and Psql > > f) Rethink our models: > > *) Improve Indexes > > *) Use Enum where it is required > > *) Fix bugs if there are. > > *) Add missing default values in schema. > > > > 10) Test should be runed on schema that is created from Models, not > > migrations (depend on 9) > > > > 11) Use alembic (or something else) instead of sqlalchemy-migrate (after > > 9.e. it will be safe > > a) make test migrations independent from sqlalchemy-migrate > > b) (If alembic) Provide work with sqlite. > > c) Replace all migrations > > > > Main part of work is in 7, 8 and 10 points. > > > > > > > ############################################################################### > > Answer for Doug's question > > > ############################################################################### > > > > Question: > > Why we should put in oslo slqlalchemy-migrate monkey patches, when we > > are planing to switch to alembic? > > > > Answer: > > If we don’t put in oslo sqlalchemy-migrate monkey patches. We won't > > be able to work on 7 point at all until 8 and 10 points will be > > implemented in every project. Also work around 8 point is not finished, > > so we are not able to implement 10 points in any of project. So this > > blocks almost all work in all projects. I think that these 100-200 lines > > of code are not so big price for saving few cycles of time. > > > > > > > > > ############################################################################### > > Current progress > > > ############################################################################### > > > > > > Nova > > > > *) 1,4,6,7 - fully finished > > > > *) 2,3,5 - waiting for oslo, but pretty simple to sync it > > > > *) 8 - almost finished > > a) Few patched around tests (on review) > > b) 9 patches that add UniqueConstraints (on review) and few missing > > c) Few patches about session (on review) and few missing > > > > *) 9 - tons of patches that syncs models with MySql on review. Also we > > are working around tests that checks that schemas that are crated by > > models and migrations are same. So it will be a lot of work here. > > > > *) 10 - pretty simple (blocked by 9) > > > > *) 11 - tons of work, candidate for I cycle. > > > > *) Also here is work around run test on all backends but it is not > > finished yet. > > > > ------- > > > > Cinder > > > > *) 1,4 - finished > > > > *) Some work around 8 (sessions and tests) > > > > *) Other things are blocked by oslo > > > > ------- > > > > Glance > > > > *) 1 - finished > > > > *) 4 - working on it > > > > > > > > Best regards, > > Boris Pavlović > > > > (irc boris-42) > > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev