It's about integration tests that aren't db-specific, so, just DATABASE/connection should be fixed ;)
On Wed, Feb 5, 2014 at 4:33 PM, Alexei Kornienko <alexei.kornie...@gmail.com > wrote: > Hi > > > I'm currently working on moving on the MySQL for savanna-ci > > We are working on same task in ceilometer so maybe you could use some of > our patches as reference: > > https://review.openstack.org/#/c/59489/ > https://review.openstack.org/#/c/63049/ > > Regards, > Alexei > > > On 02/05/2014 02:06 PM, Sergey Kolekonov wrote: > > I'm currently working on moving on the MySQL for savanna-ci > > > On Wed, Feb 5, 2014 at 3:53 PM, Sergey Lukjanov <slukja...@mirantis.com>wrote: > >> Agreed, let's move on to the MySQL for savanna-ci to run integration >> tests against production-like DB. >> >> >> On Wed, Feb 5, 2014 at 1:54 AM, Andrew Lazarev <alaza...@mirantis.com>wrote: >> >>> Since sqlite is not in the list of "databases that would be used in >>> production", CI should use other DB for testing. >>> >>> Andrew. >>> >>> >>> On Tue, Feb 4, 2014 at 1:13 PM, Alexander Ignatov <aigna...@mirantis.com >>> > wrote: >>> >>>> Indeed. We should create a bug around that and move our savanna-ci to >>>> mysql. >>>> >>>> Regards, >>>> Alexander Ignatov >>>> >>>> >>>> >>>> On 05 Feb 2014, at 01:01, Trevor McKay <tmc...@redhat.com> wrote: >>>> >>>> > This brings up an interesting problem: >>>> > >>>> > In https://review.openstack.org/#/c/70420/ I've added a migration >>>> that >>>> > uses a drop column for an upgrade. >>>> > >>>> > But savann-ci is apparently using a sqlite database to run. So it >>>> can't >>>> > possibly pass. >>>> > >>>> > What do we do here? Shift savanna-ci tests to non sqlite? >>>> > >>>> > Trevor >>>> > >>>> > On Sat, 2014-02-01 at 18:17 +0200, Roman Podoliaka wrote: >>>> >> Hi all, >>>> >> >>>> >> My two cents. >>>> >> >>>> >>> 2) Extend alembic so that op.drop_column() does the right thing >>>> >> We could, but should we? >>>> >> >>>> >> The only reason alembic doesn't support these operations for SQLite >>>> >> yet is that SQLite lacks proper support of ALTER statement. For >>>> >> sqlalchemy-migrate we've been providing a work-around in the form of >>>> >> recreating of a table and copying of all existing rows (which is a >>>> >> hack, really). >>>> >> >>>> >> But to be able to recreate a table, we first must have its >>>> definition. >>>> >> And we've been relying on SQLAlchemy schema reflection facilities for >>>> >> that. Unfortunately, this approach has a few drawbacks: >>>> >> >>>> >> 1) SQLAlchemy versions prior to 0.8.4 don't support reflection of >>>> >> unique constraints, which means the recreated table won't have them; >>>> >> >>>> >> 2) special care must be taken in 'edge' cases (e.g. when you want to >>>> >> drop a BOOLEAN column, you must also drop the corresponding CHECK >>>> (col >>>> >> in (0, 1)) constraint manually, or SQLite will raise an error when >>>> the >>>> >> table is recreated without the column being dropped) >>>> >> >>>> >> 3) special care must be taken for 'custom' type columns (it's got >>>> >> better with SQLAlchemy 0.8.x, but e.g. in 0.7.x we had to override >>>> >> definitions of reflected BIGINT columns manually for each >>>> >> column.drop() call) >>>> >> >>>> >> 4) schema reflection can't be performed when alembic migrations are >>>> >> run in 'offline' mode (without connecting to a DB) >>>> >> ... >>>> >> (probably something else I've forgotten) >>>> >> >>>> >> So it's totally doable, but, IMO, there is no real benefit in >>>> >> supporting running of schema migrations for SQLite. >>>> >> >>>> >>> ...attempts to drop schema generation based on models in favor of >>>> migrations >>>> >> >>>> >> As long as we have a test that checks that the DB schema obtained by >>>> >> running of migration scripts is equal to the one obtained by calling >>>> >> metadata.create_all(), it's perfectly OK to use model definitions to >>>> >> generate the initial DB schema for running of unit-tests as well as >>>> >> for new installations of OpenStack (and this is actually faster than >>>> >> running of migration scripts). ... and if we have strong objections >>>> >> against doing metadata.create_all(), we can always use migration >>>> >> scripts for both new installations and upgrades for all DB backends, >>>> >> except SQLite. >>>> >> >>>> >> Thanks, >>>> >> Roman >>>> >> >>>> >> On Sat, Feb 1, 2014 at 12:09 PM, Eugene Nikanorov >>>> >> <enikano...@mirantis.com> wrote: >>>> >>> Boris, >>>> >>> >>>> >>> Sorry for the offtopic. >>>> >>> Is switching to model-based schema generation is something decided? >>>> I see >>>> >>> the opposite: attempts to drop schema generation based on models in >>>> favor of >>>> >>> migrations. >>>> >>> Can you point to some discussion threads? >>>> >>> >>>> >>> Thanks, >>>> >>> Eugene. >>>> >>> >>>> >>> >>>> >>> >>>> >>> On Sat, Feb 1, 2014 at 2:19 AM, Boris Pavlovic < >>>> bpavlo...@mirantis.com> >>>> >>> wrote: >>>> >>>> >>>> >>>> Jay, >>>> >>>> >>>> >>>> Yep we shouldn't use migrations for sqlite at all. >>>> >>>> >>>> >>>> The major issue that we have now is that we are not able to ensure >>>> that DB >>>> >>>> schema created by migration & models are same (actually they are >>>> not same). >>>> >>>> >>>> >>>> So before dropping support of migrations for sqlite & switching to >>>> model >>>> >>>> based created schema we should add tests that will check that >>>> model & >>>> >>>> migrations are synced. >>>> >>>> (we are working on this) >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Best regards, >>>> >>>> Boris Pavlovic >>>> >>>> >>>> >>>> >>>> >>>> On Fri, Jan 31, 2014 at 7:31 PM, Andrew Lazarev < >>>> alaza...@mirantis.com> >>>> >>>> wrote: >>>> >>>>> >>>> >>>>> Trevor, >>>> >>>>> >>>> >>>>> Such check could be useful on alembic side too. Good opportunity >>>> for >>>> >>>>> contribution. >>>> >>>>> >>>> >>>>> Andrew. >>>> >>>>> >>>> >>>>> >>>> >>>>> On Fri, Jan 31, 2014 at 6:12 AM, Trevor McKay <tmc...@redhat.com> >>>> wrote: >>>> >>>>>> >>>> >>>>>> Okay, I can accept that migrations shouldn't be supported on >>>> sqlite. >>>> >>>>>> >>>> >>>>>> However, if that's the case then we need to fix up >>>> savanna-db-manage so >>>> >>>>>> that it checks the db connection info and throws a polite error >>>> to the >>>> >>>>>> user for attempted migrations on unsupported platforms. For >>>> example: >>>> >>>>>> >>>> >>>>>> "Database migrations are not supported for sqlite" >>>> >>>>>> >>>> >>>>>> Because, as a developer, when I see a sql error trace as the >>>> result of >>>> >>>>>> an operation I assume it's broken :) >>>> >>>>>> >>>> >>>>>> Best, >>>> >>>>>> >>>> >>>>>> Trevor >>>> >>>>>> >>>> >>>>>> On Thu, 2014-01-30 at 15:04 -0500, Jay Pipes wrote: >>>> >>>>>>> On Thu, 2014-01-30 at 14:51 -0500, Trevor McKay wrote: >>>> >>>>>>>> I was playing with alembic migration and discovered that >>>> >>>>>>>> op.drop_column() doesn't work with sqlite. This is because >>>> sqlite >>>> >>>>>>>> doesn't support dropping a column (broken imho, but that's >>>> another >>>> >>>>>>>> discussion). Sqlite throws a syntax error. >>>> >>>>>>>> >>>> >>>>>>>> To make this work with sqlite, you have to copy the table to a >>>> >>>>>>>> temporary >>>> >>>>>>>> excluding the column(s) you don't want and delete the old one, >>>> >>>>>>>> followed >>>> >>>>>>>> by a rename of the new table. >>>> >>>>>>>> >>>> >>>>>>>> The existing 002 migration uses op.drop_column(), so I'm >>>> assuming >>>> >>>>>>>> it's >>>> >>>>>>>> broken, too (I need to check what the migration test is >>>> doing). I >>>> >>>>>>>> was >>>> >>>>>>>> working on an 003. >>>> >>>>>>>> >>>> >>>>>>>> How do we want to handle this? Three good options I can think >>>> of: >>>> >>>>>>>> >>>> >>>>>>>> 1) don't support migrations for sqlite (I think "no", but >>>> maybe) >>>> >>>>>>>> >>>> >>>>>>>> 2) Extend alembic so that op.drop_column() does the right thing >>>> >>>>>>>> (more >>>> >>>>>>>> open-source contributions for us, yay :) ) >>>> >>>>>>>> >>>> >>>>>>>> 3) Add our own wrapper in savanna so that we have a >>>> drop_column() >>>> >>>>>>>> method >>>> >>>>>>>> that wraps copy/rename. >>>> >>>>>>>> >>>> >>>>>>>> Ideas, comments? >>>> >>>>>>> >>>> >>>>>>> Migrations should really not be run against SQLite at all -- >>>> only on >>>> >>>>>>> the >>>> >>>>>>> databases that would be used in production. I believe the >>>> general >>>> >>>>>>> direction of the contributor community is to be consistent >>>> around >>>> >>>>>>> testing of migrations and to not run migrations at all in unit >>>> tests >>>> >>>>>>> (which use SQLite). >>>> >>>>>>> >>>> >>>>>>> Boris (cc'd) may have some more to say on this topic. >>>> >>>>>>> >>>> >>>>>>> Best, >>>> >>>>>>> -jay >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> _______________________________________________ >>>> >>>>>>> OpenStack-dev mailing list >>>> >>>>>>> OpenStack-dev@lists.openstack.org >>>> >>>>>>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> _______________________________________________ >>>> >>>>>> OpenStack-dev mailing list >>>> >>>>>> OpenStack-dev@lists.openstack.org >>>> >>>>>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>>> >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> >>>> OpenStack-dev mailing list >>>> >>>> OpenStack-dev@lists.openstack.org >>>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>>> >>> >>>> >>> >>>> >>> _______________________________________________ >>>> >>> OpenStack-dev mailing list >>>> >>> OpenStack-dev@lists.openstack.org >>>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>>> >> >>>> >> _______________________________________________ >>>> >> OpenStack-dev mailing list >>>> >> OpenStack-dev@lists.openstack.org >>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> > >>>> > >>>> > >>>> > _______________________________________________ >>>> > OpenStack-dev mailing list >>>> > OpenStack-dev@lists.openstack.org >>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>>> _______________________________________________ >>>> OpenStack-dev mailing list >>>> OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> -- >> Sincerely yours, >> Sergey Lukjanov >> Savanna Technical Lead >> Mirantis Inc. >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > > _______________________________________________ > OpenStack-dev mailing > listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Sincerely yours, Sergey Lukjanov Savanna Technical Lead Mirantis Inc.
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev