On Mon, Apr 14, 2014 at 10:54 AM, Salvatore Orlando <[email protected]> wrote: > Resending with [Neutron] tag. > > Salvatore > > > On 14 April 2014 16:00, Salvatore Orlando <[email protected]> wrote: >> >> This is a rather long post. However the gist of it is that Neutron >> migrations are failing to correctly perform database upgrades when service >> plugins are involved, and this probably means the conditional migration path >> we designed for the Grizzly release is proving not robust enough when >> dealing with service plugins. >> >> Corrective actions are therefore required; otherwise there is a risk the >> Grenade jobs for neutron can't be made voting again. >> This issue is already being tracked with bug 1307344 which was >> independently filed and amended by Sean to reflect the actual issue. >> >> Grenade is currently failing because of this issue. Indeed the 'metering >> plugin' is enabled in devstack for Icehouse but now Havana. >> A quick fix for grenade has already been rejected (backport metering >> support to stable/havana) another quick fix instead would be running grenade >> without metering at all. >> >> However, whether we put or not a patch to make grenade running, there is >> still an issue that needs to be solved. >> Now, if you're not yet bored you might try now reading the long version of >> this post... >> >> Problem statement: >> >> Enabling or disabling a service plugin in neutron configuration might be >> done at any point and can lead migrations to build a database which is out >> of sync with expectations >> >> Assume: >> >> Current migration X (Service plugin SP enabled here) >> Service plugin SP introduced at migration F1 < X >> No auto-generation of DB schemas >> The migration F1 will be skipped >> >> When SP is enabled, the relevant DB schema won't exist and the feature >> will not work. >> >> Other failure scenarios: >> >> Current migration X >> SP introduced at migration F1 < X (but not yet enabled) >> SP enabled at migration Y > F1 > X >> SP updated at migration F2 | X < F2 < Y. >> >> F2 will fail because schemas are not yet ready for feature, even with auto >> schema generation >> >> SP introduced at migration F0, and enabled at migration F0 >> SP disabled at migration X >> SP updated at migration F1, which is skipped because it's disabled >> SP enabled again at migration Y. Feature won't work as migration X < F1 < >> Y was never executed. >> >> Requirements for a possible solution: >> >> Should solve problem not only for greenfield deployments but also for >> existing ones >> Must not make migration generation more complex or introduce error-prone >> behaviours >> >> Possible solutions: >> >> 1) Specify that all migrations must run for every plugin (*) unless they >> are really introducing schemas which are specific to a particular technology >> (such as uuid mappings between neutron and backed) >> >> This approach will probably ensure all the important migrations are >> executed. >> However, it is not back portable, unless deployers decide to tear down and >> rebuild their databases from scratch, which is unlikely to happen. >> >> To this aim "recovery" scripts might be provided to sync up the schema >> state of specific service plugins with the current alembic migration >> registered in the neutron database; or appropriate migrations can be added >> in the path to fix database schemas. >> >> (*) Neutron does not address, and probably never will address, switching >> from plugin "a" to "b" for a given service (e.g.: core features). >> >> 2) Have plugin-specific migration paths. >> >> This is not supported by alembic. In this case probably the best solution >> would be to have completely different databases for the main plugin and >> service plugins, which is however not possible because of foreign key >> relationships among tables in different service plugins. >> >> Also, there are likely to be migrations which alter core and service >> schemas at the same time. Adapting these migrations might be a problem.Extra >> work will also be needed to make this solution work with existing >> deployments. >> >> 3) Make migrations smarter >> >> When migration X is executed, it checks whether any previous migration for >> the plugins it alters has been executed. If not it will execute the missing >> migrations as well. >> This *might* work; However, alembic currently offers no support for it, as >> it's not possible to perform any form of "introspection" in the migration >> themselves. >> >> The migration engine will also be augmented as the last executed migration >> for each plugin should be stored in the database. >> This might also prove to be error prone and lead to unexpected behaviours. >> For example: >> >> Until migration X deployment uses plugin A which provides services S1 and >> S2 >> At migration Y user still uses plugin for service S1, but switches to >> plugin B service S2 >> Migration Z>Y applies to plugin B. System detects no migration for plugin >> B have been executed so far. >> However, the migrations for the service S2, offered by B were already >> executed when running plugin A >> System tries to execute migrations for plugin B which will fail as they >> were actually already executed for plugin A >> >> The system could be made smarter by storing also a list of "known" >> migrations, including whether they were executed or skipped. >> >> Summarising, in my opinion the approach #2 is not even worth being >> considered. Between approach #1 and #3, I'd prefer the simplest one and vote >> for approach #1. >>
Thanks for sending out this summary email Salvatore. I agree with your summary and I think that option #1 is the way forward to fix this. The recovery scripts are a necessity here. I'd like to hear from some deployers who may be lurking on their thoughts on this approach as well. Thanks, Kyle >> >> Salvatore >> >> >> >> >> >> > > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
