While we now have devstack external plugins, grenade (our upgrade testing framework) was really monolithic. It grew out of a last minute set of test scripts for Folsom that discovered a number of our database migrations didn't work with real data in them, and that nova compute had the annoying habit of killing off VMs when it went down. It has grown in scope since then, but very organically, and not always clearly at times.
The first step in making this plugable externally is separating out everything that's really global, vs., what's per service. That was mostly done last week (with one important part missing, resource survival testing). The top of the current unmerged stack is here - https://review.openstack.org/#/c/172648/ == New Structure == The crux of this is that all the project specific code now lives in: grenade/ projects/ 10_keystone/..... 20_ceilometer/.... 30_swift/.... .... The current (in flux) interface is as follows: * settings - similar to the devstack plugin, this is a place for initial setup. So far this has been useful to register things you'd like grenade to do for you. For instance > more projects/10_keystone/settings register_project_for_upgrade keystone register_db_to_save keystone Tells us we should register this directory for upgrade (it does a little magic when it does that). And to save off a database. * upgrade.sh - the service upgrade script, which is expected to upgrade & restart the service. It is basically what upgrade-$foo was previously. In the current patch stream ``upgrade.sh`` is also responsible for doing a service sanity check once done. The following functions are provided to help with that: - ensure_services_started - ensure_logs_exist The project also supports a local from-juno/ from-kilo/ within-juno/ directory structure just like we did before. It's just in the service directory. * shutdown.sh - the service down script, which is also responsible for doing a sanity check that the service is actually down. The following functions are provided to help with that: - ensure_services_stopped == Resource Survival == This is still in flight, and here is a preview of where this is headed this week. One of the important things that grenade does is ensure that resources (like functioning VMs) survive the upgrade unscathed. Mostly because, once upon a time they did not. This started as a simple shell script. That broke at some point and no one noticed (though, we didn't have regressions, so that's at least something). Last summer we rebuilt that tool as python using the Tempest clients (javelin2). As that was ending we realized that basically we'd just recreated ansible in the small (our yaml file and theirs are way too close to assume otherwise). This also meant we created a new coupling with Tempest, and a new global coupling of 1 tool that needed to understand all projects. So we created a new bottleneck. Grenade is going to get out of the business of dictating a tool. Instead it's going to dictate an interface: It will look something like this (exact names in flux, we'll see how the code evolves). resources.sh [create|verify_noapi|verify|destroy] - pre shutdown - create - make some stuff that we think might not survive upgrades (i.e. more than just db records) - verify - make sure that stuff is working - post shutdown - verify_noapi - make sure that stuff is working, with checks that work without any API services up. - post upgrade - verify - make sure stuff is still running - post grenade - destroy - delete everything so we don't leave crud everywhere The verify_noapi is currently a hole in our testing, and something Clark brought up in Darmstadt last summer. It's a good hole to fill. There will also be some convenience functions provided to store/fetch persistent data so that grenade can keep track of things like instance ids / ip addresses and such for resource scripts. == Upgrade Order == This remains one of the last sticking points. Today our upgrade.sh iterates every project in a specific order and does both the upgrade and restart at the same time. The good thing about this is it is simple, and more closely follows a 'rolling-ish' upgrade pattern. The problem is dependency management. Especially when we talk about libraries from one project injecting into others that aren't in requirements.txt. Like ceilometermiddleware, ironicclient. I'm starting to think we should upgrade / restart as separate steps, because it will largely get rid of the dependency ordering issues. But that's up for grabs. == External API == An external plugin definition, similar to the devstack one, is coming. But not util the rest of this settles out. My hope is it will exist by Vancouver so we can do an External Plugins in Devstack / Grenade Design Summit session there. Both as a forum to ask questions about the existing structure, as well as a discussion of what should move into external plugins for both projects. ... This was mostly an FYI for where we stand. Once the rest of the code lands, and we've crystalized our interfaces, there will be more information about how projects can plug in here and reuse this framework for their own upgrade testing. -Sean -- Sean Dague http://dague.net __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev