On Mon, May 08, 2017 at 02:45:08PM +0300, Marios Andreou wrote: > Hi folks, after some discussion locally with colleagues about improving > the upgrades experience, one of the items that came up was pre-upgrade and > update validations. I took an AI to look at the current status of > tripleo-validations [0] and posted a simple WIP [1] intended to be run > before an undercloud update/upgrade and which just checks service status. > It was pointed out by shardy that for such checks it is better to instead > continue to use the per-service  manifests where possible like [2] for > example where we check status before N..O major upgrade. There may still > be some undercloud specific validations that we can land into the > tripleo-validations repo (thinking about things like the neutron > networks/ports, validating the current nova nodes state etc?). > So do folks have any thoughts about this subject - for example the kinds > of things we should be checking - Steve said he had some reviews in > progress for collecting the overcloud ansible puppet/docker config into an > ansible playbook that the operator can invoke for upgrade of the 'manual' > nodes (for example compute in the N..O workflow) - the point being that we > can add more per-service ansible validation tasks into the service > manifests for execution when the play is run by the operator - but I'll > let Steve point at and talk about those.Â
Thanks for starting this thread Marios, sorry for the slow reply due to Summit etc. As we discussed, I think adding validations is great, but I'd prefer we kept any overcloud validations specific to services in t-h-t instead of trying to manage service specific things over multiple repos. This would also help with the idea of per-step validations I think, where e.g you could have a "is service active" test and run it after the step where we expect the service to start, a blueprint was raised a while back asking for exactly that: https://blueprints.launchpad.net/tripleo/+spec/step-by-step-validation One way we could achive this is to add ansible tasks that perform some validation after each step, where we combine the tasks for all services, similar to how we already do upgrade_tasks and host_prep_tasks: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/database/redis.yaml#L92 With the benefit of hindsight using ansible tags for upgrade_tasks wasn't the best approach, because you can't change the tags via SoftwareDeployment (e.g you need a SoftwareConfig per step), it's better if we either generate the list of tasks by merging maps e.g validation_tasks: step3: - sometask Or via ansible conditionals where we pass a step value in to each run of the tasks: validation_tasks: - sometask when: step == 3 The latter approach is probably my preference, because it'll require less complex merging in the heat layer. As you mentioned, I've been working on ways to make the deployment steps more ansible driven, so having these tasks integrated with the t-h-t model would be well aligned with that I think: https://review.openstack.org/#/c/454816/ https://review.openstack.org/#/c/462211/ Happy to discuss further when you're ready to start integrating some overcloud validations. Thanks! Steve __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev