Hi, everyone, <tl;dr>
in Fuel project we run two HA deployment tests for every commit in openstack/fuel-library repository. Currently these tests are run by Fuel CI - the standalone Third-Party CI system, which can vote on openstack/fuel* projects. But we'd like it to be the part of the gate. As these tests require several vms to be available for the test run, we'd like to use nodepool multinode support for that. </tl;dr> As I don't have deep understanding about multinode setup at the moment, and I haven't found enough info in docs or specs, I think the better way to start this discussion is to explain the workflow used in Fuel HA tests. Then we can consider which of the steps are already there, or can be implemented, and which items are potential blockers. Any feedback is welcome. Generic deployment test ======================= To run deployment for Fuel you need: - ISO image for Fuel node [1] - fuel-devops [2] - the cli tool which manages virtual machines and stores state (vm names, network interfaces..) in a PostgreSQL database - fuel-qa [3] - test framework based on proboscis Basic setup is described in [4]. And the test flow is as follows: - with fuel-devops tool: create several vm's connected via internal network - a so called 'devops environment' - with fuel-qa framework: 1. install Fuel node on first vm using the ISO image provided by the local path on the host server 2. bootstrap other vms with basic OS image provided on Fuel ISO 3. configure Fuel environment via API according to certain scenario 4. run deployment Test scenarios are described in fuel-qa documentation, see for example [5]. Deployment test on CI ===================== fuel-library code is essentially a set of puppet manifests which are used to deploy the enviroment configuration defined via Fuel interface. These manifests are delivered to Fuel node as RPM package [6]. To save time and resources on CI we don't recreate environment from scratch for every tests but regularly take a "stable enough" ISO, upload it to Jenkins slaves, create base environment (steps 1. and 2.) and snapshot all its vms. Then, on every commit we 1) rebuild a fuel-library package in a CentOS-based docker container 2) revert devops environment from snapshot 3) upload and install package on Fuel node 4) run the deployment test scenario (steps 3. and 4.) You can refer to detailed logs in [7] How this can be addressed in nodepool ===================================== The nodepool driver approach ---------------------------- fuel-devops is essentially a wrapper and vm's manager, and it was originally planned as a tool which can use multiple backends, taking libvirt as a default one. There is an still-on-discussion task to implement the 'bare-metal driver' for fuel-devops, which would make it possible to use vm's from different servers for one particular test run. We can consider implementing nodepool as a driver, so it provides vm's, which then are wrapped by fuel-devops and are sent further to fuel-qa framework. Then to run the test we would need a 'manager vm' where fuel-devops code is executed, and several empty nodes from nodepool. We'd register those empty nodes in fuel-devops database and run the test as usual. No fuel-devops approach ----------------------- Direct approach would be to use the nodepool's service for pre-built images. Given by Fuel ISO image, we regularly generate one vm with deployed Fuel node (step 1.), and one with the basic node deployed with bootstrap image (step 2.). This images are stored in Glance or another storage as usual. Then for each fuel-library test we request 1 Fuel node and several basic nodes, and then operate on them directly without wrappers. For this scenario to work we need to change a lot in fuel-qa code. But this approach seems to be aligned with the initiative [8], which is currently in development: if we manage to describe devops environments in YAML files, we'd probably be able to map these descriptions to the multinode configurations, and then support them in nodepool. Side Question ============= Can we build package from a change request so that package is then used in test? Are there any best practices? P.S. All code used for current implementation of deployment tests in Fuel CI is open, including puppet manifests for Jenkins master and slaves and Jenkins Job Builder configs for actual jobs. See links below. [1] https://ci.fuel-infra.org/view/ISO/ - nightly ISO builds [2] https://github.com/openstack/fuel-devops [3] https://github.com/openstack/fuel-qa [4] https://docs.fuel-infra.org/fuel-dev/devops.html - test environment setup [5] https://docs.fuel-infra.org/fuel-qa/base_tests.html#module-fuelweb_test.tests.test_neutron - test scenario [6] https://github.com/openstack/fuel-library/blob/master/specs/fuel-library8.0.spec [7] https://ci.fuel-infra.org/job/master.fuel-library.pkgs.ubuntu.neutron_vlan_ha/2699/console - test run example [8] https://review.openstack.org/#/c/240170/ - YAML-based templates for devops environments Fuel CI links: [A] https://github.com/fuel-infra/jenkins-jobs/tree/master/servers/fuel-ci - JJB configs [B] https://github.com/fuel-infra/puppet-manifests/tree/master/modules/jenkins/manifests - Puppet manifests [C] https://ci.fuel-infra.org - Jenkins instance -- Aleksandra Fedorova Fuel CI Team Lead bookwar __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev