Hi, Andrey I've checked this option - to use rally for configuring and running tempest test. Although it looks like great choice, unfortunately a few issues and bugs makes it not useful right now. For example it can not work with current public networks and can not create new ones, so that everything that is related to networking will fail. As I understand this bug remains already a long time unsolved: https://bugs.launchpad.net/rally/+bug/1550848 Also it doesn't have possibility to customize configuration options when running tempest configuration, like configure_tempest.py has - just count them in the command line. In rally you will need to generate tempest file and then manually to edit it for customize (for example tempest log path in DEFAULT section). Adding "interface" for tempest configuration will be great feature for rally IMHO. I think it's cool approach and we definitely should take it into account, but now it looks pretty raw and not stable enough to use it in gate jobs. Anyway, thank you for your pointing out to this great tool.
Thanks On Fri, Apr 8, 2016 at 2:33 PM, Andrey Kurilin <akuri...@mirantis.com> wrote: > Hi Sagi, > > > On Thu, Apr 7, 2016 at 5:56 PM, Sagi Shnaidman <sshna...@redhat.com> > wrote: > >> Hi, all >> >> I'd like to discuss the topic about how do we configure tempest in CI >> jobs for TripleO. >> I have currently two patches: >> support for tempest: https://review.openstack.org/#/c/295844/ >> actually run of tests: https://review.openstack.org/#/c/297038/ >> >> Right now there is no upstream tool to configure tempest, so everybody >> use their own tools. >> > > You are wrong. There is Rally in upstream:) > Basic and the most widely used Rally component is Task, which provides > benchmarking and testing tool. > But, also, Rally has Verification component(here > <https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/> > you can find is a bit outdated blog-post, but it can introduce Verification > component for you). > It can: > > 1. Configure Tempest based on public OpenStack API. > An example of config from our gates: > http://logs.openstack.org/58/285758/5/check/gate-rally-dsvm-verify-full/eabe2ff/rally-verify/5_verify_showconfig.txt.gz > . Empty options mean that rally will check these resources while running > tempest and create it if necessary) > > 2. Launch set of tests, tests which match regexp, list of tests. Also, it > supports x-fail mechanism from out of box. > An example of full run based on config file posted above - > http://logs.openstack.org/58/285758/5/check/gate-rally-dsvm-verify-full/eabe2ff/rally-verify/7_verify_results.html.gz > > 3. Compare results. > > http://logs.openstack.org/58/285758/5/check/gate-rally-dsvm-verify-light/d806b91/rally-verify/17_verify_compare_--uuid-1_9fe72ea8-bd5c-45eb-9a37-5e674ea5e5d4_--uuid-2_315843d4-40b8-46f2-aa69-fb3d5d463379.html.gz > It is not so good-looking as other rally reports, but we will fix it > someday:) > > Summarize: > - Rally is an upstream tool, which was accepted to BigTent. > - One instance of Rally can manage and run tempest for different number of > clouds > - Rally Verification component is tested in gates for every new patch. > Also it supports different APIs of services. > - You can install, configure, launch, store results, display results in > different formats. > > Btw, we are planning to refactor verification component(there is an spec > on review with several +2), so you will be able to launch whatever you want > subunit-based tools via Rally and simplify usage of it. > > However it's planned and David Mellado is working on it AFAIK. >> > Till then everybody use their own tools for tempest configuration. >> I'd review two of them: >> 1) Puppet configurations that is used in puppet modules CI >> 2) Using configure_tempest.py script from >> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py >> >> Unfortunately there is no ready puppet module or script, that configures >> tempest, you need to create your own. >> >> On other hand the config_tempest.py script provides full configuration, >> support for tempest-deployer-input.conf and possibility to add any config >> options in the command line when running it: >> >> python config_tempest.py \ >> --out etc/tempest.conf \ >> --debug \ >> --create \ >> --deployer-input ~/tempest-deployer-input.conf \ >> identity.uri $OS_AUTH_URL \ >> compute.allow_tenant_isolation true \ >> identity.admin_password $OS_PASSWORD \ >> compute.build_timeout 500 \ >> compute.image_ssh_user cirros >> >> Also it uploads images, creates necessary roles, etc. The only thing it >> requires - existence of public network. >> So finally all tempest configuration from scratch will be like: >> >> CONFIGURE_TEMPEST_DIR="$(ls >> /usr/share/openstack-tempest-*/tools/configure-tempest-directory)" >> $CONFIGURE_TEMPEST_DIR >> neutron net-create nova --shared --router:external=True >> --provider:network_type flat --provider:physical_network datacentre; >> neutron subnet-create --name ext-subnet --allocation-pool >> start=$FLOATING_IP_START,end=$FLOATING_IP_END --disable-dhcp --gateway >> $EXTERNAL_NETWORK_GATEWAY nova $FLOATING_IP_CIDR; >> python tempest/tools/install_venv.py >> python config_tempest.py \ >> --out etc/tempest.conf \ >> --debug \ >> --create \ >> --deployer-input ~/tempest-deployer-input.conf \ >> identity.uri $OS_AUTH_URL \ >> compute.allow_tenant_isolation true \ >> identity.admin_password $OS_PASSWORD \ >> compute.build_timeout 500 \ >> compute.image_ssh_user cirros >> testr init; testr run >> >> In my patch [1] I have proposed it with little changes to TripleO CI repo >> [2] >> >> Like I wrote before there is an option to use puppet for this, I spent a >> time to investigate how to do it and would like share the results with your >> in order to compare it with config_tempest.py approach. >> >> First of all it's surprising that puppet-tempest actually doesn't know to >> do almost anything. All it knows - it's to set IDs of public network (but >> not router) and images. That's all. All the rest you need to configure >> manually. >> Then comes another problem - you can use it only on overcloud controller >> node, where are all service configurations and hiera data. Most of values >> are taken directly from /etc/{service}/service.conf files, so doing it on >> undercloud you will configure undercloud itself (instead of overcloud) >> >> So first of all you need to upload this manifest to controller node of >> overcloud. >> Let's write this puppet manifest, I wrote everything in one file for >> saving a time, but of course it should be a module with usual puppet module >> structure: module_name/manifests/init.pp with module_name class. >> >> Manual configurations: >> >> class testt::config { >> $os_username = 'admin' >> $os_tenant_name = hiera(keystone::roles::admin::admin_tenant) >> $os_password = hiera(admin_password) >> $os_auth_url = hiera(keystone::endpoint::public_url) >> $keystone_auth_uri = regsubst($os_auth_url, '/v2.0', '') >> $floating_range = "192.0.2.0/24" >> $gateway_ip = "192.0.2.1" >> $floating_pool = 'start=192.0.2.50,end=192.0.2.99' >> $fixed_range = '10.0.0.0/24' >> $router_name = 'router1' >> $ca_bundle_cert_path = '/etc/ssl/certs/ca-bundle.crt' >> $cert_path = >> '/etc/pki/ca-trust/source/anchors/puppet_openstack.pem' >> $update_ca_certs_cmd = '/usr/bin/update-ca-trust force-enable && >> /usr/bin/update-ca-trust extract' >> $host_url = regsubst($keystone_auth_uri, ':5000', '') >> } >> >> Most of data is taken from hiera on the controller host. (/etc/hieradata) >> Then we start actually the tempest configuration. Surprisingly it doesn't >> have resource type to work with flavors, so all its configuration is done >> by "exec"s. We run puppet with bash to run bash within a puppet, what gives >> pretty big overhead. >> >> class testt::provision { >> include testt::config >> >> $os_auth_options = "--os-username ${config::os_username} --os-password >> ${config::os_password} --os-tenant-name ${config::os_tenant_name} >> --os-auth-url ${config::os_auth_url}/v2.0" >> >> exec { 'manage_m1.nano_nova_flavor': >> path => '/usr/bin:/bin:/usr/sbin:/sbin', >> provider => shell, >> command => "nova ${os_auth_options} flavor-delete m1.nano ||: ; nova >> ${os_auth_options} flavor-create m1.nano pup_tempest_custom_nano 128 0 1", >> unless => "nova ${os_auth_options} flavor-list | grep >> pup_tempest_custom_nano", >> } >> >> exec { 'manage_m1.micro_nova_flavor': >> path => '/usr/bin:/bin:/usr/sbin:/sbin', >> provider => shell, >> command => "nova ${os_auth_options} flavor-delete m1.micro ||: ;nova >> ${os_auth_options} flavor-create m1.micro pup_tempest_custom_micro 128 0 1", >> unless => "nova ${os_auth_options} flavor-list | grep >> pup_tempest_custom_micro", >> } >> >> Then we create public and private networks, and router between them: >> >> $neutron_deps = [Neutron_network['nova']] >> >> neutron_network { 'nova': >> ensure => 'present', >> router_external => true, >> tenant_name => "${config::os_tenant_name}", >> } >> >> neutron_subnet { 'ext-subnet': >> ensure => 'present', >> cidr => "${config::floating_range}", >> enable_dhcp => false, >> allocation_pools => ["${config::floating_pool}"], >> gateway_ip => "${config::gateway_ip}", >> network_name => 'nova', >> tenant_name => "${config::os_tenant_name}", >> } >> >> neutron_network { 'private': >> ensure => 'present', >> tenant_name => "${config::os_tenant_name}", >> } >> >> neutron_subnet { 'private_subnet': >> ensure => 'present', >> cidr => "${config::fixed_range}", >> network_name => 'private', >> tenant_name => "${config::os_tenant_name}", >> } >> >> neutron_router { "${config::router_name}": >> ensure => 'present', >> tenant_name => "${config::os_tenant_name}", >> gateway_network_name => 'nova', >> require => Neutron_subnet['ext-subnet'], >> } >> neutron_router_interface { "${config::router_name}:private_subnet": >> ensure => 'present', >> } >> >> After this it's a time to upload images: >> >> glance_image { 'cirros': >> ensure => present, >> container_format => 'bare', >> disk_format => 'qcow2', >> is_public => 'yes', >> source => ' >> http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img', >> } >> glance_image { 'cirros_alt': >> ensure => present, >> container_format => 'bare', >> disk_format => 'qcow2', >> is_public => 'yes', >> source => ' >> http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img', >> } >> >> And then you run actually puppet tempest with configuration values, most >> of them you set by yourself: >> >> class { '::tempest': >> >> >> debug => true, >> use_stderr => false, >> log_file => 'tempest.log', >> tempest_clone_owner => $::id, >> git_clone => true, >> setup_venv => true, >> tempest_clone_path => '/tmp/openstack/tempest', >> lock_path => '/tmp/openstack/tempest', >> tempest_config_file => '/tmp/openstack/tempest/etc/tempest.conf', >> configure_images => true, >> configure_networks => true, >> allow_tenant_isolation => true, >> identity_uri => "${testt::config::keystone_auth_uri}/v2.0", >> identity_uri_v3 => "${testt::config::keystone_auth_uri}/v3", >> admin_username => "${testt::config::os_username}", >> admin_tenant_name => "${testt::config::os_tenant_name}", >> admin_password => "${testt::config::os_password}", >> admin_domain_name => 'Default', >> auth_version => 'v3', >> image_name => 'cirros', >> image_name_alt => 'cirros_alt', >> cinder_available => true, >> glance_available => true, >> horizon_available => $horizon, >> nova_available => true, >> neutron_available => true, >> ceilometer_available => $ceilometer, >> aodh_available => $aodh, >> trove_available => $trove, >> sahara_available => $sahara, >> heat_available => $heat, >> swift_available => true, >> ironic_available => $ironic, >> public_network_name => 'nova', >> dashboard_url => "${testt::config::host_url}", >> flavor_ref => 'pup_tempest_custom_nano', >> flavor_ref_alt => 'pup_tempest_custom_micro', >> image_ssh_user => 'cirros', >> image_alt_ssh_user => 'cirros', >> img_file => 'cirros-0.3.4-x86_64-disk.img', >> compute_build_interval => 10, >> ca_certificates_file => "${testt::config::ca_bundle_cert_path}", >> img_dir => '/tmp/openstack/tempest', >> } >> >> But it's not enough, you need also to make some workarounds and >> additional configurations, for example: >> >> tempest_config { 'object-storage/operator_role': >> value => 'SwiftOperator', >> path => "${tempest_clone_path}/etc/tempest.conf", >> } >> } >> >> After this run puppet on controller node: >> >> sudo puppet apply --verbose --debug --detailed-exitcodes -e "include >> ::testt" | tee ~/puppet_run.log >> >> After everything is finished, you need to copy the folder with tempest to >> your node: >> scp -r -heat-admin@${CONTROLLER}:/tmp/openstack /tmp/ >> >> After this run within this directory testr init and run tests: >> /tmp/tempest/tools/with_venv.sh testr init >> /tmp/tempest/tools/with_venv.sh testr run >> >> There are still holes in this configuration and most likely you'd fix it >> by another workarounds and tempest_config runs, because it's still a few of >> skipped tests, so configuration is not full as it would be done with >> config_tempest.py. >> You don't have also any possibility to add custom configuration in >> running the manifest, for each config change you need to change the >> manifest itself which makes it maintenance harder and more complex. >> >> I would say that conclusion is quite obvious for me and it's much easier >> even to write tempest.conf manually from scratch or simple template and use >> 5 bash lines, then use puppet for things it's completely not fit to. >> >> P.S. In this script I used ideas from puppet-openstack-integration and >> packstack projects. >> >> [1] https://review.openstack.org/#/c/295844/ >> [2] https://git.openstack.org/openstack-infra/tripleo-ci >> >> -- >> Best regards >> Sagi Shnaidman >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > > -- > Best regards, > Andrey Kurilin. > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Best regards Sagi Shnaidman
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
