Bryan, Check out refstack.openstack.org and https://github.com/openstack/refstack
The refstack project provides a client which enables anyone to run tempest tests on their own clouds. It is flexible, with options for selecting all or specific test sets, or specific tests and also has an option to specify the build SHA. Although its first purpose has been to provide the community with a way to run and track interop tests on their clouds, it has been designed with flexibility and extensibility in mind. If you want to ask specific questions about the project, they can be found on IRC: #refstack and in their weekly meetings on IRC: #openstack-meeting-alt, on Mondays at 19:00 UTC<https://www.google.com/search?q=current+utc+time> And, of course, you can always ask questions on the dev list by including [refstack] in the subject. I believe Rally also provides options for running tempest tests on clouds of your choosing. --Rocky From: Bryan Sullivan [mailto:bls...@hotmail.com] Sent: Monday, April 11, 2016 10:13 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Congress] Issues with Tox testing Hi Anusha, That helps. Just one more question: in Liberty (which I'm currently based upon) have the tempest tests been run outside of devstack deployments, i.e. in an actual OpenStack deployment? The guide you reference mentions devstack but it's not clear that the same process applies outside devstack: e.g. "To list all Congress test cases, run command in /opt/stack/tempest:" references the "/opt/stack" folder which is not created outside of devstack environments. Thus to run them in a full OpenStack deployment, do I need to install tempest and create an "opt/stack/tempest" folder to which the tests are copied, on the same server where Congress is installed? I'll try Mitaka soon but I expect to have the same question there: basically, are the tempest tests expected to be usable outside a devstack deploy? I guess I could just try it, but I don't want to waste time if this is not designed to be used outside devstack environments. Thanks, Bryan Sullivan ________________________________ Date: Fri, 8 Apr 2016 09:01:29 +0530 From: anusha.ii...@gmail.com To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Congress] Issues with Tox testing Hi Bryan, tox -epy27 doesn't run tempest tests , that is tests mentioned in https://github.com/openstack/congress/tree/stable/liberty/contrib/tempest<https://github.com/openstack/congress/tree/stable/liberty/contrib/tempest?> , it runs only unit tests , tests present in https://github.com/openstack/congress/tree/stable/liberty/congress/tests . To run tempest tests, you need to manually copy the files to tempest and run the tests as mentioned in following readme https://github.com/openstack/congress/blob/stable/liberty/contrib/tempest/README.rst Mitaka supports tempest plugin, so manually copying tests to tempest can be avoided if you are using mitaka. Hope I clarified your question. Best Regards, Anusha On 8 April 2016 at 08:51, Bryan Sullivan <bls...@hotmail.com<mailto:bls...@hotmail.com>> wrote: OK, somehow I did not pick up on that, or dropped it along the way of developing the script. Thanks for the clarification, also that Tempest is not required. I should have clarified that I'm using stable/liberty as the base. I will be moving to stable/mitaka soon, as part of the OPNFV Colorado release development. One additional question then - are the tests run by "tox -epy27" the same as the tests in the folder https://github.com/openstack/congress/tree/stable/liberty/contrib/tempest? If not, how are those tests supposed to be run for a non-devstack deploy (I see reference to devstack in the readme)? I see that the folders have been reorganized for mitaka. My question is per the goal to include as much of the Congress tests as possible in the OPNFV CI/CD process. Not that I expect any to fail, I just want OPNFV to leverage the full test suite. If for liberty that's best left as the tests run by the tox command, then that's OK. Thanks, Bryan Sullivan ________________________________ Date: Thu, 7 Apr 2016 17:11:36 -0700 From: ekcs.openst...@gmail.com<mailto:ekcs.openst...@gmail.com> To: openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [Congress] Issues with Tox testing Thanks for the feedback, Bryan. Glad you got things working! 1. The instructions asking to install those packages are missing from kilo (we'll fix that), but they have been there since liberty. Was it perhaps unclear because the line is too long? * Additionally: $ sudo apt-get install git gcc python-dev libxml2 libxslt1-dev libzip-dev mysql-server python-mysqldb build-essential libssl-dev libffi-dev 2. Tempest should not be required by the tox tests. Thanks! From: Bryan Sullivan <bls...@hotmail.com<mailto:bls...@hotmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Date: Thursday, April 7, 2016 at 4:29 PM To: "openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [Congress] Issues with Tox testing An update: I found that there were two dependencies needed that were not clear in the guide at https://github.com/openstack/congress. I also installed Tempest which was not referenced before. If these additions are correct (they worked for me), they should be added to https://github.com/openstack/congress/blob/master/README.rst. $ sudo apt-get install libffi-dev libssl-dev $ cd ~/git $ git clone https://github.com/openstack/tempest/ $ cd tempest $ ~/git/congress/bin/pip install -r requirements.txt $ ~/git/congress/bin/pip install . (not sure if both pip commands are needed - I'm not an expert on pip install) After that, "tox -epy27" ran thru fine: ------------------------------------------------------------------------------------------------------- ----------- congress.tests.policy_engines.test_vmplacement.TestComputeVmAssignment.test_set_policy_with_dashes 27.623 congress.tests.policy_engines.test_vmplacement.TestComputeVmAssignment.test_set_policy 27.212 congress.tests.policy_engines.test_agnostic_performance.TestRuntimePerformance.test_simulate_latency 1.325 congress.tests.dse.test_dse.TestDSE.test_policy_tables 1.229 congress.tests.policy_engines.test_agnostic_performance.TestRuntimePerformance.test_select_100matches 1.184 congress.tests.test_congress.TestCongress.test_policy_execute 1.127 congress.tests.test_congress.TestCongress.test_datasource_api_model_execute 1.067 congress.tests.policy_engines.test_agnostic_performance.TestRuntimePerformance.test_update_nonrecursive 0.967 congress.tests.dse.test_dse.TestDSE.test_policy_table_publish 0.681 congress.tests.datasources.test_neutron_driver.TestDataSourceDriver.test_poll_subscribe 0.671 ______________________________________________________________ summary _______________________________________________________________ py27: commands succeeded congratulations :) Thanks, Bryan Sullivan ________________________________ From: bls...@hotmail.com<mailto:bls...@hotmail.com> To: openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org> Date: Thu, 7 Apr 2016 11:16:48 -0700 Subject: [openstack-dev] [Congress] Issues with Tox testing Hi Congress team, A question for tox testing expert on the Congress team. I'm trying to run the tox tests as described at https://github.com/openstack/congress, specifically the two commands: $ sudo pip install 'tox<1.7' $ tox -epy27 Due to conflicts with the OS-owned python config, I run these under my virtualenv created in the congress repo as: $ cd ~/git/congress $ bin/pip install 'tox<1.7' $ bin/tox -epy27 But in any event (whether I try to run the tox within the virtualenv or not), I get errors such as: c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory What's missing in the setup for running these tests? Note that I have all the config needed to run bash/CLI-based test scripts such as https://git.opnfv.org/cgit/copper/tree/tests/adhoc/dmz01.sh Thanks, Bryan Sullivan __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org<mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org<mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
