Re: [openstack-dev] [tempest] How to run tempest tests
Hi Folks, I would like to know about the best practices followed for skipping tests not applicable for my environment. I know one of the ways is to use the below decorator over the test method: @test.skip_because(bug=BUG_ID) However, what if my deployment doesn't support VPNAAS and I want to skip those tests. Similarly, what if I want to skip the entire suite of sahara(data processing) tests. Are there any options in testr to customize running of tempest tests as per my environment/requirements? Regards, Om On Wed, Nov 26, 2014 at 3:13 AM, Vineet Menon mvineetme...@gmail.com wrote: Hi, Thanjs for clearing that up... I had a hard time understanding the screws before I went with testr. Regards, Vineet On 25 Nov 2014 17:46, Matthew Treinish mtrein...@kortar.org wrote: On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote: Sorry for my previous message with wrong subject Hi all, By reading the tempest documentation page [1] a user can run tempest tests by using whether testr or run_tempest.sh or tox. What is the best practice? run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is my preferred way, currently. Any thought? So the options are there for different reasons and fit different purposes. The run_tempest.sh script exists mostly for legacy reasons as some people prefer to use it, and it predates the usage of tox in tempest. It also has some advantages like that it can run without a venv and provides some other options. Tox is what we use for gating, and we keep most of job definitions for gating in the tox.ini file. If you're trying to reproduce a gate run locally using tox is what is recommended to use. Personally I use it to run everything just because I often mix unit tests and tempest runs and I like having separate venvs for both being created on demand. Calling testr directly is just what all of these tools are doing under the covers, and it'll always be an option. One thing we're looking to do this cycle is to add a single entry point for running tempest which will hopefully clear up this confusion, and make the interface for interacting with tempest a bit nicer. When this work is done, the run_tempest.sh script will most likely disappear and tox will probably just be used for gating job definitions and just call the new entry-point instead of testr directly. BR, Angelo [1] http://docs.openstack.org/developer/tempest/overview.html#quickstart -Matt Treinish ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tempest] How to run tempest tests
Thanks Sridhar. I'm aware of using skipException for skipping tests at class level. However, this involves putting it in every class to skip tests which are not desired. I was looking for a way to control the tests I want to run through some kind of a configuration file, the options to pass to test runner. Regards Om On Fri, Nov 28, 2014 at 2:02 PM, Sridhar Gaddam sridhar.gad...@enovance.com wrote: If the deployment does not support IPv6, we use the following convention to skip the tests at class level. https://github.com/openstack/tempest/blob/master/tempest/api/network/base.py#L65 Regards, --Sridhar. On 11/28/2014 01:50 PM, om prakash pandey wrote: Hi Folks, I would like to know about the best practices followed for skipping tests not applicable for my environment. I know one of the ways is to use the below decorator over the test method: @test.skip_because(bug=BUG_ID) However, what if my deployment doesn't support VPNAAS and I want to skip those tests. Similarly, what if I want to skip the entire suite of sahara(data processing) tests. Are there any options in testr to customize running of tempest tests as per my environment/requirements? Regards, Om On Wed, Nov 26, 2014 at 3:13 AM, Vineet Menon mvineetme...@gmail.com wrote: Hi, Thanjs for clearing that up... I had a hard time understanding the screws before I went with testr. Regards, Vineet On 25 Nov 2014 17:46, Matthew Treinish mtrein...@kortar.org wrote: On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote: Sorry for my previous message with wrong subject Hi all, By reading the tempest documentation page [1] a user can run tempest tests by using whether testr or run_tempest.sh or tox. What is the best practice? run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is my preferred way, currently. Any thought? So the options are there for different reasons and fit different purposes. The run_tempest.sh script exists mostly for legacy reasons as some people prefer to use it, and it predates the usage of tox in tempest. It also has some advantages like that it can run without a venv and provides some other options. Tox is what we use for gating, and we keep most of job definitions for gating in the tox.ini file. If you're trying to reproduce a gate run locally using tox is what is recommended to use. Personally I use it to run everything just because I often mix unit tests and tempest runs and I like having separate venvs for both being created on demand. Calling testr directly is just what all of these tools are doing under the covers, and it'll always be an option. One thing we're looking to do this cycle is to add a single entry point for running tempest which will hopefully clear up this confusion, and make the interface for interacting with tempest a bit nicer. When this work is done, the run_tempest.sh script will most likely disappear and tox will probably just be used for gating job definitions and just call the new entry-point instead of testr directly. BR, Angelo [1] http://docs.openstack.org/developer/tempest/overview.html#quickstart -Matt Treinish ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Rally] Tempest + Rally: first success
Hi Andrey, Thanks a ton for putting together this blog on using tempest + rally. I followed all the steps listed and managed to get tempest successfully installed. However, I was not able to proceed beyond and couldn't manage to run tempest even once. I am getting the below error: om@desktop2:~/rally$ rally verify start --set image Command failed, please check log for more info 2014-06-03 18:14:56.029 8331 CRITICAL rally [-] IndexError: list index out of range What did I mess up while following the blog? Regards Om On Sat, May 31, 2014 at 3:46 AM, Andrey Kurilin akuri...@mirantis.com wrote: Hi stackers, I would like to share with you great news. We all know that it's quite hard to use Tempest out of gates, especially when you are going to benchmark different clouds, run just part of tests and would like to store somewhere results. As all this stuff doesn't belong to Tempest, we decided to make it in Rally. More details about how to use Tempest in one click in my tutorial: http://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/ -- Best regards, Andrey Kurilin. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Rally] Tempest + Rally: first success
220, in generate 2014-06-04 02:32:14.071 1939 TRACE rally self._set_network() 2014-06-04 02:32:14.071 1939 TRACE rally File /usr/local/lib/python2.7/dist-packages/rally/verification/verifiers/tempest/config.py, line 191, in _set_network 2014-06-04 02:32:14.071 1939 TRACE rally subnet = neutron.list_subnets(network_id=net_id)['subnets'][0] 2014-06-04 02:32:14.071 1939 TRACE rally IndexError: list index out of range 2014-06-04 02:32:14.071 1939 TRACE rally Command failed, please check log for more info On Tue, Jun 3, 2014 at 7:09 PM, Andrey Kurilin akuri...@mirantis.com wrote: Hey, Om! Can you launch Rally in debug mode and share logs? rally -vd verify start --set image On Tue, Jun 3, 2014 at 3:49 PM, om prakash pandey pande...@gmail.com wrote: Hi Andrey, Thanks a ton for putting together this blog on using tempest + rally. I followed all the steps listed and managed to get tempest successfully installed. However, I was not able to proceed beyond and couldn't manage to run tempest even once. I am getting the below error: om@desktop2:~/rally$ rally verify start --set image Command failed, please check log for more info 2014-06-03 18:14:56.029 8331 CRITICAL rally [-] IndexError: list index out of range What did I mess up while following the blog? Regards Om On Sat, May 31, 2014 at 3:46 AM, Andrey Kurilin akuri...@mirantis.com wrote: Hi stackers, I would like to share with you great news. We all know that it's quite hard to use Tempest out of gates, especially when you are going to benchmark different clouds, run just part of tests and would like to store somewhere results. As all this stuff doesn't belong to Tempest, we decided to make it in Rally. More details about how to use Tempest in one click in my tutorial: http://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/ -- Best regards, Andrey Kurilin. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best regards, Andrey Kurilin. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
