Re: [openstack-dev] [QA] Running Tempest tests for a customized cloud
On Wed, Aug 17, 2016 at 12:27:29AM +, Elancheran Subramanian wrote: > Hello Punal, > We do support both V2 and V3, that’s just a example I’ve stated BTW. We do > have our own integration tests which are pretty much covers all our > integration points with openstack. But we would like to leverage the tempest > while doing our upstream merge for openstack components in CI. > > I believe the tests support the include list, how can I exclude test? Any > pointer would be a great help. > It depends on the test runner you're using. The tempest run command supports several methods of excluding tests: http://docs.openstack.org/developer/tempest/run.html#test-selection If you use ostestr it offers the same options that tempest run offers: http://docs.openstack.org/developer/os-testr/ostestr.html#test-selection If you're using testr plain it can take a regex filter that will run any tests that match regex filter: http://testrepository.readthedocs.io/en/latest/MANUAL.html#running-tests You can also use a negative lookahead in the regex to exclude something. This is how tempest's tox jobs skip slow tests in the normal gate run jobs: http://git.openstack.org/cgit/openstack/tempest/tree/tox.ini#n80 Other test runners have similar selection mechanisms. -Matt Treinish signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA] Running Tempest tests for a customized cloud
On Tue, Aug 16, 2016 at 10:40:07PM +, Elancheran Subramanian wrote: > Hello There, > I’m currently playing with using Tempest as our integration tests for our > internal and external clouds, facing some issues with api which are not > supported in our cloud. For ex, listing domains isn’t supported for any user, > due to this V3 Identity tests are failing. So I would like to know what’s the > best practice? Like fix those tests, and apply those fix as patch? Or just > exclude those tests? > It really depends on the configuration of your cloud. It could be a bug in tempest or it could be a tempest configuration issue. You also could have configured your cloud in a way that is invalid and breaks API end user expectations from tempest's POV. It's hard to say without out knowing the specifics of your deployment. I'd start with filing a bug with more details. You can file a bug here: https://bugs.launchpad.net/tempest If it's a valid tempest bug then submitting a patch to fix the bug in tempest is the best path forward. -Matt Treinish signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA] Running Tempest tests for a customized cloud
Hello Punal, We do support both V2 and V3, that’s just a example I’ve stated BTW. We do have our own integration tests which are pretty much covers all our integration points with openstack. But we would like to leverage the tempest while doing our upstream merge for openstack components in CI. I believe the tests support the include list, how can I exclude test? Any pointer would be a great help. Thanks, Cheran From: punal patel <punal.pa...@gmail.com<mailto:punal.pa...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Date: Tuesday, August 16, 2016 at 4:16 PM To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [QA] Running Tempest tests for a customized cloud Hi Cheran, Best practice depends on your test plan and what coverage you. Does your test plan covers V3 Identity tests ? If it does and its failing, you should modify to make it work for your environment. Fast way to move forward is to exclude those tests. -Punal On Tue, Aug 16, 2016 at 3:40 PM, Elancheran Subramanian <esubraman...@godaddy.com<mailto:esubraman...@godaddy.com>> wrote: Hello There, I’m currently playing with using Tempest as our integration tests for our internal and external clouds, facing some issues with api which are not supported in our cloud. For ex, listing domains isn’t supported for any user, due to this V3 Identity tests are failing. So I would like to know what’s the best practice? Like fix those tests, and apply those fix as patch? Or just exclude those tests? Would be great if anyone could share their experience on this. Thanks & Regards, Cheran __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA] Running Tempest tests for a customized cloud
Hi Cheran, Best practice depends on your test plan and what coverage you. Does your test plan covers V3 Identity tests ? If it does and its failing, you should modify to make it work for your environment. Fast way to move forward is to exclude those tests. -Punal On Tue, Aug 16, 2016 at 3:40 PM, Elancheran Subramanian < esubraman...@godaddy.com> wrote: > Hello There, > I’m currently playing with using Tempest as our integration tests for our > internal and external clouds, facing some issues with api which are not > supported in our cloud. For ex, listing domains isn’t supported for any > user, due to this V3 Identity tests are failing. So I would like to know > what’s the best practice? Like fix those tests, and apply those fix as > patch? Or just exclude those tests? > > Would be great if anyone could share their experience on this. > > Thanks & Regards, > Cheran > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [QA] Running Tempest tests for a customized cloud
Hello There, I’m currently playing with using Tempest as our integration tests for our internal and external clouds, facing some issues with api which are not supported in our cloud. For ex, listing domains isn’t supported for any user, due to this V3 Identity tests are failing. So I would like to know what’s the best practice? Like fix those tests, and apply those fix as patch? Or just exclude those tests? Would be great if anyone could share their experience on this. Thanks & Regards, Cheran __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
