Re: [openstack-dev] [qa] [infra] [placement] tempest plugins virtualenv
On Fri, 28 Sep 2018 23:10:06 +0900 Matthew Treinish wrote > On Fri, Sep 28, 2018 at 02:39:24PM +0100, Chris Dent wrote: > > > > I'm still trying to figure out how to properly create a "modern" (as > > in zuul v3 oriented) integration test for placement using gabbi and > > tempest. That work is happening at > > https://review.openstack.org/#/c/601614/ > > > > There was lots of progress made after the last message on this > > topic > > http://lists.openstack.org/pipermail/openstack-dev/2018-September/134837.html > > > > but I've reached another interesting impasse. > > > > From devstack's standpoint, the way to say "I want to use a tempest > > plugin" is to set TEMPEST_PLUGINS to alist of where the plugins are. > > devstack:lib/tempest then does a: > > > > tox -evenv-tempest -- pip install -c > > $REQUIREMENTS_DIR/upper-constraints.txt $TEMPEST_PLUGINS > > > > http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_12_58_138163 > > > > > > I have this part working as expected. > > > > However, > > > > The advice is then to create a new job that has a parent of > > devstack-tempest. That zuul job runs a variety of tox environments, > > depending on the setting of the `tox_envlist` var. If you wish to > > use a `tempest_test_regex` (I do) the preferred tox environment is > > 'all'. > > > > That venv doesn't have the plugin installed, thus no gabbi tests are > > found: > > > > http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_13_25_798683 > > > > Right above this line it shows that the gabbi-tempest plugin is installed in > the venv: > > http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_13_25_650661 > > > at version 0.1.1. It's a bit weird because it's line wrapped in my browser. > The devstack logs also shows the plugin: > > http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/controller/logs/devstacklog.txt.gz#_2018-09-28_11_13_13_076 > > > All the tempest tox jobs that run tempest (and the tempest-venv command used > by > devstack) run inside the same tox venv: > > https://github.com/openstack/tempest/blob/master/tox.ini#L52 > > My guess is that the plugin isn't returning any tests that match the regex. > > I'm also a bit alarmed that tempest run is returning 0 there when no tests > are > being run. That's definitely a bug because things should fail with no tests > being successfully run. Tempest run fail on "no test" run [1] .. [1] https://github.com/openstack/tempest/blob/807f0dec66689aced05c2bb970f344cbb8a3c6a3/tempest/cmd/run.py#L182 -gmann > > -Matt Treinish > > > > > How do I get my plugin installed into the right venv while still > > following the guidelines for good zuul behavior? > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [infra] [placement] tempest plugins virtualenv
On Fri, Sep 28, 2018 at 03:31:10PM +0100, Chris Dent wrote: > On Fri, 28 Sep 2018, Matthew Treinish wrote: > > > > http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_13_25_798683 > > > > Right above this line it shows that the gabbi-tempest plugin is installed in > > the venv: > > > > http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_13_25_650661 > > Ah, so it is, thanks. My grepping and visual-grepping failed > because of the weird linebreaks. Le sigh. > > For curiosity: What's the processing that is making it be installed > twice? I ask because I'm hoping to (eventually) trim this to as > small and light as possible. And then even more eventually I hope to > make it so that if a project chooses the right job and has a gabbits > directory, they'll get run. The plugin should only be installed once. From the logs here is the only place the plugin is being installed in the venv: http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_13_01_027151 The rest of the references are just tox printing out the packages installed in the venv before running a command. > > The part that was confusing for me was that the virtual env that > lib/tempest (from devstack) uses is not even mentioned in tempest's > tox.ini, so is using its own directory as far as I could tell. It should be, devstack should be using the venv-tempest tox job to do venv prep (like installling the plugins) and run commands (like running tempest list-plugins for the log). This tox env is defined here: https://github.com/openstack/tempest/blob/master/tox.ini#L157-L162 It's sort of a hack, devstack is just using tox as venv manager for setting up tempest. But, then we use tox in the runner (what used to be devstack-gate) so this made sense. -Matt Treinish > > > My guess is that the plugin isn't returning any tests that match the regex. > > I'm going to run it without a regex and see what it produces. > > It might be that pre job I'm using to try to get the gabbits in the > right place is not working as desired. > > A few patchsets ago when I was using the oogly way of doing things > it was all working. signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [infra] [placement] tempest plugins virtualenv
On Fri, 28 Sep 2018, Matthew Treinish wrote: http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_13_25_798683 Right above this line it shows that the gabbi-tempest plugin is installed in the venv: http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_13_25_650661 Ah, so it is, thanks. My grepping and visual-grepping failed because of the weird linebreaks. Le sigh. For curiosity: What's the processing that is making it be installed twice? I ask because I'm hoping to (eventually) trim this to as small and light as possible. And then even more eventually I hope to make it so that if a project chooses the right job and has a gabbits directory, they'll get run. The part that was confusing for me was that the virtual env that lib/tempest (from devstack) uses is not even mentioned in tempest's tox.ini, so is using its own directory as far as I could tell. My guess is that the plugin isn't returning any tests that match the regex. I'm going to run it without a regex and see what it produces. It might be that pre job I'm using to try to get the gabbits in the right place is not working as desired. A few patchsets ago when I was using the oogly way of doing things it was all working. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [infra] [placement] tempest plugins virtualenv
On Fri, Sep 28, 2018 at 02:39:24PM +0100, Chris Dent wrote: > > I'm still trying to figure out how to properly create a "modern" (as > in zuul v3 oriented) integration test for placement using gabbi and > tempest. That work is happening at https://review.openstack.org/#/c/601614/ > > There was lots of progress made after the last message on this > topic > http://lists.openstack.org/pipermail/openstack-dev/2018-September/134837.html > but I've reached another interesting impasse. > > From devstack's standpoint, the way to say "I want to use a tempest > plugin" is to set TEMPEST_PLUGINS to alist of where the plugins are. > devstack:lib/tempest then does a: > > tox -evenv-tempest -- pip install -c > $REQUIREMENTS_DIR/upper-constraints.txt $TEMPEST_PLUGINS > > http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_12_58_138163 > > I have this part working as expected. > > However, > > The advice is then to create a new job that has a parent of > devstack-tempest. That zuul job runs a variety of tox environments, > depending on the setting of the `tox_envlist` var. If you wish to > use a `tempest_test_regex` (I do) the preferred tox environment is > 'all'. > > That venv doesn't have the plugin installed, thus no gabbi tests are > found: > > http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_13_25_798683 Right above this line it shows that the gabbi-tempest plugin is installed in the venv: http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_13_25_650661 at version 0.1.1. It's a bit weird because it's line wrapped in my browser. The devstack logs also shows the plugin: http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/controller/logs/devstacklog.txt.gz#_2018-09-28_11_13_13_076 All the tempest tox jobs that run tempest (and the tempest-venv command used by devstack) run inside the same tox venv: https://github.com/openstack/tempest/blob/master/tox.ini#L52 My guess is that the plugin isn't returning any tests that match the regex. I'm also a bit alarmed that tempest run is returning 0 there when no tests are being run. That's definitely a bug because things should fail with no tests being successfully run. -Matt Treinish > > How do I get my plugin installed into the right venv while still > following the guidelines for good zuul behavior? > signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa] [infra] [placement] tempest plugins virtualenv
I'm still trying to figure out how to properly create a "modern" (as in zuul v3 oriented) integration test for placement using gabbi and tempest. That work is happening at https://review.openstack.org/#/c/601614/ There was lots of progress made after the last message on this topic http://lists.openstack.org/pipermail/openstack-dev/2018-September/134837.html but I've reached another interesting impasse. From devstack's standpoint, the way to say "I want to use a tempest plugin" is to set TEMPEST_PLUGINS to alist of where the plugins are. devstack:lib/tempest then does a: tox -evenv-tempest -- pip install -c $REQUIREMENTS_DIR/upper-constraints.txt $TEMPEST_PLUGINS http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_12_58_138163 I have this part working as expected. However, The advice is then to create a new job that has a parent of devstack-tempest. That zuul job runs a variety of tox environments, depending on the setting of the `tox_envlist` var. If you wish to use a `tempest_test_regex` (I do) the preferred tox environment is 'all'. That venv doesn't have the plugin installed, thus no gabbi tests are found: http://logs.openstack.org/14/601614/21/check/placement-tempest-gabbi/f44c185/job-output.txt.gz#_2018-09-28_11_13_25_798683 How do I get my plugin installed into the right venv while still following the guidelines for good zuul behavior? -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
