Re: [openstack-dev] [nova][testing] How to run a subset of py34 unit tests

[melanie witt]

On Jan 25, 2016, at 17:57, melanie witt  wrote:

> Thanks!! That's indeed the same problem and removing the '$' from the 
> exclude_regex worked for me. (ostestr --blacklist_file tests-py3.txt --regex 
> "nova.tests.unit.network" ran all non-blacklisted network tests)

Argh, I spoke too soon, removing '$' stops it from excluding tests in the 
blacklist. I had gotten mixed up when I looked at the output of the ostestr 
--list with the difference in the exclude_regex.

For example:

testr list-tests '^((?!nova.tests.unit.volume.test_cinder.CinderApiTestCase).)*'

will *include* all of the nova.tests.unit.volume.test_cinder.CinderApiTestCase 
tests.

Whereas:

testr list-tests 
'^((?!nova.tests.unit.volume.test_cinder.CinderApiTestCase).)*$'

will exclude the nova.tests.unit.volume.test_cinder.CinderApiTestCase tests.

:(

-melanie








signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][testing] How to run a subset of py34 unit tests

[melanie witt]

On Jan 22, 2016, at 19:14, Matthew Treinish  wrote:

> Although, now that I look at the list it definitely looks like this one,
> which I had forgotten about:
> 
> https://bugs.launchpad.net/os-testr/+bug/1506215
> 
> It sounds like that bug proposes a direction so if you could patch os-testr
> to give that a try and see if that fixes your issue. If it does then we can
> just land that in os-testr.

Thanks!! That's indeed the same problem and removing the '$' from the 
exclude_regex worked for me. (ostestr --blacklist_file tests-py3.txt --regex 
"nova.tests.unit.network" ran all non-blacklisted network tests)

> However, I'm wondering if it makes more sense to change how the selection
> actually works with os-testr. This regex generation approach does get pretty
> unwieldy and seems very error prone. I'm thinking it might make more sense to
> generate a test-list and then have os-testr filter that list itself and then
> pass the pruned list to --load-list in testr run.

I could see that because conceptually the test selection has two stages: 
produce a list of non-blacklisted tests, then filter that list further as 
desired. Personally, using --blacklist_file and --regex seemed straightforward 
to me since I could specify the filter regex as I'm used to doing. It worked in 
my case but there might be cases I can't think of where the append method won't 
be able to give the desired result.


-melanie








signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][testing] How to run a subset of py34 unit tests

[Matthew Treinish]

On Fri, Jan 22, 2016 at 06:46:48PM -0800, melanie witt wrote:
> Hi everyone,
> 
> I noticed because of the way we run the py34 tests in tox.ini, I'm not able 
> to specify a filter regex the way I normally do as a positional arg, for 
> example: 'tox -epy34 nova.tests.unit.network' doesn't filter and it runs 
> everything. ('tox -epy27 nova.tests.unit.network' will only run tests that 
> match nova.tests.unit.network).
> 
> I couldn't figure out how we could add something to tox.ini to make it work 
> -- we're calling ostestr with the --blacklist_file option. I'm not completely 
> clear on what 'ostestr --blacklist_file  --regex ' does but I 
> couldn't get it to do what I want. From the documentation [1], it adds 
> --regex to the regex created from the --blacklist_file. The regex from the 
> blacklist file looks something like this '^((?!blacklistedstuff).)*$' and if 
> I can only append to it, the best I could do was a positive lookbehind but 
> that can't match at the beginning of a line. (For example, I tried "ostestr 
> --blacklist_file tests-py3.txt --regex '(?<=network)'" and it matched all the 
> non-blacklisted tests that ended with the word "network"). It seems like what 
> I would need is for --regex to do another re.search() and match the line only 
> if the previous regex from the blacklist also matched.

So os-testr literally just generates a regex [2] and passes that to testr
directly when it's called with subprocess. [3] os-testr still relies on testr
to do the actual test selection.

This does sound like a real bug in os-testr's regex generation, can you file
a bug here:

https://bugs.launchpad.net/os-testr

Although, now that I look at the list it definitely looks like this one,
which I had forgotten about:

https://bugs.launchpad.net/os-testr/+bug/1506215

It sounds like that bug proposes a direction so if you could patch os-testr
to give that a try and see if that fixes your issue. If it does then we can
just land that in os-testr.

However, I'm wondering if it makes more sense to change how the selection
actually works with os-testr. This regex generation approach does get pretty
unwieldy and seems very error prone. I'm thinking it might make more sense to
generate a test-list and then have os-testr filter that list itself and then
pass the pruned list to --load-list in testr run.

Thanks,

Matt Treinish


> [1] http://docs.openstack.org/developer/os-testr/ostestr.html#test-selection
[2] 
https://github.com/openstack/os-testr/blob/master/os_testr/os_testr.py#L145-L174
[3] https://github.com/openstack/os-testr/blob/master/os_testr/os_testr.py#L191


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][testing] How to run a subset of py34 unit tests

[melanie witt]

Hi everyone,

I noticed because of the way we run the py34 tests in tox.ini, I'm not able to 
specify a filter regex the way I normally do as a positional arg, for example: 
'tox -epy34 nova.tests.unit.network' doesn't filter and it runs everything. 
('tox -epy27 nova.tests.unit.network' will only run tests that match 
nova.tests.unit.network).

I couldn't figure out how we could add something to tox.ini to make it work -- 
we're calling ostestr with the --blacklist_file option. I'm not completely 
clear on what 'ostestr --blacklist_file  --regex ' does but I 
couldn't get it to do what I want. From the documentation [1], it adds --regex 
to the regex created from the --blacklist_file. The regex from the blacklist 
file looks something like this '^((?!blacklistedstuff).)*$' and if I can only 
append to it, the best I could do was a positive lookbehind but that can't 
match at the beginning of a line. (For example, I tried "ostestr 
--blacklist_file tests-py3.txt --regex '(?<=network)'" and it matched all the 
non-blacklisted tests that ended with the word "network"). It seems like what I 
would need is for --regex to do another re.search() and match the line only if 
the previous regex from the blacklist also matched.

As a workaround to run only network tests, I did:

source .tox/py34/bin/activate
OS_TEST_PATH=./nova/tests/unit/network ostestr --blacklist_file tests-py3.txt

Does anyone know a better way?

Thanks,
-melanie


[1] http://docs.openstack.org/developer/os-testr/ostestr.html#test-selection



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][testing] How to run a subset of py34 unit tests

[Davanum Srinivas]

Melanie,

The following should work as well

source .tox/py34/bin/activate
ostestr --regex *.network.*

Thanks,
Dims

On Fri, Jan 22, 2016 at 9:46 PM, melanie witt  wrote:
> Hi everyone,
>
> I noticed because of the way we run the py34 tests in tox.ini, I'm not able 
> to specify a filter regex the way I normally do as a positional arg, for 
> example: 'tox -epy34 nova.tests.unit.network' doesn't filter and it runs 
> everything. ('tox -epy27 nova.tests.unit.network' will only run tests that 
> match nova.tests.unit.network).
>
> I couldn't figure out how we could add something to tox.ini to make it work 
> -- we're calling ostestr with the --blacklist_file option. I'm not completely 
> clear on what 'ostestr --blacklist_file  --regex ' does but I 
> couldn't get it to do what I want. From the documentation [1], it adds 
> --regex to the regex created from the --blacklist_file. The regex from the 
> blacklist file looks something like this '^((?!blacklistedstuff).)*$' and if 
> I can only append to it, the best I could do was a positive lookbehind but 
> that can't match at the beginning of a line. (For example, I tried "ostestr 
> --blacklist_file tests-py3.txt --regex '(?<=network)'" and it matched all the 
> non-blacklisted tests that ended with the word "network"). It seems like what 
> I would need is for --regex to do another re.search() and match the line only 
> if the previous regex from the blacklist also matched.
>
> As a workaround to run only network tests, I did:
>
> source .tox/py34/bin/activate
> OS_TEST_PATH=./nova/tests/unit/network ostestr --blacklist_file tests-py3.txt
>
> Does anyone know a better way?
>
> Thanks,
> -melanie
>
>
> [1] http://docs.openstack.org/developer/os-testr/ostestr.html#test-selection
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev