Re: [openstack-dev] [Fuel] py.test vs testrepository

[Roman Prykhodchenko]

Thank you guys for all your help! Special thanks to Robert who helped to find a 
workaround for an issue [1] that didn’t let us use testr for Fuel Client. The 
patch [2] was merged and both unit and functional tests are launched by subunit 
and the data is maintained by testrepository.

Please also note that in order to facilitate debugging two additional tox 
environments —  dbgunit or dbgfunc,  were introduced for either unit or 
functional tests.


1. https://bugs.launchpad.net/testrepository/+bug/1504310 

2. https://review.openstack.org/#/c/227895/


- romcheg


> 9 жовт. 2015 р. о 01:51 Roman Prykhodchenko  написав(ла):
> 
> Folks,
> 
> Since we’ve reached the consensus here I’d like to invite you to review the 
> patch [1] that replaces py.test with testr without making debuging or running 
> specific tests harder. Please also note that it has a dependency which needs 
> to be reviewed and merged first one.
> 
> 1. https://review.openstack.org/#/c/227895
> 
> 
> - romcheg
> 
> 
>> 7 жовт. 2015 р. о 14:41 Roman Prykhodchenko  написав(ла):
>> 
>> Michał,
>> 
>> some comments in-line
>> 
 - testrepository and related components are used in OpenStack Infra
 environment for much more tasks than just running tests
>>> 
>>> If by "more tasks" you mean parallel testing, py.test also has a
>>> possibility to do that by pytest-xdist.
>> 
>> As Monthy mentioned, it’s not only about testing, it’s more about deeper 
>> integration with OpenStack Infra.
>> 
>> 
 - py.test won’t be added to global-requirements so there always be a chance
 of another dependency hell
>>> 
>>> As Igor Kalnitsky said, py.test doesn't have much requirements.
>>> https://github.com/pytest-dev/pytest/blob/master/setup.py#L58
>>> It's only argparse, which already is in global requirements without
>>> any version pinned.
>> 
>> It’s not only about py.test, there is an up-to-date objective of sticking 
>> all requirements to global-requirements because we have big problems because 
>> of that every release.
>> 
>>> 
>>> Cheers,
>>> Michal
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Roman Prykhodchenko]

Folks,

Since we’ve reached the consensus here I’d like to invite you to review the 
patch [1] that replaces py.test with testr without making debuging or running 
specific tests harder. Please also note that it has a dependency which needs to 
be reviewed and merged first one.

1. https://review.openstack.org/#/c/227895


- romcheg


> 7 жовт. 2015 р. о 14:41 Roman Prykhodchenko  написав(ла):
> 
> Michał,
> 
> some comments in-line
> 
>>> - testrepository and related components are used in OpenStack Infra
>>> environment for much more tasks than just running tests
>> 
>> If by "more tasks" you mean parallel testing, py.test also has a
>> possibility to do that by pytest-xdist.
> 
> As Monthy mentioned, it’s not only about testing, it’s more about deeper 
> integration with OpenStack Infra.
> 
> 
>>> - py.test won’t be added to global-requirements so there always be a chance
>>> of another dependency hell
>> 
>> As Igor Kalnitsky said, py.test doesn't have much requirements.
>> https://github.com/pytest-dev/pytest/blob/master/setup.py#L58
>> It's only argparse, which already is in global requirements without
>> any version pinned.
> 
> It’s not only about py.test, there is an up-to-date objective of sticking all 
> requirements to global-requirements because we have big problems because of 
> that every release.
> 
>> 
>> Cheers,
>> Michal
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Roman Prykhodchenko]

Michał,

some comments in-line

>> - testrepository and related components are used in OpenStack Infra
>> environment for much more tasks than just running tests
> 
> If by "more tasks" you mean parallel testing, py.test also has a
> possibility to do that by pytest-xdist.

As Monthy mentioned, it’s not only about testing, it’s more about deeper 
integration with OpenStack Infra.


>> - py.test won’t be added to global-requirements so there always be a chance
>> of another dependency hell
> 
> As Igor Kalnitsky said, py.test doesn't have much requirements.
> https://github.com/pytest-dev/pytest/blob/master/setup.py#L58
> It's only argparse, which already is in global requirements without
> any version pinned.

It’s not only about py.test, there is an up-to-date objective of sticking all 
requirements to global-requirements because we have big problems because of 
that every release.

> 
> Cheers,
> Michal
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Michal Rostecki]

On Wed, Oct 7, 2015 at 12:59 PM, Roman Prykhodchenko  wrote:
> What I can extract now from this thread is that Fuel should switch to testr
> because of the following reasons:
>
> - Diversity of tools is a bad idea on a project scale

We already have diversity about frameworks (or lack of them) in
OpenStack. We have Pecan, Flask, wsgiref, Django.

> - testrepository and related components are used in OpenStack Infra
> environment for much more tasks than just running tests

If by "more tasks" you mean parallel testing, py.test also has a
possibility to do that by pytest-xdist.

> - py.test won’t be added to global-requirements so there always be a chance
> of another dependency hell

As Igor Kalnitsky said, py.test doesn't have much requirements.
https://github.com/pytest-dev/pytest/blob/master/setup.py#L58
It's only argparse, which already is in global requirements without
any version pinned.

> - Sticking to global requirements is an idea which is in the scope of
> discussions around Fuel.
>
> Sounds like that’s the point when we should just file appropriate bugs and
> use testr in smaller components, e. g., Fuel Client, first and then try in
> in Nailgun.
>
>
> - romcheg
>

Cheers,
Michal

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Roman Prykhodchenko]

Yuri,

sticking to global requirements and interacting deeper with OpenStack Infra are 
up-to-date objectives for Fuel and those are pretty much technical question. 
However, software development is not only solving technical tasks, it also 
incorporates interaction between people and other teams so you cannot separate 
those thinks, even if it sounds too much like politics.

- romcheg

> 7 жовт. 2015 р. о 13:20 Yuriy Taraday  написав(ла):
> 
> On Wed, Oct 7, 2015 at 12:51 AM Monty Taylor  > wrote:
> On 10/06/2015 10:52 AM, Sebastian Kalinowski wrote:
> > I've already wrote in the review that caused this thread that I do not want
> > to blindly follow rules for using one or another. We should always consider
> > technical requirements. And I do not see a reason to leave py.test (and
> > nobody
> > show me such reason) and replace it with something else.
> 
> Hi!
> 
> The reason is that testrepository is what OpenStack uses and as I
> understand it, Fuel wants to join the Big Tent.
> 
> It saddens me that once again choice of library is being forced upon a 
> project based on what other projects use, not on technical merit. py.test is 
> more than just a (way better) test runner, it allows to write tests with less 
> boilerplate and more power. While its features are not extensively used in 
> Fuel code, switching to testr would still require changing test logic which 
> is generally bad (that's why mox is still in use in OpenStack). Can we avoid 
> that?
> 
> The use of testr is documented in the Project Testing Interface:
> 
> http://git.openstack.org/cgit/openstack/governance/tree/reference/project-testing-interface.rst#n78
>  
> 
> 
> There are many reasons for it, but in large part we are continually
> adding more and more tools to process subunit output across the board in
> the Gate. subunit2sql is an important one, as it will be feeding into
> expanded test result dashboards.
> 
> We also have zuul features in the pipeline to be able to watch the
> subunit streams in real time to respond more quickly to issues in test runs.
> 
> We also have standard job builders based around tox and testr. Having
> project divergence in this area is a non-starter when there are over 800
> repositories.
> 
> So it seems that all that's needed to keep py.test as an option is a plugin 
> for py.test that generates subunit stream like Robert said, is that right?
> 
> In short, while I understand that this seems like an area where a
> project can do whatever it wants to, it really isn't. If it's causing
> you excessive pain, I recommend connecting with Robert on ways to make
> improvements to testrepository. Those improvements will also have the
> effect of improving life for the rest of OpenStack, which is also a
> great reason why we all use the same tools rather than foster an
> environment of per-project snowflakes.
> 
> I wouldn't call py.test a snowflake. It's a very well-established testing 
> tool and OpenStack projects could benefit from using it if we integrate it 
> with testr well.
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org 
> ?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Yuriy Taraday]

On Wed, Oct 7, 2015 at 3:14 AM Monty Taylor  wrote:

> On 10/06/2015 06:01 PM, Thomas Goirand wrote:
> > On 10/06/2015 01:14 PM, Yuriy Taraday wrote:
> >> On Mon, Oct 5, 2015 at 5:40 PM Roman Prykhodchenko  >> > wrote:
> >>
> >>  Atm I have the following pros. and cons. regarding testrepository:
> >>
> >>  pros.:
> >>
> >>  1. It’s ”standard" in OpenStack so using it gives Fuel more karma
> >>  and moves it more under big tent
> >>
> >>
> >> I don't think that big tent model aims at eliminating diversity of tools
> >> we use in our projects. A collection of web frameworks used in big tent
> >> is an example of that.
> >
> >  From the downstream distro point of view, I don't agree in general, and
> > with the web framework in particular. (though it's less a concern for
> > the testr vs pbr). We keep adding dependencies and duplicates, but never
> > remove them. For example, tablib and suds/sudsjurko need to be removed
> > because they are not maintainable, there's not much work to do so, but
> > nobody does the work...
>
> The Big Tent has absolutely no change in opinion about eliminating
> diversity of tools. OpenStack has ALWAYS striven to reduce diversity of
> tools. Big Tent applies OpenStack to more things that request to be part
> of OpenStack.
>
> Nothing has changed in the intent.
>
> Diversity of tools in a project this size is a bad idea. Always has
> been. Always will be.
>
> The amount of web frameworks in use is a bug.
>

I'm sorry, that was my mistake. I just can't remember any project that was
declined place under big tent (or integrated) because of a library in use.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Thomas Goirand]

On 10/07/2015 02:06 AM, Monty Taylor wrote:
> The Big Tent has absolutely no change in opinion about eliminating
> diversity of tools. OpenStack has ALWAYS striven to reduce diversity of
> tools. Big Tent applies OpenStack to more things that request to be part
> of OpenStack.
> 
> Nothing has changed in the intent.
> 
> Diversity of tools in a project this size is a bad idea. Always has
> been. Always will be.
> 
> The amount of web frameworks in use is a bug.

Thanks a lot Monty. I am very happy that you have this opinion.

In such a case, could Zaqar get rid of Falcon? :)

Thomas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Yuriy Taraday]

On Wed, Oct 7, 2015 at 12:51 AM Monty Taylor  wrote:

> On 10/06/2015 10:52 AM, Sebastian Kalinowski wrote:
> > I've already wrote in the review that caused this thread that I do not
> want
> > to blindly follow rules for using one or another. We should always
> consider
> > technical requirements. And I do not see a reason to leave py.test (and
> > nobody
> > show me such reason) and replace it with something else.
>
> Hi!
>
> The reason is that testrepository is what OpenStack uses and as I
> understand it, Fuel wants to join the Big Tent.
>

It saddens me that once again choice of library is being forced upon a
project based on what other projects use, not on technical merit. py.test
is more than just a (way better) test runner, it allows to write tests with
less boilerplate and more power. While its features are not extensively
used in Fuel code, switching to testr would still require changing test
logic which is generally bad (that's why mox is still in use in OpenStack).
Can we avoid that?

The use of testr is documented in the Project Testing Interface:
>
>
> http://git.openstack.org/cgit/openstack/governance/tree/reference/project-testing-interface.rst#n78
>
> There are many reasons for it, but in large part we are continually
> adding more and more tools to process subunit output across the board in
> the Gate. subunit2sql is an important one, as it will be feeding into
> expanded test result dashboards.
>
> We also have zuul features in the pipeline to be able to watch the
> subunit streams in real time to respond more quickly to issues in test
> runs.
>

We also have standard job builders based around tox and testr. Having
> project divergence in this area is a non-starter when there are over 800
> repositories.
>

So it seems that all that's needed to keep py.test as an option is a plugin
for py.test that generates subunit stream like Robert said, is that right?

In short, while I understand that this seems like an area where a
> project can do whatever it wants to, it really isn't. If it's causing
> you excessive pain, I recommend connecting with Robert on ways to make
> improvements to testrepository. Those improvements will also have the

effect of improving life for the rest of OpenStack, which is also a
> great reason why we all use the same tools rather than foster an
> environment of per-project snowflakes.
>

I wouldn't call py.test a snowflake. It's a very well-established testing
tool and OpenStack projects could benefit from using it if we integrate it
with testr well.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Thomas Herve]

On Wed, Oct 7, 2015 at 12:59 PM, Roman Prykhodchenko  wrote:

> What I can extract now from this thread is that Fuel should switch to
> testr because of the following reasons:
>
> - Diversity of tools is a bad idea on a project scale
> - testrepository and related components are used in OpenStack Infra
> environment for much more tasks than just running tests
> - py.test won’t be added to global-requirements so there always be a
> chance of another dependency hell
> - Sticking to global requirements is an idea which is in the scope of
> discussions around Fuel.
>
> Sounds like that’s the point when we should just file appropriate bugs and
> use testr in smaller components, e. g., Fuel Client, first and then try in
> in Nailgun.
>

I'd say that using testr in the default tox targets and thus in the gate is
the reasonable choice, for the reasons mentioned elsewhere. That said, I
also think that it's also fair to allow alternate test runners on the local
developer environment. You may have to make some tweaks from time to time,
but most of the time py.test should be able to run the test suite supported
by testr (this is not necessarily true for nose for example which doesn't
support testscenarios AFAIK).

This way you standardize on the tools (testr remains the "source of
truth"), but make local debugging much nicer.

-- 
Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Roman Prykhodchenko]

What I can extract now from this thread is that Fuel should switch to testr 
because of the following reasons:

- Diversity of tools is a bad idea on a project scale
- testrepository and related components are used in OpenStack Infra environment 
for much more tasks than just running tests
- py.test won’t be added to global-requirements so there always be a chance of 
another dependency hell
- Sticking to global requirements is an idea which is in the scope of 
discussions around Fuel.

Sounds like that’s the point when we should just file appropriate bugs and use 
testr in smaller components, e. g., Fuel Client, first and then try in in 
Nailgun.


- romcheg

> 7 жовт. 2015 р. о 02:06 Monty Taylor  написав(ла):
> 
> On 10/06/2015 06:01 PM, Thomas Goirand wrote:
>> On 10/06/2015 01:14 PM, Yuriy Taraday wrote:
>>> On Mon, Oct 5, 2015 at 5:40 PM Roman Prykhodchenko >> > wrote:
>>> 
>>> Atm I have the following pros. and cons. regarding testrepository:
>>> 
>>> pros.:
>>> 
>>> 1. It’s ”standard" in OpenStack so using it gives Fuel more karma
>>> and moves it more under big tent
>>> 
>>> 
>>> I don't think that big tent model aims at eliminating diversity of tools
>>> we use in our projects. A collection of web frameworks used in big tent
>>> is an example of that.
>> 
>> From the downstream distro point of view, I don't agree in general, and
>> with the web framework in particular. (though it's less a concern for
>> the testr vs pbr). We keep adding dependencies and duplicates, but never
>> remove them. For example, tablib and suds/sudsjurko need to be removed
>> because they are not maintainable, there's not much work to do so, but
>> nobody does the work...
> 
> The Big Tent has absolutely no change in opinion about eliminating diversity 
> of tools. OpenStack has ALWAYS striven to reduce diversity of tools. Big Tent 
> applies OpenStack to more things that request to be part of OpenStack.
> 
> Nothing has changed in the intent.
> 
> Diversity of tools in a project this size is a bad idea. Always has been. 
> Always will be.
> 
> The amount of web frameworks in use is a bug.
> 
>>> 2. It’s in global requirements, so it doesn’t cause dependency hell
>>> 
>>> That can be solved by adding py.test to openstack/requirements.
> 
> No, it cannot. py.test/testr is not about dependency management. It's about a 
> much bigger picture of how OpenStack does development and how that 
> development can be managed.
> 
>> I'd very much prefer if we could raise the barrier for getting a 3rd
>> party new dependency in. I hope we can talk about this in Tokyo. That
>> being said, indeed, adding py.test isn't so much of a problem, as it is
>> widely used, already packaged, and maintained upstream. I'd still prefer
>> if all projects were using the same testing framework and test runner
>> though.
> 
> As I said earlier in this thread, it has already been decided by the TC long 
> ago that we will use testr. Barring a (very unlikely) TC rescinding of that 
> decision, OpenStack projects use testr. There is zero value in expanding the 
> number of test runners.
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org 
> ?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Monty Taylor]

On 10/06/2015 06:01 PM, Thomas Goirand wrote:

On 10/06/2015 01:14 PM, Yuriy Taraday wrote:

On Mon, Oct 5, 2015 at 5:40 PM Roman Prykhodchenko mailto:m...@romcheg.me>> wrote:

 Atm I have the following pros. and cons. regarding testrepository:

 pros.:

 1. It’s ”standard" in OpenStack so using it gives Fuel more karma
 and moves it more under big tent


I don't think that big tent model aims at eliminating diversity of tools
we use in our projects. A collection of web frameworks used in big tent
is an example of that.


 From the downstream distro point of view, I don't agree in general, and
with the web framework in particular. (though it's less a concern for
the testr vs pbr). We keep adding dependencies and duplicates, but never
remove them. For example, tablib and suds/sudsjurko need to be removed
because they are not maintainable, there's not much work to do so, but
nobody does the work...


The Big Tent has absolutely no change in opinion about eliminating 
diversity of tools. OpenStack has ALWAYS striven to reduce diversity of 
tools. Big Tent applies OpenStack to more things that request to be part 
of OpenStack.


Nothing has changed in the intent.

Diversity of tools in a project this size is a bad idea. Always has 
been. Always will be.


The amount of web frameworks in use is a bug.


 2. It’s in global requirements, so it doesn’t cause dependency hell

That can be solved by adding py.test to openstack/requirements.


No, it cannot. py.test/testr is not about dependency management. It's 
about a much bigger picture of how OpenStack does development and how 
that development can be managed.



I'd very much prefer if we could raise the barrier for getting a 3rd
party new dependency in. I hope we can talk about this in Tokyo. That
being said, indeed, adding py.test isn't so much of a problem, as it is
widely used, already packaged, and maintained upstream. I'd still prefer
if all projects were using the same testing framework and test runner
though.


As I said earlier in this thread, it has already been decided by the TC 
long ago that we will use testr. Barring a (very unlikely) TC rescinding 
of that decision, OpenStack projects use testr. There is zero value in 
expanding the number of test runners.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Thomas Goirand]

On 10/06/2015 01:14 PM, Yuriy Taraday wrote:
> On Mon, Oct 5, 2015 at 5:40 PM Roman Prykhodchenko  > wrote:
> 
> Atm I have the following pros. and cons. regarding testrepository:
> 
> pros.:
> 
> 1. It’s ”standard" in OpenStack so using it gives Fuel more karma
> and moves it more under big tent
> 
> 
> I don't think that big tent model aims at eliminating diversity of tools
> we use in our projects. A collection of web frameworks used in big tent
> is an example of that.

>From the downstream distro point of view, I don't agree in general, and
with the web framework in particular. (though it's less a concern for
the testr vs pbr). We keep adding dependencies and duplicates, but never
remove them. For example, tablib and suds/sudsjurko need to be removed
because they are not maintainable, there's not much work to do so, but
nobody does the work...

> 2. It’s in global requirements, so it doesn’t cause dependency hell
> 
> That can be solved by adding py.test to openstack/requirements.

I'd very much prefer if we could raise the barrier for getting a 3rd
party new dependency in. I hope we can talk about this in Tokyo. That
being said, indeed, adding py.test isn't so much of a problem, as it is
widely used, already packaged, and maintained upstream. I'd still prefer
if all projects were using the same testing framework and test runner
though.

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Davanum Srinivas]

Sebastian,

I am really hoping that the items Monty listed below are enough. Also, if
you are interested, folks do use other things for running their tests
especially to find problems when testr hides some errors. Example, please
see:

https://davanum.wordpress.com/2015/01/13/quickly-running-a-single-openstack-nova-test/

Thanks,
Dims



On Tue, Oct 6, 2015 at 5:47 PM, Monty Taylor  wrote:

> On 10/06/2015 10:52 AM, Sebastian Kalinowski wrote:
>
>> I've already wrote in the review that caused this thread that I do not
>> want
>> to blindly follow rules for using one or another. We should always
>> consider
>> technical requirements. And I do not see a reason to leave py.test (and
>> nobody
>> show me such reason) and replace it with something else.
>>
>
> Hi!
>
> The reason is that testrepository is what OpenStack uses and as I
> understand it, Fuel wants to join the Big Tent.
>
> The use of testr is documented in the Project Testing Interface:
>
>
> http://git.openstack.org/cgit/openstack/governance/tree/reference/project-testing-interface.rst#n78
>
> There are many reasons for it, but in large part we are continually adding
> more and more tools to process subunit output across the board in the Gate.
> subunit2sql is an important one, as it will be feeding into expanded test
> result dashboards.
>
> We also have zuul features in the pipeline to be able to watch the subunit
> streams in real time to respond more quickly to issues in test runs.
>
> We also have standard job builders based around tox and testr. Having
> project divergence in this area is a non-starter when there are over 800
> repositories.
>
> In short, while I understand that this seems like an area where a project
> can do whatever it wants to, it really isn't. If it's causing you excessive
> pain, I recommend connecting with Robert on ways to make improvements to
> testrepository. Those improvements will also have the effect of improving
> life for the rest of OpenStack, which is also a great reason why we all use
> the same tools rather than foster an environment of per-project snowflakes.
>
> Additionally other folks showed that this is not a blocker for moving
>> under big tent.
>>
>
> I apologize for any confusion that may have resulted from you being given
> erroneous information.
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Monty Taylor]

On 10/06/2015 10:52 AM, Sebastian Kalinowski wrote:

I've already wrote in the review that caused this thread that I do not want
to blindly follow rules for using one or another. We should always consider
technical requirements. And I do not see a reason to leave py.test (and
nobody
show me such reason) and replace it with something else.


Hi!

The reason is that testrepository is what OpenStack uses and as I 
understand it, Fuel wants to join the Big Tent.


The use of testr is documented in the Project Testing Interface:

http://git.openstack.org/cgit/openstack/governance/tree/reference/project-testing-interface.rst#n78

There are many reasons for it, but in large part we are continually 
adding more and more tools to process subunit output across the board in 
the Gate. subunit2sql is an important one, as it will be feeding into 
expanded test result dashboards.


We also have zuul features in the pipeline to be able to watch the 
subunit streams in real time to respond more quickly to issues in test runs.


We also have standard job builders based around tox and testr. Having 
project divergence in this area is a non-starter when there are over 800 
repositories.


In short, while I understand that this seems like an area where a 
project can do whatever it wants to, it really isn't. If it's causing 
you excessive pain, I recommend connecting with Robert on ways to make 
improvements to testrepository. Those improvements will also have the 
effect of improving life for the rest of OpenStack, which is also a 
great reason why we all use the same tools rather than foster an 
environment of per-project snowflakes.



Additionally other folks showed that this is not a blocker for moving
under big tent.


I apologize for any confusion that may have resulted from you being 
given erroneous information.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Sebastian Kalinowski]

I've already wrote in the review that caused this thread that I do not want
to blindly follow rules for using one or another. We should always consider
technical requirements. And I do not see a reason to leave py.test (and
nobody
show me such reason) and replace it with something else.

Additionally other folks showed that this is not a blocker for moving under
big tent.

Best,
Sebastian
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Igor Kalnitsky]

Hey Roman,

> It’s ”standard" in OpenStack so using it gives Fuel more karma
> and moves it more under big tent

As far as I understand it doesn't affect our movement under big tent.

> It’s in global requirements, so it doesn’t cause dependency hell

Honestly I have no idea how py.test caused a dependency hell. It
almost doesn't have any dependencies:

* py
* colorama for windows
* argparse for python 2.6 and 3.0

So the only possible intersection with global-requirements may is a
'py' package. Well, I checked and I didn't find 'py' package in there.

Summarizing, let's be practical - using py.test is convenient. It has
beautiful reports out-of-box, could be extended with a lot of
available plugins and so on. I'd prefer to keep it.

Thanks,
Igor

On Tue, Oct 6, 2015 at 2:14 PM, Yuriy Taraday  wrote:
> On Mon, Oct 5, 2015 at 5:40 PM Roman Prykhodchenko  wrote:
>>
>> Atm I have the following pros. and cons. regarding testrepository:
>>
>> pros.:
>>
>> 1. It’s ”standard" in OpenStack so using it gives Fuel more karma and
>> moves it more under big tent
>
>
> I don't think that big tent model aims at eliminating diversity of tools we
> use in our projects. A collection of web frameworks used in big tent is an
> example of that.
>
>> 2. It’s in global requirements, so it doesn’t cause dependency hell
>
>
> That can be solved by adding py.test to openstack/requirements.
>
>> cons.:
>> 1. Debugging is really hard
>
>
> I'd say that debugging here is not the right term. Every aspect of
> developing with testr is harder than with py.test. py.test tends to just
> work where you need additional tools and effort with testr.
>
> In general I don't see any benefit the project can get from using testr
> while its limitations will bite developers at every turn.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Yuriy Taraday]

On Mon, Oct 5, 2015 at 5:40 PM Roman Prykhodchenko  wrote:

> Atm I have the following pros. and cons. regarding testrepository:
>
> pros.:
>
> 1. It’s ”standard" in OpenStack so using it gives Fuel more karma and
> moves it more under big tent
>

I don't think that big tent model aims at eliminating diversity of tools we
use in our projects. A collection of web frameworks used in big tent is an
example of that.

2. It’s in global requirements, so it doesn’t cause dependency hell
>

That can be solved by adding py.test to openstack/requirements.

cons.:
> 1. Debugging is really hard
>

I'd say that debugging here is not the right term. Every aspect of
developing with testr is harder than with py.test. py.test tends to just
work where you need additional tools and effort with testr.

In general I don't see any benefit the project can get from using testr
while its limitations will bite developers at every turn.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Robert Collins]

On 6 October 2015 at 03:34, Roman Prykhodchenko  wrote:
> Disclaimer:
> I didn’t want to fire up this war but it silently hit one of my patches so 
> now I think it’s better to spread it to a wide audience.
>
>
> When I was dealing with one of the regular dependency hell in Fuel Client I 
> noticed, that stuff which is not in global requirements may make the 
> mentioned hell hotter, even if those requirements are test requirements. 
> After that discovery I started aligning all *requirements.txt to Kilo’s 
> global requirements and trying to remove everything which it not there. A 
> special dependency is of course py.test: replacing it is a very controversial 
> thing which I’d like to discuss here.
>
> Atm I have the following pros. and cons. regarding testrepository:
>
> pros.:
>
> 1. It’s ”standard" in OpenStack so using it gives Fuel more karma and moves 
> it more under big tent
> 2. It’s in global requirements, so it doesn’t cause dependency hell

I'd also add that it enables the big data approach to test analysis
that the subunit2sql and test dashboard folk are building up, and
thats pretty important. A subunit py.test plugin would enable that for
py.test test authors (and in fact that would enable testing via
py.test under testrepository - test repository is *not* a test runner
- its a data-store + a meta-runner).

> cons.:
>
> 1. Debugging is really hard

It is really hard indeed, and I'd like love to fix that. In fact I'd
like to make it as nice as it is in py.test - I have a plan, and I'd
be delighted to turn it into a spec that someone can follow if anyone
would like to fix this - I think it would be quite high leverage a
thing. OTOH, because its an exercise in distributed systems
programming, its not truely trivial.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] py.test vs testrepository

[Roman Prykhodchenko]

Disclaimer:
I didn’t want to fire up this war but it silently hit one of my patches so now 
I think it’s better to spread it to a wide audience.


When I was dealing with one of the regular dependency hell in Fuel Client I 
noticed, that stuff which is not in global requirements may make the mentioned 
hell hotter, even if those requirements are test requirements. After that 
discovery I started aligning all *requirements.txt to Kilo’s global 
requirements and trying to remove everything which it not there. A special 
dependency is of course py.test: replacing it is a very controversial thing 
which I’d like to discuss here.

Atm I have the following pros. and cons. regarding testrepository:

pros.:

1. It’s ”standard" in OpenStack so using it gives Fuel more karma and moves it 
more under big tent
2. It’s in global requirements, so it doesn’t cause dependency hell

cons.:

1. Debugging is really hard


I’d like to head your thoughts.


- romcheg



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev