Re: [Python-ideas] Official site-packages/test directory

[Stephane Wirtel via Python-ideas]

Hi Barry,

Sometimes, I need to read the tests of a package because I don't
understand the usage of a function/method/class and unfortunately, there
is no documentation. In this case, and only in this case, I will try to
find the tests and in the worst case, download the source and try to
understand with the 'tests' directory.

my 0.02 € ;-)

Stephane


On 01/26, Barry Warsaw wrote:

Guido van Rossum wrote:

IIUC another common layout is to have folders named test or tests inside
each package. This would avoid requiring any changes to the site-packages
layout.


That's what I do for all my personal code.  Yes, it means the test
directories are shipped with the sdist, but really who cares?  I don't
think I've had a single complaint about it, even with large-ish projects
like Mailman.  I can see you wanting to do something different if your
project has truly gargantuan test suites, but even with 100% coverage
(or nearly so), I think size just isn't usually a big deal.

In another message, Giampaolo describes being able to run tests with -m
psutil.test.  That's a neat idea which I haven't tried.  But I do think
including the tests can be instructive, and I know that on more than one
occasion, I've cracked open a project's test suite to get a better sense
of the semantics and usage of a particular API.

Finally, I'll disagree with pytest's recommendation to not put
__init__.py files in your test directories.  Although I'm not a heavy
pytest user (we use it exclusive at work, but I don't use it much with
my own stuff), having __init__.py files can be useful, especially if you
also have test data you want to access through pkg_resources, or now,
importlib_resources (importlib.resources in Python 3.7).

Cheers,
-Barry


--
Stéphane Wirtel - http://wirtel.be - @matrixise
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Barry Warsaw]

Guido van Rossum wrote:
> IIUC another common layout is to have folders named test or tests inside
> each package. This would avoid requiring any changes to the site-packages
> layout.

That's what I do for all my personal code.  Yes, it means the test
directories are shipped with the sdist, but really who cares?  I don't
think I've had a single complaint about it, even with large-ish projects
like Mailman.  I can see you wanting to do something different if your
project has truly gargantuan test suites, but even with 100% coverage
(or nearly so), I think size just isn't usually a big deal.

In another message, Giampaolo describes being able to run tests with -m
psutil.test.  That's a neat idea which I haven't tried.  But I do think
including the tests can be instructive, and I know that on more than one
occasion, I've cracked open a project's test suite to get a better sense
of the semantics and usage of a particular API.

Finally, I'll disagree with pytest's recommendation to not put
__init__.py files in your test directories.  Although I'm not a heavy
pytest user (we use it exclusive at work, but I don't use it much with
my own stuff), having __init__.py files can be useful, especially if you
also have test data you want to access through pkg_resources, or now,
importlib_resources (importlib.resources in Python 3.7).

Cheers,
-Barry


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Nick Coghlan]

On 20 January 2018 at 04:39, Chris Barker  wrote:
> So maybe the way to go is to come up with recommendations for a standard way
> to do it -- maybe published by PyPa?

I don't think the trade-offs here are clear enough for us to add an
opinionated guide to packaging.python.org, but it could be an
appropriate topic for a discussion page (e.g. "Publishing Test
Suites") that frames the problem, and lays out some of the options for
handling it:

- tests published as part of the package (generally not ideal, but can
make post-install testing easier)
- tests in the sdist only (most suitable for unit tests)
- tests published as a separate package (often suitable for integration tests)

One of the things that makes the 3rd option a bit awkward currently is
that a lot of tools assume "1 repo -> 1 sdist", so splitting your test
suite out to its own sdist can be fairly annoying in practice.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Chris Barker]

On Fri, Jan 19, 2018 at 11:21 AM, Giampaolo Rodola' 
wrote:
>
> I personally include them in psutil distribution so that users can test
> the installation with "python -m psutil.test". I have even this documented
> as I think it's an added value.
>

or:

pytest --pyargs pkg_name

It Is really handy, and sometimes required to test the distribution /
installation itself.

So I do that most  of the time these days -- but it gets ugly if the tests
get really huge.

-CHB





>
>
>
> --
> Giampaolo - http://grodola.blogspot.com
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Giampaolo Rodola']

On Fri, Jan 19, 2018 at 5:23 PM, Paul Moore  wrote:

> Another common approach is to not ship tests as part of your (runtime)
> package at all - they are in the sdist but not the wheels nor are they
> deployed with "setup.py install". In my experience, this is the usual
> approach projects take if they don't have the tests in the package
> directory. (I don't think I've *ever* seen a project try to install
> tests except by including them in the package directory...)


I personally include them in psutil distribution so that users can test the
installation with "python -m psutil.test". I have even this documented as I
think it's an added value.


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Chris Barker]

hmm,

I've struggled for ages with this problem -- I have some packages with
REALLY big test suites. so I don't put the tests in the package.

But their are also numerous issues with building and installing the package
(C code, lots of dependencies, etc), so it would be really nice to have a
way to test the actual installed package after the fact (and, be able to
properly test conda packages as well -- easy if the tests are in the
package, hard if not...)

So I like the idea of having a standard way / place to install tests.

However, somehow I never thought to make a my_package_tests package --
d'uh! seems the obvious way to handle teh "optionally install the tests"
problem.

I still like the idea of separate location, but Paul is right that it's a
change that would have to filter out through a lot of infrastructure, so
maybe not practical.

So maybe the way to go is to come up with recommendations for a standard
way to do it -- maybe published by PyPa?

-CHB




On Fri, Jan 19, 2018 at 10:19 AM, Stefan Krah  wrote:

> On Fri, Jan 19, 2018 at 05:30:43PM +, Paul Moore wrote:
> [cut]
> > I'd think that the idea of a site-packages/stest directory would need
> > a much more compelling use case to justify it.
>
> Thanks for the detailed explanation!  It sounds that there's much more work
> involved than I thought, so it's probably better to drop this proposal.
>
>
> > PS There's nothing stopping a (distribution) package FOO from
> > installing (Python) packages foo and foo-tests. It's not common, and
> > probably violates people's expectations, but it's not *illegal* (the
> > setuptools distribution installs pkg_resources as well as setuptools,
> > for a well-known example). So in theory, if people wanted this enough,
> > they could have implemented it right now, without needing any change
> > to Python or the packaging ecosystem.
>
> If people don't come with pitchforks, that's a good solution. I suspected
> that people would complain both if foo-tests were installed automatically
> like pkg_resources but also if foo-tests were a separate optional package
> (too much hassle).
>
>
>
> Stefan Krah
>
>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Paul Moore]

On 19 January 2018 at 18:19, Stefan Krah  wrote:
> On Fri, Jan 19, 2018 at 05:30:43PM +, Paul Moore wrote:
> [cut]
>> I'd think that the idea of a site-packages/stest directory would need
>> a much more compelling use case to justify it.
>
> Thanks for the detailed explanation!  It sounds that there's much more work
> involved than I thought, so it's probably better to drop this proposal.
>
>
>> PS There's nothing stopping a (distribution) package FOO from
>> installing (Python) packages foo and foo-tests. It's not common, and
>> probably violates people's expectations, but it's not *illegal* (the
>> setuptools distribution installs pkg_resources as well as setuptools,
>> for a well-known example). So in theory, if people wanted this enough,
>> they could have implemented it right now, without needing any change
>> to Python or the packaging ecosystem.
>
> If people don't come with pitchforks, that's a good solution. I suspected
> that people would complain both if foo-tests were installed automatically
> like pkg_resources but also if foo-tests were a separate optional package
> (too much hassle).

Personally, I prefer packages that don't install their tests (I'm just
about willing to tolerate the tests-inside-the package-approach) so I
actually dislike this option myself - I was just saying it's possible.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Stefan Krah]

On Fri, Jan 19, 2018 at 05:30:43PM +, Paul Moore wrote:
[cut]
> I'd think that the idea of a site-packages/stest directory would need
> a much more compelling use case to justify it.

Thanks for the detailed explanation!  It sounds that there's much more work
involved than I thought, so it's probably better to drop this proposal.


> PS There's nothing stopping a (distribution) package FOO from
> installing (Python) packages foo and foo-tests. It's not common, and
> probably violates people's expectations, but it's not *illegal* (the
> setuptools distribution installs pkg_resources as well as setuptools,
> for a well-known example). So in theory, if people wanted this enough,
> they could have implemented it right now, without needing any change
> to Python or the packaging ecosystem.

If people don't come with pitchforks, that's a good solution. I suspected
that people would complain both if foo-tests were installed automatically
like pkg_resources but also if foo-tests were a separate optional package
(too much hassle).



Stefan Krah




___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Petr Viktorin]

FWIW, I've had very good experience with putting tests for package `foo` 
in a directory/package called `test_foo`.


This combines the best of both worlds -- it can be easily separated for 
distribution (like `tests`), and it doesn't cause name conflicts (like 
`foo.tests`).



On 01/19/2018 05:23 PM, Paul Moore wrote:

Another common approach is to not ship tests as part of your (runtime)
package at all - they are in the sdist but not the wheels nor are they
deployed with "setup.py install". In my experience, this is the usual
approach projects take if they don't have the tests in the package
directory. (I don't think I've *ever* seen a project try to install
tests except by including them in the package directory...)

Paul

On 19 January 2018 at 16:10, Guido van Rossum  wrote:

IIUC another common layout is to have folders named test or tests inside
each package. This would avoid requiring any changes to the site-packages
layout.

On Fri, Jan 19, 2018 at 6:27 AM, Stefan Krah  wrote:



Hello,

I wonder if we could get an official site-packages/test directory.
Currently
it seems to be problematic to distribute tests if they are outside the
package
directory.  Here is a nice overview of the two main layout possibilities:


http://pytest.readthedocs.io/en/reorganize-docs/new-docs/user/directory_structure.html


I like the outside-the-package approach, mostly for reasons described very
eloquently here:


http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html


CPython itself of course also uses Lib/foo.py and Lib/test/test_foo.py, so
it
would make sense to have site-packages/foo.py and
site-packages/test/test_foo.py.

For me, this is the natural layout.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Paul Moore]

On 19 January 2018 at 17:08, Stefan Krah  wrote:
> On Fri, Jan 19, 2018 at 04:23:23PM +, Paul Moore wrote:
>> Another common approach is to not ship tests as part of your (runtime)
>> package at all - they are in the sdist but not the wheels nor are they
>> deployed with "setup.py install". In my experience, this is the usual
>> approach projects take if they don't have the tests in the package
>> directory. (I don't think I've *ever* seen a project try to install
>> tests except by including them in the package directory...)
>
> Yes, given the current situation not shipping is definitely the best
> approach in that case.
>
> I just thought that if we did have something like site-packages/stest
> (Guido correctly noted that "test" wouldn't work), people might use it.
>
>
> But it is all very speculative and I'm not really sure myself.

To be usable, tools like pip, wheel, setuptools, flit, etc, would all
need to be updated to take into account this option, as well as the
relevant standards (the wheel spec for one). Add to that the changes
needed to places like the sysconfig package to allow introspecting the
location of the new test directory. Would there be a test directory in
user-site as well? What about in virtual environments? (If only in
site-packages, then it'll likely be read-only in a lot of
environments). Also, would we need to reserve the directory name
chosen to prohibit 3rd party packages using it? As we've seen the
stdlib test package clashes with the original proposal, who's to say
there's nothing on PyPI that uses stest?

The idea isn't a bad one in principle - there's a proposal from some
time back on distutils-sig that Python packaging support more "target
locations" matching the POSIX style locations - for docs, config, etc.
A test directory would fit in with this idea. But it's a pretty big
change in practice, and no-one has yet done much beyond talk about it.
And the proposal would likely have put the test directory *outside*
site-packages, which avoids the name clash problem.

I'd think that the idea of a site-packages/stest directory would need
a much more compelling use case to justify it.

Paul

PS There's nothing stopping a (distribution) package FOO from
installing (Python) packages foo and foo-tests. It's not common, and
probably violates people's expectations, but it's not *illegal* (the
setuptools distribution installs pkg_resources as well as setuptools,
for a well-known example). So in theory, if people wanted this enough,
they could have implemented it right now, without needing any change
to Python or the packaging ecosystem.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Wolfgang Maier]

On 01/19/2018 05:48 PM, Guido van Rossum wrote:
On Fri, Jan 19, 2018 at 8:30 AM, Wolfgang Maier 
> wrote:



I think that's a really nice idea.
With an official site-packages/test directory there could be pip
support for optionally installing tests alongside a package if its
layout allows it. So end users could just install things without
tests, but developers could do: pip install  --with-tests
or something to get everything?


Oh, I just realized there's another problem here. The existing 'test' 
package (which is not a namespace package) would hide the 
site-packages/test directory.




Well, that shouldn't be a big obstacle since one could just as well 
choose another name ( __tests__ for example?).
Alternatively, package-specific test directories could exist *inside* 
site-packages. So much like today's .dist-info directories 
there could be .test dirs?

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Stefan Krah]

On Fri, Jan 19, 2018 at 04:23:23PM +, Paul Moore wrote:
> Another common approach is to not ship tests as part of your (runtime)
> package at all - they are in the sdist but not the wheels nor are they
> deployed with "setup.py install". In my experience, this is the usual
> approach projects take if they don't have the tests in the package
> directory. (I don't think I've *ever* seen a project try to install
> tests except by including them in the package directory...)

Yes, given the current situation not shipping is definitely the best
approach in that case.

I just thought that if we did have something like site-packages/stest
(Guido correctly noted that "test" wouldn't work), people might use it.


But it is all very speculative and I'm not really sure myself.



Stefan Krah



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Guido van Rossum]

On Fri, Jan 19, 2018 at 8:30 AM, Wolfgang Maier <
wolfgang.ma...@biologie.uni-freiburg.de> wrote:

>
> I think that's a really nice idea.
> With an official site-packages/test directory there could be pip support
> for optionally installing tests alongside a package if its layout allows
> it. So end users could just install things without tests, but developers
> could do: pip install  --with-tests or something to get everything?


Oh, I just realized there's another problem here. The existing 'test'
package (which is not a namespace package) would hide the
site-packages/test directory.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Wolfgang Maier]

On 01/19/2018 03:27 PM, Stefan Krah wrote:


Hello,

I wonder if we could get an official site-packages/test directory.  Currently
it seems to be problematic to distribute tests if they are outside the package
directory.  Here is a nice overview of the two main layout possibilities:

http://pytest.readthedocs.io/en/reorganize-docs/new-docs/user/directory_structure.html


I like the outside-the-package approach, mostly for reasons described very
eloquently here:

http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html


CPython itself of course also uses Lib/foo.py and Lib/test/test_foo.py, so it
would make sense to have site-packages/foo.py and 
site-packages/test/test_foo.py.

For me, this is the natural layout.



I think that's a really nice idea.
With an official site-packages/test directory there could be pip support 
for optionally installing tests alongside a package if its layout allows 
it. So end users could just install things without tests, but developers 
could do: pip install  --with-tests or something to get everything?


Wolfgang

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Paul Moore]

Another common approach is to not ship tests as part of your (runtime)
package at all - they are in the sdist but not the wheels nor are they
deployed with "setup.py install". In my experience, this is the usual
approach projects take if they don't have the tests in the package
directory. (I don't think I've *ever* seen a project try to install
tests except by including them in the package directory...)

Paul

On 19 January 2018 at 16:10, Guido van Rossum  wrote:
> IIUC another common layout is to have folders named test or tests inside
> each package. This would avoid requiring any changes to the site-packages
> layout.
>
> On Fri, Jan 19, 2018 at 6:27 AM, Stefan Krah  wrote:
>>
>>
>> Hello,
>>
>> I wonder if we could get an official site-packages/test directory.
>> Currently
>> it seems to be problematic to distribute tests if they are outside the
>> package
>> directory.  Here is a nice overview of the two main layout possibilities:
>>
>>
>> http://pytest.readthedocs.io/en/reorganize-docs/new-docs/user/directory_structure.html
>>
>>
>> I like the outside-the-package approach, mostly for reasons described very
>> eloquently here:
>>
>>
>> http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html
>>
>>
>> CPython itself of course also uses Lib/foo.py and Lib/test/test_foo.py, so
>> it
>> would make sense to have site-packages/foo.py and
>> site-packages/test/test_foo.py.
>>
>> For me, this is the natural layout.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Guido van Rossum]

IIUC another common layout is to have folders named test or tests inside
each package. This would avoid requiring any changes to the site-packages
layout.

On Fri, Jan 19, 2018 at 6:27 AM, Stefan Krah  wrote:

>
> Hello,
>
> I wonder if we could get an official site-packages/test directory.
> Currently
> it seems to be problematic to distribute tests if they are outside the
> package
> directory.  Here is a nice overview of the two main layout possibilities:
>
> http://pytest.readthedocs.io/en/reorganize-docs/new-docs/
> user/directory_structure.html
>
>
> I like the outside-the-package approach, mostly for reasons described very
> eloquently here:
>
> http://python-notes.curiousefficiency.org/en/
> latest/python_concepts/import_traps.html
>
>
> CPython itself of course also uses Lib/foo.py and Lib/test/test_foo.py, so
> it
> would make sense to have site-packages/foo.py and
> site-packages/test/test_foo.py.
>
> For me, this is the natural layout.
>
>
>
> Stefan Krah
>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Official site-packages/test directory

[Stefan Krah]

Hello,

I wonder if we could get an official site-packages/test directory.  Currently
it seems to be problematic to distribute tests if they are outside the package
directory.  Here is a nice overview of the two main layout possibilities:

http://pytest.readthedocs.io/en/reorganize-docs/new-docs/user/directory_structure.html


I like the outside-the-package approach, mostly for reasons described very
eloquently here:

http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html


CPython itself of course also uses Lib/foo.py and Lib/test/test_foo.py, so it
would make sense to have site-packages/foo.py and 
site-packages/test/test_foo.py.

For me, this is the natural layout.



Stefan Krah



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/