[Distutils] Uploading stdeb built debs ?

2015-10-09 Thread Stuart Axon via Distutils-SIG 
Hi,    I built a package of using stdeb, any idea if these are acceptable to 
debian + ubuntu, and where to start / who to contact to get a package included 
in those distros ? S++___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

2015-10-09 Thread Michael Merickel 
At the pylons project we've had a history of keeping our tests inside the
packages. However, keeping them outside has proven to be nicer in some
projects as well. 1) It reduces the size of the binary wheels that do not
need to package the tests. 2) It still allows you to run the tests on an
arbitrary installation if you want to by pulling down the repo and running
them against the version installed in the virtualenv. Distributing the
tests to every single installation is definitely not a requirement in order
to be able to run them for people who want to run them, and decoupling them
can really help with that. Projects that are shipping tests inside the
package may have them removed upstream and then it is very difficult to run
the suite against some binary distribution.

On Tue, Oct 6, 2015 at 6:14 PM, David Wilson 
wrote:

> On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote:
>
> > They should be inside the module. That way, you can check an installed
> > module is ok by running e.g. "python -m mypackage.tests". Any other
> > choice makes testing installed modules more cumbersome.
>
> As Donald mentioned, this doesn't work in the general case since many
> packages ship quite substantial test data around that often doesn't end
> up installed, and in other cases since the package requires significant
> fixture setup or external resources (e.g. running SQLAlchemy tests
> without a working database server would be meaningless).
>
> The option of always shipping test data as a standard part of a package
> in a vein attempt to always ensure it can be tested (which is not always
> likely given the SQLAlchemy example above) strikes me as incredibly
> wasteful, not from some oh-precious-bytes standpoint, but from the
> perspective of distributing a Python application of any size where the
> effects of always shipping half-configured test suites has increased the
> resulting distribution size potentially by 3 or 4x.
>
> https://github.com/bennoleslie/pexif is the first hit on Google for a
> module I thought would need some test data. It's actually quite
> minimally tested, yet already the tests + data are 3.6x the size of the
> module itself.
>
>
> I appreciate arguments for inlining tests alongside a package in order
> to allow reuse of the suite's functionality by consuming applications'
> test suites, but as above, in the general case this simply isn't
> something that will always work and can be relied on by default.
>
>
> Is there perhaps a third option that was absent from the original post?
> e.g. organizing tests in a separate, optional, potentially
> pip-installable package.
>
>
> >
> > > outside the module like this:
> > >
> > >  https://github.com/pypa/sampleproject/tree/master/tests
> >
> > There is no actual reason to do that except win a couple kilobytes if
> > you are distributing your package on floppy disks for consumption on
> > Z80-based machines with 64KB RAM.
> >
> > Even Python *itself* puts its test suite inside the standard library,
> > not outside it (though some Linux distros may strip it away).
> > Try "python -m test.regrtest" (again, this may fail if your distro
> > decided to ship the test suite in a separate package).
> >
> > The PyP"A" should definitely fix its sample project to reflect good
> > practices.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] tests location: Use case: new comers and docs.

2015-10-09 Thread Erik Bray 
On Wed, Oct 7, 2015 at 2:11 AM, Chris Barker  wrote:
> On Tue, Oct 6, 2015 at 10:38 PM, Thomas Güttler
>  wrote:
>>
>> Yes, there is not generic "one right way here".
>>
>> Yes, let's consider individual use cases.
>>
>> My use case are the docs for new comers:
>>
>>  - https://github.com/pypa/sampleproject
>>  - https://packaging.python.org/en/latest/distributing/
>>
>> That's why started the thread.
>
>
> unfortunately, that isn't  a use-case -- every newcomer has a different use
> case.

Indeed--I've helped newcomers whose very first attempt at packaging
Python code includes Cython code for simulations.  I think even for
newcomers what we should be providing is not a "this is the way to do
it" because then they get confused when that way doesn't work for
them.

Better, in the long term (and I'm happy to contribute to such an
effort if it will help) is to provide a sort of Choose Your Own
Adventure story.  It can't all go on one page because that would be a
mess, but a sort of "If you need to do this, read this.  If you need
to do this, read this.  Now if you need to include some data files
that are installed in your package read on, because there's really
only one right way to do that.  But now you have some options if you
want to include tests: ..."

> I was happy to see this thread, because I thought maybe I"d learn what i
> should teach my students - new to python.
>
> But alas - there clearly really is no consensus.
>
> What i've told newbies in the past is somethig like:
>
> """
> if you want your user to be able to install you package, and then run
> something like:
>
> import my_package
> my_package.test()
>
> then put your tests inside the package.
>
> If you are fine with only being able to run the tests from the source tree
> -- then put your tests outside the package.
> """
>
> but really, newbies have no idea how to make this decsion.
>
> Maybe we could come up with a decision tree for this -- some guidance for
> knowing what to do, when?

Exactly.  I think it could even be fun :)

How could we get started to add something like this to the packaging docs?

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Chris Barker 
On Fri, Oct 9, 2015 at 11:44 AM, Oscar Benjamin 
wrote:

> Assuming it's tied to the operating system it doesn't matter surely. When
> pip runs on Windows it can ignore dependencies that apply to other
> platforms so I don't see how this case makes it more complex.
>
does pip currently support platfrom-specific dependencies? That would solve
part of the problem.

but yes, there are dependencies that are a function of how the wheel was
built, beyond what platform it is on.

So those could be static in binary wheel, but not static in the sdist (or
source wheel, or...)

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Chris Barker 
On Fri, Oct 9, 2015 at 11:28 AM, Oscar Benjamin 
wrote:

> Why would it need dynamic metadata for the windows matplotlib wheel to
> have different metadata from the OSX matplotlib wheel? The platform
> Windows/OSX is static and each wheel declares its own dependencies
> statically but differently. Am I missing something?
>
I think the metadata can be static for the binary wheels, yes. but the
dependencies would be different for the two wheels.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Chris Barker 
On Fri, Oct 9, 2015 at 10:28 AM, Paul Moore  wrote:

> ... or thinking again, maybe you mean having multiple packages
> (numpy_mkl, numpy_openblas, ...) all of which satisfy a "numpy"
> requirement?


yes, that is EXACTLY what I meant.

the idea here is that if you build a package that requires both numpy with
a particular BLAS, then you'd do: eg:

numpy|mkl >= 1.9

but if you build one that only required numpy, and doesn't care which BLAS
it's using, then you simply do:

numpy >= 1.9

So pip, when asked to install said package, would look and see if any
package called numpy|* was installed (the ther right version). if so, it
would move along. If not, then it would go look on PyPi for "numpy" -- and
this is where it gets tricky -- which numpy| should it use? At this stage,
it wouldn't matter, anyone would do.

But say it installed numpy|mkl. Then next the user goes to install a
package that depends on numpy|openblas -- now pip goes and looks, and  find
numpy|mkl, but not numpy|openblas -- so it need to go install
numpy|openblas. which overwrites numpy|mkl, which is still OK. Until the
user goes to install somethign that DOES depend on numpy|mkl. No we are
stuck.

But is this any different that two packages that depend on two different
specific versions of the same package?

But all this is making me think that the way this could be handled is by
numpy NOT building a BLAS into the package at all. But rather, having the
package depend on another package that provides the BLAS, so:

numpy on PyPi would depend on this theoretical py_openblas

any package that depends on only numpy would be easy.

any package that depends on openblas would then depend on py_openblas.

So the question is: if I compile my third party package against numpy and
mkl, it would depend on numpy and py_mkl.

If I went to installed this on a system that had a numpy that depends on
openblas, pip would install py_mkl (having already installed numpy and
py_openblas). And I'd have a numpy extension calling into a different BLAS
than numpy itself is calling into -- would that cause any problems? Python
would be linked to two libs with the same names -- would that cause
conflict? I'm way out of my depth here!

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Carl Meyer 
On 10/09/2015 12:44 PM, Oscar Benjamin wrote:
> 
> On Fri, 9 Oct 2015 19:35 Carl Meyer  > wrote:
> 
> On 10/09/2015 12:28 PM, Oscar Benjamin wrote:
> > Why would it need dynamic metadata for the windows matplotlib wheel to
> > have different metadata from the OSX matplotlib wheel? The platform
> > Windows/OSX is static and each wheel declares its own dependencies
> > statically but differently. Am I missing something?
> 
> I didn't say that required dynamic metadata (wheel metadata is already
> static). I just said that it works fine currently, and that it becomes
> an open question with the move towards static metadata in both source
> and binary releases, because we have to answer questions like "what
> information beyond just package/version makes up a complete node in a
> dependency graph."
> 
> 
> 
> Assuming it's tied to the operating system it doesn't matter surely.
> When pip runs on Windows it can ignore dependencies that apply to other
> platforms so I don't see how this case makes it more complex.

Sure. "Assuming it's tied to the operating system" is an assumption that
can't be made here, though, if I understand the examples that have
already been given repeatedly regarding numpy and BLAS-linking.

If "OS" were actually sufficient to distinguish all cases, then the
existing wheel platform tags would already be an adequate solution to
this problem.

Carl



signature.asc
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Oscar Benjamin 
On Fri, 9 Oct 2015 19:35 Carl Meyer  wrote:

On 10/09/2015 12:28 PM, Oscar Benjamin wrote:
> Why would it need dynamic metadata for the windows matplotlib wheel to
> have different metadata from the OSX matplotlib wheel? The platform
> Windows/OSX is static and each wheel declares its own dependencies
> statically but differently. Am I missing something?

I didn't say that required dynamic metadata (wheel metadata is already
static). I just said that it works fine currently, and that it becomes
an open question with the move towards static metadata in both source
and binary releases, because we have to answer questions like "what
information beyond just package/version makes up a complete node in a
dependency graph."



Assuming it's tied to the operating system it doesn't matter surely. When
pip runs on Windows it can ignore dependencies that apply to other
platforms so I don't see how this case makes it more complex.

--
Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Carl Meyer 
On 10/09/2015 12:28 PM, Oscar Benjamin wrote:
> Why would it need dynamic metadata for the windows matplotlib wheel to
> have different metadata from the OSX matplotlib wheel? The platform
> Windows/OSX is static and each wheel declares its own dependencies
> statically but differently. Am I missing something?

I didn't say that required dynamic metadata (wheel metadata is already
static). I just said that it works fine currently, and that it becomes
an open question with the move towards static metadata in both source
and binary releases, because we have to answer questions like "what
information beyond just package/version makes up a complete node in a
dependency graph."

Carl



signature.asc
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

2015-10-09 Thread Chris Barker 
On Fri, Oct 9, 2015 at 10:44 AM, Thomas Güttler <
guettl...@thomas-guettler.de> wrote:

> The question remains: If a new comer asks you "How to package my
> python code and its tests?", there should be one default answer
> which works for 80% of all cases.
>

Should be, maybe -- but clearly there is no consensus as to what the "one"
answer shoudl be.

But I _think_ there are two answers:

1) inside the package, so it can be installed and the tests run on the
installed version.

2) outside the package, so that potentially large or complex test
requirements are not required for installation.

So the intro docs can lay out those two options, with a bit of text that
helps a newbie make a decision.

from above, -- I think that EITHER option would work fine for 80% of cases.

which makes me think -- why not pick one as the default while clearly
documenting the other.

And despite the fact that I have always used option (2) in my work, I
think, if there is going to be one in the simple example, it should be (1)
-- it's a fine way to get started, and users can move their tests outside
of the package later if they start getting big. As a package grows to that
extent, it will likely need other structural changes anyway.

-Chris











> I think the confusion gets worse by creating new public accessible
> repos which explain "Hey that's my way to package stupid simple
> python code".
>
> Regards,
>   Thomas Güttler
>
>
>
>
>
> --
> http://www.thomas-guettler.de/
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Oscar Benjamin 
On Fri, 9 Oct 2015 19:01 Carl Meyer  wrote:

On 10/09/2015 11:18 AM, Paul Moore wrote:
> On 9 October 2015 at 18:04, Chris Barker  wrote:
>> 1) what in the world is a "source wheel"? And how is it different than an
>> sdist (other than maybe in a different file format.
>
> A "source wheel" is the proposed name for a to-be-defined replacement
> for sdists. For now, you can think of "source wheel" and "sdist" as
> the same.
>
>> 2) Is it indeed "OK" with the current PEPs and tools for different binary
>> wheels to have different dependencies? This would be the example of, for
>> instance the Matplotlib binary wheel for Windows depends on a py_zlib,
>> whereas the binary wheel for OS-X relies on the the system lib, and
therefor
>> does not have that dependency?
>>  (and has anyone worked out the linking issues so that that would all
work
>> with virtualenv and friends...)
>
> It's not *currently* OK for different binary wheels to have different
> dependencies. At least I don't think it is. It's basically not
> something that as far as I'm aware anyone has ever considered an
> option up till now, and so it's quite likely that there are
> assumptions baked into the tools that would break if different builds
> of (a given version of) a package had different dependencies.

AFAIK this is actually just fine currently, it's just not considered
ideal for a hopeful future static-metadata world.



Why would it need dynamic metadata for the windows matplotlib wheel to have
different metadata from the OSX matplotlib wheel? The platform Windows/OSX
is static and each wheel declares its own dependencies statically but
differently. Am I missing something?

--
Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Carl Meyer 
On 10/09/2015 11:18 AM, Paul Moore wrote:
> On 9 October 2015 at 18:04, Chris Barker  wrote:
>> 1) what in the world is a "source wheel"? And how is it different than an
>> sdist (other than maybe in a different file format.
> 
> A "source wheel" is the proposed name for a to-be-defined replacement
> for sdists. For now, you can think of "source wheel" and "sdist" as
> the same.
> 
>> 2) Is it indeed "OK" with the current PEPs and tools for different binary
>> wheels to have different dependencies? This would be the example of, for
>> instance the Matplotlib binary wheel for Windows depends on a py_zlib,
>> whereas the binary wheel for OS-X relies on the the system lib, and therefor
>> does not have that dependency?
>>  (and has anyone worked out the linking issues so that that would all work
>> with virtualenv and friends...)
> 
> It's not *currently* OK for different binary wheels to have different
> dependencies. At least I don't think it is. It's basically not
> something that as far as I'm aware anyone has ever considered an
> option up till now, and so it's quite likely that there are
> assumptions baked into the tools that would break if different builds
> of (a given version of) a package had different dependencies.

AFAIK this is actually just fine currently, it's just not considered
ideal for a hopeful future static-metadata world.

Today, in the all-metadata-is-dynamic world that we actually live in,
the tools aren't able to make any assumptions at all, and just have to
download whatever best-match wheel or sdist they find for a given
package/version requirement, unpack it, and see what it says about its
own metadata (including dependencies). So different binary wheels of the
same package at the same version having different dependencies works
just fine. It's not like any of the tooling would actually know the
difference, since they will only deal with one of those wheels at a
given time.

But of course this dynamic-metadata world prevents the tools from doing
all kinds of useful things (like actually building dependency graphs in
advance so they can do proper dependency conflict resolution), which is
why we want to move towards static metadata. And it's in that transition
that issues like "are different wheels for the same project at the same
version allowed to have different dependencies" become relevant.

Carl



signature.asc
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Brett Cannon 
Listening in to an On Air session is unbounded, but direct participants are
capped at 10.

On Fri, 9 Oct 2015 at 09:17 Ionel Cristian Mărieș 
wrote:

>
> On Fri, Oct 9, 2015 at 7:05 PM, Chris Barker 
> wrote:
>
>> How many people can join a hangout? we may be bumping up against that
>> limit :-)
>
>
> ​AFAIK there's no limit on the number of people that can listen in. Also,
> Hangouts can record video (the "On Air" thing).​
>
>
>
> Thanks,
> -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

2015-10-09 Thread Thomas Güttler 
Am 07.10.2015 um 01:14 schrieb David Wilson:
> On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote:
> 
>> They should be inside the module. That way, you can check an installed
>> module is ok by running e.g. "python -m mypackage.tests". Any other
>> choice makes testing installed modules more cumbersome.
> 
> As Donald mentioned, this doesn't work in the general case since many
> packages ship quite substantial test data around that often doesn't end
> up installed, and in other cases since the package requires significant
> fixture setup or external resources (e.g. running SQLAlchemy tests
> without a working database server would be meaningless).

Should work with a temporary sqlite db.
 
> The option of always shipping test data as a standard part of a package
> in a vein attempt to always ensure it can be tested (which is not always
> likely given the SQLAlchemy example above) strikes me as incredibly
> wasteful, not from some oh-precious-bytes standpoint, but from the
> perspective of distributing a Python application of any size where the
> effects of always shipping half-configured test suites has increased the
> resulting distribution size potentially by 3 or 4x.
> 
> https://github.com/bennoleslie/pexif is the first hit on Google for a
> module I thought would need some test data. It's actually quite
> minimally tested, yet already the tests + data are 3.6x the size of the
> module itself.
> 
> 
> I appreciate arguments for inlining tests alongside a package in order
> to allow reuse of the suite's functionality by consuming applications'
> test suites, but as above, in the general case this simply isn't
> something that will always work and can be relied on by default.
> 
> 
> Is there perhaps a third option that was absent from the original post?
> e.g. organizing tests in a separate, optional, potentially
> pip-installable package.

Yes, this third way is plausible. I guess there is even a fourth way.

The question remains: If a new comer asks you "How to package my
python code and its tests?", there should be one default answer
which works for 80% of all cases.

I think the confusion gets worse by creating new public accessible
repos which explain "Hey that's my way to package stupid simple
python code".

Regards,
  Thomas Güttler





-- 
http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Paul Moore 
On 9 October 2015 at 18:04, Chris Barker  wrote:
> Which seems to point more to having an optional "binary_build" component to
> the name. Not sure what syntax is available, but the idea would be that:
>
> "numpy|mkl"
>
> and
>
> "numpy|openblas"
>
> would both match "numpy"
>
> which, of course would require changes to the whole stack...

This sounds more like it's an extension of the wheel "compatibility
tag" approach. That's another option I can imagine being worth looking
at.

Basically, wheels currently encode in their names the Python version,
ABI and architecture(s) they work on. That's sufficient for simple
uses, but for more complicated scenarios you need more (such as, "this
wheel is only valid if library FOO is present). It's not impossible to
extend the compatibility tag mechanism to include such things, but
there are a number of issues that would need to be thrashed out (spot
the common theme here? :-)) Specifically, it's horribly cumbersome to
encode everything in the filename, so some more scalable mechanism is
needed. Also, you need a notion of compatibility order (is wheel X
"more compatible" with your system than wheel Y?). And of course,
compatibility is about "does it work with what's there?" which doesn't
allow for the possibility of downloading and installing a dependency
(that's what dependency resolution does, compatibility checking is at
a later stage in the process).

So again this is a possible solution, but needs someone to work on the details.


... or thinking again, maybe you mean having multiple packages
(numpy_mkl, numpy_openblas, ...) all of which satisfy a "numpy"
requirement? That's definitely viable, the Metadata 2.0 spec allows
for one package "providing" an implementation of another
(https://www.python.org/dev/peps/pep-0426/#provides). But that part of
Metadata 2.0 isn't implemented yet - it needs some people with time to
work on it again, and it will probably be implemented alongside "sdist
2.0", which is what we've been calling "when we get round to
reworking/rethinking the sdist format".

Paul

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Paul Moore 
On 9 October 2015 at 18:04, Chris Barker  wrote:
> 1) what in the world is a "source wheel"? And how is it different than an
> sdist (other than maybe in a different file format.

A "source wheel" is the proposed name for a to-be-defined replacement
for sdists. For now, you can think of "source wheel" and "sdist" as
the same.

> 2) Is it indeed "OK" with the current PEPs and tools for different binary
> wheels to have different dependencies? This would be the example of, for
> instance the Matplotlib binary wheel for Windows depends on a py_zlib,
> whereas the binary wheel for OS-X relies on the the system lib, and therefor
> does not have that dependency?
>  (and has anyone worked out the linking issues so that that would all work
> with virtualenv and friends...)

It's not *currently* OK for different binary wheels to have different
dependencies. At least I don't think it is. It's basically not
something that as far as I'm aware anyone has ever considered an
option up till now, and so it's quite likely that there are
assumptions baked into the tools that would break if different builds
of (a given version of) a package had different dependencies.

One of the proposed approaches to binary dependencies (the one I've
been referring to as "the pyopenblas approach") is based on the idea
that different wheels could have different dependency metadata. I've
tried to enumerate the questions that need to be looked at if we were
to go down that route, but to my knowledge, no-one has yet either
tested how well things would work if this happened, or audited the
code for problematic assumptions. So it's a possibility for the
future, but like many things it depends on someone doing the work to
make it happen (and as far as I know it's mostly the numpy community
who have the need, so I assume someone in that community would need to
move this forward).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Chris Barker 
I"ve been following this thread and gotten a bit lost. But I do care,
because I'm a heavy numpy user, and also because I was involved for years
in building pacakges for OS-X, and currently need distribute some of my own
stuff that has semi-ugly C lib dependencies (Not as ugly as BLAS.,though
:-) )

In the last couple years I gave up on PyPi and pip (and went with Anaconda
and conda)-- that ecosystem simply doesn't currently support my use-cases.
But it would be nice to go back, and it looks like there are some idea on
the table that will make that possible.

But could someone clarify a thing or two for me??

1) what in the world is a "source wheel"? And how is it different than an
sdist (other than maybe in a different file format.

2) Is it indeed "OK" with the current PEPs and tools for different binary
wheels to have different dependencies? This would be the example of, for
instance the Matplotlib binary wheel for Windows depends on a py_zlib,
whereas the binary wheel for OS-X relies on the the system lib, and
therefor does not have that dependency?
 (and has anyone worked out the linking issues so that that would all work
with virtualenv and friends...)

if (2) then it seems the issue is what to do with the same package on the
same platform having potentially different dependencies -- i.e. numpy w/
mkl and numpy w/some_other_blas. In that case, I think that this could
completely explode into m**n possible wheels if we try to accommodate it in
a fully flexible manner -- so making it a name-thing (like was proposed
here), makes sense to me -- numpy_mkl and numpy_openblas are, as far as pip
is concerned, different packages. I think this is OK, as we probably only
want some small subset of possible build up on PyPi anyway (only one?).

But it does get a bit tricky if you want to put up a package that depends
on "pure" numpy -- i.e. it doesn't care which BLAS the numpy it uses, but
it DOES need numpy.

This could this be Accomidated byt have an "or" option for dependencies:

numpy_mkl>=1.9 or numpy_openblas >= 1.9

However, that would mean that the author of that wheel would need to know
all of the wheels that might be available up front - less than ideal.

Which seems to point more to having an optional "binary_build" component to
the name. Not sure what syntax is available, but the idea would be that:

"numpy|mkl"

and

"numpy|openblas"

would both match "numpy"

which, of course would require changes to the whole stack...

and off the top of my head, I'm wondering whether having one "binary_build'
flag would be enough, or if we'd find that we wanted n options, and n*m
combinations, and this would all blow up.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Ionel Cristian Mărieș 
On Fri, Oct 9, 2015 at 7:05 PM, Chris Barker  wrote:

> How many people can join a hangout? we may be bumping up against that
> limit :-)


​AFAIK there's no limit on the number of people that can listen in. Also,
Hangouts can record video (the "On Air" thing).​



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Chris Barker 
How many people can join a hangout? we may be bumping up against that limit
:-)

-CHB

On Thu, Oct 8, 2015 at 12:58 PM, Nathaniel Smith  wrote:

> On Oct 8, 2015 09:33, "Nathaniel Smith"  wrote:
> >
> > On Oct 8, 2015 8:14 AM, "Nathaniel Smith"  wrote:
> > >
> > > On Oct 7, 2015 2:58 PM, "Donald Stufft"  wrote:
> > > >
> > > > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (n...@pobox.com)
> wrote:
> > > > > > Yeah, that's not a good long term solution -- it needs to be
> moved
> > > > > into the metadata (probably by creating an MKL wheel and then
> > > > > making
> > > > > the numpy wheel depend on it). That's exactly the problem :-)
> > > > >
> > > >
> > > > Are you available on IRC or for a video call or something? I feel
> like there's
> > > > something foundational from both sides that we're each missing here
> and it'd be
> > > > easier to just hash it out in real time rather than lobbying random
> emails
> > > > coming from places of confusion (at least on my side).
> > > >
> > > > I'm not sure if Paul (or anyone else!) would want to jump in on it
> too, though
> > > > I feel like probably if it's me and you then the two "sides" will
> probably be
> > > > reasonably well represented so if more folks don't want to join
> that's probably
> > > > OK too, particularly since we wouldn't be making any actual
> decisions there :D
> > >
> > > This does sound like it would be a good idea -- couldn't hurt, anyway
> :-). I'll contact you offlist. If anyone else wants to join in, email me...
> >
> > Looks like this is happening tomorrow (Fri Oct 9) at 11 am California /
> 2 pm New York / 7 pm London. Since there's been at least some interest,
> we'll do it as a Google hangout and send a link around in case anyone wants
> to listen in, and of course summarize back to the list.
>
> Correction: happening *Monday Oct 12* at 11 am California / 2 pm New York
> / 7 pm London.
>
> -n
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

2015-10-09 Thread David Wilson 
On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote:

> They should be inside the module. That way, you can check an installed
> module is ok by running e.g. "python -m mypackage.tests". Any other
> choice makes testing installed modules more cumbersome.

As Donald mentioned, this doesn't work in the general case since many
packages ship quite substantial test data around that often doesn't end
up installed, and in other cases since the package requires significant
fixture setup or external resources (e.g. running SQLAlchemy tests
without a working database server would be meaningless).

The option of always shipping test data as a standard part of a package
in a vein attempt to always ensure it can be tested (which is not always
likely given the SQLAlchemy example above) strikes me as incredibly
wasteful, not from some oh-precious-bytes standpoint, but from the
perspective of distributing a Python application of any size where the
effects of always shipping half-configured test suites has increased the
resulting distribution size potentially by 3 or 4x.

https://github.com/bennoleslie/pexif is the first hit on Google for a
module I thought would need some test data. It's actually quite
minimally tested, yet already the tests + data are 3.6x the size of the
module itself.


I appreciate arguments for inlining tests alongside a package in order
to allow reuse of the suite's functionality by consuming applications'
test suites, but as above, in the general case this simply isn't
something that will always work and can be relied on by default.


Is there perhaps a third option that was absent from the original post?
e.g. organizing tests in a separate, optional, potentially
pip-installable package.


> 
> > outside the module like this:
> > 
> >  https://github.com/pypa/sampleproject/tree/master/tests
> 
> There is no actual reason to do that except win a couple kilobytes if
> you are distributing your package on floppy disks for consumption on
> Z80-based machines with 64KB RAM.
> 
> Even Python *itself* puts its test suite inside the standard library,
> not outside it (though some Linux distros may strip it away).
> Try "python -m test.regrtest" (again, this may fail if your distro
> decided to ship the test suite in a separate package).
> 
> The PyP"A" should definitely fix its sample project to reflect good
> practices.
> 
> Regards
> 
> Antoine.
> 
> 
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] sampleproject: use tox?

2015-10-09 Thread Paul Moore 
Well, https://packaging.python.org/en/latest/distributing/ already says

"""
The section does not aim to cover best practices for Python project
development as a whole. For example, it does not provide guidance or
tool recommendations for version control, documentation, or testing
"""

It may be good to add a footnote with your "For more exhaustive
examples..." comment - but it would need links, and good luck on
getting any sort of agreement on what constitutes good examples :-)

Paul


On 9 October 2015 at 11:11, Ionel Cristian Mărieș  wrote:
>
> On Fri, Oct 9, 2015 at 12:43 PM, Paul Moore  wrote:
>>
>>
>> The sample project is not intended for that purpose. It's a good
>> starting point for something like that, but that's not its purpose.
>>
>> Paul
>>
>> PS I sympathise with your wish for a "do this to set up your first
>> project" beginner's guide. I've wanted one of those myself for a long
>> time now. But I don't believe such a thing exists, unfortunately.
>
>
> Should we at least give some pointers as to what people should look at for a
> fully fledged project? Like a lists with examples/templates covering all the
> variants. Eg:
>
> "PyPA's sample projects only covers the packaging aspect of a project. For
> more exhaustive examples take a look at cookiecutter templates, barry's
> sample, ogrisel's appveyor demo, this and that etc etc"
>
> How about something like that?
>
>
>
> Thanks,
> -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] sampleproject: use tox?

2015-10-09 Thread Ionel Cristian Mărieș 
On Fri, Oct 9, 2015 at 12:43 PM, Paul Moore  wrote:

>
> The sample project is not intended for that purpose. It's a good
> starting point for something like that, but that's not its purpose.
>
> Paul
>
> PS I sympathise with your wish for a "do this to set up your first
> project" beginner's guide. I've wanted one of those myself for a long
> time now. But I don't believe such a thing exists, unfortunately.


​Should we at least give some pointers as to what people should look at for
a fully fledged project? Like a lists with examples/templates covering all
the variants. Eg:

"PyPA's sample projects only covers the packaging aspect of a project. For
more exhaustive examples take a look at cookiecutter templates, barry's
sample, ogrisel's appveyor demo, this and that etc etc"

How about something like that?



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] sampleproject: use tox?

2015-10-09 Thread Paul Moore 
On 9 October 2015 at 10:32, Thomas Güttler  wrote:
> My question was: Any reasons not to use tox in the sample project?

1. Because the sample project is intended to explain how to *package*
a project, not how to test it, or document it, or any of a number of
other non-packaging things.
2. Because tox is not universally agreed as best practice, and the
sample project is intended to demonstrate agreed best practice.

> I search a simple guideline for new comers to set up
> a python project including testing.

The sample project is not intended for that purpose. It's a good
starting point for something like that, but that's not its purpose.

Paul

PS I sympathise with your wish for a "do this to set up your first
project" beginner's guide. I've wanted one of those myself for a long
time now. But I don't believe such a thing exists, unfortunately.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] sampleproject: use tox?

2015-10-09 Thread Thomas Güttler 



Am 08.10.2015 um 23:33 schrieb Antoine Pitrou:

On Thu, 8 Oct 2015 22:05:16 +0200
Thomas Güttler  wrote:

This is a follow up to the thread "Where should I put tests when packaging python 
modules?"

I have never used tox up to now. But reading the mails of the thread, it seems
that tox is the current state of the art.


I don't see why that would be the case.  tox is certainly "the state of
the art" in certain circles, but many projects don't mandate it at all
or even don't use it at all and rely on other mechanisms (e.g. CI with
a rich configuration matrix).


My question was: Any reasons not to use tox in the sample project?

I read your answer like this:

 Many projects don't mandate it at all
 or even don't use it at all and rely on other mechanisms

Yes, it is correct that many projects is a different way.

I search a simple guideline for new comers to set up
a python project including testing.

 "Since many projects don't mandate it at all ..."

I don't know if this a good argument against it.

Regards,
  Thomas Güttler


--
Thomas Guettler http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

2015-10-09 Thread Ronny Pfannschmidt 
I'd also like to join

Am 9. Oktober 2015 06:29:58 MESZ, schrieb Robert Collins 
:
>Toss me an invite too - I'm very interested in this, for all that I
>havent' kibbitzed on the thread yet :)
>
>On 9 October 2015 at 05:33, Nathaniel Smith  wrote:
>> On Oct 8, 2015 8:14 AM, "Nathaniel Smith"  wrote:
>>>
>>> On Oct 7, 2015 2:58 PM, "Donald Stufft"  wrote:
>>> >
>>> > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (n...@pobox.com)
>wrote:
>>> > > > Yeah, that's not a good long term solution -- it needs to be
>moved
>>> > > into the metadata (probably by creating an MKL wheel and then
>>> > > making
>>> > > the numpy wheel depend on it). That's exactly the problem :-)
>>> > >
>>> >
>>> > Are you available on IRC or for a video call or something? I feel
>like
>>> > there's
>>> > something foundational from both sides that we're each missing
>here and
>>> > it'd be
>>> > easier to just hash it out in real time rather than lobbying
>random
>>> > emails
>>> > coming from places of confusion (at least on my side).
>>> >
>>> > I'm not sure if Paul (or anyone else!) would want to jump in on it
>too,
>>> > though
>>> > I feel like probably if it's me and you then the two "sides" will
>>> > probably be
>>> > reasonably well represented so if more folks don't want to join
>that's
>>> > probably
>>> > OK too, particularly since we wouldn't be making any actual
>decisions
>>> > there :D
>>>
>>> This does sound like it would be a good idea -- couldn't hurt,
>anyway :-).
>>> I'll contact you offlist. If anyone else wants to join in, email
>me...
>>
>> Looks like this is happening tomorrow (Fri Oct 9) at 11 am California
>/ 2 pm
>> New York / 7 pm London. Since there's been at least some interest,
>we'll do
>> it as a Google hangout and send a link around in case anyone wants to
>listen
>> in, and of course summarize back to the list.
>>
>> -n
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>

MFG Ronny
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig