Re: [Distutils] pip on windows experience

[Noah Kantrowitz]

On Jan 23, 2014, at 4:17 PM, Oscar Benjamin  wrote:

> On 23 January 2014 23:58, Nick Coghlan  wrote:
>> 
>> I really think that's our best near term workaround - still room for
>> improvement, but " pip install numpy assumes SSE2" is a much better
>> situation than "pip install numpy doesn't work on Windows".
> 
> Is it? Do you have any idea what proportion of (the relevant) people
> would be using Windows with hardware that doesn't support SSE2? I feel
> confident that it's less than 10% but I don't know how to justify a
> tighter bound than that.
> 
> You need to bear in mind that people currently have a variety of ways
> to install numpy on Windows that do work already without limitations
> on CPU instruction set. Most numpy users will not get any immediate
> benefit from the fact that "it works using pip" rather than "it works
> using the .exe installer" (or any of a number of other options). It's
> the unfortunate end users and the numpy folks who would have to pick
> up the pieces if/when the SSE2 assumption fails.
> 

This all sounds very similar to the issues with Linux binary wheels and varying 
system ABIs. Should probably keep in mind for any solution that might apply to 
both.

--Noah




signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[Oscar Benjamin]

On 23 January 2014 23:58, Nick Coghlan  wrote:
>
> I really think that's our best near term workaround - still room for
> improvement, but " pip install numpy assumes SSE2" is a much better
> situation than "pip install numpy doesn't work on Windows".

Is it? Do you have any idea what proportion of (the relevant) people
would be using Windows with hardware that doesn't support SSE2? I feel
confident that it's less than 10% but I don't know how to justify a
tighter bound than that.

You need to bear in mind that people currently have a variety of ways
to install numpy on Windows that do work already without limitations
on CPU instruction set. Most numpy users will not get any immediate
benefit from the fact that "it works using pip" rather than "it works
using the .exe installer" (or any of a number of other options). It's
the unfortunate end users and the numpy folks who would have to pick
up the pieces if/when the SSE2 assumption fails.

> Such a change would help a lot of people *right now*, while still leaving
> room to eventually figure out something more sophisticated (like postinstall
> hooks or simpler runtime multi-build support or NumPy changing to a
> dependency that internally makes this decision at runtime).

Postinstall hooks are not that sophisticated and most packaging
systems have them. You're advocating for numpy to take a dodgy
compromise here but can it not be the other way round? Wheel, pip etc.
could quickly agree on and implement a postinstall hook that would
work for numpy and then that could be made more sophisticated later
on.


Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[Nick Coghlan]

On 24 Jan 2014 08:03, "Chris Barker"  wrote:
>
> On Thu, Jan 23, 2014 at 12:25 PM, Thomas Heller 
wrote:
>>
>> Did I say this before?  I would suggest that numpy develops a way
>> where all the SSE binary variations would be installed, and the
>> appropriate ones be loaded at runtime, depending on the user's CPU
>> capabilities.  This would also allow py2exe'd distributions to include
>> them all.
>
>
> That was discussed on the numpy list, and would be really nice, but it
may also be really difficult. OS-X has built-in support for
multi-architecture binaries, but Windows does not, and while selecting a
particular .dll (or .pyd) to load at run-time would be
fairly straightforward  numpy  has more than one, and then there is the
whole scipy stack, and all the third-party stuff compiled against it.
>
> I suspect this wold have to be built-in to the python importing and
distutils build system to be workable. But maybe someone smarter than me
will figure it out.
>
>
>>> Some feedback from the people who did try those wheels would help. I
>>> asked for that on the numpy list after creating them, but didn't get
>>> much. So I haven't been in a hurry to move them over to PyPi.
>
>
> Serious chicken-egg problem there
>
>>
>> I would have tried wheels for windows, python 3.3 or 3.4, but there
aren't any.
>
>
> Yeah we need to get those up -- SSE2 only ones would work for MOST people.

I really think that's our best near term workaround - still room for
improvement, but " pip install numpy assumes SSE2" is a much better
situation than "pip install numpy doesn't work on Windows".

Such a change would help a lot of people *right now*, while still leaving
room to eventually figure out something more sophisticated (like
postinstall hooks or simpler runtime multi-build support or NumPy changing
to a dependency that internally makes this decision at runtime).

Cheers,
Nick.

>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[Chris Barker]

On Thu, Jan 23, 2014 at 12:25 PM, Thomas Heller  wrote:

> Did I say this before?  I would suggest that numpy develops a way
> where all the SSE binary variations would be installed, and the
> appropriate ones be loaded at runtime, depending on the user's CPU
> capabilities.  This would also allow py2exe'd distributions to include
> them all.


That was discussed on the numpy list, and would be really nice, but it may
also be really difficult. OS-X has built-in support for
multi-architecture binaries, but Windows does not, and while selecting a
particular .dll (or .pyd) to load at run-time would be
fairly straightforward  numpy  has more than one, and then there is the
whole scipy stack, and all the third-party stuff compiled against it.

I suspect this wold have to be built-in to the python importing and
distutils build system to be workable. But maybe someone smarter than me
will figure it out.


Some feedback from the people who did try those wheels would help. I
>> asked for that on the numpy list after creating them, but didn't get
>> much. So I haven't been in a hurry to move them over to PyPi.
>
>
Serious chicken-egg problem there


>  I would have tried wheels for windows, python 3.3 or 3.4, but there
> aren't any.


Yeah we need to get those up -- SSE2 only ones would work for MOST people.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[Nick Coghlan]

On 24 January 2014 06:25, Thomas Heller  wrote:
> Am 23.01.2014 19:52, schrieb Ralf Gommers:
>> On Thu, Jan 23, 2014 at 3:42 PM, Oscar Benjamin
>> mailto:oscar.j.benja...@gmail.com>> wrote:
>>
>> On Thu, Jan 23, 2014 at 12:16:02PM +, Paul Moore wrote:
>>  >
>>  > The official numpy installer uses some complex magic to select the
>>  > right binaries based on your CPU, and this means that the official
>>  > numpy "superpack" wininst files don't convert (at least I don't
>> think
>>  > they do, it's a while since I tried).
>>
>> It's probably worth noting that numpy are toying around with wheels
>> and
>> have uploaded a number of them to PyPI for testing:
>> http://sourceforge.net/projects/numpy/files/wheels_to_test/
>>
>> Currently there are only OSX wheels there (excluding the puer Python
>> ones) and they're not available on PyPI. I assume that they're waiting
>> for a solution for the Windows installer (a post-install script for
>> wheels). That would give a lot more impetus to put wheels up on PyPI.
>>
>>
>> Indeed. We discussed just picking the SSE2 or SSE3 build and putting
>> that up as a wheel, but that was deemed a not so great idea:
>> http://article.gmane.org/gmane.comp.python.numeric.general/56072
>
>
> Did I say this before?  I would suggest that numpy develops a way
> where all the SSE binary variations would be installed, and the
> appropriate ones be loaded at runtime, depending on the user's CPU
> capabilities.  This would also allow py2exe'd distributions to include
> them all.

I believe I suggested that at one point, but the dependencies were
scattered throughout NumPy rather than being in one place (so you had
to toggle several modules, rather than just one), and it made for some
interesting build problems further up the stack.

Agreed that runtime selection would be better, though - the current
approach not only makes it difficult to do things like create
universal py2exe and wheel binaries, it also makes it difficult to run
NumPy from portable media like a USB key, because different systems
may need different SSE modules.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels, metadata

[Thomas Heller]

Are the specifications in the PEPs above implemented somewhere
or are they only 'specifications'?


The Packaging User Guide is maintaining a summary of the relevant PEPs
that gives a brief description of the "user impact" and "implementation"
(i.e. whether projects are actually implementing any of this).
https://python-packaging-user-guide.readthedocs.org/en/latest/peps.html
For those working on the various projects, please help keep this up to date.


Thanks for the pointer; bookmarked.

Thomas


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[Thomas Heller]

Am 23.01.2014 19:52, schrieb Ralf Gommers:




On Thu, Jan 23, 2014 at 3:42 PM, Oscar Benjamin
mailto:oscar.j.benja...@gmail.com>> wrote:

On Thu, Jan 23, 2014 at 12:16:02PM +, Paul Moore wrote:
 >
 > The official numpy installer uses some complex magic to select the
 > right binaries based on your CPU, and this means that the official
 > numpy "superpack" wininst files don't convert (at least I don't think
 > they do, it's a while since I tried).

It's probably worth noting that numpy are toying around with wheels and
have uploaded a number of them to PyPI for testing:
http://sourceforge.net/projects/numpy/files/wheels_to_test/

Currently there are only OSX wheels there (excluding the puer Python
ones) and they're not available on PyPI. I assume that they're waiting
for a solution for the Windows installer (a post-install script for
wheels). That would give a lot more impetus to put wheels up on PyPI.


Indeed. We discussed just picking the SSE2 or SSE3 build and putting
that up as a wheel, but that was deemed a not so great idea:
http://article.gmane.org/gmane.comp.python.numeric.general/56072


Did I say this before?  I would suggest that numpy develops a way
where all the SSE binary variations would be installed, and the
appropriate ones be loaded at runtime, depending on the user's CPU
capabilities.  This would also allow py2exe'd distributions to include
them all.



The Sourceforge OSX wheels are presumably not getting that much use
right now. The OSX-specific numpy wheel has been downloaded 4 times in
the last week: twice on Windows and twice on Linux!


Some feedback from the people who did try those wheels would help. I
asked for that on the numpy list after creating them, but didn't get
much. So I haven't been in a hurry to move them over to PyPi.


I would have tried wheels for windows, python 3.3 or 3.4, but there 
aren't any.


Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels, metadata

[Vinay Sajip]

> certainly mention the distlib implementations,
> but also let's be clear if there is a pypa-recommend
> tool that is user-facing (like pip), that is using those
> parts of distlib.  
> In most cases, that is not true currently.
> As for mentioning distil,  I'm inclined to
> say no.   Up to this point, you've presented it as a
> proof of concept.If you're wanting to mention
> "distil" as a real option for users, I'm
> concerned about fracturing the mind of users, but it's
> something to discuss I guess.

I hear what you're saying. I've positioned distil as a proof of concept purely 
because it hasn't had widespread use, but I certainly expect it to fulfil the 
same role as pip functionally (which it must do to be an effective test-bed for 
distlib). I understand that pip is the officially recommended tool, and don't 
want to muddy the waters, there being enough confusion about packaging in the 
wider community. It seems a shame that some of the improvements over pip won't 
be more widely available, but such is life. I have the use of them, so there's 
that :-)

Regards,

Vinay Sajip

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels, metadata

[Daniel Holth]

We need pip's build to become more like conda build.
http://docs.continuum.io/conda/build.html

A simple recipe, not necessarily part of the to-be-built package
source, invokes build.sh to create a package. In that kind of system
things like distil can be used at will.

On Thu, Jan 23, 2014 at 2:12 PM, Marcus Smith  wrote:
>>
>> A lot of this stuff is implemented in distlib and distil (distlib itself
>> is just a library that provides the underpinnings, and some of the
>> functionality required, such as actually installing stuff, is implemented in
>> distil. I assume it's OK to add references to these? For example, distlib
>> implements PEP 376, PEP 425, PEP 426, PEP 427, PEP 440, but only the last of
>> these is mentioned on the PEP summaries page.
>
>
> certainly mention the distlib implementations, but also let's be clear if
> there is a pypa-recommend tool that is user-facing (like pip), that is using
> those parts of distlib.
> In most cases, that is not true currently.
>
> As for mentioning distil,  I'm inclined to say no.   Up to this point,
> you've presented it as a proof of concept.
> If you're wanting to mention "distil" as a real option for users, I'm
> concerned about fracturing the mind of users, but it's something to discuss
> I guess.
>
> Marcus
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels, metadata

[Marcus Smith]

>
>
> A lot of this stuff is implemented in distlib and distil (distlib itself
> is just a library that provides the underpinnings, and some of the
> functionality required, such as actually installing stuff, is implemented
> in distil. I assume it's OK to add references to these? For example,
> distlib implements PEP 376, PEP 425, PEP 426, PEP 427, PEP 440, but only
> the last of these is mentioned on the PEP summaries page.
>

certainly mention the distlib implementations, but also let's be clear if
there is a pypa-recommend tool that is user-facing (like pip), that is
using those parts of distlib.
In most cases, that is not true currently.

As for mentioning distil,  I'm inclined to say no.   Up to this point,
you've presented it as a proof of concept.
If you're wanting to mention "distil" as a real option for users, I'm
concerned about fracturing the mind of users, but it's something to discuss
I guess.

Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels, metadata

[Vinay Sajip]

On Thu, 23/1/14, Marcus Smith  wrote:

> The Packaging User Guide is maintaining a summary
> of the relevant PEPs that gives a brief description of the
> "user impact" and "implementation" (i.e.
> whether projects are actually implementing any of
> this).
> https://python-packaging-user-guide.readthedocs.org/en/latest/peps.html
> For those working on the various projects, please
> help keep this up to date.
 
A lot of this stuff is implemented in distlib and distil (distlib itself is 
just a library that provides the underpinnings, and some of the functionality 
required, such as actually installing stuff, is implemented in distil. I assume 
it's OK to add references to these? For example, distlib implements PEP 376, 
PEP 425, PEP 426, PEP 427, PEP 440, but only the last of these is mentioned on 
the PEP summaries page.

Regards,

Vinay Sajip

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[Ralf Gommers]

On Thu, Jan 23, 2014 at 3:42 PM, Oscar Benjamin
wrote:

> On Thu, Jan 23, 2014 at 12:16:02PM +, Paul Moore wrote:
> >
> > The official numpy installer uses some complex magic to select the
> > right binaries based on your CPU, and this means that the official
> > numpy "superpack" wininst files don't convert (at least I don't think
> > they do, it's a while since I tried).
>
> It's probably worth noting that numpy are toying around with wheels and
> have uploaded a number of them to PyPI for testing:
> http://sourceforge.net/projects/numpy/files/wheels_to_test/
>
> Currently there are only OSX wheels there (excluding the puer Python
> ones) and they're not available on PyPI. I assume that they're waiting
> for a solution for the Windows installer (a post-install script for
> wheels). That would give a lot more impetus to put wheels up on PyPI.
>

Indeed. We discussed just picking the SSE2 or SSE3 build and putting that
up as a wheel, but that was deemed a not so great idea:
http://article.gmane.org/gmane.comp.python.numeric.general/56072

The Sourceforge OSX wheels are presumably not getting that much use
> right now. The OSX-specific numpy wheel has been downloaded 4 times in
> the last week: twice on Windows and twice on Linux!
>

Some feedback from the people who did try those wheels would help. I asked
for that on the numpy list after creating them, but didn't get much. So I
haven't been in a hurry to move them over to PyPi.

Ralf
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels, metadata

[Marcus Smith]

>
>
> Are the specifications in the PEPs above implemented somewhere
> or are they only 'specifications'?
>

The Packaging User Guide is maintaining a summary of the relevant PEPs that
gives a brief description of the "user impact" and "implementation" (i.e.
whether projects are actually implementing any of this).
https://python-packaging-user-guide.readthedocs.org/en/latest/peps.html
For those working on the various projects, please help keep this up to date.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels, metadata

[Vinay Sajip]

On Thu, 23/1/14, Daniel Holth  wrote:
 
 supports_environments is not implemented.

Perhaps not in pip and wheel, but there is support in distlib and distil. 
Specifically, distil should present a list of failing clauses in 
supports_environments for any distributions to be installed (including 
dependent ones). Of course, there's not much on PyPI one can test with, since 
that metadata is absent, so testing of this feature has been limited to local 
test data.

Regards,

Vinay Sajip
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels, metadata

[Daniel Holth]

On Thu, Jan 23, 2014 at 9:23 AM, Thomas Heller  wrote:
> I'm reading up and done all those peps:
>
> 425 - compaibility tags
> 426 - metadata 2.0
> 427 - wheel binary format 1.0
> 440 - version identification and dependency specifications
>
> and trying to make sense from them.  Well, they make sense to me,
> but to what do they apply?
>
> Example: I can now build, upload, and install a wheel for my distribution.
>
> I have imported setuptools in my setup.py script,
> and have run 'setup.py bdist_wheel'.
>
> The resulting file is named 'py2exe-0.9.0-py34-none-any.whl',
> which means:
>
> python tag 'py34', abi tag 'none', platform tag 'any'
>
> What this file really is: it is a library that only works on
> CPython 3.3 and 3.4, win32 and win_amd64.  So I would have expected
> a filename like py2exe-0.9.0-py33.py34-none-win.whl or similar.
>
> Hoever, the only thing that I found which did change the filename
> is this section in the setup.cfg file (which is not appopriate for
> this distribution):
>
> [wheel]
> universal=1
>
> So, the question is:
>
> Are the specifications in the PEPs above implemented somewhere
> or are they only 'specifications'?
>
> There are very detailed descriptions how to specify requirements
> in the metadata (in pep426), something like
>
> "supports_environments": ["python_version >= '3.3'
>and 'win' in sys_platform"]
>
> Where can I specify this?  Does distutils support it? Do
> I have to manually edit some pdist.json file, or do I have
> to rename the created .whl?
>
> Thomas

supports_environments is not implemented.

Currently you should be able to just rename the created .whl to change
the tags. In the future the wheel metadata will have to match the
filename so you would need a 'retag' command or just better setup.cfg
options.

bdist_wheel supports this syntax for replacing the requirements
metadata and including the full text of a license in the metadata, in
setup.cfg. One drawback to this approach is that the wheel can have
different dependencies than the non-wheel because vanilla setuptools
doesn't generate the .dist-info style metadata.

distutils sucks, but I haven't had the opportunity to write a replacement.

[metadata]
provides-extra =
tool
signatures
faster-signatures
requires-dist =
argparse; python_version == '2.6'
keyring; extra == 'signatures'
dirspec; sys.platform != 'win32' and extra == 'signatures'
ed25519ll; extra == 'faster-signatures'
license-file = LICENSE.txt

[wheel]
# use py2.py3 tag for pure-python dist:
universal=1
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[Oscar Benjamin]

On Thu, Jan 23, 2014 at 12:16:02PM +, Paul Moore wrote:
> 
> The official numpy installer uses some complex magic to select the
> right binaries based on your CPU, and this means that the official
> numpy "superpack" wininst files don't convert (at least I don't think
> they do, it's a while since I tried).

It's probably worth noting that numpy are toying around with wheels and
have uploaded a number of them to PyPI for testing:
http://sourceforge.net/projects/numpy/files/wheels_to_test/

Currently there are only OSX wheels there (excluding the puer Python
ones) and they're not available on PyPI. I assume that they're waiting
for a solution for the Windows installer (a post-install script for
wheels). That would give a lot more impetus to put wheels up on PyPI.

The Sourceforge OSX wheels are presumably not getting that much use
right now. The OSX-specific numpy wheel has been downloaded 4 times in
the last week: twice on Windows and twice on Linux!

> But happily, Christoph Gohlke hosts a huge list of readymade wininst
> installers for hard-to-build projects, and the 3 you mention are all
> there. He's very good about building for latest Pythons, too (3.4 is
> already there for many packages). Anyone working on Windows who
> doesn't know his site
> (http://www.lfd.uci.edu/~gohlke/pythonlibs/) should check it out.

Also I've seen Cristoph mention on the numpy-discussion list that he was
at leasting testing building wheels although none seem to be available
on his site at the moment.


Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] wheels, metadata

[Thomas Heller]

I'm reading up and done all those peps:

425 - compaibility tags
426 - metadata 2.0
427 - wheel binary format 1.0
440 - version identification and dependency specifications

and trying to make sense from them.  Well, they make sense to me,
but to what do they apply?

Example: I can now build, upload, and install a wheel for my distribution.

I have imported setuptools in my setup.py script,
and have run 'setup.py bdist_wheel'.

The resulting file is named 'py2exe-0.9.0-py34-none-any.whl',
which means:

python tag 'py34', abi tag 'none', platform tag 'any'

What this file really is: it is a library that only works on
CPython 3.3 and 3.4, win32 and win_amd64.  So I would have expected
a filename like py2exe-0.9.0-py33.py34-none-win.whl or similar.

Hoever, the only thing that I found which did change the filename
is this section in the setup.cfg file (which is not appopriate for
this distribution):

[wheel]
universal=1

So, the question is:

Are the specifications in the PEPs above implemented somewhere
or are they only 'specifications'?

There are very detailed descriptions how to specify requirements
in the metadata (in pep426), something like

"supports_environments": ["python_version >= '3.3'
   and 'win' in sys_platform"]

Where can I specify this?  Does distutils support it? Do
I have to manually edit some pdist.json file, or do I have
to rename the created .whl?

Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[Thomas Heller]

Am 23.01.2014 13:16, schrieb Paul Moore:

On 23 January 2014 11:48, Thomas Heller  wrote:

Is there a solution to this?  I've seen that the wheel tool can convert
bdist_wininst installers into wheels - does this work for the packages
I mentioned above?  Do we have to build or convert to wheel those
packages, and setup a central place where we store them and make them
available to our developers?


The good news is that "wheel convert XXX.exe" works on all the above
for creating a wheel from the wininst installer. The official numpy
installer uses some complex magic to select the right binaries based
on your CPU, and this means that the official numpy "superpack"
wininst files don't convert (at least I don't think they do, it's a
while since I tried). But happily, Christoph Gohlke hosts a huge list
of readymade wininst installers for hard-to-build projects, and the 3
you mention are all there. He's very good about building for latest
Pythons, too (3.4 is already there for many packages). Anyone working
on Windows who doesn't know his site
(http://www.lfd.uci.edu/~gohlke/pythonlibs/) should check it out.

So, to summarise, yes, you can get wheels for pretty much everything
you need by using wheel convert on wininst installers. You do need to
manually download, run wheel convert, and host the wheels locally (a
simple directory is fine though).


Thanks Paul, for this info.

Thomas
(over to the next topic)

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup.py issue - some files are included as intended, but one is not

[Dan Stromberg]

I've redone it so it doesn't require make, and renamed the Makefile.
However, it's quite a bit slower now.  Also, "python setup.py clean"
doesn't appear to remove all the build artifacts.

Is there some way I can speed it up?

Is there some way I can automatically remove the build artifacts
without relying on make?  When they're left lying around, "svn status"
isn't quite as quick and easy.

Thanks!

PS: The link is http://stromberg.dnsalias.org/svn/treap/trunk/

On Fri, Jan 17, 2014 at 5:06 PM, Dan Stromberg  wrote:
> Hi all.
>
> Yes, it does currently require make.  I'm very comfortable with make,
> but I'm open to redoing the make rules in setup.py if that'll help get
> this problem solved.  It'd probably be less puzzling to Python people
> anyway.
>
> I kept increasing the version number as I tried different ways of
> getting setup.py to do what I wanted.  That's why the version numbers
> don't appear to match up.  If there's a better way of testing
> setup.py's, please let me know!
>
> Thanks!
>
>>hello, I checked out your project from svn.
>>looks like it requires some make commands, before it can be packaged?
>>I'm getting tripped up on that.
>>also confused in that your src tree is at 1.30, but the tar you mention is
>>1.35, and then your install dir was 1.31.
>>maybe get your src tree up to date with where your working copy, and post
>>any make commands that need to be run.
>>
>>On Tue, Jan 14, 2014 at 1:32 PM, Ethan Furman  wrote:
>>
>>> On 01/14/2014 01:26 PM, Dan Stromberg wrote:
>>>
 On Sat, Jan 11, 2014 at 2:04 PM, Dan Stromberg 
 wrote:

> Hi folks.
>
> I have a setup.py problem that's driving me nuts.
>

 Anyone?  I've received 0 responses.

>>>
>>> I have no answer, but forwarding to Distutils (hopefully it's an
>>> appropriate topic ;)
>>>
>>>
>>>  I have a treap.py file that tries to "import * from pyx_treap.so", and
> failing that, it'll "import * from py_treap.py" (sans extensions of
> course).  Naturally, all 3 of these should be included - although in
> the case of pyx_treap.so, it probably should be a .c that's included.
>
> It is also intended to include nest.py, which has a simple form.
>
> Well, treap.py, nest.py and pyx_treap.c are included fine, but I can't
> seem to get py_treap.py to be included for some reason.
>
> I get no errors during "python setup.py sdist upload", but upon
> "python $(which pip) install treap", I get:
> $ sudo /usr/bin/python $(which pip) install treap
> Downloading/unpacking treap
>Running setup.py egg_info for package treap
>
>  file py_treap.py (for module py_treap) not found
> Installing collected packages: treap
>Running setup.py install for treap
>  file py_treap.py (for module py_treap) not found
>  file py_treap.py (for module py_treap) not found
>
>  file py_treap.py (for module py_treap) not found
>  file py_treap.py (for module py_treap) not found
> Successfully installed treap
> Cleaning up...
>
> And it's not successfully installed - py_treap.py is missing.  The pyx
> code does its job, so the problem is masked, other than the messages
> above, and the absence of py_treap.py from
> /usr/local/lib/python2.7/dist-packages
>
> I can clearly see py_treap.py in ./dist/treap-1.35.tar.gz - it's at
> least getting packaged up that much.  It's not listed in
> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/SOURCES.txt
> , but I can see it in
> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/top_level.txt
> .
>
> My setup.py is at:
> http://stromberg.dnsalias.org/svn/treap/trunk/setup.py
>
> I've tried that setup.py and several variations on that theme, but
> none seem to include py_treap.py .
>
> Please make some suggestions?  How can I get py_treap.py included in
> the pip install?
>
> Thanks!
>
 ___
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>-- next part --
>>An HTML attachment was scrubbed...
>>URL: 
>>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows experience

[Paul Moore]

On 23 January 2014 11:48, Thomas Heller  wrote:
> After having used and built bdist_wininst installers for years (well, I
> should say decades) I have recently started playing with pip.

Thanks for trying this - it's really useful to have a newcomer's
perspective to the pip experience.

> However, does it work in real life?
>
> There are no wheels for important packages like pywin32, numpy, lxml,
> at least not on PyPi.  'pip install' using the source packages fails to
> build for most but the simplest stuff; probably because distutils has
> been extended too much, or because header files or libraries are
> missing, or whatever...

This is the key point - pip install from source fails on Windows for
pretty much any package with C code and non-trivial dependencies.
Building any of the above packages is murder on Windows, and it's 100%
critical to the user experience that you shouldn't have to.

> Is there a solution to this?  I've seen that the wheel tool can convert
> bdist_wininst installers into wheels - does this work for the packages
> I mentioned above?  Do we have to build or convert to wheel those
> packages, and setup a central place where we store them and make them
> available to our developers?

The good news is that "wheel convert XXX.exe" works on all the above
for creating a wheel from the wininst installer. The official numpy
installer uses some complex magic to select the right binaries based
on your CPU, and this means that the official numpy "superpack"
wininst files don't convert (at least I don't think they do, it's a
while since I tried). But happily, Christoph Gohlke hosts a huge list
of readymade wininst installers for hard-to-build projects, and the 3
you mention are all there. He's very good about building for latest
Pythons, too (3.4 is already there for many packages). Anyone working
on Windows who doesn't know his site
(http://www.lfd.uci.edu/~gohlke/pythonlibs/) should check it out.

So, to summarise, yes, you can get wheels for pretty much everything
you need by using wheel convert on wininst installers. You do need to
manually download, run wheel convert, and host the wheels locally (a
simple directory is fine though).

I'd love to see some or all of these projects host wheels themselves
on PyPI, or someone like Christoph host a pypi-style index of wheels,
so we could just point pip at that index as well as PyPI (note: I
don't know if pip works when a package is split across 2 index sites,
that's something I should check, but if it doesn't, I'd like to see
that added). But that's in the future for now - we probably need more
uptake of wheels before the demand becomes sufficient to persuade
anyone to do this.

> BTW: It seems the pip 1.4.1 doesn't download the wheel built with the
> most recent setuptools/wheel version, instead it downloads and tries
> to build from the source package.

Pip 1.4 doesn't use wheels by default - you should add "--use-wheel"
to the install command (or put it in your pip.ini). "--use-wheel is
the default from pip 1.5 onwards.

Hope this helps,
Paul.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] pip on windows experience

[Thomas Heller]

After having used and built bdist_wininst installers for years (well, I
should say decades) I have recently started playing with pip.

In theory, it is a grat solution.  Respects virtual environments
(although I haven't tested them yet), allows to freeze to a requirements
file, and so on.

It would allow to freeze and then commit a requirements file into our
version control system, add a 'pip install -r requirements.txt' to our
build scripts, and any user in out company as well as the build server
would automatically have installed all the required stuff. (We have 8
developers working on our source code, plus a build server that does
automatic builds).

However, does it work in real life?

There are no wheels for important packages like pywin32, numpy, lxml,
at least not on PyPi.  'pip install' using the source packages fails to
build for most but the simplest stuff; probably because distutils has
been extended too much, or because header files or libraries are
missing, or whatever...

Is there a solution to this?  I've seen that the wheel tool can convert
bdist_wininst installers into wheels - does this work for the packages
I mentioned above?  Do we have to build or convert to wheel those
packages, and setup a central place where we store them and make them
available to our developers?


BTW: It seems the pip 1.4.1 doesn't download the wheel built with the
most recent setuptools/wheel version, instead it downloads and tries
to build from the source package.

Thanks,
Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] error message from testpypi.python.org

[Chris Jerdonek]

On Thu, Jan 23, 2014 at 1:42 AM, Thomas Heller  wrote:
> Am 23.01.2014 10:05, schrieb Paul Moore:
>
>> On 23 January 2014 08:28, Thomas Heller  wrote:
>>>
>>> I'm trying out to submit a wheel to testpypi.python.org and then
>>> install it from there with pip.  It doesn't work; what does the
>>> following error message mean?
>>>
>>> c:\>py -3.4 -m pip install py2exe -r https://testpypi.python.org/pypi/
>>
>>
>> The flag -r is for a requirements file. Did you mean -i (--index-url)?
>
>
> Sure - thanks for that, now it works.
> My only excuse is that setup.py uses the -r flag for that purpose...

Well, that and the fact that the resulting error didn't seem to be of
much help in pointing you in the right direction.  I wonder if it
could have failed earlier with a more helpful message.

--Chris
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] error message from testpypi.python.org

[Thomas Heller]

Am 23.01.2014 10:05, schrieb Paul Moore:

On 23 January 2014 08:28, Thomas Heller  wrote:

I'm trying out to submit a wheel to testpypi.python.org and then
install it from there with pip.  It doesn't work; what does the
following error message mean?

c:\>py -3.4 -m pip install py2exe -r https://testpypi.python.org/pypi/


The flag -r is for a requirements file. Did you mean -i (--index-url)?


Sure - thanks for that, now it works.
My only excuse is that setup.py uses the -r flag for that purpose...

Thomas


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] error message from testpypi.python.org

[Paul Moore]

On 23 January 2014 08:28, Thomas Heller  wrote:
> I'm trying out to submit a wheel to testpypi.python.org and then
> install it from there with pip.  It doesn't work; what does the
> following error message mean?
>
> c:\>py -3.4 -m pip install py2exe -r https://testpypi.python.org/pypi/

The flag -r is for a requirements file. Did you mean -i (--index-url)?

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] error message from testpypi.python.org

[Thomas Heller]

I'm trying out to submit a wheel to testpypi.python.org and then
install it from there with pip.  It doesn't work; what does the
following error message mean?

c:\>py -3.4 -m pip install py2exe -r https://testpypi.python.org/pypi/
Exception:
Traceback (most recent call last):
  File "C:\Python34-64\lib\site-packages\pip\basecommand.py", line 122, 
in main

status = self.run(options, args)
  File "C:\Python34-64\lib\site-packages\pip\commands\install.py", line 
258, in run
for req in parse_requirements(filename, finder=finder, 
options=options, session=session):
  File "C:\Python34-64\lib\site-packages\pip\req.py", line 1572, in 
parse_requirements
req = InstallRequirement.from_line(line, comes_from, 
prereleases=getattr(options, "pre", None))
  File "C:\Python34-64\lib\site-packages\pip\req.py", line 172, in 
from_line

return cls(req, comes_from, url=url, prereleases=prereleases)
  File "C:\Python34-64\lib\site-packages\pip\req.py", line 70, in __init__
req = pkg_resources.Requirement.parse(req)
  File "C:\Python34-64\lib\site-packages\pip\_vendor\pkg_resources.py", 
line 2612, in parse

reqs = list(parse_requirements(s))
  File "C:\Python34-64\lib\site-packages\pip\_vendor\pkg_resources.py", 
line 2538, in parse_requirements

raise ValueError("Missing distribution spec", line)
ValueError: ('Missing distribution spec', 'encoding="utf-8"?>')


Storing debug log for failure in C:\Users\thomas\AppData\Roaming\pip\pip.log


Thanks,
Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig