Re: [Distutils] distutils.util.get_platform() - Linux vs Windows
-Original Message- From: Chris Barker - NOAA Federal [mailto:chris.bar...@noaa.gov] Sent: Monday, August 19, 2013 7:13 PM To: Ferencik, Samuel: Markets (PRG) Cc: distutils-sig@python.org Subject: Re: [Distutils] distutils.util.get_platform() - Linux vs Windows On Fri, Aug 16, 2013 at 2:18 AM, samuel.feren...@barclays.com wrote: It seems distutils.util.get_platform() semantically differs on Windows and Linux. Windows: the return value is derived from the architecture of the *interpreter*, hence for 32-bit Python running on 64-bit Windows get_platform() = 'win32' (32-bit). Linux: the return value is derived from the architecture of the *OS*, hence for 32-bit Python running on 64-bit Linux get_platform() = 'linux-x86_64' (64-bit). Is this intentional? This seems just plain wrong to me. For the record, running a 32 bit Python on a 64 bit OS_X box: In [5]: distutils.util.get_platform() Out[5]: 'macosx-10.6-i386' which is the answer I want. -Chris Chris, What does your 'uname -m' return? Is it possible you're really running a 32-bit Python on a *32-bit* OS X kernel? [http://superuser.com/q/161195] Basically, you're saying that the return value is wrong on Linux and correct on Windows, right? That get_platform() should return 32-bit for a 32-bit process running on a 64-bit system. TBH, I was expecting the opposite; to me, platform means the OS, which would mean that Linux does well to derive the return value from the OS's architecture. Sam ___ This message is for information purposes only, it is not a recommendation, advice, offer or solicitation to buy or sell a product or service nor an official confirmation of any transaction. It is directed at persons who are professionals and is not intended for retail customer use. Intended for recipient only. This message is subject to the terms at: www.barclays.com/emaildisclaimer. For important disclosures, please see: www.barclays.com/salesandtradingdisclaimer regarding market commentary from Barclays Sales and/or Trading, who are active market participants; and in respect of Barclays Research, including disclosures relating to specific issuers, please see http://publicresearch.barclays.com. ___ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distutils.util.get_platform() - Linux vs Windows
AFAIK Wheel is using that to determine binary compatibility, so if a 32bit Python on a 64bit Linux needs to be linux_32 that could be a problem. On Aug 20, 2013, at 2:15 AM, samuel.feren...@barclays.com wrote: -Original Message- From: Chris Barker - NOAA Federal [mailto:chris.bar...@noaa.gov] Sent: Monday, August 19, 2013 7:13 PM To: Ferencik, Samuel: Markets (PRG) Cc: distutils-sig@python.org Subject: Re: [Distutils] distutils.util.get_platform() - Linux vs Windows On Fri, Aug 16, 2013 at 2:18 AM, samuel.feren...@barclays.com wrote: It seems distutils.util.get_platform() semantically differs on Windows and Linux. Windows: the return value is derived from the architecture of the *interpreter*, hence for 32-bit Python running on 64-bit Windows get_platform() = 'win32' (32-bit). Linux: the return value is derived from the architecture of the *OS*, hence for 32-bit Python running on 64-bit Linux get_platform() = 'linux-x86_64' (64-bit). Is this intentional? This seems just plain wrong to me. For the record, running a 32 bit Python on a 64 bit OS_X box: In [5]: distutils.util.get_platform() Out[5]: 'macosx-10.6-i386' which is the answer I want. -Chris Chris, What does your 'uname -m' return? Is it possible you're really running a 32-bit Python on a *32-bit* OS X kernel? [http://superuser.com/q/161195] Basically, you're saying that the return value is wrong on Linux and correct on Windows, right? That get_platform() should return 32-bit for a 32-bit process running on a 64-bit system. TBH, I was expecting the opposite; to me, platform means the OS, which would mean that Linux does well to derive the return value from the OS's architecture. Sam ___ This message is for information purposes only, it is not a recommendation, advice, offer or solicitation to buy or sell a product or service nor an official confirmation of any transaction. It is directed at persons who are professionals and is not intended for retail customer use. Intended for recipient only. This message is subject to the terms at: www.barclays.com/emaildisclaimer. For important disclosures, please see: www.barclays.com/salesandtradingdisclaimer regarding market commentary from Barclays Sales and/or Trading, who are active market participants; and in respect of Barclays Research, including disclosures relating to specific issuers, please see http://publicresearch.barclays.com. ___ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
Paul wrote: Given that the installer includes the py.exe launcher, if you leave the defaults, then at a command prompt python doesn't work. But that's fine, because py does. And if you have multiple versions of Python installed, you don't *want* python on PATH, because then you have to manage your PATH. Why bother when py -2.7 or py -3.3 does what you want with no path management? Once you want any *other* executables, though, you have to deal with PATH (especially in the multiple Pythons case). That is a new issue, and one that hasn't been thought through yet, and we don't have a good solution. From a user perspective I think that 'py -3.4 -m pip ...' is an improvement as it means I can easily install or upgrade for a particular python installation (I tend to have a few). There's no need to put Scripts on PATH just to run pip. I think this should be the recommended invocation for Windows users. Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
On 20 August 2013 09:13, Oscar Benjamin oscar.j.benja...@gmail.com wrote: Paul wrote: Given that the installer includes the py.exe launcher, if you leave the defaults, then at a command prompt python doesn't work. But that's fine, because py does. And if you have multiple versions of Python installed, you don't *want* python on PATH, because then you have to manage your PATH. Why bother when py -2.7 or py -3.3 does what you want with no path management? Once you want any *other* executables, though, you have to deal with PATH (especially in the multiple Pythons case). That is a new issue, and one that hasn't been thought through yet, and we don't have a good solution. From a user perspective I think that 'py -3.4 -m pip ...' is an improvement as it means I can easily install or upgrade for a particular python installation (I tend to have a few). There's no need to put Scripts on PATH just to run pip. I think this should be the recommended invocation for Windows users. Thanks - I agree with you (IMO it would be nice to have pip.exe in PATH, but it's not important enough to change the install process). Some other points on how the various bundling approaches fit with the Windows python installer: 1. Will the bundled pip go into the system site-packages or the user site-packages? Does this depend on whether the user selects install for all users or install for just me? 2. If pip goes into system site-packages, what happens with the uninstaller? It doesn't know about pip, so it won't uninstall Python cleanly. (Not a major point, you can delete the directory manually after uninstalling, but it's untidy). Maybe the uninstaller should just unconditionally delete all of site-packages as well as whatever files it knows were installed. This is a normal issue when installing into the system Python, but for people who avoid that and use virtualenvs (e.g. me :-)) it's new (and annoying, as we'll never use the system pip in any case...) This raises another point - to an extent, I don't care about any of this, as I routinely use virtualenvs. But if using pip to manage the system python is becoming the recommended approach, I'd like to understand what precisely the recommendation is so that I can see if it's better than what I currently do - for instance, I've never used --user so I don't know if it will be of benefit to me. I assume that this will go in the packaging user guide in due course, but I don't know who will write it (does anyone have the relevant experience? most people I know recommend virtualenv...) Maybe the whole automatically bootstrapping on install should be optional (albeit likely on by default) for people who don't want to install stuff into their system python anyway? Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Back from holidays, I read this very interesting discussion... Am 14.08.2013 16:33, schrieb Nick Coghlan: Aside from the lack of embedded C extension support (which could likely be fixed if zipimport was migrated to Python code for 3.5), ...but I don't understand what you mean by this. Can you please explain? Thanks, Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
On 20 August 2013 09:51, Paul Moore p.f.mo...@gmail.com wrote: 1. Will the bundled pip go into the system site-packages or the user site-packages? Does this depend on whether the user selects install for all users or install for just me? If you have get-pip then why not choose at that point whether you want to install for the system or for all users e.g.: 'py -3.4 -m get-pip --user' (or perhaps reverse the default)? 2. If pip goes into system site-packages, what happens with the uninstaller? It doesn't know about pip, so it won't uninstall Python cleanly. (Not a major point, you can delete the directory manually after uninstalling, but it's untidy). Maybe the uninstaller should just unconditionally delete all of site-packages as well as whatever files it knows were installed. This is a normal issue when installing into the system Python, but for people who avoid that and use virtualenvs (e.g. me :-)) it's new (and annoying, as we'll never use the system pip in any case...) Can you not just teach the Python installer to check for pip and remove it if found? This raises another point - to an extent, I don't care about any of this, as I routinely use virtualenvs. But if using pip to manage the system python is becoming the recommended approach, I'd like to understand what precisely the recommendation is so that I can see if it's better than what I currently do - for instance, I've never used --user so I don't know if it will be of benefit to me. I assume that this will go in the packaging user guide in due course, but I don't know who will write it (does anyone have the relevant experience? most people I know recommend virtualenv...) If I could install everything I wanted with pip then virtualenvs would be more practical. Maybe when wheel distribution becomes commonplace I'll start doing that. I basically always want to install a large number of third party packages before I do anything though. So for me the procedure on ubuntu is something like: 1) install ubuntu 2) sudo apt-get install python-numpy python-scipy python-matplotlib ipython python-sympy python-dev cython python-pygraph python-tables python-wxgtk2.8 python-pywt python-sphinx ... On Windows the procedure is: 1) Install Python 2) Get MSIs for numpy, scipy, wxPython, matplotlib, PyQt, numexpr, ... 3) Setup PATH or create a shell/batch script called 'python' that does the right thing. 4) Run ez_setup.py and Install pip 5) Patch distutils (http://bugs.python.org/issue12641) 6) Use pip for cython, sympy, ipython, pyreadline, spyder, sphinx, docutils, line_profiler, coverage, ... 7) Build and install my own commonly used private packages. 8) Get more prebuilt binaries for other awkward packages when necessary: pytables, numexpr, mayavi, ... (You can see why some people just install Python(x, y) or EPD right?) It takes quite a while to do all this and then I have basically all the packages I want minus a few pippable ones. At this point I don't really see the point in creating a virtualenv except to test something that I'm personally developing. Or am I missing something? Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
On 20 August 2013 11:25, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On 20 August 2013 09:51, Paul Moore p.f.mo...@gmail.com wrote: 1. Will the bundled pip go into the system site-packages or the user site-packages? Does this depend on whether the user selects install for all users or install for just me? If you have get-pip then why not choose at that point whether you want to install for the system or for all users e.g.: 'py -3.4 -m get-pip --user' (or perhaps reverse the default)? That's not how get-pip is being proposed. It will run automatically on installation of Python. If it were manually run, *and* if a --user flag was included, then you would be correct. 2. If pip goes into system site-packages, what happens with the uninstaller? It doesn't know about pip, so it won't uninstall Python cleanly. (Not a major point, you can delete the directory manually after uninstalling, but it's untidy). Maybe the uninstaller should just unconditionally delete all of site-packages as well as whatever files it knows were installed. This is a normal issue when installing into the system Python, but for people who avoid that and use virtualenvs (e.g. me :-)) it's new (and annoying, as we'll never use the system pip in any case...) Can you not just teach the Python installer to check for pip and remove it if found? I'm not sure. That's my point, essentially. Who knows enough about the Windows installer to do this correctly? If the answer is nobody, then is a best-efforts approach with some issues that we don't have anyone who knows how to solve, an acceptable approach? Maybe it is, I'm not claiming that this is a major issue, but we should note it in the PEP as a limitation. This raises another point - to an extent, I don't care about any of this, as I routinely use virtualenvs. But if using pip to manage the system python is becoming the recommended approach, I'd like to understand what precisely the recommendation is so that I can see if it's better than what I currently do - for instance, I've never used --user so I don't know if it will be of benefit to me. I assume that this will go in the packaging user guide in due course, but I don't know who will write it (does anyone have the relevant experience? most people I know recommend virtualenv...) If I could install everything I wanted with pip then virtualenvs would be more practical. Maybe when wheel distribution becomes commonplace I'll start doing that. I basically always want to install a large number of third party packages before I do anything though. So for me the procedure on ubuntu is something like: 1) install ubuntu 2) sudo apt-get install python-numpy python-scipy python-matplotlib ipython python-sympy python-dev cython python-pygraph python-tables python-wxgtk2.8 python-pywt python-sphinx ... On Windows the procedure is: 1) Install Python 2) Get MSIs for numpy, scipy, wxPython, matplotlib, PyQt, numexpr, ... 3) Setup PATH or create a shell/batch script called 'python' that does the right thing. 4) Run ez_setup.py and Install pip 5) Patch distutils (http://bugs.python.org/issue12641) 6) Use pip for cython, sympy, ipython, pyreadline, spyder, sphinx, docutils, line_profiler, coverage, ... 7) Build and install my own commonly used private packages. 8) Get more prebuilt binaries for other awkward packages when necessary: pytables, numexpr, mayavi, ... (You can see why some people just install Python(x, y) or EPD right?) It takes quite a while to do all this and then I have basically all the packages I want minus a few pippable ones. At this point I don't really see the point in creating a virtualenv except to test something that I'm personally developing. Or am I missing something? :-) Yes, it's a pain. My experience is better because I don't use that many packages that need binaries. For those that I do, I maintain a local cache of wheels that I convert from bdist_wininst installers using wheel convert and then pip install works for everything. This is a really slick experience, but it relies on me maintaining my local repo, and having an appropriate -i flag added to pip (or I put it in pip.ini). As I work on multiple PCs, it's still fiddly (I put the cache on my skydrive for ease). But yes, if I made extensive use of binary extensions, I'd hate this approach. That's why I keep saying that the biggest win for wheels will be when they become the common means of distributing Windows binaries on PyPI, in place of wininst/msi. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Comments on PEP 426
Hello,
Some comments about PEP 426:
The information defined in this PEP is serialised to pydist.json files for
some use cases. These are files containing UTF-8 encoded JSON metadata.
Perhaps add that on-disk pydist.json files may/should be generated in printed
form with sorted keys, to ease direct inspection by users and developers?
Source labels MUST be unique within each project and MUST NOT match any
defined version for the project.
Is there a motivation for the not matching any defined version?
AFAICT it makes it necessary to have two different representation
schemes, e.g. X.Y.Z for source labels and vX.Y.Z for versions.
For source archive references, an expected hash value may be specified by
including a ``hash-algorithm=expected-hash`` entry as part of the URL
fragment.
Why only source archive references (and not e.g. binary)?
project_urls: {
Documentation: https://distlib.readthedocs.org;
Home: https://bitbucket.org/pypa/distlib;
Repository: https://bitbucket.org/pypa/distlib/src;
Tracker: https://bitbucket.org/pypa/distlib/issues;
}
This example lacks commas.
An abbreviation of metadistribution requires. This is a list of
subdistributions that can easily be installed and used together by
depending on this metadistribution.
I don't understand what it means :-) Care to explain and/or clarify
the purpose?
(for me, meta-requires sounds like something that setup.py depends
on for its own operation, but that the installed software doesn't need)
(edit: I now see this is clarified in Appendix C. The section ordering
in the PEP makes it look like meta_requires are the primary type of
requires, though, while according to that appendix they're a rather
exotic use case. Would be nice to spell that out *before* the appendices :-)).
* MAY allow direct references
What is a direct reference?
Automated tools MUST NOT allow strict version matching clauses or direct
references in this field - if permitted at all, such clauses should appear
in ``meta_requires`` instead.
Why so?
[test requires]
Public index servers SHOULD NOT allow strict version matching clauses or
direct references in this field.
Again, why? Is it important for public index servers that test
dependencies be not pinned?
Note that while these are build dependencies for the distribution being
built, the installation is a *deployment* scenario for the dependencies.
But there are no deployment requires, right? :)
(or is what meta requires are for?)
For example, multiple projects might supply
PostgreSQL bindings for use with SQL Alchemy: each project might declare
that it provides ``sqlalchemy-postgresql-bindings``, allowing other
projects to depend only on having at least one of them installed.
But the automated installer wouldn't be able to suggest the various
packages providing ``sqlalchemy-postgresql-bindings`` if none is
installed, which should IMO discourage such a scheme.
To handle this case in a way that doesn't allow for name hijacking, the
authors of the distribution that first defines the virtual dependency
should
create a project on the public index server with the corresponding name,
and
depend on the specific distribution that should be used if no other
provider
is already installed. This also has the benefit of publishing the default
provider in a way that automated tools will understand.
But then the alternatives needn't provide the virtual dependency.
They can just provide the default provider, which saves the time and
hassle of defining a well-known virtual dependency for all similar
projects.
A string that indicates that this project is no longer being developed.
The
named project provides a substitute or replacement.
How about a project that is no longer being developed but has no
direct substitution? :)
Can it use an empty string (or null / None perhaps?)
Examples indicating supported operating systems::
# Windows only
supports_environments: [sys_platform == 'win32']
Hmm, which syntax is it exactly? In a previous section, you used
the following example:
environment: sys.platform == 'win32'
(note dot vs. underscore)
modules: [chair, chair.cushions, (...)]
The example is a bit intriguing. Is it expected that both chair and
chair.cushions be specified there, or is chair sufficient?
When installing from an sdist, source archive or VCS checkout,
installation
tools SHOULD create a binary archive using ``setup.py bdist_wheel`` and
then install binary archive normally (including invocation of any install
hooks). Installation tools SHOULD NOT invoke ``setup.py install``
directly.
Interesting. Is setup.py install meant to die, or will it be redefined
as bdist_wheel + install_wheel?
(also, why is this mentioned in the postinstall hooks section, or
even in a metadata-related PEP?)
Installation tools SHOULD treat an exception thrown by a preuninstall
hook as an indication the removal of the distribution
Re: [Distutils] How to handle launcher script importability?
On 20 Aug 2013 04:22, Thomas Heller thel...@ctypes.org wrote: Back from holidays, I read this very interesting discussion... Am 14.08.2013 16:33, schrieb Nick Coghlan: Aside from the lack of embedded C extension support (which could likely be fixed if zipimport was migrated to Python code for 3.5), ...but I don't understand what you mean by this. Can you please explain? Importing C extensions requires extracting them to a temp directory and loading them from there. Trivial in Python, a pain in C. zipimport is currently still written in C. Cheers, Nick. Thanks, Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
On 20 Aug 2013 05:51, Paul Moore p.f.mo...@gmail.com wrote: On 20 August 2013 11:25, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On 20 August 2013 09:51, Paul Moore p.f.mo...@gmail.com wrote: 1. Will the bundled pip go into the system site-packages or the user site-packages? Does this depend on whether the user selects install for all users or install for just me? If you have get-pip then why not choose at that point whether you want to install for the system or for all users e.g.: 'py -3.4 -m get-pip --user' (or perhaps reverse the default)? That's not how get-pip is being proposed. It will run automatically on installation of Python. If it were manually run, *and* if a --user flag was included, then you would be correct. 2. If pip goes into system site-packages, what happens with the uninstaller? It doesn't know about pip, so it won't uninstall Python cleanly. (Not a major point, you can delete the directory manually after uninstalling, but it's untidy). Maybe the uninstaller should just unconditionally delete all of site-packages as well as whatever files it knows were installed. This is a normal issue when installing into the system Python, but for people who avoid that and use virtualenvs (e.g. me :-)) it's new (and annoying, as we'll never use the system pip in any case...) Can you not just teach the Python installer to check for pip and remove it if found? I'm not sure. That's my point, essentially. Who knows enough about the Windows installer to do this correctly? If the answer is nobody, then is a best-efforts approach with some issues that we don't have anyone who knows how to solve, an acceptable approach? Maybe it is, I'm not claiming that this is a major issue, but we should note it in the PEP as a limitation. This raises another point - to an extent, I don't care about any of this, as I routinely use virtualenvs. But if using pip to manage the system python is becoming the recommended approach, I'd like to understand what precisely the recommendation is so that I can see if it's better than what I currently do - for instance, I've never used --user so I don't know if it will be of benefit to me. I assume that this will go in the packaging user guide in due course, but I don't know who will write it (does anyone have the relevant experience? most people I know recommend virtualenv...) If I could install everything I wanted with pip then virtualenvs would be more practical. Maybe when wheel distribution becomes commonplace I'll start doing that. I basically always want to install a large number of third party packages before I do anything though. So for me the procedure on ubuntu is something like: 1) install ubuntu 2) sudo apt-get install python-numpy python-scipy python-matplotlib ipython python-sympy python-dev cython python-pygraph python-tables python-wxgtk2.8 python-pywt python-sphinx ... On Windows the procedure is: 1) Install Python 2) Get MSIs for numpy, scipy, wxPython, matplotlib, PyQt, numexpr, ... 3) Setup PATH or create a shell/batch script called 'python' that does the right thing. 4) Run ez_setup.py and Install pip 5) Patch distutils (http://bugs.python.org/issue12641) 6) Use pip for cython, sympy, ipython, pyreadline, spyder, sphinx, docutils, line_profiler, coverage, ... 7) Build and install my own commonly used private packages. 8) Get more prebuilt binaries for other awkward packages when necessary: pytables, numexpr, mayavi, ... (You can see why some people just install Python(x, y) or EPD right?) It takes quite a while to do all this and then I have basically all the packages I want minus a few pippable ones. At this point I don't really see the point in creating a virtualenv except to test something that I'm personally developing. Or am I missing something? :-) Yes, it's a pain. My experience is better because I don't use that many packages that need binaries. For those that I do, I maintain a local cache of wheels that I convert from bdist_wininst installers using wheel convert and then pip install works for everything. This is a really slick experience, but it relies on me maintaining my local repo, and having an appropriate -i flag added to pip (or I put it in pip.ini). As I work on multiple PCs, it's still fiddly (I put the cache on my skydrive for ease). But yes, if I made extensive use of binary extensions, I'd hate this approach. That's why I keep saying that the biggest win for wheels will be when they become the common means of distributing Windows binaries on PyPI, in place of wininst/msi. Scientific users will always be better off with something like hashdist/conda, since that ignores platform interoperability and easy security updates in favour of hash based reproducibility. Continuum Analytics also already take care of providing the prebuilt binary versions. The pip ecosystem is more appropriate for pure Python code and relatively simple C extensions
Re: [Distutils] How to handle launcher script importability?
Am 20.08.2013 15:42, schrieb Nick Coghlan: Importing C extensions requires extracting them to a temp directory and loading them from there. Trivial in Python, a pain in C. zipimport is currently still written in C. So what - zipimport is a builtin module (on Windows at least). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
Paul Moore p.f.moore at gmail.com writes: That's not how get-pip is being proposed. It will run automatically on installation of Python. If it were manually run, *and* if a --user flag was included, then you would be correct. +1 for providing a --user flag to python -m getpip. It is important to be able to install things without root access, and without having to create a venv. Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
On 20 August 2013 14:49, Nick Coghlan ncogh...@gmail.com wrote: On 20 Aug 2013 05:51, Paul Moore p.f.mo...@gmail.com wrote: But yes, if I made extensive use of binary extensions, I'd hate this approach. That's why I keep saying that the biggest win for wheels will be when they become the common means of distributing Windows binaries on PyPI, in place of wininst/msi. Scientific users will always be better off with something like hashdist/conda, since that ignores platform interoperability and easy security updates in favour of hash based reproducibility. Continuum Analytics also already take care of providing the prebuilt binary versions. Hashdist looks useful but it's for people who will build everything from source (as is basically required in the HPC environments for which it is designed). This is still problematic on Windows (which is never used for HPC). Conda looks interesting though, I'll give that a try soon. The pip ecosystem is more appropriate for pure Python code and relatively simple C extensions (including cffi bindings). The core extensions that I would want to put into each and every virtualenv are things like numpy and matplotlib. These projects have been reliably providing binary installers for Windows for years and I'm sure that they will soon be distributing wheels. The current PyPI binaries for numpy are here: https://pypi.python.org/pypi/numpy Is it not a fairly simple change to make it so that they're also uploading wheels? BTW is there any reason for numpy et al not to start distributing wheels now? Is any part of the wheel specification/tooling/infrastructure not complete yet? Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
Oscar Benjamin wrote:
Paul wrote:
Given that the installer includes the py.exe launcher, if you leave the
defaults, then at a command prompt python doesn't work. But that's fine,
because py does. And if you have multiple versions of Python installed, you
don't *want* python on PATH, because then you have to manage your PATH. Why
bother when py -2.7 or py -3.3 does what you want with no path
management?
Once you want any *other* executables, though, you have to deal with PATH
(especially in the multiple Pythons case). That is a new issue, and one that
hasn't been thought through yet, and we don't have a good solution.
From a user perspective I think that 'py -3.4 -m pip ...' is an improvement as
it means I can easily install or upgrade for a particular python installation
(I
tend to have a few). There's no need to put Scripts on PATH just to run pip. I
think this should be the recommended invocation for Windows users.
Crazy idea:
py install other args
(or 'py --install ...', but I think 'py install' is currently invalid and could
be used?)
which the launcher executes identically to:
py -m pip install other args
(Implicitly extended to other relevant commands... I'm not proposing a complete
list.)
Pros:
* allows implicit bootstrapping on first use (from a bundled pip, IMO, in case
no network is available)
* multiple Python versions are handled nicely and consistently ('py -3.3
install ...')
* can minimize officially supported API surface (as Paul described at the start
of this thread)
* pip becomes an internal implementation detail that can be entirely replaced
* one less character of typing (slightly tongue-in-cheek, but some people count
:) )
Cons:
* doesn't apply on *nix (or does/could it?)
* requires the most new code of any of the options
* more difficult to update the launcher than a user-installed package
* others that I can't think of because I'm suffering from confirmation bias?
Thoughts?
Cheers,
Steve
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
On 20 August 2013 16:09, Oscar Benjamin oscar.j.benja...@gmail.com wrote: BTW is there any reason for numpy et al not to start distributing wheels now? Is any part of the wheel specification/tooling/infrastructure not complete yet? Not really. It's up to them to do so, though. Maybe their toolset makes that more difficult, I don't believe they use setuptools, for example - that's their problem, but it may not be one they are interested in solving :-( The biggest issues outstanding are: 1. Script handling, which is a bit flaky still (but I don't think that affects numpy) 2. Tags (not in general, but AIUI numpy distribute a fancy installer that decides what compiled code to use depending on whether you have certain CPU features - they may want to retain that, and to do so may prefer to have more fine-grained tags, which in turn may or may not be possible to support). I don't think that's a critical issue though. Getting numpy et al on board would be a huge win - if wheels can satisfy their needs, we could be pretty sure we haven't missed anything. And it gets a big group of users involved, giving us a lot more real world use cases. Feel like sounding the numpy community out on the idea? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
On Tue, Aug 20, 2013 at 11:10 AM, Steve Dower steve.do...@microsoft.comwrote:
Oscar Benjamin wrote:
Paul wrote:
Given that the installer includes the py.exe launcher, if you leave the
defaults, then at a command prompt python doesn't work. But that's
fine,
because py does. And if you have multiple versions of Python
installed, you
don't *want* python on PATH, because then you have to manage your PATH.
Why
bother when py -2.7 or py -3.3 does what you want with no path
management?
Once you want any *other* executables, though, you have to deal with
PATH
(especially in the multiple Pythons case). That is a new issue, and one
that
hasn't been thought through yet, and we don't have a good solution.
From a user perspective I think that 'py -3.4 -m pip ...' is an
improvement as
it means I can easily install or upgrade for a particular python
installation (I
tend to have a few). There's no need to put Scripts on PATH just to run
pip. I
think this should be the recommended invocation for Windows users.
Crazy idea:
py install other args
(or 'py --install ...', but I think 'py install' is currently invalid and
could be used?)
which the launcher executes identically to:
py -m pip install other args
(Implicitly extended to other relevant commands... I'm not proposing a
complete list.)
Pros:
* allows implicit bootstrapping on first use (from a bundled pip, IMO, in
case no network is available)
Nick already killed this idea. Richard's original PEP proposed this and the
idea eventually was shot down.
-Brett
* multiple Python versions are handled nicely and consistently ('py -3.3
install ...')
* can minimize officially supported API surface (as Paul described at the
start of this thread)
* pip becomes an internal implementation detail that can be entirely
replaced
* one less character of typing (slightly tongue-in-cheek, but some people
count :) )
Cons:
* doesn't apply on *nix (or does/could it?)
* requires the most new code of any of the options
* more difficult to update the launcher than a user-installed package
* others that I can't think of because I'm suffering from confirmation
bias?
Thoughts?
Cheers,
Steve
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
On 20 August 2013 16:21, Paul Moore p.f.mo...@gmail.com wrote: On 20 August 2013 16:09, Oscar Benjamin oscar.j.benja...@gmail.com wrote: BTW is there any reason for numpy et al not to start distributing wheels now? Is any part of the wheel specification/tooling/infrastructure not complete yet? Not really. It's up to them to do so, though. Maybe their toolset makes that more difficult, I don't believe they use setuptools, for example - that's their problem, but it may not be one they are interested in solving :-( They seem to be using setuptools commands here: https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py#L48 https://github.com/numpy/numpy/blob/master/setupegg.py The biggest issues outstanding are: 1. Script handling, which is a bit flaky still (but I don't think that affects numpy) I think that they are responsible for installing the f2py script in each of my Scripts directories. I never use this script and I don't know what numpy wants with it (my understanding is that the Fortran parts of numpy were all shifted over to scipy). 2. Tags (not in general, but AIUI numpy distribute a fancy installer that decides what compiled code to use depending on whether you have certain CPU features - they may want to retain that, and to do so may prefer to have more fine-grained tags, which in turn may or may not be possible to support). I don't think that's a critical issue though. I guess this is what you mean: https://github.com/numpy/numpy/blob/master/tools/win32build/cpuid/test.c Is there no way for them to run a post-install script when pip installing wheels from PyPI? Getting numpy et al on board would be a huge win - if wheels can satisfy their needs, we could be pretty sure we haven't missed anything. And it gets a big group of users involved, giving us a lot more real world use cases. Feel like sounding the numpy community out on the idea? Maybe. I'm not usually on their mailing lists but I'd be willing to find out what they think. First I need to be clear that I know what I'm talking about though! Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
Brett Cannon wrote: Steve Dower wrote: Oscar Benjamin wrote: Paul wrote: Given that the installer includes the py.exe launcher, if you leave the defaults, then at a command prompt python doesn't work. But that's fine, because py does. And if you have multiple versions of Python installed, you don't *want* python on PATH, because then you have to manage your PATH. Why bother when py -2.7 or py -3.3 does what you want with no path management? Once you want any *other* executables, though, you have to deal with PATH (especially in the multiple Pythons case). That is a new issue, and one that hasn't been thought through yet, and we don't have a good solution. From a user perspective I think that 'py -3.4 -m pip ...' is an improvement as it means I can easily install or upgrade for a particular python installation (I tend to have a few). There's no need to put Scripts on PATH just to run pip. I think this should be the recommended invocation for Windows users. Crazy idea: py install other args (or 'py --install ...', but I think 'py install' is currently invalid and could be used?) which the launcher executes identically to: py -m pip install other args (Implicitly extended to other relevant commands... I'm not proposing a complete list.) Pros: * allows implicit bootstrapping on first use (from a bundled pip, IMO, in case no network is available) Nick already killed this idea. Richard's original PEP proposed this and the idea eventually was shot down. Are you referring to the whole idea or just the implicit bootstrap? I followed the discussions about the latter, and it seemed that the problem was trying to bootstrap pip using pip. This is different (bootstrap pip from py) and I believe it does not have the same issues. -Brett ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 20 Aug 2013 09:18, Thomas Heller thel...@ctypes.org wrote: Am 20.08.2013 15:42, schrieb Nick Coghlan: Importing C extensions requires extracting them to a temp directory and loading them from there. Trivial in Python, a pain in C. zipimport is currently still written in C. So what - zipimport is a builtin module (on Windows at least). Huh? That's irrelevant to the fact that doing the tempdir creation, file extraction and subsequent import entirely in C code would be painfully tedious. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to bundle pip?
On 20 Aug 2013 11:02, Steve Dower steve.do...@microsoft.com wrote: Brett Cannon wrote: Steve Dower wrote: Oscar Benjamin wrote: Paul wrote: Given that the installer includes the py.exe launcher, if you leave the defaults, then at a command prompt python doesn't work. But that's fine, because py does. And if you have multiple versions of Python installed, you don't *want* python on PATH, because then you have to manage your PATH. Why bother when py -2.7 or py -3.3 does what you want with no path management? Once you want any *other* executables, though, you have to deal with PATH (especially in the multiple Pythons case). That is a new issue, and one that hasn't been thought through yet, and we don't have a good solution. From a user perspective I think that 'py -3.4 -m pip ...' is an improvement as it means I can easily install or upgrade for a particular python installation (I tend to have a few). There's no need to put Scripts on PATH just to run pip. I think this should be the recommended invocation for Windows users. Crazy idea: py install other args (or 'py --install ...', but I think 'py install' is currently invalid and could be used?) which the launcher executes identically to: py -m pip install other args (Implicitly extended to other relevant commands... I'm not proposing a complete list.) Pros: * allows implicit bootstrapping on first use (from a bundled pip, IMO, in case no network is available) Nick already killed this idea. Richard's original PEP proposed this and the idea eventually was shot down. Are you referring to the whole idea or just the implicit bootstrap? I followed the discussions about the latter, and it seemed that the problem was trying to bootstrap pip using pip. That and potentially relying on network access at runtime, as well as potentially running into permissions issues depending on where Python was installed. Implicit bootstrap just seems like a recipe for hard to debug failure modes. This is different (bootstrap pip from py) and I believe it does not have the same issues. It certainly avoids some of them, but not all. I think Paul Moore has the right idea: treat scripts on Windows as a distinct problem deserving of a separate PEP. That general solution will then apply to pip as well. In the meantime py -m pip feels like a tolerable Windows specific workaround. Cheers, Nick. -Brett ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Am 20.08.2013 18:22, schrieb Nick Coghlan: On 20 Aug 2013 09:18, Thomas Heller thel...@ctypes.org mailto:thel...@ctypes.org wrote: Am 20.08.2013 15:42, schrieb Nick Coghlan: Importing C extensions requires extracting them to a temp directory and loading them from there. Trivial in Python, a pain in C. zipimport is currently still written in C. So what - zipimport is a builtin module (on Windows at least). Huh? That's irrelevant to the fact that doing the tempdir creation, file extraction and subsequent import entirely in C code would be painfully tedious. Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): def __load(): import imp, os path = os.path.join(__loader__.archive, --EXTENSIONS--, 'bz2.pyd') data = __loader__.get_data(path) dstpath = os.path.join(TEMPDIR, 'bz2.pyd') with open(dstpath, wb) as dll: dll.write(data) imp.load_dynamic(__name__, dstpath) __load() del __load (py2exe for Python 3, which is work in progress, uses this approach) Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distutils.util.get_platform() - Linux vs Windows
On Tue, Aug 20, 2013 at 9:00 AM, samuel.feren...@barclays.com wrote: That's strange. I'm on Python 3.3.1, and it seems to me that get_platform() derives the value from uname for OS X, similar to Linux. (osname, host, release, version, machine) = os.uname() ... elif osname[:6] == darwin: import _osx_support, distutils.sysconfig osname, release, machine = _osx_support.get_platform_osx( distutils.sysconfig.get_config_vars(), osname, release, machine) return %s-%s-%s % (osname, release, machine) so I would expect uname -m to be in line with get_plaform(). But maybe I'm misreading that... Also, I don't have access to the _osx_support source code. yup -- _osx_support.get_platform_osx() does special magic return value is wrong on Linux and correct on Windows, right? no -- I'm saying that it's right on Windows (and OS-X), but wrong on Linux. I think you have misread my sentence, and we actually agree here. duh! yes, we do. What's the next action? Report a Python bug? (That's a cultural question; I'm new to Python.) not sure -- this seems like the right place to report it, but an offical bug report may make sense. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distutils.util.get_platform() - Linux vs Windows
On Mon, Aug 19, 2013 at 11:15 PM, samuel.feren...@barclays.com wrote: What does your 'uname -m' return? x86_64 Is it possible you're really running a 32-bit Python on a *32-bit* OS X kernel? [http://superuser.com/q/161195] nope -- I am quite deliberately running a 32 bit Python on my 64 bit OS (I have some custom code C++ Im using that is not yet 64 bit safe). return value is wrong on Linux and correct on Windows, right? no -- I'm saying that it's right on Windows (and OS-X), but wrong on Linux. That get_platform() should return 32-bit for a 32-bit process running on a 64-bit system. yes, it should. TBH, I was expecting the opposite; to me, platform means the OS, which would mean that Linux does well to derive the return value from the OS's architecture. except what would be the utility of that? this is a call made within python, and it's part of distutils, so what the caller wants to know is the platform that this particular python was build for, NOT the platform is happens to be running on. i.e. what platform do I want to build binary extensions for, and/or what platform do I want to download binary wheels for. So I'm pretty sure that currently Windows and OS-X have it right, and Linux is broken. I'm guessing running 32 bit python on a 64 bit LInux is not that common, however. (and it's less common to download binaries...) To add complexity, if I run the Apple-supplied python2.7.1 (which is 32_64 bit universal, but runs 64 bit on my machine), I get: distutils.util.get_platform() 'macosx-10.7-intel' Which is more useful than it may look at first -- intel means both intel platforms, i.e. i386 and x86_64. and 10.7 means -- built for OS-X 10.7 and above. so I think it's doing the right thing. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 20, 2013 at 12:39 PM, Thomas Heller thel...@ctypes.org wrote: Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): ... (py2exe for Python 3, which is work in progress, uses this approach) Setuptools has also done this since the egg format was developed, but it has some well-known problems, which unfortunately your example has worse versions of. ;-) Setuptools takes the approach of keeping a per-user cache directory (so that cleanup isn't required, and so there are no security issues where somebody can replace a tempfile between you writing it and importing it), and it uses a unique subdirectory per egg so that different (say) bz2.pyd files can't conflict with each other. Even with these adjustments, Unix users frequently run into issues where the user a process is running as doesn't have access to a suitable cache directory, and so it's a common complaint about the use of zipped eggs. I thought that at one point you (Thomas) had come up with a way to load modules into memory from a zipfile without needing to extract them. Was that you? If so, how did that work out? (ISTR that there was some sort of licensing issue, too.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Am 20.08.2013 19:39, schrieb PJ Eby: On Tue, Aug 20, 2013 at 12:39 PM, Thomas Heller thel...@ctypes.org wrote: Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): ... (py2exe for Python 3, which is work in progress, uses this approach) Setuptools has also done this since the egg format was developed, but it has some well-known problems, which unfortunately your example has worse versions of. ;-) The code I posted was some 'pseudocode' how to extract and import an extension from a zip-file without coding it in C ;-). For example, TEMPDIR is not the usual TEMP directory, instead py2exe will use a per-process temp directory and cleanup after process exit. So, at least some of the problems you list below should be solved or solvable. Setuptools takes the approach of keeping a per-user cache directory (so that cleanup isn't required, and so there are no security issues where somebody can replace a tempfile between you writing it and importing it), and it uses a unique subdirectory per egg so that different (say) bz2.pyd files can't conflict with each other. Even with these adjustments, Unix users frequently run into issues where the user a process is running as doesn't have access to a suitable cache directory, and so it's a common complaint about the use of zipped eggs. I thought that at one point you (Thomas) had come up with a way to load modules into memory from a zipfile without needing to extract them. Was that you? If so, how did that work out? (ISTR that there was some sort of licensing issue, too.) Yes, that was me. It worked out so-so, fine for extensions from wx-python and numpy, for example, not so good for others. Until recently it did only work for win32, not win64, but this will be fixed. And, of course, it is windows only! The code it is based on is MPL2.0 licensed: https://github.com/fancycode/MemoryModule Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Thomas Heller theller at ctypes.org writes: Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): In distlib, I've built on top of the zipfile support to allow C extensions to be available. Ordinary zipfiles contain no metadata indicating the extensions available, but wheels built with distil do. The wheel handling code in distlib takes advantage of this. The Wheel.mount() API [1] takes care of this (adding the wheel to sys.path, extracting the extensions to a user-specific directory and using an import hook to call imp.load_dynamic when required, so that both Python modules and extensions are available for import). It seems to work, though when I introduced the Wheel.mount() API I was told that it was very dangerous, and the sky would fall, or something :-) Regards, Vinay Sajip [1] http://distlib.readthedocs.org/en/latest/tutorial.html#mounting-wheels ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Aug 20, 2013, at 8:57 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Thomas Heller theller at ctypes.org writes: Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): In distlib, I've built on top of the zipfile support to allow C extensions to be available. Ordinary zipfiles contain no metadata indicating the extensions available, but wheels built with distil do. The wheel handling code in distlib takes advantage of this. The Wheel.mount() API [1] takes care of this (adding the wheel to sys.path, extracting the extensions to a user-specific directory and using an import hook to call imp.load_dynamic when required, so that both Python modules and extensions are available for import). It seems to work, though when I introduced the Wheel.mount() API I was told that it was very dangerous, and the sky would fall, or something :-) Regards, Vinay Sajip [1] http://distlib.readthedocs.org/en/latest/tutorial.html#mounting-wheels ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig Mounting Wheels seems like a bad idea, it was one of the things Daniel explicitly removed (since Wheels are basically cleaned up eggs). Adding it back in ex post facto seems like it's an idea that's going down the wrong track. If Wheels are to be importable it should be enshrined in the PEP not an adhoc feature of one possible implementation. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
