Re: [Python-Dev] Python install layout and the PATH on win32
I heard back from Enthought on this part of the proposal. They could
accommodate this change.
1) The layout for the python root directory for all platforms should be
as follows:
stdlib = {base/userbase}/lib/
platstdlib = {base/userbase}/lib/
purelib = {base/userbase}/lib/site-packages
platlib = {base/userbase}/lib/site-packages
include = {base/userbase}/include/python{py_version_short}
scripts = {base/userbase}/bin
data = {base/userbase}
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On 03/23/2012 09:22 PM, PJ Eby wrote: On Mar 23, 2012 3:53 PM, Carl Meyer c...@oddbird.net On 03/23/2012 12:35 PM, PJ Eby wrote: AFAICT, virtualenvs are overkill for most development anyway. If you're not using distutils except to install dependencies, then configure distutils to install scripts and libraries to the same directory, and then do all your development in that directory. Presto! You now have a cross-platform virtualenv. Want the scripts on your path? Add that directory to your path... or if on Windows, don't bother, since the current directory is usually on the path. (In fact, if you're only using easy_install to install your dependencies, you don't even need to edit the distutils configuration, just use -md targetdir.) Creating and using a virtualenv is, in practice, _easier_ than any of those alternatives, Really? As I said, I've never seen the need to try, since just installing stuff to a directory on PYTHONPATH seems quite easy enough for me. that the isolation from system site-packages feature is quite popular (the outpouring of gratitude when virtualenv went isolated-by-default a few months ago was astonishing), and AFAIK none of your alternative proposals support that at all. What is this isolation for, exactly? If you don't want site-packages on your path, why not use python -S? (Sure, nobody knows about these things, but surely that's a documentation problem, not a tooling problem.) Don't get me wrong, I don't have any deep objection to virtualenvs, I've just never seen the *point* (outside of the scenarios I mentioned), No problem. I was just responding to the assertion that people only use virtualenvs because they aren't aware of the alternatives, which I don't believe is true. It's likely many people aren't aware of python -S, or of everything that's possible via distutils.cfg. But even if they were, for the cases where I commonly see virtualenv in use, it solves the same problems more easily and with much less fiddling with config files and environment variables. Case in point: libraries that also install scripts for use in development or build processes. If you're DIY, you have to figure out where to put these, too, and make sure it's on your PATH. And if you want isolation, not only do you have to remember to run python -S every time, you also have to edit every script wrapper to put -S in the shebang line. With virtualenv+easy_install/pip, all of these things Just Simply Work, and (mostly) in an intuitive way. That's why people use it. thus don't see what great advantage will be had by rearranging layouts to make them shareable across platforms, when throw stuff in a directory seems perfectly serviceable for that use case already. Tools that *don't* support just throw it in a directory as a deployment option are IMO unpythonic -- practicality beats purity, after all. ;-) No disagreement here. I think virtualenv's sweet spot is as a convenient tool for development environments (used in virtualenvwrapper fashion, where the file structure of the virtualenv itself is hidden away and you never see it at all). I think it's fine to deploy _into_ a virtualenv, if you find that convenient too (though I think there are real advantages to deploying just a big ball of code with no need for installers). But I see little reason to make virtualenvs relocatable or sharable across platforms. I don't think virtualenvs as on on-disk file structure make a good distribution/deployment mechanism at all. IOW, I hijacked this thread (sorry) to respond to a specific denigration of the value of virtualenv that I disagree with. I don't care about making virtualenvs consistent across platforms. Carl signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On Mon, Mar 26, 2012 at 12:58 PM, Carl Meyer c...@oddbird.net wrote: No disagreement here. I think virtualenv's sweet spot is as a convenient tool for development environments (used in virtualenvwrapper fashion, where the file structure of the virtualenv itself is hidden away and you never see it at all). I think it's fine to deploy _into_ a virtualenv, if you find that convenient too (though I think there are real advantages to deploying just a big ball of code with no need for installers). But I see little reason to make virtualenvs relocatable or sharable across platforms. I don't think virtualenvs as on on-disk file structure make a good distribution/deployment mechanism at all. IOW, I hijacked this thread (sorry) to respond to a specific denigration of the value of virtualenv that I disagree with. I don't care about making virtualenvs consistent across platforms. Well, if you're the virtualenv maintainer (or at least the PEP author), and you're basically shooting down the principal rationale for reorganizing the Windows directory layout, then it's not really much of a hijack - it's pretty darn central to the thread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On 3/26/2012 12:27 PM, PJ Eby wrote: On Mon, Mar 26, 2012 at 12:58 PM, Carl Meyer c...@oddbird.net mailto:c...@oddbird.net wrote: No disagreement here. I think virtualenv's sweet spot is as a convenient tool for development environments (used in virtualenvwrapper fashion, where the file structure of the virtualenv itself is hidden away and you never see it at all). I think it's fine to deploy _into_ a virtualenv, if you find that convenient too (though I think there are real advantages to deploying just a big ball of code with no need for installers). But I see little reason to make virtualenvs relocatable or sharable across platforms. I don't think virtualenvs as on on-disk file structure make a good distribution/deployment mechanism at all. IOW, I hijacked this thread (sorry) to respond to a specific denigration of the value of virtualenv that I disagree with. I don't care about making virtualenvs consistent across platforms. Well, if you're the virtualenv maintainer (or at least the PEP author), and you're basically shooting down the principal rationale for reorganizing the Windows directory layout, then it's not really much of a hijack - it's pretty darn central to the thread! What I read here is a bit different than Mr Eby read, it seems. I read Carl as suggesting that keeping deployment copies of virtualenvs as foolish, but thinking it is fine to deploy into a virtualenv file structure (although preferring to deplay a big ball of code, himself). Personally, I see application deployment as a big ball of code the preferred technique also, but library/module deployment is harder to do that way... it sort of defeats the ability to then bundle the library/module into the big ball of code for the application. But if the goal is to deploy a big ball of code, that would run on top of an installed Python or virtualenv Python, then that is a lot easier if the only modules used are Python modules (no C extensions). Such can be bundled into a zip file, with little support, such that a relative Python novice as myself can figure it out and implement it quickly. C extensions cannot be run from a zip file, so then one needs support code to unzip the C binaries dynamically, and (possibly) delete them when done. Or am I missing something? Hmm. And here's something else that might be missing: integration of the launcher with .py files that are actually ZIP archives... where does it find the #! line? (probably it can't, currently -- I couldn't figure out how to make it do it). Is it possible to add a #! line at the beginning of a ZIP archive for the launcher to use, and still have Python recognize the result as a ZIP archive? I know self-extracting archives put an executable program in front of a ZIP file, and the result is still recognized by most ZIP archivers, but I tried just putting a #! line followed by a ZIP archive, and Python gave me SyntaxError: unknown decode error. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On 3/26/2012 1:21 PM, Glenn Linderman wrote: Hmm. And here's something else that might be missing: integration of the launcher with .py files that are actually ZIP archives... where does it find the #! line? (probably it can't, currently -- I couldn't figure out how to make it do it). Is it possible to add a #! line at the beginning of a ZIP archive for the launcher to use, and still have Python recognize the result as a ZIP archive? I know self-extracting archives put an executable program in front of a ZIP file, and the result is still recognized by most ZIP archivers, but I tried just putting a #! line followed by a ZIP archive, and Python gave me SyntaxError: unknown decode error. OK, my first try there, I forgot the stupid Windows /b switch on copy, so apparently the ZIP archive got mangled. When I use ropy /b to join #!/usr/bin/python3.2 and a zip file, it now works. Sorry for the noise. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On Sat, Mar 24, 2012 at 4:35 AM, PJ Eby p...@telecommunity.com wrote: Just dumping things in a directory adjacent to the corresponding scripts is the original virtualenv, and it still works just dandy -- most people just don't *know* this. (And again, if there are tools out there that *don't* support single-directory virtualenvs, the best answer is probably to fix them.) Not to mention that CPython gained native support for that layout in 2.6 via __main__.py files (although I stuffed up and forgot to add it to the What's New before the release). I'll chime in on the -1 side here as well. If you want *easy* cross-platform execution of __main__, use the -m switch. I'm obviously biased, since I'm the original author and primary maintainer of that switch, but it just makes all these problems with cross-platform questions and running from an installed copy vs running from source *go away*. Indeed, avoiding such cross-platform inconsistencies with regards to the location of stdlib modules was one of the major arguments in favour of adding the original incarnation of the switch way back in Python 2.4. To run the main clients (one for repo management, one for Django site management) in my current work project, I use python -m pulpdist.manage_repos and python -m pulpdist.manage_site. It works from a source checkout (so long as I cd into src/ first), on an installed version, in a virtualenv, anywhere. I can easily run it on a different Python version just by changing the version I invoke. The commands would probably also work on at least Mac OS X and maybe even Windows (although I've never actually tried either of those, since PulpDist is targeted specifically at Linux systems). I may get around to installing at least the repo management client as a real script some day (since it will be more convenient for system administrators that way), but direct execution will never be the main way of executing it from a source checkout. So Van's proposal still smacks too much to me of change for change's sake. If you want an execution mechanism that is completely consistent across platforms (including virtual environments), then -m already exists. For direct execution, the proposal trades cross-version inconsistencies for cross-platform consistency. When we *already have* a consistent cross-platform mechanism in -m, the inevitable disruption involved in changing the Windows layout seems very hard to justify. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On Sat, Mar 24, 2012 at 07:19, Nick Coghlan ncogh...@gmail.com wrote: On Sat, Mar 24, 2012 at 4:35 AM, PJ Eby p...@telecommunity.com wrote: Just dumping things in a directory adjacent to the corresponding scripts is the original virtualenv, and it still works just dandy -- most people just don't *know* this. (And again, if there are tools out there that *don't* support single-directory virtualenvs, the best answer is probably to fix them.) Not to mention that CPython gained native support for that layout in 2.6 via __main__.py files (although I stuffed up and forgot to add it to the What's New before the release). I'll chime in on the -1 side here as well. If you want *easy* cross-platform execution of __main__, use the -m switch. I love the -m option but what does it have to do with unifying the install layout? One is about executing __main__ and one is about a directory structure. Indeed, avoiding such cross-platform inconsistencies with regards to the location of stdlib modules was one of the major arguments in favour of adding the original incarnation of the switch way back in Python 2.4. Ok, so it is about directory structure, but about the standard library. Since part of this proposal is about Scripts vs. bin, how does -m help you there? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
By dodging the issue entirely - anything I might want to regularly run from a source checkout I execute with -m. It gets sys.path right automatically and I don't need to care about platform specific executable naming conventions. -- Sent from my phone, thus the relative brevity :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)
On 23 March 2012 03:20, Brian Curtin br...@python.org wrote: Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden have all expressed that they have existing tools that would break and would need to be adjusted to match the new location of the python.exe, because that location is assumed to be at the root of the python install. Isn't the proposed BinaryDir registry key helpful here? It's not like we're telling people to fend for themselves -- we'll tell you where it's at. It won't help me much. I either check a key and fall back on the old method, or check in bin and fall back on the old method. No major difference. The key is slightly worse, as I'm already looking in the filesystem, so why open a registry key, but it's mostly irrelevant. I still don't really get how this portion of the proposal, the python.exe move to bin, is holding people up. If you're using the launcher, the change is invisible. If you're using a setup where bin is on the Path, the change is invisible. File associations? Invisible. If you're typing out the full path, you have to type bin in the middle -- this kind of sucks but I think we'll live. Agreed, it's irrelevant for end users. It's only going to affect tools. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
VanL wrote:
As this has been brought up a couple times in this subthread, I figured that
I would lay out the
rationale here.
There are two proposals on the table: 1) Regularize the install layout, and
2) move the python
binary to the binaries directory. This email will deal with the first, and a
second email will deal
with the second.
1) Regularizing the install layout:
One of Python's strengths is its cross-platform appeal. Carefully-
written Python programs are frequently portable between operating
systems and Python implementations with very few changes. Over the
years, substantial effort has been put into maintaining platform
parity and providing consistent interfaces to available functionality,
even when different underlying implementations are necessary (such
as with ntpath and posixpath).
One place where Python is unnecessarily different, however, is in
the layout and organization of the Python environment. This is most
visible in the name of the directory for binaries on the Windows platform
(Scripts) versus the
name of the directory for binaries on every other platform (bin), but a
full listing of the
layouts shows
substantial differences in layout and capitalization across platforms.
Sometimes the include is capitalized (Include), and sometimes not; and
the python version may or may not be included in the path to the
standard library or not.
This may seem like a harmless inconsistency, and if that were all it was, I
wouldn't care. (That
said, cross-platform consistency is its own good). But it becomes a real pain
when combined with
tools like virtualenv or the new pyvenv to create cross-platform development
environments.
In particular, I regularly do development on both Windows and a Mac, and then
deploy on Linux. I do
this in virtualenvs, so that I have a controlled and regular environment. I
keep them in sync using
source control.
The problem comes when I have executable scripts that I want to include in my
dvcs - I can't have it
in the obvious place - the binaries directory - because *the name of the
directory changes when you
move between platforms.* More concretely, I can't hg add Scripts/runner.py?
on my windows
environment (where it is put in the PATH by virtualenv) and thendo a pull on
Mac or Linux and have
it end up properly in bin/runner.py which is the correct PATH for those
platforms.
This applies anytime there are executable scripts that you want to manage
using source control
across platforms. Django projects regularly have these, and I suspect we will
be seeing more of this
with the new project support in virtualenvwrapper.
While a few people have wondered why I would want this -- hopefully answered
above -- I have not
heard any opposition to this part of the proposal.
This first proposal is just to make the names of the directories match across
platforms. There are
six keys defined in the installer files (sysconfig.cfg and
distutils.command.install): 'stdlib',
'purelib', 'platlib', 'headers', 'scripts', and 'data'.
Currently on Windows, there are two different layouts defined:
'nt': {
'stdlib': '{base}/Lib',
'platstdlib': '{base}/Lib',
'purelib': '{base}/Lib/site-packages',
'platlib': '{base}/Lib/site-packages',
'include': '{base}/Include',
'platinclude': '{base}/Include',
'scripts': '{base}/Scripts',
'data' : '{base}',
},
'nt_user': {
'stdlib': '{userbase}/Python{py_version_nodot}',
'platstdlib': '{userbase}/Python{py_version_nodot}',
'purelib': '{userbase}/Python{py_version_nodot}/site-packages',
'platlib': '{userbase}/Python{py_version_nodot}/site-packages',
'include': '{userbase}/Python{py_version_nodot}/Include',
'scripts': '{userbase}/Scripts',
'data' : '{userbase}',
},
The proposal is to make all the layouts change to:
'nt': {
'stdlib': '{base}/lib',
'platstdlib': '{base}/lib',
'purelib': '{base}/lib/site-packages',
'platlib': '{base}/lib/site-packages',
'include': '{base}/include',
'platinclude': '{base}/include',
'scripts': '{base}/bin',
'data' : '{base}',
},
The change here is that 'Scripts' will change to 'bin' and the capitalization
will be removed. Also,
user installs of Python will have the same internal layout as system
installs of Python. This
will also, not coincidentally, match the install layout for posix, at least
with regard to the
'bin', 'lib', and 'include' directories.
Again, I have not heard *anyone* objecting to this part of the proposal as it
is laid out here.
(Paul had a concern with the lib directory earlier, but he said he was ok
with the above).
Please let me know if you have any problems or concerns with this part 1.
Since userbase will usually be a single directory in the home
dir of a user, the above would loose the possibility to support
multiple Python versions
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)
On 23/03/2012 7:10 PM, Paul Moore wrote: On 23 March 2012 03:20, Brian Curtinbr...@python.org wrote: Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden have all expressed that they have existing tools that would break and would need to be adjusted to match the new location of the python.exe, because that location is assumed to be at the root of the python install. Isn't the proposed BinaryDir registry key helpful here? It's not like we're telling people to fend for themselves -- we'll tell you where it's at. It won't help me much. I either check a key and fall back on the old method, or check in bin and fall back on the old method. No major difference. The key is slightly worse, as I'm already looking in the filesystem, so why open a registry key, but it's mostly irrelevant. That's a really good point. On reflection, the 2 tools I've been using as examples are already sniffing around the file-system relative to the install path, looking in the root and the PCBuild directories. The simplest approach for these tools to take is to simply sniff the bin directory too - so they are unlikely to refer to the BinaryDir key at all. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)
On 22 March 2012 23:15, VanL van.lindb...@gmail.com wrote: Another use case was just pointed out to me: making things consistent with buildout. Given a similar use case (create repeatable cross platform environments), they create and use a 'bin' directory for executable files. Another problem case: cx_Freeze. This currently breaks when installed in a viretualenv, as it locates the Scripts directory by appending Scripts to the directory of the python executable. So the proposed change *will* break cx_Freeze. The Scripts-bin change will also break it. Paul. PS Yes, I need to report the existing bug. The point remains, however... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On Friday, March 23, 2012 at 11:39 AM, PJ Eby wrote: Even if you are using tools that don't use distutils' configuration settings for these directories, why not simply fix those tools so that they do? Thats what I do currently - I set things to bin and patch Python and the tools so that they work. However, have considered this to be a little bit of a wart anyway for a long time - even before I adopted my current method of working - because it is a pointless inconsistency. The fact that it makes virtual environments consistent across platforms, together with pyvenv going into 3.3, gave me enough of a push to elevate this from private annoyance to should fix. Thanks, Van ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On Friday, March 23, 2012 at 1:35 PM, PJ Eby wrote: Tool developers are going meh about your proposal because it doesn't actually solve any problems for them: they still have to support the old layout, and if their code already uses distutils' facilities for obtaining paths, there's nothing they gain from the change. Three notes. FIrst, distutils.cfg doesn't always work because it is centered around the idea of set paths that are the same each time - which doesn't always work with virtualenvs. Further, a number of tools find that it doesn't work (haven't seen it myself, but look at the comments in pypm's installer). So yes, I patch python. Second, most installer tools don't follow distutils.cfg. Even if that helps for python setup.py install, the other tools are still broken when you want to specify a layout. That is why changing the defaults is the only effective way to make this change - because the defaults drive what is actually implemented. I know, because I have looked at and patched these tools to make them work. Third, there are some tool makers going meh - because you are right, this is not a problem they have. This is a problem associated with using those tools. And regardless of there being other ways to do it, including your 'dump it in a directory' method, development in virtualenvs is convenient, widespread, and on the rise. Given that pyvenv will go into 3.3, it will be the 'one obvious way to do it' - making going-forward cross-platform compatibility a positive good. Again I note the example of buildout. And fourth, (because nobody expects the spanish inquisition), isn't the gratuitous difference a (small but) obvious wart? Does anybody positively like 'Scripts'? The most common comment I have received when talking to people off this list has been, 'yeah, that was always sort of weird.'___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On Mar 23, 2012 4:19 PM, VanL van.lindb...@gmail.com wrote: Three notes. FIrst, distutils.cfg doesn't always work because it is centered around the idea of set paths that are the same each time - which doesn't always work with virtualenvs. And the virtualenv doesn't contain its own copy of distutils.cfg? Second, most installer tools don't follow distutils.cfg. Even if that helps for python setup.py install, the other tools are still broken when you want to specify a layout. So, we should change Python to fix the broken tools that don't follow documented standards for configuring installation locations? If the tools are that broken, aren't they going to break even *harder* when you change the paths for Windows? And fourth, (because nobody expects the spanish inquisition), isn't the gratuitous difference a (small but) obvious wart? It's hardly the only wart we keep around for backwards compatibility. If it's going to change, it needs a proper transition period at the least. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On Mar 23, 2012 3:53 PM, Carl Meyer c...@oddbird.net wrote: Hi PJ, On 03/23/2012 12:35 PM, PJ Eby wrote: AFAICT, virtualenvs are overkill for most development anyway. If you're not using distutils except to install dependencies, then configure distutils to install scripts and libraries to the same directory, and then do all your development in that directory. Presto! You now have a cross-platform virtualenv. Want the scripts on your path? Add that directory to your path... or if on Windows, don't bother, since the current directory is usually on the path. (In fact, if you're only using easy_install to install your dependencies, you don't even need to edit the distutils configuration, just use -md targetdir.) Creating and using a virtualenv is, in practice, _easier_ than any of those alternatives, Really? As I said, I've never seen the need to try, since just installing stuff to a directory on PYTHONPATH seems quite easy enough for me. that the isolation from system site-packages feature is quite popular (the outpouring of gratitude when virtualenv went isolated-by-default a few months ago was astonishing), and AFAIK none of your alternative proposals support that at all. What is this isolation for, exactly? If you don't want site-packages on your path, why not use python -S? (Sure, nobody knows about these things, but surely that's a documentation problem, not a tooling problem.) Don't get me wrong, I don't have any deep objection to virtualenvs, I've just never seen the *point* (outside of the scenarios I mentioned), and thus don't see what great advantage will be had by rearranging layouts to make them shareable across platforms, when throw stuff in a directory seems perfectly serviceable for that use case already. Tools that *don't* support just throw it in a directory as a deployment option are IMO unpythonic -- practicality beats purity, after all. ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On Mar 23, 2012 10:21 PM, PJ Eby p...@telecommunity.com wrote: On Mar 23, 2012 4:19 PM, VanL van.lindb...@gmail.com wrote: Three notes. FIrst, distutils.cfg doesn't always work because it is centered around the idea of set paths that are the same each time - which doesn't always work with virtualenvs. And the virtualenv doesn't contain its own copy of distutils.cfg? It can, but a new one. Virtualenvs don't carry over the distutils.cfg from the main installation. Thus, using distutils.cfg in the virtualenv would require editing the .cfg for every new virtualenv-and it still wouldn't work all the time for the other reasons discussed. Second, most installer tools don't follow distutils.cfg. Even if that helps for python setup.py install, the other tools are still broken when you want to specify a layout. So, we should change Python to fix the broken tools that don't follow documented standards for configuring installation locations? If the documented functions don't work for the use cases, there is nothing else. Again, see the comments in PyPM. If the tools are that broken, aren't they going to break even *harder* when you change the paths for Windows? If people substitute on hard-coded value for another, does cross platform consistency help? And ifthat focuses attention on the new packaging APIs and the correct way to do it, isn't that even better? And fourth, (because nobody expects the spanish inquisition), isn't the gratuitous difference a (small but) obvious wart? It's hardly the only wart we keep around for backwards compatibility. If it's going to change, it needs a proper transition period at the least. Already proposed, making a transition over three releases with it starting as an off by default option in 3.3. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 21/03/2012 23:03, Paul Moore wrote: On 21 March 2012 22:43, Mark Hammondskippy.hamm...@gmail.com wrote: On 22/03/2012 1:22 AM, Lindberg, Van wrote: Mark, MAL, Martin, Tarek, Could you comment on this? Eric is correct - tools will be broken by this change. However, people seem willing to push forward on this and accept such breakage as the necessary cost. MAL, in his followup, asks what the advantages are of such a change. I've actually been asking for the same thing in this thread and the only real answer I've got is consistency. So while I share MAL's concerns, people seem willing to push forward on this anyway, without the benefits having been explained. IOW, this isn't the decision I would make, but I think I've already made that point a number of times in this thread. Beyond that, there doesn't seem much for me to add... I agree on all points here. I don't understand quite why backward compatibility is being treated so lightly here. But equally, I've made my points and have little further to add. Well I've gone through (and deleted) three draft contributions to the ideas proposed here over the last week or so. In short, I'm with Paul Mark. The OP seems far more casual towards breakage than would be the case if, eg, code were involved. If this had been proposed for Python 3k I'd have said: go for it - why not? But for this to drop in now means, as others have said, that I'll have to adjust various small tools which assume the location of python.exe according to the (minor) version I'm running. I can certainly cope with the change and without too much difficulty, but I'm afraid it does smack of a too foolish consistency. And it's not as though I've seen crowds of people chiming in with a me-too! The only person strongly supporting the change (as distinct from not opposing it) is VanL, who appears to need it for his particular setup. In short, I'm -1 but I'm not going to storm off in a huff if it goes ahead, merely be a little bewildered at why this was needed by anyone else and exactly what real-world problem it's solving for thousands of Windows Python users. TJG ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
As this has been brought up a couple times in this subthread, I figured
that I would lay out the rationale here.
There are two proposals on the table: 1) Regularize the install layout,
and 2) move the python binary to the binaries directory. This email will
deal with the first, and a second email will deal with the second.
1) Regularizing the install layout:
One of Python's strengths is its cross-platform appeal. Carefully-
written Python programs are frequently portable between operating
systems and Python implementations with very few changes. Over the
years, substantial effort has been put into maintaining platform
parity and providing consistent interfaces to available functionality,
even when different underlying implementations are necessary (such
as with ntpath and posixpath).
One place where Python is unnecessarily different, however, is in
the layout and organization of the Python environment. This is most
visible in the name of the directory for binaries on the Windows
platform (Scripts) versus the name of the directory for binaries on
every other platform (bin), but a full listing of the layouts shows
substantial differences in layout and capitalization across platforms.
Sometimes the include is capitalized (Include), and sometimes not; and
the python version may or may not be included in the path to the
standard library or not.
This may seem like a harmless inconsistency, and if that were all it
was, I wouldn't care. (That said, cross-platform consistency is its own
good). But it becomes a real pain when combined with tools like
virtualenv or the new pyvenv to create cross-platform development
environments.
In particular, I regularly do development on both Windows and a Mac, and
then deploy on Linux. I do this in virtualenvs, so that I have a
controlled and regular environment. I keep them in sync using source
control.
The problem comes when I have executable scripts that I want to include
in my dvcs - I can't have it in the obvious place - the binaries
directory - because *the name of the directory changes when you move
between platforms.* More concretely, I can't hg add Scripts/runner.py?
on my windows environment (where it is put in the PATH by virtualenv)
and thendo a pull on Mac or Linux and have it end up properly in
bin/runner.py which is the correct PATH for those platforms.
This applies anytime there are executable scripts that you want to
manage using source control across platforms. Django projects regularly
have these, and I suspect we will be seeing more of this with the new
project support in virtualenvwrapper.
While a few people have wondered why I would want this -- hopefully
answered above -- I have not heard any opposition to this part of the
proposal.
This first proposal is just to make the names of the directories match
across platforms. There are six keys defined in the installer files
(sysconfig.cfg and distutils.command.install): 'stdlib', 'purelib',
'platlib', 'headers', 'scripts', and 'data'.
Currently on Windows, there are two different layouts defined:
'nt': {
'stdlib': '{base}/Lib',
'platstdlib': '{base}/Lib',
'purelib': '{base}/Lib/site-packages',
'platlib': '{base}/Lib/site-packages',
'include': '{base}/Include',
'platinclude': '{base}/Include',
'scripts': '{base}/Scripts',
'data' : '{base}',
},
'nt_user': {
'stdlib': '{userbase}/Python{py_version_nodot}',
'platstdlib': '{userbase}/Python{py_version_nodot}',
'purelib': '{userbase}/Python{py_version_nodot}/site-packages',
'platlib': '{userbase}/Python{py_version_nodot}/site-packages',
'include': '{userbase}/Python{py_version_nodot}/Include',
'scripts': '{userbase}/Scripts',
'data' : '{userbase}',
},
The proposal is to make all the layouts change to:
'nt': {
'stdlib': '{base}/lib',
'platstdlib': '{base}/lib',
'purelib': '{base}/lib/site-packages',
'platlib': '{base}/lib/site-packages',
'include': '{base}/include',
'platinclude': '{base}/include',
'scripts': '{base}/bin',
'data' : '{base}',
},
The change here is that 'Scripts' will change to 'bin' and the
capitalization will be removed. Also, user installs of Python will
have the same internal layout as system installs of Python. This will
also, not coincidentally, match the install layout for posix, at least
with regard to the 'bin', 'lib', and 'include' directories.
Again, I have not heard *anyone* objecting to this part of the proposal
as it is laid out here. (Paul had a concern with the lib directory
earlier, but he said he was ok with the above).
Please let me know if you have any problems or concerns with this part 1.
Thanks,
Van
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)
[PART 2: Moving the python binary] There are two proposals on the table: 1) Regularize the install layout, and 2) move the python binary to the binaries directory. This email deals with the second issue exclusively. This has been the more contentious issue. 2) Moving the Python exe: A regular complaint of those new to Python on windows (and new to programming generally) has been that one of the first things that they need to do is to edit the PATH to allow Python to be run. In particular, this is the difficult sequence: 1. Install python. 2. Open up a shell and run python 3. Use pip or easy_install to install regetron (a package that installs an executable file). 4. Run regetron. For step #2, the python exe needs to be on the PATH. For steps 3 and 4, the binaries directory needs to be on the PATH. Currently, neither of these are true, so the path needs to be edited to include both the python root (where python.exe is, for step 2) and the Scripts (hopefully soon bin) directory where pip and regetron are (for steps 3 and 4). You can substitute regetron for nose, cython, or other packages as well. MvL asked why anyone would want to run python directly from a cmd shell instead of running it from the start menu. There are two immediate responses to that: 1) observed behavior is that people prefer to run python from the cmd shell when developing (as observed by various people teaching Python, including Brian Curtin in this thread), and 2) running python or python programs from the shell is sometimes the only way to get a proper traceback when developing, making it a better way to work. The proposal here is to move the python.exe into the binaries directory (whatever it is called) and add an option to the windows installer to add that one directory to the PATH on install (and clean up the PATH on uninstall). A new registry key would be added pointing to the location of the python binary (wherever it is). Brian Curtin suggested this part of the proposal and has implemented it in a branch. MvL suggested a gradual transition to this over a three-release period. Open Issues: The PEP 397 Installer: As pointed out by Paul Moore, it may not matter once PEP 397 lands if python.exe is in the PATH or not - and it may be better if it is not. As he put it: If we do put python.exe on PATH (whether it's in bin or not), we have to debate how to handle people having multiple versions of python on their machine. In a post-PEP 397 world, no Python is the machine default - .py files are associated with py.exe, not python.exe, so we have to consider the following 3 commands being run from a shell prompt: 1. myprog.py 2. py myprog.py 3. python myprog.py 1 and 2 will always do the same thing. However, 3 could easily do something completely different, if the Python in the #! line differs from the one found on PATH. To me, this implies that it's better for (3) to need explicit user action (setting PATH) if it's to do anything other than give an error. But maybe that's just me. I've been hit too often by confusion caused by *not* remembering this fact. One possible response here is that the moving of the python.exe binary and the setting of the PATH would be tied to an unchecked-by-default installer option, making an explicit user choice needed to invoke the new functionality. Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden have all expressed that they have existing tools that would break and would need to be adjusted to match the new location of the python.exe, because that location is assumed to be at the root of the python install. A related issue is that this portion of the proposal has met with some resistance, but not much support here on Python-dev. The reason for that is selection bias: Those who are on Python-dev are much more likely to have tools that do advanced things with Python, such as introspect on the location of the binary, and are also much more likely to be comfortable with things like editing the PATH on windows. In contrast, the people that have trouble with this issue are those that are newest to Python and programming generally - those for whom editing the PATH is a challenge and whom are likely to be confused by the distinction between python.exe and a python program - and why, even after they add python to the path, the python program is not directly executable. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On 22 March 2012 14:17, VanL van.lindb...@gmail.com wrote: As this has been brought up a couple times in this subthread, I figured that I would lay out the rationale here. I'm repeating myself here after I promised not to. My apologies, but I don't think this posting captures the debate completely. One reason I suggested a PEP is to better ensure that the arguments both pro and con were captured, as that is a key part of the PEP process. One place where Python is unnecessarily different, however, is in the layout and organization of the Python environment. This is most visible in the name of the directory for binaries on the Windows platform (Scripts) versus the name of the directory for binaries on every other platform (bin), First of all, this difference is almost entirely *invisible*. Apart from possibly setting PATH (once!) users should not be digging around in the Python installation directory. Certainly on Windows, it is a very unusual application that expects users to even care how the application is laid out internally. And I suspect that is also true on Unix and Mac. Secondly, the layouts are not as similar as you claim here, if I understand things correctly. on Unix Python is installed in /usr/local/bin so there isn't even a Python installation directory there. And Macs use some sort of Mac-specific bundle technology as I understand it. To be honest, I don't think that there's a lot of similarity even with your proposed changes. but a full listing of the layouts shows substantial differences in layout and capitalization across platforms. That's true, but largely incidental. And given that (a) Windows is case insensitive and (b) the capitalisation, although inconsistent, follows platform standards (Unix all lowercase, Windows capitalised) it makes little practical difference. Sometimes the include is capitalized (Include), and sometimes not; and the python version may or may not be included in the path to the standard library or not. Given that on Windows the Python library is usually in something like C:\Python32\Lib whereas on Unix it's in /usr/lib/python3.2 (I think), the difference is reasonable because the Python *base* location (C:\Python32 on Windows vs /usr on Unix) is version specific in one case but not the other. To keep the correspondence complete, you should be suggesting installing in /python32 on Unix (which I doubt would gain much support :-)) This may seem like a harmless inconsistency, and if that were all it was, I wouldn't care. (That said, cross-platform consistency is its own good). But it becomes a real pain when combined with tools like virtualenv or the new pyvenv to create cross-platform development environments. The issue with virtualenv and pyvenv may be more significant. But you're only mentioning those incidentally. As a straw-man suggestion, why can virtualenv not be changed to maintain a platform-neutral layout in spite of what core Python does? This is a straw-man because I know the answer, but can we get the point out in the open - it's related to how distutils installs code, and that in turn hits you straight up against the distutils freeze. If distutils' behaviour is the issue here, then argue for more flexibility in packaging, and use that extra flexibility to argue for changes to virtualenv and pyvenv to maintain a standard cross-platform layout. Breaking the Python installation layout isn't the only option here, and I'd like to see a clear analysis of the tradeoffs. (I also get a sense of undue haste - we can change the Python layout for 3.3, but changing packaging and virtualenv is a much longer process...) In particular, I regularly do development on both Windows and a Mac, and then deploy on Linux. I do this in virtualenvs, so that I have a controlled and regular environment. I keep them in sync using source control. The problem comes when I have executable scripts that I want to include in my dvcs - I can't have it in the obvious place - the binaries directory - because *the name of the directory changes when you move between platforms.* More concretely, I can't hg add Scripts/runner.py? on my windows environment (where it is put in the PATH by virtualenv) and thendo a pull on Mac or Linux and have it end up properly in bin/runner.py which is the correct PATH for those platforms. This presupposes that your development workflow - developing in place in the virtualenv itself - is the obvious approach. From what I've seen of tools like virtualenvwrapper, you're not alone in this. And I'm pretty new to using virtualenv so I wouldn't like to claim to be any expert. But can I respectfully suggest that other ways of working wouldn't hit these issues? WhatI do is develop my project in a project specific directory, just as I would if I were using the system Python. And I have an activated virtualenv *located somewhere else* that I install required 3rd party modules in, etc. I then do standard pip install to install and test
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
Hi Paul, To start with, I appreciate your comments, and it is worth having both sides expressed. On 3/22/2012 12:59 PM, Paul Moore wrote: I'm repeating myself here after I promised not to. My apologies, but I don't think this posting captures the debate completely. One reason I suggested a PEP is to better ensure that the arguments both pro and con were captured, as that is a key part of the PEP process. I would be happy to write up a PEP. First of all, this difference is almost entirely *invisible*. Apart from possibly setting PATH (once!) users should not be digging around in the Python installation directory. Certainly on Windows, it is a very unusual application that expects users to even care how the application is laid out internally. And I suspect that is also true on Unix and Mac. This is a good point; it is mostly visible in the virtualenvs. If it only changed in virtualenvs, I would be happy. The policy, though, is that the virtualenv follows the platform policy. Secondly, the layouts are not as similar as you claim here, if I understand things correctly. on Unix Python is installed in /usr/local/bin so there isn't even a Python installation directory there. And Macs use some sort of Mac-specific bundle technology as I understand it. To be honest, I don't think that there's a lot of similarity even with your proposed changes. I was summarizing here because, frankly, there are hardly any OS/2 users, so it would be mostly Windows users affected by this change. Also as noted, I suggest that all platforms standardize on bin, lib, and include, just as I laid out. That said, while I think that the above is a good idea, my personal ambitions are more modest: If the names of the top-level directories only were changed to 'bin', 'lib', and 'include' - never mind differences under 'lib' - I would be happy. In fact, even the one change of 'Scripts' to 'bin' everywhere would get 90% of my uses. The issue with virtualenv and pyvenv may be more significant. But you're only mentioning those incidentally. I am approaching it from the platform level because of the policy that virtualenvs match the platform install layout. If instead virtualenv layouts were standardized, that would end up making me just as happy. (I also get a sense of undue haste - we can change the Python layout for 3.3, but changing packaging and virtualenv is a much longer process...) Honestly, I didn't expect that much resistance. None of the people I talked to in person even cared, or if they did, they thought that consistency was a benefit. But now that virtualenvs are going in in 3.3, I see this as the last good chance to change this. This presupposes that your development workflow - developing in place in the virtualenv itself - is the obvious approach. From what I've seen of tools like virtualenvwrapper, you're not alone in this. [...] But can you acknowledge (and document) that change your way of working is another alternative to change Python. Acknowledged. What you say is true - but people wanted to know what the benefit would be. I laid out my concrete use-case as a rationale. And as you note, I am not alone in this type of development. Sure, I care here because it affects my style of development, and there are other styles that have other benefits (and tradeoffs). I don't see that this part of the proposal would negatively affect those styles. While a few people have wondered why I would want this -- hopefully answered above -- I have not heard any opposition to this part of the proposal. See above. There has been opposition from a number of people. It's relatively mild, simply because it's a niche area and doesn't cause huge pain, but it's there. And you seem (based on the above analysis) to be overstating the benefits, so the summary here is weighted heavily in favour of change. If I have misrepresented anyone, I am sorry - but to the best of my understanding, no one had (prior to you, right now) objected to *this part* of the proposal. Mark, at least, specified that his concern was with the moving of the python binary and that he didn't care about this part. I believe Tim indicated that too, but perhaps I have on my rose-colored glasses and misunderstood him. Also, you have made no attempt that I've seen to address the question of why this is important enough to break backward compatibility. Maybe it is - but why? Backward compatibility is a very strong rule, and should be broken only with good justification. Consistency, and it makes my way of working easier really shouldn't be sufficient. In general, yes, I agree with you. However, the break with backwards compatibility is, as you point out, minor, and there is a benefit to consistency - especially given virtualenv-centric development. Has anyone checked whether this will affect people like Enthought and ActiveState who distribute their own versions of Python? Is ActiveState's PPM tool affected?
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)
Another use case was just pointed out to me: making things consistent with buildout. Given a similar use case (create repeatable cross platform environments), they create and use a 'bin' directory for executable files. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
(resending, only sent to Van the first time) FWIW, I avoid the directory naming problems Van describes entirely by including my scripts in the source package and running them with the -m switch. So python -m pulpdist.manage_site, for example, is PulpDist's Django administration client wrapper. I do it that way mainly to get sys.path right automatically, but it avoids several other installed vs checked out differences too. -- Sent from my phone, thus the relative brevity :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)
Given the cost of the change, and the advent of the PEP-397 Launcher, I also vote -1. ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On Thu, Mar 22, 2012 at 12:59, Paul Moore p.f.mo...@gmail.com wrote: Note - that is not Regularizing the layout. You have not made any changes to OS/2 (which matches Windows at the moment). I think that would be a wasted effort with OS/2 entering unsupported mode in 3.3, and OS/2 specific code being removed in 3.4. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)
On Thu, Mar 22, 2012 at 13:57, VanL van.lindb...@gmail.com wrote: Honestly, I didn't expect that much resistance. None of the people I talked to in person even cared, or if they did, they thought that consistency was a benefit. But now that virtualenvs are going in in 3.3, I see this as the last good chance to change this. I was one of these people, first finding out just about the Scripts/bin change, and my thought was JFDI. The rest of it seems fine to me - I say let's go for it. Personally, my main concerns are around procedure and policy. The more the discussion goes on, the more I feel that there should be a PEP to capture the details of the debate clearly. Too much is getting lost in the noise. And I think you should provide a clear statement of why this issue is important enough to justify violating the backward compatibility policies. As Mark said (I think it was Mark...) if this had been proposed for 3.0, it would have been fine. Now we're at 3.2 with 3.3 close to release, and it just seems too late to be worth the risk. One plus point about your posting this separately. It's made me think through the issue in a bit more detail, and I'm now a solid -1 on the proposal. I have been trying at various PyCons and in various conversations to move this for years. No one cares. The current urgency is driven by pyvenv - changes now will be much, much easier than changes later. Again, I am happy to write a PEP. If I were to summarize (on this issue only): 1. The current backwards compatibility hit is minimal; I would be happy to contact and provide patches to the four packages I have found (and anyone else who wants one). Backwards compatibility in the future will probably be harder to deal with. 2. There are advantages to cross-platform consistency and to virtualenv-based development. I believe that these will grow in the future. 3. Most people won't care. To the extent that people notice, I think they will appreciate the consistency. The virtualenv point, to me, is a strong one. I think we have an opportunity right now to make an adjustment, otherwise we're locked in again. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)
2012/3/22 VanL van.lindb...@gmail.com: Open Issues: If we do put python.exe on PATH (whether it's in bin or not), we have to debate how to handle people having multiple versions of python on their machine. In a post-PEP 397 world, no Python is the machine default - .py files are associated with py.exe, not python.exe, so we have to consider the following 3 commands being run from a shell prompt: 1. myprog.py 2. py myprog.py 3. python myprog.py 1 and 2 will always do the same thing. However, 3 could easily do something completely different, if the Python in the #! line differs from the one found on PATH. To me, this implies that it's better for (3) to need explicit user action (setting PATH) if it's to do anything other than give an error. But maybe that's just me. I've been hit too often by confusion caused by *not* remembering this fact. I'm not sure how widely used #1 is. I can't remember coming across any bug reports or posts around the web where the example command line just uses the Python chosen by the file association. I would suspect it's especially rare in the current time when many people are running a lot of versions of Python. Right now I have 2.6, 2.7, 3.1, 3.2, and 3.3, all installed in some different order, and I couldn't tell you which of those I installed the latest bugfix release for. That last one wins the race when it comes to file associations, and I've never paid attention to the installer option. #3 *will* require explicit user action - the Path setting is off by default. For as much as it's an advanced feature, it's really helpful to beginners. If you just want to type in python and have it work, the Path option is great. That's not to say the launcher isn't *also* a good thing. If you're a first timer and install Python 3.3 and want to run a tutorial - add Python to the path, type python, and you're on your way. If you're an advanced user and you want to write and run code on Python 3.3, do the same. If you're even more advanced and are doing multi-version work, the launcher is a helpful alternative. One possible response here is that the moving of the python.exe binary and the setting of the PATH would be tied to an unchecked-by-default installer option, making an explicit user choice needed to invoke the new functionality. I ended up typing out the above while missing this paragraph...but, bingo. Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden have all expressed that they have existing tools that would break and would need to be adjusted to match the new location of the python.exe, because that location is assumed to be at the root of the python install. Isn't the proposed BinaryDir registry key helpful here? It's not like we're telling people to fend for themselves -- we'll tell you where it's at. A related issue is that this portion of the proposal has met with some resistance, but not much support here on Python-dev. The reason for that is selection bias: Those who are on Python-dev are much more likely to have tools that do advanced things with Python, such as introspect on the location of the binary, and are also much more likely to be comfortable with things like editing the PATH on windows. In contrast, the people that have trouble with this issue are those that are newest to Python and programming generally - those for whom editing the PATH is a challenge and whom are likely to be confused by the distinction between python.exe and a python program - and why, even after they add python to the path, the python program is not directly executable. I still don't really get how this portion of the proposal, the python.exe move to bin, is holding people up. If you're using the launcher, the change is invisible. If you're using a setup where bin is on the Path, the change is invisible. File associations? Invisible. If you're typing out the full path, you have to type bin in the middle -- this kind of sucks but I think we'll live. I get that tools could be affected. I had two IDE makers at PyCon immediately throw up red flags to this change. I think one of them was about to charge the stage during my talk. When it was mentioned that we could point them to the proper location, they breathed a sigh of relief and said cool, do it. If a registry key pointing you to python.exe (rather, the directory) right now in Python 3.3 works, why doesn't another one pointing you to python.exe in Python = 3.3 work? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)
On Thu, Mar 22, 2012 at 18:26, Ethan Furman et...@stoneleaf.us wrote: Given the cost of the change, and the advent of the PEP-397 Launcher, I also vote -1. Can you provide some justification other than a number? It's a pretty cheap change and the launcher solves somewhat of a different problem. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)
[snipped some CCs] On 23/03/2012 2:20 PM, Brian Curtin wrote: ... I get that tools could be affected. I had two IDE makers at PyCon immediately throw up red flags to this change. I think one of them was about to charge the stage during my talk. When it was mentioned that we could point them to the proper location, they breathed a sigh of relief and said cool, do it. If a registry key pointing you to python.exe (rather, the directory) right now in Python 3.3 works, why doesn't another one pointing you to python.exe in Python= 3.3 work? It will work. The fact MvL is proposing the conservative approach of landing this in 3.5+ and have 3.3+ include the *new* registry key means I'm willing to reluctantly accept it rather than aggressively oppose it. Tools then have a chance to adapt to the new key. If the proposal moved any faster, existing tools which only use the old key would break without warning. The fact they need to change at all is unfortunate, but the timescale proposed means we can at least say we warned them. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
Mark, MAL, Martin, Tarek, Could you comment on this? This is in the context of changing the name of the 'Scripts' directory on windows to 'bin'. Éric brings up the point (explained more below) that if we make this change, packages made/installed the new packaging infrastructure and those made/installed with bdist_winist and the old (frozen) distutils will be inconsistent. The reason why is that the old distutils has a hard-coded dict in distutils.command.install that would point to the old locations. If we were to make this change in sysconfig.cfg, we would probably want to make a corresponding change in the INSTALL_SCHEMES dict in distutils.command.install. More context: On 3/20/2012 10:41 PM, Éric Araujo wrote: Le 20/03/2012 21:40, VanL a écrit : On Tuesday, March 20, 2012 at 5:07 PM, Paul Moore wrote: It's worth remembering Éric's point - distutils is frozen and changes are in theory not allowed. This part of the proposal is not possible without an exception to that ruling. Personally, I don't see how making this change could be a problem, but I'm definitely not an expert. If distutils doesn't change, bdist_wininst installers built using distutils rather than packaging will do the wrong thing with regard to this change. End users won't be able to tell how an installer has been built. Looking at the code in bdist_wininst, it loops over the keys in the INSTALL_SCHEMES dict to find the correct locations. If the hard-coded dict were changed, then the installer would 'just work' with the right location - and this matches my experience having made this sort of change. When I change the INSTALL_SCHEMES dict, things get installed according to the new scheme without difficulty using the standard tools. The only time when something is trouble is if it does its own install routine and hard-codes 'Scripts' as the name of the install directory - and I have only seen that in PyPM a couple versions ago. From the top of my head the developers with the most experience about Windows deployment are Martin v. Löwis, Mark Hammond and Marc-André Lemburg (not sure about the Windows part for MAL, but he maintains a library that extends distutils and has been broken in the past). I think their approval is required for this kind of huge change. Note the above - this is why I would like your comment. The point of the distutils freeze (i.e. feature moratorium) is that we just can’t know what complicated things people are doing with undocumented internals, because distutils appeared unmaintained and under-documented for years and people had to work with and around it; since the start of the distutils2 project we can Just Say No™ to improvements and features in distutils. “I don’t see what could possibly go wrong” is a classic line in both horror movies and distutils developmentwink. Renaming Scripts to bin on Windows would have effects on some tools we know and surely on many tools we don’t know. We don’t want to see again people who use or extend distutils come with torches and pitchforks because internals were changed and we have to revert. So in my opinion, to decide to go ahead with the change we need strong +1s from the developers I named above and an endorsement by Tarek, or if he can’t participate in the discussion, Guido. As a footnote, distutils is already broken in 3.3. Now we give users or system administrators the possibility to edit the install schemes at will in sysconfig.cfg, but distutils hard-codes the old scheme. I tend to think it should be fixed, to make the distutils-packaging transition/cohabitation possible. Any comment? Thanks, Van CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
Lindberg, Van wrote: Mark, MAL, Martin, Tarek, Could you comment on this? This is in the context of changing the name of the 'Scripts' directory on windows to 'bin'. Éric brings up the point (explained more below) that if we make this change, packages made/installed the new packaging infrastructure and those made/installed with bdist_winist and the old (frozen) distutils will be inconsistent. The reason why is that the old distutils has a hard-coded dict in distutils.command.install that would point to the old locations. If we were to make this change in sysconfig.cfg, we would probably want to make a corresponding change in the INSTALL_SCHEMES dict in distutils.command.install. I'm not sure I understand the point in making that change. Could you expand on the advantage of using bin instead of Scripts ? Note that distutils just provides defaults for these installation locations. All of them can be overridden using command line arguments to the install command. FWIW: I've dropped support for bdist_wininst in mxSetup.py since bdist_msi provides much better system integration. More context: On 3/20/2012 10:41 PM, Éric Araujo wrote: Le 20/03/2012 21:40, VanL a écrit : On Tuesday, March 20, 2012 at 5:07 PM, Paul Moore wrote: It's worth remembering Éric's point - distutils is frozen and changes are in theory not allowed. This part of the proposal is not possible without an exception to that ruling. Personally, I don't see how making this change could be a problem, but I'm definitely not an expert. If distutils doesn't change, bdist_wininst installers built using distutils rather than packaging will do the wrong thing with regard to this change. End users won't be able to tell how an installer has been built. Looking at the code in bdist_wininst, it loops over the keys in the INSTALL_SCHEMES dict to find the correct locations. If the hard-coded dict were changed, then the installer would 'just work' with the right location - and this matches my experience having made this sort of change. When I change the INSTALL_SCHEMES dict, things get installed according to the new scheme without difficulty using the standard tools. The only time when something is trouble is if it does its own install routine and hard-codes 'Scripts' as the name of the install directory - and I have only seen that in PyPM a couple versions ago. From the top of my head the developers with the most experience about Windows deployment are Martin v. Löwis, Mark Hammond and Marc-André Lemburg (not sure about the Windows part for MAL, but he maintains a library that extends distutils and has been broken in the past). I think their approval is required for this kind of huge change. Note the above - this is why I would like your comment. The point of the distutils freeze (i.e. feature moratorium) is that we just can’t know what complicated things people are doing with undocumented internals, because distutils appeared unmaintained and under-documented for years and people had to work with and around it; since the start of the distutils2 project we can Just Say No™ to improvements and features in distutils. “I don’t see what could possibly go wrong” is a classic line in both horror movies and distutils developmentwink. Renaming Scripts to bin on Windows would have effects on some tools we know and surely on many tools we don’t know. We don’t want to see again people who use or extend distutils come with torches and pitchforks because internals were changed and we have to revert. So in my opinion, to decide to go ahead with the change we need strong +1s from the developers I named above and an endorsement by Tarek, or if he can’t participate in the discussion, Guido. As a footnote, distutils is already broken in 3.3. Now we give users or system administrators the possibility to edit the install schemes at will in sysconfig.cfg, but distutils hard-codes the old scheme. I tend to think it should be fixed, to make the distutils-packaging transition/cohabitation possible. Any comment? Thanks, Van CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar
Re: [Python-Dev] Python install layout and the PATH on win32
On 22/03/2012 1:22 AM, Lindberg, Van wrote: Mark, MAL, Martin, Tarek, Could you comment on this? Eric is correct - tools will be broken by this change. However, people seem willing to push forward on this and accept such breakage as the necessary cost. MAL, in his followup, asks what the advantages are of such a change. I've actually been asking for the same thing in this thread and the only real answer I've got is consistency. So while I share MAL's concerns, people seem willing to push forward on this anyway, without the benefits having been explained. IOW, this isn't the decision I would make, but I think I've already made that point a number of times in this thread. Beyond that, there doesn't seem much for me to add... Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 21 March 2012 22:43, Mark Hammond skippy.hamm...@gmail.com wrote: On 22/03/2012 1:22 AM, Lindberg, Van wrote: Mark, MAL, Martin, Tarek, Could you comment on this? Eric is correct - tools will be broken by this change. However, people seem willing to push forward on this and accept such breakage as the necessary cost. MAL, in his followup, asks what the advantages are of such a change. I've actually been asking for the same thing in this thread and the only real answer I've got is consistency. So while I share MAL's concerns, people seem willing to push forward on this anyway, without the benefits having been explained. IOW, this isn't the decision I would make, but I think I've already made that point a number of times in this thread. Beyond that, there doesn't seem much for me to add... I agree on all points here. I don't understand quite why backward compatibility is being treated so lightly here. But equally, I've made my points and have little further to add. One thought though - maybe this should need a PEP at least, to document the proposal and record the various arguments made in this thread? Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
Cleaning up the absurd CC line On Thu, Mar 22, 2012 at 8:03 AM, Paul Moore p.f.mo...@gmail.com wrote: I agree on all points here. I don't understand quite why backward compatibility is being treated so lightly here. But equally, I've made my points and have little further to add. As a non-Windows user who occasionally is the only one available to help Windows users do something (other than install Linux and learn to live freewink/), consistency would be nice. I often have trouble finding the right advice for Windows, even if I feel like looking for it. Dunno if that's a common or important use case, though. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
For those who missed it, in http://bugs.python.org/issue14302, Martin recently commented: After more discussion, it appears that this change is too incompatible to be done in a single release. Therefore, I propose a long-term change into this direction, with the actual change not happening until 3.5. snip more detail While I'm still unclear on the actual benefits of this, Martin's approach strikes a reasonable compromise so I withdraw my objections. Thanks, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 5:48 AM, Mark Hammond wrote: While I'm still unclear on the actual benefits of this, Martin's approach strikes a reasonable compromise so I withdraw my objections. Ok. I was out of town and so could not respond to most of the latest discussion. A question for you Mark, Paul, (and anyone else): Éric correctly points out that there are actually two distinct changes proposed here: 1. Moving the Python binary 2. Changing from Scripts to bin So far, the primary resistance seems to be to item #1 - moving the python binary. There have been a few people who have noted that #2 will require some code to change (i.e. Paul), but I don't see lots of resistance. Am I reading you correctly? Thanks, VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 20/03/2012 14:08, VanL wrote: On 3/20/2012 5:48 AM, Mark Hammond wrote: While I'm still unclear on the actual benefits of this, Martin's approach strikes a reasonable compromise so I withdraw my objections. Ok. I was out of town and so could not respond to most of the latest discussion. A question for you Mark, Paul, (and anyone else): Éric correctly points out that there are actually two distinct changes proposed here: 1. Moving the Python binary 2. Changing from Scripts to bin So far, the primary resistance seems to be to item #1 - moving the python binary. There have been a few people who have noted that #2 will require some code to change (i.e. Paul), but I don't see lots of resistance. Speaking for myself, I think that's true. At present I tend to add /scripts to my path and I can just as easily add /bin (for whatever version I'm running most often on that machine). TJG ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
Germane to this discussion, I reached out for feedback. Most people didn't care about the issue, or were slightly inclined to have it be uniform across platforms. As Terry mentioned, I think that long-term uniformity will benefit everybody down the line, and that is the way to go. The most interesting feedback, though, related to moving the Python exe and placing it on the PATH. I got one argument back that I thought was persuasive here: We want things to 'just work.' Specifically, the following sequence of events should not require any fiddling on Windows: 1. Install python. 2. Open up a shell and run python 3. Use pip or easy_install to install regetron (a package that installs an executable file). 4. Run regetron. For step #2, the python exe needs to be on the PATH. For steps 3 and 4, the binaries directory needs to be on the PATH. In hearing from a couple people who teach python to beginners, this is a substantial hurdle - the first thing they need to do is to edit their environment to add these directories to the PATH. This is orthogonal to the Scripts/bin issue, but I thought it should be brought up. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
In hearing from a couple people who teach python to beginners, this is a substantial hurdle - the first thing they need to do is to edit their environment to add these directories to the PATH. This is something I never understood. On Windows, it's custom to launch programs from the start menu, and Python is easy enough to find on the start menu (e.g. by typing Python). Why do people want to launch it by opening a shell window, then typing python? In any case, I have given up my resistance to the feature request for automatic path fiddling several years ago, and was since waiting for a contribution of a patch that makes it happen. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 10:52 AM, Martin v. Löwis wrote: In hearing from a couple people who teach python to beginners, this is a substantial hurdle - the first thing they need to do is to edit their environment to add these directories to the PATH. This is something I never understood. On Windows, it's custom to launch programs from the start menu, and Python is easy enough to find on the start menu (e.g. by typing Python). Why do people want to launch it by opening a shell window, then typing python? Because the workflow you suggest is broken when you are developing with Python. Assume that you are iteratively building up a program in Python. You aren't sure if it is right yet, so you want to get it into python to test it and see the output. There are three ways to do this. 1. Run python from the start menu. - Import sys, fiddle with sys.path to add my module, import/run my module, do my tests. When you exit /hard error out, the python window disappears. 2. Double-click the .py file - Runs the file, but then disappears immediately (unless you put in something like input/raw_input just to keep the window open) - and if it errors out, you never see the traceback - it disappears too fast. 3. Get a shell and run python. This requires cd'ing to the directory where my .py file is, but then I run/import it and I see the information. To repeat the process, either type python again or just press arrow-up. 4. (Not relevant here) - do it in an IDE that does #3 for you. #3 is the only reasonable way to do development if you are not in an IDE. Thanks, Van ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On Tue, Mar 20, 2012 at 10:52, Martin v. Löwis mar...@v.loewis.de wrote: In hearing from a couple people who teach python to beginners, this is a substantial hurdle - the first thing they need to do is to edit their environment to add these directories to the PATH. This is something I never understood. On Windows, it's custom to launch programs from the start menu, and Python is easy enough to find on the start menu (e.g. by typing Python). Why do people want to launch it by opening a shell window, then typing python? I've never thought about doing it otherwise. If I want to run the C:\Users\brian\example\sample.py script, I'd open a CMD and move to the example directory and execute the sample script. The class of about 60 people I taught a few years back at a previous employer all did the same thing without me specifying. Everyone was used to working in the command line for other tasks, from using other languages to running our products, so it was natural to them to run it that way. In any case, I have given up my resistance to the feature request for automatic path fiddling several years ago, and was since waiting for a contribution of a patch that makes it happen. I'm working on the changes we discussed at PyCon. http://bugs.python.org/issue3561 has an version of the patch doing it the old way - I hope to have the new way figured out soon. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
1. Run python from the start menu. - Import sys, fiddle with sys.path to add my module, import/run my module, do my tests. When you exit /hard error out, the python window disappears. 2. Double-click the .py file - Runs the file, but then disappears immediately (unless you put in something like input/raw_input just to keep the window open) - and if it errors out, you never see the traceback - it disappears too fast. 3. Get a shell and run python. This requires cd'ing to the directory where my .py file is, but then I run/import it and I see the information. To repeat the process, either type python again or just press arrow-up. 4. (Not relevant here) - do it in an IDE that does #3 for you. #3 is the only reasonable way to do development if you are not in an IDE. No - there is an version #3a: 3.a) Get a shell and run the script CD into the directory, then directly run foo.py, without prefixing it with python.exe. This doesn't require any changes to the path, and is shorter in usage than having the path specified. With PEP 397, you will be able to run py foo.py without path modification, and it will get the correct Python version even (which neither the path manipulation nor the file association could achieve). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 11:19 AM, Martin v. Löwis wrote: No - there is an version #3a: 3.a) Get a shell and run the script CD into the directory, then directly run foo.py, without prefixing it with python.exe. This doesn't require any changes to the path, and is shorter in usage than having the path specified. Fair enough - but notice: 1) You are still in the shell, instead of running from the start menu; and 2) what if you want to import it and test a function with various inputs? You either implement a request/response in a __main__ block, or type python and then import foo. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 12:02 PM, VanL wrote: On 3/20/2012 10:52 AM, Martin v. Löwis wrote: In hearing from a couple people who teach python to beginners, this is a substantial hurdle - the first thing they need to do is to edit their environment to add these directories to the PATH. This is something I never understood. On Windows, it's custom to launch programs from the start menu, and Python is easy enough to find on the start menu (e.g. by typing Python). Why do people want to launch it by opening a shell window, then typing python? Perhaps as the number of *nix users increases, the number of (*nix windows) developer/users increases. 3. Get a shell and run python. This requires cd'ing to the directory where my .py file is, but then I run/import it and I see the information. When IDLE crashes, running it from a cmd window is the only way to get a traceback to help diagnose the problem. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 20 March 2012 14:08, Lindberg, Van van.lindb...@haynesboone.com wrote: On 3/20/2012 5:48 AM, Mark Hammond wrote: While I'm still unclear on the actual benefits of this, Martin's approach strikes a reasonable compromise so I withdraw my objections. Ok. I was out of town and so could not respond to most of the latest discussion. A question for you Mark, Paul, (and anyone else): Éric correctly points out that there are actually two distinct changes proposed here: 1. Moving the Python binary 2. Changing from Scripts to bin So far, the primary resistance seems to be to item #1 - moving the python binary. There have been a few people who have noted that #2 will require some code to change (i.e. Paul), but I don't see lots of resistance. Am I reading you correctly? Somewhat. I don't really object to #1, but mildly object to #2. I also note that the proposals round the Lib directory seem to have disappeared. I assume those have been dropped - they were the ones I did object to. I also note that I'm assuming virtualenv will change to match whatever the Python version it's referencing does. I don't see how you can guarantee that, but if there are discrepancies between virtualenvs and installed Pythons, my level of objection goes up a little more. Martin's suggestion of an intermediate registry entry to ease transition doesn't help me. So I don't care about that. See a later message for my comments on PATH as it affects this discussion, though. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
When IDLE crashes, running it from a cmd window is the only way to get a traceback to help diagnose the problem. Certainly. In this case, there is no PATH issue, though: you have to CD into the Python installation, anyway, to start IDLE - and there you have python.exe in the current directory. Plus, you can still launch Lib\idlelib\idle.py without prefixing it with python.exe. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 03/20/2012 12:19 PM, Paul Moore wrote: I also note that I'm assuming virtualenv will change to match whatever the Python version it's referencing does. I don't see how you can guarantee that, but if there are discrepancies between virtualenvs and installed Pythons, my level of objection goes up a little more. Virtualenv will follow whatever Python does, yes. Carl signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 13 March 2012 20:43, VanL van.lindb...@gmail.com wrote:
Following up on conversations at PyCon, I want to bring up one of my
personal hobby horses for change in 3.3: Fix install layout on Windows, with
a side order of making the PATH work better.
As this is being considered an 'incompatible change' on the bug
tracker item [1] in any case, I'd like to mention that this might also
be a convenient moment to re-think the default install location. After
all, software is supposed to be installed in %programfiles% on
windows, not in c:\program name.
I asked a question about this on IRC, to which the response was that
there were two main reasons to install python in c:\pythonxy:
1 - issues due to spaces ('Program Files') or non-ascii characters in
the path ('Fișiere Program' on a Romanian windows). These issues are
supposed to be fixed by now (?).
2 - issues due to permissions - installing python / packages in
%programfiles% may require administrator rights.
Historical note: in python 1.5 the install location was changed to
\program files\.., but in python 1.6/2.0 it was changed (back?) to
\pythonxy. [2 @ 10618, 10850, 13804]
[1] http://bugs.python.org/issue14302
[2] http://hg.python.org/cpython/file/a5add01e96be/Misc/HISTORY
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 20 March 2012 14:27, VanL van.lindb...@gmail.com wrote: Germane to this discussion, I reached out for feedback. Most people didn't care about the issue, or were slightly inclined to have it be uniform across platforms. As Terry mentioned, I think that long-term uniformity will benefit everybody down the line, and that is the way to go. The most interesting feedback, though, related to moving the Python exe and placing it on the PATH. I got one argument back that I thought was persuasive here: We want things to 'just work.' Specifically, the following sequence of events should not require any fiddling on Windows: 1. Install python. 2. Open up a shell and run python 3. Use pip or easy_install to install regetron (a package that installs an executable file). 4. Run regetron. For step #2, the python exe needs to be on the PATH. For steps 3 and 4, the binaries directory needs to be on the PATH. This is covered (better, IMO) by PEP 397 - Python Launcher for Windows. Step 2, just run py. If you prefer a particular version, run py -2 or py -3 or py -3.2. Adding python.exe to PATH actually makes this message *worse* as it confuses the issue. In a post-PEP 397 world, I would say that we should be telling Windows users *not* to run python.exe at all. (Unless they are using virtualenvs - maybe PEP 397 could do with an extra option to run the Python from the currently active virtualenv, but that's a side issue). If we do put python.exe on PATH (whether it's in bin or not), we have to debate how to handle people having multiple versions of python on their machine. In a post-PEP 397 world, no Python is the machine default - .py files are associated with py.exe, not python.exe, so we have to consider the following 3 commands being run from a shell prompt: 1. myprog.py 2. py myprog.py 3. python myprog.py 1 and 2 will always do the same thing. However, 3 could easily do something completely different, if the Python in the #! line differs from the one found on PATH. To me, this implies that it's better for (3) to need explicit user action (setting PATH) if it's to do anything other than give an error. But maybe that's just me. I've been hit too often by confusion caused by *not* remembering this fact. Note: I am using Vinay's py.exe all the time these days, so my comments about a post-PEP 397 world are from my direct experience. For your steps 3 and 4, there is certainly user intervention required as things stand. It would indeed be nice if regetron just worked as expected. But I'd argue a few points here: 1. On Windows, if regetron was not explicitly an application for working with my Python installation (like pip, easy_install, nose, etc) then I'd prefer it to be packaged as a standalone application using cx_Freeze or py2exe. I've had too many applications break because I accidentally uninstalled a dependency from my Python installation to want anything that is an end-user application installed into my Python scripts/bin directory. 2. If regetron is not an end-user application, it should really be getting installed in, and run from, a virtualenv. And in that case, activating the right virtualenv is part of the workflow. And that sets up PATH as needed, so there's no problem here. The problem with your example is that it depends on the package/executable. I looked at regetron (I thought it was a made up example at first!) and it seems clear to me that it should either be packaged up with py2exe/cx_Freeze, or (if it's sufficiently version-independent) sit outside of Python's installation tree. I can't see any reason why I'd expect a regetron command to work or not depending on which copy of Python on my PC I have active. But other applications may differ, I guess. I concede that the picture is much simpler when people only ever have a single version of Python on their machine. So for that case alone, maybe the Make this Python the default option in the installer should add the bin directory (or Scripts and the root, under the current layout) to the PATH. But equally, if the installer detects another copy of Python already installed, it should probably warn the user loudly that it's important to understand the implications of setting make this Python the default, and should not set it by default (this may be the current behaviour, I don't know). I have no idea what proportion of Python users on Windows have multiple versions installed. I also have no idea how many such users work on the command line. My guess would be not that many and not that many of the first group respectively... But there are big groups like scientists and web developers who could sway these figures a lot. In hearing from a couple people who teach python to beginners, this is a substantial hurdle - the first thing they need to do is to edit their environment to add these directories to the PATH. I'd be curious as to how much PEP 397's py.exe would have helped those people. But yes, it's an issue.
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 1:50 PM, Merlijn van Deen wrote: On 13 March 2012 20:43, VanLvan.lindb...@gmail.com wrote: Following up on conversations at PyCon, I want to bring up one of my personal hobby horses for change in 3.3: Fix install layout on Windows, with a side order of making the PATH work better. As this is being considered an 'incompatible change' on the bug tracker item [1] in any case, I'd like to mention that this might also be a convenient moment to re-think the default install location. After all, software is supposed to be installed in %programfiles% on windows, not in c:\program name. I don't particularly care about this issue, as I always install to my own location (c:\lib\python\X.Y), but I don't think that the default location of the install should be confounded with this issue - or should I say these issues, because we already have two. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 1:56 PM, Paul Moore wrote: This is covered (better, IMO) by PEP 397 - Python Launcher for Windows. Step 2, just run py. If you prefer a particular version, run py -2 or py -3 or py -3.2. Interesting. I haven't played around with that at all, so can't comment. I will have to try it out. Adding python.exe to PATH actually makes this message *worse* as it confuses the issue. In a post-PEP 397 world, I would say that we should be telling Windows users *not* to run python.exe at all. (Unless they are using virtualenvs - maybe PEP 397 could do with an extra option to run the Python from the currently active virtualenv, but that's a side issue). I think that having the PATH manipulation be optional might address this issue. I also think that the PEP 397 launcher should respect virtualenvs - at least the built-in pyvenvs - or else there will be lots of confusion. For your steps 3 and 4, there is certainly user intervention required as things stand. It would indeed be nice if regetron just worked as expected. But I'd argue a few points here: 1. On Windows, if regetron was not explicitly an application for working with my Python installation (like pip, easy_install, nose, etc) then I'd prefer it to be packaged as a standalone application using cx_Freeze or py2exe. I've had too many applications break because I accidentally uninstalled a dependency from my Python installation to want anything that is an end-user application installed into my Python scripts/bin directory. Maybe so - and I would probably agree that for any packaged application, bundling it into its own environment (or at least its own virtualenv) is the best practice. But what about pip, easy_install, nose, cython, pygments, PIL, etc, that do this and are meant to be associated with a particular python version? Substitute nose for regetron if you want, and there is still a problem. The problem with your example is that it depends on the package/executable. I looked at regetron (I thought it was a made up example at first!) ...! I got the name from the feedback I received. I thought it was made up too. I have no idea what proportion of Python users on Windows have multiple versions installed. I also have no idea how many such users work on the command line. My guess would be not that many and not that many of the first group respectively... But there are big groups like scientists and web developers who could sway these figures a lot. There are a number of casual users that probably only have one version installed, but every python user/dev on windows that I know has one python that they consider to be python, and everything else needs to be launched with a suffix (e.g., python26.exe). This is usually put earlier on the PATH so that it gets picked up first. For example, right now I have 2.6, 2.7, 3.2, jython, and pypy all installed, and I have python pointing to 2.7. I'd be curious as to how much PEP 397's py.exe would have helped those people. But yes, it's an issue. Although someone at some point will have to introduce those beginners to the question of Python 2 vs Python 3, and PATH pain will hit them then, anyway. I would imagine that it would help steps 1 and 2, but 3 and 4 would be problematic (how can you pip install something using py?) unless you were in a virtualenv, and then (unless py respected the virtualenv) the whole thing would be problematic, because there wouldn't be one clear way to do it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 1:19 PM, Paul Moore wrote:
Somewhat. I don't really object to #1, but mildly object to #2. I also
note that the proposals round the Lib directory seem to have
disappeared. I assume those have been dropped - they were the ones I
did object to.
They are of secondary importance to me, and I would be mostly ok to drop
them - but I would like to understand your objection.
I would like to know if you would object to user lib installs matching
the system install. I.e., would it cause problems with you if it were
just 'lib' everywhere, with no 'lib/python{version}'? It sounded like
adding the version directory was the issue.
Thanks,
VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by
U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any
U.S. tax advice contained in this communication (including any
attachments) was not intended or written to be used, and cannot be
used, for the purpose of (i) avoiding penalties under the Internal
Revenue Code or (ii) promoting, marketing or recommending to another
party any transaction or matter addressed herein.
CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential,
may be privileged and should be read or retained only by the intended
recipient. If you have received this transmission in error, please
immediately notify the sender and delete it from your system.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 20 March 2012 19:35, Lindberg, Van van.lindb...@haynesboone.com wrote:
I would like to know if you would object to user lib installs matching
the system install. I.e., would it cause problems with you if it were
just 'lib' everywhere, with no 'lib/python{version}'? It sounded like
adding the version directory was the issue.
User lib installs don't bother me as I don't use them. But yes, it's
the version directory that bothers me.
So if you're proposing simply making the user lib install match the
system install, both being lust lib, then that's fine. I was
somewhat confused about what you were proposing, that's all.
Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 20 March 2012 19:22, VanL van.lindb...@gmail.com wrote: There are a number of casual users that probably only have one version installed, but every python user/dev on windows that I know has one python that they consider to be python, and everything else needs to be launched with a suffix (e.g., python26.exe). This is usually put earlier on the PATH so that it gets picked up first. For example, right now I have 2.6, 2.7, 3.2, jython, and pypy all installed, and I have python pointing to 2.7. But no Python I am aware of *has* a suffixed version (python26.exe). Renaming/copying is (in my view) a far more invasive change than simply modifying PATH (and it doesn't help the whole nose/regetron situation either). Serious question: Given a brand new PC, if you were installing Python 2.7, 3.2, 3.3a1, jython, and pypy, what would you do (beyond simply running 5 installers) to get your environment set up the way you want? For me, I'd 1. Install the Python launcher (only until 3.3 includes it) 2. Edit py.ini to tailor py.exe to my preferred defaults for Python and Python3. 3. Install my powershell module which allows me to switch which Python is on PATH Done. (That doesn't cater for pypy or jython, as I don't use them. But I'd probably use a couple of aliases for the rare uses I'd make of them) I'd be curious as to how much PEP 397's py.exe would have helped those people. But yes, it's an issue. Although someone at some point will have to introduce those beginners to the question of Python 2 vs Python 3, and PATH pain will hit them then, anyway. I would imagine that it would help steps 1 and 2, but 3 and 4 would be problematic (how can you pip install something using py?) unless you were in a virtualenv, and then (unless py respected the virtualenv) the whole thing would be problematic, because there wouldn't be one clear way to do it. There isn't one clear way right now. And adding one particular version to PATH only helps if you only *have* one version. My current preference is as follows: 1. If you only ever have one Python on your machine, add it (and its scripts dir) to PATH and be done with it. Unfortunately, we're in the throes of the Python 2-3 transition, and not many people can manage with the one-Python restriction (I certainly can't). Also the Python installer can't detect if that's what you want. 2. Otherwise, use virtualenvs for anything that isn't being packaged up as a standalone environment. Activate as needed. 3. To access your system python(s) use py.exe with a version flag if needed. Never (or nearly never) install packages in the system Python. 4. To run scripts, use #! lines and the py.exe association (and set PATHEXT if you want) to associate the precise Python you want with the script. I have to say, I've recently discovered virtualenv, so the above is the opinion of a newly-converted zealot - so take with a pinch of salt :-) Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 3:15 PM, Paul Moore wrote:
On 20 March 2012 19:35, Lindberg, Vanvan.lindb...@haynesboone.com wrote:
I would like to know if you would object to user lib installs matching
the system install. I.e., would it cause problems with you if it were
just 'lib' everywhere, with no 'lib/python{version}'? It sounded like
adding the version directory was the issue.
User lib installs don't bother me as I don't use them. But yes, it's
the version directory that bothers me.
So if you're proposing simply making the user lib install match the
system install, both being lust lib, then that's fine. I was
somewhat confused about what you were proposing, that's all.
I was originally going to make it match posix-user installs, but just
plain posix (no version directory) is just fine with me too.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 11:50 AM, Merlijn van Deen wrote:
As this is being considered an 'incompatible change' on the bug
tracker item [1] in any case, I'd like to mention that this might also
be a convenient moment to re-think the default install location. After
all, software is supposed to be installed in %programfiles% on
windows, not in c:\program name.
I asked a question about this on IRC, to which the response was that
there were two main reasons to install python in c:\pythonxy:
1 - issues due to spaces ('Program Files') or non-ascii characters in
the path ('Fișiere Program' on a Romanian windows). These issues are
supposed to be fixed by now (?).
2 - issues due to permissions - installing python / packages in
%programfiles% may require administrator rights.
I also wondered about %programfiles%, and had heard of issue #1, and
would hope that it is not a real issue in modern times, but haven't
attempted to test to determine otherwise.
However, the in the first quoted paragraph there is an incorrect
statement... the last sentence is simply not true. While software that
is installed for everyone on the computer is supposed to be installed
in %programfiles%, software that is installed for user only need not
be, and in fact, it is recommended (at least by installer software I've
used) that the alternate path is (XP) C:\Documents and
Settings\user\Local Settings\Application Data or (7)
C:\Users\user\AppData\Local (I think, I haven't found certain
documentation about this).
Or is it even possible to install something for user only anymore? I
haven't been involved with installers lately (have been doing portable
apps, no install needed). Certainly the program files (x86) business
adds an extra wrinkle to it, somehow, on 64 bit machines, and I'm not
hitting the right sites on my Google searches to discover anything about
that, so that's why I'm wondering if it has been deprecated.
Speaking of which, it would be nice to have Portable Python be part of
the standard reportoire of packages available.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 3:31 PM, Paul Moore wrote: Serious question: Given a brand new PC, if you were installing Python 2.7, 3.2, 3.3a1, jython, and pypy, what would you do (beyond simply running 5 installers) to get your environment set up the way you want? I install each python in its own directory: C:/lib/python/2.7 C:/lib/python/3.2 C:/lib/python/3.3 C:/lib/jython C:/lib/pypy Jython and Pypy get their own directories because they can have different version compatibilities. I then edit my distutils.command.install and patch pip/virtualenv so that all my directories are 'bin'/'lib'/'include'. I have never used the py.exe runner, but I then choose whichever Python is my default (right now 2.7, but hoping that I will be able to switch during the 3.3 timeframe) and that gets put on the PATH, along with its 'bin' directory. The other root dirs/bin directories get put on the PATH after the default Python. I don't remember whether I did it or whether it is installed that way, but I have a python2.6.exe and an pythonw.2.6.exe, etc, and all the individual installers include both a pip and a pip-2.6 version (or whatever that install has). I honestly don't remember - I would love to have someone else check. With this setup, I get my default choice anytime I type python and a specific interpreter version when I specify it. Same with installers, etc. I then install virtualenv and virtualenvwrapper-powershell and do all of my development out of virtualenvs. Occasionally I will install something to the system python if it is a pain to compile and I am installing a binary version from somewhere, but I generally try to keep the system python(s) clean.CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 21/03/2012 1:08 AM, Lindberg, Van wrote: On 3/20/2012 5:48 AM, Mark Hammond wrote: While I'm still unclear on the actual benefits of this, Martin's approach strikes a reasonable compromise so I withdraw my objections. Ok. I was out of town and so could not respond to most of the latest discussion. A question for you Mark, Paul, (and anyone else): Éric correctly points out that there are actually two distinct changes proposed here: 1. Moving the Python binary 2. Changing from Scripts to bin So far, the primary resistance seems to be to item #1 - moving the python binary. There have been a few people who have noted that #2 will require some code to change (i.e. Paul), but I don't see lots of resistance. Am I reading you correctly? Well - as Paul implies, there are 2 distinct changes being proposed, but in 2 different environments. For an installed Python: If it has to move, it may as well move to somewhere consistent with other platforms. IOW, moving to bin seems preferable to moving to Scripts. My initial objection was to moving it at all in an installed Python. For a virtual env, we are talking about moving it *from* Scripts to bin, which may cause some people different problems. However, that isn't the concern I was expressing and I'd hate to see virtual envs remain inconsistent with an installed Python after this effort. So I'm assuming that: * The executable (and DLL) are moved to a bin directory in an installed Python. * distutils etc will change to install all scripts (or executables generated from scripts) into that same directory. IOW, Scripts would die. * A virtual-env would have an almost identical layout to an installed Python. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 4:49 PM, Mark Hammond wrote: So I'm assuming that: * The executable (and DLL) are moved to a bin directory in an installed Python. * distutils etc will change to install all scripts (or executables generated from scripts) into that same directory. IOW, Scripts would die. * A virtual-env would have an almost identical layout to an installed Python. Yes. I would make your point #3 stronger - I would like a virtualenv to have an identical layout to an installed python, at least with reference to the names of directories and the location of binaries. The base directory would be the only difference. Thanks, Van ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 20 March 2012 22:00, VanL van.lindb...@gmail.com wrote: On 3/20/2012 4:49 PM, Mark Hammond wrote: So I'm assuming that: * The executable (and DLL) are moved to a bin directory in an installed Python. * distutils etc will change to install all scripts (or executables generated from scripts) into that same directory. IOW, Scripts would die. It's worth remembering Éric's point - distutils is frozen and changes are in theory not allowed. This part of the proposal is not possible without an exception to that ruling. Personally, I don't see how making this change could be a problem, but I'm definitely not an expert. If distutils doesn't change, bdist_wininst installers built using distutils rather than packaging will do the wrong thing with regard to this change. End users won't be able to tell how an installer has been built. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 21/03/2012 5:50 AM, Merlijn van Deen wrote:
On 13 March 2012 20:43, VanLvan.lindb...@gmail.com wrote:
Following up on conversations at PyCon, I want to bring up one of my
personal hobby horses for change in 3.3: Fix install layout on Windows, with
a side order of making the PATH work better.
As this is being considered an 'incompatible change' on the bug
tracker item [1] in any case, I'd like to mention that this might also
be a convenient moment to re-think the default install location. After
all, software is supposed to be installed in %programfiles% on
windows, not in c:\program name.
I asked a question about this on IRC, to which the response was that
there were two main reasons to install python in c:\pythonxy:
1 - issues due to spaces ('Program Files') or non-ascii characters in
the path ('Fișiere Program' on a Romanian windows). These issues are
supposed to be fixed by now (?).
2 - issues due to permissions - installing python / packages in
%programfiles% may require administrator rights.
Apart from personal preference (ie, I prefer the status quo here), the
second issue is a bit of a killer. Even an administrator can not write
to Program Files unless they are using an elevated process (ie,
explicitly use Run as Administrator and confirm the prompt.
This means that any installer wanting to write .py files into the Python
install must be elevated, and any Python process wanting to generate
.pyc files must also be elevated. So even if an installer does arrange
elevation, unless that installer also compiles all .pyc and .pyo files
at install time, Python would fail to generate the .pyc files on first
use. While Python will probably fail silently and still continue to
work, it will have a significant performance impact.
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 21/03/2012 9:45 AM, R. David Murray wrote:
On Wed, 21 Mar 2012 09:09:38 +1100, Mark Hammondskippy.hamm...@gmail.com
wrote:
On 21/03/2012 5:50 AM, Merlijn van Deen wrote:
I asked a question about this on IRC, to which the response was that
there were two main reasons to install python in c:\pythonxy:
1 - issues due to spaces ('Program Files') or non-ascii characters in
the path ('Fișiere Program' on a Romanian windows). These issues are
supposed to be fixed by now (?).
2 - issues due to permissions - installing python / packages in
%programfiles% may require administrator rights.
Apart from personal preference (ie, I prefer the status quo here), the
second issue is a bit of a killer. Even an administrator can not write
to Program Files unless they are using an elevated process (ie,
explicitly use Run as Administrator and confirm the prompt.
This means that any installer wanting to write .py files into the Python
install must be elevated, and any Python process wanting to generate
.pyc files must also be elevated. So even if an installer does arrange
elevation, unless that installer also compiles all .pyc and .pyo files
at install time, Python would fail to generate the .pyc files on first
use. While Python will probably fail silently and still continue to
work, it will have a significant performance impact.
So windows requires admin privileges to install to Program Files, but
not to install to '/'? How novel. (You can perhaps tell that I'm
not a windoows user). My understanding, though, is that Python
does make a distinction between a system install of Python and
a per-user one, so I don't think your objection really applies.
I think it does. Consider I've installed Python as a system install.
Now I want to install some other package - ideally that installer will
request elevation - all well and good - the .py files are installed.
However, next time I want to run Python, it will fail to generate the
.pyc files - even though I'm an administrator. I would need to
explicitly tell Python to execute as administrator (or run it from an
already elevated command-prompt) to have things work as expected. Thus,
the usual case would be that Python is unable to update any files in
its install directory.
If Python installed for a single user didn't install into Program Files
(which it probably couldn't do without an administrator providing
credentials anyway) then it wouldn't be a problem - but then we have
multiple possible default install locations, which sounds like more
trouble than it is worth...
That said, there is an open bug in the tracker about the insecurity
of a system install of python (exactly that the files are writable
by anyone). So that would have to be solved first. I'd say this
is definitely a separate issue from Van's discussion, and the *only*
reason one might want to tie them together at all is well, we're
changing the directory layout anyway.
Agreed.
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 4:25 PM, Mark Hammond wrote: I think it does. Consider I've installed Python as a system install. Now I want to install some other package - ideally that installer will request elevation - all well and good - the .py files are installed. However, next time I want to run Python, it will fail to generate the .pyc files - even though I'm an administrator. I would need to explicitly tell Python to execute as administrator (or run it from an already elevated command-prompt) to have things work as expected. Thus, the usual case would be that Python is unable to update any files in its install directory. If Python installed for a single user didn't install into Program Files (which it probably couldn't do without an administrator providing credentials anyway) then it wouldn't be a problem - but then we have multiple possible default install locations, which sounds like more trouble than it is worth... That said, there is an open bug in the tracker about the insecurity of a system install of python (exactly that the files are writable by anyone). So that would have to be solved first. I'd say this is definitely a separate issue from Van's discussion, and the *only* reason one might want to tie them together at all is well, we're changing the directory layout anyway. Indeed, the single user place isn't a single place, unless you consider the per-user $APPDATA environment variable sufficient to determine it (or the Windows API that returns the initial boot up value of $APPDATA/ %APPDATA%, which is the preferred technique for code). But it does solve the security problem (stuff in APPDATA is accessible only to a single login by default). So that might be justification for putting it there, for single users. For multi-user installs, %PROGRAMFILES% is appropriate, but, like I've heard some Linux distributions do, *.pyc might have to be prebuilt and installed along with Python (or generated during install, instead of waiting for first use). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On Tuesday, March 20, 2012 at 5:07 PM, Paul Moore wrote: It's worth remembering Éric's point - distutils is frozen and changes are in theory not allowed. This part of the proposal is not possible without an exception to that ruling. Personally, I don't see how making this change could be a problem, but I'm definitely not an expert. If distutils doesn't change, bdist_wininst installers built using distutils rather than packaging will do the wrong thing with regard to this change. End users won't be able to tell how an installer has been built. This is a good point. Who can make this call - Guido, or someone else? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
Hi, Le 20/03/2012 21:40, VanL a écrit : On Tuesday, March 20, 2012 at 5:07 PM, Paul Moore wrote: It's worth remembering Éric's point - distutils is frozen and changes are in theory not allowed. This part of the proposal is not possible without an exception to that ruling. Personally, I don't see how making this change could be a problem, but I'm definitely not an expert. If distutils doesn't change, bdist_wininst installers built using distutils rather than packaging will do the wrong thing with regard to this change. End users won't be able to tell how an installer has been built. This is a good point. Who can make this call - Guido, or someone else? From the top of my head the developers with the most experience about Windows deployment are Martin v. Löwis, Mark Hammond and Marc-André Lemburg (not sure about the Windows part for MAL, but he maintains a library that extends distutils and has been broken in the past). I think their approval is required for this kind of huge change. The point of the distutils freeze (i.e. feature moratorium) is that we just can’t know what complicated things people are doing with undocumented internals, because distutils appeared unmaintained and under-documented for years and people had to work with and around it; since the start of the distutils2 project we can Just Say No™ to improvements and features in distutils. “I don’t see what could possibly go wrong” is a classic line in both horror movies and distutils development wink. Renaming Scripts to bin on Windows would have effects on some tools we know and surely on many tools we don’t know. We don’t want to see again people who use or extend distutils come with torches and pitchforks because internals were changed and we have to revert. So in my opinion, to decide to go ahead with the change we need strong +1s from the developers I named above and an endorsement by Tarek, or if he can’t participate in the discussion, Guido. As a footnote, distutils is already broken in 3.3. Now we give users or system administrators the possibility to edit the install schemes at will in sysconfig.cfg, but distutils hard-codes the old scheme. I tend to think it should be fixed, to make the distutils-packaging transition/cohabitation possible. Regards ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
R. David Murray wrote: My understanding, though, is that Python does make a distinction between a system install of Python and a per-user one, so I don't think your objection really applies. Seems to me that for Python at least, the important distinction is not so much where the files are placed, but whether the registry entries are made machine-wide or user-local. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 17 March 2012 05:28, Mark Hammond skippy.hamm...@gmail.com wrote: I hate to seem like I'm piling on now after panning your last brainstorm :-), but... this would be quite problematic for virtualenv users, many of whom do rely on the fact that the virtualenv stuff is confined to within a limited set of known subdirectories, and they can overlay a virtualenv and their own project code with just a few virtualenv directories vcs-ignored. I would prefer either the status quo or the proposed cross-platform harmonization. I work purely Windows-only, and I have a few scripts that manage virtualenvs for myself (for example, sort of a personal virtalenv-wrapper for Powershell - see https://bitbucket.org/pmoore/poshpy for a work-in-progress version). They have special casing for the differences in layout between standard installs, build directories, and virtualenvs. Changing the layout would cause these tools to need to change. In theory, putting python.exe/pythonw.exe into Scripts would simplify them (no need to cater for the cases where I need to put 2 directories on PATH), and changing Scripts - bin would be trivial. But in practice, it would mean that I need to check (somehow) the Python version and adjust the layout used accordingly. As there is no way of knowing the Python version without running Python, this is too slow to be practical. So while the changes are in theory harmless in isolation (except the library locations - changing those *would* cause pain) the need to support multiple versions would make this a major issue for me. So, I prefer the status quo. If necessary, I can live with the change to rename scripts as bin and put the Python executables in there (some cost, but some small benefit as well) but I oppose changing the library locations (all cost, no gain for me). All of this presupposes that both the standard installer *and* virtualenv change. I suspect that having virtualenv respect the old layout for 3.2 and earlier, and the new one for 3.3+, could be messy, though, so that's not guaranteed, I guess... Breaking the few tools I'm concerned about vs asking Van etc to continue taking the pain he feels isn't going to mean the end of the world for any of us. Agreed. I can't say my pain is any more important than Van's, but the same applies the other way round :-) So given the stakes in this particular discussion aren't that high, I'll try and summarize the thread over the next few days (or someone can beat me to it if they prefer) and we can ask someone semi-impartial to make a decision. I'd be happy to nominate MvL if he feels so inclined (even though I haven't asked him). Sounds reasonable. I'd suggest that Van should probably report any other examples where someone would benefit from this change - at the moment unless I've misread the thread, it seems like he's the only example of someone who would gain. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
17.03.12 13:57, Paul Moore написав(ла): As there is no way of knowing the Python version without running Python, this is too slow to be practical. Cold start: $ time python3 --version Python 3.1.2 real0m0.073s user0m0.004s sys 0m0.004s Hot start: $ time python3 --version Python 3.1.2 real0m0.007s user0m0.004s sys 0m0.004s ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 17 March 2012 14:07, Serhiy Storchaka storch...@gmail.com wrote:
17.03.12 13:57, Paul Moore написав(ла):
As there is no
way of knowing the Python version without running Python, this is too
slow to be practical.
Cold start:
$ time python3 --version
Python 3.1.2
real 0m0.073s
user 0m0.004s
sys 0m0.004s
Hot start:
$ time python3 --version
Python 3.1.2
real 0m0.007s
user 0m0.004s
sys 0m0.004s
Blame it on Windows or my overloaded PC if you must :-) but I get
perceptible delays on a cold start at times. Plus, I'd probably need
to do this in code that enumerates all installed Pythons and
virtualenvs - that could be 10 installations. I've never tried it in
anger, so I could be worrying over nothing, but to an extent that's my
point - I don't *need* to know the version unless I have to have
version-specific code to define the layout.
And if I were starting Python up, I'd probably be better just
importing sys and sysconfig, and using sys.executable and
sysconfig.get_path('scripts'). And there's the chicken-and-egg problem
- if I don't know the version, I don't know where python.exe is, so
how can I run it to find the version?
Meh. None of this is a real issue. It's just some extra messy coding.
But Van's point is that this proposal gives him less hard coding.
Beyond pointing out that it gives me more, I don't have much to add.
Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/17/2012 10:50 AM, Paul Moore wrote: Meh. None of this is a real issue. It's just some extra messy coding. But Van's point is that this proposal gives him less hard coding. Beyond pointing out that it gives me more, I don't have much to add. I suspect a case could be made that harmonization now will benefit multiple people in, say, 5 years, especially if by then one only supported 3.3+ while supporting multiple platforms. It would be the same rationale as that for 3.0, and especially the bytes to unicode change for text. (As I remember, we are only 3 years in on that one ;-). I leave it to Van to actually explain and make the argument. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 16 March 2012 00:12, Carl Meyer c...@oddbird.net wrote: Changing the directory name is in fact a new and different (and much more invasive) special case, because distutils et al install scripts there, and that directory name is part of the distutils install scheme. Installers don't care where the Python binary is located, so moving it in with the other scripts has very little impact. Two thoughts: 1. The incompatibilities between platforms is precisely the problem that sysconfig is designed to solve, isn't it? So tools in Python will either use sysconfig (and be correct regardless of layout) or should be encouraged to change to use sysconfig (so they are layout-independent). And tools *not* in Python will be platform-specific anyway (I assume no-one is writing Perl scripts to manipulate their Python installation :-)) 2. The differences in layout between a installed Python, uninstalled builds and virtualenvs, on the same platform, are more annoying in practice than any cross-platform differences (at least for me). But again, these are known issues that can be dealt with easily enough (trivially via sysconfig from within Python). If I were tidying up, I would consider renaming Scripts to bin on Windows, and putting the Python executables in there (so there's only one directory to add to PATH, and it uses the common name bin rather than a name that implies that it doesn't contain exes). But that offers no practical benefit, and as Mark says does break existing code, so I don't think it's worth it. If you can get Guido to lend you the time machine keys, I'd support putting it in from Python 1.5 onwards :-) Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/15/2012 6:19 PM, Mark Hammond wrote:
[At least I think that is your proposal - can you confirm that the
directory layouts in your proposal exactly match the directory
layouts in virtual envs on all other platforms? ie, that
inconsistencies like the python{py_version_short} suffix will not
remain?]
Yes, that is the ideal.
Also, I'm still yet to see what exactly becomes easier in your
model? As you mention, most Python code will not care; distutils and
other parts of the stdlib will do the right thing - and indeed,
already do for Windows.
Again, I have stated my use case - cross platform development where the
tools use the directory layout in some way, or where the environment
should be checked into source control.
It is a shame this wasn't done as part of py3k in the first place.
But I assume you would be looking at Python 3.4 for this, right?
No, I would like this for 3.3.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
Carl - Changing the directory name is in fact a new and different (and much more invasive) special case, because distutils et al install scripts there, and that directory name is part of the distutils install scheme. Installers don't care where the Python binary is located, so moving it in with the other scripts has very little impact. So would changing the distutils install scheme in 3.3 - as defined and declared by distutils - lead to a change in your code? Alternatively stated, do you independently figure out that your virtualenv is on Windows and then put things in Scripts, etc, or do you use sysconfig? If sysconfig gave you different (consistent) values across platforms, how would that affect your code? Thanks, VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/16/2012 3:38 AM, Paul Moore wrote: On 16 March 2012 00:12, Carl Meyerc...@oddbird.net wrote: Changing the directory name is in fact a new and different (and much more invasive) special case, because distutils et al install scripts there, and that directory name is part of the distutils install scheme. Installers don't care where the Python binary is located, so moving it in with the other scripts has very little impact. This is very interesting, as it seems to argue against Mark's point. If moving the Python binary is not an issue here, then would this change make it any more/less of an issue? 1. The incompatibilities between platforms is precisely the problem that sysconfig is designed to solve, isn't it? So tools in Python will either use sysconfig (and be correct regardless of layout) or should be encouraged to change to use sysconfig (so they are layout-independent). Right. I want to change the default layout in sysconfig.cfg. 2. The differences in layout between a installed Python, uninstalled builds and virtualenvs, on the same platform, are more annoying in practice than any cross-platform differences (at least for me). But again, these are known issues that can be dealt with easily enough (trivially via sysconfig from within Python). These differences are a major pain for me - and it doesn't make sense that they should need to be worked around each and every time. If I were tidying up, I would consider renaming Scripts to bin on Windows, and putting the Python executables in there (so there's only one directory to add to PATH, and it uses the common name bin rather than a name that implies that it doesn't contain exes). But that offers no practical benefit... This is not a we should be consistent argument - I know that would never fly. I do cross-platform dev all the time (develop on Windows and Mac, deploy on Linux) and so this bites me *every single time* I want to get a consistent layout between these three. That could be because I want my deployment environment to match my development environment(s), it could be because I need to introspect the layout to find some data, or because I want to check in an entire environment into source control. This is not purely aesthetics - this is an issue I deal with all the time. Thanks, VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
Hi, Le 16/03/2012 16:17, Lindberg, Van a écrit : On 3/16/2012 3:38 AM, Paul Moore wrote: 1. The incompatibilities between platforms is precisely the problem that sysconfig is designed to solve, isn't it? So tools in Python will either use sysconfig (and be correct regardless of layout) or should be encouraged to change to use sysconfig (so they are layout-independent). Right. I want to change the default layout in sysconfig.cfg. A few notes: - sysconfig was extracted from distutils to the top level in 2.7 and 3.2, but distutils does not use it (due to the Great Revert two years ago after it was decided at PyCon to freeze distutils and start distutils2); there are unfortunately a few subtle differences between the install schemes in sysconfig and distutils. So even if virtualenv uses sysconfig in 2.7/3.2+, I’m not sure that the venv’s pip will install distutils-based projects in the right places. - packaging uses only sysconfig.cfg - I think a change to distutils install schemes in 3.3 would violate the freeze. Regards ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 16 March 2012 15:17, Lindberg, Van van.lindb...@haynesboone.com wrote: This is not a we should be consistent argument - I know that would never fly. I do cross-platform dev all the time (develop on Windows and Mac, deploy on Linux) and so this bites me *every single time* I want to get a consistent layout between these three. That could be because I want my deployment environment to match my development environment(s), it could be because I need to introspect the layout to find some data, or because I want to check in an entire environment into source control. The only way I can read this to make sense is that you somehow consider the Python installation as part of your development environment (you mentioned source control earlier in the thread - surely you don't manage your Python installation in source control - binaries, stdlib, etc?). I can't see why you would do this, and it certainly doesn't seem like a reasonable thing to do to me. Can you clarify? Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/16/2012 10:53 AM, Paul Moore wrote: The only way I can read this to make sense is that you somehow consider the Python installation as part of your development environment (you mentioned source control earlier in the thread - surely you don't manage your Python installation in source control - binaries, stdlib, etc?). I can't see why you would do this, and it certainly doesn't seem like a reasonable thing to do to me. Can you clarify? I don't check in the python binary itself, nor the stdlib, but I *do* check in the whole installation, including the binaries directory. I like having my deploy environment exactly match my develop environment. So if I do have an executable program, I put it in the binaries directory and check it in. My .hgignore includes python, python.exe, pip, easy_install, etc. - things that are owned by the installation - but it includes everything else. As for the stdlib, I don't check that in, so that portion of the proposal (standardize lib naming) is nice to have, but not essential to me. For example, in the following environment: env/ bin/ python pip easy_install my_script lib/ [stuff] data/ [stuff] src/ my_package I would include bin/my_script, src/, and data/ in my version control. This breaks cross-platform development if bin is named Scripts. CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/16/2012 9:22 AM, Lindberg, Van wrote: On 3/16/2012 10:53 AM, Paul Moore wrote: The only way I can read this to make sense is that you somehow consider the Python installation as part of your development environment (you mentioned source control earlier in the thread - surely you don't manage your Python installation in source control - binaries, stdlib, etc?). I can't see why you would do this, and it certainly doesn't seem like a reasonable thing to do to me. Can you clarify? I don't check in the python binary itself, nor the stdlib, but I*do* check in the whole installation, including the binaries directory. I like having my deploy environment exactly match my develop environment. So I think I'm finally beginning to see the underlying reason why Van is desiring this consistency: It is not that he wants to check in his installation of Python, but that he wants to check in his installation of his packages and scripts into a source control environment, and then be able to check out that source control environment into an installation of Python on another machine of a different architecture. In an environment where a source control system is pervasive and well used, this would be an effective deployment alternative to developing a packaging/distribution solution using distutils, distutels2, packaging, easy_install, eggs, or peanuts, or any other such scheme. But! Source control environments don't lend themselves to being used for anything except exact replication of file and directory structure, so when the different architectures have different directory structures, this deployment technique cannot easily work except, as Van has discussed, by tweaking the development machine's environment to match that of the deployment machines... and that only works in the case where the deployment happens to only one architecture, and the development machine can be tweaked to match... but deploying to multiple machine having different architectures and directory structures would be impossible using the source control deployment technique, because of the different directory structures. If Van stated this goal in this thread, I missed it, and I think it is the missing link in the discussions. If I'm wrong, apologies for the noise. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/16/2012 11:57 AM, Glenn Linderman wrote: So I think I'm finally beginning to see the underlying reason why Van is desiring this consistency: It is not that he wants to check in his installation of Python, but that he wants to check in his installation of his packages and scripts into a source control environment, and then be able to check out that source control environment into an installation of Python on another machine of a different architecture. In an environment where a source control system is pervasive and well used, this would be an effective deployment alternative to developing a packaging/distribution solution using distutils, distutels2, packaging, easy_install, eggs, or peanuts, or any other such scheme. But! Source control environments don't lend themselves to being used for anything except exact replication of file and directory structure, so when the different architectures have different directory structures, this deployment technique cannot easily work except, as Van has discussed, by tweaking the development machine's environment to match that of the deployment machines... and that only works in the case where the deployment happens to only one architecture, and the development machine can be tweaked to match... but deploying to multiple machine having different architectures and directory structures would be impossible using the source control deployment technique, because of the different directory structures. This is exactly correct.CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/16/2012 12:22 PM, Lindberg, Van wrote: env/ bin/ python pip easy_install my_script In http://bugs.python.org/issue14302 Brian Curtin claims After talks at PyCon with several people, python.exe will live in C:\Python33\bin rather than C:\Python33 to come more in line with the Unix layout. This will also simplify another issue with the Path option for the 3.3 installer as well as packaging's target directory for top-level scripts (used to be Scripts/, will be bin/). -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
Hi Van, On 03/16/2012 08:08 AM, Lindberg, Van wrote: Changing the directory name is in fact a new and different (and much more invasive) special case, because distutils et al install scripts there, and that directory name is part of the distutils install scheme. Installers don't care where the Python binary is located, so moving it in with the other scripts has very little impact. So would changing the distutils install scheme in 3.3 - as defined and declared by distutils - lead to a change in your code? Alternatively stated, do you independently figure out that your virtualenv is on Windows and then put things in Scripts, etc, or do you use sysconfig? If sysconfig gave you different (consistent) values across platforms, how would that affect your code? Both virtualenv and PEP 405 pyvenv figure out the platform at venv-creation time, and hard-code certain information about the correct layout for that platform (Scripts vs bin, as well as lib/pythonx.x vs Lib), so the internal layout of the venv matches the system layout on that platform. The key fact is that there is then no special-casing necessary when code runs within the virtualenv (particularly installer code); the same install scheme that would work in the system Python will also Just Work in the virtualenv. I'm not concerned about changes to distutils/sysconfig install schems to make them more compatible across platforms from the POV of virtualenv; we can easily update the current platform-detection code to do the right thing depending on both platform and Python version. I do share Éric's concern about whether distutils' legacy install schemes would be updated or not, and how this would affect backwards compatibility for older installer code, but that's pretty much orthogonal to virtualenv/pyvenv. I don't want to make the internal layout of a virtualenv differ from the system Python layout on the same platform, which (IIUC) was Mark's proposal. Hope that clarifies, Carl signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 17/03/2012 7:22 AM, Carl Meyer wrote: ... I don't want to make the internal layout of a virtualenv differ from the system Python layout on the same platform, which (IIUC) was Mark's proposal. Just to be clear, I made that suggestion in an effort to keep both myself and Van - that the Python executable would remain in the same place for installed Pythons in the interests of b/w compat, but change it in the virtual env in an effort to keep Van happy when working in such environments. I now fully concede that was a dumb idea :) Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 14/03/2012 6:43 AM, VanL wrote: Following up on conversations at PyCon, I want to bring up one of my personal hobby horses for change in 3.3: Fix install layout on Windows, with a side order of making the PATH work better. ... For the sake of brain-storming, how about this: * All executables and scripts go into the root of the Python install. This directory is largely empty now - it is mainly a container for other directories. This would solve the problem of needing 2 directories on the PATH and mean existing tools which locate the executable would work fine. * If cross-platform consistency was desired, then we could consider making other platforms match this. However, if there are issues which might prevent this happening for other platforms (eg, the risk of breaking other 3rd party tools, conventions on the platform ,etc) then it might be worth conceding these considerations apply equally to the Windows installs and we just live with this platform difference. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On Fri, Mar 16, 2012 at 19:53, Mark Hammond skippy.hamm...@gmail.com wrote: For the sake of brain-storming, how about this: * All executables and scripts go into the root of the Python install. This directory is largely empty now - it is mainly a container for other directories. This would solve the problem of needing 2 directories on the PATH and mean existing tools which locate the executable would work fine. How are existing tools locating the executable which would break with a change to bin? As I posted on the tracker, the way which pops in my mind would be to look for C:\\Python%d%d % (x, y) but that's already pretty broken. The people I talked to at PyCon about this were Dino from Microsoft and he nudged the guy next to him to ask the same question (I seem to remember this guy worked for an IDE) -- both of them just wanted to be sure they can still find python.exe's location via the registry, which will be fine. I think we'll add a key to go along with InstallPath - BinaryPath probably makes sense. * If cross-platform consistency was desired, then we could consider making other platforms match this. However, if there are issues which might prevent this happening for other platforms (eg, the risk of breaking other 3rd party tools, conventions on the platform ,etc) then it might be worth conceding these considerations apply equally to the Windows installs and we just live with this platform difference. I don't think we're going to defeat the Unix army with their fleets of distro packagers and torch wielding purists. If anyone's going to move, my money's on Windows. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 17/03/2012 12:07 PM, Brian Curtin wrote: On Fri, Mar 16, 2012 at 19:53, Mark Hammondskippy.hamm...@gmail.com wrote: For the sake of brain-storming, how about this: * All executables and scripts go into the root of the Python install. This directory is largely empty now - it is mainly a container for other directories. This would solve the problem of needing 2 directories on the PATH and mean existing tools which locate the executable would work fine. How are existing tools locating the executable which would break with a change to bin? As I posted on the tracker, the way which pops in my mind would be to look for C:\\Python%d%d % (x, y) but that's already pretty broken. As I just replied in the tracker :) They typically look up the InstallPath key in the registry and look for python.exe there - see the link to that activate.bat file I posted early in the thread. The people I talked to at PyCon about this were Dino from Microsoft and he nudged the guy next to him to ask the same question (I seem to remember this guy worked for an IDE) -- both of them just wanted to be sure they can still find python.exe's location via the registry, which will be fine. I think we'll add a key to go along with InstallPath - BinaryPath probably makes sense. While I wouldn't object to that, it would seem redundant - if the whole point of this is to standardize the locations, then looking for bin/python.exe relative to the existing InstallPath key should also be reliable and hopefully permanent. At the risk of repeating myself too many times, my concern is with 3rd party tools who (a) will break with the new scheme and need to be updated and (b) even after updating will still need the burden of supporting both the old and the new schemes. I simply don't see the benefit that makes this worthwhile. * If cross-platform consistency was desired, then we could consider making other platforms match this. However, if there are issues which might prevent this happening for other platforms (eg, the risk of breaking other 3rd party tools, conventions on the platform ,etc) then it might be worth conceding these considerations apply equally to the Windows installs and we just live with this platform difference. I don't think we're going to defeat the Unix army with their fleets of distro packagers and torch wielding purists. If anyone's going to move, my money's on Windows. Right - but why? Who wins? Where is the evidence of the pain this has caused people over the last 18 years or so since Windows has been doing this? Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
Hi Mark, On 03/16/2012 05:53 PM, Mark Hammond wrote: * All executables and scripts go into the root of the Python install. This directory is largely empty now - it is mainly a container for other directories. This would solve the problem of needing 2 directories on the PATH and mean existing tools which locate the executable would work fine. I hate to seem like I'm piling on now after panning your last brainstorm :-), but... this would be quite problematic for virtualenv users, many of whom do rely on the fact that the virtualenv stuff is confined to within a limited set of known subdirectories, and they can overlay a virtualenv and their own project code with just a few virtualenv directories vcs-ignored. I would prefer either the status quo or the proposed cross-platform harmonization. Carl signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/16/2012 6:25 PM, Carl Meyer wrote: Hi Mark, On 03/16/2012 05:53 PM, Mark Hammond wrote: * All executables and scripts go into the root of the Python install. This directory is largely empty now - it is mainly a container for other directories. This would solve the problem of needing 2 directories on the PATH and mean existing tools which locate the executable would work fine. I hate to seem like I'm piling on now after panning your last brainstorm :-), but... this would be quite problematic for virtualenv users, many of whom do rely on the fact that the virtualenv stuff is confined to within a limited set of known subdirectories, and they can overlay a virtualenv and their own project code with just a few virtualenv directories vcs-ignored. I would prefer either the status quo or the proposed cross-platform harmonization. Yes, it seems fruitless to make directory structure changes without achieving cross-platform consistency. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 17/03/2012 12:25 PM, Carl Meyer wrote: Hi Mark, On 03/16/2012 05:53 PM, Mark Hammond wrote: * All executables and scripts go into the root of the Python install. This directory is largely empty now - it is mainly a container for other directories. This would solve the problem of needing 2 directories on the PATH and mean existing tools which locate the executable would work fine. I hate to seem like I'm piling on now after panning your last brainstorm :-), but... this would be quite problematic for virtualenv users, many of whom do rely on the fact that the virtualenv stuff is confined to within a limited set of known subdirectories, and they can overlay a virtualenv and their own project code with just a few virtualenv directories vcs-ignored. I would prefer either the status quo or the proposed cross-platform harmonization. Yeah, fair enough. I should have indicated it was 1/2 tongue-in-cheek, but figured it worth throwing it out there anyway :) OTOH, the part that wasn't tongue-in-cheek was the part that said why not change the other platforms instead of windows (then wait for the inevitable replies), then so those same reasons apply to Windows too - eg fleets of torch wielding windows admins :) Breaking the few tools I'm concerned about vs asking Van etc to continue taking the pain he feels isn't going to mean the end of the world for any of us. So given the stakes in this particular discussion aren't that high, I'll try and summarize the thread over the next few days (or someone can beat me to it if they prefer) and we can ask someone semi-impartial to make a decision. I'd be happy to nominate MvL if he feels so inclined (even though I haven't asked him). Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/14/2012 6:30 PM, Mark Hammond wrote: So why not just standardize on that new layout for virtualenvs? That sounds like the worst of all worlds - keep all the existing special cases, and add one. The fact is that most code doesn't know about this, only installers or virtual environments. Most code just assumes that distutils does its thing correctly and that binaries are installed... wherever the binaries go. Again, I have experience with this, as I have edited my own install to do this for a couple of years. The breakage is minimal and it makes things much more consistent and easier to use for cross-platform development. Right now we are in front of the knee on major 3.x adoption - I would like to have things be standardized going forth everywhere. Thanks, Van ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 16/03/2012 8:57 AM, VanL wrote:
On 3/14/2012 6:30 PM, Mark Hammond wrote:
So why not just standardize on that new layout for virtualenvs?
That sounds like the worst of all worlds - keep all the existing special
cases, and add one.
I'm not so sure. My concern is that this *will* break external tools
that attempt to locate the python executable from an installed
directory. However, I'm not sure this requirement exists for virtual
environments - such tools probably will not attempt to locate the
executable in a virtual env as there is no standard place for a virtual
env to live.
So having a standard layout in the virtual envs still seems a win - we
keep the inconsistency in the layout of the installed Python, but
tools which work with virtualenvs still get a standardized layout.
[At least I think that is your proposal - can you confirm that the
directory layouts in your proposal exactly match the directory layouts
in virtual envs on all other platforms? ie, that inconsistencies like
the python{py_version_short} suffix will not remain?]
Just to be completely clear, my current concern is only with the
location of the executable in an installed Python.
The fact is that most code doesn't know about this, only installers or
virtual environments. Most code just assumes that distutils does its
thing correctly and that binaries are installed... wherever the binaries
go.
Of course - but this raises 2 points:
* I'm referring to *external* tools that launch Python. They obviously
need to know where the binaries are to launch them. Eg, the PEP397
launcher; the (admittedly few) people who use the launcher would need to
upgrade it to work under your scheme. Ditto *all* other such tools that
locate and launch Python.
* most code isn't a high enough bar. If we only considered such
anecdotes, most backwards compatibility concerns would be moot.
Again, I have experience with this, as I have edited my own install to
do this for a couple of years. The breakage is minimal and it makes
things much more consistent and easier to use for cross-platform
development.
All due respect, but I'm not sure that is a large enough sample to draw
any conclusions from. I've offered 2 concrete examples of things that
*will* break and I haven't looked for others.
Also, I'm still yet to see what exactly becomes easier in your model?
As you mention, most Python code will not care; distutils and other
parts of the stdlib will do the right thing - and indeed, already do
for Windows. So the proposal wants to change distutils and other parts
of the stdlib even though most code won't notice. But the code that
will notice will be broken!
So I dispute it is easier for anyone; I agree it is more consistent,
but given the *certainty* external tools will break, I refer to you the
Zen of Python's thoughts on consistency ;)
Right now we are in front of the knee on major 3.x adoption - I would
like to have things be standardized going forth everywhere.
It is a shame this wasn't done as part of py3k in the first place. But
I assume you would be looking at Python 3.4 for this, right? So if
people start working with Python 3.3 now and finds this change in 3.4,
we are still asking them to take the burden of supporting the multiple
locations. I guess I'd be less concerned if we managed to get it into
3.3 and also recommended to people that they should ignore 3.2 and
earlier when porting their tools/libraries to 3.x.
I think I've made all the points I can make in this discussion. As I
mentioned at the start, I'm not quite -1 on the idea, so I'm not going
to push this barrow any further (although I'm obviously happy to clarify
anything I've said...)
Cheers,
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 03/15/2012 04:19 PM, Mark Hammond wrote: On 16/03/2012 8:57 AM, VanL wrote: On 3/14/2012 6:30 PM, Mark Hammond wrote: So why not just standardize on that new layout for virtualenvs? That sounds like the worst of all worlds - keep all the existing special cases, and add one. I'm not so sure. My concern is that this *will* break external tools that attempt to locate the python executable from an installed directory. However, I'm not sure this requirement exists for virtual environments - such tools probably will not attempt to locate the executable in a virtual env as there is no standard place for a virtual env to live. So having a standard layout in the virtual envs still seems a win - we keep the inconsistency in the layout of the installed Python, but tools which work with virtualenvs still get a standardized layout. The implementation of virtualenv (and especially PEP 405 pyvenv) are largely based around making sure that the internal layout of a virtualenv is identical to the layout of an installed Python on that same platform, to avoid any need to special-case virtualenvs in distutils. The one exception to this is the location of the python binary itself in Windows virtualenvs; we do place it inside Scripts\ so that the virtualenv can be activated by adding only a single path to the shell PATH. But I would be opposed to any additional special-casing of the internal layout of a virtualenv that would require tools installing software inside virtualenv to use a different install scheme than when installing to a system Python. In other words, I would much rather that tools have to understand a different layout between Windows virtualenvs and Unixy virtualenvs (because most tools don't have to care anyway, distutils just takes care of it, and to the extent they do have to care, they have to adjust anyway in order to work with installed Pythons) than that they have to understand a different layout between virtualenv and non- on the same platform. To as great an extent as possible, tools shouldn't have to care whether they are dealing with a virtualenv. A consistent layout all around would certainly be nice, but I'm not venturing any opinion on whether it's worth the backwards incompatibility. Carl signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 16/03/2012 10:48 AM, Carl Meyer wrote: ... The implementation of virtualenv (and especially PEP 405 pyvenv) are largely based around making sure that the internal layout of a virtualenv is identical to the layout of an installed Python on that same platform, to avoid any need to special-case virtualenvs in distutils. The one exception to this is the location of the python binary itself in Windows virtualenvs; we do place it inside Scripts\ so that the virtualenv can be activated by adding only a single path to the shell PATH. But I would be opposed to any additional special-casing of the internal layout of a virtualenv ... Unless I misunderstand, that sounds like it should keep everyone happy; there already is a special case for the executable on Windows being in a different place in an installed layout vs a virtual-env layout. Changing this to use bin instead of Scripts makes the virtualenv more consistent across platforms and doesn't impose any additional special-casing for Windows (just slightly changes the existing special-case :) Thanks, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 03/15/2012 05:10 PM, Mark Hammond wrote: On 16/03/2012 10:48 AM, Carl Meyer wrote: The implementation of virtualenv (and especially PEP 405 pyvenv) are largely based around making sure that the internal layout of a virtualenv is identical to the layout of an installed Python on that same platform, to avoid any need to special-case virtualenvs in distutils. The one exception to this is the location of the python binary itself in Windows virtualenvs; we do place it inside Scripts\ so that the virtualenv can be activated by adding only a single path to the shell PATH. But I would be opposed to any additional special-casing of the internal layout of a virtualenv ... Unless I misunderstand, that sounds like it should keep everyone happy; there already is a special case for the executable on Windows being in a different place in an installed layout vs a virtual-env layout. Changing this to use bin instead of Scripts makes the virtualenv more consistent across platforms and doesn't impose any additional special-casing for Windows (just slightly changes the existing special-case :) Changing the directory name is in fact a new and different (and much more invasive) special case, because distutils et al install scripts there, and that directory name is part of the distutils install scheme. Installers don't care where the Python binary is located, so moving it in with the other scripts has very little impact. Carl signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 14/03/2012 6:43 AM, VanL wrote:
Following up on conversations at PyCon, I want to bring up one of my
personal hobby horses for change in 3.3: Fix install layout on Windows,
with a side order of making the PATH work better.
Short version:
1) The layout for the python root directory for all platforms should be
as follows:
stdlib = {base/userbase}/lib/python{py_version_short}
platstdlib = {base/userbase}/lib/python{py_version_short}
purelib = {base/userbase}/lib/python{py_version_short}/site-packages
platlib = {base/userbase}/lib/python{py_version_short}/site-packages
include = {base/userbase}/include/python{py_version_short}
As per comments later in the thread, I'm -1 on including
python{py_version_short} in the lib directories for a number of
reasons; one further reason not outlined is that it would potentially
make running Python directly from a built tree difficult. For the same
reason, I'm also -1 on having that in the include dir.
scripts = {base/userbase}/bin
We should note that this may cause pain for a number of projects - I've
seen quite a few projects that already assume Scripts on Windows - eg,
virtualenv and setuptools IIRC - and also assume the executable is where
it currently lives - one example off the top of my head is the mozilla
jetpack project - see:
https://github.com/mozilla/addon-sdk/blob/master/bin/activate.bat#L117
This code (and any other code looking in Scripts on Windows) will fail
and need to be updated with this change. Further, assuming such
projects want to target multiple Python versions, it will need to keep
the old code and check the new location.
Another bit of code which would be impacted is the PEP397 launcher; it
too would have to grow version specific logic to locate the executable.
So while I'm not (yet) -1 on the general idea, I'm close. I guess I
don't understand how the benefits this offers outweigh the costs to 3rd
parties. Given the work on making Python more virtualenv friendly,
can't we just wear the costs of the existing scheme in the stdlib and
avoid breaking the code already out there? IOW, who exactly will
benefit from this, and how does the cost of them supporting the existing
scheme compare to the cost of the breakage to multiple 3rd parties?
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python install layout and the PATH on win32
In
view-source:http://mail.python.org/pipermail/python-dev/2012-March/117586.html
van.lindberg at gmail.com posted:
1) The layout for the python root directory for all platforms should be
as follows:
stdlib = {base/userbase}/lib/python{py_version_short}
platstdlib = {base/userbase}/lib/python{py_version_short}
purelib = {base/userbase}/lib/python{py_version_short}/site-packages
platlib = {base/userbase}/lib/python{py_version_short}/site-packages
include = {base/userbase}/include/python{py_version_short}
scripts = {base/userbase}/bin
data = {base/userbase}
Why?
Pure python vs compiled C doesn't need to be separated at the directory
level, except for cleanliness.
Some (generally unix) systems prefer to split the libraries into several
additional pieces depending on CPU architecture.
The structure listed above doesn't have a location for docs.
Some packages (such as tcl) may be better off in their own area.
What is data? Is this an extra split compared to today, or does it
refer to things like LICENSE.txt, README.txt, and NEWS.txt?
And even once I figure out where files have moved, and assume that
the split is perfect -- what does this buy me over the current
situation? I was under the impression that programs like distutils
already handled finding the appropriate directories for a program;
if you're rewriting that logic, you're just asking for bugs on a
strange platform that you don't use.
If you're looking for things interactively, then platform conventions
are probably more important than consistency across platforms. If you
disagree, you are welcome to reorganize your personal linux installation
so that it matches windows, and see whether it causes you any problems.
... We *already* have this. The only difference in this proposal is
that we go from py_version_nodot to py_version_short, i.e. from
c:\python33\lib\python33
to
c:\python33\lib\python3.3
I have not seen that redundancy before on windows.
I'm pretty sure that it is a relic of your Linux provider wanting
to support multiple python versions using shared filesystems. The
Windows standard is to use a local disk, and to bundle it all up
into its own directory, similar to the way that java apps sometimes
ship with their own JVM.
Also note that using the dot in a directory name is incautious.
I haven't personally had trouble in several years, but doing so is
odd enough that some should be expected. Python already causes
some grief by not installing in Program Files, but that is at
least justified by the spaces in filenames problem; what is the
advantange of 3.3?
I'm using windows, and I just followed the defaults at installation.
It is possible that the installer continued to do something based
on an earlier installation, but I don't think this machine has ever
had a customized installation of any python version.
C:\python32\*
Everything is under here; I assume {base/userbase} would be
set to C:\python32
As is customary for windows, the base directory contains the
license/readme/news and all executables that the user is
expected to use directly. (python.exe, pythonw.exe. It also
contains w9xpopen.exe that users do not use, but that too is
fairly common.)
There is no data directory.
Subdirectories are:
C:\python32\DLLs
In additional to regular DLL files, it contains .pyd files
and icons. It looks like modules from the stdlib that happen
to be written in C. Most users will never bother to look here.
C:\python32\Doc
A .chm file; full html would be fine too, but removing it
would be a bad idea.
C:\python32\include
These are the header files, though most users will never have
any use for them, as there isn't generally a compiler.
C:\python32\Lib
The standard library -- or at least the portion implemented
in python.
Note that site-packages is a subdirectory here. It doesn't
happen to have an __init__.py, but to an ordinary user it
looks just like any other stdlib package, such as xml or
multiprocessing.
I personally happen to keep things in subdirectories of
site-packages, but I can't say what is standard.
Moving site-packages out of the Lib directory might make
sense, but probably isn't worth the backward compatibility hit.
C:\python32\libs
.lib files. I'm not entirely sure what these (as opposed to
the DLLs) are for; lib files aren't that common on windows.
My machine does not appear to have any that aren't associated
with cross-platform tools or unix emulation.
C:\python32\tcl
Note that this is in addition to associated files under DLLs
and libs. I would prefer to see them in one place, but
moving it in with non-tcl stuff would not be an improvement.
Most users will never look (or care); those that do usually
appreciate knowing that, for example, the dde subdirectory
is for tcl.
C:\python32\Tools
This has three subdirectories (i18n,
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/13/2012 9:58 PM, Terry Reedy wrote: Given that we already repeat it, isn't it better to be consistent? But there is no repetition currently on Windows installations. I though you were just proposing to switch lib (lower-cased, and scripts renamed as bin, and pythonxx). So I do not think I yet understand what the proposal is and how it would be different from what I have now. Aaah, I was looking at my local installations, which happen to be nt-user. Looking at the system installation (nt) I see that there is no repetition. I am fine with keeping the distinction between base installs (no py_version) and user installs (including a py_version). I would just suggest that when you have a py_version, it be the same py_version (not dots sometimes, nodot other times). It also begs the question as to whether the py_version is *ever* needed. Thanks, Van ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/13/2012 9:57 PM, VanL wrote:
On Mar 13, 2012, at 8:37 PM, Martin v. Löwis mar...@v.loewis.de wrote:
The installation will end up in
c:\python33\lib\python3.3
which has the software name and version twice in the path.
Do we *really* need this?
We *already* have this. The only difference in this proposal is that we go
from py_version_nodot to py_version_short, i.e. from
c:\python33\lib\python33
to
c:\python33\lib\python3.3
Given that we already repeat it, isn't it better to be consistent?
Is it?
I think you are confusing two different configuration sections in
sysconfig.cfg:
[nt]
stdlib = {base}/Lib
platstdlib = {base}/Lib
purelib = {base}/Lib/site-packages
platlib = {base}/Lib/site-packages
include = {base}/Include
platinclude = {base}/Include
scripts = {base}/Scripts
data = {base}
[nt_user]
stdlib = {userbase}/Python{py_version_nodot}
platstdlib = {userbase}/Python{py_version_nodot}
purelib = {userbase}/Python{py_version_nodot}/site-packages
platlib = {userbase}/Python{py_version_nodot}/site-packages
include = {userbase}/Python{py_version_nodot}/Include
scripts = {userbase}/Scripts
data = {userbase}
--
Scott Dial
sc...@scottdial.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/14/2012 9:53 AM, Jim J. Jewett wrote:
In
view-source:http://mail.python.org/pipermail/python-dev/2012-March/117586.html
van.lindberg at gmail.com posted:
1) The layout for the python root directory for all platforms should be
as follows:
stdlib = {base/userbase}/lib/python{py_version_short}
platstdlib = {base/userbase}/lib/python{py_version_short}
purelib = {base/userbase}/lib/python{py_version_short}/site-packages
platlib = {base/userbase}/lib/python{py_version_short}/site-packages
include = {base/userbase}/include/python{py_version_short}
scripts = {base/userbase}/bin
data = {base/userbase}
Why?
Pure python vs compiled C doesn't need to be separated at the directory
level, except for cleanliness.
I am deliberately being cautious here. I actually agree with you. I am
only suggesting we maintain all of these different distinctions because
that is what we have already.
You can see what we have currently at
http://hg.python.org/distutils2/file/2cec52b682a9/distutils2/_backport/sysconfig.cfg
I am *not* suggesting that docs, etc change at all - that is included in
a different part of the configuration and is not modified by what I
propose here (lines 1-26).
As noted earlier in the thread, I also change my proposal to maintain
the existing differences between system installs and user installs.
Thus, the only place I am proposing changing are the values for the keys
listed above. Specifically, this (lines 57-65 in the file above):
[nt]
stdlib = {base}/Lib
platstdlib = {base}/Lib
purelib = {base}/Lib/site-packages
platlib = {base}/Lib/site-packages
include = {base}/Include
platinclude = {base}/Include
scripts = {base}/Scripts
data = {base}
Would become this:
[nt]
stdlib = {base}/lib
platstdlib = {base}/lib
purelib = {base}/lib/site-packages
platlib = {base}/lib/site-packages
include = {base}/include
platinclude = {base}/include
scripts = {base}/bin
data = {base}
and this (lines 86-93):
[nt_user]
stdlib = {userbase}/Python{py_version_nodot}
platstdlib = {userbase}/Python{py_version_nodot}
purelib = {userbase}/Python{py_version_nodot}/site-packages
platlib = {userbase}/Python{py_version_nodot}/site-packages
include = {userbase}/Python{py_version_nodot}/Include
scripts = {userbase}/Scripts
data = {userbase}
would become this:
[nt_user]
stdlib = {userbase}/python{py_version_short}
platstdlib = {userbase}/python{py_version_short}
purelib = {userbase}/python{py_version_nodot}/site-packages
platlib = {userbase}/python{py_version_nodot}/site-packages
include = {userbase}/python{py_version_nodot}/include
scripts = {userbase}/bin
data = {userbase}
... if you're rewriting that logic, you're just asking for bugs on a
strange platform that you don't use.
I am not rewriting the logic - the logic is driven by these
configuration values. And this is a platform I use, and that is why this
drives me crazy!
Subdirectories are:
You forgot one:
C:\python32\Scripts
Would change to C:\python32\bin. The python binary and scripts meant to
be run direction (easy_install, etc) would all go in this directory.
C:\python32\DLLs
Would not change.
C:\python32\Doc
Would not change.
C:\python32\include
Would be specified as lower case only - but otherwise would not change.
C:\python32\Lib
Would be specified as lower case only - but otherwise would not change.
C:\python32\libs
Would not change.
C:\python32\tcl
Would not change.
C:\python32\Tools
This proposal does not change this, although I do think that this could
be eliminated or made into examples.
Thanks,
Van
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/14/2012 10:09 AM, Scott Dial wrote:
I think you are confusing two different configuration sections in
sysconfig.cfg:
[nt]
stdlib = {base}/Lib
platstdlib = {base}/Lib
purelib = {base}/Lib/site-packages
platlib = {base}/Lib/site-packages
include = {base}/Include
platinclude = {base}/Include
scripts = {base}/Scripts
data = {base}
[nt_user]
stdlib = {userbase}/Python{py_version_nodot}
platstdlib = {userbase}/Python{py_version_nodot}
purelib = {userbase}/Python{py_version_nodot}/site-packages
platlib = {userbase}/Python{py_version_nodot}/site-packages
include = {userbase}/Python{py_version_nodot}/Include
scripts = {userbase}/Scripts
data = {userbase}
I was lumping them together, yes, but now note that I modify the
proposal to maintain this distinction. These would change to:
[nt]
stdlib = {base}/lib
platstdlib = {base}/lib
purelib = {base}/lib/site-packages
platlib = {base}/lib/site-packages
include = {base}/include
platinclude = {base}/include
scripts = {base}/bin
data = {base}
[nt_user]
stdlib = {userbase}/python{py_version_short}
platstdlib = {userbase}/python{py_version_short}
purelib = {userbase}/python{py_version_short}/site-packages
platlib = {userbase}/python{py_version_short}/site-packages
include = {userbase}/python{py_version_short}/include
scripts = {userbase}/bin
data = {userbase}
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
