Re: [Python-Dev] Accepting PEP 397

[Mark Hammond]

On 22/06/2012 8:14 AM, Brian Curtin wrote:

On Wed, Jun 20, 2012 at 11:54 AM, Brian Curtin  wrote:

As the PEP czar for 397, after Martin's final updates, I hereby
pronounce this PEP "accepted"!

Thanks to Mark Hammond for kicking it off, Vinay Sajip for writing up
the code, Martin von Loewis for recent updates, and everyone in the
community who contributed to the discussions.

I will begin integration work this evening.


It's in. http://hg.python.org/cpython/rev/a7ecbb2ad967

Thanks all!


Awesome - thank you!

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 397 - Last Comments

[Mark Hammond]

Sorry, but I missed the announcement of an updated PEP.

It looks good to me!  Also, I see no reason not to always use a 32bit 
version of the launcher other than (a) the 64bit code already exists and 
works and (b) it might mean it is no longer possible to do a complete 
build of a 64bit Python without the 32bit compilers installed.  But (b) 
is really only a theoretical problem so I think in practice it would be 
fine either way.


Thanks to Martin for updating it - I agree it is vastly improved!

Cheers,

Mark

On 19/06/2012 2:31 PM, Brian Curtin wrote:

Martin approached me earlier and requested that I act as PEP czar for
397. I haven't been involved in the writing of the PEP and have been
mostly observing from the outside, so I accepted and hope to get this
wrapped up quickly and implemented in time for the beta. The PEP is
pretty complete, but there are a few outstanding issues.

On Mon, Jun 18, 2012 at 1:05 PM, Terry Reedy  wrote:

"Independent installations will always only overwrite newer versions of the
launcher with older versions." 'always only' is a bit awkward and the
sentence looks backwards to me. I would expect only overwriting older
versions with newer versions.


Agreed, I would expect the same. I would think taking out the word
"only" and then flipping newer and older in the sentence would correct
it.

On Mon, Jun 18, 2012 at 1:05 PM, Terry Reedy  wrote:

These seem contradictory:

"The 32-bit distribution of Python will not install a 32-bit version of the
launcher on a 64-bit system."

I presume this mean that it will install the 64-bit version and that there
will always be only one version of the launcher on the system.

"On 64bit Windows with both 32bit and 64bit implementations of the same
(major.minor) Python version installed, the 64bit version will always be
preferred.  This will be true for both 32bit and 64bit implementations of
the launcher - a 32bit launcher will prefer to execute a 64bit Python
installation of the specified version if available."

This implies to me that the 32bit installation *will* install a 32bit
launcher and that there could be both versions of the launcher installed.


I took that as covering an independently-installed launcher.

You could always install your own 32-bit launcher, and it'd prefer to
launch a binary matching the machine type. So yes, there could be
multiple launchers installed for different machine types, and I'm not
sure why we'd want to (or how we could) prevent people from installing
them. You could have a 64-bit launcher available system-wide in your
Windows folder, then you could have a 32-bit launcher running out of
C:\Users\Terry for some purposes.

Martin - is that correct?

===

Outside of Terry's concerns, I find the updated PEP almost ready to go
as-is. Many of the updates were in line with what Martin and I briefly
talked about at PyCon, and I believe some of them came out of previous
PEP discussions on here, so I see nothing unexpected at this point.

My only additional comment would be to have the "Configuration file"
implementation details supplemented with a readable example of where
the py.ini file should be placed. On my machine that is
"C:\Users\brian\AppData\Local", rather than making people have to run
that parameter through the listed function via pywin32.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/skippy.hammond%40gmail.com



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 405 (Python Virtual Environments) and Windows script support

[Mark Hammond]

Vinay originally wrote:

PEP 397 (Python launcher for Windows) has not yet been accepted, so there still
needs to be some way of natively launching scripts in Windows which is
equivalent to /path/to/venv/bin/foo. The way setuptools (and hence Distribute)
does this is to shadow each script with an executable: whereas a script foo
would be simply placed in /path/to/venv/bin/ on POSIX, on Windows, the files
foo.exe and foo-script.py (or foo-script.pyw) are placed in
\path\to\venv\Scripts. The user can run \path\to\venv\Scripts\foo just as on
POSIX.

The foo.exe file is just a copy of a stock launcher executable which finds its
name from the C argv[0], and based in that name (foo in this case), invokes
foo-script.py or foo-script.pyw with the appropriate Python interpreter.


I don't understand the relationship between this "stock launcher" and 
the PEP 397 launcher.  They seem to have quite distinct requirements 
without much overlap.  Specifically, I'm not aware that the current PEP 
397 implementation could perform the same role as the "stock launcher" - 
IIUC, it has no special handling of the "-script" suffix or special 
logic based around its argv[0].


So unless I'm mistaken, even with PEP 397 accepted, either this "stock 
launcher" is still necessary anyway or the PEP398 launcher would need 
the addition of new features so it could replace the stock launcher.


FWIW, Vinay and I exchanged some private mail recently about how to best 
integrate the PEP397 launcher with virtualenvs - and while we both 
agreed it would be nice, we couldn't come up with anything worthwhile. 
Having the launcher be aware of a virtualenv when invoked via file 
associations is problematic - for example, Windows Explorer is unlikely 
to have a virtualenv configured in its environment.  Even when 
considering just command-line usage there are some edge-cases that make 
things problematic (eg, what if a script in a venv nominates a specific 
Python version via a shebang line?  What if a venv is activated but the 
user/launcher attempts to execute a script outside the venv? etc.)


On 29/05/2012 9:24 AM, Nick Coghlan wrote:
...


It would have been better if the issue of script management on Windows
had been raised in PEP 405 itself - I likely would have declared PEP
397 a dependency *before* accepting it (even if that meant the feature
missed the alpha 4 deadline and first appeared in beta 1, or
potentially even missed 3.3 altogether).

However, I'm not going to withdraw the acceptance of the PEP over this
- while I would have made a different decision at the time given the
additional information (due to the general preference to treat Windows
as a first class deployment target), I think reversing my decision now
would make the situation worse rather than better.

That means the important question is what needs to happen before beta
1 at the end of June. As I see it, we have two ways forward:

1. My preferred option: bring PEP 397 up to scratch as a specification
for the behaviour of the Python launcher (perhaps with Vinay stepping
up as a co-author to help Mark if need be), find a BDFL delegate (MvL?
Brian Curtin?) and submit that PEP for acceptance within the next few
weeks. The updated PEP 397 should include an explanation of exactly
how it will help with the correct implementation of PEP 405 on Windows
(this may involve making the launcher pyvenv aware).


As above, it isn't clear to me how the additional complexity and list of 
caveats in real use make it worthwhile to have the PEP397 launcher 
pyvenv aware.



2. The fallback option: remove the currently checked in build
artifacts from source control and incorporate them into the normal
Windows build processes (both the main VS 2010 process, and at least
the now-legacy VS 2008 process)

For alpha 4, I suggest going with MvL's suggestion - drop the binaries
from Mercurial and accept that this aspect of PEP 405 simply won't
work on Windows until the first beta.


Agreed - ISTM that this stock launcher is probably going to need to 
co-exist with the PEP397 launcher for the long term.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)

[Mark Hammond]

On 23/03/2012 7:10 PM, Paul Moore wrote:

On 23 March 2012 03:20, Brian Curtin  wrote:

Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden have
all expressed that they have existing tools that would break and would need
to be adjusted to match the new location of the python.exe, because that
location is assumed to be at the root of the python install.


Isn't the proposed "BinaryDir" registry key helpful here? It's not
like we're telling people to fend for themselves -- we'll tell you
where it's at.


It won't help me much. I either check a key and fall back on the old
method, or check in bin and fall back on the old method. No major
difference. The key is slightly worse, as I'm already looking in the
filesystem, so why open a registry key, but it's mostly irrelevant.


That's a really good point.  On reflection, the 2 tools I've been using 
as examples are already sniffing around the file-system relative to the 
install path, looking in the root and the PCBuild directories.  The 
simplest approach for these tools to take is to simply sniff the bin 
directory too - so they are unlikely to refer to the BinaryDir key at all.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)

[Mark Hammond]

[snipped some CCs]

On 23/03/2012 2:20 PM, Brian Curtin wrote:
...

I get that tools could be affected. I had two IDE makers at PyCon
immediately throw up red flags to this change. I think one of them was
about to charge the stage during my talk. When it was mentioned that
we could point them to the proper location, they breathed a sigh of
relief and said "cool, do it". If a registry key pointing you to
python.exe (rather, the directory) right now in Python<  3.3 works,
why doesn't another one pointing you to python.exe in Python>= 3.3
work?


It will work.  The fact MvL is proposing the conservative approach of 
landing this in 3.5+ and have 3.3+ include the *new* registry key means 
I'm willing to reluctantly accept it rather than aggressively oppose it. 
 Tools then have a chance to adapt to the new key.  If the proposal 
moved any faster, existing tools which only use the old key would break 
without warning.  The fact they need to change at all is unfortunate, 
but the timescale proposed means we can at least say we warned them.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)

[Mark Hammond]

I'm responding to both of Van's recent messages in one:

On 23/03/2012 1:47 AM, VanL wrote:

[PART 2: Moving the python binary]

...

A regular complaint of those new to Python on windows (and new to
programming generally) has been that one of the first things that
they need to do is to edit the PATH to allow Python to be run. In
particular, this is the difficult sequence:

1. Install python. 2. Open up a shell and run "python" 3. Use pip or
easy_install to install regetron (a package that installs an
executable file). 4. Run regetron.

...

One possible response here is that the moving of the python.exe
binary and the setting of the PATH would be tied to an
unchecked-by-default installer option, making an explicit user choice
needed to invoke the new functionality.


Given an off-by-default setting, I fail to see how it fixes your
"difficult sequence" above.  What would the instructions above now say?
That the user should re-install Python ensuring to set that checkbox?
Cover both cases, including how the user can tell if it is on the PATH
and how to fix it otherwise?  Something else?


Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden
have all expressed that they have existing tools that would break
and would need to be adjusted to match the new location of the
python.exe, because that location is assumed to be at the root of the
python install.

A related issue is that this portion of the proposal has met with
some resistance, but not much support here on Python-dev. The reason
for that is selection bias: Those who are on Python-dev are much more
likely to have tools that do advanced things with Python, such as
introspect on the location of the binary, and are also much more
likely to be comfortable with things like editing the PATH on
windows. In contrast, the people that have trouble with this issue
are those that are newest to Python and programming generally - those
for whom editing the PATH is a challenge and whom are likely to be
confused by the distinction between python.exe and a python program -
and why, even after they add python to the path, the python program
is not directly executable.


Here you are referring to the PATH, but that isn't really where the
objections are.  I would claim a selection bias on Python-dev, where
subscribers are less likely to use Windows regularly for development and
therefore less likely to have developed or use tools for finding and
launching Python.  IMO, the lack of objections on Python-dev to renaming
the binary directory is the same reason you aren't seeing overwhelming 
*support* for the change either.  Without the perspective of being 
regular Windows users, people are happy to agree "consistent is better". 
 All other things being equal, I'd agree too.


Really, we have just one anecdote from you about your process and as
Paul says, no attempt to outline other alternatives.  For example,
couldn't your "activate.bat" add both Scripts *and* bin to the PATH
whereas your "activate.sh" adds just "bin"?


I have been running like this for several years across multiple
Python versions, so I have experience with the "breakage" from this
part of the proposal. I have found four packages that would need to
be updated: Pip, virtualenv, PyPM, and Egginst would need 1-2 line
patches.


With all due respect, I find this disingenuous.  Your lack of experience 
with the tools that are out there doesn't mean they don't exist and I've 
already offered a couple of examples. I certainly can't claim to know 
what most of them are; I expect that I am underestimating them.  IMO, 
your list is a fraction of the tools impacted.



I have these patches, I would/could provide them. Generally these tools have 
something like:

if platform == 'win32':
  bin_dir = 'Scripts'
else:
  bin_dir = 'bin'

The patches just remove the special casing - bin_dir just gets set to 'bin'.


So none of those tools need to work with previous Python versions?  But 
even if what you say is strictly true, I don't think a reasonable 
response to "but what about backwards compatibility and tool breakage" 
is "the breakage is simple and the fix is trivial" - the bar has never 
been that low for changes to the language itself.  I don't see why 
tooling around the language shouldn't be held to any less account.


So my summary of the situation is:

* There has been *exactly one* concrete case listed that would benefit 
from this, and I believe that one case can be mitigated by you having 2 
directories on the PATH in Windows and one on other platforms.


* You yourself listed 4 tools that would need to change to support this. 
 I've listed a further 2, and Paul and Tim both indicated they would be 
impacted.  ActiveState and Enthought haven't been canvassed.  I suspect 
this is the t

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 22/03/2012 1:22 AM, Lindberg, Van wrote:

Mark, MAL, Martin, Tarek,

Could you comment on this?


Eric is correct - tools will be broken by this change.  However, people 
seem willing to push forward on this and accept such breakage as the 
necessary cost.


MAL, in his followup, asks what the advantages are of such a change. 
I've actually been asking for the same thing in this thread and the only 
real answer I've got is "consistency".  So while I share MAL's concerns, 
people seem willing to push forward on this anyway, without the benefits 
having been explained.


IOW, this isn't the decision I would make, but I think I've already made 
that point a number of times in this thread.  Beyond that, there doesn't 
seem much for me to add...


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 21/03/2012 9:45 AM, R. David Murray wrote:

On Wed, 21 Mar 2012 09:09:38 +1100, Mark Hammond  
wrote:

On 21/03/2012 5:50 AM, Merlijn van Deen wrote:

I asked a question about this on IRC, to which the response was that
there were two main reasons to install python in c:\pythonxy:

1 - issues due to spaces ('Program Files') or non-ascii characters in
the path ('Fișiere Program' on a Romanian windows). These issues are
supposed to be fixed by now (?).
2 - issues due to permissions - installing python / packages in
%programfiles% may require administrator rights.


Apart from personal preference (ie, I prefer the status quo here), the
second issue is a bit of a killer.  Even an administrator can not write
to Program Files unless they are using an "elevated" process (ie,
explicitly use "Run as Administrator" and confirm the prompt.

This means that any installer wanting to write .py files into the Python
install must be elevated, and any Python process wanting to generate
.pyc files must also be elevated.  So even if an installer does arrange
elevation, unless that installer also compiles all .pyc and .pyo files
at install time, Python would fail to generate the .pyc files on first
use.  While Python will probably fail silently and still continue to
work, it will have a significant performance impact.


So windows requires admin privileges to install to Program Files, but
not to install to '/'?  How novel.  (You can perhaps tell that I'm
not a windoows user).  My understanding, though, is that Python
does make a distinction between a system install of Python and
a per-user one, so I don't think your objection really applies.


I think it does.  Consider I've installed Python as a "system install". 
 Now I want to install some other package - ideally that installer will 
request elevation - all well and good - the .py files are installed. 
However, next time I want to run Python, it will fail to generate the 
.pyc files - even though I'm an administrator.  I would need to 
explicitly tell Python to execute "as administrator" (or run it from an 
already elevated command-prompt) to have things work as expected.  Thus, 
the "usual" case would be that Python is unable to update any files in 
its install directory.


If Python installed for a single user didn't install into Program Files 
(which it probably couldn't do without an administrator providing 
credentials anyway) then it wouldn't be a problem - but then we have 
multiple possible default install locations, which sounds like more 
trouble than it is worth...



That said, there is an open bug in the tracker about the insecurity
of a system install of python (exactly that the files are writable
by anyone).  So that would have to be solved first.  I'd say this
is definitely a separate issue from Van's discussion, and the *only*
reason one might want to tie them together at all is "well, we're
changing the directory layout anyway".


Agreed.

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 21/03/2012 5:50 AM, Merlijn van Deen wrote:

On 13 March 2012 20:43, VanL  wrote:

Following up on conversations at PyCon, I want to bring up one of my
personal hobby horses for change in 3.3: Fix install layout on Windows, with
a side order of making the PATH work better.


As this is being considered an 'incompatible change' on the bug
tracker item [1] in any case, I'd like to mention that this might also
be a convenient moment to re-think the default install location. After
all, software is supposed to be installed in %programfiles% on
windows, not in c:\.

I asked a question about this on IRC, to which the response was that
there were two main reasons to install python in c:\pythonxy:

1 - issues due to spaces ('Program Files') or non-ascii characters in
the path ('Fișiere Program' on a Romanian windows). These issues are
supposed to be fixed by now (?).
2 - issues due to permissions - installing python / packages in
%programfiles% may require administrator rights.


Apart from personal preference (ie, I prefer the status quo here), the 
second issue is a bit of a killer.  Even an administrator can not write 
to Program Files unless they are using an "elevated" process (ie, 
explicitly use "Run as Administrator" and confirm the prompt.


This means that any installer wanting to write .py files into the Python 
install must be elevated, and any Python process wanting to generate 
.pyc files must also be elevated.  So even if an installer does arrange 
elevation, unless that installer also compiles all .pyc and .pyo files 
at install time, Python would fail to generate the .pyc files on first 
use.  While Python will probably fail silently and still continue to 
work, it will have a significant performance impact.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 21/03/2012 1:08 AM, Lindberg, Van wrote:

On 3/20/2012 5:48 AM, Mark Hammond wrote:

While I'm still unclear on the actual benefits of this, Martin's
approach strikes a reasonable compromise so I withdraw my objections.



Ok. I was out of town and so could not respond to most of the latest
discussion.

A question for you Mark, Paul, (and anyone else): Éric correctly points
out that there are actually two distinct changes proposed here:

1. Moving the Python binary
2. Changing from "Scripts" to "bin"

So far, the primary resistance seems to be to item #1 - moving the
python binary. There have been a few people who have noted that #2 will
require some code to change (i.e. Paul), but I don't see lots of resistance.

Am I reading you correctly?


Well - as Paul implies, there are 2 distinct changes being proposed, but 
in 2 different environments.


For an installed Python:  If it has to move, it may as well move to 
somewhere consistent with other platforms.  IOW, moving to "bin" seems 
preferable to moving to Scripts.  My initial objection was to moving it 
at all in an installed Python.


For a virtual env, we are talking about moving it *from* Scripts to bin, 
which may cause some people different problems.  However, that isn't the 
concern I was expressing and I'd hate to see virtual envs remain 
inconsistent with an installed Python after this effort.


So I'm assuming that:
* The executable (and DLL) are moved to a "bin" directory in an 
installed Python.
* distutils etc will change to install all "scripts" (or executables 
generated from scripts) into that same directory.  IOW, "Scripts" would die.
* A virtual-env would have an almost identical layout to an installed 
Python.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

For those who missed it, in http://bugs.python.org/issue14302, Martin 
recently commented:


"""
After more discussion, it appears that this change is too incompatible 
to be done in a single release.


Therefore, I propose a long-term change into this direction, with the 
actual change not happening until 3.5.



"""

While I'm still unclear on the actual benefits of this, Martin's 
approach strikes a reasonable compromise so I withdraw my objections.


Thanks,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 17/03/2012 12:25 PM, Carl Meyer wrote:

Hi Mark,

On 03/16/2012 05:53 PM, Mark Hammond wrote:

* All executables and scripts go into the root of the Python install.
This directory is largely empty now - it is mainly a container for other
directories.  This would solve the problem of needing 2 directories on
the PATH and mean existing tools which locate the executable would work
fine.


I hate to seem like I'm piling on now after panning your last brainstorm
:-), but... this would be quite problematic for virtualenv users, many
of whom do rely on the fact that the virtualenv "stuff" is confined to
within a limited set of known subdirectories, and they can overlay a
virtualenv and their own project code with just a few virtualenv
directories vcs-ignored.

I would prefer either the status quo or the proposed cross-platform
harmonization.


Yeah, fair enough.  I should have indicated it was 1/2 tongue-in-cheek, 
but figured it worth throwing it out there anyway :)


OTOH, the part that wasn't tongue-in-cheek was the part that said "why 
not change the other platforms instead of windows" (then wait for the 
inevitable replies), then "so those same reasons apply to Windows too" - 
eg "fleets of torch wielding windows admins" :)


Breaking the few tools I'm concerned about vs asking Van etc to continue 
taking the pain he feels isn't going to mean the end of the world for 
any of us.  So given the stakes in this particular discussion aren't 
that high, I'll try and summarize the thread over the next few days (or 
someone can beat me to it if they prefer) and we can ask someone 
semi-impartial to make a decision.  I'd be happy to nominate MvL if he 
feels so inclined (even though I haven't asked him).


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 17/03/2012 12:07 PM, Brian Curtin wrote:

On Fri, Mar 16, 2012 at 19:53, Mark Hammond  wrote:

For the sake of brain-storming, how about this:

* All executables and scripts go into the root of the Python install. This
directory is largely empty now - it is mainly a container for other
directories.  This would solve the problem of needing 2 directories on the
PATH and mean existing tools which locate the executable would work fine.


How are existing tools locating the executable which would break with
a change to bin? As I posted on the tracker, the way which pops in my
mind would be to look for "C:\\Python%d%d" % (x, y) but that's already
pretty broken.


As I just replied in the tracker :)  They typically look up the 
InstallPath key in the registry and look for python.exe there - see the 
link to that activate.bat file I posted early in the thread.



The people I talked to at PyCon about this were Dino
from Microsoft and he nudged the guy next to him to ask the same
question (I seem to remember this guy worked for an IDE) -- both of
them just wanted to be sure they can still find python.exe's location
via the registry, which will be fine. I think we'll add a key to go
along with InstallPath - BinaryPath probably makes sense.


While I wouldn't object to that, it would seem redundant - if the whole 
point of this is to standardize the locations, then looking for 
"bin/python.exe" relative to the existing InstallPath key should also be 
reliable and hopefully permanent.


At the risk of repeating myself too many times, my concern is with 3rd 
party tools who (a) will break with the new scheme and need to be 
updated and (b) even after updating will still need the burden of 
supporting both the old and the new schemes.  I simply don't see the 
benefit that makes this worthwhile.



* If cross-platform consistency was desired, then we could consider making
other platforms match this.  However, if there are issues which might
prevent this happening for other platforms (eg, the risk of breaking other
3rd party tools, conventions on the platform ,etc) then it might be worth
conceding these considerations apply equally to the Windows installs and we
just live with this platform difference.


I don't think we're going to defeat the Unix army with their fleets of
distro packagers and torch wielding purists. If anyone's going to
move, my money's on Windows.


Right - but why?  Who wins?  Where is the evidence of the pain this has 
caused people over the last 18 years or so since Windows has been doing 
this?


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 14/03/2012 6:43 AM, VanL wrote:

Following up on conversations at PyCon, I want to bring up one of my
personal hobby horses for change in 3.3: Fix install layout on Windows,
with a side order of making the PATH work better.


...

For the sake of brain-storming, how about this:

* All executables and scripts go into the root of the Python install. 
This directory is largely empty now - it is mainly a container for other 
directories.  This would solve the problem of needing 2 directories on 
the PATH and mean existing tools which locate the executable would work 
fine.


* If cross-platform consistency was desired, then we could consider 
making other platforms match this.  However, if there are issues which 
might prevent this happening for other platforms (eg, the risk of 
breaking other 3rd party tools, conventions on the platform ,etc) then 
it might be worth conceding these considerations apply equally to the 
Windows installs and we just live with this platform difference.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 17/03/2012 7:22 AM, Carl Meyer wrote:
...


I don't want to make the internal layout of a virtualenv differ from the
system Python layout on the same platform, which (IIUC) was Mark's proposal.


Just to be clear, I made that suggestion in an effort to keep both 
myself and Van - that the Python executable would remain in the same 
place for installed Pythons in the interests of b/w compat, but change 
it in the virtual env in an effort to keep Van happy when working in 
such environments.  I now fully concede that was a dumb idea :)


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 16/03/2012 10:48 AM, Carl Meyer wrote:
...


The implementation of virtualenv (and especially PEP 405 pyvenv) are
largely based around making sure that the internal layout of a
virtualenv is identical to the layout of an installed Python on that
same platform, to avoid any need to special-case virtualenvs in
distutils. The one exception to this is the location of the python
binary itself in Windows virtualenvs; we do place it inside Scripts\ so
that the virtualenv can be "activated" by adding only a single path to
the shell PATH. But I would be opposed to any additional special-casing
of the internal layout of a virtualenv

...

Unless I misunderstand, that sounds like it should keep everyone happy; 
there already is a special case for the executable on Windows being in a 
different place in an installed layout vs a virtual-env layout. 
Changing this to use "bin" instead of "Scripts" makes the virtualenv 
more consistent across platforms and doesn't impose any additional 
special-casing for Windows (just slightly changes the existing 
special-case :)


Thanks,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 16/03/2012 8:57 AM, VanL wrote:

On 3/14/2012 6:30 PM, Mark Hammond wrote:


So why not just standardize on that new layout for virtualenvs?


That sounds like the worst of all worlds - keep all the existing special
cases, and add one.


I'm not so sure.  My concern is that this *will* break external tools 
that attempt to locate the python executable from an installed 
directory.  However, I'm not sure this requirement exists for virtual 
environments - such tools probably will not attempt to locate the 
executable in a virtual env as there is no standard place for a virtual 
env to live.


So having a standard layout in the virtual envs still seems a win - we 
keep the inconsistency in the layout of the "installed" Python, but 
tools which work with virtualenvs still get a standardized layout.


[At least I think that is your proposal - can you confirm that the 
directory layouts in your proposal exactly match the directory layouts 
in virtual envs on all other platforms?  ie, that inconsistencies like 
the python{py_version_short} suffix will not remain?]


Just to be completely clear, my current concern is only with the 
location of the executable in an installed Python.



The fact is that most code doesn't know about this, only installers or
virtual environments. Most code just assumes that distutils does its
thing correctly and that binaries are installed... wherever the binaries
go.


Of course - but this raises 2 points:

* I'm referring to *external* tools that launch Python.  They obviously 
need to know where the binaries are to launch them.  Eg, the PEP397 
launcher; the (admittedly few) people who use the launcher would need to 
upgrade it to work under your scheme.  Ditto *all* other such tools that 
locate and launch Python.


* "most code" isn't a high enough bar.  If we only considered such 
anecdotes, most backwards compatibility concerns would be moot.



Again, I have experience with this, as I have edited my own install to
do this for a couple of years. The breakage is minimal and it makes
things much more consistent and easier to use for cross-platform
development.


All due respect, but I'm not sure that is a large enough sample to draw 
any conclusions from.  I've offered 2 concrete examples of things that 
*will* break and I haven't looked for others.


Also, I'm still yet to see what exactly becomes "easier" in your model? 
 As you mention, most Python code will not care; distutils and other 
parts of the stdlib will "do the right thing" - and indeed, already do 
for Windows.  So the proposal wants to change distutils and other parts 
of the stdlib even though "most code" won't notice.  But the code that 
will notice will be broken!


So I dispute it is "easier" for anyone; I agree it is more consistent, 
but given the *certainty* external tools will break, I refer to you the 
Zen of Python's thoughts on consistency ;)



Right now we are in front of the knee on major 3.x adoption - I would
like to have things be standardized going forth everywhere.


It is a shame this wasn't done as part of py3k in the first place.  But 
I assume you would be looking at Python 3.4 for this, right?  So if 
people start working with Python 3.3 now and finds this change in 3.4, 
we are still asking them to take the burden of supporting the multiple 
locations.  I guess I'd be less concerned if we managed to get it into 
3.3 and also recommended to people that they should ignore 3.2 and 
earlier when porting their tools/libraries to 3.x.


I think I've made all the points I can make in this discussion.  As I 
mentioned at the start, I'm not quite -1 on the idea, so I'm not going 
to push this barrow any further (although I'm obviously happy to clarify 
anything I've said...)


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

[resending - original reply went only to Van]

On 15/03/2012 10:15 AM, Lindberg, Van wrote:
> On 3/14/2012 5:39 PM, Mark Hammond wrote:
>> Can you offer any examples of 3rd party tools which could unify code
>> in this scheme, and particularly, where this scheme would cause them
>> to have less code, not more?
>
> How about virtualenv:
>
> """
> def path_locations(home_dir):
> """Return the path locations for the environment (where libraries 
are,

> where scripts go, etc)"""
> # XXX: We'd use distutils.sysconfig.get_python_inc/lib but its
> # prefix arg is broken: http://bugs.python.org/issue3386
> if sys.platform == 'win32':
>   [snip a bit about spaces in the path]
>   lib_dir = join(home_dir, 'Lib')
>   inc_dir = join(home_dir, 'Include')
>   bin_dir = join(home_dir, 'Scripts')
> if is_jython:
>   lib_dir = join(home_dir, 'Lib')
>   inc_dir = join(home_dir, 'Include')
>   bin_dir = join(home_dir, 'bin')
> elif is_pypy:
>   lib_dir = home_dir
>   inc_dir = join(home_dir, 'include')
>   bin_dir = join(home_dir, 'bin')
> elif sys.platform != 'win32':
>   lib_dir = join(home_dir, 'lib', py_version)
>   inc_dir = join(home_dir, 'include', py_version + abiflags)
>   bin_dir = join(home_dir, 'bin')
> return home_dir, lib_dir, inc_dir, bin_dir
> """

So what would that look like in your scheme?  I'd expect you wind up with:

if sys.platform == 'win32' and sys.versioninfo < (3,4):
  ... existing layout
else:
  ... new layout.

So it actually ends up as slightly *more* code.

>
>
>> I think you misunderstand the .bat file - there is no python
>> executable in the bin directory. The bat file is locating your
>> already installed Python and attempting to use it.
>
> My only point here is that it would still find the already-installed
> Python (I think).

I'm fairly sure it would not - it doesn't look in %PYTHONINSTALL%\bin.

>
>> My other questions still remain: who specifically will benefit from
>> this, and what would be the cost to those beneficiaries of sticking
>> with the existing scheme?
>
> I will benefit, for one. My use case is that I do cross-platform
> development and deployment, and I occasionally want to put an entire
> environment in source control. Currently the case changing and
> Scripts/bin distinction make this a distinct pain, such that I go in 
> and edit my Windows python installation in the way that I am

> describing right now.
>
>   From my actual experience with this layout, pip, virtualenv, and
> pypm are the only three major packages that hard-code this logic and
> would need to be changed slightly.

So why not just standardize on that new layout for virtualenvs?

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 15/03/2012 3:03 AM, Lindberg, Van wrote:

On 3/14/2012 1:32 AM, Mark Hammond wrote:

As per comments later in the thread, I'm -1 on including
"python{py_version_short}" in the lib directories for a number of
reasons; one further reason not outlined is that it would potentially
make running Python directly from a built tree difficult. For the same
reason, I'm also -1 on having that in the include dir.


A built tree would look the same as always - the directories would only
be moved (if at all) during installation. Thus, you will still be able
to run python directly from a built installation.


So long as the location of the "lib" dir doesn't change, that is 
probably true.  If lib was to change though, then PC/getpathp.c would 
need to change which is where my concern came from.



scripts = {base/userbase}/bin


We should note that this may cause pain for a number of projects - I've
seen quite a few projects that already assume "Scripts" on Windows - eg,
virtualenv and setuptools IIRC


If you look at these projects, though, they *special case* Windows to
account for the different layout. Removing this difference will allow
these projects to remove their special-casing code.


I don't think that is true.  One of the examples I offered was a .bat 
file - it wouldn't be possible to remove the .bat file with your 
proposal.  The other example was the Windows specific launcher.


Most things that need to locate the Python executable aren't actually 
Python code - once Python is running, locating the executable is as 
simple as sys.executable.  So by their very nature, tools needing to 
locate Python will tend to be platform specific in the first place.


Can you offer any examples of 3rd party tools which could unify code in 
this scheme, and particularly, where this scheme would cause them to 
have less code, not more?



- and also assume the executable is where
it currently lives - one example off the top of my head is the mozilla
"jetpack" project - see:

https://github.com/mozilla/addon-sdk/blob/master/bin/activate.bat#L117


This code actually reinforces my point:

First, this code would actually still work. The section ":FoundPython"
sets the PATH to "%VIRTUAL_ENV%\bin;%PYTHONINSTALL%;%PATH%" (L80), which
would still allow python.exe to be found and run.

Second, look at that line again. Mozilla actually has edited this code
so that the jetpack uses a cross-platform "bin" convention, just as I am
proposing.


I think you misunderstand the .bat file - there is no python executable 
in the bin directory.  The bat file is locating your already installed 
Python and attempting to use it.



Third, one element of this proposal is that there would be a key placed
in the registry that points directly to the location of the python
executable, making locating it trivial to locate programmatically on
Windows.


That sounds reasonable, but it still causes breakage, and still causes 
extra code for tools needing to support earlier versions.  Saying "hey, 
it's easy to fix" are just words to someone frustrated trying to get 
things working with a later version of Python.


Don't get me wrong - the scheme you propose is how it should have been 
done in the first place, no question.  My issue is the breakage this 
will cause versus the benefit.


My other questions still remain: who specifically will benefit from 
this, and what would be the cost to those beneficiaries of sticking with 
the existing scheme?  The only benefit I've seen suggested is 
aesthetics, and while that is laudable, I don't think it is enough to 
justify breakage.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[Mark Hammond]

On 14/03/2012 6:43 AM, VanL wrote:

Following up on conversations at PyCon, I want to bring up one of my
personal hobby horses for change in 3.3: Fix install layout on Windows,
with a side order of making the PATH work better.

Short version:

1) The layout for the python root directory for all platforms should be
as follows:

stdlib = {base/userbase}/lib/python{py_version_short}
platstdlib = {base/userbase}/lib/python{py_version_short}
purelib = {base/userbase}/lib/python{py_version_short}/site-packages
platlib = {base/userbase}/lib/python{py_version_short}/site-packages

> include = {base/userbase}/include/python{py_version_short}

As per comments later in the thread, I'm -1 on including 
"python{py_version_short}" in the lib directories for a number of 
reasons; one further reason not outlined is that it would potentially 
make running Python directly from a built tree difficult.  For the same 
reason, I'm also -1 on having that in the include dir.



scripts = {base/userbase}/bin


We should note that this may cause pain for a number of projects - I've 
seen quite a few projects that already assume "Scripts" on Windows - eg, 
virtualenv and setuptools IIRC - and also assume the executable is where 
it currently lives - one example off the top of my head is the mozilla 
"jetpack" project - see:


https://github.com/mozilla/addon-sdk/blob/master/bin/activate.bat#L117

This code (and any other code looking in "Scripts" on Windows) will fail 
and need to be updated with this change.  Further, assuming such 
projects want to target multiple Python versions, it will need to keep 
the old code and check the new location.


Another bit of code which would be impacted is the PEP397 launcher; it 
too would have to grow version specific logic to locate the executable.


So while I'm not (yet) -1 on the general idea, I'm close.  I guess I 
don't understand how the benefits this offers outweigh the costs to 3rd 
parties.  Given the work on making Python more virtualenv friendly, 
can't we just wear the costs of the existing scheme in the stdlib and 
avoid breaking the code already out there?  IOW, who exactly will 
benefit from this, and how does the cost of them supporting the existing 
scheme compare to the cost of the breakage to multiple 3rd parties?


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 397 - Python launcher for Windows

[Mark Hammond]

On 22/02/2012 2:50 AM, Vinay Sajip wrote:

Mark Hammond  gmail.com>  writes:


think there is something that could be added to those docs - the use of
PATHEXT and the fact that once the shebang line is in place, a
command-prompt could do just "hello.py" rather than needing "py hello.py".


Or even just "hello" should work.


Ooops - right.  IIRC, "hello.py" will work without anything special in 
PATHEXT and just "hello" would work with a modified PATHEXT.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 397 - Python launcher for Windows

[Mark Hammond]

On 21/02/2012 2:54 AM, Mark Lawrence wrote:

On 18/02/2012 05:24, Mark Hammond wrote:

...


* Write some user-oriented docs.


The section in the docs "Using Python on Windows" would need to be
updated, but would this have to happen for every current version of Python?


I'm not sure what docs you are referring to here?


The docs here
https://bitbucket.org/vinay.sajip/pylauncher/src/tip/Doc/launcher.rst
are in my view possibly overkill, what do the rest of you think?


Even though I had no input into those docs, I actually think they are 
fairly good and can't see what should be dropped.  It may make sense to 
split the docs so there is a separate "advanced" doc page.  Further, I 
think there is something that could be added to those docs - the use of 
PATHEXT and the fact that once the shebang line is in place, a 
command-prompt could do just "hello.py" rather than needing "py hello.py".



The ouput from py --help seems fine but nothing happens when pyw --help
is entered, is this by accident or design?


I guess "accident" - or more accurately, the lack of doing anything 
special.  It could be useful to have that display a message box with the 
usage - while that would break "pyw --help > out.txt", I doubt that 
really is useful for anyone.  Alternatively, instead of trying to 
display all the usage in "pyw --help", it could display a short message 
indicating what the program is for and refer to "py.exe --help" for more 
information.  Possibly a plain "pyw" (with no args) could do the same 
thing - nothing useful happens in that case either.



Thoughts or comments?

Mark


A cracking bit of kit :)


Thanks!  Vinay's implementation is great, I agree.

Thanks,

Mark


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 397 - Python launcher for Windows

[Mark Hammond]

On 18/02/2012 11:08 PM, mar...@v.loewis.de wrote:


Zitat von Mark Hammond :


I'm wondering what thoughts are on PEP 397, the Python launcher for
Windows. I've been using the implementation for a number of months now
and I find it incredibly useful.


I wonder what the rationale for the PEP (as opposed to the rationale
for the launcher) is - why do you need to have a PEP for it? As
written, it specifies some "guidelines" that some software package
of yours might adhere to. You don't need a PEP for that, just write
the software and adhere to the guidelines, possibly putting them into
the documentation.

A PEP needs to have controversial issues, or else there would not
have been a point in writing it in the first place. Also, it needs
to concern CPython, or the Python language, else it does not need to
be a *P*EP.


The launcher was slightly controversial when the pep was initially 
written 12 months ago.  If you believe the creation of the PEP was 
procedurally incorrect I'm happy to withdraw it - obviously I just want 
the launcher, with or without a PEP.  Alternatively, if you think the 
format of the PEP needs to change before it can be accepted, then I'm 
happy to do that too if you can be very specific about what you want 
changed.  If you mean something else entirely then please be very 
specific - I admit I'm not clear on the point of your message at all.




To be a proper PEP, you need to include these things:
- what is the action that you want to see taken?
- what is the Python version (or versions) that you
want to see the action taken for?
- what alternative actions have been proposed, and what
are (in your opinion, and the opinion of readers) pros
and cons of each action?

Assuming you are proposing some future action for CPython,
I'm opposed to the notion that the implementation of the
launcher is the specification. The specification needs to be
in the PEP. It may leave room, in which case the remaining
details need to be specified in the documentation.


I'm really not sure what you are trying to say here.  That the PEP 
should remove all references to an implementation specification, or that 
the PEP simply should be withdrawn?  As above, I don't care - I just 
want the launcher with the least amount of bureaucracy possible.



A critical question (IMO) is the question how the launcher
gets onto systems. Will people have to download and install
it themselves, or will it come as part of some Python
distribution?


This is addressed in the PEP: "The launcher will be distributed with all 
future versions of Python ..."



If it comes with the Python distribution,
how get multiple copies of the launcher coordinated?


This may not be specified as well as it could, but: "Future versions of 
the launcher should remain backwards compatible with older versions, so 
later versions of Python can install an updated version of the launcher 
without impacting how the previously installed version of the launcher 
is used."



Also: what's the name of the launcher? How can I actually use
it?


This too is there: "The console launcher will be named 'py.exe' and the 
Windows one named 'pyw.exe'" and there is discussion of the command-line 
args.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 397 - Python launcher for Windows

[Mark Hammond]

On 18/02/2012 4:37 PM, Brian Curtin wrote:

On Fri, Feb 17, 2012 at 23:24, Mark Hammond  wrote:

I'm wondering what thoughts are on PEP 397, the Python launcher for Windows.
  I've been using the implementation for a number of months now and I find it
incredibly useful.

To my mind, the specific steps would be:

* Arrange for it to be installed with the next release of 3.2 and all future
versions - I'm happy to try and help with that, but will probably need some
help from Martin.


I've been doing some installer work lately and would be willing to
help out if I can.


Great.


Thoughts or comments?


Will you be at PyCon, specifically at the language summit? I proposed
a side-track to discuss this PEP, and I say side-track since a great
majority of the group are not Windows users, so I don't think it's a
topic to bring before the entire group.


Unfortunately not, but if you can get a few people together to discuss 
this, I'm happy to wait and see what consensus they arrive at.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Status of PEP 397 - Python launcher for Windows

[Mark Hammond]

I'm wondering what thoughts are on PEP 397, the Python launcher for 
Windows.  I've been using the implementation for a number of months now 
and I find it incredibly useful.


To my mind, the specific steps would be:

* Have someone pronounce it as accepted (or suggest steps to be taken 
before such a pronouncement).  I can't recall the current process - does 
Guido have to pronounce personally or formally delegate to a czar?


* Move the source into the Python tree and update the build process.

* Arrange for it to be installed with the next release of 3.2 and all 
future versions - I'm happy to try and help with that, but will probably 
need some help from Martin.


* Write some user-oriented docs.

Thoughts or comments?

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dll name for embedding?

[Mark Hammond]

On 17/02/2012 7:44 PM, Egon Smiwa wrote:

Hi all,
I'm an app developer with a CPython dll in the folder of that app.
In general, are there strict requirements about the dll name
(a preference would be "python.dll" (easy to update (simple replace) ).
I successfully used "python.dll" and a few standard modules,
then I tried to use the sympy library and its import fails with an
AV exception, unless I rename the dll back to the original "python32.dll"
Is there an intrinsic filename requirement inside the CPython dll, modules,
or are name-restrictions to be presumed only in case of third-party libs?


Note that this is off-topic for python-dev, which is for the development 
of Python - python-list would be a better choice.  But the short story 
is that given Python extensions have a link-time dependency on the core 
Python DLL, it isn't possible to rename the DLL without breaking all 
extensions built against the original name - this is just how link-time 
dependencies work on Windows.


You may also find http://www.python.org/dev/peps/pep-0384 of interest, 
but this still includes the major version in the DLL name and also 
depends on the authors of the extensions you want to use opting in.


As mentioned above, please followup on python-list.

Cheers,

Mark.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: Anyone still using Python 2.5?

[Mark Hammond]

FWIW, the most recent version of pywin32 has the following download 
counts (rounded to the nearest thousand)


Version 32bit   64bit
-
3.2 -  75,000   9,000
3.1 -   4,000   1,000
2.7 - 126,000  16,000
2.6 -  46,000   6,000
2.5 -  21,000 n/a
2.4 -   3,000 n/a
2.3 -   1,000 n/a

So ISTM that 2.5 isn't hugely popular these days, but also isn't 
insignificant.  It probably means I could "safely" drop 2.3 and 2.4 
support though...


Mark

On 21/12/2011 6:16 PM, Chris Withers wrote:

What's the python-dev view on this?

 Original Message 
Subject: Anyone still using Python 2.5?
Date: Wed, 21 Dec 2011 07:15:46 +
From: Chris Withers 
To: Python List ,
"testing-in-pyt...@lists.idyll.org" ,
simplis...@googlegroups.com

Hi All,

What's the general consensus on supporting Python 2.5 nowadays?

Do people still have to use this in commercial environments or is
everyone on 2.6+ nowadays?

I'm finally getting some continuous integration set up for my packages
and it's highlighting some 2.5 compatibility issues. I'm wondering
whether to fix those (lots of ugly "from __future__ import
with_statement" everywhere) or just to drop Python 2.5 support.

What do people feel?

cheers,

Chris



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Emit a BytesWarning on bytes filenames on Windows

[Mark Hammond]

On 31/10/2011 8:39 AM, Victor Stinner wrote:

Le 29/10/2011 07:47, Mark Hammond a écrit :

When previously discussing this issue, I was under the impression that
the problem was unencodable bytes passed from the Python code to Windows
- but the reverse is true - only the data coming back from Windows isn't
encodable.


The undecodable filenames issue occurs mostly on os.listdir(bytes) and
os.getcwdb().

Unencodable filenames issue occurs on the rest of my function list.


As the data came externally, the only solution the programmer
has is to change to the unicode version of the api
- so we recommend the bytes version not be used by anyone,
anytime - which means it is conceptually deprecated already.


I proposed to raise a Unicode error on undecodable filenames, instead of
returning invalid filenames (with question marks), to force the
developer to move to the Unicode API. But as I explained in my previous
message, you have to wait for an user having the problem to be noticied
of the problem.

Terry J. Reedy is also concerned about backward compatibility (3.2 ->
3.3). Emiting a warning, disabled by default, is a softer solution :-)


Right - and just to be clear, we are all now agreeing that the 
UnicodeDecodeError isn't appropriate and a warning will be issued instead?





Therefore, as you imply, I think the solution to this issue is to start
the process of deprecating the bytes version of the api in py3k with a
view to removing it completely - possibly with a less aggressive
timeline than normal.


If there is a warning, I don't really care of removing the bytes API
before Python 4.


Agreed - I was trying to say that I think we should start the 
deprecation process of the bytes API, so a [Pending]DeprecationWarning 
would then be appropriate.  The actual timing of the removal isn't 
important.




PendingDeprecationgWarning can be used, or maybe a DeprecationWarning
mentioning that the code will stay for long time.


In Python 2.7, I think documenting the issue and a
recommendation to always use unicode is sufficient (ie, we can't
deprecate it and a new BytesWarning seems gratuitous.)


Sorry, I don't understand "gratuitous" here: do you mean that a new
warning would annoying, and that it is cheap and useful to add it to
Python 2.7.x?


I mean "Uncalled for; lacking good reason; unwarranted."  IOW, I don't 
think we need to take any action for 2.7, apart from possibly 
documentation changes.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Emit a BytesWarning on bytes filenames on Windows

[Mark Hammond]

On 29/10/2011 9:52 AM, Victor Stinner wrote:

Hi,

I am not more conviced that raising a UnicodeEncodeError on unencodable
characters is the right fix for the issue #13247. The problem with this
solution is that you have to wait until an user get a UnicodeEncodeError.

I have yet another proposition: emit a warning when a bytes filename is used.
So it doesn't affect the default behaviour, but you can use -Werror to test if
your program is fully Unicode compliant on Windows (without having to test
invalid filenames).

I don't know if a BytesWarning or a DeprecationWarning is more apropriate. It
depends if we plan to drop support of bytes filenames on Windows later (in
Python 3.5 or later).


When previously discussing this issue, I was under the impression that 
the problem was unencodable bytes passed from the Python code to Windows 
- but the reverse is true - only the data coming back from Windows isn't 
encodable.


This changes my opinion significantly :)  I don't think raising an error 
is the right choice - there are almost certainly use-cases where the 
current behaviour works OK and we would break them (eg, not all files in 
a directory are likely to be unencodable).  As the data came externally, 
the only solution the programmer has is to change to the unicode version 
of the api - so we recommend the bytes version not be used by anyone, 
anytime - which means it is conceptually deprecated already.


Therefore, as you imply, I think the solution to this issue is to start 
the process of deprecating the bytes version of the api in py3k with a 
view to removing it completely - possibly with a less aggressive 
timeline than normal.  In Python 2.7, I think documenting the issue and 
a recommendation to always use unicode is sufficient (ie, we can't 
deprecate it and a new BytesWarning seems gratuitous.)


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use our strict mbcs codec instead of the Windows ANSI API

[Mark Hammond]

+1 from me!

Mark

On 25/10/2011 9:57 AM, Victor Stinner wrote:

Hi,

I propose to raise Unicode errors if a filename cannot be decoded on Windows,
instead of creating a bogus filenames with questions marks. Because this change
is incompatible with Python 3.2, even if such filenames are unusable and I
consider the problem as a (Python?) bug, I would like your opinion on such
change before working on a patch.

--

Windows works internally on Unicode strings since Windows 95 (or something
like that), but provides also an "ANSI" API using the ANSI code page and byte
strings for backward compatibility. It was already proposed to drop completly
the bytes API in our nt (os) module, but it may break the Python backward
compatibility (and it is difficult to list Python programs using the bytes API
to access the file system).

The ANSI API uses MultiByteToWideChar (decode) and WideCharToMultiByte
(encode) functions in the default mode (flags=0): MultiByteToWideChar()
replaces undecodable bytes by '?' and WideCharToMultiByte() ignores
unencodable characters (!!!). This behaviour produces invalid filenames (see
for example the issue #13247) and *the user is unable to detect codec errors*.

In Python 3.2, I changed the MBCS codec to make it strict: it raises a
UnicodeEncodeError if a character cannot be encoded to the ANSI code page
(e.g. encode Ł to cp1252) and a UnicodeDecodeError if a character cannot be
decoded from the ANSI code page (e.g. b'\xff' from cp932).

I propose to reuse our MBCS codec in strict mode (error handler="strict"), to
notice directly encode/decode errors, with the Windows native (wide character)
API. It should simplify the source code: replace 2 versions of a function by 1
version + optional code to decode arguments and/or encode the result.

--

Read also the previous thread:

[Python-Dev] Byte filenames in the posix module on Windows
Wed Jun 8 00:23:20 CEST 2011
http://mail.python.org/pipermail/python-dev/2011-June/111831.html

--

FYI I patched again Python MBCS codec: it now handles correclty ignore and
replace mode (to encode and decode), but now also supports any error handler.

--

We might use the PEP 383 to store undecoable bytes as surrogates (U+DC80-
U+DCFF). But the situation is the opposite of the situtation on UNIX: on
Windows, the problem is more on encoding (text->bytes) than on decoding
(bytes->text). On UNIX, problems occur when the system is misconfigured (e.g.
wrong locale encoding). On Windows, problems occur when your application uses
the old (ANSI) API, whereas your filesystem is fully Unicode compliant and you
created Unicode filenames with a program using the new (Windows) API.

Only few programs are fully Unicode compliant. A lot of programs fail if a
filename cannot be encoded to the ANSI code page (just 2 examples: Mercurial
and Visual Studio).

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/skippy.hammond%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP397 no command line options to python?

[Mark Hammond]

On 24/10/2011 12:56 PM, Michael Urman wrote:

On Sun, Oct 23, 2011 at 17:15, Mark Hammond  wrote:

How about abusing the existing flags for this purpose - eg:

% py -3?
% py -2.7?


I would have expected that to launch an interactive python shell of
the appropriate version. Does it do something else today?


That is what it does today without the trailing '?' character.  My idea 
was to allow the trailing '?' to behave like the proposed --which.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP397 no command line options to python?

[Mark Hammond]

On 24/10/2011 11:46 AM, Nick Coghlan wrote:

On Mon, Oct 24, 2011 at 10:00 AM, Mark Hammond  wrote:

* The "magic" symbol is somewhat self-documenting - it implies a question.
  Using  --which adds another special case that people would need to
understand isn't passed to Python.  IOW, I like that there is only 1 special
option and that one special option can be expressed in the form of a
question.


This may be a difference in what we're used to. To me, the "-?" is
strongly associated with "-h" and "--help"


Fair enough - and I admit to thinking -? didn't work for Python - but it 
does!



, whereas "--which" maps directly to the *nix "which" command:


Sure, but this isn't for *nix, so I'm not sure it is safe to assume the 
users of the launcher will make that association.



So I don't actually see any particularly *new* design decisions to be
made in relation to a "--which" option - it's just a workaround for
the lack of a native 'which' equivalent on Windows,


Actually I don't think that is true - even with a 'which' on Windows you 
can't get this information from it.  Indeed, this functionality is quite 
distinct from that offered by which.


TBH I'm not that bothered - I just have a slight uneasiness to this new 
special option which really just helps describe what a different special 
option does.


So - in an effort to talk myself out of my idea... :)  one advantage 
--which would have is that it could work without any version qualifiers 
at all - eg:


% py --which path/to/script.py

could also tell you what version of Python would be used to execute the 
named script, taking into account the current defaults, environment 
variables and shebang line found in the script.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP397 no command line options to python?

[Mark Hammond]

On 24/10/2011 10:36 AM, Nick Coghlan wrote:

On Mon, Oct 24, 2011 at 8:15 AM, Mark Hammond  wrote:

How about abusing the existing flags for this purpose - eg:

% py -3?
% py -2.7?


What does using the magic symbol offer over an explicit separate flag?


* The "magic" symbol is somewhat self-documenting - it implies a 
question.  Using  --which adds another special case that people would 
need to understand isn't passed to Python.  IOW, I like that there is 
only 1 special option and that one special option can be expressed in 
the form of a question.


* Simplicity - does "py -2.3 --which" work the same as "py --which 
-2.3"?  If not, that's not at all intuitive.  If so, it adds complexity 
to the launcher and the PEP text.


* Extensibility - While I've resisted, I predict that due to popular 
demand, we will wind up supporting additional arguments which are passed 
directly to Python (eg, "py -2.3 -W scriptName").  If we did, how would 
we treat --which when it is specified with additional options?


So to turn the question back around - why introduce a new special option 
when the existing single special option can be leveraged?  Are we 
opening the door to further special options?


I guess the key downside to this suggestion is that it doesn't allow you 
ask where the default Python is without using "-2?" (or maybe just -?)


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP397 no command line options to python?

[Mark Hammond]

On 23/10/2011 12:27 AM, Paul Moore wrote:

(Sorry, should have gone to the list...)

On 22 October 2011 13:15, Vinay Sajip  wrote:

Nick Coghlan  gmail.com>  writes:


As a simpler alternative, I suggest the launcher just gain a "--which"
long option that displays the full path to the interpreter it found.

So:

C:\>  py -2 --which
C:\Python27\python.exe

C:\>  py -3 --which
C:\Python32\python.exe

No significant complexity in the launcher, and if you want to add
additional arguments like -m, -c, or -i you can do it by running
'--which' and switching to invoking that interpreter directly.


Perhaps even simpler would be for the -h option to print the interpreter paths
which would be returned for -2 and -3, on separate lines, even without the
--which, e.g.

Currently configured:
-2: c:\Python27\python.exe
-3: c:\Python32\python.exe


--which is nice for people who can use Unix-style $() or Powershell&
to directly execute the output as a command.

&  (py -3 --which)


How about abusing the existing flags for this purpose - eg:

% py -3?
% py -2.7?

etc.

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP397 no command line options to python?

[Mark Hammond]

On 18/10/2011 8:59 PM, Sam Partington wrote:
...


I added shebangs to
all files as appropriate for devel/stable branch, and initially I
changed the python build targets from "python -utt build.py" to
"./build.py" and I lost the -utt functionality which I could live
with.


Can't you just put the -utt options in the shebang line of build.py?


But on some of the windows machines the default action of python
files was to open an editor with the build.py. So I changed it to "py
-utt build.py". Everything seemed fine initially until tests started
to fail which ensued some head scratching.  I actually didn't figure
out what was going on until I noticed that SCiTe was also calling the
wrong python because it also had "py -utt" to run python scripts.
Incidentally, one of my colleagues also discovered the same issue in
his eclipse pydev setup. I also notice that Editra also does "python
-u" by default, and I can imagine lots of users swapping "python" with
"py".


Why would users choose to do that?  Using "python" presumably already 
works for them, so what benefit do they get?  If the main advantage is 
they can now use shebang lines, then the specific options the script 
wants can be expressed in that line.


I wonder if we just need to make it clear that py.exe is not designed to 
simply be a replacement for python.exe - a simple replacement adds no 
value.  It is designed to bring shebang processing to Python on Windows 
and the shebang line is where these args should live.  If you want finer 
control over things, just keep using python.exe.


Also, seeing it is much easier to add a feature later than to remove it, 
we should err on the side of not adding the feature until it is clear 
that many people want it and ensure we aren't creating other 
inconsistencies or issues when we do.  If it turns out to be true that 
even with clear documentation people come to the same conclusion as you 
(ie, that py.exe supports arguments the same way as python.exe) we still 
have the option of adding it.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP397 no command line options to python?

[Mark Hammond]

On 18/10/2011 3:24 AM, PJ Eby wrote:



On Mon, Oct 17, 2011 at 8:55 AM, Sam Partington
mailto:sam.parting...@gmail.com>> wrote:

Yes it is a bit annoying to have to treat those specially, but other
than -c/-m it does not need to understand pythons args, just check
that the arg is not an explicit version specifier.  -q/-Q etc have no
impact on how to treat the file.

In fact there's no real need to treat -c differently as it's extremely
unlikely that there is a file that might match. But for -m you can
come up with a situation where if you it gets it wrong. e.g. 'module'
and 'module.py' in the cwd.

I would suggest that it is also unlikely that there will be any future
options would need any special consideration.


What about -S (no site.py) and -E (no environment)?  These are needed
for secure setuid scripts on *nix; I don't know how often they'd be used
in practice on Windows.  (Basically, they let you isolate a script's
effective sys.path; there may be some use case overlap with virtual envs.


It is worth pointing out that options can be specified directly in the 
shebang line - eg, a line like "#! /usr/bin/python -S" in a foo.py works 
as expected.  What doesn't work is explicitly using a command like:


% py -E foo.py

Using the foo.py above as an example, that would need to result in 
spawning Python with both -E and -S options.  For my money, that doesn't 
seem worthwhile - eg, someone else may have the expectation that 
specifying the args to py.exe should override the args on the shebang 
line.  Then someone else will have different expectations about the 
specific order of the args that should be used (eg, compare using 
"python -m somemodule -v" versus "python -v -m somemodule".  etc.


For these reasons I'm still advocating we don't support such 
command-lines, but as usual I'll go with the consensus :)


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP397 no command line options to python?

[Mark Hammond]

On 17/10/2011 9:10 PM, Sam Partington wrote:

Hello all,

I was surprised to see that the excellent pylauncher doesn't do the
magic shebang processing if you give it any python command line
options.  e.g. Given

#!/usr/bin/env python2.6
import sys
print(sys.executable)

C:\>py test.py
C:\Python26\python.exe
C:\>py -utt test.py
C:\Python27\python.exe

It is spelled out that it shouldn't do so in the pep :

   "Only the first command-line argument will be checked for a shebang line
and only if that argument does not start with a '-'."

But I can't really see why that should be the case.  What is the
rational behind this?


It really is a combination of 2 things:

* The key use-case for the launcher is to be executed implicitly - ie, 
the user types just "foo.py".  In that scenario there is no opportunity 
for the user to specify any args between the name of the executable and 
of the script.  IOW, the expectation is that people will not type "py 
foo.py", but instead just type "foo.py".


* A desire to avoid command-line parsing in the launcher or to make some 
options "more equal" then others.  Eg, you mention later that -c and -m 
should be special, but what about -w or -Q?  What about new options in 
future versions?



It is very surprising to the user that adding a
simple option like -tt should change the way the launcher behaves.
The PEP also states that the launcher should show the python help if
'-h' is specified :

 "If the only command-line argument is "-h" or "--help", the launcher will
 print a small banner and command-line usage, then pass the argument to
 the default Python.  This will cause help for the launcher being printed
 followed by help for Python itself.  The output from the launcher will
 clearly indicate the extended help information is coming from the
 launcher and not Python."

To me that would suggest to end users that they can use any of the
command line options with the launcher, and they should behave as if
you had called python directly.


I think the language is fairly clear - only the help options are special 
and no other options will work.


...


Incidentally whilst implementing this I also noticed a bug in the
pylauncher whereby the py launcher would incorrectly treat "py t3" as
a request for python version as if '-3' had been specified.  I have a
small patch that fixes this and implements the above for pylauncher
with extra tests if there is interest.


That certainly sounds like a bug and a patch sent to 
https://bitbucket.org/vinay.sajip/pylauncher will be appreciated!



PS I have been lurking for a while, hello everyone.


Hi and cheers! :)

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.2.1 encoding surprise

[Mark Hammond]

On 22/07/2011 9:02 AM, Glenn Linderman wrote:

Bad logic is get_configured_value!  get_configured_value only looks in
the global configuration file if there is a local configuration file
that doesn't have the setting.  It should look in the global
configuration file if there is no local configuration file _OR_ the is a
local configuration file without the setting.

I'll await a new launcher.


I just pushed a fix and hopefully Vinay will push a new MSI soon.

Thanks,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New update to PEP397 - Python launcher for Windows

[Mark Hammond]

On 21/07/2011 10:13 PM, anatoly techtonik wrote:

If you're going to include this into standard Python distribution, it
needs more attention from _users_. As a user, I can not find any
references to any user stories in this PEP article. Abstract chapter
is totally useless


Could you please be a little more constructive and offer concrete 
improvements?


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New update to PEP397 - Python launcher for Windows

[Mark Hammond]

On 21/07/2011 6:32 PM, Glenn Linderman wrote:

On 7/20/2011 11:35 PM, Mark Hammond wrote:

 * If the command starts with the definition of a customized command
   followed by a space character, the customized command will be used.
   See below for a description of customized commands.



 Then a shebang line of '#! vpython' in a script named 'doit.py' will
 result in the launcher using the command-line 'c:\bin\vpython.exe -foo
 doit.py'


Shouldn't the second paragraph include a space before the 2nd ' ?  In
other words, the command as quoted does not show a "customized command
followed by a space character" as the definition requires.  I don't know
why the space character is required, or maybe "white space" was meant,
so that the line terminating newline character qualifies also to delimit
the customized command?


Yes, thanks for pointing that out - I've pushed a new version of the PEP 
with that paragraph reading as:


"""
If the command starts with the definition of a customized command 
followed by a whitespace character (including a newline), the customized 
command will be used.  See below for a description of customized commands.

"""


Thanks,

Mark




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/skippy.hammond%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] New update to PEP397 - Python launcher for Windows

[Mark Hammond]

I've updated PEP 397 - "Python launcher for Windows" based on recent 
discussions and Vinay's implementation.


http://www.python.org/dev/peps/pep-0397/ and a copy is attached for your 
convenience.


The main changes are:

* All mentions of the Python reference implementation have been removed 
now that a C implementation exists and is a more accurate implementation 
of the PEP than the Python version.


* Some "implementation details" have been removed and links added to the 
launcher docs.  This was done mainly so the implementation is free to 
make changes based on user feedback and still stay true to the PEP. 
Note that the launcher doc link doesn't exist right now, but should 
magically appear over the next couple of days as Vinay pushes some 
changes I just sent him.


* The recommendation to install into System32 has been changed to a 
recommendation to install directly into \Windows, as the System32 
directory is not on the default path for 32bit processes on a 64bit 
Windows.  Vinay even has a version of an MSI installer which installs 
into this directory.  The PEP also gives installers more leeway on where 
to install the launcher if this directory is not writable.


I think this PEP is getting close to being finalized - please suggest 
whatever changes you feel are necessary ASAP as soon I'll be asking for 
pronouncement.


Thanks - especially to Vinay for taking on the C implementation!

Mark
PEP: 397
Title: Python launcher for Windows
Version: $Revision$
Last-Modified: $Date$
Author: Mark Hammond 
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created: 15-Mar-2011
Post-History: 21-July-2011, 17-May-2011, 15-Mar-2011

Abstract

This PEP describes a Python launcher for the Windows platform.  A 
Python launcher is a single executable which uses a number of 
heuristics to locate a Python executable and launch it with a
specified command line.


Rationale

Windows provides "file associations" so an executable can be associated
with an extension, allowing for scripts to be executed directly in some
contexts (eg., double-clicking the file in Windows Explorer.)  Until now,
a strategy of "last installed Python wins" has been used and while not
ideal, has generally been workable due to the conservative changes in
Python 2.x releases.  As Python 3.x scripts are often syntactically
incompatible with Python 2.x scripts, a different strategy must be used
to allow files with a '.py' extension to use a different executable based
on the Python version the script targets.  This will be done by borrowing
the existing practices of another operating system - scripts will be able
to nominate the version of Python they need by way of a "shebang" line, as
described below.

Unix-like operating systems (referred to simply as "Unix" in this
PEP) allow scripts to be executed as if they were executable images
by examining the script for a "shebang" line which specifies the
actual executable to be used to run the script.  This is described in
detail in the evecve(2) man page [1] and while user documentation will
be created for this feature, for the purposes of this PEP that man
page describes a valid shebang line.

Additionally, these operating systems provide symbolic-links to
Python executables in well-known directories.  For example, many
systems will have a link /usr/bin/python which references a
particular version of Python installed under the operating-system.
These symbolic links allow Python to be executed without regard for
where Python it actually installed on the machine (eg., without
requiring the path where Python is actually installed to be
referenced in the shebang line or in the PATH.)  PEP 394 'The "python"
command on Unix-Like Systems' [2] describes additional conventions
for more fine-grained specification of a particular Python version.

These 2 facilities combined allow for a portable and somewhat 
predictable way of both starting Python interactively and for allowing
Python scripts to execute.  This PEP describes an implementation of a 
launcher which can offer the same benefits for Python on the Windows 
platform and therefore allows the launcher to be the executable
associated with '.py' files to support multiple Python versions
concurrently.

While this PEP offers the ability to use a shebang line which should
work on both Windows and Unix, this is not the primary motivation for
this PEP - the primary motivation is to allow a specific version to be
specified without inventing new syntax or conventions to describe it.

An overview of the launcher.

This PEP outlines the general functionality and key guidelines of a
launcher for Windows.  It is accompanied by an implementat

Re: [Python-Dev] 3.2.1 encoding surprise

[Mark Hammond]

On 21/07/2011 10:18 AM, Glenn Linderman wrote:

On 7/20/2011 5:07 PM, Nick Coghlan wrote:

On Thu, Jul 21, 2011 at 8:51 AM, Ethan Furman  wrote:

I would say that would be a cool enhancement, as it could save a bit of
typing, but I think the launcher is quite useful even without path
traversal.

Two related points:

1. Walking PATH isn't necessary, but the cwd of the py process should
be inherited from the shell correctly. If it is, then 'py foo.py'
shouldn't need path traversal, it should just look in the current
directory. Using PATHEXT to turn 'foo.py' directly into an executable
command on PATH from any directory is different and out of scope for
the launcher.


Sorry, I disagree that it is out of scope.  Looking in the current
directory is fine, when the script is there, but my scripts are seldom
in my data directories, and I want to run scripts (from the PATH) on
data that is in the CWD.  I consider this a _very common_ use case for
using scripts/programs, but then if you want to use py from the command
line to tweak which version of Python gets used to execute the script
(if the default one didn't work, for example, and you want to try a
different one), then suddenly, you have to find the path to the script,
and specify it explicitly.


The arguments above apply equally to python.exe.  The launcher's job is 
to find an appropriate python.exe and launch it, not to locate the 
scripts and all the command-line parsing that would entail.  If you want 
this feature you should advocate for it to be added to Python itself and 
it will then automatically work in the launcher too.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.2.1 encoding surprise

[Mark Hammond]

On 21/07/2011 10:13 AM, Glenn Linderman wrote:

On 7/20/2011 2:43 PM, Glenn Linderman wrote:

It's not py's job to walk the path: the shell does that when you just type
"foo". It locates foo.py, and then invokes py because of file association - py
then checks the file for a shebang to decide which Python to dispatch it to.


Certainly when the launcher is invoked via an association, this would
be the case.  However, when the launcher is invoked via the command
line, then the unqualified name is passed through.  To be useful from
the command line, the launcher should walk the PATH to find the .py file.


The Windows SearchPath API

makes it a pretty easy job to walk the PATH.  Would even allow low cost
additional feature of searching for both   foo and foo.py  at the same time.


But that would also require us to parse the command-line, understand 
which options require 2 args and which just require 1 (making it fragile 
in the face of later versions introducing new options), then 
re-stitching a modified command-line back together for the child 
process.  IMO that is all out of scope.


Are you maybe forgetting about the PATHEXT functionality?  If you 
include .py in your PATHEXT and file associations are set so .py files 
are handled by the launcher, you should be able to directly execute just 
'foo' and have the command processor search the PATH for a foo.py and 
invoke it via the launcher.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python launcher command line usage (Was: 3.2.1 encoding surprise)

[Mark Hammond]

On 21/07/2011 10:08 AM, Antoine Pitrou wrote:

On Thu, 21 Jul 2011 09:55:28 +1000
Mark Hammond  wrote:


  >  The two proposals

overlap but are not mutually exclusive. For future pythons, 'python33'
is easier to remember and type than 'py -v 3.3' or whatever the proposed
encantation is.


'py -3.3' - less chars to type than 'python33' and with no need to have
every Python directory on your PATH.  IMO it is also simple enough that
people will remember it fairly easily.


Given that Python 2.x has a "-3" option, isn't "py -3.3" kind of
confusing, at least to the eye?


A little, yeah, but IMO practicality beats purity here.  I'd probably 
feel different if I felt 'python -3' was regularly used and would 
continue to be so in the future.  Also, I think most people who would 
potentially use 'python -3' will be aware that running 'py' is a totally 
different command and will adjust accordingly (either by continuing to 
use 'python -3' or adjusting to running 'py -2 -3'.)


The PEP does make explicit mention of this...

Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.2.1 encoding surprise

[Mark Hammond]

On 21/07/2011 10:02 AM, Glenn Linderman wrote:

So this tells me that it didn't find a local py.ini (no surprise, I
don't have one) but doesn't tell me that it did find or read
c:\Windows\system32\py.ini much less whether I have the syntax correct
for my [defaults] section.  It doesn't tell me that it didn't find a #!
line (but there isn't one).


I'll have a go at enhancing the debug output for most of the above 
(although note that if it *did* find a shebang line extra output would 
have been generated.)



Perhaps the problem is that there isn't one?  If it finds no virtual
command, maybe it doesn't obey the [defaults] python=3 directive?  The
PEP says it should act like '!#python' (I think the PEP meant
'#!python', though).  There is no " " after '!#python' in the PEP,
so that disqualifies it from being a customized command, there is no
'/usr/bin/' nor '/usr/bin/env ' in front of the 'python' so that means
it is not a virtual command; and it is not a fully-qualified name, so it
doesn't mean that case either looks like the PEP needs a bit of
clarification here.


I'm not sure the PEP needs clarification - possibly just the 
implementation ;)  But let me know if you think otherwise.



I do have a python on my path, but it is 3.1.2, not 3.2.1 or 2.6.4, and
it runs 2.6.4 as the output shows.


FYI, what is on the path isn't relevant to the launcher.


And I would expect it to run 3.2.1
with the [defaults] python=3 directive, since that is newer than 3.1.2,
which is on my PATH.


It may be that your copy of the launcher is a little old - some changes 
were pushed just yesterday (but I'm not sure if Vinay made a new 
installer yet).  It has slightly better debug output (although generally 
not what you are asking for yet) and better "cross-bittedness" support.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python launcher command line usage (Was: 3.2.1 encoding surprise)

[Mark Hammond]

On 21/07/2011 4:38 AM, Terry Reedy wrote:


Many installers first make an organization directory and then an app
directory within that. This annoys me sometimes when they only have one
app to ever install, but is useful when there might really be multiple
directories, as in our case. (Ditto for start menu entries.) This is
what python should have done a decade ago.


I disagree.  If we followed that advice we would also be in "\Program 
Files".  I have no problem with multiple Python versions installed 
directly off the root, especially given most users probably have a very 
small number of such installations.  I think Python being a developer 
tool rather than a user app is a reasonable justification for that (and 
the justification used when the existing scheme was decided)


> The two proposals

overlap but are not mutually exclusive. For future pythons, 'python33'
is easier to remember and type than 'py -v 3.3' or whatever the proposed
encantation is.


'py -3.3' - less chars to type than 'python33' and with no need to have 
every Python directory on your PATH.  IMO it is also simple enough that 
people will remember it fairly easily.


Also, the launcher supports the ability to select either the 32 or 64bit 
implementation - so maybe 'python33.exe' isn't really good enough and 
should reflect the bittedness?



A python directory also gives a sensible (though optional) place to put
other interpreters and even python-based apps. The launcher does not.


What other interpreters?  IMO it doesn't make sense to have IronPython, 
jython etc be installed there.  Ditto for apps - especially given most 
apps tend to be tied to a subset of all possible Python versions.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.2.1 encoding surprise

[Mark Hammond]

On 21/07/2011 7:43 AM, Glenn Linderman wrote:
...


I still get the same behavior.  Is there any debugging output produced
by py.exe that would tell what py.ini it looks in, and what the content
is?  What diagnostic steps might I take to produce additional
information that would help you (or me, along the way) analyze the
issue?  I do not presently have a C compiler installed on this machine,
however, unless it came along for a ride with something else.


It doesn't do exactly what you ask, but setting an environment variable 
PYLAUNCH_DEBUG to any value will cause some debug output to be generated 
to stdout.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python launcher command line usage (Was: 3.2.1 encoding surprise)

[Mark Hammond]

On 20/07/2011 1:00 AM, Paul Moore wrote:

On 19 July 2011 02:41, Vinay Sajip  wrote:

The use of py from the command line is merely a convenience for developers (as
the PEP says) - it's better to rely on shebang lines together with settings in
the .ini to get the behaviour you want.


But it's a *huge* convenience for running multiple Python versions,
particularly as no existing Python versions install executables with
the version in the name (python3.exe, python3.2.exe, etc).And BAT
files aren't a suitable option (I'll rant about the issues with BAT
files if you want, but I recommend you don't ask :-))

Being able to say py -3, py -2.7, etc, rather than having to hack
PATH, create renamed copies of exes, etc, is arguably more of a
benefit to me than shebang support.


Ditto for me.


This may explain why I'd like to see a command-line means of invoking
custom commands. Something like py.exe looking at an initial argument,
and if it's of the form "-cmd" for a command in py.ini, then run that
command, passing remaining arguments just as for py -3. (Maybe --cmd
to match standard long option usage would be better?)

Presumably, if this idea is to go anywhere, it would need adding to
the PEP. Mark, do you think it would be useful?


I doubt I will find it useful - but I'm on record as saying I wont find 
the custom command support itself useful :)  But similarly with that 
support, evidence that enough people *will* find it useful is enough for 
me to support the concept.


My current thinking re the PEP is to make it much smaller - just 
describe the concepts and try to avoid as much implementation detail as 
possible - I see no reason the PEP needs to take a stance on issues like 
that - this feature really could be treated just like any other feature 
request in Python - a loose consensus and acceptable patch is all that 
is needed.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation

[Mark Hammond]

On 2/07/2011 5:16 PM, I wrote:


Given [the C implementation] is now ahead of the Python
reference impl, I wonder if we should just drop all wording about that
reference impl and just treat the C impl as canonical?


I'm looking to update the PEP based on this discussion - does anyone 
object to the above?  Or to put it another way, does anyone believe 
dropping the Python reference implementation is to the detriment of the PEP?


Thanks,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation

[Mark Hammond]

On 5/07/2011 11:23 AM, Greg Ewing wrote:

Vinay Sajip wrote:

the installation of a pre-3.3 version of Python after Python 3.3 is
installed with the launcher will, if the user selects "Register
Extensions",
hijack the laumcher's associations to that earlier Python. Then bye
bye launcher


I don't see how anything can be done about that. It
also doesn't seem like too serious a problem -- it's
no worse than what currently happens if you install an
older version over a newer one, i.e. associations revert
to the old version.

Making the launcher a separate install at least offers
a way of fixing that if it happens and isn't desired,
i.e. reinstall the launcher.


Or an MSI installer may be able to offer a "repair" feature without too 
much pain.


However, I'm a bit torn on the separate installer issue.  I really think 
it should be installed with later Python versions even if it is made 
available as a separate installer for the benefit of earlier ones as it 
is one less step for people to get confused about - eg, in a few years 
when 3.3+ is the most common Python being downloaded and used, books or 
people helping on python-list could start referring to the launcher 
under the assumption it is installed, rather than avoiding mention of it 
simply to avoid the whole spiel about "this will only work if you have 
installed an optional package."  IOW, it will be impossible to give 
unconditional advice on certain aspects of launching Python.


If the launcher is such that we can unconditionally recommend its use, 
IMO we should just install it with Python.  I'll go with the consensus 
though...


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation

[Mark Hammond]

On 4/07/2011 3:59 PM, Greg Ewing wrote:

Mark Hammond wrote:


On 2/07/2011 7:08 PM, Vinay Sajip wrote:



perhaps we could remember the last non-launcher association when we
install the launcher,


It might be better to look in the registry for other Python
installations and ask the user which one to restore if there
is more than one. Trying to restore the "last" one would be
prone to breakage if the user didn't uninstall versions in
precisely the reverse of the order of installation.


While that makes alot of sense, the fact we are already "broken" in 
exactly the same way means I hope we can treat the restoration of 
associations as a separate issue - a worthwhile one, but not a 
pre-requisite for this PEP being approved.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation

[Mark Hammond]

On 2/07/2011 7:08 PM, Vinay Sajip wrote:

Mark Hammond  gmail.com>  writes:


The PEP does say "if possible, should be installed somewhere likely to
already be on the system PATH (eg., the Windows System32) directory."
It is silent about what to do when that isn't possible, but I'd think it
OK if the launcher was installed directly in the Python directory - IOW,
I'd think it OK if the PEP said "should be installed next to the
PythonXX.dll being installed" - but an important point in the above
working is the "already be on the system PATH" - ie, I don't really want
it put in a newly created directory unless the installer also adds that
directory to the PATH - and what to do on uninstall then becomes an issue.


But "next to PythonXY.dll" implies multiple copies, which we want to avoid,
right?


I believe this will be most useful when people select "install for all 
users", which will force it into system32.  If people select "just for 
me", then there will be multiple copies on disk, but only one will be 
active (ie, we will not support different ones being registered for 
different users.


It isn't ideal, but I think it good enough, avoids some complexity and 
should meet the needs of the majority of users.



One problem with all of this is uninstallation and specifically if the
user is uninstalling the most recent Python installation while leaving
earlier ones.  I guess there are 2 general answers to this:

* The installer process could remember the previous association and
restore that on uninstall.


We'd need to do that in the case of the earlier Python not having come with a
launcher, i.e. when its version is<  3.3.


OTOH, we don't do that now AFAIK.


* We treat this as a "wont-fix" and document a work-around of asking the
user to reinstall the previous version to restore the file association.


This is not ideal from a user's perspective.


Sure, but neither is the current situation - I don't recall enough users 
complaining about that to make the effort worthwhile.


I'm not fundamentally opposed to doing something better here - I'm just 
trying to avoid this kind of stuff being a requirement for acceptance. 
If you are more familiar with the installer than I and feel it can be 
done simply, then I'm happy for you to go for it!



We probably need to be mindful of adding too much extra work for the
installer process as that may well end up blocking us on getting this
into the next appropriate release.  In particular, Martin's thoughts
here would be very useful.

This would force the user to reinstall that older one to re-establish
the associations correctly


It sounds onerous for users to have to reinstall an older Python.


But this only happens when they install a later version, then uninstall 
the later one and continue to use the old one.  I'd suggest that is (a) 
rare and (b) possibly already broken (ie, the old association may not be 
restored now).  If the old association currently is correctly restored, 
then I'd expect things to just magically work in this new model without 
any effort.



I'm not that
familiar with Windows Installer features, so I don't know if this is too fancy,
but perhaps we could remember the last non-launcher association when we install
the launcher, and either restore that when the launcher is uninstalled (if it's
still pointing to an installed Python) or remove the association altogether,
otherwise. If this logic is too fancy to include in the installer itself,
perhaps we can provide this logic in the launcher itself or via an ancillary
executable or DLL that the installer can just call into.


I'm still inclined to call YAGNI, but as above, I don't fundamentally 
oppose such bells-and-whistles.



Is there some mechanism like the SharedDLLs registry key for executables, which
could be used to reference count launcher installations so that it could be
uninstalled at the appropriate time? Could we perhaps used the SharedDLLs
feature itself?


Yeah - I wonder if we can leverage the "job" api here and refuse to
start if there are already 2 processes in the job?  OTOH, that is tricky
as it would also prevent someone using os.startfile with a .py file


If there's only ever one launcher installed (which we could ensure by
installing it in a Windows rather than a Python location)


We probably can't ensure this while the installer supports a non-UAC 
installer (ie, the "just for me" option.)  OTOH, I'm not sure the "just 
for me" option currently works without needing UAC elevation anyway. 
But assuming it does (or the intention is to fix things if it does not), 
then we can't guarantee a writable system32.


> then perhaps we could

perhaps check for the value of a customised command pointing at the one-and-
only launcher, but this 

Re: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation

[Mark Hammond]

On 1/07/2011 7:20 PM, Vinay Sajip wrote:

Mark Hammond  gmail.com>  writes:


The intention is that there only be a single launcher, as only one app
can be associated with .py files.  OTOH though, file associations can be
configured per-user IIRC, and assuming that is the case, we could avoid
my multiple-ini-file usecase above by just allowing a different launcher
to be registered for a specific user.  This is sounding difficult from
the UI perspective though (ie, does the installer then need to ask that
question, will there be a post-install technique for per-user
registration, etc?)


I don't like this, for the reasons you state. I think it would be better if the
PEP was changed to say that there is intended to be just one launcher
installation per machine. As the intention is for the launcher to ship with
Python, and there can be multiple Pythons installed, I presume we'll have to
handle this by installing the launcher in some common-to-all-Pythons location
somewhere outside a Python installation location, such as "c:\Program
Files\Python Launcher". This should be stated in the PEP, and we'll also need to
indicate how the launcher will be cleaned up if for some strange reason someone
uninstalls all Pythons from a machine. Then we can just state that there's a
global .ini file (where the launcher is installed) and a local one (in the
user's APPDATA). From that perspective, it makes sense to have items in the
local (APPDATA) override the global (launcher installation location).


The PEP does say "if possible, should be installed somewhere likely to 
already be on the system PATH (eg., the Windows System32) directory." 
It is silent about what to do when that isn't possible, but I'd think it 
OK if the launcher was installed directly in the Python directory - IOW, 
I'd think it OK if the PEP said "should be installed next to the 
PythonXX.dll being installed" - but an important point in the above 
working is the "already be on the system PATH" - ie, I don't really want 
it put in a newly created directory unless the installer also adds that 
directory to the PATH - and what to do on uninstall then becomes an issue.


One problem with all of this is uninstallation and specifically if the 
user is uninstalling the most recent Python installation while leaving 
earlier ones.  I guess there are 2 general answers to this:


* The installer process could remember the previous association and 
restore that on uninstall.


* We treat this as a "wont-fix" and document a work-around of asking the 
user to reinstall the previous version to restore the file association.


We probably need to be mindful of adding too much extra work for the 
installer process as that may well end up blocking us on getting this 
into the next appropriate release.  In particular, Martin's thoughts 
here would be very useful.



This would force the user to reinstall that older one to re-establish 
the associations correctly



BTW I thought of another thing that perhaps needs handling: what if a customized
command points to the launcher itself? It'd be turtles all the way down :-)


Yeah - I wonder if we can leverage the "job" api here and refuse to 
start if there are already 2 processes in the job?  OTOH, that is tricky 
as it would also prevent someone using os.startfile with a .py file


From your second mail:


I've assumed that if a customised command is provided with arguments in the
shebang line, these will be ignored - if people want to run with different
options they can always define more customised commands. If you agree with this,
the PEP should probably explicitly state this.


I'm not too bothered to be honest - the customized commands exist purely 
for alternative implementations, so my initial thoughts are that 
additional args would be as useful for them as they are for cpython 
invocations.  IOW, if they don't need it, then CPython invocations don't 
need it either, so maybe it can be dropped completely?



In a couple of cases I've implemented using fixed size arrays - for the lists of
installed Pythons and customised commands. Of course these can be made dynamic,
but what's there is good enough for the moment for exploration.


Sure - I think there is some policy that a Python version number will 
never be > 10, so that sounds fine to me.  So long as the launcher 
doesn't blindly run off the end of such arrays I think it is fine - 
limitations can be addressed in later versions.


It will be a few days until I can look at the implementation, but I'm 
very happy to see it started.  Given it is now ahead of the Python 
reference impl, I wonder if we should just drop all wording about that 
reference impl and just treat the C impl as canonical?


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation

[Mark Hammond]

On 30/06/2011 10:09 PM, Vinay Sajip wrote:

There's a lot to like in the PEP, and I have some questions relating to the
latest version:

1. In the section on shebang line parsing, it says "If the command starts with
the definition of a customized command followed by a space character, the
customized command will be used." Sorry if I'm being dense, but what's the
significance of the trailing space character? In fact, your vpython example in
the customeised comments section doesn't show a trailing space - you've used
'#! vpython' as the shebang line.


This is just clumsy wording on my part - what I'm trying to say is 
simply that 'vpython3' will not match a customized command 'vpython'.



2. The section on .ini files says that "Commands specified in the [.ini file
in the] "application directory" [APPDATA] will have precedence over the one
next to the [launcher] executable." What's the benefit of this?


The idea is that a user without admin permissions can customize the 
behaviour - so the admin can add a .ini file, but the user can override 
any commands in that file with their own definitions.



If you have
only one launcher executable of one type (say console, 32-bit) on a system,
then there's no point in having two locations for .ini files.


The idea is that some users will not have permission to edit the one 
next to the launcher.  I'd be open to considering this yagni if others 
agree.


> If you have

multiple copies of the launcher and multiple .ini files next to them, then
with this precedence order, you can't specialise the behaviour of anything in
a specific launcher .ini that's also specified in the APPDATA .ini.


The intention is that there only be a single launcher, as only one app 
can be associated with .py files.  OTOH though, file associations can be 
configured per-user IIRC, and assuming that is the case, we could avoid 
my multiple-ini-file usecase above by just allowing a different launcher 
to be registered for a specific user.  This is sounding difficult from 
the UI perspective though (ie, does the installer then need to ask that 
question, will there be a post-install technique for per-user 
registration, etc?)



Doesn't it
make more sense to look for a setting in any file next to the launcher, and if
not found there, look in the APPDATA? That way common things can be defined in
the APPDATA .ini and overridden for special cases in the launcher-adjacent
.ini. Or have I got completely the wrong end of the stick?


I came at this from the other angle under the assumption there will only 
be one launcher installed - so common things can be stored in the 
launcher-adjacent file and per-user special cases can be in the per-user 
APPDATA dir.


If the expectation was multiple launchers be installed, then I'd 
probably then prefer to KISS and only have the launcher-adjacent file 
supported at all.




By the way, I've already converted the existing Python reference implementation
to C (I did it before I saw your response to my post). It basically works in
that it does what the Python version does, but doesn't include any handling of
-32 suffixes or .ini files. I can post this on BitBucket if anyone's interested,
but it's a work in progress. I'm working on some simple tests.


Sure, that would be awesome!  I think that will mean your impl is fairly 
close to the first draft of the PEP I checked into HG, which is nice and 
still quite useful to use :)


Thanks!

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation

[Mark Hammond]

On 30/06/2011 3:00 AM, Vinay Sajip wrote:

PEP 397 (Python launcher for Windows) has a reference implementation in Python.
Does anyone know of a C implementation, or is planning/working on one? I realise
this is the final objective, so such implementation might be premature, but
perhaps someone has been experimenting ...


Not yet - my last update of the PEP has made the existing reference 
implementation out-of-date, so I want to work on that before starting on 
the C version.  However, seeing as my note about the most recent PEP 
update attracted zero comments, I admit I lost any sense of urgency on 
actually doing this...  I'll make an effort to update that reference 
impl in the next week or so (but obviously anyone else is free to help ;)


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Updated version of PEP-0397 - Python launcher for Windows.

[Mark Hammond]

Hi all,
  I've updated PEP-0397 to try and address some of the comments from 
the last draft.  I've checked the new version into hg, so you can find a 
full diff there, but the key items I've changed are:


* Spelled out the "version qualifier" rules for the shebang lines.
* Spelled out some customization options, both for fine-tuning the 
specific Python version selected and for supporting other Python 
implementations via "custom" commands.

* Indicated the launcher is not supported at all on Win2k or earlier.
* Removed some cruft.

The new version is attached and I welcome all comments, including 
bike-shedding on the environment variable names and INI section/value names.


Note that the reference implementation has not changed - I'll update 
that once there is general agreement on the functionality described in 
the PEP.


Thanks,

Mark
PEP: 397
Title: Python launcher for Windows
Version: $Revision$
Last-Modified: $Date$
Author: Mark Hammond 
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created: 15-Mar-2011
Post-History: 17-May-2011, 15-Mar-2011

Abstract

This PEP describes a Python launcher for the Windows platform.  A 
Python launcher is a single executable which uses a number of 
heuristics to locate a Python executable and launch it with a
specified command line.


Rationale

Windows provides "file associations" so an executable can be associated
with an extension, allowing for scripts to be executed directly in some
contexts (eg., double-clicking the file in Windows Explorer.)  Until now,
a strategy of "last installed Python wins" has been used and while not
ideal, has generally been workable due to the conservative changes in
Python 2.x releases.  As Python 3.x scripts are often syntactically
incompatible with Python 2.x scripts, a different strategy must be used
to allow files with a '.py' extension to use a different executable based
on the Python version the script targets.  This will be done by borrowing
the existing practices of another operating system - scripts will be able
to nominate the version of Python they need by way of a "shebang" line, as
described below.

Unix-like operating systems (referred to simply as "Unix" in this
PEP) allow scripts to be executed as if they were executable images
by examining the script for a "shebang" line which specifies the
actual executable to be used to run the script.  This is described in
detail in the evecve(2) man page [1] and while user documentation will
be created for this feature, for the purposes of this PEP that man
page describes a valid shebang line.

Additionally, these operating systems provide symbolic-links to
Python executables in well-known directories. For example, many
systems will have a link /usr/bin/python which references a
particular version of Python installed under the operating-system.
These symbolic links allow Python to be executed without regard for
where Python it actually installed on the machine (eg., without
requiring the path where Python is actually installed to be
referenced in the shebang line or in the PATH.)  PEP 394 'The "python"
command on Unix-Like Systems' [2] describes additional conventions
for more fine-grained specification of a particular Python version.

These 2 facilities combined allow for a portable and somewhat 
predictable way of both starting Python interactively and for allowing
Python scripts to execute.  This PEP describes an implementation of a 
launcher which can offer the same benefits for Python on the Windows 
platform and therefore allows the launcher to be the executable
associated with '.py' files to support multiple Python versions
concurrently.

While this PEP offers the ability to use a shebang line which should
work on both Windows and Unix, this is not the primary motivation for
this PEP - the primary motivation is to allow a specific version to be
specified without inventing new syntax or conventions to describe it.

An overview of the launcher.

This PEP outlines the general functionality and key guidelines of a
launcher for Windows.  It is accompanied by a reference implementation
[3], written in Python, which defines the detailed implementation.  Over
time, changes to the implementation may be desired - if the changes 
adhere to the guidelines in this PEP and have been made following 
the standard Python development model this PEP need not change.  
In other words, this PEP makes no attempt to describe in detail every 
feature offered by the launcher but instead to offer guidelines the 
launcher should adhere to.

The actual implementation of the launcher will be written in C and
will not link directly or i

Re: [Python-Dev] sys.settrace: behavior doesn't match docs

[Mark Hammond]

On 2/05/2011 9:27 PM, Ned Batchelder wrote:
...

Maybe the fact no one noticed the docs
were wrong proves that no one ever tried returning None from a local
trace function.


Or if they did, they should have complained by now.  IMO, if the 
behaviour regresses from how it is documented and how it previously 
worked and no reports of the regression exist, we should just fix it 
without regard to people relying on the "new" functionality...


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Non-code changes on "old" branches

[Mark Hammond]

Hi,
  There are a couple of changes I'd like to make and would like some 
guidance on policy:


http://bugs.python.org/issue6498 is a documentation bug which exists in 
Python 2.6 and later.  The patch in that bug touches the docs and a 
comment in one source file.  Is it acceptable to push that change to the 
2.6 branch, or should I start with 2.7?


My request re .hgignore from yesterday didn't get any complaints, so I 
intend opening a bug, asking for review here and if I don't get 
objections in a day or so, pushing the change.  This really should go 
all the way back to 2.5 even though that release has long been closed. 
Is it acceptable to push a change to .hgignore to the 2.5 branch?  If 
not, where should I start with such a change?


I ask these questions primarily as the dev guide tells me I should 
forward-port all changes - thus, I need to know the earliest versions I 
can use before I can even start the process...


Thanks,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] .hgignore including site-packages and scripts directories?

[Mark Hammond]

On 30/03/2011 1:37 PM, R. David Murray wrote:

On Wed, 30 Mar 2011 12:17:05 +1100, Mark Hammond  
wrote:

On 30/03/2011 12:09 PM, R. David Murray wrote:

The solution is to add such
directories and/or files to your personal ignore list See the 'ignore'
entry under 'ui' in the hgrc documentation.


Yeah - but I was wondering if it could be made more convenient by
default given the downside seems quite small...


I suppose I wouldn't care about site-packages.  Nothing except the
existing README should ever get checked in there, I think.  And I don't
seem to have a 'Scripts' directory, just Tools/scripts, which shouldn't
be ignored.  Is Scripts windows specific?  (I also have a build/scripts,
but build is ignored.)


Yeah, "Scripts" is indeed Windows specific - which I admit I had 
forgotten until a couple of hours ago when debugging why a script using 
virtualenv failed on Windows due to assuming stuff went into a 'bin' 
directory and not the 'Scripts' directory.  The directory is normally 
populated by the distutils 'install' command, easy_install, etc


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] .hgignore including site-packages and scripts directories?

[Mark Hammond]

On 30/03/2011 12:09 PM, R. David Murray wrote:

On Wed, 30 Mar 2011 11:11:45 +1100, Mark Hammond  
wrote:

I'm wondering if it is a reasonable idea to have .hgignore exclude all
files from 'Lib/site-packages' and 'Scripts'?  As I install packages
into my source builds, a 'hg status' lists *many* files in both those
directories forcing me to scroll up a number of pages to see files which
have actually changed.


I hardly ever install things into my source build.  The first time I've
done that, in fact, was to run coverage.


Windows doesn't really have an install process integrated into the 
build, so it is probably fairly common there.



The solution is to add such
directories and/or files to your personal ignore list See the 'ignore'
entry under 'ui' in the hgrc documentation.


Yeah - but I was wondering if it could be made more convenient by 
default given the downside seems quite small...


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] .hgignore including site-packages and scripts directories?

[Mark Hammond]

I'm wondering if it is a reasonable idea to have .hgignore exclude all 
files from 'Lib/site-packages' and 'Scripts'?  As I install packages 
into my source builds, a 'hg status' lists *many* files in both those 
directories forcing me to scroll up a number of pages to see files which 
have actually changed.


IIUC, listing a directory in .hgignore doesn't preclude files from that 
directory being added to hg, and doesn't prevent files in those 
directories already under hg from being detected as changed.  The only 
downside I can see if that if new files are added to those directories 
which should be added to hg, a simple "hg st" will not show it - someone 
must remember and explicitly add it.  However, ISTM those files are 
already likely to be missed given the large amount of noise 'hg st' 
shows in that directory - the files are likely to be in the middle of a 
very long list which my brain will be trained to habitually skip over. 
The number of new files which legitimately need to be added to those 
directories seem so small that this risk seems worthwhile.


Any thoughts?

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Second draft: PEP397: Python launcher for Windows

[Mark Hammond]

On 24/03/2011 1:20 PM, Michael Foord wrote:

On 24/03/2011 00:44, Dj Gilcrease wrote:

On Wed, Mar 23, 2011 at 8:14 PM, Mark
Hammond wrote:

If you guys (or anyone) would like to agree on some precise rules for
both
the location of the config file and its contents and express this as
a patch
to the PEP text, I have no problem supporting it in the implementations.
I'd like to insist that the format of the config file was such that the
GetPrivateProfileString() Windows function could be used to extract
the data
(eg, only '=' can be used to separate the name/value pair,
case-insensitive
and no support for string interpolation) as I have no interest in
writing my
own config file parser in C :)

In the user directory much like TortoiseHG adds a murcurial.ini file
to the users directory for basic globa config the launch could look
for a python.ini there and use to to add known paths or version
overrides on a user by user basis.


A single global location (for shared installs) or a single per-user
location for per-user installs would seem to be sensible if the config
file route is chosen.


My concern with that would be that an administrator may install Python, 
but the user may not have write access to that global location, leaving 
that user unable to customize the launcher.  OTOH, that is how things 
work on Unix today - such users could not change what /usr/bin/python 
points to).  Always using a per-user location would mean there is no 
opportunity to have global settings, but it is unclear which is the 
lesser of 2 evils.  Supporting both might even be reasonable if the 
rules are very straightforward.


I'd still very much like to see the change to the wording of the PEP to 
describe this feature though, otherwise I fear different people will 
have different assumptions about exactly what it does and how it does it...


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Second draft: PEP397: Python launcher for Windows

[Mark Hammond]

On 24/03/2011 4:34 AM, Ethan Furman wrote:

Michael Foord wrote:

To be honest I would be happy with registry entries instead as the
alternative implementations can then add to their installers (or
provide a script) to add the right registry entry.


I'm partial to the config file method myself. I have this vague hope
where this tool could end up being a shebang-script-launching tool for
other scripting languages as well.


That is an explicit non-goal for the launcher - but if it happens to be 
able to work that way for you, that's great :)


If you guys (or anyone) would like to agree on some precise rules for 
both the location of the config file and its contents and express this 
as a patch to the PEP text, I have no problem supporting it in the 
implementations.  I'd like to insist that the format of the config file 
was such that the GetPrivateProfileString() Windows function could be 
used to extract the data (eg, only '=' can be used to separate the 
name/value pair, case-insensitive and no support for string 
interpolation) as I have no interest in writing my own config file 
parser in C :)


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Second draft: PEP397: Python launcher for Windows

[Mark Hammond]

On 23/03/2011 6:12 AM, Michael Foord wrote:

On 22/03/2011 07:21, Mark Hammond wrote:

Hi all,
I've made some changes to the draft PEP and checked it into the PEP
repository as PEP397. The reference implementation is currently being
tracked at http://bugs.python.org/issue11629.



Hey Mark,

One way of supporting alternative implementations (that may not even
have a standard install directory) is allowing configuration. e.g.
config.ini

[paths]
ironpython = c:\Program Files\IronPython 2.7\ipy.exe
ironpython2.7 = c:\Program Files\IronPython 2.7\ipy.exe
jython = c:\Users\foobar\jython2.5\jython.exe


Hi Michael,

I'd have no problem with that in general, but how would you feel about 
letting the PEP stand as it is without this additional requirement and 
then treat this as an additional feature to be thrashed out separately? 
 I intentionally worded the PEP to specifically allow these kinds of 
features to be added outside the PEP process.


For example, I guess the name of the INI file wouldn't be config.ini, 
and I guess there might need to be a strategy to allow it to exist in 
multiple places for when users want this feature but don't have write 
access to the location of py.exe.  Then people might want it to be in 
the cwd, or in any parent of the cwd, etc... Further, it might also be 
possible to support this with simple environment variables which might 
wind up being just as (or more) reasonable (the concept of per-user 
environment variables already exist and a UI already exists for editing 
them, and it would allow different cmd-prompts to have different "rules" 
with minimal complexity) - not that I am necessarily advocating this - 
I'd just prefer the PEP to not get bogged down with those kinds of issues.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Second draft: PEP397: Python launcher for Windows

[Mark Hammond]

Hi all,
  I've made some changes to the draft PEP and checked it into the PEP 
repository as PEP397.  The reference implementation is currently being 
tracked at http://bugs.python.org/issue11629.


Most of the changes should not be too controversial, but of note:

* I've added lots of discussion about the child-process issue and the 
wording now states, basically, that in-process is not feasible, but if 
we use the win32 Job API we can arrange for the child to be killed with 
the parent on Windows XP and later.  The reference implementation 
demonstrates this thanks to an example from Curt Hagenlocher.


* 64bit and 32bit considerations have been added, along with basic 
information about other distros and implementations.


* The somewhat controversial optional first command-line parameter which 
allows you to specify a particular version for interactive use has grown 
an almost-certain-to-be-controversial option to specify the bittedness 
of the Python you wish to start.  I've added this mainly as it would 
scratch my personal itch as I tend to have 32bit and 64bit 
implementations of most versions available and while this isn't 
particularly common, I'm going to attempt to defend it on the principle 
of "just don't use it if you don't like it" :)  Note the reference 
implementation does not yet implement this, nor does it implement the 
general policy about 32bit vs 64bit considerations.


Please let me know what you think and what objections/comments/etc remain.

Thanks,

Mark
PEP: 397
Title: Python launcher for Windows
Version: $Revision$
Last-Modified: $Date$
Author: Mark Hammond 
Status: Draft
Type: Standards Track or Informational ?
Content-Type: text/x-rst
Created: 15-Mar-2011


Abstract

This PEP describes a Python launcher for the Windows platform.  A 
Python launcher is a single executable which uses a number of 
heuristics to locate a Python executable and launch it with a
specified command line.


Rationale

Windows provides "file associations" so an executable can be associated
with an extension, allowing for scripts to be executed directly in some
contexts (eg., double-clicking the file in Windows Explorer.)  Until now,
a strategy of "last installed Python wins" has been used and while not
ideal, has generally been workable due to the conservative changes in
Python 2.x releases.  As Python 3.x scripts are often syntactically
incompatible with Python 2.x scripts, a different strategy must be used
to allow files with a '.py' extension to use a different executable based
on the Python version the script targets.  This will be done by borrowing
the existing practices of another operating system - scripts will be able
to nominate the version of Python they need by way of a "shebang" line, as
described below.

Unix-like operating systems (referred to simply as "Unix" in this
PEP) allow scripts to be executed as if they were executable images
by examining the script for a "shebang" line which specifies the
actual executable to be used to run the script.  This is described in
detail in the evecve(2) man page [1] and while user documentation will
be created for this feature, for the purposes of this PEP that man
page describes a valid shebang line.

Additionally, these operating systems provide symbolic-links to
Python executables in well-known directories. For example, many
systems will have a link /usr/bin/python which references a
particular version of Python installed under the operating-system.
These symbolic links allow Python to be executed without regard for
where Python it actually installed on the machine (eg., without
requiring the path where Python is actually installed to be
referenced in the shebang line or in the PATH.)  PEP 394 'The "python"
command on Unix-Like Systems' [2] describes additional conventions
for more fine-grained specification of a particular Python version.

These 2 facilities combined allow for a portable and somewhat 
predictable way of both starting Python interactively and for allowing
Python scripts to execute.  This PEP describes an implementation of a 
launcher which can offer the same benefits for Python on the Windows 
platform and therefore allows the launcher to be the executable
associated with '.py' files to support multiple Python versions
concurrently.

While this PEP offers the ability to use a shebang line which should
work on both Windows and Unix, this is not the primary motivation for
this PEP - the primary motivation is to allow a specific version to be
specified without inventing new syntax or conventions to describe it.

An overview of the launcher.

This PEP outlines the general functionality and key guidelines of a
launch

Re: [Python-Dev] Module version variable

[Mark Hammond]

On 22/03/2011 8:09 AM, Barry Warsaw wrote:

I started an Informational PEP on this at Pycon, and will try to finish a
draft of it this week.  (I'm claiming 396 for it.)


We nearly had a race for that number! FYI, I just checked in pep-0397 
leaving 396 for you :)


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 22/03/2011 12:04 AM, Paul Moore wrote:

I haven't had time to read the PEP yet, so my apologies if this is
made explicit there, but is the launcher expected to be solely for
implementing file associations? I thought there had been discussions
of using it to start the interactive interpreter, and for it having
command line arguments (implying direct command line usage). If it can
be used directly, there are many other scenarios that might be
impacted. Consider a service implemented using SRVANY which uses the
launcher. Stopping the service kills the launcher, leaving Python (and
the script, ie the service) running...


The intention is for the script to be used to interactively start a 
Python interpreter for the convenience of the developers working 
interactively.  It is not intended to replace all current uses of 
python.exe though, hence the different name.


Problems with things like SVRANY will be caused if those things use file 
associations to implicitly launch the executable associated with .py 
files - but those problems are caused by having py.exe associated with 
.py files, not with the fact that py.exe can also be used to launch 
python interactively.  IOW, SVRANY type use-cases should continue to use 
python.exe.  If problems are caused due to the change in associations, 
those problems would be caused even if py.exe did not have facilities 
designed for interactive use.


FWIW, I do accept that if an in-process model was used there may be less 
problems if use-cases like SVRANY happened to use py.exe, so should be 
supported if possible.  But while updating the PEP last night to 
reference 64bit and 32bit considerations, this idea hit another snag.  I 
changed the PEP to say:


On 64bit Windows with both 32bit and 64bit implementations of the
same (major.minor) Python version installed, the 64bit version will
always be preferred.  This will be true for both 32bit and 64bit
implementations of the launcher - a 32bit launcher will prefer to
execute a 64bit Python installation of the same version if
available.  This is so the behaviour of the launcher can be
predicted knowing only what versions are installed on the PC and
without regard to the order in which they were installed.

I think that is sensible as needing to know the order of past 
installations to predict the behaviour is undesirable.  However, that 
would also be impossible to achieve in an in-process model - a 32bit 
launcher could not load a 64bit Python into its process, and vice-versa. 
 Even if the wording of the above paragraph changes, if there remains a 
case where a 64bit launcher needs to load a 32bit Python or vice-versa, 
the same issue will arise.


I'm starting to think the only reasonable compromise will be to use 
child processes but use the win32 "job" API to arrange for Windows to 
kill the child should be the parent be killed - this will need more 
research and experimentation though - see 
http://msdn.microsoft.com/en-us/library/ms684161%28v=vs.85%29.aspx for 
an overview.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 21/03/2011 1:04 PM, "Martin v. Löwis" wrote:

Can you please add a summary of this discussion to
the PEP? (also, can you please check in the PEP, and

> give it a number?)

OK, I'll check it in once I get a PEP number allocated as per PEP1, 
updated to reflect some of the discussions in this thread.


Should I also check the reference implementation in?  Maybe next to the 
PEP text as "pep--reference.py"?


Thanks,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 21/03/2011 3:50 PM, Greg Ewing wrote:

Martin v. Löwis wrote:


Windows doesn't support exec.


Hmmm. In that case, if the launcher works by loading a pythonXY.dll,
I'd say that sys.executable should point to whatever version of
python.exe corresponds to that dll.

Generally, things should be made to look as much as possible as
if that python.exe had been executed in the first place.


Yeah, that would have to be the plan - if there are ways it *doesn't* 
look like python.exe had been executed it is likely to cause problems.


Initially I thought we could just adjust sys.executable after it is set 
to the "inappropriate" default value, but that would be tricky - this is 
called by Python during Py_Main() - which also executes the specified 
script.  Thus, the launcher would either:


* Call Py_Initialize(), adjust sys.executable, then call Py_Main().  I'm 
not sure what side-effects this would have though - Py_Main does a 
number of things before calling Py_Initialize() and having it called 
before those things are done might have other side-effects.


* Re-implement Py_Main and adjust sys.executable immediately after it is 
set.  This would be a huge burden given the size of that routine and the 
number of different versions we want to support.


The first option is the only one I'd really consider reasonable and I'll 
probably experiment with it once we have enough agreement that the C 
implementation can proceed - but I'm not confident enough we can make it 
work correctly to base the entire success or failure of the PEP on that 
detail.


Any other ideas/thoughts about this?

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 21/03/2011 12:27 PM, "Martin v. Löwis" wrote:

I remain -1 on any proposal that uses subprocesses. It absolutely must
be the launcher that is running Python. In fact, I'd call it
"python.exe".


For clarity, could you please tell us which scenarios you have in mind
that cause you to take that position, and why those scenarios couldn't
ignore the existence of the launcher and stick with doing exactly what
they do now (which presumably is to use python.exe directly)?


I'm primarily bothered about the failure to implement TerminateProcess
correctly. I don't actually know what use cases would be affected, other
then saying that anything launching py.exe could be affect, in
particular applications using ShellExecuteEx. I don't think it is
feasible to change them all to launch something different instead; some
may be out of our control. Potential candidates would be web servers.


This fear seems overblown to me as I doubt any such programs exist.  A 
program could not use ShellExecuteEx on an arbitrary file extension and 
hope to obtain a handle it can use to kill the process - the docs make 
it clear no handle is returned in a number of common cases and implies 
that even if you get a handle back you should not kill it (as it 
explicitly states that a single process may be the target for any number 
of such calls.)  For the same reason, you can't even wait on it unless 
you know the implementation of the associated program.


So this scenario would need to be a program which used ShellExecuteEx 
only to launch Python programs and therefore felt it could rely on the 
ability to (a) always get a handle back and (b) use TerminateProcess to 
kill it.  ISTM the likelihood of such a program existing is so low that 
it should not kill the chances of this PEP being accepted if the 
sub-process model is the only reasonable way to implement the launcher.


ie, let's say we are forced to choose between the following 3 options:

* No launcher at all (the status-quo), causing demonstrable breakage in 
Windows file associations whenever Python 2.x and Python 3.x scripts 
exist on the same box.


* An in-process launcher which caused breakage in a number of reasonably 
common scenarios for Python programmers, and such breakage could easily 
be demonstrated.


* An out-of-process launcher which caused breakage for the hypothetical 
program mentioned above, of which no instance can be found and no 
breakage actually demonstrated.


I personally would conclude that the last option is the least worst 
scenario by a wide margin.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 21/03/2011 11:15 AM, "Martin v. Löwis" wrote:

Am 21.03.2011 00:43, schrieb Greg Ewing:

Mark Hammond wrote:


The above raises an interesting question - if the launcher executed
Python in-process, what would sys.executable be?


I think it should be the actual Python executing at that
moment, not the launcher. This is the least change from
current behaviour and therefore least likely to break
existing code.


I remain -1 on any proposal that uses subprocesses. It absolutely must
be the launcher that is running Python. In fact, I'd call it
"python.exe".


For clarity, could you please tell us which scenarios you have in mind 
that cause you to take that position, and why those scenarios couldn't 
ignore the existence of the launcher and stick with doing exactly what 
they do now (which presumably is to use python.exe directly)?


Thanks,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 21/03/2011 11:10 AM, "Martin v. Löwis" wrote:

Am 21.03.2011 00:52, schrieb Mark Hammond:

On 21/03/2011 10:32 AM, "Martin v. Löwis" wrote:

The above raises an interesting question - if the launcher executed
Python in-process, what would sys.executable be?  I can imagine there
are few scenarios where it would be desirable to have it refer to the
launcher and a number of scenarios where it would be undesirable and
possibly break existing scripts.


Interesting question. What is it in COM and ISAPI applications?


ISAPI isn't a good example - that uses DLLs.


Actually, this is precisely the example i was looking at.


For COM objects hosted in
a .exe it will be pythonw.exe.  For Windows services it will be
pythonservice.exe (which is part of pywin32 and is located simply by
assuming it is next to win32service.pyd)

Regardless, I don't think they are very likely to break - my concern is
more for "normal" scripts which construct a child process cmdline/argv
using sys.executable...


If we find out how other cases of just loading the DLL deal with
sys.executable, it may help us in guiding how it should be set in the
launcher.


I guess it is all about context; in an ISAPI application or in a COM 
object implemented in a DLL, sys.executable will be the name of the host 
process (which is determined in PC/getpathp.c by calling 
GetModuleFileName with a NULL handle).  People writing ISAPI apps or COM 
objects are likely to understand the implications of them existing in 
some process which isn't python[w].exe so will ignore sys.executable and 
use some other technique to locate a python[w].exe if they need it.


"Normal" scripts are less likely to do so - a code search at google for 
sys.executable shows it used in ways which would possibly break if 
sys.executable referred to a launcher and the launcher didn't use the 
"current" version of Python as the default.  Often these are in tests or 
setup.py, but we would need to avoid any such breakage.


[Which isn't to suggest we can't avoid such breakage with an "in-process 
launcher" - it just isn't immediately clear how we would do so]


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 21/03/2011 10:32 AM, "Martin v. Löwis" wrote:

The above raises an interesting question - if the launcher executed
Python in-process, what would sys.executable be?  I can imagine there
are few scenarios where it would be desirable to have it refer to the
launcher and a number of scenarios where it would be undesirable and
possibly break existing scripts.


Interesting question. What is it in COM and ISAPI applications?


ISAPI isn't a good example - that uses DLLs.  For COM objects hosted in 
a .exe it will be pythonw.exe.  For Windows services it will be 
pythonservice.exe (which is part of pywin32 and is located simply by 
assuming it is next to win32service.pyd)


Regardless, I don't think they are very likely to break - my concern is 
more for "normal" scripts which construct a child process cmdline/argv 
using sys.executable...


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 20/03/2011 11:56 PM, Dj Gilcrease wrote:
...

Before Mark wrote up this pep I had started experimenting with how to
make the launcher and I was able to get it to launch python while
exiting py.exe and as far as I could tell it behaved just as if I had
launched the app directly by double clicking it.


In addition to Martin's point, this approach would mean the exit code of 
the child process probably isn't available to whoever started the 
launcher.  For example, a batch file that tried to use something like 
"if errorlevel ..." to check the result of the Python process would 
probably not work as expected.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

Following up myself here...
On 20/03/2011 9:25 PM, Mark Hammond wrote:

On 20/03/2011 8:37 PM, "Martin v. Löwis" wrote:
...
Some of these uses probably shouldn't use the launcher directly - eg,
ISAPI apps and COM objects which have a separate registration step could
register a specific python.exe - the installation script itself could by
launched by the launcher, but that script could continue to use
sys.executable to record the specific python.exe.


The above raises an interesting question - if the launcher executed 
Python in-process, what would sys.executable be?  I can imagine there 
are few scenarios where it would be desirable to have it refer to the 
launcher and a number of scenarios where it would be undesirable and 
possibly break existing scripts.


Consider when a script with a "../python3" shebang line is started via 
the launcher, but that script itself uses sys.executable to launch its 
own child.  If sys.executable was the launcher, that child would likely 
be started using a python 2.x interpreter...


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 20/03/2011 8:37 PM, "Martin v. Löwis" wrote:
...

... problems with creating child processes:
- applications using the debug API, PSAPI, etc. will be confused if
   the action all happens in a child process. I can accept that they
   have to adjust, though.


Some of these uses probably shouldn't use the launcher directly - eg, 
ISAPI apps and COM objects which have a separate registration step could 
register a specific python.exe - the installation script itself could by 
launched by the launcher, but that script could continue to use 
sys.executable to record the specific python.exe.


A side effect of this though is that if a newer version of python is 
installed such that "py.exe register.py" now used the later version, the 
existing registration would still use the older version until 
re-registered.  I think that is reasonable though.



- TerminateProcess must work correctly, and I see no way to achieve
   that (i.e. terminating the parent process won't terminate the child)


I understand this concern (and meant to say this in reply to Eugene's 
mail), but wonder if it isn't similar to the above - that if some 
program wants to launch python and note the id of the child process so 
it can terminate the child later, then python.exe could continue to be 
used.  OTOH, if you are terminating the process by using Process Manager 
or something else to find a 'python.exe', that will continue to work and 
the parent py.exe will automatically silently terminate.



That wouldn't bother me.


* There are subtle differences when Python is loaded by an executable in
the "install" directory versus by an executable that is not.
Specifically, I'm thinking about the differences in how the default
sys.path is populated.


This would need to be fixed. I.e. the launcher would have to arrange it
so it gives the illusion of having been started from the executable.


It isn't immediately obvious how that could be done though - although I 
admit I haven't looked into it.


But in general, I'd be happy for py.exe to avoid CreateProcess for 
CPython if that could be reasonably done and was the consensus...


Thanks,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 20/03/2011 8:36 PM, Terry Reedy wrote:

On 3/20/2011 3:22 AM, Glenn Linderman wrote:
On 3/19/2011 7:38 PM, Mark Hammond wrote:
[snip]

As both a writer and reader, I would like to just add, for instance,

#! python3

(or 3.3 or whatever) and have the launcher do the 'right thing'.


If there was general agreement, I'd be happy for the PEP to support that 
variation.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 20/03/2011 6:22 PM, Glenn Linderman wrote:

On 3/19/2011 7:38 PM, Mark Hammond wrote:

...

A Windows user who has only learned Python 2.x programming would not
necessarily have ever heard of execve, would not realize execve(2) means
it is from the 2nd chapter of the Unix man pages, meaning an API call,
would not know C programming, would not know what #include 
means, would not understand syntax like "const" or "[]". The description
is a bit friendlier, if they get that far, but they may still be quite
confused by wondering how to create and provide argv and envp in the
right forms, wondering what SIGCHLD, PID, set-user-ID bit, SIG_IGN,
SIG_DFL, SIGTRAP, set-group-ID, effective id, a.out, binary executable,
shared library stubs, ld.so(8), ELF executable, PT_INTERP, and
/lib/ld-unix.so.1 (and .2) are.


Sorry, I didn't mean to imply we would refer people to the execve man 
page, but simply that the man page defines the behaviour from the POV of 
the PEP.



Ah, but Mark! You are confusing (and maybe I do in some of my comments
too) the complexity of the launcher versus the complexity of describing
the launcher.


I believe it will be simple to describe it -as you mention, the execve 
man page needs only a couple of sentences - I believe with just a few 
more it can be documented with the same clarity and quality as the rest 
of the Python docs for Windows users.



The PEP could therefore be clearer by discussing the CPython
implementation strategy, even if it points out that other
implementations may need a different strategy.


I believe it states fairly clearly that the reference implementation 
defines much of this, and the reference implementation I referred to 
does indeed implement (and therefore define) it.  The PEP does make 
mention of other implementations.



So I think we are agreeing here, mostly. And you mentioned having the
CPython implementation PythonX.Y get on the PATH somehow. So just keep
the python launcher off the PATH, remove its then unnecessary option
syntax, and we agree totally :)


Not quite - my point is that *if* a CPython implementation is on the 
path (which many users arrange), then there is no advantage to using the 
launcher for that specific Python.  I personally don't have any CPython 
implementation on my path (actually, that isn't totally true - cygwin's 
Python is for my command-prompts), and therefore would find the 
behaviour described in the launcher incredibly useful.  This scratches 
my itch and unless the option does harm for people who choose not to use 
it, I see no problem with supporting it.



Does anyone see any benefit to using a launcher, if PythonX.Y is on the
PATH? I don't.


Fair enough.  But users who do not have a PythonX.Y on the PATH (even if 
they have another X.Y on it) will.



Per the beginning of this email, I think you need to stay far away from
referencing the execve man page in a description of a Windows launcher,
except perhaps as a footnote.


The PEP is not intended to be user documentation, but if it is to be 
used in that way I'd be happy to document it directly in the PEP.



Not using the launcher from the command line, and it not having any
available options, would sidestep this issue nicely.


I don't see what issue needs to be side-stepped.  IMO the feature can be 
clearly described (even if my initial PEP didn't do so) and easily 
avoided if not desired.



No, the workaround of using fully-qualified executable names in the
shebang line makes the script now dysfunctional on Unix.


No - as the PEP says, if it can be expected to work on Unix, it should 
be expected to work on Windows.  Otherwise, the script is inherently 
dysfunctional on Unix, so the fully-qualified path is perfectly reasonable.



I don't think you'll get "always", because of environments that use
different implementations on different platforms. To get always, you
need the flexibility to specify implementation and version for each
platform separately... two lines.


Sorry, but I just don't go for this at all.  Please don't continue to 
advocate for that in a PEP I am sponsoring - just propose your own.



Let's enumerate... and maybe if we keep talking we'll increase the
agreement list and decrease the disagreement list.

New ideas in this message:
1) If launcher doesn't find a #!/#@ line, it should run python2 to
support scripts without such lines.


Sorry if the PEP didn't make it clear, but that is how the reference 
implementation works.  You might like to try it.



Agreements:
1) Python needs PythonX.Y to be on the PATH


Nope.


2) Need a launcher to handle Windows associations that otherwise only
allow one version for one extension
3) Need a separate launcher for .py and .pyw


Yep.


4) Support for as many environments as possible, and as many
installations as possible for each environment.


Nope - "supports standard Python

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 20/03/2011 5:49 AM, "Martin v. Löwis" wrote:

* Is this really PEP material, or will turning the PEP into a regular
spec be suitable?


It's PEP material if it is contentious, which I believe it is.


Of course it is - this is python-dev .


I, for example, will find issues with it if the implementation uses
CreateProcess at some point - the launcher should IMO run Python
directly, rather than creating a new process. This, in turn, will
probably mean that you cannot actually implement the PEP in Python.


Out of curiosity, what is your objection to having the child process?  I 
guess it must be the system resources needed for the parent process 
while it waits for the child to terminate so the exit code can be 
reflected correctly "up stream"?


I would have 3 main problems with dynamically loading Python into the 
launcher process:


* It may be difficult or impossible to do for implementations other than 
CPython.


* There are subtle differences when Python is loaded by an executable in 
the "install" directory versus by an executable that is not. 
Specifically, I'm thinking about the differences in how the default 
sys.path is populated.


* The registry doesn't currently tell us exactly where pythonxx.dll can 
be located, but does tell is where python.exe is.  Thus things would get 
a little more complex as we go snooping for the location of the DLL.


Note that these are almost certainly surmountable, but I wonder if the 
benefits outweigh the costs.



* If it is a PEP, is it "Standards Track" or "Informational"?


If you are proposing to change the CPython code base (which I think you
should), it's standards track. If you are talking about a tool provided
by you on PyPI, you actually don't need to discuss it with anybody.


Thanks - "Standards Track" it is then :)  FWIW, I have no interest in 
making this a stand-alone project or distribution.  If there is 
agreement that this shouldn't be part of Python itself (which may well 
be a reasonable decision to make), I'll let the idea die.





* Does the functionality described strike the right balance between
simplicity and usefulness?


It leaves a number of issues still open, so it's hard to tell yet.
For example, the 32-bit vs. 64-bit issue should be resolved (which I
think comes down to having 4 binaries).


This is an interesting one and one which I don't have a firm opinion 
about - but the soft opinion I have is something along the lines of "the 
functionality should not be dictated by the bittedness of the launcher 
itself".  IOW, if we decide that a 64bit implementation of Python should 
be chosen over a 32bit implementation of the exact same version, this 
should be done by both the 32bit and 64bit launchers.


Of course, I welcome more thoughts on this and could be easily swayed.


Plus, it should talk about
installation of multiple copies of the same Python version, e.g.
from ActivePython, EPD, etc.


Good point - this is something I haven't considered at all.  Any 
thoughts on this from anyone?



Notice that the PEP doesn't talk about file associations yet (but it
should).


Actually, it does say '...the "console" version of the launcher should 
be associated with .py files and the "windows" version associated with 
.pyw files.'


What extra level of detail do you feel is necessary?


I'm fine with the strategy, but I feel that the devil's in the detail.


Indeed!  But general approval for the strategy is an encouraging first 
step :)


Thanks!

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

On 20/03/2011 4:15 AM, Dj Gilcrease wrote:

On Sat, Mar 19, 2011 at 4:44 AM, Glenn Linderman  wrote:



2) If the launcher provides command line options for the "benefit" of
launching interactive Python interpreters, those command line options can
have data puns with script names, or can conflict with actual Python
options.  I believe Python 2 already has a -3 option, for example.


FWIW, the point about Python 2 supporting a -3 arg is explicitly 
mentioned in the PEP.



Windows users are not trained that "-" introduces an option syntax, but
rather "/".  Most _programmer_ users would probably be aware of "-" as an
option syntax, but Python is used as a language for non-programmers in some
circles, and few Windows non-programmers understand "/" much less "-" and
not even command lines very well.  So not using a launcher for launching
interactive Python sidesteps all that: Python itself is introduced and
taught, and no need to teach about (or even have) launcher options that
could possibly conflict and confuse, in addition to Python options that may
conflict with script names already.  (I have seen lots of Windows users use
leading punctuation characters on filenames to affect sort order and
grouping of files in My Documents, not knowing they can create
subdirectories/subfolders, or not wanting to bother with them, since all
their applications default to opening things from My Documents.)


The command line options I disagree with as well. If the user wants to
test a script that has a shebang of python2 with python3 then they
should explicitly launch it that way just like you would have to do on
*nix


See my reply to Glenn on this - the optional -N.N argument must be the 
first arg, as must the file with the shebang - so if you specify -N.N, 
no shebang processing is done at all.


I expect that in general, the key advantage of shebang parsing will be 
when the script is launched via file association - eg, entering the 
command "foo.py" directly at the command-prompt or double-clicking from 
explorer.  Windows itself will lookup the association and execute 
"py.exe foo.py" - there is no opportunity for the user to arrange for 
arguments to be inserted in that command (only at the tail).


If a user explicitly specifies the command "py.exe -3 foo.py", I 
personally think it is fine the shebang line is ignored as the user has 
expressed a clear intention that a specific version of Python be used.



No I think there should be at max 2 environment variables and they
should be explicitly set by the user not added by default
PYTHON_2
 If set would override the latest version of python2 that is
launched via a shebang line ending in python or python2 but not
python2.x
PYTHON_3
 if set would override the latest version of python3 that is
launched via a shebang line ending in python3 but not python3.x


The reference implementation currently supports 3 environment variables 
- while they have different names, 2 of them work as you describe above. 
 The third is used to nominate how a simple 'python' with no version 
qualifiers should be treated - I wanted the ability for the simple 
'python' to execute a python 3.x version if desired by the user.



Thank you Mark for writing up this pep. It is almost identical to the
one I was working up but covers more details and actually provides an
implementation example. Other then the command line args I agree with
everything covered.


Thanks for the feedback!

Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

Thanks for the feedback!

On 19/03/2011 7:44 PM, Glenn Linderman wrote:

Not all of the ideas below are complementary to each other, some are
either or, to allow different thoughts to be inspired or different
directions to be taken.

Thanks for starting a PEP.

On 3/18/2011 11:02 PM, Mark Hammond wrote:


 The launcher should be as simple as possible (but no simpler.)


The launcher could be simpler if it isn't used for launching interactive
interpreters as well as script references via Windows associations (more
about that after the next quote).

The launcher could be simpler if it just read through the file of its
first parameter until it finds a line starting with #@ (process as a
Windows version of Unix #!) or starting without a # (error case). This
avoids the need parse such lines.


I don't agree with that at all.  The rules for the shebang parsing are 
very simple - see the man-page I reference in the pep - the file must 
start with the characters '#!' and the entire command must be < 127 
chars.  The only real "parsing" needed is to check if the specified 
command starts with one of 2 fixed strings.  I believe this is simpler 
than parsing multiple lines of the file.


> Remember, the typical

Windows user is not likely to place a #! line in their scripts in the
first place, so teaching them what a Unix #! line is, and how the
parameter after it should be something that Windows doesn't even use,
and the launcher has to work hard to interpret, is not as simple as
possible.


I disagree with various aspects of that - the "typical Windows user" is 
not going to add a shebang or variation of to their Python source files, 
period.  IMO, the kind of user who would is already somewhat likely to 
know of the shebang convention, so it would not be foreign.  Those 
remaining who are not familiar with it can simply be pointed at existing 
docs etc for the shebang line and their new knowledge now works on more 
than Windows (and more than Python on non-Windows platforms)



The launcher could be simpler if the Python installer placed versioned
Python executables on the PATH. Unfortunately, historically it hasn't.
If it did, would, or the launcher installer would place them there for
pre-existing versions of Python, then the launcher could work by
launching the appropriate version of Python, expecting Windows to find
it on the PATH. The PEP doesn't address the level of internal complexity
of the launcher necessary to find where Python has been installed,
neither for CPython, nor for the alternate implementations to be supported.


The PEP intentionally doesn't, but the implementation does - it already 
does exactly that for CPython.  Other implementation may need a 
different strategy, but we can cross that bridge when we come to it.



The launcher could be simpler if a directory \usr\bin were created under
Windows Program Files, placed on the PATH, and %ProgramFiles% prepended
to the Unix #! line, with the Python/Jython/Cython installers placing
appropriately versioned executables in that directory. Could even start
a trend for programs ported from Unix. One could even place an "env"
program there, for more simplicity.


Again, I disagree - I think in practice the code would be more complex, 
and having Python assert it "owns" such directories and executables is a 
can of worms we should avoid.





 * When used to launch an interactive Python interpreter, the launcher
   will support the first command-line argument optionally be a
   version specifier in the form "-n[.n]" (where n is a single
   integer) to nominate a specific version be used.  For example,
   while "py.exe" may locate and launch the latest Python 2.x
   implementation installed, a command-line such as "py.exe -3" could
   specify the latest Python 3.x implementation be launched, while
   "py.exe -2.6" could specify Python 2.6 be located and launched.
   If a Python 2.x implementation is desired to be launched with the
   -3 flag, the command-line would need to be similar to "py.exe -2
   -3" (or the specific version of Python could obviously be
   launched manually without use of this launcher.)


I think that a python launcher that is "on the PATH" that could be used
to launch an interactive Python, should be different than one that is
used to launch .py[w] scripts.


I believe that if you know the Python you want is already on the PATH, 
you should just use 'python.exe' instead of 'py.exe'.  I don't see any 
reason to use this launcher for interactive Python sessions where this 
is the case.



1) python should be invoked interactively by typing "python" or
"pythonX[.Y]" at the CMD prompt, not "py". This can be done without a
launcher, if appropriate versioned pythons are placed on th

[Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows

[Mark Hammond]

Hi all,
  During the huge thread about PEP 394, a suggestion was made that a 
"launcher" for Python on Windows could be implemented which would allow 
for some of the guidelines in that PEP to apply to the Windows version 
of Python.


I've attached the first draft of a PEP for such a launcher and put the 
first draft of the "reference implementation" (implemented in Python 3) 
at http://starship.python.net/crew/mhammond/downloads/python_launcher.py


The reference implementation is fairly complete WRT the features 
described in the PEP.  However, I'm sure there will be comments on both 
the PEP itself and the implementation, so I'm putting them up for 
discussion now rather than spending more time polishing things which may 
end up changing.  I'll start work on a C implementation once there is 
broad agreement on the functionality in the reference implementation.


All comments welcome, including, but not limited to:

* Is this really PEP material, or will turning the PEP into a regular 
spec be suitable?


* If it is a PEP, is it "Standards Track" or "Informational"?

* Does the functionality described strike the right balance between 
simplicity and usefulness?


* Does the strategy have general support from Martin, who as the person 
making Python distributions would need to be involved in at least some 
aspects of it (specifically, having the installer MSI install the 
launcher and update the file associations to reference it).


Thanks,

Mark
PEP: ??
Title: Python launcher for Windows
Version: $Revision$
Last-Modified: $Date$
Author: Mark Hammond 
Status: Draft
Type: Standards Track or Informational ?
Content-Type: text/x-rst
Created: 15-Mar-2011


Abstract

This PEP describes a Python launcher for the Windows platform.  A 
Python launcher is a single executable which uses a number of 
heuristics to locate a Python executable and launch it with a
specified command line.


Rationale

Unix-like operating systems (referred to simply as "Unix" in this
PEP) allow scripts to be executed as if they were executable images
by examining the script for a "shebang" line which specified the
actual executable to be used to run the script.  This is described in
detail in the evecve(2) man page [1].

Additionally, these operating systems provide symbolic-links to
Python executables in well-known directories. For example, many
systems will have a link /usr/bin/python which references a
particular version of Python installed under the operating-system.
These symbolic links allow Python to be executed without regard for
where Python it actually installed on the machine (eg, without
requiring the path where Python is actually installed to be
referenced in the shebang line or in the PATH.  PEP 394 'The "python"
command on Unix-Like Systems' [2] describes additional conventions
for more fine-grained specification of a particular Python version.

These 2 facilities combined allow for a portable and somewhat 
predictable way of both starting Python interactively and for allowing
Python scripts to execute.  This PEP describes an implementation of a 
launcher which can offer the same benefits for Python on the Windows 
platform.

An overview of the launcher.

This PEP outlines the general functionality and key guidelines of a
launcher for Windows.  It is accompianied by reference implemention,
written in Python, which defines the detailed implementation.  Over
time, changes to the implementation may be desired - if the changes 
adhere to the guidelines in this PEP and have been made following 
the standard Python development model, this PEP need not change.  
In other words, this PEP makes no attempt to describe in detail every 
feature offered by the launcher, but instead to offer guidelines the 
launcher should adhere to.

The actual implementation of the launcher will be written in C and
will not link directly or indirectly with Python, but the Python
based reference implementation should always remain the canonical
description of the functionality.  Any changes proposed to the
launcher functionality should first be made and agreed to in the
Python implementation, after which time the C implementation can be
changed to reflect the new reference.  Any deviations between the
functionality of the reference implementation versus the C
implementation due should imply the C implementation is in error.

It is expected that 2 versions of the launcher will be needed - one
which is a console program and one which is a "windows" (ie, GUI)
program.  These 2 launchers correspond to the 'python.exe' and
'pythonw.exe' executables which currently ship with Python.  The
console launcher will be named 'py.exe'

Re: [Python-Dev] Bugs in thread_nt.h

[Mark Hammond]

These issues are best put in the tracker so they don't get lost - 
especially at the moment with lots of regulars at pycon.


It would also be good to know if there is an actual behaviour bug caused 
by this (ie, what problems can be observed which are caused by the 
current code?)


Cheers,

Mark

On 10/03/2011 12:25 PM, Sturla Molden wrote:


Atomic operations (InterlockedCompareExchange, et al.) are used on the
field 'owned' in NRMUTEX. These methods require the memory to be aligned
on 32-byte boundaries. They also require the volatile qualifer. Three
small changes are therefore needed (see below).


Regards,
Sturla Molden





typedef struct NRMUTEX {
volatile LONG owned ; /* Bugfix: remember volatile */
DWORD thread_id ;
HANDLE hevent ;
} NRMUTEX, *PNRMUTEX;


NRMUTEX
AllocNonRecursiveMutex(void)
{
PNRMUTEX mutex = (PNRMUTEX)_aligned_malloc(sizeof(NRMUTEX),32) ; /*
Bugfix: align to 32-bytes */
if (mutex && !InitializeNonRecursiveMutex(mutex))
{
free(mutex) ;
mutex = NULL ;
}
return mutex ;
}

void
FreeNonRecursiveMutex(PNRMUTEX mutex)
{
if (mutex)
{
DeleteNonRecursiveMutex(mutex) ;
_aligned_free(mutex) ; /* Bugfix: align to 32-bytes */
}
}







___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/skippy.hammond%40gmail.com



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 9/03/2011 5:05 PM, Glenn Linderman wrote:


Standard installation paths are accepted by about 99% of the users, so
embedding standard installation paths can work well for that batch of
users. Of course, Windows changes the standard path periodically, so
that it different from versions prior to and including WinXP versus
versions after WinXP. And they had no standard before WinNT (or if they
did, few followed it). Your comment does point out a possible need for
multiple standard installation paths just for different versions of
Windows, though :(


The python installer ignores "Program Files" - as much as anything I'm 
concerned about the drive letter - IIUC, Python will default to 
installing on the system drive, which may not be C:


...


Your premise that using a standard installation path in #@ restricts the
script to be targeted to one machine is fallacious, so the conclusions
are also.


Sorry - I should have said "some machines with Python installed by the 
installer".  Contrast this to the option of looking up the registry and 
you have "all machines with Python installed by the installer".  I'm not 
going to quibble about the percentages in the first category as I have 
no data to guide me, but I'm reasonably confident about the "all" in the 
second.


...


The #! line is clearly is NOT "the only clue Windows needs to make it
work automagically" or we wouldn't be having this discussion at all.


Sorry - I should have been clear - I meant "the only clue a windows 
specific Python launcher needs".


> And

it is not at all clear if a Windows machine contains Jython, IronPython,
and Cython which one should be launched by a launcher.


I think it is reasonably clear: it would be the same version which would 
be launched on most *nix systems.  Are there any systems where 
"/usr/bin/python" would use Jython, for example?



You could, of
course, argue that python-dev is only concerned with CPython, and the
launcher can be specific to CPython, and that argument might carry the
day, but the CPython registry lookup necessary to make that happen
doesn't help the users of alternate implementations a bit, forcing each
of them to implement their own launcher as well...


I'd have no problem with the Windows launcher supporting other 
implementations - I expect something like "/usr/bin/jython" would be 
reasonable.  How we locate the installed jython is indeed an interesting 
problem, but I'd guess it is surmountable - especially if we can get the 
buy-in of these alternate implementations for their future releases.



#!/usr/bin/env python2.6
#@"C:\Program Files\Env.exe" ipy2.6.3.exe


As above, the C: assumes Windows is installed on C: - that may be 
common, but not universal.  Further, a 64bit machine would probably want 
"\Program Files (x86)" (which is becoming more and more common)




in .py files and

#!/usr/bin/env python2.6
#@"C:\Program Files\Env.exe" ipy2.6.3w.exe

in .pyw files.


I'm slightly skeptical about that - .pyw files are surely rare on *nix 
systems and as we've discussed, pyw files are generally not meant to be 
cmd-line scripts.  IOW, I'm skeptical scripts exist which are meant to 
work with '/usr/bin/env python2.6' on *nix and pythonw.exe on windows.


...


So no matter what is done, it won't solve world hunger. Probably that's
why Windows users have been left high and dry for years in this
regard... there is no "system" Python on it, out of the box, so the only
users are those that are smart enough to download and install and
configure things, and those who use a Python embedded into an
application, at the cost of a Python installation per application,
because there has been no "system" Python, and because Windows users are
not perceived to be capable of, or willing to bother with, downloading
and installing various requisite dependencies.

The resulting vacuum has been filled by people with different points of
view, because there has been no technique implemented, none declared to
be "best", and anarchy and variant workarounds have abounded.


I'm glad solving world hunger is out of scope for this :)  I understand 
your position but my personal opinion is that simple support for #! is 
more desirable.  I'd be happy to go with the consensus though...


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 9/03/2011 1:43 PM, Glenn Linderman wrote:

I'm of the opinion that attempting to parse a Unix #! line, and intuit
what would be the equivalent on Windows is unnecessarily complex and
error prone, and assumes that the variant systems are configured using
the same guidelines (which the Python community may espouse, but may not
be followed by all distributions, sysadmins, or users). That's why I
propose a different line for Windows... it is as simple as the
long-proven Unix #! line, but imposes no restrictions on or requirements
that there be a #! line; it has more flexibility in that it could invoke
different versions or provide different options on Unix and Windows if
necessary for some environments.

#!/usr/bin/env python2.6 -B
#@c:\python26\python2.6.exe

or

#!/usr/bin/python2.5
#@"C:\Program Files (x86)\IronPython 2.6 for .NET 4.0\ipy.exe"


I don't think that buys us much.  A script with "#!/usr/bin/env python" 
could be distributed with an expectation it will work across various 
different machines (and possibly even different operating systems).  A 
script with "c:\..." could not be distributed and expected to work 
reliably anywhere.  ie, any script with a #! line (or even a #@ line) 
with a fully qualified Windows path can only be expected to work on a 
single machine - so there is no need to support both #! and #@ as the 
script is not even cross-machine portable, let alone cross-platform 
portable.


The only way to expect a #! line to work across machines would be to 
have a "virtual" path - eg just "python2.6" without any path specifier 
at all.  In that case, I see no problem with reusing the #! from *nix 
systems and treating "/usr/bin" etc as a "virtual" specifier on Windows. 
 If people find a need on Windows to add a fully-qualified path to this 
line (whatever the spelling), they are implicitly saying this script 
works only on the current machine.


IOW, as soon as someone has:

#!/usr/bin/env python2.6 -B
#@c:\python26\python2.6.exe

in their script, the script is targeted at exactly 1 specific machine, 
so why not just reuse the #! syntax?  OTOH, if an existing script has:


#!/usr/bin/env python2.6 -B

They are attempting to declare in a portable way that Python 2.6 is 
necessary - so why force them to add a #@ line to make it work on 
Windows when the #! line is the only clue Windows needs to make it work 
automagically?


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 8/03/2011 10:15 AM, Greg Ewing wrote:

Mark Hammond wrote:


Yup - although I think a pythonw.exe launcher would be needed too


Couldn't the launcher look at the extension of the file being
launched to decide about this?


Nope - the launcher itself must be marked as "console" or "windows", and 
hence would suffer the exact same problem which needs us to have 
python.exe and pythonw.exe in the first place (ie, in some cases you get 
a "temp" console where you don't want one, and in other cases you get a 
new console where you wanted an existing one to be used)


Mark


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 8/03/2011 7:33 AM, Michael Foord wrote:

A python launcher as you describe is a *great* idea.

A few concerns:

* we're missing an opportunity to do something easy (Martin is happy to
modify the installer and says it is easy) for something that may or may
not happen


Don't let my -0 stop anyone :)


* will you call it python.exe? will it be installed by the python
installer?

- I doubt calling it python.exe will fly, but I'm not sure. If so what
will you call what is currently 'python.exe'? - if not then "python
foo.py" on the command line will *still* not work...


Calling it python.exe would make the most sense for people who don't 
look behind the curtain, but I agree it could potentially be confusing 
for people.  Further, we would need to ensure we didn't create an 
infinite loop where the launcher python.exe found a python.exe it 
thought was an appropriate sub-process, but where it turns out it is 
actually another launcher.


Having it installed by the Python installer also makes sense to me but 
I'd be very interested in Martin's take on this (and also on everything 
else we are discussing here).



* we're (yet again) making instructions for running stuff on Windows
*different* to other platforms (and making tutorials written for other
pythons "not work" in certain ways)


Actually, I'd argue we are making them more similar as we would be 
honoring a shebang line and allowing the exact same mechanism be used to 
denote the major version of Python needed.



* as I work with multiple platforms it would be really nice if the same
invocations worked across all of them - whilst I say again that I really
like the idea of the launcher it doesn't conflict with the other
suggestions (creating multiple binaries) and as you (Mark) say it
wouldn't hurt...

So why not do both? We could create the extra binaries to bring Python
on Windows inline with the unix conventions for command line
invocations, and the new launcher can follow on as a nice addition.

Note that the discussions about the Python installer adding to the PATH
won't be *ended* by the creation of the installer. A typical install on
a Unix-like system adds various other utilities to the path that merely
adding the top-level of your Python install on Windows still doesn't
add. In particular distutils installed scripts go into a subdirectory of
your Python install.


The distutils scripts are a good point.  To be honest, apart from the 
"file association" issue, I can't see much advantage in doing both - if 
all Python directories end up on your path such that "python3 foo.py" 
magically works, then the launcher script is adding complexity without 
bringing much to the table.  Further, I'm somewhat skeptical that the 
file associations are used by that many people in the real world - 
currently when you double-click on a script you get a temp console 
created, and should a traceback be raised, you get to see it for about 
2ms before the console is destroyed.  Some people may be .pyw to avoid 
that, but then their script has to go to extraordinary lengths to 
display such errors in a UI of some kind.  I suspect most people just 
find it more convenient to launch such scripts from a console.  Maybe a 
quick poll on python-list would be reasonable...


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 7/03/2011 9:33 PM, Paul Moore wrote:

That sounds like a fairly cool idea. So if I follow what you're
suggesting, we'd have a single python.exe, probably installed in
system32, which did the necessary command line juggling and shebang
parsing, then simply redirected to the appropriate Python interpreter?
Presumably that launcher would be pretty version independent, so (a)
the one that gets installed with Python 3.3 would support older
versions even though they didn't include the launcher themselves, (b)
overwriting the launcher when a new version of Python is installed
wouldn't be too big a deal, and (c) it could be released as a
standalone package for people with only older versions of Python
installed?


Yup - although I think a pythonw.exe launcher would be needed too (for 
the same reasons we need python.exe and pythonw.exe today)



I like this idea. If I had the spare time (I don't :-() I'd work on this myself.

+1 from me.


Cool - I may have a go at this over the next few weekends... :)

Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 7/03/2011 4:34 PM, Nick Coghlan wrote:

On Mon, Mar 7, 2011 at 3:19 PM, Mark Hammond  wrote:

Without putting too much thought into it, I think a simple scheme could work
where the path must either be "/usr/bin/python[23]?" or a fully-qualified
path to a Python executable.  IIUC, this should allow most scripts to have a
shebang line that "does the right thing" on *nix and Windows systems while
still offering maximum flexibility on Windows.


Such a scheme should be able to handle "/usr/bin/env python2" and
variants as well.


Agreed - that is just my ignorance showing through :)

Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 7/03/2011 3:30 PM, Glenn Linderman wrote:

I'm only against the overhead of a helper written in Python, since it
would have to launch Python (some explicit version) to run the helper
script, and then launch the "right" version of Python to execute the
real script. You mention a thin executable wrapper, and I have no
problem with the overhead of that, probably.


Yeah - I think any such wrapper could be a reasonably small C program - 
although I guess that depends on the decided functionality :)


...



Since Windows does not use a #! line, then the solutions for (2) must be
different. The ones that I can think of are:

(A) declare the Python version in the name of the script file. By doing
so in the extension, additional Windows associations could be created to
launch different versions of Python. Switching versions would be as
simple as renaming the file, changing the extension to include a
different version. However, various tools would have to learn about
additional extensions that mean Python (syntax directed editors, etc.)
And it would be hard to have a script in a module, unless the importer
recognized all those extensions too. So this "solution" has ripple
effects that make it unattractive, although it is simple to implement
the basic feature.


Agreed - I find this unattractive.


(B) declare the Python version in the content of the script file. This
cures most of the ripple effects of the above, but requires a "launcher"
or "wrapper" program to be designed, implemented, and installed. There
are a variety of subsolutions for different ways of declaring the version.




Without putting too much thought into it, I think a simple scheme could 
work where the path must either be "/usr/bin/python[23]?" or a 
fully-qualified path to a Python executable.  IIUC, this should allow 
most scripts to have a shebang line that "does the right thing" on *nix 
and Windows systems while still offering maximum flexibility on Windows.



By default, for Python, the wrapper would get associated only with .py
files.


There would probably also need to be the same for ".pyw", but it would 
be almost identical to the .py handling.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 7/03/2011 10:04 AM, Michael Foord wrote:

Paul Moore was +1 on Windows being included. Mark did accept that some
of the changes were "desirable", but was also concerned they didn't
address all the issues on Windows. I *would* like to see all the issues
addressed but I think that is outside the scope of this PEP.


I simply think Windows is outside the scope of this PEP - the title is 
"The "python" command on Unix-Like Systems" - IMO it should be kept 
short and focus on exactly what the title says.


...


A generic Python launcher that could be used for the file association
and honours the Unix style shebang line, delegating to the appropriate
version of python, is an interesting idea but out of the scope of this PEP.


As above - I think Windows should be kept out of scope.


It would be interesting to see if Mark, or others, are strongly opposed
to these ideas or have strong preferences. If the biggest downside is
really that it still leaves some issues unresolved then lets do this and
tackle the other issues separately.


As mentioned in my other email, I'm -0 as I think they are slightly 
misguided.  Why not leave the PEP with "Unix-like" in the title to 
Unix-like systems and create one with Windows in the title for Windows? :)


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 7/03/2011 1:55 AM, Paul Moore wrote:

On 6 March 2011 02:33, Mark Hammond  wrote:

IIUC, the PEP language is referring to links which point to a specific
version of Python and that there is no suggestion a 'python3' will live in
the Python 3 binary tree.  If that is correct and assuming we don't want to
investigate using links on Windows, I'd suggest the best analogy to the *nix
situation could be simply for the installers to generate "python2.bat" and
"python3.bat" files and put them in System32 with a "last installed wins"
policy.


No, no, no! Please no! Bat files are a lousy way of writing wrappers
on Windows, as they don't nest. I've lost count of the number of times
I've been tripped up by putting a series of commands into a batch
file, only to have them fail because somewhere in the middle is a bat
file wrapper which makes the "outer" bat file stop part way through
:-(


To be clear, I was suggesting that using .bat files in system32 is a 
close analogy to the *nix situation - I didn't mean to advocate for it 
to actually happen :)  Further, I see the creation of a python3.exe in 
the Python install directory as quite different than the *nix situation 
(where the 'python3' link is not in the install dir, but instead in a 
'system' dir).  IOW, I was trying to point out the solution to the 
problem on *nix doesn't translate that well to Windows, so Windows 
should not be considered as part of this PEP.  That isn't to suggest 
Windows should not be considered at all.


That said though, I'm only -0 on python2.exe/python3.exe - I don't think 
it will hurt, but also don't think it will help that much in practice. 
It may also turn out to be unnecessary should a "complete" solution be 
implemented - eg, a "python launcher" which (a) read the shebang lines 
and (b) allowed something like "python -3" on the command-line would 
render both python3.exe and requests to have multiple installed Python 
versions on the PATH redundant.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 6/03/2011 11:51 PM, Dj Gilcrease wrote:
> Why not modify the windows installers to install into C:\Python\X.Y
> and have the .bat files generated in C:\Python which is what I have
> been doing manually since py25. I just add C:\Python to the system
> Path then create/modify the bat files for new versions of python I
> install.

That sounds like a reasonable scheme people may choose to use - however, 
it doesn't really address the basic issue - something still needs to add 
c:\Python to PATH, and the scheme itself doesn't really necessitate the 
common parent directory - ie, it is fundamentally the same (if less 
"clean") as the existing "c:\pythonx" with a (eg) "c:\python-cmd" or 
"c:\python-scripts" directory with the .bat files - especially given the 
various installations can be located via the registry.


...

> Something I have been thinking about recently though is outside the
> scope of the pep is writing a python.exe, to replace the python.bat,
> that would  try to read the shebang line of the file to send it to the
> right version of python. Then I just associate py files with the
> dispatcher exe and everything should work as intended.

But where would such a python.exe live and how would that directory end 
up on the PATH?


On the more general idea though, it could have legs as it would solve 
the file association issue for files which include the shebang and 
arrange for the status-quo (or better) for files which don't...


But this sounds like a different PEP ;)

Cheers,

Mark.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 6/03/2011 12:37 AM, Michael Foord wrote:

On 05/03/2011 07:02, Nick Coghlan wrote:

On Sat, Mar 5, 2011 at 10:47 AM, Mark
Hammond wrote:

I think this discussion should be divorced from this PEP and taken up
with
the discussion about the PATH and the "last installed wins" issue Martin
mentions - only all of them taken together will "fix" this issue -
not that
I personally consider it particularly broken - more like
"sub-optimal" :)

I updated the draft PEP, explicitly bringing Mac OS X and Cygwin
within the scope of the recommendation, and excluding PYTHON*
environment variable considerations and Windows-related proposals. The
Windows section does include a hit-list that may serve as a useful
starting point if someone else felt like starting a Windows specific
PEP, though.


The issues are relevant to Windows (there are *other* issues as well but
that doesn't mean that the same issue doesn't apply). Guido also said he
would like to see Windows addressed.


What issues specifically?  If I look at the current PEP language, it 
refers to "the default version of the Python interpreter" and "all new 
code that needs to invoke the Python interpreter".  Neither of these 
apply in any meaningful way to Windows.  It could be argued that the 
latter could include .bat files which use Python but that sounds like a 
theoretical problem (ie, I haven't heard much noise about that) where 
PEP394 is addressing an observed practical problem.


On Windows it seems to be just about 'python2' and 'python3' doing the 
correct thing in a default command prompt, which while desirable ignores 
the broader issues (eg, the file associations and anything else 
"windowsy" and not driven by the command-prompt)




I'll be happy to try and come to some consensus with Martin (and others)
on what we *can* do for Windows (as similar to the other platforms as
possible would be my goal) and provide appropriate text for the PEP.


IIUC, the PEP language is referring to links which point to a specific 
version of Python and that there is no suggestion a 'python3' will live 
in the Python 3 binary tree.  If that is correct and assuming we don't 
want to investigate using links on Windows, I'd suggest the best analogy 
to the *nix situation could be simply for the installers to generate 
"python2.bat" and "python3.bat" files and put them in System32 with a 
"last installed wins" policy.  The fact this doesn't help users 
installing Python "just for me" is fine too from the POV of matching 
*nix systems - a user installing a private Python build on *nix also 
doesn't get the python2 and python3 conveniences, nor the automatic PATH 
convenience.  But in practice I expect this would *not* be OK - which 
just highlights the risks of trying to generalize a specific observed 
problem on one OS to others.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEPs] Support the /usr/bin/python2 symlink upstream

[Mark Hammond]

On 5/03/2011 8:21 AM, "Martin v. Löwis" wrote:
...

As for Windows support: we currently don't install a python3.exe binary,
let alone python2.exe or pythonw2.exe (or is that python2w.exe?). I'll
adjust the installer if the PEP asks me to. For the reasons discussed,
I'm -0 on the change (i.e. double-clicking .py will continue to launch
the most-recently installed Python, rather than the "right" one, and
setting PYTHONPATH will continue to break installations).


I agree with the -0 - I simply don't think it will, in practice, make 
anyone's life much easier.  To run python2 and python3 based scripts in 
the same environment already requires some manual work by the machine 
owner (both directories will need to be added to the path) so the 
additional burden of some other steps (eg, .bat files, doskey alaises 
etc) doesn't seem that great.  There is also a small risk of confusion - 
people may believe python.exe and python2.exe/python3.exe have different 
purposes and be confused about when to use which.


I think this discussion should be divorced from this PEP and taken up 
with the discussion about the PATH and the "last installed wins" issue 
Martin mentions - only all of them taken together will "fix" this issue 
- not that I personally consider it particularly broken - more like 
"sub-optimal" :)


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 384 status

[Mark Hammond]

On 31/08/2010 7:54 AM, Nick Coghlan wrote:

On Tue, Aug 31, 2010 at 12:47 AM, Michael Foord
  wrote:

An extension compiled for one version of Python will be linked against the
version of the C runtime used by that version of Python (if it is compiled
with the same version of Visual Studio of course).

If the extension binary is to remain compatible with a later version of
Python, compiled against a different version of the C runtime, then it
*must* be possible for multiple C runtimes to be loaded. If the stable ABI
doesn't allow this then binaries will *still* have to be recompiled when we
update the version of Visual Studio used to compile Python - defeating the
purpose of the PEP. Right?

If this is the case then I agree that it should be explicit in the PEP.


Ah, I knew it was explicit in the PEP somewhere. The following is
currently mentioned in the "Excluded Functions" section:

"In addition, functions expecting FILE* are not part of the ABI, to
avoid depending on a specific version of the Microsoft C runtime DLL
on Windows."


It would be interesting to know how, in practice, these FILE pointers 
come to life.  In my experience they are generally obtained via fopen. 
If that is broadly true, then a middle-ground may be for Python to 
expose something like Py_fopen, Py_fclose and a PyFILE opaque "handle". 
 API elements which currently take a FILE * could be exposed using a 
PyFILE * in the ABI.  People who didn't care about this level of 
portability could continue to use the non-ABI FILE * functions, but 
people who do could use Py_fopen/Py_fclose in place of fopen/fclose but 
otherwise work as now.


A downside is that as mentioned, in practice these FILE pointers may 
come from more than simply fopen, so few people would be able to 
leverage this.  It also assumes that people open files before handing 
them to Python, but otherwise don't use that file - it would be a 
slippery-slope to wind up with Py_fread etc.


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows

[Mark Hammond]

On 4/08/2010 11:08 AM, Steve Holden wrote:

It's a little disappointing to discover that despite the relatively
large number of developers who have received MSDN licenses from
Microsoft, none if us have the time to make sure that the buildbots are
green for the 2.6.6 release.

I wonder if anyone can think of a way we can get some Windows skillz
into the group that could assist at ties like this. Some brainstorming
might find a way through.


I never go looking at the buildbots to look for problems - maybe some 
way of explicitly bringing such failures to peoples attention would be 
good - I don't recall seeing anything recently on python-dev which would 
prompt me to take a look.


Visiting http://www.python.org/dev/buildbot/2.6/ shows a single Windows 
buildbot that seems to have been green for the last few builds - am I 
looking in the wrong place?


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows registry path not ignored with Py_IgnoreEnvironmentFlag set

[Mark Hammond]

On 2/06/2010 11:32 AM, Farshid Lashkari wrote:

Hello,

I noticed that if Py_IgnoreEnvironmentFlag is enabled, the Windows
registry is still used to initialize sys.path during startup. Is this an
oversight or intentional?


I guess it falls somewhere in the middle - the flag refers to the 
'environment' so I believe it hasn't really been considered as applying 
to the registry - IOW, the reference to 'environment' probably refers to 
the specific 'environment variables' rather than the more general 
'execution environment'.



I assumed one of the intentions of this flag is to prevent embedded
Python interpreters from being affected by other Python installations.
Ignoring the Window registry as well as environment variables seems to
make sense in this situation.


I agree.


If this is an oversight, would it be too late to have this fixed in
Python 2.7?


Others will have opinions which carry more weight than mine, but I see 
no reason it should not be fixed for *some* Python version.  Assuming no 
objections from anyone else, I suggest the best way to get this to 
happen in the short to medium term would be to open a bug with a patch. 
 A bug without a patch would also be worthwhile but would almost 
certainly cause it to be pushed back to a future 3.x version...


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385 progress report

[Mark Hammond]

Hi Dirkjan,

On 8/02/2010 8:35 AM, Dirkjan Ochtman wrote:
...


In fact, a few weeks ago I talked to Brett and we figured that we
should probably pin down a deadline. We discussed aiming at May 1, and
at this time I think that should be feasible. That also seems to
coincide with the release of 2.7b2, though, so maybe we need to do it
one week later (or sooner?). Anyway, we figured that a weekend would
probably be a good time. If we manage to find a good date, I'll put it
in the PEP.


Isn't setting a date premature while outstanding issues remain without a 
timetable for their resolution?



As for the current state of The Dreaded EOL Issue, there is an
extension which seems to be provide all the needed features, but it
appears there are some nasty corner cases still to be fixed. Martin
Geisler has been working on it over the sprint, but I think there's
more work to be done here. Anyone who wants to jump in would be quite
welcome (maybe Martin will clarify here what exactly the remaining
issues are).


See http://mercurial.selenic.com/wiki/EOLTranslationPlan#TODO - of 
particular note:


* There are transient errors in the tests which Martin is yet to 
identify.  These tests do not require windows to reproduce or fix.


* The mercurial tests do not run on Windows.

Given the above, most sane Windows developers would hold off on "live" 
testing of the extension until at least the first issue is resolved - 
but the second issue makes it very difficult for them to help resolve that.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] First draft of "sysconfig"

[Mark Hammond]

On 15/12/2009 3:09 PM, David Lyon wrote:

On Tue, 15 Dec 2009 15:05:18 +1100, Mark Hammond

wrote:

But under windows, an application developer might (as in probably
would) like to install an application in \Program Files\someapp
rather than hidden in the bowels of the python interpretor.


I agree - but in that case you are talking about an application built
with Python - that is a different set of requirements.


Building an application with python.. that's right. Of course. Why not?


I'm missing your point - many applications exist written in Python.


IOW, this isn't designed for applications which happen to be written in
Python.  There might be a case for such a module to be created, but this
PEP doesn't attempt to solve that particular problem.


But programmers might want to write an application with python. It
doesn't seem like such an edge-case thing to do.


They can, and they have.  So again your point is lost on me.


They might like their data in "Application Data", which is where
support people get trained to look for application data. Not down
in \pythonX.Y\ ...


Nothing is stopping them from doing that - but this PEP isn't intended
to provide that information.


Distutils is stopping them.


I don't agree with that and I can present many applications as evidence. 
 You yourself mentioned mercurial and it looks for mercurial.ini in the 
user's appdata directory.


Regardless, this discussion isn't about distutils.


It does - many applications written in Python exist which do exactly
that.


Yes. And they don't use any of the built in facilities, under windows.


To continue the mercurial example - mercurial will not use sysconfig to 
determine where to look for mercurial.ini on *any* operating system. 
sysconfig is not about solving that particular problem.



So under windows, then, what is it trying to solve? Thats what I am
asking.


The same thing it is trying to solve for non-Windows users - various 
threads here have articulated this well.  You needn't feel bad about not 
having such use-cases yourself - that simply means sysconfig isn't 
targetted at you - it isn't targetted at application developers on any 
operating system.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com