[issue38999] Python launcher on Windows does not detect active venv

[Eryk Sun]

Eryk Sun  added the comment:

> I realize one could interpret the python3 in the shebang as an 
> "explicit version specification", but a) that's not on the 
> command line, which seems the more natural reading and b) it's 
> only a partial specification, since it doesn't say which python3
> to use.

The wanted version is specified by the command line, else by the script 
shebang. The default version to use for `py -3` or a "python3" virtual command 
in a shebang (e.g. "#!/usr/bin/python3") is determined by the PY_PYTHON3 
environment variable or the "python3" setting in "py.ini". If a default 3.x 
version isn't configured, the launcher uses the latest-version installation of 
3.x, of which the 64-bit version is preferred (e.g. 3.10 is preferred over 
3.10-32, which is preferred over 3.9).

For just `py` or a "python" virtual command in a shebang -- i.e. when no 
version is specified -- an active virtual environment is preferred. Else it 
uses the default that's set in PY_PYTHON environment variable or the "python" 
setting in py.ini. If no default is set, and there's no shebang, the launcher 
prefers the latest-version 3.x that's installed. If there's a shebang, the 
launcher prefers the latest-version 2.x that's installed.

Originally, the non-versioned "/usr/bin/env python" virtual command was handled 
the same as all other non-versioned "python" virtual commands. But later on the 
"env" form was changed to first try a path search for "python.exe" via WinAPI 
SearchPathW() before falling back on the common behavior. In most cases this 
search will find an active virtual environment. But it could be that another 
directory with "python.exe" is added to the front of PATH after a virtual 
environment is activated.

---

How about generalizing "/usr/bin/env python*" to support virtual environments 
by getting the "version" (venv) or "version_info" (virtualenv) value from 
"%VIRTUAL_ENV%\pyvenv.cfg"? I'd prefer to get that value in 
locate_venv_python(). It can still set the `version` string to "venv", but 
extend the INSTALLED_PYTHON record with a `venv_version` string that can be 
checked in the result of find_python_by_version(L"venv").

Extend locate_python() with a `search` parameter. If `search` is true, then do 
a path search [1] for `wanted_ver`, with the searched "python*.exe" executable 
name set depending on the launcher build (i.e. py[w][_d].exe). If searching is 
skipped or doesn't find a match, then run an active virtual environment if 
either `wanted_ver` is empty or `search` is true and `wanted_ver` is compatible 
with the virtual environment. Otherwise locate a registered Python installation 
based on `wanted_ver` and `from_shebang`.

---

For GUI and debug builds, the PYTHON_EXECUTABLE macro could be split into a 
base "python[w]" name and an extension "[_d].exe" name, which makes it easy to 
construct the name to search for given a valid version string. For example, 
with a pyw_d.exe debug build, "python3" would be searched for as 
"pythonw3_d.exe".

---

[1] I'd prefer to expand the path search to also check the user and machine 
"App Paths", which are like "$HOME/.local/bin" and "/usr/bin" for the Windows 
shell.

--
versions: +Python 3.10 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Ross Boylan]

Ross Boylan  added the comment:

As someone who finds the current behavior surprising, and having lost 
significant time because of it, I have a couple of comments.

1) If the venv created a python3 (or 2, as appropriate) file, would the 
expected behavior with !#/usr/bin/env python3 be restored?  I think the 
previous analysis said that python3 doesn't match the venv because there is no 
python3 in the venv path.  So putting one there would fix it.  Of course, it 
would only help for newly created venvs.  The pythonN file would need to be in 
addition to the python file, not instead of it, so that invocations of python 
without a number continued to work as expected.

2) There appear to be two separate sections of the documentation which govern 
this case. 3.7.1.2 of Python Setup and Usage says "If the launcher is run with 
no explicit Python version specification, and a virtual environment (created 
with the standard library venv module or the external virtualenv tool) active, 
the launcher will run the virtual environment’s interpreter rather than the 
global one. "
If py myscript.py is invoked that seems to fit this case, but obviously that's 
not what's happening.  I realize one could interpret the python3 in the shebang 
as an "explicit version specification", but a) that's not on the command line, 
which seems the more natural reading and b) it's only a partial specification, 
since it doesn't say which python3 to use.

Then section 3.7.2 is about !# lines.  So if you invoke py without a version 
specifier, but the invoked file has a !#, does 3.7.1.2 or 3.2 apply?

--
nosy: +rossb

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Steve Dower]

Steve Dower  added the comment:

> Do you mean all shebangs, including those with a native file path?

I like the idea as a whole, but ultimately end up disliking all the available 
options (in the context of py.exe, that is).

* Unix/virtual shebangs don't work reliably on Windows (as we can see :) )
* Native file path shebangs on Windows don't work on Unix

The scripts installed by pip &co. are their own executable wrappers, so the 
shebang shouldn't matter.

But, the python.org Windows installer associates .py files with py.exe, so it 
_does_ matter for double-click and PATHEXT scenarios.

Perhaps it is reasonable to redefine "/usr/bin/env python*" as "use 
%VIRTUALENV% python.exe if set" regardless of the specific version? I really 
don't want to add checks that require running Python - I'd rather bail out and 
recommend using "py" directly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Eryk Sun]

Eryk Sun  added the comment:

> I think supporting the shebang line in py.exe is a misfeature and 
> would prefer we'd never done it (though it predates my involvement). 

Do you mean all shebangs, including those with a native file path?
Or do you mean just virtual shebangs, and in particular those that search PATH 
via "/usr/bin/env"? It's not without controversy even in Unix. For example, see 
[1] and the rebuttal to it in the comments.

In the case of this issue, "env" is used to run a script in an active virtual 
environment. This doesn't rely on the user or system PATH, assuming the command 
can be found in the active scripts directory.

As an alternative to modifying the launcher, pip (via distlib) could be updated 
to transform "#!/usr/bin/env pythonX[.Y]" shebangs into "#!/usr/bin/env python" 
when installing scripts in Windows. pip and distlib are third-party tools, 
however, so that suggestion is beyond the scope of this issue tracker.

[1] https://jmmv.dev/2016/09/env-considered-harmful.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Steve Dower]

Steve Dower  added the comment:

> This makes it simple for users to intervene via the PATH (which is the whole 
> point of /usr/bin/env) and launch whatever they want.

That's one point of view, but the other is that the whole point of the shebang 
line is for users to be able to run the script without having to use a terminal 
at all. And once you're there, you no longer have a customisable PATH to work 
with.

"Simple for users to intervene via the PATH" is not true on Windows. Modifying 
PATH globally can break applications or (some) system components, and modifying 
it temporarily requires becoming a terminal user, which is not the majority.

Personally, I think supporting the shebang line in py.exe is a misfeature and 
would prefer we'd never done it (though it predates my involvement). But if 
someone wants to implement support for detecting a venv and matching it to the 
shebang, I won't actively block it going in.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Alexandros Karypidis]

Alexandros Karypidis  added the comment:

My suggestion to gradually fall back from most specific to least specific 
version, was an attempt to reconcile the input from various viewpoints.

I think that ideally the behaviour should be as close to UNIX as possible. This 
means that it should just search the path for that binary (which is the whole 
point of using /usr/bin/env) and use it (even if it is the wrong version).

While I understand that one can do more (verify the version in whatever 
executable is located, etc) there is a lot in favor of consistency across 
platforms (versus even better behavior).

So my vote if the community decides to address this would be to just "search 
the path for .exe when encountering #!/usr/bin/env " and 
then launch it and hope for the best.

This makes it simple for users to intervene via the PATH (which is the whole 
point of /usr/bin/env) and launch whatever they want. 

Currently, creatign a venv on windows only places "python.exe" in the Scripts 
folder (another difference to Linux). I would argue this is "correct" though 
because I think this was a decision of DISTROS (to use python for version 2 and 
python3 for version 3, while undertransition). Most distros still do that to 
this day and it will likely stick around. One day there will be only "python" 
and it will mean "version 3" but until then, it's better for the py launcher to 
do the same as on Linux and say "could not find python3 in the path" where the 
user can intervene (e.g. copy python.exe as python3.exe) and "fix" it manually.

Of course, doing "smarter" things on Windows is always an option but I would 
say that at least the "minimal" should be done to support consistency. (As I 
said, I would also suggest that no more than this minimal be done, but not in 
order to keep implementation simple, but to keep it consistent with other 
platforms).

Then again, that's merely one point of view from someone who is a user and does 
not even contribute to the project...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Eryk Sun]

Eryk Sun  added the comment:

I don't think the "/usr/bin/env" case needs to limit qualified names like 
"python3[.x]" to registered installations. This is a choice made to simplify 
the implementation. If it finds a generic "python.exe" executable in PATH, for 
3.5+ it is possible to query version information directly from it. Older 
versions lack this metadata, but it's also possible to inspect the PE import 
table (i.e. IMAGE_DIRECTORY_ENTRY_IMPORT) for a dependency on "python2x.dll" or 
"python3x.dll".

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Alexandros Karypidis]

Alexandros Karypidis  added the comment:

I think this situation is not ideal and the limitation seems artificial.

If I understand correctly, there are two parameters that govern the behaviour 
of the python launcher:

1) Is the shebang using /usr/bin/env or not?
2) IS the shebang specifying a version qualifier or not?

There are 4 combinations available:

a) env with version qualifier
b) env without version qualifier
c) non-env with version qualifier
d) non-env without version qualifier

Of the above, the python launcher supports (b) and (d).

For case (c) the launcher is configurable with py.ini

For case (a) it was decided to treat it as case (b) and search for "python.exe" 
in the path. To quote:

In the case of (1) I read:

"The launcher also looks for the specific shebang line #!/usr/bin/env python. 
On Unix, the env program searches for a command on $PATH and runs the command 
so located. Similarly, with this shebang line, the launcher will look for a 
copy of python.exe on the user's current %PATH% and will run that copy.

As activating a virtualenv means that it is added to PATH, no special handling 
is needed to run scripts with the active virtualenv - they just need to use the 
#!/usr/bin/env python shebang line, exactly as on Unix. (If there is no 
activated virtualenv, and no python.exe on PATH, the launcher will look for a 
default Python exactly as if the shebang line had said #!python)."

Why does the launcher search for "python.exe" specifically when using a version 
qualifier with env? This is very different to what happens on UNIX and causes 
the issues I reported. I see no reason why using #!/usr/bin/evn pythonX where X 
is a major version should not check the PATH for that specific version (e.g. 
#!/usr/bin/evn python3 should check for python3.exe in the PATH) and only fall 
back to plain "python.exe" if no match is found. This would be much closer to 
UNIX behaviour and allow for "common" shebangs across Windows/Linux/Darwin etc.

I would also note that if I create a python2 virtual environment and use a 
#!/usr/bin/env python3 shebang, a search on the path for python.exe would yield 
the version 2 executable from the virtual environment folder. So the argument 
about there being a technical issue is anyway true (the same wrong thing can 
happen in the current implementation).

Do you have any suggestion on how to work around this limitation using the 
current implementation of python launcher? Is there any way to use 
#!/usr/bin/env python3 across both Windows and non-Windows?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Paul Moore]

Paul Moore  added the comment:

See PEP 486 (https://www.python.org/dev/peps/pep-0486/). This is intended 
behaviour, as it is assumed that explicitly specifying a Python version means 
that the user had a specific Python interpreter in mind.

There are also technical issues - if the shebang says #!/usr/bin/python2, and a 
virtual environment is active, how do we know if it's a Python 2 or 3 
environment (without launching the interpreter, which is a significant extra 
cost for the launcher)? It would definitely be wrong to launch a Python 3 
interpreter in that case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Paul Moore]

Change by Paul Moore :


--
Removed message: https://bugs.python.org/msg358060

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Paul Moore]

Paul Moore  added the comment:

See PEP 486 
(https://www.youtube.com/watch?v=Lmq13nrEFaM&list=PLuE3ABw6vYeR0gcPT6uj25ZZhmSP9vRB2&index=28&t=0s).
 This is intended behaviour, as it is assumed that explicitly specifying a 
Python version means that the user had a specific Python interpreter in mind.

There are also technical issues - if the shebang says #!/usr/bin/python2, and a 
virtual environment is active, how do we know if it's a Python 2 or 3 
environment (without launching the interpreter, which is a significant extra 
cost for the launcher)? It would definitely be wrong to launch a Python 3 
interpreter in that case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Alexandros Karypidis]

Alexandros Karypidis  added the comment:

As confirmed by debug information when setting PYLAUNCH_DEBUG=1, the shebang 
seems to be ignored when using '#!//usr/bin/env python3' but works fine when 
using '#!//usr/bin/env python'.

This is with '#!//usr/bin/env python' (proper):

(.venv) PS C:\pytest> .\script.py
launcher build: 32bit
launcher executable: Console
File 'C:\Users\karypid\AppData\Local\py.ini' non-existent
File 'C:\WINDOWS\py.ini' non-existent
Called with command line: "C:\pytest\script.py"
maybe_handle_shebang: read 167 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python
searching PATH for python executable
Python on path: C:\pytest\.venv\Scripts\python.exe
...


This is with '#!//usr/bin/env python3' (wrong):

(.venv) PS C:\pytest> .\script.py
launcher build: 32bit
launcher executable: Console
File 'C:\Users\karypid\AppData\Local\py.ini' non-existent
File 'C:\WINDOWS\py.ini' non-existent
Called with command line: "C:\pytest\script.py"
maybe_handle_shebang: read 168 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python3
locating Pythons in 64bit registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
...


As you can see in the second case even though it regognises the python3 
command, it makes no attempt to find it in the path.

Note that it appears that Windows installations only carry 'python.exe' and do 
not have a 'python3.exe' (neither in the native installation folder, nor in the 
virtual environment folder) so searching on the path would only 'work' if the 
user has copied python.exe to python3.exe in their \Scripts folder. (I 
actually tried that and it did not work).

A proper solution would probably need to search for 'python.exe' even for the 
'#!//usr/bin/env python3' shebang to detect if a virtual environment is 
present, possibly even confirming that the virtual environment is of the 
appropriate version.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Alexandros Karypidis]

Alexandros Karypidis  added the comment:

Forgot the simple script:

#!/usr/bin/env python3

import os, sys, platform

print('EXECUTABLE: ' + sys.executable)
print('PREFIX: ' + sys.prefix)
print('BASE PREFIX: ' + sys.base_prefix)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38999] Python launcher on Windows does not detect active venv

[Alexandros Karypidis]

New submission from Alexandros Karypidis :

When you activate a venv on Windows and use a shebang with a major verion 
qualifier, the python launcer does not properly detect that a venv is active 
and uses the system installation instead.

The incorrect behavior is documented in this SO question where another user has 
confirmed and suggested it is a bug: 
https://stackoverflow.com/questions/59238326

Steps to reproduce (needs script.py attached below):

1. Install Python 3.7 on Windows 10 (64 bit)
2. Run script.py you should see:
PS C:\pytest> .\script.py
EXECUTABLE: C:\Program Files\Python37\python.exe
PREFIX: C:\Program Files\Python37
BASE PREFIX: C:\Program Files\Python37

3. Create and activate a virtual environment with:
PS C:\pytest> python -m venv .venv
PS C:\pytest> . .\.venv\Scripts\Activate.ps1

4. Run script.py you should see it ignore the active virtual environment:
(.venv) PS C:\pytest> .\script.py
EXECUTABLE: C:\Program Files\Python37\python.exe
PREFIX: C:\Program Files\Python37
BASE PREFIX: C:\Program Files\Python37

I am using Windows 10 64-bit, update 1903 and Python 3.7.5-64

--
components: Windows
messages: 358017
nosy: Alexandros Karypidis, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Python launcher on Windows does not detect active venv
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com