[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Change by Steve Dower : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

miss-islington added the comment: New changeset 6a9c0fca3f2f93681468b51929472f4433753f25 by Miss Islington (bot) in branch '3.7': bpo-35797: Fix default executable used by the multiprocessing module (GH-11676) https://github.com/python/cpython/commit/6a9c0fca3f2f93681468b51929472f4433753f25

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Steve Dower added the comment: I realised while doing the fix that changing sys.executable to point to the "real" python.exe would break scenarios that involve generating scripts. All of those have been relying on sys.executable launching the venv, which would break if we changed it there.

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Change by miss-islington : -- pull_requests: +11503, 11504, 11505 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Change by miss-islington : -- pull_requests: +11503 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Steve Dower added the comment: New changeset 4e02f8f8b4baab63f927cfd87b401200ba2969e9 by Steve Dower in branch 'master': bpo-35797: Fix default executable used by the multiprocessing module (GH-11676) https://github.com/python/cpython/commit/4e02f8f8b4baab63f927cfd87b401200ba2969e9

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Change by miss-islington : -- pull_requests: +11503, 11504 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Change by Steve Dower : -- keywords: +patch, patch, patch pull_requests: +11497, 11498, 11499 stage: test needed -> patch review ___ Python tracker ___

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Change by Steve Dower : -- keywords: +patch, patch pull_requests: +11497, 11498 stage: test needed -> patch review ___ Python tracker ___

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Change by Steve Dower : -- keywords: +patch pull_requests: +11497 stage: test needed -> patch review ___ Python tracker ___ ___

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Steve Dower added the comment: > How about making the py[w].exe launcher unset __PYVENV_LAUNCHER__ whenever it > runs a registered version explicitly? I think this is a good idea, though it can be unset unconditionally. If we're in a virtual environment, then it will run the intermediate

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Eryk Sun added the comment: > The current solution is the simplest one that ensures the most > compatibility for the least amount of risk, even though that > was not zero risk, as we've seen. How about making the py[w].exe launcher unset __PYVENV_LAUNCHER__ whenever it runs a registered

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Steve Dower added the comment: > I would prefer a -X command-line option, such as "VIRTUAL_ENV". > multiprocessing could propagate this option to worker processes. Right, so would I, but it needs to be propagated basically everywhere (unless "everywhere" all defers through multiprocessing,

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Eryk Sun added the comment: multiprocessing could be redesigned to make the child process responsible for duplicating all handles from its logical parent (by PID passed on the command line -- regardless of whatever its real parent is). That avoids the problem of the parent mistakenly

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Steve Dower added the comment: > So I think "breaking if environment variables are not inherited" is a less > severe failure mode than this issue is. ISTR that having sys.executable point to a path outside of sys.prefix breaks the site module in some way, so that would move the fix over

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Antoine Pitrou added the comment: Hmm, if multiprocessing exhibits the problem, I wouldn't be surprised if some third-party libraries also do. I think `sys.executable` should really point to the proper executable. > But changing sys.executable to _not_ be the venv one will break anyone who

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Steve Dower added the comment: The first idea makes sense, but because of how we've already architected things (and the direction we're trying to rearchitect things) it isn't really that feasible. The second idea could be good. It isn't that hard to make globally named handles that can be

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Christian Ullrich added the comment: I have two ideas, but no idea if either is remotely feasible: 1. Replicate the launcher's selection logic in multiprocessing and avoid the intermediate step via sys.executable. Perhaps put it into pythonXX.dll or export it from the launcher

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Steve Dower added the comment: > We need a way to allow multiprocessing to spawn the real python.exe instead > of the launcher executable that's set as sys.executable. Got to a computer and had just reached the same conclusion. Given the environment is inherited, it's easy to do: >>>

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Eryk Sun added the comment: The launchers have inheritance enabled (for the duplicated standard handles, which should be made inheritable via SetHandleInformation instead, but that's a separate issue). However, multiprocessing doesn't use inheritable handles. Spawning processes with

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Steve Dower added the comment: You can't create symlinks on Windows without additional user permissions, and the old method of copying most of the binaries was slow and made DLL hell worse, as well as simply not working with the Windows Store model that does not let random executables load

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Antoine Pitrou added the comment: Hmm... Out of curiosity, why is the venv's python.exe not simply a symlink to the original python.exe? What is the point of going through py.exe here? Also, is this a regression? Otherwise I don't think it has to be a release blocker. --

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

Steve Dower added the comment: Good catch! I'm surprised we don't have any tests for this, but I guess we don't really create any virtual environments in our test suite. A shame nobody hit it during RC. I don't actually know the best fix for this. The venv python.exe script is the same as

[issue35797] concurrent.futures.ProcessPoolExecutor does not work in venv on Windows

New submission from Christian Ullrich : Using concurrent.futures.ProcessPoolExecutor on Windows fails immediately with a lot of exceptions of the "access denied", "file not found", and "invalid handle" varieties. Running the script that creates the ProcessPoolExecutor from the main