[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18150
pull_request: https://github.com/python/cpython/pull/18792

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[miss-islington]

miss-islington  added the comment:


New changeset e696b15a62dd0c85fe6ed3c9698b5f889c0bb1b3 by Miss Islington (bot) 
in branch '3.8':
bpo-35537: Rewrite setsid test for os.posix_spawn (GH-11721)
https://github.com/python/cpython/commit/e696b15a62dd0c85fe6ed3c9698b5f889c0bb1b3


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[miss-islington]

Change by miss-islington :


--
pull_requests: +13947
pull_request: https://github.com/python/cpython/pull/14093

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 5884043252473ac733aba1d3251d4debe72511e5 by Victor Stinner in 
branch 'master':
bpo-35537: Rewrite setsid test for os.posix_spawn (GH-11721)
https://github.com/python/cpython/commit/5884043252473ac733aba1d3251d4debe72511e5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

Python 3.8 beta 1 is released with this feature. I documented the change. 
Thnaks everybody who was involved to make this possible. I close the issue. 
Open new issues for follow-up.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11263

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11264

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11586

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -10917

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -10918

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11257

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11584

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11583

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11476

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11587

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11524

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11523

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11258

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11335

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11334

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11248

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11595

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11249

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11593

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11594

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11626

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11627

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -11628

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:


New changeset d7befad328ad1a6d1f812be2bf154c1cd1e01fbc by Victor Stinner in 
branch 'master':
bpo-35537: Document posix_spawn() change in subprocess (GH-11668)
https://github.com/python/cpython/commit/d7befad328ad1a6d1f812be2bf154c1cd1e01fbc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Joannah Nanjekye]

Change by Joannah Nanjekye :


--
pull_requests: +11942

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Nina Zakharenko]

Change by Nina Zakharenko :


--
pull_requests: +11626, 11627

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Nina Zakharenko]

Change by Nina Zakharenko :


--
pull_requests: +11626, 11627, 11628

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Nina Zakharenko]

Change by Nina Zakharenko :


--
pull_requests: +11626

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 325e4bac5ab49f47ec60242d3242647605193a2e by Victor Stinner in 
branch 'master':
bpo-35537: Fix function name in os.posix_spawnp() errors (GH-11719)
https://github.com/python/cpython/commit/325e4bac5ab49f47ec60242d3242647605193a2e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11593, 11594, 11595, 11596

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11593, 11594

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11593, 11594, 11595

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11593

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 1e39b83f24ffade3e0ca1a8004b461354f64ae08 by Victor Stinner in 
branch 'master':
bpo-35537: Skip test_start_new_session() of posix_spawn (GH-11718)
https://github.com/python/cpython/commit/1e39b83f24ffade3e0ca1a8004b461354f64ae08


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11586, 11587, 11588

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11586, 11587

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11586

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11583, 11584, 11585

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11583, 11584

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11583

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 80c5dfe74b4402d0a220c9227f262ec6fde1d7fc by Victor Stinner 
(Joannah Nanjekye) in branch 'master':
bpo-35537: Add setsid parameter to os.posix_spawn() and os.posix_spawnp() 
(GH-11608)
https://github.com/python/cpython/commit/80c5dfe74b4402d0a220c9227f262ec6fde1d7fc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Giampaolo Rodola']

Change by Giampaolo Rodola' :


--
pull_requests: +11528

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +11523, 11524, 11525

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +11523, 11524

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +11523

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

> Is sys.platform equal to 'linux' on WSL? Sorry, I don't know WSL. If it's 
> equal, is it possible to explicitly exclude WSL in the subprocess test, 
> _use_posix_spawn()?

I don't have immediate access to WSL right now, but I'll try to get one and 
investigate what we have there wrt. posix_spawn() and vfork().

> So the bug only affects "QEMU User Space". I never used that before. 

Yes, I was specifically referring to QEMU user-space. One use case that heavily 
relies on it is Tizen. Its packages are built in a chroot jail containing the 
build environment with binaries native to the target architecture, making an 
illusion that packages are built on the target system and are not 
cross-compiled. So the binaries are run under QEMU user-space emulation. But in 
reality, because of unacceptable performance of binary translation, many 
frequently-used binaries, like coreutils and compilers, are replaced with 
host-native binaries in a way transparent to the build system (so it has no 
idea whether it runs host-native or target-native binaries).

> Does the difference matters? The bug only occurs in some very specific cases:

> * WSL
> * QEMU User Emulation

> Are these use cases common enough to block the whole idea of using 
> posix_spawn() on Linux. I don't think so. I really want to use posix_spawn() 
> for best performances! Moreover, I expect that glibc implementation of 
> posix_spawn() is safer than Python _posixsubprocess. The glibc has access to 
> low-level stuff like it's internal signals, cancellation points, etc. 
> _posixsubprocess is more generic and doesn't worry about such low-level 
> stuff, whereas they might cause bad surprised.

It's true that a C library is in a better position to implement something like 
posix_spawn(), but I still think that vfork()/exec() is worth to consider at 
least on Linux. See bpo-35823, which should also work under QEMU user-mode and 
WSL (but needs testing).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

I wrote PR 11668 to propose to *document* the subtle behavior change when 
posix_spawn() is used on QEMU User Emulation and WSL platforms, rather than 
trying to opt-out for posix_spawn() on these platforms.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11476, 11477

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11476

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

I discussed the following issue with Florian Weimer (who works on the glibc):
---
$ cat test.c
#include 

int main(int argc, char *argv[], char *envp[]) {
return posix_spawn(0, "non-existing", 0, 0, argv, envp);
}

$ aarch64-linux-gnu-gcc -static test.c
$ qemu-aarch64 ./a.out
$ echo $?
0
---

While the parent gets a "success", in subprocess we get the pid and later read 
the exit status of the child process. Even if posix_spawn() doesn't report a 
failure, waitpid() will still report a failure. It's a subtle behavior change. 
I'm not sure yet if it's acceptable for Python in term of semantics:

* without posix_spawn(), subprocess.Popen constructor raises FileNotFoundError: 
the parent knows that the execution failed because the program doesn't exist
* with posix_spawn() with the vfork bug, subprocess.Popen succeed but 
Popen.wait() returns something like exit code 127. The parent doesn't know if 
the child process explcitly used exit(127) or if execv() failed.

Does the difference matters? The bug only occurs in some very specific cases:

* WSL
* QEMU User Emulation

Are these use cases common enough to block the whole idea of using 
posix_spawn() on Linux. I don't think so. I really want to use posix_spawn() 
for best performances! Moreover, I expect that glibc implementation of 
posix_spawn() is safer than Python _posixsubprocess. The glibc has access to 
low-level stuff like it's internal signals, cancellation points, etc. 
_posixsubprocess is more generic and doesn't worry about such low-level stuff, 
whereas they might cause bad surprised.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

> Another problem with posix_spawn() on glibc: it doesn't report errors to the 
> parent process when run under QEMU user-space emulation and Windows Subsystem 
> for Linux. This is because starting with commit [1] (glibc 2.25) 
> posix_spawn()  relies on address space sharing semantics of clone(CLONE_VM) 
> to report errors, but it's not implemented in QEMU and WSL, so 
> clone(CLONE_VM) and vfork() behave like fork(). See also [2], [3].

Oh, you know a lot of stuff about vfork and posix_spawn, you are impressive :-)

Is sys.platform equal to 'linux' on WSL? Sorry, I don't know WSL. If it's 
equal, is it possible to explicitly exclude WSL in the subprocess test, 
_use_posix_spawn()?

For QEMU, I was very surprised that a full VM wouldn't implement vfork. So I 
tried Fedora Rawhide and I didn't notice the bug that you described.

---
$ cat test.c
#include 

int main(int argc, char *argv[], char *envp[]) {
return posix_spawn(0, "non-existing", 0, 0, argv, envp);
}

$ gcc test.c
$ ./a.out
$ echo $?
2
---

In the VM, I get exit status 2 as expected.

You wrote:

---
$ aarch64-linux-gnu-gcc -static test.c
$ qemu-aarch64 ./a.out
$ echo $?
0
---

So the bug only affects "QEMU User Space". I never used that before. I 
understand that it's a bug in the glibc and in "QEMU User Emulation" which 
doesn't implement vfork "properly".


> There is no problem with musl (it doesn't rely on address space sharing).

I'm not interested to support musl at this point because I don't see any 
obvious way to detect that we are running musl nor get the musl version.

The history of this issue shows the posix_spawn() only works in some very 
specific cases, so we have to be careful and only opt-in for specific libc, on 
specific platform with specific libc version (read at runtime if possible).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

Another problem with posix_spawn() on glibc: it doesn't report errors to the 
parent process when run under QEMU user-space emulation and Windows Subsystem 
for Linux. This is because starting with commit [1] (glibc 2.25) posix_spawn()  
relies on address space sharing semantics of clone(CLONE_VM) to report errors, 
but it's not implemented in QEMU and WSL, so clone(CLONE_VM) and vfork() behave 
like fork(). See also [2], [3].

This can be easily demonstrated:
$ cat test.c
#include 

int main(int argc, char *argv[], char *envp[]) {
return posix_spawn(0, "non-existing", 0, 0, argv, envp);
}

$ gcc test.c
$ ./a.out
$ echo $?
2
$ aarch64-linux-gnu-gcc -static test.c
$ qemu-aarch64 ./a.out
$ echo $?
0

There is no problem with musl (it doesn't rely on address space sharing).

[1] 
https://sourceware.org/git/?p=glibc.git;a=commit;h=4b4d4056bb154603f36c6f8845757c1012758158
[2] https://bugs.launchpad.net/qemu/+bug/1673976
[3] https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg04890.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:


New changeset f6243ac1e4828299fe5a8e943d7bd41cab1f34cd by Victor Stinner in 
branch 'master':
bpo-35537: subprocess can use posix_spawn with pipes (GH-11575)
https://github.com/python/cpython/commit/f6243ac1e4828299fe5a8e943d7bd41cab1f34cd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

>> * pass_fds: there is not API to mark a fd as inheritable (clear O_CLOEXEC 
>> flag)

> POSIX has a bug for this [5]. It's marked fixed, but the current POSIX docs 
> doesn't reflect the changes. The idea is to make 
> posix_spawn_file_actions_adddup2() clear O_CLOEXEC if the source and the 
> destination are the same (unlike dup2(), which would do nothing). musl has 
> already implemented the new behavior [6], but glibc hasn't.

Update: glibc has implemented it for 2.29 
(https://sourceware.org/bugzilla/show_bug.cgi?id=23640).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Joannah Nanjekye]

Change by Joannah Nanjekye :


--
pull_requests: +11334, 11335

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Joannah Nanjekye]

Change by Joannah Nanjekye :


--
pull_requests: +11334, 11335, 11336

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Joannah Nanjekye]

Change by Joannah Nanjekye :


--
pull_requests: +11334

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

> It should be compared to the current code. Currently, _posixsubprocess uses a 
> loop calling execv(). I don't think that calling posix_spawn() in a loop 
> until one doesn't fail is more inefficient.

> The worst case would be when applying process attributes and run file actions 
> would dominate performances... I don't think that such case exists. Syscalls 
> like dup2() and close() should be way faster than the final successful 
> execv() in the overall posix_spawn() call. I'm talking about the case when 
> the program exists.

I had vfork() used internally by posix_spawn() in mind when I considered 
repeatedly calling it "prohibitively expensive". While vfork() is much cheaper 
than fork(), it doesn't mean that its performance is comparable to dup2() and 
close(). But on systems where posix_spawn is a syscall the overhead could 
indeed be lesser. Anyway, it should be measured.

>> Iterate over PATH entries and perform a quick check for common exec errors 
>> (ENOENT, ENOTDIR, EACCES) manually (e.g. by stat()).

> I dislike this option. There is a high risk of race condition if the program 
> is created, deleted or modified between the check and exec. It can cause 
> subtle bugs, or worse: security vulnerabilities. IMHO the only valid check, 
> to test if a program exists, is to call exec().

While exec() is of course cleaner, IMHO races don't matter in this case. If our 
stat()-like check fails, we could as well imagine that it is exec() that failed 
(while doing the same checks as our stat()), so it doesn't matter what happens 
with the tested entry afterwards. If the check succeeds, we simply call 
posix_spawn() that will perform the same checks again. If any race condition 
occurs, the problem will be detected by posix_spawn(). 

> Alexey: Do you want to work on a PR to reimplement the "executable_list" and 
> loop used by subprocess with _posixsubproces?

I'm currently focused on researching whether it's possible to use vfork()-like 
approach instead of fork() on Linux, so if anyone wants to implement the PR you 
suggested, feel free to do it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

Thank you for the answers, Kyle!

> I'll be preparing a patch for our posix_spawn's signal handling.

Great!

> My mistake in my setuid assessment was pointed out to me- it doesn't seem 
> like a highly likely attack vector, but it is indeed possible.

The specific scenario with non-synchronized posix_spawn/setuid is certainly not 
a good practice and could probably be considered an application bug (such 
application effectively spawns a child with "random" privileges -- depending on 
whether setuid() completed before or after vfork()).

That said, in Linux C libraries protection from that scenario is usually 
achieved automatically if all signals are blocked before vfork(). This is the 
result of a Linux-specific detail: at syscall level, setuid() affects a single 
thread only, but setuid() library function must affect the process as a whole. 
This forces C libraries to signal all threads when setuid() is called and wait 
until all threads perform setuid() syscall. This waiting can't complete until 
vfork() returns (because signals are blocked), so setuid() is effectively 
postponed. I don't know how setuid() behaves on FreeBSD (so the above may be 
not applicable at all).

> If the child errored out, they will indeed be properly reapable at that point 
> due to how vfork is implemented -- any observation to the contrary is to be 
> considered a bug.

Ah, that's good, thanks for the confirmation!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

Alexey Izbyshev
> So, if we can't change os.execvp() and/or current subprocess behavior, 
> posix_spawnp seems to be ruled out.

My main motivation for using posix_spawn() is performance. An optimization 
should not justify to introduce a backward incompatible change. I agree wth 
posix_spawnp() cannot be used directly in subprocess because of that. But 
os.posix_spawnp() addition in Python 3.8 remains useful because it allows to 
use it directly (avoid subprocess).


> A naive emulation of posix_spawnp would be repeatedly calling posix_spawn for 
> each PATH entry, but that's prohibitively expensive.

It should be compared to the current code. Currently, _posixsubprocess uses a 
loop calling execv(). I don't think that calling posix_spawn() in a loop until 
one doesn't fail is more inefficient.

The worst case would be when applying process attributes and run file actions 
would dominate performances... I don't think that such case exists. Syscalls 
like dup2() and close() should be way faster than the final successful execv() 
in the overall posix_spawn() call. I'm talking about the case when the program 
exists.


> Iterate over PATH entries and perform a quick check for common exec errors 
> (ENOENT, ENOTDIR, EACCES) manually (e.g. by stat()).

I dislike this option. There is a high risk of race condition if the program is 
created, deleted or modified between the check and exec. It can cause subtle 
bugs, or worse: security vulnerabilities. IMHO the only valid check, to test if 
a program exists, is to call exec().

Alexey: Do you want to work on a PR to reimplement the "executable_list" and 
loop used by subprocess with _posixsubproces? You should keep the latest 
exception to re-raise it if no posix_spawn() successed. Don't forget to clear 
the exception on success, to not create a reference cycle:

commit acb9fa79fa6453c2bbe3ccfc9cad2837feb90093
Author: Victor Stinner 
Date:   Wed Sep 13 10:10:10 2017 -0700

bpo-31234, socket.create_connection(): Fix ref cycle (#3546)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Kyle Evans]

Kyle Evans  added the comment:

I'll be preparing a patch for our posix_spawn's signal handling.

My mistake in my setuid assessment was pointed out to me- it doesn't seem like 
a highly likely attack vector, but it is indeed possible.

If the child errored out, they will indeed be properly reapable at that point 
due to how vfork is implemented -- any observation to the contrary is to be 
considered a bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Kyle Evans]

Kyle Evans  added the comment:

On Wed, Jan 16, 2019 at 5:47 PM Alexey Izbyshev  wrote:
> > Hi,
>
> > As a disclaimer, I'm a FreeBSD developer interested in making sure we're 
> > doing the right thing here. =)
>
> > May I ask what the above assessment is based on, and specifically what we 
> > need to address?
>
> Hello, Kyle! That assessment is based on my quick and incorrect reading of 
> posix_spawn source code. I didn't notice that "error" is visible in the 
> parent due to address space sharing. Sorry for that, and thank you for the 
> correction. A proper error notification is required for posix_spawn to be 
> used in subprocess as a replacement for fork/exec, and FreeBSD does it right.

Oh, good to hear. =) koobs had pointed this thread out in hopes that we can try 
to reconcile and figure out what work needs to be done here. =)

> While we're here, would you be kind to answer several questions about 
> posix_spawn on FreeBSD?

Most definitely, if I can!

> 1) On Linux, without taking additional precautions, signals may be delivered 
> to a child process created via vfork(). If a signal handler runs in such a 
> child, it can leave traces of its work in the (shared) memory, potentially 
> surprising the parent process. To avoid this, glibc[1] and musl[2] disable 
> all signals (even signals used internally for thread cancellation) before 
> creating a child, but FreeBSD calls vfork() right away. Is this not 
> considered a problem, or is something hidden in vfork() implementation?
>

Right, after glancing over our implementation details- this is indeed a problem 
here. Our manpage indicates that we only block SIGTTOU for SIGTTIN for children 
in vfork(), and some light testing seems to indicate that's the case. Signal 
handlers are inherited from the parent, so that's less than stellar.

> 2) Another problem with vfork() is potential sharing of address space between 
> processes with different privileges (see Rich Felker's post[3] about this and 
> the previous problem). Is this a valid concern on FreeBSD?

My initial read-through of the relevant bits seem to indicate that image 
activation will cause the child process to be allocated a new process space, so 
it's looking kind of like a 'no' -- I'm outsourcing the answer to this one to 
someone more familiar with those bits, though, so I'll get back to you.


> 3) Does FreeBSD kernel guarantee that when waitpid(WNOHANG) is called at [4], 
> the child is ready for reaping? On Linux, it's not always the case[5].

I want to say vfork semantics guarantee it- we got to this point, so the child 
hit an error and _exit(127). I'm double-checking this one, though, to verify 
the child's properly marked a zombie before we could possibly reschedule the 
parent.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

> One of the issue that I have with using posix_spawn() is that the *exact* 
> behavior of subprocess is not properly defined by test_subprocess. Should we 
> more more tests, or document that the exact behavior is "an implementation 
> detail"?

Regarding using PATH from "env" instead of parent's environment, it may be 
considered documented because subprocess docs refer to os.execvp(), where it's 
explicitly documented:

"""
The variants which include a “p” near the end (execlp(), execlpe(), execvp(), 
and execvpe()) will use the PATH environment variable to locate the program 
file. When the environment is being replaced (using one of the exec*e variants, 
discussed in the next paragraph), the new environment is used as the source of 
the PATH variable.
"""

The problem is that it differs from execvp() in libc (and POSIX), so I would 
consider such behavior a bug if it wasn't so old to become a feature. Thanks to 
Victor for noticing that, I missed it.

So, if we can't change os.execvp() and/or current subprocess behavior, 
posix_spawnp seems to be ruled out. (I don't consider temporarily changing the 
parent environment as a solution). A naive emulation of posix_spawnp would be 
repeatedly calling posix_spawn for each PATH entry, but that's prohibitively 
expensive. Could we use a following hybrid approach instead?

Iterate over PATH entries and perform a quick check for common exec errors 
(ENOENT, ENOTDIR, EACCES) manually (e.g. by stat()). If the check fails, exec 
would also fail so just skip the entry. (It's important not to get false 
negatives here, but the child process would have the same privileges as the 
parent since we don't use POSIX_SPAWN_RESETIDS, so I can't think of one). 
Otherwise, call posix_spawn with the absolute path. If it fails, skip the entry.

Looks ugly, but are there other problems? This would seem to work just as well 
as posix_spawnp in the common case, but in the worst (contrived) case it would 
be equivalent to calling posix_spawn for each PATH entry.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

> Hi,

> As a disclaimer, I'm a FreeBSD developer interested in making sure we're 
> doing the right thing here. =)

> May I ask what the above assessment is based on, and specifically what we 
> need to address?

Hello, Kyle! That assessment is based on my quick and incorrect reading of 
posix_spawn source code. I didn't notice that "error" is visible in the parent 
due to address space sharing. Sorry for that, and thank you for the correction. 
A proper error notification is required for posix_spawn to be used in 
subprocess as a replacement for fork/exec, and FreeBSD does it right.

While we're here, would you be kind to answer several questions about 
posix_spawn on FreeBSD?

1) On Linux, without taking additional precautions, signals may be delivered to 
a child process created via vfork(). If a signal handler runs in such a child, 
it can leave traces of its work in the (shared) memory, potentially surprising 
the parent process. To avoid this, glibc[1] and musl[2] disable all signals 
(even signals used internally for thread cancellation) before creating a child, 
but FreeBSD calls vfork() right away. Is this not considered a problem, or is 
something hidden in vfork() implementation?

2) Another problem with vfork() is potential sharing of address space between 
processes with different privileges (see Rich Felker's post[3] about this and 
the previous problem). Is this a valid concern on FreeBSD?

3) Does FreeBSD kernel guarantee that when waitpid(WNOHANG) is called at [4], 
the child is ready for reaping? On Linux, it's not always the case[5].

[1] 
https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/spawni.c;h=353bcf5b333457d191320e358d35775a2e9b319b;hb=HEAD#l372
[2] 
http://git.musl-libc.org/cgit/musl/tree/src/process/posix_spawn.c?id=5ce3737931bb411a8d167356d4d0287b53b0cbdc#n171
[3] https://ewontfix.com/7
[4] 
https://svnweb.freebsd.org/base/head/lib/libc/gen/posix_spawn.c?view=markup#l229
[5] 
https://sourceware.org/git/?p=glibc.git;a=commit;h=aa95a2414e4f664ca740ad5f4a72d9145abbd426

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 8c349565e8a442e17f1a954d1a9996847749d778 by Victor Stinner in 
branch 'master':
Revert "bpo-35537: subprocess can now use os.posix_spawnp (GH-11579)" (GH-11582)
https://github.com/python/cpython/commit/8c349565e8a442e17f1a954d1a9996847749d778


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> One of the issue that I have with using posix_spawn() is that the *exact* 
> behavior of subprocess is not properly defined by test_subprocess. Should we 
> more more tests, or document that the exact behavior is "an implementation 
> detail"? I guess that the best for users is get the same behavior on all 
> platforms, but can we really warranty that? Python rely on the operating 
> system and the libc, but each platform has subtle behavior differneces. 
> Handling time and date is a good example: strptime() and strftime() have big 
> differences between platforms. posix_spawn() is another example where the 
> implementation is very different depending on the platform.

I don't think we can get the same behaviour in all platforms, but I would want 
to know what Gregory P. Smith thinks about this potential divergence in 
behaviour and what are the guarantees that posix_subprocess should maintain.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

One of the issue that I have with using posix_spawn() is that the *exact* 
behavior of subprocess is not properly defined by test_subprocess. Should we 
more more tests, or document that the exact behavior is "an implementation 
detail"? I guess that the best for users is get the same behavior on all 
platforms, but can we really warranty that? Python rely on the operating system 
and the libc, but each platform has subtle behavior differneces. Handling time 
and date is a good example: strptime() and strftime() have big differences 
between platforms. posix_spawn() is another example where the implementation is 
very different depending on the platform.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

Oh... Using directly posix_spawnp() in subprocess to locate the executable in 
the PATH introduces subtle behavior changes...
https://github.com/python/cpython/pull/11579/#pullrequestreview-193261299

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11263, 11264, 11265

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11263, 11264

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11263

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 07858894689047c77f9c12ddc061d30681368d19 by Victor Stinner in 
branch 'master':
bpo-35537: subprocess can now use os.posix_spawnp (GH-11579)
https://github.com/python/cpython/commit/07858894689047c77f9c12ddc061d30681368d19


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11257

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11257, 11258

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11257, 11258, 11259

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Kyle Evans]

Kyle Evans  added the comment:

> * On FreeBSD, if setting posix_spawn() "attributes" or execute posix_spawn() 
> "file actions" fails, posix_spawn() succeed but the child process exits 
> immediately with exit code 127 without trying to call execv(). If execv() 
> fails, posix_spawn() succeed, but the child process exit with exit code 127.

Hi,

As a disclaimer, I'm a FreeBSD developer interested in making sure we're doing 
the right thing here. =)

May I ask what the above assessment is based on, and specifically what we need 
to address?

As far as I can tell, our implementation is as POSIX describes -- errors 
processing the file actions or attrs triggers a 127 exit [1][2] which get 
bubbled up via the return value to posix_spawn [3]. exec failures capture errno 
at [4] and bubble the error up to the return value of posix_spawn as well via 
[3]. POSIX explicitly does not require an implementation to use errno for this, 
only return values, and we seem to have gone the route of not using errno to 
match OpenSolaris behavior.

What do I need to do to reproduce the results for deriving the results seen in 
the above quote, so that I can fix us and we can also see this improvement?

I threw together a minimal C reproducer for posix-spawn on -current (this 
particular bit being unchanged since FreeBSD 11.x times) and was returned 
ENOENT for a bad exec and otherwise given a pid for successful exec with a 
return of 0.

[1] 
https://svnweb.freebsd.org/base/head/lib/libc/gen/posix_spawn.c?view=markup#l214
[2] 
https://svnweb.freebsd.org/base/head/lib/libc/gen/posix_spawn.c?view=markup#l219
[3] 
https://svnweb.freebsd.org/base/head/lib/libc/gen/posix_spawn.c?view=markup#l232
[4] 
https://svnweb.freebsd.org/base/head/lib/libc/gen/posix_spawn.c?view=markup#l225

--
nosy: +kevans

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

subprocess_bench_stdout.py: benchmark for PR 11575 using 
stdout=subprocess.PIPE, /usr/bin/pwd, and allocate 2 GiB of memory in the 
parent process. Result on my laptop:

Mean +- std dev: [fork_exec] 28.2 ms +- 0.3 ms -> [posix_spawn] 561 us +- 209 
us: 50.25x faster (-98%)

--
Added file: https://bugs.python.org/file48057/subprocess_bench_stdout.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11248, 11249, 11250

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11248, 11249

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +11248

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

More benchmarks.

I modified subprocess_bench.py to use:
ARGS = ["/usr/bin/python3", "-S", "-E", "-c", "pass"]
=> Mean +- std dev: [fork_exec] 34.1 ms +- 0.4 ms -> [posix_spawn] 6.85 ms +- 
0.08 ms: 4.97x faster (-80%)

Benchmark using:
ARGS = ["/usr/bin/python3", "-c", "pass"]
=> Mean +- std dev: [fork_exec] 44.5 ms +- 0.6 ms -> [posix_spawn] 17.2 ms +- 
0.2 ms: 2.58x faster (-61%)

Copy of the previous benchmark using /usr/bin/true:
Mean +- std dev: [fork_exec] 27.1 ms +- 0.4 ms -> [posix_spawn] 447 us +- 163 
us: 60.55x faster (-98%)

The benchmark is less impressive with Python which has a longer startup time (7 
to 17 ms depending on the -S option).

The speedup is between 2.6x (default) and 5.0x (-S option) faster for Python... 
to execute "pass" (do nothing).

In short, I understand that vfork removes a fixed cost of 27 ms which is the 
cost of duplicating the 2 GiB of memory pages.

The speedup depends on the memory footprint of the parent process and the 
execution time of the child process. The best speedup is when the parent is the 
largest and the child is the fastest.

--

Another benchmark, I modified subprocess_bench.py with:

BIG_ALLOC = b'x' * (10 * 1024 * 1024 * 1024)   # 10 GiB
ARGS = ["/bin/true"]

Mean +- std dev: [fork_exec] 139 ms +- 9 ms -> [posix_spawn] 420 us +- 208 us: 
331.40x faster (-100%)

Here the cost of copying 10 GiB of memory pages is around 138 ms. It's 331x 
faster ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 9daecf37a571e98aaf43a387bcc9e41a7132f477 by Victor Stinner in 
branch 'master':
bpo-35537: subprocess uses os.posix_spawn in some cases (GH-11452)
https://github.com/python/cpython/commit/9daecf37a571e98aaf43a387bcc9e41a7132f477


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

Gregory P. Smith:
"""
Thanks for all your research and reference links on this!  As a 
_posixsubprocess maintainer, I am not against either posix_spawn or vfork being 
used directly in the future when feasible.

A challenge, especially with platform specific vfork, is making sure we 
understand exactly which platforms it can work properly on and checking for 
those both at compile time _and_ runtime (running kernel version and 
potentially the runtime libc version?) so that we can only use it in situations 
we are sure it is supposed to behave as desired in.  My guiding philosophy: Be 
conservative on choosing when such a thing is safe to use.
"""

About "My guiding philosophy: Be conservative on choosing when such a thing is 
safe to use.", I modified my PR 11452 to only use posix_spawn() on a very small 
subset of platforms where we know that the implementation is safe. It's 
different than early implementations which tried to use it as soon as it's 
available.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy:  -pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

Serhiy Storchaka:
> I mean that after writing tests they can be tested manually by disabling 
> conditions for posix_spawn one by one. I.e. some tests should fail if remove 
> "stdout is None" and some tests should fail if remove "not close_fds", etc.

I made some manual tests on my PR 11452. I changed close_fds default value from 
True to False. I also modified my change to use posix_spawnp using Joannah's PR 
11554 of bpo-35674.

The following tests fail *as expected*:

* test_close_fds_when_max_fd_is_lowered
* test_exception_errpipe_normal
* test_exception_errpipe_bad_data

The 2 errpipe tests mock subprocess to inject errors in the error pipe... but 
posix_spawn() doesn't expose its private "error pipe", so the test is not 
relevant for posix_spawn().

test_close_fds_when_max_fd_is_lowered() tests close_fds=True behavior. It's 
expected that it fails.

At least, I didn't notice any bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

If someone wants to implement a runtime check for musl, here is an heuristic 
based on ldd output and libc filenames:

https://github.com/lovell/detect-libc/blob/master/lib/detect-libc.js

var GLIBC = 'glibc';
var MUSL = 'musl';

// Try ldd
var ldd = spawnSync('ldd', ['--version'], spawnOptions);
if (ldd.status === 0 && ldd.stdout.indexOf(MUSL) !== -1) {
  family = MUSL;
  version = versionFromMuslLdd(ldd.stdout);
  method = 'ldd';
} else if (ldd.status === 1 && ldd.stderr.indexOf(MUSL) !== -1) {
  family = MUSL;
  version = versionFromMuslLdd(ldd.stderr);
  method = 'ldd';
} else {
  // Try filesystem (family only)
  var lib = safeReaddirSync('/lib');
  if (lib.some(contains('-linux-gnu'))) {
family = GLIBC;
method = 'filesystem';
  } else if (lib.some(contains('libc.musl-'))) {
family = MUSL;
method = 'filesystem';
  } else if (lib.some(contains('ld-musl-'))) {
family = MUSL;
method = 'filesystem';
  } else {
var usrSbin = safeReaddirSync('/usr/sbin');
if (usrSbin.some(contains('glibc'))) {
  family = GLIBC;
  method = 'filesystem';
}
  }
}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

> FYI, I'm researching how to use vfork(), focusing on Linux, but I'll need 
> more time to study the current state of affairs of libcs.

Using os.posix_spawn() in pure Python is a first step forward :-)


> (...) There are also additional subtle issues related to signal handling (and 
> pthread cancellation in particular) which I need to study.

That's why vfork was an opt-in option in the first glibc implementation, 
whereas glibc is now able to detect when using vfork is safe or not.


> If I find vfork()-like approach feasible, I'll open a separate issue. The 
> bonus of this approach is that it doesn't depend on a particular libc, so 
> both glibc (including older versions) and musl could be supported.

I like it, but it seems to be *very* tricky to implement!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

> Until muscl decides to provide an "#ifdef __MUSL__"-like or any way that it's 
> musl, I propose to not support musl: don't use os.posix_spawn() but 
> _posixsubprocess.

FYI, I'm researching how to use vfork(), focusing on Linux, but I'll need more 
time to study the current state of affairs of libcs. Both musl and glibc don't 
use "pure" vfork() now because of perceived danger of miscompilation due to 
sharing of the stack between the parent and the child. They switched to 
clone(CLONE_VM|CLONE_VFORK), which is basically the same but allows the caller 
to provide a separate stack for the child. There are also additional subtle 
issues related to signal handling (and pthread cancellation in particular) 
which I need to study.
 
If I find vfork()-like approach feasible, I'll open a separate issue. The bonus 
of this approach is that it doesn't depend on a particular libc, so both glibc 
(including older versions) and musl could be supported.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

> https://wiki.musl-libc.org/faq.html

"""
Q: Why is there no __MUSL__ macro?

It’s a bug to assume a certain implementation has particular properties rather 
than testing. So far, every time somebody’s asked for this with a particular 
usage case in mind, the usage case was badly wrong, and would have broken 
support for the next release of musl. The official explanation: 
http://openwall.com/lists/musl/2013/03/29/13
"""

IMHO that's wrong. A software like Python heavily rely on the *exact* 
implementation of a libc.

https://github.com/python/cpython/pull/9224/files looks like a coarse heuristic 
to detect musl for example.

Until muscl decides to provide an "#ifdef __MUSL__"-like or any way that it's 
musl, I propose to not support musl: don't use os.posix_spawn() but 
_posixsubprocess.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Kubilay Kocak]

Change by Kubilay Kocak :


--
nosy: +koobs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

> * On FreeBSD, if setting posix_spawn() "attributes" or execute posix_spawn() 
> "file actions" fails, posix_spawn() succeed but the child process exits 
> immediately with exit code 127 without trying to call execv(). If execv() 
> fails, posix_spawn() succeed, but the child process exit with exit code 127.
> * The worst seems to be: "In my test on Ubuntu 14.04, ./python -c "import 
> subprocess; subprocess.call(['/xxx'], close_fds=False, 
> restore_signals=False)" silently returns with zero exit code."

To clarify, in the Ubuntu example only the parent process returns with zero 
exit code. The child exits with 127, so the behavior is the same as on FreeBSD, 
as demonstrated by check_call():

$ ./python -c "import subprocess; subprocess.check_call(['/xxx'], 
close_fds=False, restore_signals=False)"
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/izbyshev/workspace/cpython/Lib/subprocess.py", line 348, in 
check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/xxx']' returned non-zero exit status 
127.

> According to Alexey Izbyshev, musl should be safe as well, but I don't know 
> how to test musl on my Fedora, nor how to check if Python is linked to musl, 
> nor what is the minimum musl version which is safe.

musl doesn't have an equivalent of __GLIBC__ macro by design as explained in 
its FAQ [1], and the same FAQ suggests to use something like "#ifndef __GLIBC__ 
portable_code() #else glibc_code()". So far musl-related fixes followed this 
advice (see PR 9288, PR 9224), but in this case it doesn't seem like a good 
idea to me. POSIX allows asynchronous error notification, so "portable" code 
shouldn't make assumptions about that, and technically old glibc and FreeBSD 
are conforming implementations, but not suitable as a replacement for 
_posixsubprocess.

Maybe we could use configure-time musl detection instead? It seems like 
config.guess has some support for musl[2], but it uses "ldd" from PATH and 
hence appears to work out-of-the-box only on musl-based distros like Alpine. 
See also [3].

Regarding the minimum musl version, in this case it's not important since the 
relevant change was committed in 2013[4], and given that musl is relatively 
young, such old versions shouldn't have users now. 

[1] https://wiki.musl-libc.org/faq.html
[2] 
https://github.com/python/cpython/blob/5bb146aaea1484bcc117ab6cb38dda39ceb5df0f/config.guess#L154
[3] https://github.com/RcppCore/Rcpp/pull/449
[4] 
https://git.musl-libc.org/cgit/musl/commit/?id=fb6b159d9ec7cf1e037daa974eeeacf3c8b3b3f1

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

I created bpo-35674: "Expose os.posix_spawnp()" to later support executable 
with no directory in subprocess.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

"The native spawn(2) system introduced in Oracle Solaris 11.4 shares a lot of 
code with the forkx(2) and execve(2)."
https://blogs.oracle.com/solaris/posix_spawn-as-an-actual-system-call

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor  added the comment:

I wrote PR 11452 which is based on PR 11242 but adds support for 'env' and 
'restore_signals' parameters and checks the operating system and libc version 
to decide if posix_spawn() can be used by subprocess.

In my implementation, posix_spawn() is only used on macOS and glibc 2.26 or 
newer (and glibc 2.24 and newer on Linux).

According to Alexey Izbyshev, musl should be safe as well, but I don't know how 
to test musl on my Fedora, nor how to check if Python is linked to musl, nor 
what is the minimum musl version which is safe.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +10917
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com