[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +18150 pull_request: https://github.com/python/cpython/pull/18792 ___ Python tracker ___

[issue35537] use os.posix_spawn in subprocess

miss-islington added the comment: New changeset e696b15a62dd0c85fe6ed3c9698b5f889c0bb1b3 by Miss Islington (bot) in branch '3.8': bpo-35537: Rewrite setsid test for os.posix_spawn (GH-11721) https://github.com/python/cpython/commit/e696b15a62dd0c85fe6ed3c9698b5f889c0bb1b3 -- nosy:

[issue35537] use os.posix_spawn in subprocess

Change by miss-islington : -- pull_requests: +13947 pull_request: https://github.com/python/cpython/pull/14093 ___ Python tracker ___

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: New changeset 5884043252473ac733aba1d3251d4debe72511e5 by Victor Stinner in branch 'master': bpo-35537: Rewrite setsid test for os.posix_spawn (GH-11721) https://github.com/python/cpython/commit/5884043252473ac733aba1d3251d4debe72511e5 --

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: Python 3.8 beta 1 is released with this feature. I documented the change. Thnaks everybody who was involved to make this possible. I close the issue. Open new issues for follow-up. -- resolution: -> fixed stage: patch review -> resolved status:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11263 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11264 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11586 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -10917 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -10918 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11257 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11584 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11583 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11476 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11587 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11524 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11523 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11258 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11335 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11334 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11248 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11595 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11249 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11593 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11594 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11626 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11627 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: -11628 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: New changeset d7befad328ad1a6d1f812be2bf154c1cd1e01fbc by Victor Stinner in branch 'master': bpo-35537: Document posix_spawn() change in subprocess (GH-11668) https://github.com/python/cpython/commit/d7befad328ad1a6d1f812be2bf154c1cd1e01fbc --

[issue35537] use os.posix_spawn in subprocess

Change by Joannah Nanjekye : -- pull_requests: +11942 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by Nina Zakharenko : -- pull_requests: +11626, 11627 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by Nina Zakharenko : -- pull_requests: +11626, 11627, 11628 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35537] use os.posix_spawn in subprocess

Change by Nina Zakharenko : -- pull_requests: +11626 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: New changeset 325e4bac5ab49f47ec60242d3242647605193a2e by Victor Stinner in branch 'master': bpo-35537: Fix function name in os.posix_spawnp() errors (GH-11719) https://github.com/python/cpython/commit/325e4bac5ab49f47ec60242d3242647605193a2e --

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11593, 11594, 11595, 11596 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11593, 11594 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11593, 11594, 11595 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11593 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: New changeset 1e39b83f24ffade3e0ca1a8004b461354f64ae08 by Victor Stinner in branch 'master': bpo-35537: Skip test_start_new_session() of posix_spawn (GH-11718) https://github.com/python/cpython/commit/1e39b83f24ffade3e0ca1a8004b461354f64ae08 --

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11586, 11587, 11588 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11586, 11587 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11586 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11583, 11584, 11585 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11583, 11584 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11583 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: New changeset 80c5dfe74b4402d0a220c9227f262ec6fde1d7fc by Victor Stinner (Joannah Nanjekye) in branch 'master': bpo-35537: Add setsid parameter to os.posix_spawn() and os.posix_spawnp() (GH-11608)

[issue35537] use os.posix_spawn in subprocess

Change by Giampaolo Rodola' : -- pull_requests: +11528 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by Gregory P. Smith : -- pull_requests: +11523, 11524, 11525 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35537] use os.posix_spawn in subprocess

Change by Gregory P. Smith : -- pull_requests: +11523, 11524 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by Gregory P. Smith : -- pull_requests: +11523 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Alexey Izbyshev added the comment: > Is sys.platform equal to 'linux' on WSL? Sorry, I don't know WSL. If it's > equal, is it possible to explicitly exclude WSL in the subprocess test, > _use_posix_spawn()? I don't have immediate access to WSL right now, but I'll try to get one and

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: I wrote PR 11668 to propose to *document* the subtle behavior change when posix_spawn() is used on QEMU User Emulation and WSL platforms, rather than trying to opt-out for posix_spawn() on these platforms. -- ___

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11476, 11477 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11476 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: I discussed the following issue with Florian Weimer (who works on the glibc): --- $ cat test.c #include int main(int argc, char *argv[], char *envp[]) { return posix_spawn(0, "non-existing", 0, 0, argv, envp); } $ aarch64-linux-gnu-gcc -static test.c $

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: > Another problem with posix_spawn() on glibc: it doesn't report errors to the > parent process when run under QEMU user-space emulation and Windows Subsystem > for Linux. This is because starting with commit [1] (glibc 2.25) > posix_spawn() relies on

[issue35537] use os.posix_spawn in subprocess

Alexey Izbyshev added the comment: Another problem with posix_spawn() on glibc: it doesn't report errors to the parent process when run under QEMU user-space emulation and Windows Subsystem for Linux. This is because starting with commit [1] (glibc 2.25) posix_spawn() relies on address

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: New changeset f6243ac1e4828299fe5a8e943d7bd41cab1f34cd by Victor Stinner in branch 'master': bpo-35537: subprocess can use posix_spawn with pipes (GH-11575) https://github.com/python/cpython/commit/f6243ac1e4828299fe5a8e943d7bd41cab1f34cd --

[issue35537] use os.posix_spawn in subprocess

Alexey Izbyshev added the comment: >> * pass_fds: there is not API to mark a fd as inheritable (clear O_CLOEXEC >> flag) > POSIX has a bug for this [5]. It's marked fixed, but the current POSIX docs > doesn't reflect the changes. The idea is to make > posix_spawn_file_actions_adddup2()

[issue35537] use os.posix_spawn in subprocess

Change by Joannah Nanjekye : -- pull_requests: +11334, 11335 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by Joannah Nanjekye : -- pull_requests: +11334, 11335, 11336 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35537] use os.posix_spawn in subprocess

Change by Joannah Nanjekye : -- pull_requests: +11334 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Alexey Izbyshev added the comment: > It should be compared to the current code. Currently, _posixsubprocess uses a > loop calling execv(). I don't think that calling posix_spawn() in a loop > until one doesn't fail is more inefficient. > The worst case would be when applying process

[issue35537] use os.posix_spawn in subprocess

Alexey Izbyshev added the comment: Thank you for the answers, Kyle! > I'll be preparing a patch for our posix_spawn's signal handling. Great! > My mistake in my setuid assessment was pointed out to me- it doesn't seem > like a highly likely attack vector, but it is indeed possible. The

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: Alexey Izbyshev > So, if we can't change os.execvp() and/or current subprocess behavior, > posix_spawnp seems to be ruled out. My main motivation for using posix_spawn() is performance. An optimization should not justify to introduce a backward incompatible

[issue35537] use os.posix_spawn in subprocess

Kyle Evans added the comment: I'll be preparing a patch for our posix_spawn's signal handling. My mistake in my setuid assessment was pointed out to me- it doesn't seem like a highly likely attack vector, but it is indeed possible. If the child errored out, they will indeed be properly

[issue35537] use os.posix_spawn in subprocess

Kyle Evans added the comment: On Wed, Jan 16, 2019 at 5:47 PM Alexey Izbyshev wrote: > > Hi, > > > As a disclaimer, I'm a FreeBSD developer interested in making sure we're > > doing the right thing here. =) > > > May I ask what the above assessment is based on, and specifically what we > >

[issue35537] use os.posix_spawn in subprocess

Alexey Izbyshev added the comment: > One of the issue that I have with using posix_spawn() is that the *exact* > behavior of subprocess is not properly defined by test_subprocess. Should we > more more tests, or document that the exact behavior is "an implementation > detail"? Regarding

[issue35537] use os.posix_spawn in subprocess

Alexey Izbyshev added the comment: > Hi, > As a disclaimer, I'm a FreeBSD developer interested in making sure we're > doing the right thing here. =) > May I ask what the above assessment is based on, and specifically what we > need to address? Hello, Kyle! That assessment is based on my

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: New changeset 8c349565e8a442e17f1a954d1a9996847749d778 by Victor Stinner in branch 'master': Revert "bpo-35537: subprocess can now use os.posix_spawnp (GH-11579)" (GH-11582) https://github.com/python/cpython/commit/8c349565e8a442e17f1a954d1a9996847749d778

[issue35537] use os.posix_spawn in subprocess

Pablo Galindo Salgado added the comment: > One of the issue that I have with using posix_spawn() is that the *exact* > behavior of subprocess is not properly defined by test_subprocess. Should we > more more tests, or document that the exact behavior is "an implementation > detail"? I guess

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: One of the issue that I have with using posix_spawn() is that the *exact* behavior of subprocess is not properly defined by test_subprocess. Should we more more tests, or document that the exact behavior is "an implementation detail"? I guess that the best

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: Oh... Using directly posix_spawnp() in subprocess to locate the executable in the PATH introduces subtle behavior changes... https://github.com/python/cpython/pull/11579/#pullrequestreview-193261299 -- ___ Python

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11263, 11264, 11265 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11263, 11264 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11263 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: New changeset 07858894689047c77f9c12ddc061d30681368d19 by Victor Stinner in branch 'master': bpo-35537: subprocess can now use os.posix_spawnp (GH-11579) https://github.com/python/cpython/commit/07858894689047c77f9c12ddc061d30681368d19 --

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11257 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11257, 11258 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11257, 11258, 11259 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35537] use os.posix_spawn in subprocess

Kyle Evans added the comment: > * On FreeBSD, if setting posix_spawn() "attributes" or execute posix_spawn() > "file actions" fails, posix_spawn() succeed but the child process exits > immediately with exit code 127 without trying to call execv(). If execv() > fails, posix_spawn() succeed,

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: subprocess_bench_stdout.py: benchmark for PR 11575 using stdout=subprocess.PIPE, /usr/bin/pwd, and allocate 2 GiB of memory in the parent process. Result on my laptop: Mean +- std dev: [fork_exec] 28.2 ms +- 0.3 ms -> [posix_spawn] 561 us +- 209 us: 50.25x

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11248, 11249, 11250 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11248, 11249 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- pull_requests: +11248 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: More benchmarks. I modified subprocess_bench.py to use: ARGS = ["/usr/bin/python3", "-S", "-E", "-c", "pass"] => Mean +- std dev: [fork_exec] 34.1 ms +- 0.4 ms -> [posix_spawn] 6.85 ms +- 0.08 ms: 4.97x faster (-80%) Benchmark using: ARGS =

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: New changeset 9daecf37a571e98aaf43a387bcc9e41a7132f477 by Victor Stinner in branch 'master': bpo-35537: subprocess uses os.posix_spawn in some cases (GH-11452) https://github.com/python/cpython/commit/9daecf37a571e98aaf43a387bcc9e41a7132f477 --

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: Gregory P. Smith: """ Thanks for all your research and reference links on this! As a _posixsubprocess maintainer, I am not against either posix_spawn or vfork being used directly in the future when feasible. A challenge, especially with platform specific

[issue35537] use os.posix_spawn in subprocess

Change by Antoine Pitrou : -- nosy: -pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: Serhiy Storchaka: > I mean that after writing tests they can be tested manually by disabling > conditions for posix_spawn one by one. I.e. some tests should fail if remove > "stdout is None" and some tests should fail if remove "not close_fds", etc. I made

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: If someone wants to implement a runtime check for musl, here is an heuristic based on ldd output and libc filenames: https://github.com/lovell/detect-libc/blob/master/lib/detect-libc.js var GLIBC = 'glibc'; var MUSL = 'musl'; // Try ldd var ldd =

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: > FYI, I'm researching how to use vfork(), focusing on Linux, but I'll need > more time to study the current state of affairs of libcs. Using os.posix_spawn() in pure Python is a first step forward :-) > (...) There are also additional subtle issues

[issue35537] use os.posix_spawn in subprocess

Alexey Izbyshev added the comment: > Until muscl decides to provide an "#ifdef __MUSL__"-like or any way that it's > musl, I propose to not support musl: don't use os.posix_spawn() but > _posixsubprocess. FYI, I'm researching how to use vfork(), focusing on Linux, but I'll need more time

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: > https://wiki.musl-libc.org/faq.html """ Q: Why is there no __MUSL__ macro? It’s a bug to assume a certain implementation has particular properties rather than testing. So far, every time somebody’s asked for this with a particular usage case in mind, the

[issue35537] use os.posix_spawn in subprocess

Change by Kubilay Kocak : -- nosy: +koobs ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35537] use os.posix_spawn in subprocess

Alexey Izbyshev added the comment: > * On FreeBSD, if setting posix_spawn() "attributes" or execute posix_spawn() > "file actions" fails, posix_spawn() succeed but the child process exits > immediately with exit code 127 without trying to call execv(). If execv() > fails, posix_spawn()

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: I created bpo-35674: "Expose os.posix_spawnp()" to later support executable with no directory in subprocess. -- ___ Python tracker ___

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: "The native spawn(2) system introduced in Oracle Solaris 11.4 shares a lot of code with the forkx(2) and execve(2)." https://blogs.oracle.com/solaris/posix_spawn-as-an-actual-system-call -- ___ Python tracker

[issue35537] use os.posix_spawn in subprocess

STINNER Victor added the comment: I wrote PR 11452 which is based on PR 11242 but adds support for 'env' and 'restore_signals' parameters and checks the operating system and libc version to decide if posix_spawn() can be used by subprocess. In my implementation, posix_spawn() is only used

[issue35537] use os.posix_spawn in subprocess

Change by STINNER Victor : -- keywords: +patch pull_requests: +10917 stage: -> patch review ___ Python tracker ___ ___

  1   2   >