[issue37790] subprocess.Popen() is extremely slow (with close_fds=True which is the default) on Illumos

STINNER Victor added the comment: I merged FreeBSD changes: New changeset 162c567d164b5742c0d1f3f8bd8c8bab9c117cd0 by Victor Stinner in branch 'master': bpo-38061: os.closerange() uses closefrom() on FreeBSD (GH-19696)

[issue37790] subprocess.Popen() is extremely slow (with close_fds=True which is the default) on Illumos

Antoine Pitrou added the comment: I think we would accept a PR since it would probably be trivial, but someone has to submit it :-) -- ___ Python tracker ___

[issue37790] subprocess.Popen() is extremely slow (with close_fds=True which is the default) on Illumos

John Levon added the comment: closefrom() is on both Solaris and illumos too - and might even have originated there as an API - so if that's the issue, it should be trivially fixable -- ___ Python tracker

[issue37790] subprocess.Popen() is extremely slow (with close_fds=True which is the default) on Illumos

STINNER Victor added the comment: > We've moved illumos-gate wsdiff python tool from Python 2 to Python 3. I guess that you're talking about https://github.com/illumos/illumos-gate/blob/master/usr/src/tools/scripts/wsdiff.py This project is part of illumos: "An open-source Unix operating

[issue37790] subprocess.Popen() is extremely slow

STINNER Victor added the comment: I create bpo-38061 "FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()". -- ___ Python tracker ___

[issue37790] subprocess.Popen() is extremely slow

Nick Henderson added the comment: We have recently bumped into a similar problem. Using FreeBSD, subprocess calls were taking more than 10 times the usual time to execute after migrating to python3.6. After some digging, the default for 'close_fds' was changed to 'True'. On linux, this

[issue37790] subprocess.Popen() is extremely slow

STINNER Victor added the comment: This issue lacks a lot of information: * What is your operating system (name and version)? On Linux, what is your Linux kernel version? * Which Python version did you try? * Which command are you running? * Do you use a shell? * Do you use bytes (default) or

[issue37790] subprocess.Popen() is extremely slow

Antoine Pitrou added the comment: I don't seen any significant difference here (Ubuntu 18.04): $ time python2 sp.py 10.3433089256 real0m10,362s user0m6,565s sys 0m4,372s $ time python3 sp.py 11.746907234191895 real0m11,781s user0m7,356s sys 0m5,239s --

[issue37790] subprocess.Popen() is extremely slow

Alexander Pyhalov added the comment: Even if I use import subprocess process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,

[issue37790] subprocess.Popen() is extremely slow

Alexander Pyhalov added the comment: I couldn't reproduce entire test, as wsdiff script is rather large, but here's the simplified version. If Popen is used more often, difference is much more significant. # Using workaround $ python3.5 ~/tmp/1.py 1 10.780487537384033 # Without workaround

[issue37790] subprocess.Popen() is extremely slow

Antoine Pitrou added the comment: Can you post a simple script showcasing the performance degradation? -- nosy: +pitrou ___ Python tracker ___

[issue37790] subprocess.Popen() is extremely slow

Change by John Levon : -- nosy: +movement ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37790] subprocess.Popen() is extremely slow

Alexander Pyhalov added the comment: I've tried to rewrite subporcess.getstatusoutput() usage with subprocess.Popen() and switch to shell=False, it didn't help, so I doubti it getstatusoutput() overhead, it's Popen() issue. -- ___ Python tracker

[issue37790] subprocess.Popen() is extremely slow

Josh Rosenberg added the comment: Have you tried switching to using Popen itself (or run, which keeps it to one layer of convenience wrapping)? subprocess.getstatusoutput is three layers of wrapping (it calls check_output, which in turn calls run, which in turn calls Popen), and (unlike

[issue37790] subprocess.Popen() is extremely slow

New submission from Alexander Pyhalov : We've moved illumos-gate wsdiff python tool from Python 2 to Python 3. The tool does the following - for each file from old and new proto area compares file attributes to find differences in binary otput (spawning elfdump, dump and other utilities).