[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

STINNER Victor added the comment: test_subprocess.test_leaking_fds_on_error takes more than 5 minutes on x86 FreeBSD 7.2 build slave! This tests creates 1,024 subprocesses, and subprocess has to close 655,000 file descriptors just to create on child process (whereas there are only something l

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Ross Lagerwall added the comment: Closing this as a duplicate of #8052 -- nosy: +rosslagerwall resolution: -> duplicate status: open -> closed ___ Python tracker ___ __

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Charles-Francois Natali added the comment: > Your posix_closefrom() implementation as written today is not safe to call > between fork() and exec() due to the opendir/readdir implementation. It can > and will hang processes at unexpected times. Yeah, I remove the patch when I realized that.

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Gregory P. Smith added the comment: According to http://www.unix.com/man-page/All/3c/closefrom/ closefrom() is not async-signal-safe. :( -- ___ Python tracker ___ _

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Gregory P. Smith added the comment: There is no OS API call to provide a *safe* way to get a list of all open file descriptors as part of POSIX in general that can be called after the fork() and before the exec(). It must be async signal safe. The closefrom() call available in Solaris and re

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Changes by Charles-Francois Natali : Removed file: http://bugs.python.org/file20980/py3k_closefrom.diff ___ Python tracker ___ ___ Python-bugs

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Changes by s7v7nislands : Removed file: http://bugs.python.org/file20835/python27.patch ___ Python tracker ___ ___ Python-bugs-list mailing li

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Changes by s7v7nislands : Removed file: http://bugs.python.org/file20834/py3k.patch ___ Python tracker ___ ___ Python-bugs-list mailing list U

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Charles-Francois Natali added the comment: Attached is a new version falling back to /proc/self/fd when closefrom(2) is not available (on Unix), working on Linux. It's indeed much faster than the current approach. Note that it's only used if _posixsubprocess is not available, because in that c

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Changes by Charles-Francois Natali : Added file: http://bugs.python.org/file20980/py3k_closefrom.diff ___ Python tracker ___ ___ Python-bugs-l

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Changes by Charles-Francois Natali : Removed file: http://bugs.python.org/file20979/py3k_closefrom.diff ___ Python tracker ___ ___ Python-bugs

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Charles-Francois Natali added the comment: Attached is a patch adding os.closefrom. If closefrom(2) is available, it's used. Otherwise, two options: - if sysconf and _SC_OPEN_MAX are defined, we close each file descriptor up to _SC_OPEN_MAX - if not, we choose a default value (256), and close e

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

STINNER Victor added the comment: Benchmark extracted from #11314, Python 3.2 on Linux: subprocess("/bin/false", close_fds=True) is 22% slower than subprocess("/bin/false", close_fds=False). -- stage: -> patch review ___ Python tracker

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Antoine Pitrou added the comment: The patch is wrong, as explained by Charles-François. If someone wants subprocess to use closefrom(), a patch is required :) But I think at least a mention in the documentation is warranted. -- nosy: +gregory.p.smith, pitrou priority: normal -> high st

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

STINNER Victor added the comment: As written by Charles-Francois Natali (msg129890), we can use closefrom(). Pseudo-code: find the biggest fd than that be kept open, call closefrom(highest+1), and then use close() (os.closerange) for fd in 0..highest that have to be closed. closefrom() is av

[issue11284] slow close file descriptors in subprocess, popen2, os.popen*

Changes by Terry J. Reedy : -- nosy: +loewis stage: -> patch review title: slow close file descriptors in subprocess, popen2, os.pepen* -> slow close file descriptors in subprocess, popen2, os.popen* versions: -Python 2.5, Python 2.6 ___ Python tra