I'd like to propose a new function "open_noinherit" or maybe even a new mode flag "n" for the builtin "open" (see footnote for the names).
The new function should work exactly like the builtin "open", with one difference: The open file is not inherited to any child processes (whereas files opened with "open" will be inherited). The new function can be implemented (basically) using os.O_NOINHERIT on MS Windows resp. fcntl / FD_CLOEXEC on Posix. I will post a working Python implementation next week. There are five reasons for the proposal: 1) The builtin "open" causes unexpected problems in conjunction with subprocesses, in particular in multi-threaded programs. It can cause file permission errors in the subprocess or in the current process. On Microsoft Windows, some of the possible file permission errors are not documented by Microsoft (thus very few programs written for Windows will react properly). 2) Inheriting open file handles to subprocesses is a security risk. 3) For the developer, finding "cause and effect" is *very* hard, in particular in multi-threaded programs, when the errors occur only in race-conditions. 4) The problems arise in some of the standard library modules as well, i.e. shutil.filecopy. 5) Very few developers are aware of the possible problems. As a work-around, one can replace open with os.fdopen (os.open (..., + os.O_NOINHERIT), ... ) on Windows, but that's really ugly, hard to read, may raise a different exception than open (IOError instead of OSError), and needs careful work to take platform-specific code into account Here is a single-threaded example to demonstrate the effect: import os import subprocess outf = open ("blah.tmp", "wt") subprocess.Popen("notepad.exe") # or whatever program you like, but # It must be a program that does not exit immediately! # Now the subprocess has inherited the open file handle # We can still write: outf.write ("Hello world!\n") outf.close() # But we can not rename the file (at least on Windows) os.rename ("blah.tmp", "blah.txt") # this fails with OSError: [Errno 13] Permission denied # Similar problems with other file operations on non-Windows platforms. Ok, in this little program one can see what is going wrong easily. But what if the subprocess exits very quickly? Then perhaps you see the OSError, perhaps not - depending on the process scheduler of your operation system. In a commercial multi-theaded daemon application, the error only occured under heavy load and was hard to reproduce - and it was even harder to find the cause. That's because cause and effect were in two different threads in two completely different parts of the program: - Thread A opens a file and starts to write data - Thread B starts a subprocess (which inherits the file handle from thread A!) - Thread A continues writing to the file and closes it. - And now it's a race condition: - a) Thread A wants to rename the file - b) the subprocess exits. If a) is first: Error, if b) is first: no error. To make things more complicated, even two subprocesses can disturb each other. The new function should be implemented in C ideally, because the GIL could prevent a thread-switch between os.open and the fcntl.F_SETFD call. Note that the problem described here arises not only for files, but for sockets as well. See bug 1222790: SimpleXMLRPCServer does not set FD_CLOEXEC Once there is an easy-to-use, platform-independent, documented builtin "open_noinherit" (or a new mode flag for "open"), the standard library should be considered. For each occurence of "open" or "file", it should be considered if it necessary to inherit the file to subprocesses. If not, it should be replaced with open_noinherit. One example is shutil.filecopy, where open_noiherit should be used instead of open. The socket module is another candidate, I think - but I'm not sure about that. A nice effect of using "open_noinherit" is that - in many cases - one no longer needs to speficy close_fds = True when calling subprocess.Popen. [Note that close_fds is *terribly* slow if MAX_OPEN_FILES is "big", e.g. 800, see bug 1663329] Footnote: While writing this mail, at least 3 times I typed "nonherit" instead of "noinherit". So maybe someone can propose a better name? Or a new mode flag character could be "p" (like "private" or "protected"). Henning _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com