[issue16850] Atomic open + close-and-exec
Charles-François Natali added the comment: Windows provides O_NOINHERIT (_O_NOINHERIT) flag which has a similar purpose. ... and even then, many Unices don't support it. Yes, but I bet that more and more OSes will support it :-) For example, it looks like O_CLOEXEC is part of the POSIX standard 2008: Hum, OK, but many operating systems don't support it to this day. So if we expose it and the underlying operating system doesn't support it, do you want to fallback to fcntl (which wouldb't be atomic anymore, but let's pretend the GIL protection is enough). Also, I'm not sure exactly if this flag is useful enough to be exposed: there are many flags that can be passed when opening a file: O_DIRECT, O_SYNC, O_NONBLOCK, O_DSYNC... Amusingly, Java exposes some of them (but not O_CLOEXEC): http://docs.oracle.com/javase/7/docs/api/java/nio/file/StandardOpenOption.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: So if we expose it and the underlying operating system doesn't support it, do you want to fallback to fcntl (which wouldb't be atomic anymore, but let's pretend the GIL protection is enough). At the beginning, I was convinced that the atomicity was important than the portability. But after having read that even fopen() uses a fallback, it is maybe much more convinient to have a fallback. For example, it would be annoying that a program works on Linux 2.6.23, but not on Linux 2.6.22 whereas the atomicity is not a must-have. Said differently: the manual fallback described in msg178957 now seems annoying to me :-) So let's say that a fallback is preferable to improve the portability, but that open(name, e) would still fail with NotImplementedError if O_CLOEXEC, O_NOINHERIT and fcntl(FD_CLOEXEC) are all missing on a platform. I guess that all platform provide at least one flag/function. You already implemented a similar fallback for subprocess: use pipe2(O_CLOEXEC) if available, or fallback to pipe()+fcntl(FD_CLOEXEC). I'm not sure exactly if this flag is useful enough to be exposed I would like to benefit of the atomicity feature of O_CLOEXEC, without having to implement myself all the tricky things to check if the platform supports it or not. O_CLOEXEC solves for example a race condition in tempfile._mkstemp_inner(): fd = _os.open(file, flags, 0o600) _set_cloexec(fd) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Charles-François Natali added the comment: O_CLOEXEC solves for example a race condition in tempfile._mkstemp_inner(): fd = _os.open(file, flags, 0o600) _set_cloexec(fd) Hum... diff --git a/Lib/tempfile.py b/Lib/tempfile.py --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -57,6 +57,8 @@ _allocate_lock = _thread.allocate_lock _text_openflags = _os.O_RDWR | _os.O_CREAT | _os.O_EXCL +if hasattr(_os, 'O_CLOEXEC'): +_text_openflags |= _os.O_CLOEXEC if hasattr(_os, 'O_NOINHERIT'): _text_openflags |= _os.O_NOINHERIT if hasattr(_os, 'O_NOFOLLOW'): We should probably add this to 3.3 and default (IIRC O_CLOEXEC was added to the os module in 3.3). Also, I think O_NOFOLLOW is useless: if the file is created (O_EXCL|O_CREAT), then by definition it's not a symlink (juste check glibc's mkstemp() implementation, and it only passes O_CREAT|O_EXCL). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: We should probably add this to 3.3 and default (IIRC O_CLOEXEC was added to the os module in 3.3). I created the issue #16860. I just realized that tempfile doesn't use open() but os.open(), so this issue help to fix #16860. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: Here is a work-in-progress patch to test my idea: add e flag to open(). Limitations/TODO: * The unit test doesn't work on Windows (it requires fcntl) * e mode and the opener argument are exclusive: if O_CLOEXEC and O_NOINHERINT are missing, I don't see how the opener can be aware of the e flag. Or should we call fcntl(F_SETFD, flags|FD_CLOEXEC) after the opener? It would be strange: the opener is supposed to be the only one to decide how the file is opened, isn't it? * NotImplementedError is raised if O_CLOEXEC, O_NOINHERIT and fcntl() are missing I only tested my patch on Linux (3.6). -- keywords: +patch Added file: http://bugs.python.org/file28561/open_mode_e.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
New submission from STINNER Victor: Recent version on different operating systems support opening a file with close-on-exec flag set immediatly (atomic). This feature fixes a race condition when the process calls execv() between open() and fcntl() (to set the FD_CLOEXEC flag to the newly opened file). It would be nice to expose this feature in Python. The problem is the find a portable and safe way to expose the feature: neologix is against a best-effort function. For example, Linux kernel older than 2.6.22 simply ignores O_CLOEXEC flag (while the libc may expose it). The feature looks to be supported by at least: * Linux kernel = 2.6.23 * FreeBSD 8+ * Windows: _open(filename, _O_NOINHERIT). Is it supported by Windows XP and older versions? http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx See also: * Issue #12760 (closed): This issue added an x mode to open() to create a file in exclusive mode * Issue #12103: Document how to use open with os.O_CLOEXEC * Issue #12105: It was proposed to add an e mode to open() for O_CLOEXEC -- components: Interpreter Core messages: 178949 nosy: alexey-smirnov, amaury.forgeotdarc, haypo, neologix, sbt priority: normal severity: normal status: open title: Atomic open + close-and-exec versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: The problem is the find a portable and safe way to expose the feature A solution is to add a e mode to open() which would raise a NotImplementedError if the platform is not known to support this feature. For example, if the OS is linux, we would check if the kernel version is at least 2.6.23, otherwise an exception would be raised. The check (on the OS/version) would be done at the first call the function (if the e mode if used). We already have such behaviour on other functions. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Christian Heimes added the comment: You could do both: use the O_CLOEXEC flag and do a fcntl() call on POSIX. In my opinion it's enough to document that the x flag may be affected by a race condition issue on some operation systems. -- nosy: +christian.heimes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: You could do both: use the O_CLOEXEC flag and do a fcntl() call on POSIX This is the best-effort option. It was already discussed and rejected in the issue #12760. We had a similar discussion for the PEP 418 on monotonic clock. The final decision is not only provide time.monotonic() if the OS supports monotonic clock. Using an exception, a developer can develop its own best-effort function: try: fileobj = open(filename, e) except NotImplementedError: fileobj = open(filename, r) # may also fail here if the fcntl module is missing import fcntl flags = fcntl.fcntl(self.socket, fcntl.F_GETFD) fcntl.fcntl(fileobj, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC) We may expose the best-effort function somewhere else, but not in open() which must be atomic in my opinion. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Christian Heimes added the comment: Do you mean #12105? I didn't know about the ticket before. How about two options, like 'e' for guaranteed atomic CLOEXEC and 'E' for CLOEXEC with or without atomic ops? It's not much additional work and lowers the burden on the user. It's going to be hard to get a list of platforms and versions where O_CLOEXEC is supported. We should go for a white list approach and only enable it on platforms that are either documented to support it or have been tested. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Changes by Ross Lagerwall rosslagerw...@gmail.com: -- nosy: +rosslagerwall ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Antoine Pitrou added the comment: How about two options, like 'e' for guaranteed atomic CLOEXEC and 'E' for CLOEXEC with or without atomic ops? Why would you want that? Best effort is sufficient. Also, I'm not sure why e. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
Charles-François Natali added the comment: I don't comfortable exposing this. The main reason is that this flag is really non-portable. Having open() fail at runtime because the platform doesn't support it looks really wrong to me. And silently ignore it is even worse. The 'x' flag was added because it is useful, on available on all platforms (I mean, even Windows has it). Here's, it's by definition Unix-specific, and even then, many Unices don't support it. So I'm -1. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: The feature looks to be supported by at least: * FreeBSD 8+ Hum, it looks like it was only added to FreeBSD 8.3: http://www.freebsd.org/cgi/man.cgi?query=openapropos=0sektion=0manpath=FreeBSD+8.3-RELEASEarch=defaultformat=html (O_CLOEXEC doesn't appear in FreeBSD 8.2 manual page) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16850] Atomic open + close-and-exec
STINNER Victor added the comment: Also, I'm not sure why e. The choice of the e letter comes from the GNU version of fopen(). Extract of fopen manual page on Linux: e (since glibc 2.7) Open the file with the O_CLOEXEC flag. See open(2) for more information. Oh, by the way: e mode is a best-effort approach. It uses fcntl() if O_CLOEXEC is not supported by the running kernel. fopen() doesn't check the kernel version, but pass O_CLOEXEC and check (once) if FD_CLOEXEC is set: http://sourceware.org/git/?p=glibc.git;a=blob;f=libio/fileops.c;h=61b61b30d88d0fca9e6b20a2fe62a2406cc027cf;hb=HEAD#l319 It's interesting to know that the glibc chose the best-effort approach. The 'x' flag was added because it is useful, on available on all platforms (I mean, even Windows has it). Here's, it's by definition Unix-specific, ... Windows provides O_NOINHERIT (_O_NOINHERIT) flag which has a similar purpose. ... and even then, many Unices don't support it. Yes, but I bet that more and more OSes will support it :-) For example, it looks like O_CLOEXEC is part of the POSIX standard 2008: http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html (O_CLOEXEC: If set, the FD_CLOEXEC flag for the new file descriptor shall be set.) The O_CLOEXEC flag of Linux comes from QNX and BeOS which support it since many years (I don't know when, at least since 2004). So O_CLOEXEC is not really specific to Linux. -- For more information of the Linux support, this article of Ulrich Drepper contains many useful information: http://udrepper.livejournal.com/20407.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
