A NOTE has been added to this issue. ====================================================================== http://austingroupbugs.net/view.php?id=1016 ====================================================================== Reported By: izabera Assigned To: ====================================================================== Project: 1003.1(2013)/Issue7+TC1 Issue ID: 1016 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Isabella Organization: --- User Reference: --- Section: 2.7.2 Page Number: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_02
Line Number: last paragraph Interp Status: --- Final Accepted Text: ====================================================================== Date Submitted: 2015-12-28 13:52 UTC Last Modified: 2016-10-31 16:23 UTC ====================================================================== Summary: race condition with set -C ====================================================================== ---------------------------------------------------------------------- (0003481) geoffclare (manager) - 2016-10-31 16:23 http://austingroupbugs.net/view.php?id=1016#c3481 ---------------------------------------------------------------------- In the Oct 27 teleconference we agreed that we should change the standard to match what modern shells are doing, and I took an action to propose wording changes. However, while working on the changes, it occurred to me that there is a better solution: add an O_NOCLOBBER flag to open(). Adding this to the standard would require that it is added to an implementation first and therefore I present two options below, one which requires O_NOCLOBBER and one which allows either current behaviour or (the equivalent of) O_NOCLOBBER but encourages using O_NOCLOBBER. Rather than choose between these options when resolving this bug, we should defer the choice until the time comes to apply this bug during work on the Issue 8 drafts. If, by then, O_NOCLOBBER has been added to open() by at least one widely used implementation, we should require it (by applying option 2). The detailed changes for the two options are below, but first there are two minor editorial changes that should be made regardless of which option ends up being applied. (All page and line numbers are for the 2016 edition.) On page 2361 line 75346 section 2.7.3, change: ... called with the O_APPEND flag. to: ... called with the O_APPEND flag set. On page 2410 line 77125 section 2.14 set, change: Prevent existing files from being overwritten ... to: Prevent existing regular files from being overwritten ... <b>Option 1: encourage O_NOCLOBBER</b> On page 2361 line 75337 section 2.7.2, change:<blockquote>Output redirection using the '>' format shall fail if the <i>noclobber</i> option is set (see the description of <i>set</i> -<b>C</b>) and the file named by the expansion of <i>word</i> exists and is a regular file. Otherwise, redirection using the '>' or ">|" formats shall cause the file whose name results from the expansion of <i>word</i> to be created and opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created; otherwise, it shall be truncated to be an empty file after being opened.</blockquote> to:<blockquote> Output redirection using the '>' format shall fail if the <i>noclobber</i> option is set (see the description of <i>set</i> -<b>C</b>) and the file named by the expansion of <i>word</i> exists and is either a regular file or a symbolic link that resolves to a regular file; it may also fail if the file is a symbolic link that does not resolve to an existing file. Implementations need not perform the check for the existence and type of the file and the creation of the file if it does not exist as a single atomic operation; however, creation of the file shall be performed as if the <i>open</i>() function as defined in the System Interfaces volume of POSIX.1-2008 was called with the O_CREAT and O_EXCL flags set, except that if the file exists and is a symbolic link, the open operation need not fail unless the symbolic link resolves to an existing regular file. Performing this step atomically ensures that creation of lock files using '>' with <i>noclobber</i> set is reliable. Since the overall operation may involve a race condition, redirection failure caused by concurrent creation or removal of files (particularly non-regular files) may be diagnosed misleadingly by the shell when it reports a redirection failure. In all other cases (<i>noclobber</i> not set, redirection using '>' does not fail for the reasons stated above, or redirection using the ">|" format), output redirection shall cause the file whose name results from the expansion of <i>word</i> to be opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created as an empty file; otherwise, it shall be opened as if the <i>open</i>() function was called with the O_TRUNC flag set.</blockquote> Cross-volume change to XBD ... On page 241 line 8132 section <fcntl.h> change FUTURE DIRECTIONS from:<blockquote>None.</blockquote>to:<blockquote>A future version of this standard may add an O_NOCLOBBER file creation flag - see the FUTURE DIRECTIONS section for [xref to open()].</blockquote> Cross-volume change to XSH ... On page 1416 line 47098 section open() change FUTURE DIRECTIONS from:<blockquote>None.</blockquote>to:<blockquote>A future version of this standard may add an O_NOCLOBBER flag, specified as follows, for use by shells when the <i>noclobber</i> option is set (see [xref to XRAT C.2.7.2]): O_NOCLOBBER<blockquote>If O_CREAT and O_NOCLOBBER are set, <i>open</i>() shall fail if the file exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing <i>open</i>() naming the same filename in the same directory with O_NOCLOBBER and O_CREAT set or with O_EXCL and O_CREAT set. If O_NOCLOBBER and O_CREAT are set, and the file exists and is either a non-regular file or a symbolic link that resolves to a non-regular file, the file shall be opened as if neither flag was set. If O_NOCLOBBER and O_CREAT are set, and <i>path</i> names a symbolic link that does not resolve to an existing file, an empty file shall be created such that <i>path</i> resolves to the newly created file. If O_NOCLOBBER is set and O_CREAT is not set, the result is undefined.</blockquote></blockquote> Cross-volume change to XRAT ... On page 3736 line 128235 section C.2.7.2, change:<blockquote>There is no additional rationale provided for this section.</blockquote>to:<blockquote>Earlier versions of this standard did not require redirection using '>' when <i>noclobber</i> is set to perform the file creation step as an atomic operation. Historical shells just called <i>stat</i>() to check if a regular file existed and then called <i>creat</i>(). The operation thus involved a race condition which meant that it could not be used for reliable creation of lock files. Many shell implementations improved on this by using <i>open</i>() with the O_CREAT and O_EXCL flags set as one step in a multi-step process which still meant that an existing non-regular file (for example <b>/dev/null</b>, <b>/dev/tty</b>, or a FIFO) was opened successfully. However, the methods employed still involved a race condition and could produce misleading diagnostics if there is concurrent creation or removal of files. An ideal solution would be an O_NOCLOBBER flag for <i>open</i>() which the shell could use in order to perform the entire operation atomically, and implementations are encouraged to adopt this solution, adding the flag as described in the FUTURE DIRECTIONS sections of [xref to XSH open()], and using it in the implementation's POSIX shell and in other shells. Authors of portable shells should make use of <tt>#ifdef O_NOCLOBBER</tt> so that it is used on implementations that provide it. If O_NOCLOBBER is not used, shells can use one of the following methods:<ol><li>The "stat first" method.<ol type="a"><li>Call <i>stat</i>() and if the file exists and is a regular file, the redirection fails. Otherwise:</li> <li>Call <i>open</i>() without O_CREAT or O_TRUNC to open an existing file. If the open succeeds, use <i>fstat</i>() to check whether the opened file is a regular file. If it is, close it and fail the redirection. If it is a non-regular file, the redirection succeeds. Otherwise:</li> <li>Call <i>open</i>() with O_CREAT|O_EXCL. The redirection succeeds or fails depending on whether the open succeeds or fails.</li></ol></li> <li>The "exclusive create first" method.<ol type="a"><li>Call <i>open</i>() with O_CREAT|O_EXCL. If the open succeeds, the redirection succeeds. If the open fails with [EMFILE] or [ENFILE], use <i>stat</i>() to check whether a regular file exists; if it does, fail the redirection. Otherwise:</li> <li>Call <i>open</i>() without O_CREAT or O_TRUNC to open an existing file. If the open succeeds, use <i>fstat</i>() to check whether the opened file is a regular file. If it is, close it and fail the redirection. If it is a non-regular file, the redirection succeeds. If the second open fails, the redirection fails with a diagnostic based on the <i>errno</i> value set by the first open.</li></ol> (A minor variation of this method could also be used whereby step 2.b is only done if the <i>open</i>() in step 2.a fails with [EEXIST].)</li></ol> Method 1 is in widespread use. Method 2 has not been observed exactly as described, although an implementation which omits the <i>stat</i>() in step 2.a has been observed. Without the <i>stat</i>(), this method has a problem in that if a regular file exists but the <i>open</i>() fails with [EMFILE] or [ENFILE] instead of [EEXIST] (which is to be expected if those conditions exist, because detecting [EEXIST] is more expensive), then the shell will give an incorrect diagnostic. (Reporting that no file descriptors are available implies that a non-regular file exists, because the shell tried to open the file and it is not supposed to open an existing regular file.) A variant of method 1 which omits the initial <i>stat</i>() call has also been observed; this has the same problem with [EMFILE] and [ENFILE]. With the <i>stat</i>(), this misleading diagnostic can also happen, but only if a regular file is created in the timing window between steps 1.a and 1.b, which makes it an allowed case. (The standard allows a misleading diagnostic when there is concurrent creation or removal of files.) Both methods have cases where a misleading diagnostic is given when a non-regular file is concurrently created or removed. With method 1 it occurs if no file exists at steps 1.a and 1.b, and a non-regular file is created before step 1.c. With method 2 it occurs if a non-regular file exists at step 2.a and is removed before step 2.b. (In both cases, the diagnostic misleadingly implies that a regular file exists). Both methods differ from historical shell behavior in that the redirection fails if there is an existing symbolic link whose target does not exist, instead of the link's target being created as a regular file. The standard developers consider this to be of less importance than ensuring that the creation of lock files is reliable.</blockquote> <b>Option 2: require O_NOCLOBBER</b> On page 2361 line 75337 section 2.7.2, change:<blockquote>Output redirection using the '>' format shall fail if the <i>noclobber</i> option is set (see the description of <i>set</i> -<b>C</b>) and the file named by the expansion of <i>word</i> exists and is a regular file. Otherwise, redirection using the '>' or ">|" formats shall cause the file whose name results from the expansion of <i>word</i> to be created and opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created; otherwise, it shall be truncated to be an empty file after being opened.</blockquote>to:<blockquote>Output redirection using the '>' format shall fail if the <i>noclobber</i> option is set (see the description of <i>set</i> -<b>C</b>) and the file named by the expansion of <i>word</i> exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be performed as a single atomic operation, as if the <i>open</i>() function as defined in the System Interfaces volume of POSIX.1-2008 was called with the O_CREAT and O_NOCLOBBER flags set. Performing the operation atomically ensures that creation of lock files using '>' with <i>noclobber</i> set is reliable. In all other cases (<i>noclobber</i> not set, redirection using '>' does not fail for the reasons stated above, or redirection using the ">|" format), output redirection shall cause the file whose name results from the expansion of <i>word</i> to be opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created as an empty file; otherwise, it shall be opened as if the <i>open</i>() function was called with the O_TRUNC flag set.</blockquote> Cross-volume change to XBD ... On page 239 line 8036 section <fcntl.h> add:<blockquote>O_NOCLOBBER Prevent existing regular files from being overwritten.</blockquote> Cross-volume changes to XSH ... On page 1409 line 46805 section open(), change:<blockquote>... naming the same filename in the same directory with O_EXCL and O_CREAT set.</blockquote>to:<blockquote>... naming the same filename in the same directory with O_EXCL and O_CREAT set or with O_NOCLOBBER and O_CREAT set.</blockquote> On page 1409 line 46809 section open(), add:<blockquote>O_NOCLOBBER<blockquote>If O_CREAT and O_NOCLOBBER are set, <i>open</i>() shall fail if the file exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing <i>open</i>() naming the same filename in the same directory with O_NOCLOBBER and O_CREAT set or with O_EXCL and O_CREAT set. If O_NOCLOBBER and O_CREAT are set, and the file exists and is either a non-regular file or a symbolic link that resolves to a non-regular file, the file shall be opened as if neither flag was set. If O_NOCLOBBER and O_CREAT are set, and <i>path</i> names a symbolic link that does not resolve to an existing file, an empty file shall be created such that <i>path</i> resolves to the newly created file. If O_NOCLOBBER is set and O_CREAT is not set, the result is undefined.</blockquote></blockquote> Cross-volume change to XRAT ... On page 3736 line 128235 section C.2.7.2, change:<blockquote>There is no additional rationale provided for this section.</blockquote>to:<blockquote>Earlier versions of this standard did not require redirection using '>' when <i>noclobber</i> is set to open the file as if by calling <i>open</i>() with the O_CREAT and O_NOCLOBBER flags set. Historical shells just called <i>stat</i>() to check if a regular file existed and then called <i>creat</i>(). The operation thus involved a race condition which meant that it could not be used for reliable creation of lock files. Many shell implementations improved on this by using <i>open</i>() with the O_CREAT and O_EXCL flags set as one step in a multi-step process which still meant that an existing non-regular file (for example <b>/dev/null</b>, <b>/dev/tty</b>, or a FIFO) was opened successfully. However, the methods employed still involved a race condition and could produce misleading diagnostics if there is concurrent creation or removal of files. They also differed from historical shell behavior in that the redirection failed if there is an existing symbolic link whose target does not exist, instead of the link's target being created as a regular file. The ideal solution is to use the O_NOCLOBBER flag for <i>open</i>(), which this standard now requires.</blockquote> Issue History Date Modified Username Field Change ====================================================================== 2015-12-28 13:52 izabera New Issue 2015-12-28 13:52 izabera Name => Isabella 2015-12-28 13:52 izabera Organization => --- 2015-12-28 13:52 izabera User Reference => --- 2015-12-28 13:52 izabera Section => 2.7.2 2015-12-28 13:52 izabera Page Number => http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_02 2015-12-28 13:52 izabera Line Number => last paragraph 2015-12-31 05:58 shware_systems Note Added: 0002990 2015-12-31 12:52 jilles Note Added: 0002991 2015-12-31 14:21 shware_systems Note Added: 0002992 2015-12-31 17:05 nick Interp Status => --- 2015-12-31 17:05 nick Note Added: 0002993 2015-12-31 17:05 nick Description Updated 2016-10-20 16:40 geoffclare Note Added: 0003446 2016-10-31 16:23 geoffclare Note Added: 0003481 ======================================================================
