A NOTE has been added to this issue. ====================================================================== http://austingroupbugs.net/view.php?id=1016 ====================================================================== Reported By: izabera Assigned To: ====================================================================== Project: 1003.1(2013)/Issue7+TC1 Issue ID: 1016 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Isabella Organization: --- User Reference: --- Section: 2.7.2 Page Number: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_02
Line Number: last paragraph Interp Status: --- Final Accepted Text: ====================================================================== Date Submitted: 2015-12-28 13:52 UTC Last Modified: 2016-11-10 17:33 UTC ====================================================================== Summary: race condition with set -C ====================================================================== ---------------------------------------------------------------------- (0003485) rhansen (manager) - 2016-11-10 17:33 http://austingroupbugs.net/view.php?id=1016#c3485 ---------------------------------------------------------------------- We discussed this during the 2016-11-03 and 2016-11-10 teleconferences and came up with the following revised version of the wording from http://austingroupbugs.net/view.php?id=1016#c3481: <b>Option 1: encourage O_NOCLOBBER</b> On page 2361 line 75337 section 2.7.2, change: <blockquote>Output redirection using the '>' format shall fail if the <i>noclobber</i> option is set (see the description of <i>set</i> -<b>C</b>) and the file named by the expansion of <i>word</i> exists and is a regular file. Otherwise, redirection using the '>' or ">|" formats shall cause the file whose name results from the expansion of <i>word</i> to be created and opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created; otherwise, it shall be truncated to be an empty file after being opened.</blockquote> to: <blockquote>Output redirection using the '>' format shall fail if the <i>noclobber</i> option is set (see the description of <i>set</i> -<b>C</b>) and the file named by the expansion of <i>word</i> exists and is either a regular file or a symbolic link that resolves to a regular file; it may also fail if the file is a symbolic link that does not resolve to an existing file. The check for existence, file creation, and open operations shall be performed atomically as is done by the <i>open</i>() function as defined in the System Interfaces volume of POSIX.1-2008 when the O_CREAT and O_EXCL flags are set, except that if the file exists and is a symbolic link, the open operation need not fail with [EEXIST] unless the symbolic link resolves to an existing regular file. Performing these operations atomically ensures that the creation of lock files and unique (often temporary) files is reliable. The check for the type of the file need not be performed atomically with the check for existence, file creation, and open operations. If not, there is a potential race condition that may result in a misleading shell diagnostic message when redirection fails. See [xref to rationale] for more details. In all other cases (<i>noclobber</i> not set, redirection using '>' does not fail for the reasons stated above, or redirection using the ">|" format), output redirection shall cause the file whose name results from the expansion of <i>word</i> to be opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created as an empty file; otherwise, it shall be opened as if the <i>open</i>() function was called with the O_TRUNC flag set.</blockquote> Cross-volume change to XBD ... On page 241 line 8132 section <fcntl.h> change FUTURE DIRECTIONS from: <blockquote>None.</blockquote> to: <blockquote>A future version of this standard may add an O_NOCLOBBER file creation flag - see the FUTURE DIRECTIONS section for [xref to open()].</blockquote> Cross-volume change to XSH ... On page 1416 line 47098 section open() change FUTURE DIRECTIONS from: <blockquote>None.</blockquote> to: <blockquote>A future version of this standard may add an O_NOCLOBBER flag, specified as follows, for use by shells when the <i>noclobber</i> option is set (see [xref to XRAT C.2.7.2]): O_NOCLOBBER <blockquote>If O_CREAT and O_NOCLOBBER are set, <i>open</i>() shall fail if the file exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing <i>open</i>() naming the same filename in the same directory with O_NOCLOBBER and O_CREAT set or with O_EXCL and O_CREAT set. If O_NOCLOBBER and O_CREAT are set, and the file exists and is either a non-regular file or a symbolic link that resolves to a non-regular file, the file shall be opened as if neither flag was set. If O_NOCLOBBER and O_CREAT are set, and <i>path</i> names a symbolic link that does not resolve to an existing file, an empty file shall be created such that <i>path</i> resolves to the newly created file. If O_NOCLOBBER is set and O_CREAT is not set, the result is undefined.</blockquote> Cross-volume change to XRAT ... On page 3736 line 128235 section C.2.7.2, change: <blockquote>There is no additional rationale provided for this section.</blockquote> to: <blockquote>Earlier versions of this standard did not require redirection using '>' when <i>noclobber</i> is set to perform the file creation step as an atomic operation. Historical shells just called <i>stat</i>() to check if a regular file existed and then called <i>creat</i>(). The operation thus involved a race condition which meant that it could not be used for reliable creation of lock files. Many shell implementations improved on this by using <i>open</i>() with the O_CREAT and O_EXCL flags set as one step in a multi-step process which still meant that an existing non-regular file (for example <b>/dev/null</b>, <b>/dev/tty</b>, or a FIFO) was opened successfully. However, the methods employed still involved a race condition and could produce misleading diagnostics if there is concurrent creation or removal of files. An ideal solution would be an O_NOCLOBBER flag for <i>open</i>() which the shell could use in order to perform the entire operation atomically, and implementations are encouraged to adopt this solution, adding the flag as described in the FUTURE DIRECTIONS sections of [xref to XSH open()], and using it in the implementation's POSIX shell and in other shells. Authors of portable shells should make use of <tt>#ifdef O_NOCLOBBER</tt> so that it is used on implementations that provide it. If O_NOCLOBBER is not used, shells can use one of the following methods:<ol> <li>The "stat first" method.<ul> <li>a. Call <i>stat</i>() and if the file exists and is a regular file, the redirection fails. Otherwise:</li> <li>b. Call <i>open</i>() without O_CREAT or O_TRUNC to open an existing file. If the open succeeds, use <i>fstat</i>() to check whether the opened file is a regular file. If it is, close it and fail the redirection. If it is a non-regular file, the redirection succeeds. Otherwise:</li> <li>c. Call <i>open</i>() with O_CREAT|O_EXCL. The redirection succeeds or fails depending on whether the open succeeds or fails.</li></ul></li> <li>The "exclusive create first" method.<ul> <li>a. Call <i>open</i>() with O_CREAT|O_EXCL. If the open succeeds, the redirection succeeds. If the open fails with [EMFILE] or [ENFILE], use <i>stat</i>() to check whether a regular file exists; if it does, fail the redirection. Otherwise:</li> <li>b. Call <i>open</i>() without O_CREAT or O_TRUNC to open an existing file. If the open succeeds, use <i>fstat</i>() to check whether the opened file is a regular file. If it is, close it and fail the redirection. If it is a non-regular file, the redirection succeeds. If the second open fails, the redirection fails with a diagnostic based on the <i>errno</i> value set by the first open.</li></ul> (A minor variation of this method could also be used whereby step 2.b is only done if the <i>open</i>() in step 2.a fails with [EEXIST].)</li></ol> Method 1 is in widespread use. Method 2 has not been observed exactly as described, although an implementation which omits the <i>stat</i>() in step 2.a has been observed. Without the <i>stat</i>(), this method has a problem in that if a regular file exists but the <i>open</i>() fails with [EMFILE] or [ENFILE] instead of [EEXIST] (which is to be expected if those conditions exist, because detecting [EEXIST] is more expensive), then the shell will give an incorrect diagnostic. (Reporting that no file descriptors are available implies that a non-regular file exists, because the shell tried to open the file and it is not supposed to open an existing regular file.) A variant of method 1 which omits the initial <i>stat</i>() call has also been observed; this has the same problem with [EMFILE] and [ENFILE]. With the <i>stat</i>(), this misleading diagnostic can also happen, but only if a regular file is created in the timing window between steps 1.a and 1.b, which makes it an allowed case. (The standard allows a misleading diagnostic when there is concurrent creation or removal of files.) Both methods have cases where a misleading diagnostic is given when a non-regular file is concurrently created or removed. With method 1 it occurs if no file exists at steps 1.a and 1.b, and a non-regular file is created before step 1.c. With method 2 it occurs if a non-regular file exists at step 2.a and is removed before step 2.b. (In both cases, the diagnostic misleadingly implies that a regular file exists). Both methods differ from historical shell behavior in that the redirection fails if there is an existing symbolic link whose target does not exist, instead of the link's target being created as a regular file. The standard developers consider this to be of less importance than ensuring that the creation of lock files is reliable.</blockquote> <b>Option 2: require O_NOCLOBBER</b> On page 2361 line 75337 section 2.7.2, change: <blockquote>Output redirection using the '>' format shall fail if the <i>noclobber</i> option is set (see the description of <i>set</i> -<b>C</b>) and the file named by the expansion of <i>word</i> exists and is a regular file. Otherwise, redirection using the '>' or ">|" formats shall cause the file whose name results from the expansion of <i>word</i> to be created and opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created; otherwise, it shall be truncated to be an empty file after being opened.</blockquote> to: <blockquote>Output redirection using the '>' format shall fail if the <i>noclobber</i> option is set (see the description of <i>set</i> -<b>C</b>) and the file named by the expansion of <i>word</i> exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be performed as a single atomic operation, as if the <i>open</i>() function as defined in the System Interfaces volume of POSIX.1-2008 was called with the O_CREAT and O_NOCLOBBER flags set. Performing the operation atomically ensures that creation of lock files and unique (often temporary) files using '>' with <i>noclobber</i> set is reliable. In all other cases (<i>noclobber</i> not set, redirection using '>' does not fail for the reasons stated above, or redirection using the ">|" format), output redirection shall cause the file whose name results from the expansion of <i>word</i> to be opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created as an empty file; otherwise, it shall be opened as if the <i>open</i>() function was called with the O_TRUNC flag set.</blockquote> Cross-volume change to XBD ... On page 239 line 8036 section <fcntl.h> add: <blockquote>O_NOCLOBBER Prevent existing regular files from being overwritten.</blockquote> Cross-volume changes to XSH ... On page 1409 line 46805 section open(), change: <blockquote>... naming the same filename in the same directory with O_EXCL and O_CREAT set.</blockquote> to: <blockquote>... naming the same filename in the same directory with O_EXCL and O_CREAT set or with O_NOCLOBBER and O_CREAT set.</blockquote> On page 1409 line 46809 section open(), add: <blockquote>O_NOCLOBBER <blockquote>If O_CREAT and O_NOCLOBBER are set, <i>open</i>() shall fail if the file exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing <i>open</i>() naming the same filename in the same directory with O_NOCLOBBER and O_CREAT set or with O_EXCL and O_CREAT set. If O_NOCLOBBER and O_CREAT are set, and the file exists and is either a non-regular file or a symbolic link that resolves to a non-regular file, the file shall be opened as if neither flag was set. If O_NOCLOBBER and O_CREAT are set, and <i>path</i> names a symbolic link that does not resolve to an existing file, an empty file shall be created such that <i>path</i> resolves to the newly created file. If O_NOCLOBBER is set and O_CREAT is not set, the result is undefined.</blockquote> </blockquote> Cross-volume change to XRAT ... On page 3736 line 128235 section C.2.7.2, change: <blockquote>There is no additional rationale provided for this section.</blockquote> to: <blockquote>Earlier versions of this standard did not require redirection using '>' when <i>noclobber</i> is set to open the file as if by calling <i>open</i>() with the O_CREAT and O_NOCLOBBER flags set. Historical shells just called <i>stat</i>() to check if a regular file existed and then called <i>creat</i>(). The operation thus involved a race condition which meant that it could not be used for reliable creation of lock files. Many shell implementations improved on this by using <i>open</i>() with the O_CREAT and O_EXCL flags set as one step in a multi-step process which still meant that an existing non-regular file (for example <b>/dev/null</b>, <b>/dev/tty</b>, or a FIFO) was opened successfully. However, the methods employed still involved a race condition and could produce misleading diagnostics if there is concurrent creation or removal of files. They also differed from historical shell behavior in that the redirection failed if there is an existing symbolic link whose target does not exist, instead of the link's target being created as a regular file. The ideal solution is to use the O_NOCLOBBER flag for <i>open</i>(), which this standard now requires.</blockquote> Issue History Date Modified Username Field Change ====================================================================== 2015-12-28 13:52 izabera New Issue 2015-12-28 13:52 izabera Name => Isabella 2015-12-28 13:52 izabera Organization => --- 2015-12-28 13:52 izabera User Reference => --- 2015-12-28 13:52 izabera Section => 2.7.2 2015-12-28 13:52 izabera Page Number => http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_02 2015-12-28 13:52 izabera Line Number => last paragraph 2015-12-31 05:58 shware_systems Note Added: 0002990 2015-12-31 12:52 jilles Note Added: 0002991 2015-12-31 14:21 shware_systems Note Added: 0002992 2015-12-31 17:05 nick Interp Status => --- 2015-12-31 17:05 nick Note Added: 0002993 2015-12-31 17:05 nick Description Updated 2016-10-20 16:40 geoffclare Note Added: 0003446 2016-10-31 16:23 geoffclare Note Added: 0003481 2016-10-31 16:28 geoffclare Note Edited: 0003481 2016-10-31 16:33 geoffclare Note Edited: 0003481 2016-11-10 17:33 rhansen Note Added: 0003485 ======================================================================
