Req #35326 [Com]: Concurrency issue with recursive mkdir
Edit report at https://bugs.php.net/bug.php?id=35326&edit=1 ID: 35326 Comment by: gustavo dot straube at gmail dot com Reported by:bugs dot php dot net at chsc dot dk Summary:Concurrency issue with recursive mkdir Status: Not a bug Type: Feature/Change Request Package:*General Issues Operating System: Linux PHP Version:5.1.6 Block user comment: N Private report: N New Comment: > We don't need a locking mechanism. If the creation of a directory fails, we > just need to check if it exists anyway [...] In the application I experienced this bug, I do have a condition before the directory creation. 1. if (!is_dir($path)) 2. mkdir($path, 0755, true); If I have two concurrent process ("A" and "B") running the code in this order (process - line): A - 1 B - 1 B - 2 A - 2 When process "A" verified the directory existence, it really didn't exist. Previous Comments: [2013-10-30 08:49:24] bugs dot php dot net at chsc dot dk > To create the right locking mechanism to avoid such issue to happen We don't need a locking mechanism. If the creation of a directory fails, we just need to check if it exists anyway (i.e. it has been created by a concurrent thread). If it does, we can proceed to create the subdirectories. [2013-10-01 04:52:05] paj...@php.net Update the status. And adding a small note about "why is it not a php bug", what is described here can happen anywhere outside php as well, be a shell script, ftp/ssh, etc. [2013-10-01 04:21:25] paj...@php.net Should have been: ... and will drastically affect performance without actually solving the problem [2013-10-01 04:15:57] paj...@php.net I am not sure it is actually a PHP problem. To create the right locking mechanism to avoid such issue to happen between (almost) simultaneous requests is the application developer job, not the core. Adding such tests should be done in almost all file creation operations (and some other) and will drastically performance without actually solving the problem. [2013-09-30 10:39:45] me at vlastv dot ru Problem still exists in PHP 5.4.1. The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=35326 -- Edit this bug report at https://bugs.php.net/bug.php?id=35326&edit=1
Req #35326 [Com]: Concurrency issue with recursive mkdir
Edit report at https://bugs.php.net/bug.php?id=35326&edit=1 ID: 35326 Comment by: me at vlastv dot ru Reported by:bugs dot php dot net at chsc dot dk Summary:Concurrency issue with recursive mkdir Status: Open Type: Feature/Change Request Package:Feature/Change Request Operating System: Linux PHP Version:5.1.6 Block user comment: N Private report: N New Comment: Problem still exists in PHP 5.4.1. Previous Comments: [2013-07-23 22:55:07] gustavo dot straube at gmail dot com Problem still exists in PHP 5.3.3. [2006-10-10 07:25:52] bugs dot php dot net at chsc dot dk Problem still exists in PHP 5.1.6. [2005-11-22 13:52:24] bugs dot php dot net at chsc dot dk Yes, it also occurs with the CVS snapshot (this time I tried with the CGI version). AFAICS the problem is in php_plain_files_mkdir in main/streams/plain_wrapper.c. If php_mkdir() or VCWD_MKDIR() returns false, the method gives up instead of checking whether directory exists anyway, i.e. if it was created by another thread. [2005-11-22 12:13:10] tony2...@php.net Please try using this CVS snapshot: http://snaps.php.net/php5-latest.tar.gz For Windows: http://snaps.php.net/win32/php5-win32-latest.zip [2005-11-22 11:32:49] bugs dot php dot net at chsc dot dk Description: There appears to be a concurrency issue when mkdir is called on two separate threads with the following parameters: Thread 1: mkdir('/tmp/foo/bar/1', 0777, true); Thread 2: mkdir('/tmp/foo/bar/2', 0777, true); Sometimes one of the threads (e.g. thread 2) two fails with "Warning: mkdir(): File exists" and the directory /tmp/foo/bar/2 does not exist afterwards. What I think is happening in thread 2 is is this: PHP looks for /tmp/foo/bar/2 and sees that it does not exist. Neither does /tmp/foo/bar and /tmp/foo. So it tries to create /tmp/foo, but that directory was just created a millisecond second ago by another thread, so mkdir gives instead of just moving on to create /tmp/foo/bar and /tmp/foo/bar/2. Reproduce code: --- This code reproduces the problem in about every second attempt. It consists of two files, index.php and iframe.php. index.php: Nr. 1 Nr. 2 iframe.php: time()); $dir = '/tmp/' . $fire . '/foo/bar/foo/bar/' . $no; //sleep($no); var_dump(is_dir($dir)); echo "Making $dir "; mkdir($dir, 0777, true); var_dump(is_dir($dir)); ?> Expected result: Thread 2 should output this: bool(false) Making /tmp/1132653651/foo/bar/foo/bar/2 bool(true) Actual result: -- Thread 2 sometimes outputs this: bool(false) Making /tmp/1132655181/foo/bar/foo/bar/2 Warning: mkdir(): File exists in /home/chsc/public_html/mkdir/iframe.php on line 10 bool(false) Notice that the directory does not exist afterwards. If I uncomment the sleep() call above, there are no problems. A workaround: $ok = mkdir($dir, 0777, true); if (!$ok && !is_dir($dir)) { sleep(1); mkdir($dir, 0777, true); } In other words, it works if I try again. I hope that mkdir() can be changed so that it only returns false if the directory cannot be created at all (e.g. because of lack of permissions or because it already exists). -- Edit this bug report at https://bugs.php.net/bug.php?id=35326&edit=1
Req #35326 [Com]: Concurrency issue with recursive mkdir
Edit report at https://bugs.php.net/bug.php?id=35326&edit=1 ID: 35326 Comment by: gustavo dot straube at gmail dot com Reported by:bugs dot php dot net at chsc dot dk Summary:Concurrency issue with recursive mkdir Status: Open Type: Feature/Change Request Package:Feature/Change Request Operating System: Linux PHP Version:5.1.6 Block user comment: N Private report: N New Comment: Problem still exists in PHP 5.3.3. Previous Comments: [2006-10-10 07:25:52] bugs dot php dot net at chsc dot dk Problem still exists in PHP 5.1.6. [2005-11-22 13:52:24] bugs dot php dot net at chsc dot dk Yes, it also occurs with the CVS snapshot (this time I tried with the CGI version). AFAICS the problem is in php_plain_files_mkdir in main/streams/plain_wrapper.c. If php_mkdir() or VCWD_MKDIR() returns false, the method gives up instead of checking whether directory exists anyway, i.e. if it was created by another thread. [2005-11-22 12:13:10] tony2...@php.net Please try using this CVS snapshot: http://snaps.php.net/php5-latest.tar.gz For Windows: http://snaps.php.net/win32/php5-win32-latest.zip [2005-11-22 11:32:49] bugs dot php dot net at chsc dot dk Description: There appears to be a concurrency issue when mkdir is called on two separate threads with the following parameters: Thread 1: mkdir('/tmp/foo/bar/1', 0777, true); Thread 2: mkdir('/tmp/foo/bar/2', 0777, true); Sometimes one of the threads (e.g. thread 2) two fails with "Warning: mkdir(): File exists" and the directory /tmp/foo/bar/2 does not exist afterwards. What I think is happening in thread 2 is is this: PHP looks for /tmp/foo/bar/2 and sees that it does not exist. Neither does /tmp/foo/bar and /tmp/foo. So it tries to create /tmp/foo, but that directory was just created a millisecond second ago by another thread, so mkdir gives instead of just moving on to create /tmp/foo/bar and /tmp/foo/bar/2. Reproduce code: --- This code reproduces the problem in about every second attempt. It consists of two files, index.php and iframe.php. index.php: Nr. 1 Nr. 2 iframe.php: time()); $dir = '/tmp/' . $fire . '/foo/bar/foo/bar/' . $no; //sleep($no); var_dump(is_dir($dir)); echo "Making $dir "; mkdir($dir, 0777, true); var_dump(is_dir($dir)); ?> Expected result: Thread 2 should output this: bool(false) Making /tmp/1132653651/foo/bar/foo/bar/2 bool(true) Actual result: -- Thread 2 sometimes outputs this: bool(false) Making /tmp/1132655181/foo/bar/foo/bar/2 Warning: mkdir(): File exists in /home/chsc/public_html/mkdir/iframe.php on line 10 bool(false) Notice that the directory does not exist afterwards. If I uncomment the sleep() call above, there are no problems. A workaround: $ok = mkdir($dir, 0777, true); if (!$ok && !is_dir($dir)) { sleep(1); mkdir($dir, 0777, true); } In other words, it works if I try again. I hope that mkdir() can be changed so that it only returns false if the directory cannot be created at all (e.g. because of lack of permissions or because it already exists). -- Edit this bug report at https://bugs.php.net/bug.php?id=35326&edit=1