tags 34713 notabug close 34713 thanks Hello Christoph,
Christoph Michelbach wrote: > To reproduce this bug, you need two different file systems. Adapt the > paths to fit your system. Thank you for making this bug report. However what you are experiencing is due to the race condition created by the non-atomic nature of copying files from one file system to another, removing files, and renaming files. This is not a bug in mv but is an intrinsic behavior. > Set the experimental file structure up like this: > > mkdir exp > cd exp > mkdir a > cd a > touch a{1..100000} > cd .. > mkdir b > cd b > touch b{1..10000} > mkdir /t/ae # /t has to be on a different file system Thank you for the very nice test case. > Then have two terminals open in the exp directory created above. This is a clue to the nature of the problem being a race condition. It describes simultaneous parallel processes. > In one, execute this command: > > mv a /t/ae Because /t is on a different file system mv cannot simply rename the files but must perform the action in two steps. It copies the file from source to destination. It removes source file. This is documented in the mv with: 'mv' can move any type of file from one file system to another. Prior to version '4.0' of the fileutils, 'mv' could move only regular files between file systems. For example, now 'mv' can move an entire directory hierarchy including special device files from one partition to another. It first uses some of the same code that's used by 'cp -a' to copy the requested directories and files, then (assuming the copy succeeded) it removes the originals. If the copy fails, then the part that was copied to the destination partition is removed. If you were to copy three directories from one partition to another and the copy of the first directory succeeded, but the second didn't, the first would be left on the destination partition and the second and third would be left on the original partition. The mv a /t/ae action is similar to cp -a a t/ae && rm -r a when the action is successful. Similar because there are two steps happening. A first step with the copy and a second step with the removal and there is a time skew between those actions. > In the other, execute this one while the one in the first terminal > still is running (hence the large number of files so you have time to > do this): > > mv b/* a This is the second part of the race condition. It it moving files into the a directory at the same time that files are being copied out of the directory and the directory itself being removed. > You will end up with 100 000 files in /t/ae. The 10 000 files beginning > with the letter b will be gone. Look at the two actions explicitly: Process 1: cp -a a /t/ae rm -rf a Process 2: mv b/* a Now it is more obvious that as soon as the first process copy finishes that it will remove the source location, that is having files moved into it by the second process, that the directory will be deleted by the first process. Does that make it easier to understand what is happening? The copy and remove two actions do not occur when both the source and destination are on the same file system. In that case the file can be renamed atomically without doing a copy. But when the action is across two file systems this is not possible and it is simulated (or perhaps emulated) by the copy and remove two step action. Whenever tasks are moving files into and out of the same directory at the same time this is always something to be aware of regardless because they may be an overlap of actions in that directory. In this particular example the problem can be avoided by renaming "a" first and then transfering the files to the other file system. Because it was removed then the second process can create it without collision. Something like this pseudo-code. However to safely use temporary file names will require more code than this. This is simply for illustration purposes. Process 1: mv a tmpdirname cp -a tmpdirname /t/ae rm -rf tmpdirname Process 2: mkdir a mv b/* a I hope this helps. Since this is not a bug in mv I am going to close the ticket in our bug database. But please we would like to hear back from you in this ticket for any further discussion. Bob