On 05/11/2012 06:40 PM, Mark wrote: > Hi, > > On Fri, May 11, 2012 16:45, Pádraig Brady wrote: >> On 05/11/2012 04:03 PM, Mark wrote: >>> Here's a patch for cp which adds a new --preallocate option. When >>> specified, cp allocates disk space for the destination file before >>> ... >>> >>> To-do list: >>> - Add --preallocate option to mv as well >>> - Should the option name be changed to --pre-allocate? >>> - Maybe have an option to tell cp to pre-allocate space for all >>> destination files in one go, rather than pre-allocating space for each >>> individual file before copying? >> >> I don't think there should be an option at all. >> cp should have enough info to do the right thing. >> Why would you even not want to preallocate? > > Apart from the case I mentioned where posix_fallocate() might write out > zeros on some systems, I can't think of many cases where pre-allocating > wouldn't be a good idea. So perhaps at some point in the future it would > be the default, with a --no-preallocate option to disable. > > >>> - Better handling of sparse files, e.g. don't call fallocate() if >>> source file is sparse and --sparse=always is given. >> >> That's an important consideration. > > Even for sparse files, if cp can determine where the holes are beforehand > (using SEEK_HOLE/SEEK_DATA or fiemap), it could pre-allocate the non-hole > regions. So with more work, pre-allocation could be used with sparse > files. Implementing that is beyond my abilities though... > > It would still be helpful to allow pre-allocation to be disabled. (E.g. > source file has many written/allocated all-zero regions, which the user > wants to turn into holes in the destination file.) > > Still another option: the ability to use FALLOC_FL_PUNCH_HOLE to punch > holes in (pre-allocated) destination files. Then any all-zero non-hole > regions in the source could be turned into holes in the destination in > conjunction with --sparse=always. By pre-allocating, cp would at least > guarantee the copy won't fail due to insufficient disk space, which may be > useful in some cases. > > >>> - If pre-allocation fails due to insufficient disk space, cp prints a >>> message and continues. So typically it will fill up the disk then abort >>> with an out-of-disk-space error. It would be nice to be able to tell cp >>> to abort when a pre-allocation fails, so it can exit without wasting >>> time. >> >> Yes it should exit immediately on ENOSPC > > Aborting by default would make sense. Ideally that would be configurable. > In some cases the user might prefer to get a warning, so they can Ctrl-Z > suspend cp and delete/move other files to make space, instead of having cp > abort. > > > An interesting aside: I tried using cp to pre-allocate space for a very > large file on an ext4 partition, much larger than the amount of free > space. IMHO it would be best for the filesystem to fail immediately in > that case. ext4 does a lot of work (there was a lot of disk activity and > it took a long time to fail). ext4 pre-allocates as much of the requested > region as possible, rather than succeeding or failing all-or-nothing. So > you get a disk-full condition. (Of course that's no worse than what > happens when you run cp normally. But it would happen much more quickly > with pre-allocation.)
Well that's bad as you get a delay in addition to the normal copy. However, I don't see that behavior with 2.6.40.4-5.fc15.x86_64 at least? $ df -Th . Filesystem Type Size Used Avail Use% Mounted on /dev/sdb1 ext4 97G 85G 7.1G 93% /home $ for s in 1 10 100; do time fallocate -l ${s}G testfile; rm testfile; done sys 0m0.121s fallocate: testfile: fallocate failed: No space left on device sys 0m0.345s fallocate: testfile: fallocate failed: No space left on device sys 0m0.001s > You can try that by doing something like this on an ext4 partition: > truncate -s 9999999999999 testfile # (create a sparse file larger than > free space) > cp --preallocate testfile testfile.copy Note I'mm CC'ing the bug address rather than the coreut...@gnu.org address to keep all of this thread associated with the bug. cheers, Pádraig.