Re: Cygwin api to punch a hole into a file?
On Dec 5 15:47, Thomas Wolff via Cygwin wrote: > > Am 05/12/2023 um 15:40 schrieb Thomas Wolff via Cygwin: > > Am 05/12/2023 um 14:53 schrieb Corinna Vinschen via Cygwin: > > > On Dec 5 07:36, Thomas Wolff via Cygwin wrote: > > > > > > > > Am 01.12.2023 um 12:02 schrieb Corinna Vinschen via Cygwin: > > > > > On Dec 1 11:22, Cedric Blancher via Cygwin wrote: > > > > > > Not all filesystems have a 128k block/stripe size, and certainly > > > > > > most > > > > > > filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes > > > > > > is common, ref pathconf _PC_MIN_HOLE_SIZE). > > > > > There's no _PC_MIN_HOLE_SIZE in Linux or POSIX. In Windows, a sparse > > > > > file uses chunks of 64K. You can see this even with a file of just > > > > > a single block. Try this: > > > > > > > > > > $ touch x > > > > > $ chattr +S x > > > > > > > > > > $ echo > > > > > 1234567890123456789012345678901234567890123456789012345678901234567890123456789 > > > > > >> x > > > > > $ ls -ls x > > > > > 1 -rw-r--r-- 1 corinna vinschen 80 Dec 1 11:56 x > > > > > > > > > > [repeat echo and ls -ls until...] > > > > > > > > > > $ echo > > > > > 1234567890123456789012345678901234567890123456789012345678901234567890123456789 > > > > > >> x > > > > > $ ls -ls x > > > > > 1 -rw-r--r-- 1 corinna vinschen 720 Dec 1 11:56 x > > > > > $ echo > > > > > 1234567890123456789012345678901234567890123456789012345678901234567890123456789 > > > > > >> x > > > > > $ ls -ls x > > > > > 64 -rw-r--r-- 1 corinna vinschen 800 Dec 1 11:56 x2 > > > > > ^^ > > > > > This > > > > For me, it goes up from 1 to 4, then in steps of 4KB. > > > Is that a local NTFS, did you actually call > > > > > > chattr +S x > > > > > > after touching the file, and did you check with > > > > > > lsattr x > > > > > > that x is actually sparse? > > Ups, sorry, I was just throwing my 2p into something. Yes, on a local > > NTFS; setting chattr +s does not change it but lsattr says: > > ---a .ls-s > > > > So why does chattr not seem to work? > Sorry again, I failed to test properly (I had copied your `chattr +S x` > which is not the filename I tested... blush). > With chattr +S, I reproduce your observation. No worries. Thanks for testing and confirming! Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin api to punch a hole into a file?
Am 05/12/2023 um 15:40 schrieb Thomas Wolff via Cygwin: Am 05/12/2023 um 14:53 schrieb Corinna Vinschen via Cygwin: On Dec 5 07:36, Thomas Wolff via Cygwin wrote: Am 01.12.2023 um 12:02 schrieb Corinna Vinschen via Cygwin: On Dec 1 11:22, Cedric Blancher via Cygwin wrote: Not all filesystems have a 128k block/stripe size, and certainly most filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes is common, ref pathconf _PC_MIN_HOLE_SIZE). There's no _PC_MIN_HOLE_SIZE in Linux or POSIX. In Windows, a sparse file uses chunks of 64K. You can see this even with a file of just a single block. Try this: $ touch x $ chattr +S x $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 1 -rw-r--r-- 1 corinna vinschen 80 Dec 1 11:56 x [repeat echo and ls -ls until...] $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 1 -rw-r--r-- 1 corinna vinschen 720 Dec 1 11:56 x $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 64 -rw-r--r-- 1 corinna vinschen 800 Dec 1 11:56 x2 ^^ This For me, it goes up from 1 to 4, then in steps of 4KB. Is that a local NTFS, did you actually call chattr +S x after touching the file, and did you check with lsattr x that x is actually sparse? Ups, sorry, I was just throwing my 2p into something. Yes, on a local NTFS; setting chattr +s does not change it but lsattr says: ---a .ls-s So why does chattr not seem to work? Sorry again, I failed to test properly (I had copied your `chattr +S x` which is not the filename I tested... blush). With chattr +S, I reproduce your observation. Thomas Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin api to punch a hole into a file?
Am 05/12/2023 um 14:53 schrieb Corinna Vinschen via Cygwin: On Dec 5 07:36, Thomas Wolff via Cygwin wrote: Am 01.12.2023 um 12:02 schrieb Corinna Vinschen via Cygwin: On Dec 1 11:22, Cedric Blancher via Cygwin wrote: Not all filesystems have a 128k block/stripe size, and certainly most filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes is common, ref pathconf _PC_MIN_HOLE_SIZE). There's no _PC_MIN_HOLE_SIZE in Linux or POSIX. In Windows, a sparse file uses chunks of 64K. You can see this even with a file of just a single block. Try this: $ touch x $ chattr +S x $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 1 -rw-r--r-- 1 corinna vinschen 80 Dec 1 11:56 x [repeat echo and ls -ls until...] $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 1 -rw-r--r-- 1 corinna vinschen 720 Dec 1 11:56 x $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 64 -rw-r--r-- 1 corinna vinschen 800 Dec 1 11:56 x2 ^^ This For me, it goes up from 1 to 4, then in steps of 4KB. Is that a local NTFS, did you actually call chattr +S x after touching the file, and did you check with lsattr x that x is actually sparse? Ups, sorry, I was just throwing my 2p into something. Yes, on a local NTFS; setting chattr +s does not change it but lsattr says: ---a .ls-s So why does chattr not seem to work? Thomas Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin api to punch a hole into a file?
On Dec 5 07:36, Thomas Wolff via Cygwin wrote: > > > Am 01.12.2023 um 12:02 schrieb Corinna Vinschen via Cygwin: > > On Dec 1 11:22, Cedric Blancher via Cygwin wrote: > > > Not all filesystems have a 128k block/stripe size, and certainly most > > > filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes > > > is common, ref pathconf _PC_MIN_HOLE_SIZE). > > There's no _PC_MIN_HOLE_SIZE in Linux or POSIX. In Windows, a sparse > > file uses chunks of 64K. You can see this even with a file of just > > a single block. Try this: > > > >$ touch x > >$ chattr +S x > > > >$ echo > > 1234567890123456789012345678901234567890123456789012345678901234567890123456789 > > >> x > >$ ls -ls x > >1 -rw-r--r-- 1 corinna vinschen 80 Dec 1 11:56 x > > > >[repeat echo and ls -ls until...] > > > >$ echo > > 1234567890123456789012345678901234567890123456789012345678901234567890123456789 > > >> x > >$ ls -ls x > >1 -rw-r--r-- 1 corinna vinschen 720 Dec 1 11:56 x > >$ echo > > 1234567890123456789012345678901234567890123456789012345678901234567890123456789 > > >> x > >$ ls -ls x > >64 -rw-r--r-- 1 corinna vinschen 800 Dec 1 11:56 x2 > >^^ > >This > For me, it goes up from 1 to 4, then in steps of 4KB. Is that a local NTFS, did you actually call chattr +S x after touching the file, and did you check with lsattr x that x is actually sparse? Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin api to punch a hole into a file?
Am 01.12.2023 um 12:02 schrieb Corinna Vinschen via Cygwin: On Dec 1 11:22, Cedric Blancher via Cygwin wrote: Not all filesystems have a 128k block/stripe size, and certainly most filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes is common, ref pathconf _PC_MIN_HOLE_SIZE). There's no _PC_MIN_HOLE_SIZE in Linux or POSIX. In Windows, a sparse file uses chunks of 64K. You can see this even with a file of just a single block. Try this: $ touch x $ chattr +S x $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 1 -rw-r--r-- 1 corinna vinschen 80 Dec 1 11:56 x [repeat echo and ls -ls until...] $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 1 -rw-r--r-- 1 corinna vinschen 720 Dec 1 11:56 x $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 64 -rw-r--r-- 1 corinna vinschen 800 Dec 1 11:56 x2 ^^ This For me, it goes up from 1 to 4, then in steps of 4KB. I don't know how Windows stores short files, maybe they are part of the metadata up to a point or some such. However, as soon as you raise the size over a given point, the *allocation size* will be rounded up to 64K and from that point on, it will be in 64K chunks. Also sparsifying and desparsifying of blocks only works in 64K chunks. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin api to punch a hole into a file?
Greetings, Corinna Vinschen via Cygwin! > I don't know how Windows stores short files, By Windows I Think you did mean NTFS. Then yes, file data for certain short values is stored directly in $MFT. This is not specifically file contents, and even if a file content is just 1 byte, not necessarily it will be stored in $MFT. It all depends on the entire size of file metadata. If it does not exceed some-under-1K in total, then file data may be stored in $MFT as well. > maybe they are part of the metadata up to a point or some such. However, as > soon as you raise the size over a given point, the *allocation size* will be > rounded up to 64K and from that point on, it will be in 64K chunks. Also > sparsifying and desparsifying of blocks only works in 64K chunks. -- With best regards, Andrey Repin Monday, December 4, 2023 18:56:37 Sorry for my terrible english... -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin api to punch a hole into a file?
On Dec 1 11:22, Cedric Blancher via Cygwin wrote: > I see the value of 128k (128*1024 bytes) quite often in your sparse > file commits. Yes, but they have been removed. Read the patches again, especially commit 65831f88d6c4. > Can you please make this value a per filesystem tunable? No, because we don't have a facility for that. > Not all filesystems have a 128k block/stripe size, and certainly most > filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes > is common, ref pathconf _PC_MIN_HOLE_SIZE). There's no _PC_MIN_HOLE_SIZE in Linux or POSIX. In Windows, a sparse file uses chunks of 64K. You can see this even with a file of just a single block. Try this: $ touch x $ chattr +S x $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 1 -rw-r--r-- 1 corinna vinschen 80 Dec 1 11:56 x [repeat echo and ls -ls until...] $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 1 -rw-r--r-- 1 corinna vinschen 720 Dec 1 11:56 x $ echo 1234567890123456789012345678901234567890123456789012345678901234567890123456789 >> x $ ls -ls x 64 -rw-r--r-- 1 corinna vinschen 800 Dec 1 11:56 x2 ^^ This I don't know how Windows stores short files, maybe they are part of the metadata up to a point or some such. However, as soon as you raise the size over a given point, the *allocation size* will be rounded up to 64K and from that point on, it will be in 64K chunks. Also sparsifying and desparsifying of blocks only works in 64K chunks. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
_PC_MIN_HOLE_SIZE pathconf() Re: Cygwin api to punch a hole into a file?
On Tue, 28 Nov 2023 at 11:30, Corinna Vinschen via Cygwin wrote: > > On Nov 24 12:01, Corinna Vinschen via Cygwin wrote: > > On Nov 23 23:36, Cedric Blancher via Cygwin wrote: > > > Linux has fallocate(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, ...) > > > to punch a hole into a file, i.e. deallocate the blocks given and make > > > the file a "sparse file". > > > > We don't support the Linux-specific fallocate(2) call, only ftruncate(2) > > and posix_fallocate(3). Patches, as usual, thoughtfully considered. > > The next test release cygwin-3.5.0-0.485.g65831f88d6c4 introduces > the Linux-specific fallocate(2) call. Naturally we can't support > all flags, but the following flag combinations are allowed: > > - 0 same as posix_fallocate(3) > - FALLOC_FL_KEEP_SIZE > - FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE > - FALLOC_FL_ZERO_RANGE > - FALLOC_FL_ZERO_RANGE | FALLOC_FL_KEEP_SIZE > > A few comments: > > - With 0 and FALLOC_FL_KEEP_SIZE, sparse blocks in the given range > are re-allocated as per the POSIX requirements. For that, it uses > the same code as FALLOC_FL_ZERO_RANGE, but only for the holes within > the range. > > - With FALLOC_FL_KEEP_SIZE, over-allocation is done by setting > the allocation size of a file while keeping EOF the same. > However, in contrast to your typical Linux filesystem, over- > allocation on Windows filesystems is only temporary. The > over-allocated blocks are returned to free blocks as soon as > the last HANDLE to the file is closed. > > - With FALLOC_FL_KEEP_SIZE, no over-allocation is performed on sparse > files. The reason is that over-allocation on sparse files has no > effect on Windows. > > - FALLOC_FL_ZERO_RANGE is implemented as writing zeros to the > given range. For parts of the range which are already allocated > data blocks, as much zeros are written as necessary. For holes > in sparse files, only a single zero byte is written to the > hole per 64K chunk, which is the allocation granularity of > sparse files. > > - FALLOC_FL_ZERO_RANGE is NOT atomic. > > Please give it a try. Could you please implement _PC_MIN_HOLE_SIZE in pathconf(), so that people can query the minimum hole size for a (path in a) filesystem? Should return nothing for FAT and NFSv3, should work on NTFS and NFSv4.2 Ced -- Cedric Blancher [https://plus.google.com/u/0/+CedricBlancher/] Institute Pasteur -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin api to punch a hole into a file?
On Tue, 28 Nov 2023 at 11:30, Corinna Vinschen via Cygwin wrote: > > On Nov 24 12:01, Corinna Vinschen via Cygwin wrote: > > On Nov 23 23:36, Cedric Blancher via Cygwin wrote: > > > Linux has fallocate(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, ...) > > > to punch a hole into a file, i.e. deallocate the blocks given and make > > > the file a "sparse file". > > > > We don't support the Linux-specific fallocate(2) call, only ftruncate(2) > > and posix_fallocate(3). Patches, as usual, thoughtfully considered. > > The next test release cygwin-3.5.0-0.485.g65831f88d6c4 introduces > the Linux-specific fallocate(2) call. Naturally we can't support > all flags, but the following flag combinations are allowed: > > - 0 same as posix_fallocate(3) > - FALLOC_FL_KEEP_SIZE > - FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE > - FALLOC_FL_ZERO_RANGE > - FALLOC_FL_ZERO_RANGE | FALLOC_FL_KEEP_SIZE > > A few comments: > > - With 0 and FALLOC_FL_KEEP_SIZE, sparse blocks in the given range > are re-allocated as per the POSIX requirements. For that, it uses > the same code as FALLOC_FL_ZERO_RANGE, but only for the holes within > the range. > > - With FALLOC_FL_KEEP_SIZE, over-allocation is done by setting > the allocation size of a file while keeping EOF the same. > However, in contrast to your typical Linux filesystem, over- > allocation on Windows filesystems is only temporary. The > over-allocated blocks are returned to free blocks as soon as > the last HANDLE to the file is closed. > > - With FALLOC_FL_KEEP_SIZE, no over-allocation is performed on sparse > files. The reason is that over-allocation on sparse files has no > effect on Windows. > > - FALLOC_FL_ZERO_RANGE is implemented as writing zeros to the > given range. For parts of the range which are already allocated > data blocks, as much zeros are written as necessary. For holes > in sparse files, only a single zero byte is written to the > hole per 64K chunk, which is the allocation granularity of > sparse files. > > - FALLOC_FL_ZERO_RANGE is NOT atomic. > > Please give it a try. Corinna, THANK you! Related question about commit "Cygwin: pwrite(2): sparsify file" https://cygwin.com/git/?p=newlib-cygwin.git;a=blobdiff;f=winsup/cygwin/fhandler/disk_file.cc;h=c70afed49f1ecdb11812d31a9663d18c0a5f03f7;hp=b49b25c71ad030a9391db89a157cadcd095d4f36;hb=f64f3eced8e0f51753bc199f3ba96fab06a54848;hpb=114f89caff7b9b62b0b12bc2c6d143daf47b8042 I see the value of 128k (128*1024 bytes) quite often in your sparse file commits. Can you please make this value a per filesystem tunable? Not all filesystems have a 128k block/stripe size, and certainly most filesystems have smaller minimum hole sizes than 128k (e.g. 512bytes is common, ref pathconf _PC_MIN_HOLE_SIZE). Ced -- Cedric Blancher [https://plus.google.com/u/0/+CedricBlancher/] Institute Pasteur -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin api to punch a hole into a file?
On Nov 24 12:01, Corinna Vinschen via Cygwin wrote: > On Nov 23 23:36, Cedric Blancher via Cygwin wrote: > > Linux has fallocate(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, ...) > > to punch a hole into a file, i.e. deallocate the blocks given and make > > the file a "sparse file". > > We don't support the Linux-specific fallocate(2) call, only ftruncate(2) > and posix_fallocate(3). Patches, as usual, thoughtfully considered. The next test release cygwin-3.5.0-0.485.g65831f88d6c4 introduces the Linux-specific fallocate(2) call. Naturally we can't support all flags, but the following flag combinations are allowed: - 0 same as posix_fallocate(3) - FALLOC_FL_KEEP_SIZE - FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE - FALLOC_FL_ZERO_RANGE - FALLOC_FL_ZERO_RANGE | FALLOC_FL_KEEP_SIZE A few comments: - With 0 and FALLOC_FL_KEEP_SIZE, sparse blocks in the given range are re-allocated as per the POSIX requirements. For that, it uses the same code as FALLOC_FL_ZERO_RANGE, but only for the holes within the range. - With FALLOC_FL_KEEP_SIZE, over-allocation is done by setting the allocation size of a file while keeping EOF the same. However, in contrast to your typical Linux filesystem, over- allocation on Windows filesystems is only temporary. The over-allocated blocks are returned to free blocks as soon as the last HANDLE to the file is closed. - With FALLOC_FL_KEEP_SIZE, no over-allocation is performed on sparse files. The reason is that over-allocation on sparse files has no effect on Windows. - FALLOC_FL_ZERO_RANGE is implemented as writing zeros to the given range. For parts of the range which are already allocated data blocks, as much zeros are written as necessary. For holes in sparse files, only a single zero byte is written to the hole per 64K chunk, which is the allocation granularity of sparse files. - FALLOC_FL_ZERO_RANGE is NOT atomic. Please give it a try. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin api to punch a hole into a file?
On Nov 23 23:36, Cedric Blancher via Cygwin wrote: > Linux has fallocate(fd, FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, ...) > to punch a hole into a file, i.e. deallocate the blocks given and make > the file a "sparse file". We don't support the Linux-specific fallocate(2) call, only ftruncate(2) and posix_fallocate(3). Patches, as usual, thoughtfully considered. > But how can I do that with the Cygwin API? Does anyone have an example > for Cygwin, which is the Win equivalent to Linux fallocate(fd, > FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, 1048576, 2097152);? > > I tried with Cygwin 3.5, but the file was not sparse after that. Do you remember the discussion in August, starting here: https://cygwin.com/pipermail/cygwin-developers/2023-August/012664.html especially: https://cygwin.com/pipermail/cygwin-developers/2023-August/012679.html So, did you set the sparse mount option per https://cygwin.com/cygwin-ug-net/using.html#mount-table ? Also, chattr -S, which is independent from the mount option "sparse": https://cygwin.com/cygwin-ug-net/chattr.html Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple