Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?
On Dec 18 12:10, Cedric Blancher via Cygwin wrote: > On Wed, 6 Dec 2023 at 10:39, Corinna Vinschen via Cygwin > wrote: > > Basically, with 3.4, you always have to set the "sparse" mount option, > > with 3.5, on local SSDs you don't. I don't see a problem here. > > How can I remount the existing C: mount with the "sparse" option in > Cygwin 3.4? I cannot get it working, or I am too stupid. How about you tell us what you did, so we can check? I know that it works, but apparently something crucial is missing. One thing, for instance, is, that changes in fstab and fstab.d only start working after stopping all Cygwin processes of the user or after calling `mount -a'. Another is, of course, how your fstab line actually looked like. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?
On Wed, 6 Dec 2023 at 10:39, Corinna Vinschen via Cygwin wrote: > > On Dec 6 09:49, Cedric Blancher via Cygwin wrote: > > On Fri, 1 Dec 2023 at 11:43, Corinna Vinschen via Cygwin > > VMware emulates NVME SSD these days, so this should work, yes? > > It doesn't matter, see below. > > > Primary concern is how to detect whether sparse file support is ON in > > Cygwin for a specific filesystem? Maybe /sbin/mount without options > > should have a sparse/nosparse mount flag to reflect what is used? > > What concern? I explained exactly what happens, and it all revolves > around the fact that a filesystem sets the FILE_SUPPORTS_SPARSE_FILES > flag or not. This is the only interesting fact. Nothing else > matters. > > If you want to see if a filesystem supports that flag, just use the > /usr/lib/csih/getVolInfo tool: > > $ /usr/lib/csih/getVolInfo . > Device Type: 7 > Characteristics: 20020 > Volume Name: > Serial Number : 2292001085 > Max Filenamelength : 255 > Filesystemname : > Flags : 3e72eff > FILE_CASE_SENSITIVE_SEARCH: TRUE > FILE_CASE_PRESERVED_NAMES : TRUE > FILE_UNICODE_ON_DISK : TRUE > FILE_PERSISTENT_ACLS : TRUE > FILE_FILE_COMPRESSION : TRUE > FILE_VOLUME_QUOTAS: TRUE > FILE_SUPPORTS_SPARSE_FILES: TRUE > FILE_SUPPORTS_REPARSE_POINTS : TRUE > FILE_SUPPORTS_REMOTE_STORAGE : FALSE > FILE_RETURNS_CLEANUP_RESULT_INFO : TRUE > FILE_SUPPORTS_POSIX_UNLINK_RENAME : TRUE > FILE_VOLUME_IS_COMPRESSED : FALSE > FILE_SUPPORTS_OBJECT_IDS : TRUE > FILE_SUPPORTS_ENCRYPTION : TRUE > FILE_NAMED_STREAMS: TRUE > FILE_READ_ONLY_VOLUME : FALSE > FILE_SEQUENTIAL_WRITE_ONCE: FALSE > FILE_SUPPORTS_TRANSACTIONS: TRUE > FILE_SUPPORTS_HARD_LINKS : TRUE > FILE_SUPPORTS_EXTENDED_ATTRIBUTES : TRUE > FILE_SUPPORTS_OPEN_BY_FILE_ID : TRUE > FILE_SUPPORTS_USN_JOURNAL : TRUE > FILE_SUPPORTS_INTEGRITY_STREAMS : FALSE > FILE_SUPPORTS_BLOCK_REFCOUNTING : FALSE > FILE_SUPPORTS_SPARSE_VDL : FALSE > FILE_DAX_VOLUME : FALSE > FILE_SUPPORTS_GHOSTING: FALSE > > > > What Cygwin gets to see is that you want to access a file on some > > > disk. It opens the file and fetches volume information. So it finds: > > > > > > - The filesystem returns the FILE_SUPPORTS_SPARSE_FILES > > > > > > So we can create sparse files. > > > > > > - The volume returns the SSINFO_FLAGS_NO_SEEK_PENALTY flag. > > > > > > So we're on a drive having no seek penalty. This is *probably* > > > an SSD, but may be some other type of drive. > > > > What about SMB? SMB supports sparse files, and should certainly use > > this by DEFAULT, as it makes many HPC apps operate much faster. > > BUT: SSINFO_FLAGS_NO_SEEK_PENALTY is only for filesystems with a > > drive, and not for network filesystem. > > > > Same applies to the NFSv4.1 filesystem driver, I'm going to pester > > Roland Mainz to add FILE_SUPPORTS_SPARSE_FILES support, so same issue > > as SMB here. > > Again, the filesystem doesn't matter. It either sets the > FILE_SUPPORTS_SPARSE_FILES flag or not, as simple as that. > > If it does, you can create sparse files with chattr +S, or you can rely > on the lseek/ftruncate/posix_fallocate automatism, or you stamp a hole > into the file with fallocate(FALLOC_FL_PUNCH_HOLE). > > The *only* difference is if you have to use the "sparse" mount option or > not. > > Basically, with 3.4, you always have to set the "sparse" mount option, > with 3.5, on local SSDs you don't. I don't see a problem here. How can I remount the existing C: mount with the "sparse" option in Cygwin 3.4? I cannot get it working, or I am too stupid. Ced -- Cedric Blancher [https://plus.google.com/u/0/+CedricBlancher/] Institute Pasteur -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?
On Dec 7 17:42, Shaddy Baddah via Cygwin wrote: > Hi Corinna, > > On 6/12/2023 8:38 pm, Corinna Vinschen via Cygwin wrote: > > > If you want to see if a filesystem supports that flag, just use the > > /usr/lib/csih/getVolInfo tool: > > > >$ /usr/lib/csih/getVolInfo . > >Device Type: 7 > >Characteristics: 20020 > >Volume Name: > >Serial Number : 2292001085 > >Max Filenamelength : 255 > >Filesystemname : > >Flags : 3e72eff > > FILE_CASE_SENSITIVE_SEARCH: TRUE > ... > > FILE_SUPPORTS_GHOSTING: FALSE > > Is is worth it/possible to extend this getVolInfo to also include the > SSINFO_FLAGS_NO_SEEK_PENALTY flag in the output?-- Regards, Shaddy Good idea. I created a patch and a new version of the csih package (0.9.14) will be available shortly. Thanks, Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?
Hi Corinna, On 6/12/2023 8:38 pm, Corinna Vinschen via Cygwin wrote: If you want to see if a filesystem supports that flag, just use the /usr/lib/csih/getVolInfo tool: $ /usr/lib/csih/getVolInfo . Device Type: 7 Characteristics: 20020 Volume Name: Serial Number : 2292001085 Max Filenamelength : 255 Filesystemname : Flags : 3e72eff FILE_CASE_SENSITIVE_SEARCH: TRUE ... FILE_SUPPORTS_GHOSTING: FALSE Is is worth it/possible to extend this getVolInfo to also include the SSINFO_FLAGS_NO_SEEK_PENALTY flag in the output?-- Regards, Shaddy -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?
On Dec 6 10:38, Corinna Vinschen via Cygwin wrote: > Again, the filesystem doesn't matter. It either sets the > FILE_SUPPORTS_SPARSE_FILES flag or not, as simple as that. > > If it does, you can create sparse files with chattr +S, or you can rely > on the lseek/ftruncate/posix_fallocate automatism, or you stamp a hole > into the file with fallocate(FALLOC_FL_PUNCH_HOLE). > > The *only* difference is if you have to use the "sparse" mount option or > not. > > Basically, with 3.4, you always have to set the "sparse" mount option, > with 3.5, on local SSDs you don't. I don't see a problem here. Oh, and to be very clear here: This *only* affects new files or files which are not yet sparse. As soon as a file is sparse, it stays sparse. Subsequently, the way sparse blocks are created or converted to allocated blocks during write, lseek, ftruncate/posix_fallocate depends solely on how the OS and the filesystem driver perform in this matter. Cygwin is out of the picture then. After all, it's still just a user space DLL. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?
On Dec 6 09:49, Cedric Blancher via Cygwin wrote: > On Fri, 1 Dec 2023 at 11:43, Corinna Vinschen via Cygwin > VMware emulates NVME SSD these days, so this should work, yes? It doesn't matter, see below. > Primary concern is how to detect whether sparse file support is ON in > Cygwin for a specific filesystem? Maybe /sbin/mount without options > should have a sparse/nosparse mount flag to reflect what is used? What concern? I explained exactly what happens, and it all revolves around the fact that a filesystem sets the FILE_SUPPORTS_SPARSE_FILES flag or not. This is the only interesting fact. Nothing else matters. If you want to see if a filesystem supports that flag, just use the /usr/lib/csih/getVolInfo tool: $ /usr/lib/csih/getVolInfo . Device Type: 7 Characteristics: 20020 Volume Name: Serial Number : 2292001085 Max Filenamelength : 255 Filesystemname : Flags : 3e72eff FILE_CASE_SENSITIVE_SEARCH: TRUE FILE_CASE_PRESERVED_NAMES : TRUE FILE_UNICODE_ON_DISK : TRUE FILE_PERSISTENT_ACLS : TRUE FILE_FILE_COMPRESSION : TRUE FILE_VOLUME_QUOTAS: TRUE FILE_SUPPORTS_SPARSE_FILES: TRUE FILE_SUPPORTS_REPARSE_POINTS : TRUE FILE_SUPPORTS_REMOTE_STORAGE : FALSE FILE_RETURNS_CLEANUP_RESULT_INFO : TRUE FILE_SUPPORTS_POSIX_UNLINK_RENAME : TRUE FILE_VOLUME_IS_COMPRESSED : FALSE FILE_SUPPORTS_OBJECT_IDS : TRUE FILE_SUPPORTS_ENCRYPTION : TRUE FILE_NAMED_STREAMS: TRUE FILE_READ_ONLY_VOLUME : FALSE FILE_SEQUENTIAL_WRITE_ONCE: FALSE FILE_SUPPORTS_TRANSACTIONS: TRUE FILE_SUPPORTS_HARD_LINKS : TRUE FILE_SUPPORTS_EXTENDED_ATTRIBUTES : TRUE FILE_SUPPORTS_OPEN_BY_FILE_ID : TRUE FILE_SUPPORTS_USN_JOURNAL : TRUE FILE_SUPPORTS_INTEGRITY_STREAMS : FALSE FILE_SUPPORTS_BLOCK_REFCOUNTING : FALSE FILE_SUPPORTS_SPARSE_VDL : FALSE FILE_DAX_VOLUME : FALSE FILE_SUPPORTS_GHOSTING: FALSE > > What Cygwin gets to see is that you want to access a file on some > > disk. It opens the file and fetches volume information. So it finds: > > > > - The filesystem returns the FILE_SUPPORTS_SPARSE_FILES > > > > So we can create sparse files. > > > > - The volume returns the SSINFO_FLAGS_NO_SEEK_PENALTY flag. > > > > So we're on a drive having no seek penalty. This is *probably* > > an SSD, but may be some other type of drive. > > What about SMB? SMB supports sparse files, and should certainly use > this by DEFAULT, as it makes many HPC apps operate much faster. > BUT: SSINFO_FLAGS_NO_SEEK_PENALTY is only for filesystems with a > drive, and not for network filesystem. > > Same applies to the NFSv4.1 filesystem driver, I'm going to pester > Roland Mainz to add FILE_SUPPORTS_SPARSE_FILES support, so same issue > as SMB here. Again, the filesystem doesn't matter. It either sets the FILE_SUPPORTS_SPARSE_FILES flag or not, as simple as that. If it does, you can create sparse files with chattr +S, or you can rely on the lseek/ftruncate/posix_fallocate automatism, or you stamp a hole into the file with fallocate(FALLOC_FL_PUNCH_HOLE). The *only* difference is if you have to use the "sparse" mount option or not. Basically, with 3.4, you always have to set the "sparse" mount option, with 3.5, on local SSDs you don't. I don't see a problem here. Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Sparse file support for SMB by default? Re: Comment about "Cygwin: sparse support: enable automatic sparsifying of files on SSDs", extend feature to VMware/qemu disks?
On Fri, 1 Dec 2023 at 11:43, Corinna Vinschen via Cygwin wrote: > > On Dec 1 11:12, Cedric Blancher via Cygwin wrote: > > Good morning! > > > > Small comment about 3.5.0 commit "Cygwin: sparse support: enable > > automatic sparsifying of files on SSDs" > > https://cygwin.com/git/?p=newlib-cygwin.git;a=commit;h=832e91422c4ae9b2dcd0c307779f3dd53ee9c0ac > > > > Can this functionality be extended to VMware and QEMU virtual disks, please? > > Not without a complete rewrite and lots of slowing down code. > > VMware and QEMU virtual disks are not exposed to the OS as VMware and > QEMU virtual disks as far as the volume and filesystem information is > going. They are just some arbitrary harddisks managed by some > arbitrary driver. VMware emulates NVME SSD these days, so this should work, yes? Primary concern is how to detect whether sparse file support is ON in Cygwin for a specific filesystem? Maybe /sbin/mount without options should have a sparse/nosparse mount flag to reflect what is used? > > What Cygwin gets to see is that you want to access a file on some > disk. It opens the file and fetches volume information. So it finds: > > - The filesystem returns the FILE_SUPPORTS_SPARSE_FILES > > So we can create sparse files. > > - The volume returns the SSINFO_FLAGS_NO_SEEK_PENALTY flag. > > So we're on a drive having no seek penalty. This is *probably* > an SSD, but may be some other type of drive. What about SMB? SMB supports sparse files, and should certainly use this by DEFAULT, as it makes many HPC apps operate much faster. BUT: SSINFO_FLAGS_NO_SEEK_PENALTY is only for filesystems with a drive, and not for network filesystem. Same applies to the NFSv4.1 filesystem driver, I'm going to pester Roland Mainz to add FILE_SUPPORTS_SPARSE_FILES support, so same issue as SMB here. Ced -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
