Re: Create new files as sparse on NT systems. (2nd try)
On Mon, 3 Feb 2003, Vaclav Haisman wrote:
> This is a little bit improved version of my previous post.
> By default creation of sparse files is disabled. It can be enabled by CYGWIN
> option sparse_files.
>
> Vaclav Haisman
>
> 2003-02-03 Vaclav Haisman <[EMAIL PROTECTED]>
> [snip]
> * environ.cc (parse_thing): Add new CYGWIN option.
>
> Index: cygwin/environ.cc
> ===
> RCS file: /cvs/src/src/winsup/cygwin/environ.cc,v
> retrieving revision 1.90
> diff -p -u -r1.90 environ.cc
> --- cygwin/environ.cc 30 Sep 2002 03:05:13 - 1.90
> +++ cygwin/environ.cc 3 Feb 2003 12:54:40 -
> @@ -522,6 +522,7 @@ static struct parse_thing
>{"title", {&display_title}, justset, NULL, {{FALSE}, {TRUE}}},
>{"tty", {NULL}, set_process_state, NULL, {{0}, {PID_USETTY}}},
>{"winsymlinks", {&allow_winsymlinks}, justset, NULL, {{FALSE}, {TRUE}}},
> + {"sparse_files", {&allow_sparse}, justset, NULL, {{FALSE}, {TRUE}}},
>{NULL, {0}, justset, 0, {{0}, {0}}}
> };
> [snip]
Vaclav,
I don't know if it matters, but the rest of the entries in the parse_thing
table are alphabetically ordered...
Igor
--
http://cs.nyu.edu/~pechtcha/
|\ _,,,---,,_[EMAIL PROTECTED]
ZZZzz /,`.-'`'-. ;-;;,_[EMAIL PROTECTED]
|,4- ) )-,_. ,\ ( `'-' Igor Pechtchanski
'---''(_/--' `-'\_) fL a.k.a JaguaR-R-R-r-r-r-.-.-. Meow!
Oh, boy, virtual memory! Now I'm gonna make myself a really *big* RAMdisk!
-- /usr/games/fortune
Re: Create new files as sparse on NT systems. (2nd try)
I haven't noticed until you pointed it out. This can be easily corrected. Vaclav Haisman > I don't know if it matters, but the rest of the entries in the parse_thing > table are alphabetically ordered... > Igor > --
Re: Create new files as sparse on NT systems. (2nd try)
On Mon, Feb 03, 2003 at 02:18:03PM +0100, Vaclav Haisman wrote: > >This is a little bit improved version of my previous post. >By default creation of sparse files is disabled. It can be enabled by CYGWIN >option sparse_files. > >Vaclav Haisman > > >2003-02-03 Vaclav Haisman <[EMAIL PROTECTED]> > * fhandler.h (allow_sparse): Declare new extern variable. > * fhandler.cc (METHOD_BUFFERED): New macro. > (FSCTL_SET_SPARSE): Ditto. > (allow_sparse): Define the new variable. > (fhandler_base::open): Try to set newly created or trucated files > as sparse on NT systems. > * environ.cc (parse_thing): Add new CYGWIN option. This is YA case where I don't think that a CYGWIN environment variable option is justified. UNIX has a method for producing sparse files. If this is desired functionality, Cygwin should mimic that not invent a new way of doing things. cgf
Re: Create new files as sparse on NT systems. (2nd try)
On Thu, 13 Feb 2003, Christopher Faylor wrote: > This is YA case where I don't think that a CYGWIN environment variable option is > justified. > > UNIX has a method for producing sparse files. If this is desired functionality, > Cygwin should mimic that not invent a new way of doing things. > > cgf > I am not that much knowledgeable in matters of UNIX standars. What libary/system call is it? My justificaion of this patch is that with this option enable it mimics what this FreeBSD 4.7 box does. I don't need any other syscall than open() and ftruncate() to create 100GB empty sparse file. Vaclav Haisman
Re: Create new files as sparse on NT systems. (2nd try)
> UNIX has a method for producing sparse files. If this is desired functionality,
> Cygwin should mimic that not invent a new way of doing things.
>
> cgf
Hi,
I have prepared another patch that implements parse files for Cygwin. It is
smaller and, I think, even better than the previous. No new CYGWIN options.
I have also been searching internet for some informations abut sparse files in
Unix systems. It seems that if OS and file system supports it then it supports
it without any extra system call. Some systems (SunOS) have fcntl() command
F_FREESP that is supposed to free allocated disk space. But all unices I have
had look at only support such combination of parameters that the deallocated
block of space is at the end of file. In this case it works as ftruncate(). If
I should implement this fcntl() command such that it would be able to
deallocate disk space in a middle of a file then I would be inventing
something new.
Vaclav Haisman
2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]>
* include/winioctl.h (FSCTL_SET_SPARSE): Define.
2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]>
* fhandler.h: Include winioctl.h for DeviceIoControl.
(fhandler::open): Try to set newly created and truncated files as
sparse on NT systems.
Index: cygwin/fhandler.cc
===
RCS file: /cvs/src/src/winsup/cygwin/fhandler.cc,v
retrieving revision 1.143
diff -u -p -r1.143 fhandler.cc
--- cygwin/fhandler.cc 20 Dec 2002 01:48:22 - 1.143
+++ cygwin/fhandler.cc 17 Feb 2003 16:08:30 -
@@ -27,6 +27,7 @@ details. */
#include "pinfo.h"
#include
#include
+#include
static NO_COPY const int CHUNK_SIZE = 1024; /* Used for crlf conversions */
@@ -486,6 +487,18 @@ fhandler_base::open (path_conv *pc, int
&& !allow_ntsec && allow_ntea)
set_file_attribute (has_acls (), get_win32_name (), mode);
+ /* Try to set newly created files as sparse files on NT system. */
+ if (wincap.is_winnt () && get_device () == FH_DISK
+ && (access & GENERIC_WRITE) == GENERIC_WRITE
+ && (flags & (O_CREAT | O_TRUNC)))
+{
+ DWORD dw;
+ BOOL r = DeviceIoControl (x, FSCTL_SET_SPARSE, NULL, 0, NULL, 0, &dw,
+ NULL);
+ syscall_printf ("%d = DeviceIoControl(0x%x, FSCTL_SET_SPARSE, NULL, 0, "
+ "NULL, 0, &dw, NULL)", r, x);
+}
+
set_io_handle (x);
set_flags (flags, pc ? pc->binmode () : 0);
Index: w32api/include/winioctl.h
===
RCS file: /cvs/src/src/winsup/w32api/include/winioctl.h,v
retrieving revision 1.8
diff -u -p -r1.8 winioctl.h
--- w32api/include/winioctl.h 7 Nov 2002 14:14:01 - 1.8
+++ w32api/include/winioctl.h 17 Feb 2003 16:08:34 -
@@ -69,6 +69,7 @@ extern "C" {
#define FSCTL_GET_REPARSE_POINT CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 42,
METHOD_BUFFERED, FILE_ANY_ACCESS)
#define FSCTL_SET_REPARSE_POINT CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 41,
METHOD_BUFFERED, FILE_WRITE_DATA)
#define FSCTL_DELETE_REPARSE_POINT CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 43,
METHOD_BUFFERED, FILE_WRITE_DATA)
+#define FSCTL_SET_SPARSECTL_CODE(FILE_DEVICE_FILE_SYSTEM, 49,
+METHOD_BUFFERED, FILE_SPECIAL_ACCESS)
#define DEVICE_TYPE DWORD
#define FILE_DEVICE_BEEP 1
#define FILE_DEVICE_CD_ROM 2
Re: Create new files as sparse on NT systems. (2nd try)
On Mon, Feb 17, 2003 at 05:37:20PM +0100, Vaclav Haisman wrote: >> UNIX has a method for producing sparse files. If this is desired functionality, >> Cygwin should mimic that not invent a new way of doing things. >> >> cgf > >Hi, >I have prepared another patch that implements parse files for Cygwin. It is >smaller and, I think, even better than the previous. No new CYGWIN options. > >I have also been searching internet for some informations abut sparse files in >Unix systems. It seems that if OS and file system supports it then it supports >it without any extra system call. Some systems (SunOS) have fcntl() command >F_FREESP that is supposed to free allocated disk space. But all unices I have >had look at only support such combination of parameters that the deallocated >block of space is at the end of file. In this case it works as ftruncate(). If >I should implement this fcntl() command such that it would be able to >deallocate disk space in a middle of a file then I would be inventing >something new. This looks pretty good but the cygwin convention is to use wincap settings for this kind of thing rather than using is_winnt. So, please add a wincap capability to accomplish this. cgf >2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]> > > * include/winioctl.h (FSCTL_SET_SPARSE): Define. > >2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]> > > * fhandler.h: Include winioctl.h for DeviceIoControl. > (fhandler::open): Try to set newly created and truncated files as > sparse on NT systems.
Re: Create new files as sparse on NT systems. (2nd try)
> This looks pretty good but the cygwin convention is to use wincap
> settings for this kind of thing rather than using is_winnt. So, please
> add a wincap capability to accomplish this.
>
> cgf
Modified as suggested.
Vaclav Haisman
2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]>
* wincap.h (wincaps::supports_sparse_files): New flag.
(wincapc::supports_sparse_files): New method.
* wincap.cc (wincap_unknown): Define value for the new flag.
(wincap_95): Ditto.
(wincap_95osr2): Ditto.
(wincap_98): Ditto.
(wincap_98se): Ditto.
(wincap_me): Ditto.
(wincap_nt3): Ditto.
(wincap_nt4): Ditto.
(wincap_nt4sp4): Ditto.
(wincap_2000): Ditto.
(wincap_xp): Ditto.
* fhandler.cc: Include winioctl.h for DeviceIoControl.
(fhandler::open): Try to set newly created and truncated files as
sparse.
2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]>
* include/winioctl.h (FSCTL_SET_SPARSE): Define.
Index: cygwin/fhandler.cc
===
RCS file: /cvs/src/src/winsup/cygwin/fhandler.cc,v
retrieving revision 1.143
diff -u -p -r1.143 fhandler.cc
--- cygwin/fhandler.cc 20 Dec 2002 01:48:22 - 1.143
+++ cygwin/fhandler.cc 17 Feb 2003 17:23:43 -
@@ -27,6 +27,7 @@ details. */
#include "pinfo.h"
#include
#include
+#include
static NO_COPY const int CHUNK_SIZE = 1024; /* Used for crlf conversions */
@@ -486,6 +487,18 @@ fhandler_base::open (path_conv *pc, int
&& !allow_ntsec && allow_ntea)
set_file_attribute (has_acls (), get_win32_name (), mode);
+ /* Try to set newly created files as sparse files on NT system. */
+ if (wincap.supports_sparse_files () && get_device () == FH_DISK
+ && (access & GENERIC_WRITE) == GENERIC_WRITE
+ && (flags & (O_CREAT | O_TRUNC)))
+{
+ DWORD dw;
+ BOOL r = DeviceIoControl (x, FSCTL_SET_SPARSE, NULL, 0, NULL, 0, &dw,
+ NULL);
+ syscall_printf ("%d = DeviceIoControl(0x%x, FSCTL_SET_SPARSE, NULL, 0, "
+ "NULL, 0, &dw, NULL)", r, x);
+}
+
set_io_handle (x);
set_flags (flags, pc ? pc->binmode () : 0);
Index: cygwin/wincap.cc
===
RCS file: /cvs/src/src/winsup/cygwin/wincap.cc,v
retrieving revision 1.18
diff -u -p -r1.18 wincap.cc
--- cygwin/wincap.cc15 Oct 2002 17:04:20 - 1.18
+++ cygwin/wincap.cc17 Feb 2003 17:23:44 -
@@ -47,7 +47,8 @@ static NO_COPY wincaps wincap_unknown =
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_95 = {
@@ -86,7 +87,8 @@ static NO_COPY wincaps wincap_95 = {
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_95osr2 = {
@@ -125,7 +127,8 @@ static NO_COPY wincaps wincap_95osr2 = {
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_98 = {
@@ -164,7 +167,8 @@ static NO_COPY wincaps wincap_98 = {
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_98se = {
@@ -203,7 +207,8 @@ static NO_COPY wincaps wincap_98se = {
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_me = {
@@ -242,7 +247,8 @@ static NO_COPY wincaps wincap_me = {
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_nt3 = {
@@ -281,7 +287,8 @@ static NO_COPY wincaps wincap_nt3 = {
has_64bit_file_access:true,
has_process_io_counters:false,
supports_reading_modem_output_lines:true,
- needs_memory_protection:true
+ needs_memory_protection:true,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_nt4 = {
@@ -320,7 +327,8 @@ static NO_COPY wincaps wincap_nt4 = {
has_64bit_file_access:true,
has_process_io_counters:false,
supports_reading_modem_output_lines:true,
- needs_memory_protection:true
+ needs_memory_protection:true,
+ supports_sparse_files:false
Re: Create new files as sparse on NT systems. (2nd try)
> On Mon, Feb 17, 2003 at 05:37:20PM +0100, Vaclav Haisman wrote: >> I have prepared another patch that implements sparse files for >> Cygwin. It is smaller and, I think, even better than the previous. >> No new CYGWIN options. >> 2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]> >> >> * include/winioctl.h (FSCTL_SET_SPARSE): Define. >> >> 2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]> >> >> * fhandler.h: Include winioctl.h for DeviceIoControl. >> (fhandler::open): Try to set newly created and truncated files as >> sparse on NT systems. Christopher Faylor wrote: > This looks pretty good but the cygwin convention is to use wincap > settings for this kind of thing rather than using is_winnt. So, > please add a wincap capability to accomplish this. Is it wise to set *all* new files to sparse? Surely if this was actually advantageous, Windows would do it anyway? From MSDN: "Note It is up to the application to maintain sparseness by writing zeros with FSCTL_SET_ZERO_DATA." I.e., this will gain nothing unless the application knows about sparse-ness, in which case, it should explicitly specify that the file should be sparse. So, all this patch will do is to force Windows to examine more metadata for every file read. This seems *extremely undesirable*. Max.
Re: Create new files as sparse on NT systems. (2nd try)
> Is it wise to set *all* new files to sparse? Surely if this was actually > advantageous, Windows would do it anyway? From MSDN: "Note It is up to the > application to maintain sparseness by writing zeros with > FSCTL_SET_ZERO_DATA." I.e., this will gain nothing unless the application > knows about sparse-ness, in which case, it should explicitly specify that > the file should be sparse. So, all this patch will do is to force Windows to > examine more metadata for every file read. This seems *extremely > undesirable*. > > Max. As I have written in my previous emails in FreeBSD and SunOS all files are sparse if underlying file system supports it. I doubt Windows is significantly slower/faster in inspecting file system metadata than either of these OSes, Vaclav Haisman
Re: Create new files as sparse on NT systems. (2nd try)
Vaclav Haisman wrote: >> Is it wise to set *all* new files to sparse? Surely if this was >> actually advantageous, Windows would do it anyway? From MSDN: "Note >> It is up to the application to maintain sparseness by writing zeros >> with FSCTL_SET_ZERO_DATA." I.e., this will gain nothing unless the >> application knows about sparse-ness, in which case, it should >> explicitly specify that the file should be sparse. So, all this >> patch will do is to force Windows to examine more metadata for every >> file read. This seems *extremely undesirable*. >> >> Max. > > As I have written in my previous emails in FreeBSD and SunOS all > files are sparse if underlying file system supports it. I doubt > Windows is significantly slower/faster in inspecting file system > metadata than either of these OSes, That's nice. Did you read the bit above where I quoted MSDN? Merely setting the file as sparse will NOT SAVE SPACE on Windows. So, no space gain, and a performance penalty of untested magnitude. I see only disadvantages. Max.
Re: Create new files as sparse on NT systems. (2nd try)
> That's nice. Did you read the bit above where I quoted MSDN? Merely setting > the file as sparse will NOT SAVE SPACE on Windows. So, no space gain, and a > performance penalty of untested magnitude. I see only disadvantages. > > > Max. Oh yes, it will. Some applications do lseek() on rather long distances, then write few bytes, then do another lseek() etc. Without this Windows will physicaly write zeros to the file which takes time and space. With this patch regions of the file between the written bytes are not written to the file and do not occupy any space. Judging by http://linux-ntfs.sourceforge.net/ntfs/concepts/data_runs.html there seems to be same amount of metada as in case of non-sparse file. Vaclav Haisman
Re: Create new files as sparse on NT systems. (2nd try)
On Mon, Feb 17, 2003 at 07:20:42PM +0100, Vaclav Haisman wrote: >>That's nice. Did you read the bit above where I quoted MSDN? Merely >>setting the file as sparse will NOT SAVE SPACE on Windows. So, no >>space gain, and a performance penalty of untested magnitude. I see >>only disadvantages. > >Oh yes, it will. Some applications do lseek() on rather long >distances, then write few bytes, then do another lseek() etc. Without >this Windows will physicaly write zeros to the file which takes time >and space. With this patch regions of the file between the written >bytes are not written to the file and do not occupy any space. Judging >by http://linux-ntfs.sourceforge.net/ntfs/concepts/data_runs.html there >seems to be same amount of metada as in case of non-sparse file. lseeks were what I was thinking of when I suggested just following the UNIX convention. Is anyone willing to run a few simple benchmarks to see if there is a drawback to turning sparseness on for everything on an NTFS file system? Btw, now that I've said that it occurred to me to check GetVolumeInformation. There is apparently a FILE_SUPPORTS_SPARSE_FILES flag available. That's the ultimate way to deal with this rather than adding a wincap, I believe. Check (pc->fs.flags & FILE_SUPPORTS_SPARSE_FILES) in fhandler_disk_file::open and do the appropriate thing there. Sorry I didn't think of this before. cgf
Re: Create new files as sparse on NT systems. (2nd try)
> Btw, now that I've said that it occurred to me to check > GetVolumeInformation. There is apparently a FILE_SUPPORTS_SPARSE_FILES > flag available. That's the ultimate way to deal with this rather than > adding a wincap, I believe. Check (pc->fs.flags & > FILE_SUPPORTS_SPARSE_FILES) in fhandler_disk_file::open and do the > appropriate thing there. > > Sorry I didn't think of this before. > > cgf I know about this flag and I have considered it when I started writing this patch. It has one flaw. The volume of root of the path to the file doesn't have to be the same volume as the file is physicaly stored on in case of reparse points presence. Besides I don't see it as a problem if this call fails to set the sparseness becase the file system doesn't support it. Vaclav Haisman
Re: Create new files as sparse on NT systems. (2nd try)
On Mon, Feb 17, 2003 at 07:48:57PM +0100, Vaclav Haisman wrote: >>Btw, now that I've said that it occurred to me to check >>GetVolumeInformation. There is apparently a FILE_SUPPORTS_SPARSE_FILES >>flag available. That's the ultimate way to deal with this rather than >>adding a wincap, I believe. Check (pc->fs.flags & >>FILE_SUPPORTS_SPARSE_FILES) in fhandler_disk_file::open and do the >>appropriate thing there. >> >>Sorry I didn't think of this before. > >I know about this flag and I have considered it when I started writing >this patch. It has one flaw. The volume of root of the path to the >file doesn't have to be the same volume as the file is physicaly stored >on in case of reparse points presence. Besides I don't see it as a >problem if this call fails to set the sparseness becase the file system >doesn't support it. This is consistent with the way the rest of cygwin works, however. The same argument could be applied to testing for ntsec. If this was an issue then we should be changing the fs information to reflect reparse points. cgf
Re: Create new files as sparse on NT systems. (2nd try)
> This is consistent with the way the rest of cygwin works, however. The > same argument could be applied to testing for ntsec. If this was an issue > then we should be changing the fs information to reflect reparse points. > > cgf I am not sure what is the conclusion here. Should I make it check for FILE_SUPPORTS_SPARSE_FILES even though it can be inaccurate? Vaclav Haisman
Re: Create new files as sparse on NT systems. (2nd try)
On Mon, Feb 17, 2003 at 08:23:21PM +0100, Vaclav Haisman wrote: >> This is consistent with the way the rest of cygwin works, however. The >> same argument could be applied to testing for ntsec. If this was an issue >> then we should be changing the fs information to reflect reparse points. > >I am not sure what is the conclusion here. Should I make it check for >FILE_SUPPORTS_SPARSE_FILES even though it can be inaccurate? Yes. cgf
Re: Create new files as sparse on NT systems. (2nd try)
I've moved the whole thing into fhandler_disk_file::open and made it
conditional on FILE_SUPPORTS_SPARSE_FILES. I had to add new method to path_conv
to get access to flags member of fs_info. This patch includes wincap changes
but they can be left out as the rest of the patch doesn't rely on them anymore.
Vaclav Haisman
2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]>
* include/winioctl.h (FSCTL_SET_SPARSE): Define.
2003-02-18 Vaclav Haisman <[EMAIL PROTECTED]>
* wincap.h (wincaps::supports_sparse_files): New flag.
(wincapc::supports_sparse_files): New method.
* wincap.cc (wincap_unknown): Define value for the new flag.
(wincap_95): Ditto.
(wincap_95osr2): Ditto.
(wincap_98): Ditto.
(wincap_98se): Ditto.
(wincap_me): Ditto.
(wincap_nt3): Ditto.
(wincap_nt4): Ditto.
(wincap_nt4sp4): Ditto.
(wincap_2000): Ditto.
(wincap_xp): Ditto.
* path.h (path_conv::fs_flags): New method.
* fhandler_disk_file.cc: Include winioctl.h for DeviceIoControl.
(fhandler_disk_file::open): Set newly created and truncated files as
sparse on platforms that support it.
Index: cygwin/fhandler_disk_file.cc
===
RCS file: /cvs/src/src/winsup/cygwin/fhandler_disk_file.cc,v
retrieving revision 1.40
diff -u -p -r1.40 fhandler_disk_file.cc
--- cygwin/fhandler_disk_file.cc10 Feb 2003 22:43:29 - 1.40
+++ cygwin/fhandler_disk_file.cc18 Feb 2003 21:48:39 -
@@ -26,6 +26,7 @@ details. */
#include "pinfo.h"
#include
#include
+#include
#define _COMPILING_NEWLIB
#include
@@ -386,6 +387,19 @@ fhandler_disk_file::open (path_conv *rea
return 0;
}
+ /* Set newly created and truncated files as sparse files. */
+ if ((real_path->fs_flags () & FILE_SUPPORTS_SPARSE_FILES)
+ && (get_access () & GENERIC_WRITE) == GENERIC_WRITE
+ && (get_flags () & (O_CREAT | O_TRUNC)))
+{
+ DWORD dw;
+ HANDLE h = get_handle ();
+ BOOL r = DeviceIoControl (h , FSCTL_SET_SPARSE, NULL, 0, NULL, 0, &dw,
+ NULL);
+ syscall_printf ("%d = DeviceIoControl(0x%x, FSCTL_SET_SPARSE, NULL, 0, "
+ "NULL, 0, &dw, NULL)", r, h);
+}
+
set_symlink_p (real_path->issymlink ());
set_execable_p (real_path->exec_state ());
set_socket_p (real_path->issocket ());
Index: cygwin/path.h
===
RCS file: /cvs/src/src/winsup/cygwin/path.h,v
retrieving revision 1.51
diff -u -p -r1.51 path.h
--- cygwin/path.h 13 Feb 2003 03:13:37 - 1.51
+++ cygwin/path.h 18 Feb 2003 21:38:50 -
@@ -152,6 +152,7 @@ class path_conv
short get_unitn () {return devn == FH_BAD ? 0 : unit;}
DWORD file_attributes () {return fileattr;}
DWORD drive_type () {return fs.drive_type;}
+ DWORD fs_flags () {return fs.flags;}
BOOL fs_fast_ea () {return fs.sym_opt & PC_CHECK_EA;}
void set_path (const char *p) {strcpy (path, p);}
char *return_and_clear_normalized_path ();
Index: cygwin/wincap.cc
===
RCS file: /cvs/src/src/winsup/cygwin/wincap.cc,v
retrieving revision 1.18
diff -u -p -r1.18 wincap.cc
--- cygwin/wincap.cc15 Oct 2002 17:04:20 - 1.18
+++ cygwin/wincap.cc18 Feb 2003 21:38:51 -
@@ -47,7 +47,8 @@ static NO_COPY wincaps wincap_unknown =
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_95 = {
@@ -86,7 +87,8 @@ static NO_COPY wincaps wincap_95 = {
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_95osr2 = {
@@ -125,7 +127,8 @@ static NO_COPY wincaps wincap_95osr2 = {
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_98 = {
@@ -164,7 +167,8 @@ static NO_COPY wincaps wincap_98 = {
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_COPY wincaps wincap_98se = {
@@ -203,7 +207,8 @@ static NO_COPY wincaps wincap_98se = {
has_64bit_file_access:false,
has_process_io_counters:false,
supports_reading_modem_output_lines:false,
- needs_memory_protection:false
+ needs_memory_protection:false,
+ supports_sparse_files:false
};
static NO_C
Re: Create new files as sparse on NT systems. (2nd try)
Two things - First: Please, please don't make this the default! Once a file is sparsified, it cannot be unsparsified except by copying the contents to a new file! This seems like an optimization for a corner case is trying to cause a global change. And: FSCTL_SET_SPARSE, used in the patch, is *not defined* in current w32api !!! Max.
Re: Create new files as sparse on NT systems. (2nd try)
On Tue, Feb 18, 2003 at 10:08:56PM -, Max Bowsher wrote: >Two things - First: > >Please, please don't make this the default! Once a file is sparsified, it >cannot be unsparsified except by copying the contents to a new file! This >seems like an optimization for a corner case is trying to cause a global >change. Why is it a big deal if a file is sparse? I don't get it. In 99% of the cases this won't be a big deal. In the cases where it is a big deal, cygwin will be operating more like UNIX. >And: > >FSCTL_SET_SPARSE, used in the patch, is *not defined* in current w32api !!! Calm down: 2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]> * include/winioctl.h (FSCTL_SET_SPARSE): Define. cgf
Re: Create new files as sparse on NT systems. (2nd try)
Christopher Faylor wrote: > On Tue, Feb 18, 2003 at 10:08:56PM -, Max Bowsher wrote: >> Two things - First: >> >> Please, please don't make this the default! Once a file is >> sparsified, it cannot be unsparsified except by copying the contents >> to a new file! This seems like an optimization for a corner case is >> trying to cause a global change. > > Why is it a big deal if a file is sparse? I don't get it. In 99% of > the cases this won't be a big deal. In the cases where it is a big > deal, cygwin will be operating more like UNIX. Mainly, it feels aesthetically horrible to me. And, we don't have any benchmarks for the common case, yet. What kind of program would actually benefit from sparse files? And shouldn't it be the responsibility of that program to request them? >> And: >> >> FSCTL_SET_SPARSE, used in the patch, is *not defined* in current >> w32api !!! > > Calm down: > > 2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]> > > * include/winioctl.h (FSCTL_SET_SPARSE): Define. Oops. Max.
Re: Create new files as sparse on NT systems. (2nd try)
On Tue, Feb 18, 2003 at 10:21:26PM -, Max Bowsher wrote: >Christopher Faylor wrote: >> On Tue, Feb 18, 2003 at 10:08:56PM -, Max Bowsher wrote: >>> Two things - First: >>> >>> Please, please don't make this the default! Once a file is >>> sparsified, it cannot be unsparsified except by copying the contents >>> to a new file! This seems like an optimization for a corner case is >>> trying to cause a global change. >> >> Why is it a big deal if a file is sparse? I don't get it. In 99% of >> the cases this won't be a big deal. In the cases where it is a big >> deal, cygwin will be operating more like UNIX. > >Mainly, it feels aesthetically horrible to me. > >And, we don't have any benchmarks for the common case, yet. > >What kind of program would actually benefit from sparse files? And shouldn't >it be the responsibility of that program to request them? IIRC, linux creates sparse files automatically when you do an lseek to a position beyond EOF. I believe that Windows is similar. I would like to see a benchmark but I doubt there will be any noticeable difference. cgf
RE: Create new files as sparse on NT systems. (2nd try)
[snip] > >What kind of program would actually benefit from sparse > files? And shouldn't > >it be the responsibility of that program to request them? > > IIRC, linux creates sparse files automatically when you do an lseek to > a position beyond EOF. I believe that Windows is similar. > No, Windows does nothing like that. On Windows, sparse files are completely non-automatic: "Note: It is up to the application to maintain sparseness by writing zeros with FSCTL_SET_ZERO_DATA", sez the Platform docs. Even if you do WriteFile()s with all zeros on a sparse file, you are actually hitting the disk. > I would like to see a benchmark but I doubt there will be any > noticeable > difference. The only thing this patch will do AFAICS is set a bit somewhere in the guts of NTFS that will be pretty much ignored. I'm with Max, I don't see the benefit and can only imagine the consequences. -- Gary R. Van Sickle Braemar Inc. 11481 Rupp Dr. Burnsville, MN 55337
RE: Create new files as sparse on NT systems. (2nd try)
On Tue, 18 Feb 2003, Gary R Van Sickle wrote: > "Note: It is up to the application to maintain sparseness by writing zeros > with FSCTL_SET_ZERO_DATA", sez the Platform docs. In this respect Windows are ahead of any recent Unix system. I wasn't able find any Unix/Posix syscall that would allow this unlike Windows. > Even if you do WriteFile()s with all zeros on a sparse file, you are > actually hitting the disk. Have you ever tryed the same thing in Unix environment? Writing buffer full of zeros with write syscall won't gain you anything either. All the zeros will be physicaly written onto the disk. This means it has the same behaviour as Unix systems. > The only thing this patch will do AFAICS is set a bit somewhere in the guts > of NTFS that will be pretty much ignored. I'm with Max, I don't see the > benefit and can only imagine the consequences. I don't see any negative consequences of this patch. The only one I can imagine it can slow down file operations but I very very doubt it. Vaclav Haisman
Re: Create new files as sparse on NT systems. (2nd try)
Vaclav Haisman wrote: > On Tue, 18 Feb 2003, Gary R Van Sickle wrote: > >> "Note: It is up to the application to maintain sparseness by writing >> zeros with FSCTL_SET_ZERO_DATA", sez the Platform docs. > > In this respect Windows are ahead of any recent Unix system. I wasn't > able find any Unix/Posix syscall that would allow this unlike Windows. > >> Even if you do WriteFile()s with all zeros on a sparse file, you are >> actually hitting the disk. > > Have you ever tryed the same thing in Unix environment? Writing > buffer full of zeros with write syscall won't gain you anything > either. All the zeros will be physicaly written onto the disk. This > means it has the same behaviour as Unix systems. > >> The only thing this patch will do AFAICS is set a bit somewhere in >> the guts of NTFS that will be pretty much ignored. I'm with Max, I >> don't see the benefit and can only imagine the consequences. > > I don't see any negative consequences of this patch. The only one I > can imagine it can slow down file operations but I very very doubt it. Could you do some tests, so we have more than conjecture to go on? What programs actually *benefit* from sparseness? Max.
Re: Create new files as sparse on NT systems. (2nd try)
> Could you do some tests, so we have more than conjecture to go on? > > What programs actually *benefit* from sparseness? My primary motivation to do this is that I use P2P sharing program called BitTorrent. This program is written in Python and I run it in Cygwin. This program first creates whole file that I want to download by writing very few bytes with long distances between them and then fills it as it downloads chunks of the file from various other peers. The creation of this file takes from tens of seconds to few minutes without this patch, depending on size of the file. But with this patch it takes about two seconds to create this almost empty file. The files I usually download are movies. I don't experience any extra slowness while playing such created files. Vaclav Haisman
Re: Create new files as sparse on NT systems. (2nd try)
Vaclav Haisman wrote: >> Could you do some tests, so we have more than conjecture to go on? >> >> What programs actually *benefit* from sparseness? > > My primary motivation to do this is that I use P2P sharing program > called BitTorrent. This program is written in Python and I run it in > Cygwin. This program first creates whole file that I want to download > by writing very few bytes with long distances between them and then > fills it as it downloads chunks of the file from various other peers. > The creation of this file takes from tens of seconds to few minutes > without this patch, depending on size of the file. But with this > patch it takes about two seconds to create this almost empty file. > The files I usually download are movies. I don't experience any extra > slowness while playing such created files. Well, why not have BitTorrent set the file as sparse? Why should Cygwin do this for *all* files? - if it was universally advantageous, I would imagine it would be on by default in Windows. Max.
Re: Create new files as sparse on NT systems. (2nd try)
On Wed, 19 Feb 2003, Max Bowsher wrote: > Well, why not have BitTorrent set the file as sparse? Because it runs as Cygwin app which is Unix-like environment. There is no way to set files sparse in Unix because all files are sparse if the file systems supports it. Vaclav Haisman
Re: Create new files as sparse on NT systems. (2nd try)
On Wed, Feb 19, 2003 at 02:19:50AM +0100, Vaclav Haisman wrote: >On Wed, 19 Feb 2003, Max Bowsher wrote: >>Well, why not have BitTorrent set the file as sparse? > >Because it runs as Cygwin app which is Unix-like environment. There is >no way to set files sparse in Unix because all files are sparse if the >file systems supports it. ...which is, coincidentally enough, why I was interested in the patch. cgf
Re: Create new files as sparse on NT systems. (2nd try)
Christopher Faylor wrote: Because it runs as Cygwin app which is Unix-like environment. There is no way to set files sparse in Unix because all files are sparse if the file systems supports it. ...which is, coincidentally enough, why I was interested in the patch. It seems like a win to me -- UNIX files are always sparse by default: if you seek to a location and write, blocks are filled in only at the location of the write. One thing to investigate would be what happens when Windows trys to mmap() a sparse file. It doesn't bother a UNIX box, but Windows? Perhaps that is what BitTorrent is presently doing? -- Joe Buehler
Re: Create new files as sparse on NT systems. (2nd try)
On Wed, Feb 19, 2003 at 10:39:17AM -0500, Joe Buehler wrote: >Christopher Faylor wrote: > >>>Because it runs as Cygwin app which is Unix-like environment. There is >>>no way to set files sparse in Unix because all files are sparse if the >>>file systems supports it. >> >>...which is, coincidentally enough, why I was interested in the patch. > >It seems like a win to me -- UNIX files are always sparse by default: if you >seek to a location and write, blocks are filled in only at the location of >the write. > >One thing to investigate would be what happens when Windows trys to mmap() >a sparse file. It doesn't bother a UNIX box, but Windows? Perhaps that >is what BitTorrent is presently doing? Hmm. Good point. Does some brave soul want to apply the patch and do some experimenting? cgf
Re: Create new files as sparse on NT systems. (2nd try)
On Wed, Feb 19, 2003 at 12:57:38PM -0500, Christopher Faylor wrote: > On Wed, Feb 19, 2003 at 10:39:17AM -0500, Joe Buehler wrote: > >One thing to investigate would be what happens when Windows trys to mmap() > >a sparse file. It doesn't bother a UNIX box, but Windows? Perhaps that > >is what BitTorrent is presently doing? > > Hmm. Good point. Does some brave soul want to apply the patch and do some > experimenting? I'm going to test that *shiver*. Just a hint: Since we're calling GetVolumeInformation in path.cc anyway, it could make sense to remember the sparseness of the FS at that point. So fhandler_disk_file::open() could use that result when thinking about calling that new DeviceIoControl() call. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Developermailto:[EMAIL PROTECTED] Red Hat, Inc.
Re: Create new files as sparse on NT systems. (2nd try)
On Wed, Feb 19, 2003 at 08:41:35PM +0100, Corinna Vinschen wrote: > On Wed, Feb 19, 2003 at 12:57:38PM -0500, Christopher Faylor wrote: > > On Wed, Feb 19, 2003 at 10:39:17AM -0500, Joe Buehler wrote: > > >One thing to investigate would be what happens when Windows trys to mmap() > > >a sparse file. It doesn't bother a UNIX box, but Windows? Perhaps that > > >is what BitTorrent is presently doing? > > > > Hmm. Good point. Does some brave soul want to apply the patch and do some > > experimenting? > > I'm going to test that *shiver*. Well... as a first result: I tried to write two 4K blocks with a lseek(16K), creating a 24K file which should use only 8K on disk. It didn't work. Unfortunaltely, NTFS5 seem to support sparseness only if the files are big enough or if the sparse blocks are big enough. The above test worked for me beginning with an lseek of 129K. > Just a hint: Since we're calling GetVolumeInformation in path.cc > [bla bla bla] Apparently I didn't look into the patch. :-P Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Developermailto:[EMAIL PROTECTED] Red Hat, Inc.
Re: Create new files as sparse on NT systems. (2nd try)
On Wed, Feb 19, 2003 at 09:19:25PM +0100, Corinna Vinschen wrote:
> > I'm going to test that *shiver*.
>
> Well... as a first result: I tried to write two 4K blocks with a
> lseek(16K), creating a 24K file which should use only 8K on disk.
>
> It didn't work.
>
> Unfortunaltely, NTFS5 seem to support sparseness only if the files
> are big enough or if the sparse blocks are big enough. The above
> test worked for me beginning with an lseek of 129K.
This isn't what MSDN says. They have an example which shows a 32Megs
file, taking only 8K on disk. Somehow they didn't implement it exactly
according to their advertisement texts. *shrug*
Ok, my test case:
SNIP
#include
#include
#include
#include
#include
#include
#include
#include
int
main()
{
off_t seek = 184;
int buf[4096];
char *map;
DWORD len;
int fd = open ("sparse.test", O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (fd < 0)
{
printf ("open failed: %d \"%s\"\n", errno, strerror(errno));
return 1;
}
// Write first block
memset (buf, 1, 4096);
write (fd, buf, 4096);
// Seek 184K
lseek (fd, seek * 1024, SEEK_CUR);
// Write second block
memset (buf, 2, 4096);
write (fd, buf, 4096);
// Print size values
len = GetFileSize ((HANDLE)get_osfhandle (fd), NULL);
printf ("Size: %7luK\n", len >> 10);
len = GetCompressedFileSize ("sparse.test", NULL);
printf ("Size on disk: %7luK\n", len >> 10);
close (fd);
// Reopen for mmap
fd = open ("sparse.test", O_RDWR);
if (fd < 0)
{
printf ("open 2 failed: %d \"%s\"\n", errno, strerror(errno));
return 1;
}
map = mmap (NULL, (seek + 8) * 1024, PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0);
if (!map)
{
printf ("mmap failed: %d \"%s\"\n", errno, strerror(errno));
return 1;
}
else
{
unsigned long i;
// Check contents
for (i = 0; i < 4096; ++i)
if (map[i] != 1)
printf ("first page doesn't contain 1 in byte %lu\n", i);
for (i = 4096; i < (seek + 8) * 1024 - 4096; ++i)
if (map[i] != 0)
printf ("sparse pages don't contain 0 in byte %lu\n", i);
for (i = (seek + 8) * 1024 - 4096; i < (seek + 8) * 1024; ++i)
if (map[i] != 2)
printf ("last page doesn't contain 3 in byte %lu\n", i);
// Write a 3 in the middle
map[((seek + 8) * 1024) / 2] = 3;
munmap (map, (seek + 8) * 1024);
// Print new size values
len = GetFileSize ((HANDLE)get_osfhandle (fd), NULL);
printf ("Size: %7luK\n", len >> 10);
len = GetCompressedFileSize ("sparse.test", NULL);
printf ("Size on disk: %7luK\n", len >> 10);
}
close (fd);
return 0;
}
SNAP
This testcase worked pretty well. The output of the first size infos:
Size: 192K
Size on disk: 64K
Creating the mmap works fine, checking all blocks returned the expected
result (no messages) and writing the byte in the middle of the block
changed the file size info to
Size: 192K
Size on disk: 128K
which, btw., does *not* correspond to the `Size' and `Size on disk'
info in the Explorer file properties dialog. It stated:
Size: 192K
Size on disk: 128K
before and
Size: 192K
Size on disk: 192K
after writing the byte in the middle.
However, this shows a flaw in stat(): st_blocks is just a computed
value which we get from the size of the file returned by Windows.
The actual value of bytes allocated on the FS is returned by the
GetCompressedFileSize() function, for all files, regardless of normal,
compressed or sparsed.
Since st_blocks contains the number of blocks allocated, according to
the Linux man page and SUSv3, shouldn't we change st_blocks to reflect
the value of GetCompressedFileSize() now?
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Developermailto:[EMAIL PROTECTED]
Red Hat, Inc.
Re: Create new files as sparse on NT systems. (2nd try)
On Tue, Feb 18, 2003 at 10:51:31PM +0100, Vaclav Haisman wrote: > 2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]> > > * include/winioctl.h (FSCTL_SET_SPARSE): Define. > > 2003-02-18 Vaclav Haisman <[EMAIL PROTECTED]> > > * wincap.h (wincaps::supports_sparse_files): New flag. > (wincapc::supports_sparse_files): New method. > * wincap.cc (wincap_unknown): Define value for the new flag. > (wincap_95): Ditto. > (wincap_95osr2): Ditto. > (wincap_98): Ditto. > (wincap_98se): Ditto. > (wincap_me): Ditto. > (wincap_nt3): Ditto. > (wincap_nt4): Ditto. > (wincap_nt4sp4): Ditto. > (wincap_2000): Ditto. > (wincap_xp): Ditto. > * path.h (path_conv::fs_flags): New method. > * fhandler_disk_file.cc: Include winioctl.h for DeviceIoControl. > (fhandler_disk_file::open): Set newly created and truncated files as > sparse on platforms that support it. I've applied that patch now. If anybody still has problems with it, please feel free to post a *testcase* which shows the problem. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Developermailto:[EMAIL PROTECTED] Red Hat, Inc.
Re: Create new files as sparse on NT systems. (2nd try)
On Wed, Feb 19, 2003 at 10:06:37PM +0100, Corinna Vinschen wrote: > Since st_blocks contains the number of blocks allocated, according to > the Linux man page and SUSv3, shouldn't we change st_blocks to reflect > the value of GetCompressedFileSize() now? I've checked in a patch so st_blocks is now computed using GetCompressedFileSize() on NT systems. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Developermailto:[EMAIL PROTECTED] Red Hat, Inc.
Re: Create new files as sparse on NT systems. (2nd try)
On Thu, Feb 20, 2003 at 11:16:06AM +0100, Corinna Vinschen wrote: >On Tue, Feb 18, 2003 at 10:51:31PM +0100, Vaclav Haisman wrote: >> 2003-02-17 Vaclav Haisman <[EMAIL PROTECTED]> >> >> * include/winioctl.h (FSCTL_SET_SPARSE): Define. >> >> 2003-02-18 Vaclav Haisman <[EMAIL PROTECTED]> >> >> * wincap.h (wincaps::supports_sparse_files): New flag. >> (wincapc::supports_sparse_files): New method. >> * wincap.cc (wincap_unknown): Define value for the new flag. >> (wincap_95): Ditto. >> (wincap_95osr2): Ditto. >> (wincap_98): Ditto. >> (wincap_98se): Ditto. >> (wincap_me): Ditto. >> (wincap_nt3): Ditto. >> (wincap_nt4): Ditto. >> (wincap_nt4sp4): Ditto. >> (wincap_2000): Ditto. >> (wincap_xp): Ditto. >> * path.h (path_conv::fs_flags): New method. >> * fhandler_disk_file.cc: Include winioctl.h for DeviceIoControl. >> (fhandler_disk_file::open): Set newly created and truncated files as >> sparse on platforms that support it. > >I've applied that patch now. If anybody still has problems with it, >please feel free to post a *testcase* which shows the problem. Um. I was still hoping for benchmarks indicating that there was no performance hit from this patch. Also, this patch introduces a change to wincap which is never used. We don't need to change wincap. cgf cgf
Re: Create new files as sparse on NT systems. (2nd try)
On Thu, Feb 20, 2003 at 10:25:00AM -0500, Christopher Faylor wrote: > On Thu, Feb 20, 2003 at 11:16:06AM +0100, Corinna Vinschen wrote: > >I've applied that patch now. If anybody still has problems with it, > >please feel free to post a *testcase* which shows the problem. > > Um. I was still hoping for benchmarks indicating that there was no performance > hit from this patch. Also, this patch introduces a change to wincap which is > never used. We don't need to change wincap. I can quickly remove the wincap change (I didn't realize it isn't used after all the discussions here). However, since some people were objecting so doggedly, I was thinking that committing this patch would give a hint that a testcase(tm) is more useful than just musing. Everybody who wants this patch removed can accomplish this by showing us the testcase and the oh so bad results... Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Developermailto:[EMAIL PROTECTED] Red Hat, Inc.
Re: Create new files as sparse on NT systems. (2nd try)
On Thu, Feb 20, 2003 at 04:52:22PM +0100, Corinna Vinschen wrote: >However, since some people were objecting so doggedly, I was thinking >that committing this patch would give a hint that a testcase(tm) is more >useful than just musing. Everybody who wants this patch removed can >accomplish this by showing us the testcase and the oh so bad results... Ok. Makes sense. cgf
