Re: xargs completely broken under cygwin 3.6.0-0.108.gb7f5a33200a9
No need for attachments, the results are simply the same failure: > Could you please try running: > ... $ xargs -t --show-limits < /dev/null xargs: Unexpected suffix cmdline on cmdline > and it would be useful if you could run a failing command under strace e.g. > ... $ strace -o xargs.strace xargs -t --show-limits < /dev/null xargs: Unexpected suffix cmdline on cmdline > Contents of /proc/.../cmdline is an argv array: > ... I get the same thing you showed: $ cat -A /proc/self/cmdline; echo cat^@-A^@/proc/self/cmdline^@ The runtime of the failed xargs is so short I haven't been able to capture its /proc/*/cmdline, but I did get one from the 'strace ...": D:\cygwin64\bin\strace.exe^@-o^@xargs.strace^@xargs^@-t^@--show-limits I can try older test versions of cygwin, to see where it first shows. -- Bruce -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: xargs completely broken under cygwin 3.6.0-0.108.gb7f5a33200a9
Sorry, I missed the xargs.strace file; it's included as an attachment. --- Process 7020 created --- Process 7020 loaded C:\Windows\System32\ntdll.dll at 7ff99e03 --- Process 7020 loaded C:\Windows\System32\kernel32.dll at 7ff99ca4 --- Process 7020 loaded C:\Windows\System32\KernelBase.dll at 7ff99b49 --- Process 7020 loaded C:\Windows\System32\apphelp.dll at 7ff99885 --- Process 7020 loaded D:\cygwin64\bin\cygwin1.dll at 7ff96330 --- Process 7020 thread 6896 created --- Process 7020 thread 9828 created --- Process 7020 loaded D:\cygwin64\bin\cygintl-8.dll at 0005ee2d --- Process 7020 loaded D:\cygwin64\bin\cygiconv-2.dll at 0003eee6 --- Process 7020 thread 9796 created 4 4 [main] xargs (7020) ** 651 655 [main] xargs (7020) Program name: D:\cygwin64\bin\xargs.exe (windows pid 7020) 291 946 [main] xargs (7020) OS version: Windows NT-10.0 1931139 [main] xargs (7020) ** --- Process 7020 loaded C:\Windows\System32\advapi32.dll at 7ff99cea --- Process 7020 loaded C:\Windows\System32\msvcrt.dll at 7ff99c98 --- Process 7020 loaded C:\Windows\System32\sechost.dll at 7ff99d20 --- Process 7020 loaded C:\Windows\System32\bcrypt.dll at 7ff99b3c --- Process 7020 loaded C:\Windows\System32\rpcrt4.dll at 7ff99d65 --- Process 7020 loaded C:\Windows\System32\cryptbase.dll at 7ff99aa8 --- Process 7020 loaded C:\Windows\System32\bcryptprimitives.dll at 7ff99b84 13841 14980 [main] xargs (7020) sigprocmask: 0 = sigprocmask (0, 0x0, 0x7FF9635DE350) 2184 17164 [main] xargs (7020) open_shared: name shared.5, shared 0x1A000 (wanted 0x1A000), h 0x150, m 0, created 0 369 17533 [main] xargs (7020) user_heap_info::init: heap base 0xA, heap top 0xA, heap size 0x2000 (536870912) 3005 20538 [main] xargs (7020) open_shared: name S-1-5-21-4021324372-3308885545-1226283485-1002.1, shared 0x1A001 (wanted 0x1A001), h 0x148, m 1, created 0 401 20939 [main] xargs (7020) user_info::create: opening user shared for 'S-1-5-21-4021324372-3308885545-1226283485-1002' at 0x1A001 556 21495 [main] xargs (7020) user_info::create: user shared version AB1FCCE8 496 21991 [main] xargs (7020) fhandler_pipe::create: name \\.\pipe\cygwin-f76db13c759b51fa-7020-sigwait, size 11440, mode PIPE_TYPE_MESSAGE 325 22316 [main] xargs (7020) fhandler_pipe::create: pipe read handle 0x15C 252 22568 [main] xargs (7020) fhandler_pipe::create: CreateFile: name \\.\pipe\cygwin-f76db13c759b51fa-7020-sigwait 344 22912 [main] xargs (7020) fhandler_pipe::create: pipe write handle 0x164 322 23234 [main] xargs (7020) dll_crt0_0: finished dll_crt0_0 initialization --- Process 7020 thread 9784 created 1652 24886 [sig] xargs (7020) SetThreadName: SetThreadDescription() failed. 1000 883 25769 [sig] xargs (7020) wait_sig: entering ReadFile loop, my_readsig 0x15C, my_sendsig 0x164 1437 27206 [main] xargs (7020) mount_info::conv_to_posix_path: conv_to_posix_path (D:\cygwin64\etc\setup, 0x0, no-add-slash) 502 27708 [main] xargs (7020) normalize_win32_path: D:\cygwin64\etc\setup = normalize_win32_path (D:\cygwin64\etc\setup) 217 27925 [main] xargs (7020) mount_info::conv_to_posix_path: /etc/setup = conv_to_posix_path (D:\cygwin64\etc\setup) 268 28193 [main] xargs (7020) time: 1712043393 = time(0x0) 440 28633 [main] xargs (7020) sigprocmask: 0 = sigprocmask (0, 0x0, 0xA0100) 1582 30215 [main] xargs (7020) _cygwin_istext_for_stdio: fd 0: not open 321 30536 [main] xargs (7020) _cygwin_istext_for_stdio: fd 1: not open 629 31165 [main] xargs (7020) _cygwin_istext_for_stdio: fd 2: not open 831 31996 [main] xargs (7020) open_shared: name cygpid.822, shared 0x1A002 (wanted 0x1A002), h 0x184, m 2, created 1 353 32349 [main] xargs (7020) time: 1712043393 = time(0x0) 380 32729 [main] xargs 822 pinfo::thisproc: myself dwProcessId 7020 313 33042 [main] xargs 822 environ_init: GetEnvironmentStrings returned 0x67D970 1045 34087 [main] xargs 822 win32env_to_cygenv: 0xA04B0: ALLUSERSPROFILE=C:\ProgramData 524 34611 [main] xargs 822 win32env_to_cygenv: 0xA04E0: APPDATA=C:\Users\bruce\AppData\Roaming 454 35065 [main] xargs 822 win32env_to_cygenv: 0xA0510: COMMONPROGRAMFILES=C:\Program Files\Common Files 535 35600 [main] xargs 822 win32env_to_cygenv: 0xA0550: COMPUTERNAME=BUGS-INSIDER 557 36157 [main] xargs 822 win32env_to_cygenv: 0xA0580: COMSPEC=C:\WINDOWS\system32\cmd.exe 413 36570 [main] xargs 822 win32env_to_cygenv: 0xA05B0: CYG=/E/cygwin-incoming 5217 41787 [main] xargs 822 win32env_to_cygenv: 0xA05D0: CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files 414 42201 [main] xargs 822 win32env_to_cygenv: 0xA0620: CommonProgramW6432=C:\P
Re: xargs completely broken under cygwin 3.6.0-0.108.gb7f5a33200a9
I just did binary-search regression tests on the available old test versions, and found this: OK: cygwin-3.6.0-0.86.gbfe2790e7bc4.tar.xz BAD: cygwin-3.6.0-0.92.g8bd6ba8f16ec.tar.xz - Bruce -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: util-linux-2.39.3-1: libblkid returns invalid physical_sector_size
Hi Christian, On 3/31/2024 1:11 AM, Christian Franke via Cygwin wrote: Testcase: # cygcheck -f /sbin/fdisk.exe util-linux-2.39.3-1 # /sbin/fdisk.exe -l /dev/sdd Disk /dev/sdd: 465.76 GiB, 500107862016 bytes, 976773168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 34359738880 bytes I/O size (minimum/optimal): 34359738880 bytes / 34359738880 bytes [...valuable investigation and patch suggestion elided...] Your suggested patch looks fine to me. I have added it to the patch deck for a new util-linux 2.39.3-2, which has just been uploaded. The patch allows fdisk.exe to report the three correct values in my limited testing. Thanks for the report and the patch! ..mark -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: util-linux-2.39.3-1: libblkid returns invalid physical_sector_size
Hi Mark, Mark Geisert via Cygwin wrote: Hi Christian, On 3/31/2024 1:11 AM, Christian Franke via Cygwin wrote: Testcase: # cygcheck -f /sbin/fdisk.exe util-linux-2.39.3-1 # /sbin/fdisk.exe -l /dev/sdd Disk /dev/sdd: 465.76 GiB, 500107862016 bytes, 976773168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 34359738880 bytes I/O size (minimum/optimal): 34359738880 bytes / 34359738880 bytes [...valuable investigation and patch suggestion elided...] Your suggested patch looks fine to me. I have added it to the patch deck for a new util-linux 2.39.3-2, which has just been uploaded. The patch allows fdisk.exe to report the three correct values in my limited testing. Thanks for the report and the patch! You're welcome. BTW, according to the Linux kernel sources, BLKPBSZGET etc return 'unsigned int' and not 'unsigned long' since first appearance in 2.6.32-rc3 (2009?): https://elixir.bootlin.com/linux/v2.6.32-rc3/source/block/ioctl.c#L276 https://elixir.bootlin.com/linux/v2.6.32-rc3/source/block/compat_ioctl.c#L743 https://elixir.bootlin.com/linux/v6.8.2/source/block/ioctl.c#L533 So I don't understand why the mentioned code would be correct for Linux. -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: xargs completely broken under cygwin 3.6.0-0.108.gb7f5a33200a9
On Apr 2 01:42, Bruce Jerrick via Cygwin wrote: > I just did binary-search regression tests on the available > old test versions, and found this: > > OK: cygwin-3.6.0-0.86.gbfe2790e7bc4.tar.xz > BAD: cygwin-3.6.0-0.92.g8bd6ba8f16ec.tar.xz Thanks, I found the offending patch, but I have to think about a solution. Stay tuned, Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: util-linux-2.39.3-1: libblkid returns invalid physical_sector_size
> Downgrading to util-linux-2.33.3-3 does not help. The related code > differs, but has the same problem. But it was OK in util-linux-2.33.1-3 . The only difference in output between that and the fixed 2.39.3-2 is that the latter includes one more decimal place in the reported "human" sizes (GiB, TiB), which is very welcome. -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: xargs completely broken under cygwin 3.6.0-0.108.gb7f5a33200a9
On Apr 2 12:44, Corinna Vinschen via Cygwin wrote: > On Apr 2 01:42, Bruce Jerrick via Cygwin wrote: > > I just did binary-search regression tests on the available > > old test versions, and found this: > > > > OK: cygwin-3.6.0-0.86.gbfe2790e7bc4.tar.xz > > BAD: cygwin-3.6.0-0.92.g8bd6ba8f16ec.tar.xz > > Thanks, I found the offending patch, but I have to think about a > solution. > > Stay tuned, > Corinna Just building cygwin-3.6.0-0.109.ga0a25849f9dd, should be up in an hour or two. Please give it a try. Thanks, Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin&Win32 file prefetch, block sizes?
On Apr 2 02:04, Martin Wege via Cygwin wrote: > Hello, > > Is there any document which describes how Cygwin and Win32 file > prefetch and readahead work, and which sizes are used (e.g. always > read one full page even if only 16 bytes are requested?)? I'm not aware of any docs, but again, keep in mind that Cygwin is a usersapce DLL. We basically do what Windows does for low-level file access. > Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does that > mean the file's block size is really 64k? Is this info per filesystem, > or hardcoded in Cygwin? Hardcoded in Cygwin since 2017, based on a discussion in terms of file access performance, especially when using stdio.h functions: https://cygwin.com/cgit/newlib-cygwin/commit/?id=7bef7db5ccd9c Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: xargs completely broken under cygwin 3.6.0-0.108.gb7f5a33200a9
'xargs' is back to working with cygwin-3.6.0-0.109.ga0a25849f9dd . Thanks for the quick fix! -- Bruce -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: util-linux-2.39.3-1: libblkid returns invalid physical_sector_size
Bruce Jerrick via Cygwin wrote: Downgrading to util-linux-2.33.3-3 does not help. The related code differs, but has the same problem. I take that back. The above should read "util-linux-2.33.1-3". But it was OK in util-linux-2.33.1-3 . Yes, this is correct. I possibly downgraded util-linux, but forgot libblkid1. -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: util-linux-2.39.3-1: libblkid returns invalid physical_sector_size
Christian Franke via Cygwin wrote: ,,, BTW, according to the Linux kernel sources, BLKPBSZGET etc return 'unsigned int' and not 'unsigned long' since first appearance in 2.6.32-rc3 (2009?): https://elixir.bootlin.com/linux/v2.6.32-rc3/source/block/ioctl.c#L276 https://elixir.bootlin.com/linux/v2.6.32-rc3/source/block/compat_ioctl.c#L743 https://elixir.bootlin.com/linux/v6.8.2/source/block/ioctl.c#L533 So I don't understand why the mentioned code would be correct for Linux. It is likely an upstream regression from an 1+ year old commit. I filed a GH issue: https://github.com/util-linux/util-linux/issues/2904 -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: xargs completely broken under cygwin 3.6.0-0.108.gb7f5a33200a9
On Apr 2 08:38, Bruce Jerrick via Cygwin wrote: > 'xargs' is back to working with cygwin-3.6.0-0.109.ga0a25849f9dd . > > Thanks for the quick fix! > -- Bruce Thanks for the report and testing! Corinna -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin&Win32 file prefetch, block sizes?
On Tue, Apr 2, 2024 at 3:17 PM Corinna Vinschen via Cygwin wrote: > > On Apr 2 02:04, Martin Wege via Cygwin wrote: > > Hello, > > > > Is there any document which describes how Cygwin and Win32 file > > prefetch and readahead work, and which sizes are used (e.g. always > > read one full page even if only 16 bytes are requested?)? > > I'm not aware of any docs, but again, keep in mind that Cygwin is a > usersapce DLL. We basically do what Windows does for low-level file > access. > > > Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does that > > mean the file's block size is really 64k? Is this info per filesystem, > > or hardcoded in Cygwin? > > Hardcoded in Cygwin since 2017, based on a discussion in terms of > file access performance, especially when using stdio.h functions: > > https://cygwin.com/cgit/newlib-cygwin/commit/?id=7bef7db5ccd9c OUCH. While I can understand the motivation, FAT32 on multi-GB-devices having 64k block size, and Win32 API on Win95/98/ME/Win7 being optimized to that insane block size, it is absolutely WRONG with today's NTFS and even more so with ReFS. This only works if you stream files, but as soon as you are doing random read/writes the performance is terrible due to cache thrashing. That could explain the many complaints about Cygwin's IO performance. So, what can be done? I'm not a benchmarking guru, so I'd like to propose to add a tunable called EXPERIMENTAL_PREFERRED_IO_BLKSIZE to the CYGWIN env variable (marked as "experimental"), so the benchmarking guys can do performance testing without recompiling everything, get perf results for Cygwin 3.6, and decide what to do for Cygwin 3.7. Thanks, Martin -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin&Win32 file prefetch, block sizes?
On 4/2/2024 3:35 PM, Martin Wege via Cygwin wrote: On Tue, Apr 2, 2024 at 3:17 PM Corinna Vinschen via Cygwin wrote: On Apr 2 02:04, Martin Wege via Cygwin wrote: Hello, Is there any document which describes how Cygwin and Win32 file prefetch and readahead work, and which sizes are used (e.g. always read one full page even if only 16 bytes are requested?)? I'm not aware of any docs, but again, keep in mind that Cygwin is a usersapce DLL. We basically do what Windows does for low-level file access. Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does that mean the file's block size is really 64k? Is this info per filesystem, or hardcoded in Cygwin? Hardcoded in Cygwin since 2017, based on a discussion in terms of file access performance, especially when using stdio.h functions: https://cygwin.com/cgit/newlib-cygwin/commit/?id=7bef7db5ccd9c OUCH. While I can understand the motivation, FAT32 on multi-GB-devices having 64k block size, and Win32 API on Win95/98/ME/Win7 being optimized to that insane block size, it is absolutely WRONG with today's NTFS and even more so with ReFS. This only works if you stream files, but as soon as you are doing random read/writes the performance is terrible due to cache thrashing. That could explain the many complaints about Cygwin's IO performance. No comment. So, what can be done? I'm not a benchmarking guru, so I'd like to propose to add a tunable called EXPERIMENTAL_PREFERRED_IO_BLKSIZE to the CYGWIN env variable (marked as "experimental"), so the benchmarking guys can do performance testing without recompiling everything, get perf results for Cygwin 3.6, and decide what to do for Cygwin 3.7. That kind of experiment is what folks who can build their own cygwin1.dll might do. I doubt we'd want to make a run-time global disk I/O strategy changer available like this, even temporarily. What could make sense is enhancing Cygwin's posix_fadvise() to support POSIX_FADV_RANDOM getting mapped to Windows' FILE_RANDOM_ACCESS flag. Something like this is currently done for POSIX_FADV_SEQUENTIAL -> FILE_SEQUENTIAL_ONLY. These are per-filedescriptor adjustments and due to Windows limitations would apply to a whole file rather than having the POSIX behavior of being settable for a byte range within a file. SHTDI, PTC, and all that :-). ..mark -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Implementing /bin/ionice, ioprio_set() support with FILE_IO_PRIORITY_HINT_INFO?
Hello,
could Cygwin implement support for /usr/bin/ionice and ioprio_set()
via FILE_IO_PRIORITY_HINT_INFO?
So basically implement ioprio_set() to store the value per process,
and for each file open() call this:
FILE_IO_PRIORITY_HINT_INFO priorityHint={0};
priorityHint.PriorityHint = ioniceprio2WinIoPriorityHint(ioniceprio);
result = SetFileInformationByHandle( hFile,
FileIoPriorityHintInfo,
&priorityHint,
sizeof(PriorityHint));
NTFS and ReFS support FILE_IO_PRIORITY_HINT_INFO
Thanks,
Martin
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: cygwin utils can access directory and its contents, but W10 utils claim to have no access, why?
On Mon, Jan 22, 2024 at 10:59 AM John Ruckstuhl
wrote:
>
> Thanks for the replies Brian & Corinna, I learned a lot.
>
> On Mon, Jan 22, 2024 at 1:48 AM Corinna Vinschen
> wrote:
> > Users in the Administrators group have these privileges in their user
> > token. Under UAC, both privileges are removed from the token. In an
> > elevated shell, though, both privileges are present.
> >
> > The funny thing here is this: While both privileges are present in the
> > token, they are disabled by default.
> >
> > They have to be enabled explicitely before you can exercise the
> > privileges. Usually you do this in the same application
> > programatically.
>
> Now I see...
> Local administrator Bob reports his User Access Token info
> (with Windows whoami, not cygwin whoami)
> C:\>whoami /priv
>
> PRIVILEGES INFORMATION
> --
>
> Privilege Name... State
> = ...
> ...
> SeBackupPrivilege ... Disabled
> SeRestorePrivilege... Disabled
> ...
>
> C:\>
>
> Thanks very much.
> John Ruckstuhl
The clue about UAT and disabled privileges was crucial.
Thank you again, Corinna.
I'm still doing my cleanup with ordinary Python (not the Cygwin Python
linked to the cygwin dll).
But I wrote an EnablePrivilege function to enable the privs if if necessary.
Best regards,
John Ruckstuhl
1 """
2 This module exports functions for removing old files and directories.
3
4 Functions
5 myremove -- Remove the file or dir, and return the number
of bytes freed.
6 """
7
8 # standard library imports
9 import logging
10 import os
11 import shutil
12 import subprocess
13
14 # related third party imports
15 import win32api
16 import win32security
17
18 # no local application/library specific imports
19
20
21 logger = logging.getLogger(__name__)
22
23 # this custom startupinfo is used to suppress display of the
subprocess window
24 startupinfo = subprocess.STARTUPINFO()
25 startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
26
27
28 def _remove(path):
29 """Remove the file or dir, and return the number of bytes freed."""
30
31 ...
32
33
34 def myremove(path):
35 """Remove the file or dir, and return the number of bytes freed.
36
37 Attempt removal different ways, until successful (or defeated).
38
39 1. Attempt call _remove (without additional prep).
40 2. Attempt to enable privileges SeBackupPrivilege and
41 SeRestorePrivilege, then call _remove.
42 3. Attempt to takeown, then call _remove.
43
44 The enable privileges approach is ~6x faster than the
takeown approach,
45 because takeown of an MEI dir containing ~1K files takes
time (8-9 sec).
46
47 Example results, enable privs approach
48 MEI dirs: 113/113 removed, 2047.5 MB recovered in 2.0 minutes
49
50 Example results, takeown approach
51 MEI dirs: 126/126 removed, 2453.8 MB recovered in 15.0 minutes
52
53 By default, the User Access Token for a member of local group
54 Administrators has the privileges SeBackupPrivilege and
SeRestorePrivilege,
55 but they are disabled.
56
57 The MEI dirs are typically ~1000 files occupying ~20 MB.
58 The MEI dirs are created by users running the pyinstaller -generated
59 executables ...
60
61 Presumably the process that calls this function is run as a user
62 which is a member of local group "Administrators". So
this process will
63 have privileges SeBackupPrivilege and SeRestorePrivilege granted but
64 disabled. Also this process will be able to take
ownership, if needed.
65 """
66
67 logger.debug('%s(%r)', 'myremove', path)
68
69 # first try, attempt _remove
70 try:
71 logger.debug('%s(%r)', '_remove', path)
72 nbytes = _remove(path)
73 return nbytes
74 except WindowsError as e:
75 if e.args == (5, 'Access is denied'):
76 # report the exception and carry on to next attempt
77 logger.debug(e)
78 else:
79 # re-raise any other WindowsError
80 raise
81
82 # second try, attempt enable privs then _remove
83 try:
84 logger.debug('%s(%r)', '_remove', path)
85 EnablePrivilege(win32security.SE_BACKUP_NAME)
86 EnablePrivilege(win32security.SE_RESTORE_NAME)
87 nbytes = _remove(path)
88 return nbytes
89 except Exception as e:
90 # report the exce
Re: Cygwin&Win32 file prefetch, block sizes?
On Wed, 3 Apr 2024 at 00:36, Martin Wege via Cygwin wrote: > > On Tue, Apr 2, 2024 at 3:17 PM Corinna Vinschen via Cygwin > wrote: > > > > On Apr 2 02:04, Martin Wege via Cygwin wrote: > > > Hello, > > > > > > Is there any document which describes how Cygwin and Win32 file > > > prefetch and readahead work, and which sizes are used (e.g. always > > > read one full page even if only 16 bytes are requested?)? > > > > I'm not aware of any docs, but again, keep in mind that Cygwin is a > > usersapce DLL. We basically do what Windows does for low-level file > > access. > > > > > Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does that > > > mean the file's block size is really 64k? Is this info per filesystem, > > > or hardcoded in Cygwin? > > > > Hardcoded in Cygwin since 2017, based on a discussion in terms of > > file access performance, especially when using stdio.h functions: > > > > https://cygwin.com/cgit/newlib-cygwin/commit/?id=7bef7db5ccd9c > > OUCH. > > While I can understand the motivation, FAT32 on multi-GB-devices > having 64k block size, and Win32 API on Win95/98/ME/Win7 being > optimized to that insane block size, it is absolutely WRONG with > today's NTFS and even more so with ReFS. This only works if you stream > files, but as soon as you are doing random read/writes the performance > is terrible due to cache thrashing. That could explain the many > complaints about Cygwin's IO performance. > It can also explain why Cygwin is so slow on SMB filesystems. If it really works on 64k blocks, then this would pull all small files over the wire, even if only a bit gets touched. Thinking more about this, this basically defeats every driver/tcp/ip/ethernet/net/switch optimization like jumbo frames et al. /usr/bin/stat %o (optimum IO block size hint) also returns 65536 on SMB, which is NOT GOOD: This needs to be confirmed, and if this is really true, then it should be fixed for SMB. I need a coffee to think about this... > So, what can be done? I'm not a benchmarking guru, so I'd like to > propose to add a tunable called EXPERIMENTAL_PREFERRED_IO_BLKSIZE to > the CYGWIN env variable (marked as "experimental"), so the > benchmarking guys can do performance testing without recompiling > everything, get perf results for Cygwin 3.6, and decide what to do for > Cygwin 3.7. I would agree, but I would also clamp the minimum at page size (4096 bytes on x86) and 8M (4x 2M hugepage size) to prevent abuse. Does Cygwin partake in the GSOC programm (Google Summer Of Code)? This would IMO be a high priority item. Ced -- Cedric Blancher [https://plus.google.com/u/0/+CedricBlancher/] Institute Pasteur -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin&Win32 file prefetch, block sizes?
On Wed, 3 Apr 2024 at 03:10, Mark Geisert via Cygwin wrote: > > On 4/2/2024 3:35 PM, Martin Wege via Cygwin wrote: > > On Tue, Apr 2, 2024 at 3:17 PM Corinna Vinschen via Cygwin > > wrote: > >> > >> On Apr 2 02:04, Martin Wege via Cygwin wrote: > >>> Hello, > >>> > >>> Is there any document which describes how Cygwin and Win32 file > >>> prefetch and readahead work, and which sizes are used (e.g. always > >>> read one full page even if only 16 bytes are requested?)? > >> > >> I'm not aware of any docs, but again, keep in mind that Cygwin is a > >> usersapce DLL. We basically do what Windows does for low-level file > >> access. > >> > >>> Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does that > >>> mean the file's block size is really 64k? Is this info per filesystem, > >>> or hardcoded in Cygwin? > >> > >> Hardcoded in Cygwin since 2017, based on a discussion in terms of > >> file access performance, especially when using stdio.h functions: > >> > >>https://cygwin.com/cgit/newlib-cygwin/commit/?id=7bef7db5ccd9c > > > > OUCH. > > > > While I can understand the motivation, FAT32 on multi-GB-devices > > having 64k block size, and Win32 API on Win95/98/ME/Win7 being > > optimized to that insane block size, it is absolutely WRONG with > > today's NTFS and even more so with ReFS. This only works if you stream > > files, but as soon as you are doing random read/writes the performance > > is terrible due to cache thrashing. That could explain the many > > complaints about Cygwin's IO performance. > > No comment. > > > So, what can be done? I'm not a benchmarking guru, so I'd like to > > propose to add a tunable called EXPERIMENTAL_PREFERRED_IO_BLKSIZE to > > the CYGWIN env variable (marked as "experimental"), so the > > benchmarking guys can do performance testing without recompiling > > everything, get perf results for Cygwin 3.6, and decide what to do for > > Cygwin 3.7. > > That kind of experiment is what folks who can build their own > cygwin1.dll might do. I doubt we'd want to make a run-time global disk > I/O strategy changer available like this, even temporarily. Realistically that would mean that Cygwin will forever be stuck with an insane IO block size. Building Cygwin.dll requires specialised knowledge and TIME, and no manager will waste the time of a performance engineer to produce custom binaries. Cygwin 3.6 is right now in development, so it would be better to add such a knob, so performance engineers can just grab those binaries and do benchmarking with them. BTW: A block size of 64k is CLEARLY harming performance. Have a look at https://www.zabkat.com/blog/buffered-disk-access.htm the sweet spot is somewhere between 16k and 32k, for SMB even below that. 64k is clearly on the backside of the curve, and actively harming performance, except for "linear reads". > > What could make sense is enhancing Cygwin's posix_fadvise() to support > POSIX_FADV_RANDOM getting mapped to Windows' FILE_RANDOM_ACCESS flag. > Something like this is currently done for POSIX_FADV_SEQUENTIAL -> > FILE_SEQUENTIAL_ONLY. These are per-filedescriptor adjustments and due > to Windows limitations would apply to a whole file rather than having > the POSIX behavior of being settable for a byte range within a file. Nope. Because we are talking about a sensible default for all applications, and a block size of 64k is HARMFUL, except on fat32 where the filesystem block size is already 64k for multi gigabyte disks. Ced -- Cedric Blancher [https://plus.google.com/u/0/+CedricBlancher/] Institute Pasteur -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
