Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2024-06-07 Thread Roland Mainz via Cygwin
On Mon, Jan 8, 2024 at 2:56 PM Corinna Vinschen via Cygwin
 wrote:
>
> On Dec 18 13:04, Cedric Blancher via Cygwin wrote:
> > On Fri, 1 Sept 2023 at 13:00, Corinna Vinschen via Cygwin
> >  wrote:
> > >
> > > On Sep  1 06:23, Cedric Blancher via Cygwin wrote:
> > > > Good morning!
> > > >
> > > > During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
> > > > odd when I looked at the network traffic generated by one of our
> > > > cluster nodes:
> > > > It seems that for each call to a tool (i.e. starting "sed" from
> > > > "bash") Cygwin searches for *.lnk files.
> > > >
> > > > Is this correct even when the filesystem in question has native
> > > > symlink support (e.g. NFS)?
> > >
> > > Yes.  During file handling, Cygwin doesn't know what filesystem a
> > > file is on until it could actually open the file and request file
> > > and filesystem info from the open handle.
> >
> > Why? If you have the path name you could lookup the (cached) mount
> > points, and determine the filesystem type. Same solution applies for
> > UNC paths, where you can easily lookup the filesystem type, and cache
> > it per-process or in Cygserver.
>
> No can do.
>
> We *do* filesystem info caching, but this requires to be able to
> open the file in the first place.  If the file exists, we're done, but
> if not, we have to evaluate all symbolic links in the path first.
> Simply reducing the path to the mount point and then translate it into
> a Windows path doesn't cut it.
>
> After the file (or its parent dir) could be opened, so we know the path
> is valid, we can fetch the filesystem info right from the open file
> handle.  At this point, we can shortcut the whole thing, reducing
> the three necessary calls to kernel functions to only the first one
> and skipping all the filesystem evaluation code.
>
> > > So if Cygwin couldn't open
> > > "foo" because the NtCreateFile call returned with status
> > > STATUS_OBJECT_PATH_NOT_FOUND or STATUS_OBJECT_NAME_NOT_FOUND, or
> > > STATUS_NO_SUCH_FILE, or one of the countless other status codes the
> > > kernel (or the driver) might return in case a file doesn't exist,
> > > it will tack on .lnk and .exe and, for historical reasons, .exe.lnk,
> > > and try again.
> >
> > Can this machinery please be turned off via CYGWIN env var option? As
> > discussed in https://www.mail-archive.com/cygwin@cygwin.com/msg174547.html
> > this machinery causes very bad filesystem lookup performance, and it
> > would IMO a good idea to have an option to turn this off, and just
> > allow and expect native links (for NTFS, ReFS and NFS). Maybe CYGWIN
> > env var option winsymlinks_expect:native? winsymlinks_expect takes a :
> > seperated list which symlink types are to be expected.
>
> We won't add any more options to the CYGWIN env variable if it's not
> necessary, and there's no proof at all that this is necessary in this
> case.
>
> If the file exists as specified, there will be no further check for
> .exe or .lnk.  If it doesn't exist, *and* the return code from
> the kernel is STATUS_OBJECT_PATH_NOT_FOUND (or any one of a
> number of equivalent return values) we're done here and are going
> to fall back to checking for symlinks in the leading path components.
>
> If we got over that, the check for .exe is unavoidable.
[snip]

What about doing lookups in parallel, e.g. use |NtCreateFile()| with
an |IO_STATUS_BLOCK|, and do lookups for *.exe, *.lnk etc. in parallel
?
That might not help for all filesystem drivers, but some of them are
multithreaded (e.g. SMB, ms-nfs41-client, ...) and would greatly
benefit from that since the requests would go async&¶llel over the
network, instead of sequentially like it is now.



Bye,
Roland
-- 
  __ .  . __
 (o.\ \/ /.o) roland.ma...@nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2024-01-08 Thread Brian Inglis via Cygwin

On 2024-01-08 12:57, matthew patton via Cygwin wrote:

This breaks many applications such as the java runtime among others.

In any event "unreadable files" is a problem all over the place if I use 
Cygwin's /usr/bin/ln to create links. That's why I was forced to write a wrapper. Even if 
'JUNCTION' is false/misleading as to the root cause, plenty of 'DOS' programs, like the 
AWS CLI and Golang have major problems with the default behavior.
For a "compatibility" environment to spawn a slew of unusable filesystem 
artifacts is a MAJOR bug, IMO.


That is why you have env var settings to make things more compatible.
Read the docs and make the best choice for your situation.
Cygwin is compatible with MS Windows but expecting MS Windows to be compatible 
with anything else is expecting too much.

Just avoid using non-POSIX interfaces and you will be much better off.
You can then be confident that anything you do or have can easily be moved to 
another POSIX compatible system and used with only minor differences, unless you 
go crazy and use another proprietary system!


--
Take care. Thanks, Brian Inglis  Calgary, Alberta, Canada

La perfection est atteinte   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2024-01-08 Thread matthew patton via Cygwin
> This breaks many applications such as the java runtime among others.
In any event "unreadable files" is a problem all over the place if I use 
Cygwin's /usr/bin/ln to create links. That's why I was forced to write a 
wrapper. Even if 'JUNCTION' is false/misleading as to the root cause, plenty of 
'DOS' programs, like the AWS CLI and Golang have major problems with the 
default behavior.
For a "compatibility" environment to spawn a slew of unusable filesystem 
artifacts is a MAJOR bug, IMO.

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2024-01-08 Thread Corinna Vinschen via Cygwin
On Jan  8 14:17, Jeffrey Altman via Cygwin wrote:
> On 1/8/2024 1:44 PM, matthew patton via Cygwin wrote:
> > > Cygwin does not create symlinks as junctions.  No idea where you got that 
> > > idea.
> > $ echo $CYGWINwinsymlinks:nativestrict
> > $ /usr/bin/ln -s default.GGG6q test1
> > 01/08/2024  01:24 PM         test1 [...]Type=File
> > $ (unset CYGWIN; /usr/bin/ln -s default.GGG6q test2.nocygwin)
> > 01/08/2024  01:25 PM         test2.nocygwin [...]Type=File
> 
> JUNCTIONS are a type of reparse point tag. Many tools report things as
> JUNCTIONS when they don't know what else to call it because JUNCTIONS were
> the only type of reparse tag commonly used.
> [...]
> As part of this reply I will note that the NTFS symlinks differ from POSIX
> symlinks in significant ways
> 
> 1. A pre-existing file system object is required in order to attach a
>reparse tag
> 2. The type of the target must be known when the reparse tag is applied
>to a pre-existing file system object
> 3. The reparse tag may be removed and replaced any number of times
>without deleting the pre-existing object to which it is attached.  
>Whereas a POSIX symlink inode target cannot be altered once
>created.  The inode must be deleted and replaced.
> 4. The Windows file APIs do not behave as many applications expect them
>to when a symlink reparse tag is present.  For example, opening a
>file handle will traverse the reparse tag and open the target but
>the file info api when given the same path will return the
>information belonging to the object on which the reparse tag was
>applied.   This breaks many applications such as the java runtime
>among others.
> 
> Jeffrey Altman

Thanks, Jeffrey.  I couldn't have expressed it so nicely.  Very well
summarized.


Corinna

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2024-01-08 Thread Jeffrey Altman via Cygwin

On 1/8/2024 1:44 PM, matthew patton via Cygwin wrote:

Cygwin does not create symlinks as junctions.  No idea where you got that idea.

$ echo $CYGWINwinsymlinks:nativestrict
$ /usr/bin/ln -s default.GGG6q test1
01/08/2024  01:24 PM         test1 [...]Type=File
$ (unset CYGWIN; /usr/bin/ln -s default.GGG6q test2.nocygwin)
01/08/2024  01:25 PM         test2.nocygwin [...]Type=File


JUNCTIONS are a type of reparse point tag. Many tools report things as 
JUNCTIONS when they don't know what else to call it because JUNCTIONS 
were the only type of reparse tag commonly used.


This is the output of JP Software's Take Command for an AFS volume root 
directory


[\\afs\yfs]dir Directory of \\afs\yfs\* 5/26/2021 11:49  . 
11/26/2019 14:25  amd64_linux26 
[your-file-system.com#amd64_linux26] 11/24/2019 20:29  backups 
10/20/2022 10:43  project 12/05/2011 10:14  public 
[#root.public] 9/06/2020 9:27  service 7/26/2010 20:44  
support [#root.support] 6/15/2011 11:40  test [#test.test] 
2/15/2012 8:49  u 3/28/2023 12:50  user 11/28/2011 17:01 
 usr [user] 12/10/2009 0:34  www [#root.www] 
12/13/2016 1:29 30 autorun.inf 6/07/2015 21:54 104 Desktop.ini 
11/26/2019 14:42  local [@sys\usr\local] 148 bytes in 3 files 
and 12 dirs 0 bytes free


The JUNCTIONS are AFS volume mount points using the Microsoft designated 
reparse point for OpenAFS. The SYMLINKD is a symlink reparse point to a 
directory using the NTFS symlink reparse point. The SYMLINK is a symlink 
reparse point to an object of unknown type (which Windows expects to be 
a file not a directory) using the NTFS symlink reparse point.


Here is the cygwin ls -l output

[\\afs\yfs]c:\tools\cygwin\bin\ls -l . total 38 -rw-r--r-- 1 jaltm 
mkpasswd 104 Jun 7 2015 Desktop.ini drwxr-xr-x 1 jaltm mkpasswd 0 Nov 26 
2019 amd64_linux26 -rw-r--r-- 1 jaltm mkpasswd 30 Dec 13 2016 
autorun.inf drwxr-xr-x 1 jaltm mkpasswd 0 Nov 24 2019 backups lrwxrwxrwx 
1 jaltm mkpasswd 14 Nov 26 2019 local -> @sys/usr/local drwxr-xr-x 1 
jaltm mkpasswd 0 Oct 20 2022 project drwxr-xr-x 1 jaltm mkpasswd 0 May 
21 2023 public drwxr-xr-x 1 jaltm mkpasswd 0 Sep 6 2020 service 
drwxr-xr-x 1 jaltm mkpasswd 0 Nov 17 2015 support drwxr-xr-x 1 jaltm 
mkpasswd 0 May 26 2019 test drwxr-xr-x 1 jaltm mkpasswd 0 Feb 15 2012 u 
drwxr-xr-x 1 jaltm mkpasswd 0 Mar 28 2023 user lrwxrwxrwx 1 jaltm 
mkpasswd 4 Nov 28 2011 usr -> user drwxr-xr-x 1 jaltm mkpasswd 0 Jul 10 
2017 www


As part of this reply I will note that the NTFS symlinks differ from 
POSIX symlinks in significant ways


1. A pre-existing file system object is required in order to attach a
   reparse tag
2. The type of the target must be known when the reparse tag is applied
   to a pre-existing file system object
3. The reparse tag may be removed and replaced any number of times
   without deleting the pre-existing object to which it is attached.  
   Whereas a POSIX symlink inode target cannot be altered once
   created.  The inode must be deleted and replaced.
4. The Windows file APIs do not behave as many applications expect them
   to when a symlink reparse tag is present.  For example, opening a
   file handle will traverse the reparse tag and open the target but
   the file info api when given the same path will return the
   information belonging to the object on which the reparse tag was
   applied.   This breaks many applications such as the java runtime
   among others.

Jeffrey Altman



--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2024-01-08 Thread Rainer Emrich via Cygwin

Am 08.01.2024 um 19:44 schrieb matthew patton via Cygwin:

Cygwin does not create symlinks as junctions.  No idea where you got that idea.

$ echo $CYGWINwinsymlinks:nativestrict
$ /usr/bin/ln -s default.GGG6q test1
01/08/2024  01:24 PM         test1 [...]Type=File
$ (unset CYGWIN; /usr/bin/ln -s default.GGG6q test2.nocygwin)
01/08/2024  01:25 PM         test2.nocygwin [...]Type=File

I have CYGWIN=winsymlinks:native
A JUNCTION is created if you try to ln to nonexisting file or directory.
If you ln to an existing directory you get a SYMLINKD and for an 
existing file you get SYMLINK, as expected!


Rainer


--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2024-01-08 Thread matthew patton via Cygwin
> Cygwin does not create symlinks as junctions.  No idea where you got that 
> idea.
$ echo $CYGWINwinsymlinks:nativestrict
$ /usr/bin/ln -s default.GGG6q test1
01/08/2024  01:24 PM         test1 [...]Type=File
$ (unset CYGWIN; /usr/bin/ln -s default.GGG6q test2.nocygwin)
01/08/2024  01:25 PM         test2.nocygwin [...]Type=File


# 'ln' is a wrapper function around 'mklink' since Cygwin does the wrong 
thing.https://github.com/tb3088/shell-environment/blob/ccf7aa161899c2c4ebe2d9e980e674bc726a3ef3/.functions_os.CYGWIN_NT#L10

$ ln default.GGG6q test3.mklinksymbolic link created for test3.mklink <<===>> 
default.GGG6q01/08/2024  01:25 PM          test3.mklink 
[default.GGG6q]Type=.symlink

C:\Users\xxx>type default.GGG6qfoobar
C:\Users\xxx>type test1The file cannot be accessed by the system.
C:\Users\pattonma>type test2.nocygwinThe file cannot be accessed by the system.
C:\Users\xxx>type test3.mklinkfoobar

  

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2024-01-08 Thread Corinna Vinschen via Cygwin
On Jan  8 17:11, matthew patton via Cygwin wrote:
> > For instance: Getting rid of .lnk files isn't easy with backward 
> >compatibility in mind.
> screw backward compatability! :)Why carry around bandaids on bandaids
> for an OS that is 10 years out of support?

We're using .lnk files for FIFOs. That's the compat concern, not some OS
from the past.

> Obviously not ripe for 3.6 or maybe even 3.7, but at some point we
> should just put a stick in the sand and say this .lnk crap is dead and
> you have to either compile it back in with an IFDEF or a CYGWIN var.

No ifdefs, and a Cygwin var setting unfortunately exists.

> And while we're at it use OS-native links instead of Junctions. I'm
> not sure how that came about but Junctions are unhelpful.

Cygwin does not create symlinks as junctions.  No idea where you got
that idea.


Corinna

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2024-01-08 Thread matthew patton via Cygwin
> For instance: Getting rid of .lnk files isn't easy with backward 
>compatibility in mind.
screw backward compatability! :)Why carry around bandaids on bandaids for an OS 
that is 10 years out of support?
Obviously not ripe for 3.6 or maybe even 3.7, but at some point we should just 
put a stick in the sand and say this .lnk crap is dead and you have to either 
compile it back in with an IFDEF or a CYGWIN var. And while we're at it use 
OS-native links instead of Junctions. I'm not sure how that came about but 
Junctions are unhelpful.


-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2024-01-08 Thread Corinna Vinschen via Cygwin
On Dec 18 13:04, Cedric Blancher via Cygwin wrote:
> On Fri, 1 Sept 2023 at 13:00, Corinna Vinschen via Cygwin
>  wrote:
> >
> > On Sep  1 06:23, Cedric Blancher via Cygwin wrote:
> > > Good morning!
> > >
> > > During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
> > > odd when I looked at the network traffic generated by one of our
> > > cluster nodes:
> > > It seems that for each call to a tool (i.e. starting "sed" from
> > > "bash") Cygwin searches for *.lnk files.
> > >
> > > Is this correct even when the filesystem in question has native
> > > symlink support (e.g. NFS)?
> >
> > Yes.  During file handling, Cygwin doesn't know what filesystem a
> > file is on until it could actually open the file and request file
> > and filesystem info from the open handle.
> 
> Why? If you have the path name you could lookup the (cached) mount
> points, and determine the filesystem type. Same solution applies for
> UNC paths, where you can easily lookup the filesystem type, and cache
> it per-process or in Cygserver.

No can do.

We *do* filesystem info caching, but this requires to be able to
open the file in the first place.  If the file exists, we're done, but
if not, we have to evaluate all symbolic links in the path first.
Simply reducing the path to the mount point and then translate it into
a Windows path doesn't cut it.

After the file (or its parent dir) could be opened, so we know the path
is valid, we can fetch the filesystem info right from the open file
handle.  At this point, we can shortcut the whole thing, reducing
the three necessary calls to kernel functions to only the first one
and skipping all the filesystem evaluation code.

> > So if Cygwin couldn't open
> > "foo" because the NtCreateFile call returned with status
> > STATUS_OBJECT_PATH_NOT_FOUND or STATUS_OBJECT_NAME_NOT_FOUND, or
> > STATUS_NO_SUCH_FILE, or one of the countless other status codes the
> > kernel (or the driver) might return in case a file doesn't exist,
> > it will tack on .lnk and .exe and, for historical reasons, .exe.lnk,
> > and try again.
> 
> Can this machinery please be turned off via CYGWIN env var option? As
> discussed in https://www.mail-archive.com/cygwin@cygwin.com/msg174547.html
> this machinery causes very bad filesystem lookup performance, and it
> would IMO a good idea to have an option to turn this off, and just
> allow and expect native links (for NTFS, ReFS and NFS). Maybe CYGWIN
> env var option winsymlinks_expect:native? winsymlinks_expect takes a :
> seperated list which symlink types are to be expected.

We won't add any more options to the CYGWIN env variable if it's not
necessary, and there's no proof at all that this is necessary in this
case.

If the file exists as specified, there will be no further check for
.exe or .lnk.  If it doesn't exist, *and* the return code from
the kernel is STATUS_OBJECT_PATH_NOT_FOUND (or any one of a
number of equivalent return values) we're done here and are going
to fall back to checking for symlinks in the leading path components.

If we got over that, the check for .exe is unavoidable.

So we're at the .lnk check. I'm pretty sure, if you try to measure the
performance it's not that bad. After all, at this point, the OS has
already cached the parent dir info and the NtOpenFile call returns
adequately snappy.

We might get away with dropping the .exe.lnk check.  In the past we
had the problem that these file existed, and you can still easily
create them, but theoretically nobody should do this.  Getting rid
of this check may be a minor backward incompatibility.

The additional problem with getting rid of .lnk is this: Even if you
switch to native symlinks exclusively (which may subtly break POSIX
compatibility, which is why Cygwin defaults to
IO_REPARSE_TAG_LX_SYMLINK) you won't switch to native symlinks
exclusively.  The reason is that FIFOs (and other device files but those
don;'t really matter since they were never fully utilized) are
implemented as Windows shortcuts.

Having said that.  I'm not opposed to fixes and improvements to the
path handling code.  It's long, winding and convoluted, and there's
certainly room for improvement.

We won't do that for Cygwin 3.5, though.  If all goes well, it will be
release in the next few weeks.

After that, when we cut the 3.5 branch, the main branch is open for new
stuff for 3.6.

However, before making generic suggestions, which may or may not be
helpful, it would be really great if you could have a look into the code
and try to understand why things are done the way they are done.  I'm
open to questions if you don't understand the code, but at least the
last couple of years, we tried hard to add comments everywhere to
explain things.  If you see a way to improve the code *without breaking
stuff*, I'm all ears, but it needs to be based on real, testable code
changes or at least POC code.

For instance: Getting rid of .lnk files isn't easy with backward
compatibility in mind.  Creating FIFOs differently is

Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2023-12-18 Thread Cedric Blancher via Cygwin
On Fri, 1 Sept 2023 at 13:00, Corinna Vinschen via Cygwin
 wrote:
>
> On Sep  1 06:23, Cedric Blancher via Cygwin wrote:
> > Good morning!
> >
> > During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
> > odd when I looked at the network traffic generated by one of our
> > cluster nodes:
> > It seems that for each call to a tool (i.e. starting "sed" from
> > "bash") Cygwin searches for *.lnk files.
> >
> > Is this correct even when the filesystem in question has native
> > symlink support (e.g. NFS)?
>
> Yes.  During file handling, Cygwin doesn't know what filesystem a
> file is on until it could actually open the file and request file
> and filesystem info from the open handle.

Why? If you have the path name you could lookup the (cached) mount
points, and determine the filesystem type. Same solution applies for
UNC paths, where you can easily lookup the filesystem type, and cache
it per-process or in Cygserver.

> So if Cygwin couldn't open
> "foo" because the NtCreateFile call returned with status
> STATUS_OBJECT_PATH_NOT_FOUND or STATUS_OBJECT_NAME_NOT_FOUND, or
> STATUS_NO_SUCH_FILE, or one of the countless other status codes the
> kernel (or the driver) might return in case a file doesn't exist,
> it will tack on .lnk and .exe and, for historical reasons, .exe.lnk,
> and try again.

Can this machinery please be turned off via CYGWIN env var option? As
discussed in https://www.mail-archive.com/cygwin@cygwin.com/msg174547.html
this machinery causes very bad filesystem lookup performance, and it
would IMO a good idea to have an option to turn this off, and just
allow and expect native links (for NTFS, ReFS and NFS). Maybe CYGWIN
env var option winsymlinks_expect:native? winsymlinks_expect takes a :
seperated list which symlink types are to be expected.

Ced
-- 
Cedric Blancher 
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2023-09-25 Thread Cedric Blancher via Cygwin
On Fri, 1 Sept 2023 at 13:00, Corinna Vinschen via Cygwin
 wrote:
>
> On Sep  1 06:23, Cedric Blancher via Cygwin wrote:
> > Good morning!
> >
> > During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
> > odd when I looked at the network traffic generated by one of our
> > cluster nodes:
> > It seems that for each call to a tool (i.e. starting "sed" from
> > "bash") Cygwin searches for *.lnk files.
> >
> > Is this correct even when the filesystem in question has native
> > symlink support (e.g. NFS)?
>
> Yes.  During file handling, Cygwin doesn't know what filesystem a
> file is on until it could actually open the file and request file
> and filesystem info from the open handle.  So if Cygwin couldn't open
> "foo" because the NtCreateFile call returned with status
> STATUS_OBJECT_PATH_NOT_FOUND or STATUS_OBJECT_NAME_NOT_FOUND, or
> STATUS_NO_SUCH_FILE, or one of the countless other status codes the
> kernel (or the driver) might return in case a file doesn't exist,
> it will tack on .lnk and .exe and, for historical reasons, .exe.lnk,
> and try again.

Our concern here is that this produces an absurd amount of extra
syscalls (6x more!!, compared to native POSIX), which hurts CIFS and
NFS performance.

The example with calling sed in a script is especially BAD, as trying
to execute "sed" in a bash shell script leads it try for sed.exe,
sed.sh, sed.bat, multiplied by looking for *.lnk files (i.e.
sed.exe.lnk, sed.sh.lnk, sed.bat.lnk).

And what finally made my brain go BOOM was the (IMO) absurd
combination like "sed.bat.exe.lnk"

Ced
-- 
Cedric Blancher 
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple


Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?

2023-09-01 Thread Corinna Vinschen via Cygwin
On Sep  1 06:23, Cedric Blancher via Cygwin wrote:
> Good morning!
> 
> During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
> odd when I looked at the network traffic generated by one of our
> cluster nodes:
> It seems that for each call to a tool (i.e. starting "sed" from
> "bash") Cygwin searches for *.lnk files.
> 
> Is this correct even when the filesystem in question has native
> symlink support (e.g. NFS)?

Yes.  During file handling, Cygwin doesn't know what filesystem a
file is on until it could actually open the file and request file
and filesystem info from the open handle.  So if Cygwin couldn't open
"foo" because the NtCreateFile call returned with status
STATUS_OBJECT_PATH_NOT_FOUND or STATUS_OBJECT_NAME_NOT_FOUND, or
STATUS_NO_SUCH_FILE, or one of the countless other status codes the
kernel (or the driver) might return in case a file doesn't exist,
it will tack on .lnk and .exe and, for historical reasons, .exe.lnk,
and try again.

Corinna

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple