Re: [Mingw-w64-public] How to recognize symlinks in WIN32?

[Adrien Nader]

On Thu, Jan 15, 2015, Greg Jung wrote:
> Thanks all, let me summarize the situation:
> 
>cygwin has its own trick to make a symlnk where the native OS doesn't
> cooperate, which is a simple file beginning with the char[11]='!'
> then 0xFF,0xFE, then wide-char name of where the link points.
> In windows a symlink creator needs elevated privilege and so native
> symlinks may not be sufficient.
> 
>cygwin lstat is trained to recognize this.  The first clue is that it is
> a system file - if the system attribute is erased, cygwin looks at a simple
> file.
> (It doesn't care that it is marked hidden or not). Then symlinks would have
> '!',0xff,0xfe as first 12 bytes;
> 
> incorporating winsymlinks:native invokes the cygwin "ln" command to
> create native symlinks if the shell is run as administrator, cywin links if
> not.
> winsymlinks:nativestrict would break non-native symlink creation.
> 
> That's all very useful thanks for the help.

In a few words: cygwin, and therefore msys*, are additional layers which
provide almost posix on top of win32 and do their very own stuff.

Win32 and POSIX symlinks have little in common besides the name and you
shouldn't have to take special care for files which are created by
either since they're emulating the functionality, even on systems
without reparse points, junctions and win32 symlinks (i.e. they use
dirty tricks to do it).

(and actually I think msys doesn't symlink but copies)

-- 
Adrien Nader

--
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet
___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public

Re: [Mingw-w64-public] How to recognize symlinks in WIN32?

[Greg Jung]

Thanks all, let me summarize the situation:

   cygwin has its own trick to make a symlnk where the native OS doesn't
cooperate, which is a simple file beginning with the char[11]='!'
then 0xFF,0xFE, then wide-char name of where the link points.
In windows a symlink creator needs elevated privilege and so native
symlinks may not be sufficient.

   cygwin lstat is trained to recognize this.  The first clue is that it is
a system file - if the system attribute is erased, cygwin looks at a simple
file.
(It doesn't care that it is marked hidden or not). Then symlinks would have
'!',0xff,0xfe as first 12 bytes;

incorporating winsymlinks:native invokes the cygwin "ln" command to
create native symlinks if the shell is run as administrator, cywin links if
not.
winsymlinks:nativestrict would break non-native symlink creation.

That's all very useful thanks for the help.

On Thu, Jan 15, 2015 at 12:23 AM, David Macek 
wrote:

> On 15. 1. 2015 4:11, Greg Jung wrote:
> > Yes I've seen that, if my second post appeared, the symlinks created
> with cygwin are the ones giving me trouble.  These links are invisible to
> CMD.exe, by someone's
> > design:
> >
> > CYGWIN- created links in
> >  Directory of e:\cygwin64\lib\nox
> >
> > 03/31/2014  09:39 AM  .
> > 03/31/2014  09:39 AM  ..
> > 07/01/2013  03:24 AM   336,710 libXpm-noX.a
> > 07/01/2013  03:24 AM43,690 libXpm-noX.dll.a
> >2 File(s)380,400 bytes
> >2 Dir(s)  85,657,726,976 bytes free
>
> I think these are cygwin emulated symlinks:
>
> They are visible, just not by default. I suspect they are marked with the
> system attribute. Use "dir /as" to show them. You should see a small size
> (in order of tens of bytes).
>
> You can instruct cygwin to create native NTFS symlinks, but due to a
> different design, there are some restrictions. See this: <
> https://cygwin.com/cygwin-ug-net/using.html#pathnames-symlinks>
>
> > So the question becomes, "why do cygwin symlinks look different, and how
> can a user program detect this attribute?
>
> I assume you could detect them using cygwin *stat calls. Maybe by
> compiling against cygwin headers and cygwin1.dll, or maybe by extracting
> the relevant code from cygwin sources (you'd have to check the relevant
> licenses).
>
> --
> David Macek
>
>
--
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public

Re: [Mingw-w64-public] How to recognize symlinks in WIN32?

[David Macek]

On 15. 1. 2015 4:11, Greg Jung wrote:
> Yes I've seen that, if my second post appeared, the symlinks created with 
> cygwin are the ones giving me trouble.  These links are invisible to CMD.exe, 
> by someone's
> design:
> 
> CYGWIN- created links in 
>  Directory of e:\cygwin64\lib\nox
> 
> 03/31/2014  09:39 AM  .
> 03/31/2014  09:39 AM  ..
> 07/01/2013  03:24 AM   336,710 libXpm-noX.a
> 07/01/2013  03:24 AM43,690 libXpm-noX.dll.a
>2 File(s)380,400 bytes
>2 Dir(s)  85,657,726,976 bytes free

I think these are cygwin emulated symlinks:

They are visible, just not by default. I suspect they are marked with the 
system attribute. Use "dir /as" to show them. You should see a small size (in 
order of tens of bytes).

You can instruct cygwin to create native NTFS symlinks, but due to a different 
design, there are some restrictions. See this: 


> So the question becomes, "why do cygwin symlinks look different, and how can 
> a user program detect this attribute?

I assume you could detect them using cygwin *stat calls. Maybe by compiling 
against cygwin headers and cygwin1.dll, or maybe by extracting the relevant 
code from cygwin sources (you'd have to check the relevant licenses).

-- 
David Macek



smime.p7s
Description: S/MIME Cryptographic Signature
--
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public

Re: [Mingw-w64-public] How to recognize symlinks in WIN32?

[Greg Jung]

Yes I've seen that, if my second post appeared, the symlinks created with
cygwin are the ones giving me trouble.  These links are invisible to
CMD.exe, by someone's
design:

CYGWIN- created links in
 Directory of e:\cygwin64\lib\nox

03/31/2014  09:39 AM  .
03/31/2014  09:39 AM  ..
07/01/2013  03:24 AM   336,710 libXpm-noX.a
07/01/2013  03:24 AM43,690 libXpm-noX.dll.a
   2 File(s)380,400 bytes
   2 Dir(s)  85,657,726,976 bytes free

from mingw.org/msys bash:
greg@Homerw7 /f/git/bldgdl
$ ls -la /e/cygwin64/lib/nox
total 634
drwxr-xr-x  2 greg Administrators   4096 Mar 31  2014 .
drwxr-xr-x 77 greg Administrators 262144 Sep 25 21:36 ..
-rw-r--r--  1 greg Administrators 336710 Jul  1  2013 libXpm-noX.a
-rw-r--r--  1 greg Administrators  43690 Jul  1  2013 libXpm-noX.dll.a
-rw-r--r--  1 greg Administrators 38 Mar 29  2014 libXpm.a
-rw-r--r--  1 greg Administrators 46 Mar 29  2014 libXpm.dll.a

from cygwin/bash:
$ ls -la /e/cygwin64/lib/nox
total 638
drwxrwxrwx+ 1 greg None  0 Mar 31  2014 .
drwxrwxrwx+ 1 greg None  0 Sep 25 21:36 ..
lrwxrwxrwx  1 greg None 12 Mar 29  2014 libXpm.a -> libXpm-noX.a
lrwxrwxrwx  1 greg None 16 Mar 29  2014 libXpm.dll.a -> libXpm-noX.dll.a
-rwxrwxrwx+ 1 greg None 336710 Jul  1  2013 libXpm-noX.a
-rwxrwxrwx+ 1 greg None  43690 Jul  1  2013 libXpm-noX.dll.a

  And the same result from msys2 bash.
The CMD.exe directory command of a native symlink looks like so:
 Directory of c:\msys64\opt

01/14/2015  03:02 PM  .
01/14/2015  03:02 PM  ..
12/08/2014  09:40 PM  bin
11/08/2014  02:47 PM  doc
11/06/2014  01:29 AM  etc
11/17/2014  06:22 PM  include
11/04/2014  01:09 AM  info
12/08/2014  04:37 PM  lib
12/08/2014  09:39 PM  lib64
05/24/2014  09:11 AM  man
11/18/2014  04:23 PM  pkgconfig
11/05/2014  09:16 PM  share
11/04/2014  01:09 AM  var
01/14/2015  03:02 PM  ww [yypkg.exe]
11/08/2014  08:57 AM  x86_64-w64-mingw32
01/14/2015  02:36 PM  yy [yypkg.exe]
11/04/2014  01:04 AM 2,348,046 yypkg.exe
   3 File(s)  2,348,046 bytes
  14 Dir(s)  40,575,012,864 bytes free
on cygwin the native links ww,yy look just like cygwin symlinks:
 $ls -la
total 2576
drwxrwxrwx+ 1 greg   None   0 Jan 14 15:02 .
drwxrwxrwx+ 1 greg   None   0 Nov 17 18:21 ..
drwxrwxrwx+ 1 greg   None   0 Dec  8 21:40 bin
drwxrwxrwx+ 1 greg   None   0 Nov  8 14:47 doc
drwxrwxrwx+ 1 greg   None   0 Nov  6 01:29 etc
drwxrwxrwx+ 1 greg   None   0 Nov 17 18:22 include
drwxrwxrwx+ 1 greg   None   0 Nov  4 01:09 info
drwxrwxrwx+ 1 greg   None   0 Dec  8 16:37 lib
drwxrwxrwx+ 1 greg   None   0 Dec  8 21:39 lib64
drwxrwxrwx+ 1 greg   None   0 May 24  2014 man
drwxrwxrwx+ 1 greg   None   0 Nov 18 16:23 pkgconfig
drwxrwxrwx+ 1 greg   None   0 Nov  5 21:16 share
drwxrwxrwx+ 1 greg   None   0 Nov  4 01:09 var
lrwxrwxrwx  1 Administrators None   9 Jan 14 15:02 ww -> yypkg.exe
drwxrwxrwx+ 1 greg   None   0 Nov  8 08:57 x86_64-w64-mingw32
lrwxrwxrwx  1 Administrators None   9 Jan 14 14:36 yy -> yypkg.exe
-rwxrwxrwx+ 1 greg   None 2348046 Nov  4 01:04 yypkg.exe


On Wed, Jan 14, 2015 at 6:16 PM, dw  wrote:

>  I'm not quite sure what you are looking for.  However, maybe this will
> help:
>
> Data.raw does not have the FILE_ATTRIBUTE_REPARSE_POINT attribute set.
> Similarly, ffd.dwReserved0 is 0.
> Data2.raw DOES have the FILE_ATTRIBUTE_REPARSE_POINT attribute.  And
> ffd.dwReserved0 shows IO_REPARSE_TAG_SYMLINK.
>
> FWIW.
>
> dw
>

For some reason my second post was bounced (maybe it didn't like a link I
referenced)

The links I couldn't detect earlier were created by cygwin install
> procedure, i.e. cygwin64/lib/noX has two files and
> two symlinks pointing to the files: fstat_win32 below didn't find a
> FILE_ATTRIBUTE_REPARSE_POINT in its mask.
>
> Nevertheless, cygwin and msys2 "ls -la" commands show a symlink.  They
> work like a symlink should.
> So the question becomes, "why do cygwin symlinks look different, and how
> can a user program detect this attribute?
>
> On 1/14/2015 9:33 AM, Greg Jung wrote:
>
> Hi Folks,
>   I've been trying to program a recognition of NTFS symbolic link files;
>  What I;ve gleaned
> from MSDN procedures don't seem to work.  Below are coded three methods
>
>  1) (untested) open file and call GetFileInformationByHandleEx.  I can't
> get winbase.h header definitions recognized
>  2) GetFileAttributes() works ok for Junctions doesn't recognize symlink
> files
>  3) FindFirstEx . same result as 2)
>
>
--
New Year. New Location. New Benefits. New Data 

Re: [Mingw-w64-public] How to recognize symlinks in WIN32?

[dw]

I'm not quite sure what you are looking for.  However, maybe this will help:

I created an NTFS symbolic file link with (data.raw is just a normal file):

   mklink c:\idelme\data2.raw c:\idelme\data.raw

From there, I was able to see that it was a symbolic link with:

#include 
#include 

void TestMe(const char *name)
{
   printf("name: %s\n", name);
   DWORD d = GetFileAttributes(name);
   printf("%x %d\n", d, d & FILE_ATTRIBUTE_REPARSE_POINT);

   WIN32_FIND_DATA ffd;
   HANDLE h = FindFirstFile(name, &ffd);

   printf("%x\n\n", ffd.dwReserved0);
   CloseHandle(h);
}

int main()
{
   TestMe("C:\\idelme\\data.raw");
   TestMe("C:\\idelme\\data2.raw");
}

Data.raw does not have the FILE_ATTRIBUTE_REPARSE_POINT attribute set.  
Similarly, ffd.dwReserved0 is 0.
Data2.raw DOES have the FILE_ATTRIBUTE_REPARSE_POINT attribute.  And 
ffd.dwReserved0 shows IO_REPARSE_TAG_SYMLINK.


FWIW.

dw

On 1/14/2015 9:33 AM, Greg Jung wrote:

Hi Folks,
  I've been trying to program a recognition of NTFS symbolic link 
files;  What I;ve gleaned

from MSDN procedures don't seem to work.  Below are coded three methods

1) (untested) open file and call GetFileInformationByHandleEx.  I 
can't get winbase.h header definitions recognized
 2) GetFileAttributes() works ok for Junctions doesn't recognize 
symlink files

 3) FindFirstEx . same result as 2)

There must be a way, since cygwin 'ls -a' and msys2 'ls -a'  commands 
will show links.


 in the calling routine, I modify result of stat() with system calls:
#ifdef _WIN32
int addlink = 0;
fstat_win32(filename.c_str(), addlink);
int actStat = stat(filename.c_str(), &statStruct);
if( addlink != 0) cout << " addlink came back " << endl;
statStruct.st_mode |= addlink;
#else
int actStat = lstat( testDir.c_str(), &statStruct);
#endif

where fstat_win32 now has a graveyard full of attempts: ( When a 
directory link is encountered, however, the test succeeeds.)


  void fstat_win32(const string& filepath, int& st_mode)
{
DWORD dwattrib, reparsetag;
DString st;
st=filepath;
#if 1
//
//  This method has not been tested: it will not compile - 
GetFileInformationByHandleEx

//  does not get through from the winbase.h header
//
HANDLEhFile;
BOOL success;
DWORD fattrib[2];
hFile = CreateFile( (TCHAR *)st.c_str(),
FILE_GENERIC_READ,
FILE_SHARE_READ | FILE_SHARE_WRITE,
0,
OPEN_EXISTING,
FILE_FLAG_OPEN_REPARSE_POINT,
  0);
if (hFile == INVALID_HANDLE_VALUE) {
CloseHandle(hFile);
cout << " stat_win32: could not open " + st << endl;
return;
   }
else {
success = GetFileInformationByHandleEx(hFile,
 FileAttributeTagInfo,
 &fattrib,
 sizeof(fattrib));
dwattrib = fattrib[0];
reparsetag = fattrib[1];
}
CloseHandle(hFile);

//dwattrib = GetFileAttributes( (TCHAR *)st.c_str()); // This 
doesn't work to find symlinks

if( (dwattrib & FILE_ATTRIBUTE_REPARSE_POINT) != 0 )  {
st_mode |= S_IFLNK;
cout << " fstat_win32: symbolic link detected and flagged!! " 
<< filepath ;

}
 else st_mode=0;
#elif 0
DWORD dwlen;
HANDLEhFind;
BOOLfoundnext;
WIN32_FIND_DATA FindFileData;
// This doesn't work to find symlinks
hFind = FindFirstFileEx( (TCHAR *) st.c_str(),
 FindExInfoStandard,
 &FindFileData,
 FindExSearchNameMatch, NULL, 0);
if (hFind == INVALID_HANDLE_VALUE)
return;
dwattrib = FindFileData.dwFileAttributes;
if( (dwattrib & FILE_ATTRIBUTE_REPARSE_POINT) != 0 )  {
st_mode |= S_IFLNK;
cout << " fstat_win32: symbolic link detected and flagged!! " 
<< filepath ;

dwattrib = FindFileData.dwReserved0;
// IO_REPARSE_TAG_SYMLINK _MOUNT_POINT etc.
}
   FindClose(hFind);
#endif
return;
}



--
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet


___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public


--
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public

Re: [Mingw-w64-public] How to recognize symlinks in WIN32?

[Greg Jung]

Hi all,
  I've been doing some experimentation today and discovered another twist
in the story
which relates to cygwin - so I forward this on to cygwin-user list, also.
  I have been testing the code below against symlinks created
in cygwin64 (I'm running win7 64-bit; I have mingw/msys, msys2 (64), and
cygwin64 installations).
My code hadn't detected symlink files but all of thesymlinks I was trying
were cygwin64.
I created some symlinks natively, using an ln.exe (from
http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html)
and using mklink.exe; these links *-do show up-* as reparse points in my
detection scheme 3), (but 2) should also work)  described below.
 The links I couldn't detect earlier were created by cygwin install
procedure, i.e. cygwin64/lib/noX has two files and
two symlinks pointing to the files: fstat_win32 below didn't find a
FILE_ATTRIBUTE_REPARSE_POINT in its mask.

Nevertheless, cygwin and msys2 "ls -la" commands show a symlink.  They work
like a symlink should.
So the question becomes, "why do cygwin symlinks look different, and how
can a user program detect this attribute?

(I've edited the original message to correct the lstat call )

On Wed, Jan 14, 2015 at 9:33 AM, Greg Jung  wrote:

> Hi Folks,
>   I've been trying to program a recognition of NTFS symbolic link files;
>  What I;ve gleaned
> from MSDN procedures don't seem to work.  Below are coded three methods
>
> 1) (untested) open file and call GetFileInformationByHandleEx.  I can't
> get winbase.h header definitions recognized
>  2) GetFileAttributes() works ok for Junctions doesn't recognize symlink
> files
>  3) FindFirstEx . same result as 2)
>
> There must be a way, since cygwin 'ls -a' and msys2 'ls -a'  commands will
> show links.
>
>  in the calling routine, I modify result of stat() (from glibc, I take it)
> from with system calls:
> #ifdef _WIN32
> int addlink = 0;
> fstat_win32(filename.c_str(), addlink);
> int actStat = stat(filename.c_str(), &statStruct);
> if( addlink != 0) cout << " addlink came back " << endl;
> statStruct.st_mode |= addlink;
> #else
> int actStat = lstat( filename.c_str(), &statStruct);
> #endif
>
> where fstat_win32 now has a graveyard full of attempts: ( When a directory
> link is encountered, however, the test succeeeds.)
>
>   void fstat_win32(const string& filepath, int& st_mode)
> {
> DWORD dwattrib, reparsetag;
> DString st;
> st=filepath;
> #if 1
> //
> //  This method has not been tested: it will not compile -
> GetFileInformationByHandleEx
> //  does not get through from the winbase.h header
> //
> HANDLE hFile;
> BOOL success;
> DWORD fattrib[2];
> hFile = CreateFile( (TCHAR *)st.c_str(),
> FILE_GENERIC_READ,
> FILE_SHARE_READ | FILE_SHARE_WRITE,
> 0,
> OPEN_EXISTING,
> FILE_FLAG_OPEN_REPARSE_POINT,
>   0);
> if (hFile == INVALID_HANDLE_VALUE) {
> CloseHandle(hFile);
> cout << " stat_win32: could not open " + st << endl;
> return;
>}
> else {
> success = GetFileInformationByHandleEx( hFile,
> FileAttributeTagInfo,
> &fattrib,
> sizeof(fattrib));
> dwattrib = fattrib[0];
> reparsetag = fattrib[1];
> }
> CloseHandle(hFile);
>
> //dwattrib = GetFileAttributes( (TCHAR *)st.c_str()); // This doesn't
> work to find symlinks
> if( (dwattrib & FILE_ATTRIBUTE_REPARSE_POINT) != 0 )  {
> st_mode |= S_IFLNK;
> cout << " fstat_win32: symbolic link detected and flagged!! " <<
> filepath ;
> }
>  else st_mode=0;
> #elif 0
> DWORD dwlen;
> HANDLE hFind;
> BOOL foundnext;
> WIN32_FIND_DATA FindFileData;
> // This doesn't work to find symlinks
> hFind = FindFirstFileEx( (TCHAR *) st.c_str(),
>  FindExInfoStandard,
>  &FindFileData,
>  FindExSearchNameMatch, NULL, 0);
> if (hFind == INVALID_HANDLE_VALUE)
> return;
> dwattrib = FindFileData.dwFileAttributes;
> if( (dwattrib & FILE_ATTRIBUTE_REPARSE_POINT) != 0 )  {
> st_mode |= S_IFLNK;
> cout << " fstat_win32: symbolic link detected and flagged!! " <<
> filepath ;
> dwattrib = FindFileData.dwReserved0;
> // IO_REPARSE_TAG_SYMLINK _MOUNT_POINT etc.
> }
>FindClose(hFind);
> #endif
> return;
> }
>
>
--
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public