Re: BSD file flags support in Cygwin?
Corinna, thanks for your answer. My question was largely to confirm my understanding as I probably do not have time to produce such a patch currently. For the benefit of the discussion and to assist any future attempts at doing this: Chflags(2) and friends would be relatively easy to implement. OTOH the BSD’s and OSX use stat(2) to report file flags in st_flags. Since Cygwin lacks an st_flags field, we would have to somehow change struct stat to accommodate it. Do you have guidance on how to handle such a change? Would you even want such a change? [I suspect not.] [I know some OS’es have had to deal with stat(2) changes over the years, but I am not as familiar with Cygwin history.] Bill On 11/14/17, 2:06 AM, Corinna Vinschen wrote: >On Nov 13 23:21, Bill Zissimopoulos wrote: >> Does Cygwin have any support for BSD file flags (UF_* flags, such as >> UF_HIDDEN, etc.)? These flags are often used to provide support for >> Windows file attributes (FILE_ATTRIBUTE_*, such as >>FILE_ATTRIBUTE_HIDDEN). >> >> OSX and FreeBSD provide such support during stat(2) and chflags(2). I >> expect that Cygwin does not have such support, but wanted to >>double-check. >> >> >>https://www.freebsd.org/cgi/man.cgi?query=chflags&apropos=0&sektion=2&man >>pa >> th=FreeBSD+11.1-RELEASE+and+Ports&arch=default&format=html > >No, we don't support this. We couldn't support it via stat anyway, >only via the extra chflags function group. Patches welcome, as >long as it's the entire group of functions so chflags(1) compiles >OOTB. > > >Corinna > >-- >Corinna Vinschen Please, send mails regarding Cygwin to >Cygwin Maintainer cygwin AT cygwin DOT com >Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
BSD file flags support in Cygwin?
Does Cygwin have any support for BSD file flags (UF_* flags, such as UF_HIDDEN, etc.)? These flags are often used to provide support for Windows file attributes (FILE_ATTRIBUTE_*, such as FILE_ATTRIBUTE_HIDDEN). OSX and FreeBSD provide such support during stat(2) and chflags(2). I expect that Cygwin does not have such support, but wanted to double-check. https://www.freebsd.org/cgi/man.cgi?query=chflags&apropos=0&sektion=2&manpa th=FreeBSD+11.1-RELEASE+and+Ports&arch=default&format=html Bill
Re: Id3tag and lame packages?
On 9/22/16, 3:35 PM, David Stacey wrote: >On 22/09/16 22:58, Bill Zissimopoulos wrote: >> BTW, you mentioned a legal problem. I am seeing that LAME is LGPL >>licensed >> and therefore should be eligible for inclusion in Cygwin(?). > >No, there are plenty of LGPL packages in Cygwin. The problem with LAME >is that it still has a number of active patents. [1] is a good write-up >describing the problem, and said patents are enumerated in [2]. > >Dave. > >[1] - https://lwn.net/Articles/178285/ >[2] - http://mp3licensing.com/patents/index.html Thank you, David. My apologies for the late response, my junk filters have a mind of their own. Bill
Re: Id3tag and lame packages?
On 9/23/16, 12:40 AM, Csaba Raduly wrote: >On Thu, Sep 22, 2016 at 11:58 PM, Bill Zissimopoulos wrote: >> >> BTW, you mentioned a legal problem. I am seeing that LAME is LGPL >>licensed >> and therefore should be eligible for inclusion in Cygwin(?). > >It's probably due to >https://en.wikipedia.org/wiki/LAME#Patents_and_legal_issues Makes sense. Thank you! Bill
Re: Id3tag and lame packages?
On 9/22/16, 2:06 PM, Yaakov Selkowitz wrote: >On 2016-09-22 15:55, Bill Zissimopoulos wrote: >> On 9/22/16, 12:37 PM, Yaakov Selkowitz wrote: >>> lame cannot be shipped however >>> for legal reasons, but it's easy to build yourself with cygport: >>> >>> https://github.com/cygwinports-extras/lame >> >> Alas this fails: >[snip] >> configure.in:396: error: possibly undefined macro: AM_PATH_GTK >> If this token and others are legitimate, please use >>m4_pattern_allow. >> See the Autoconf documentation. >> autoreconf-2.69: /usr/bin/autoconf-2.69 failed with exit status: 1 >> *** ERROR: autoreconf failed > >You're missing a build dependency: libgtk1.2-devel. Thanks again. That did the trick. Sorry for the newbie questions. BTW, you mentioned a legal problem. I am seeing that LAME is LGPL licensed and therefore should be eligible for inclusion in Cygwin(?). Bill
Re: Id3tag and lame packages?
On 9/22/16, 12:37 PM, Yaakov Selkowitz wrote: >lame cannot be shipped however >for legal reasons, but it's easy to build yourself with cygport: > >https://github.com/cygwinports-extras/lame Alas this fails: $ cygport lame.cygport compile >>> Compiling lame-3.99.5-2.x86_64 autoreconf-2.69: Entering directory `.' autoreconf-2.69: configure.in: not using Gettext autoreconf-2.69: running: aclocal --force main::scan_file() called too early to check prototype at /usr/bin/aclocal-1.11 line 644. configure.in:396: warning: macro `AM_PATH_GTK' not found in library autoreconf-2.69: configure.in: tracing autoreconf-2.69: running: libtoolize --copy --force libtoolize: putting auxiliary files in '.'. libtoolize: copying file './ltmain.sh' libtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.in, libtoolize: and rerunning libtoolize and aclocal. libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. main::scan_file() called too early to check prototype at /usr/bin/aclocal-1.11 line 644. configure.in:396: warning: macro `AM_PATH_GTK' not found in library autoreconf-2.69: running: /usr/bin/autoconf-2.69 --force configure.in:396: error: possibly undefined macro: AM_PATH_GTK If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. autoreconf-2.69: /usr/bin/autoconf-2.69 failed with exit status: 1 *** ERROR: autoreconf failed Bill
Re: Id3tag and lame packages?
On 9/22/16, 12:37 PM, Yaakov Selkowitz wrote: >On 2016-09-22 14:12, Bill Zissimopoulos wrote: >> Do the packages id3tag and lame exist for Cygwin? I would like to try >> compiling mp3fs on Cygwin and I cannot find them. > >If you mean libid3tag, it is available in the distribution; install >libid3tag-devel and its dependencies. You are right of course. How did I miss that? >lame cannot be shipped however >for legal reasons, but it's easy to build yourself with cygport: > >https://github.com/cygwinports-extras/lame Excellent! Thank you. Bill
Id3tag and lame packages?
Do the packages id3tag and lame exist for Cygwin? I would like to try compiling mp3fs on Cygwin and I cannot find them. Bill
Re: FUSE, symbolic links and special files
On 8/26/16, 4:59 PM, cygwin-ow...@cygwin.com Bill Zissimopoulos wrote: >On 8/26/16, 11:05 AM, Corinna Vinschen wrote: > >>On Aug 25 19:04, Bill Zissimopoulos wrote: >>>- The first case is during the processing of NtCreateFile (without the >>> FILE_OPEN_REPARSE_POINT flag set). >> >>This case doesn't matter to us. Cygwin always opens the file with >>FILE_OPEN_REPARSE_POINT set... >> >>> - The second case is through direct manipulation of the reparse point >>> using FSCTL_GET_REPARSE_POINT, FSCTL_SET_REPARSE_POINT and >>> FSCTL_DELETE_REPARSE_POINT. >>> >>> Let us consider the expected behavior of an NFS_SPECFILE_LNK reparse >>>point >>> (this is speculation) during NtCreateFile: >>> >>> - On NTFS prior to Win8: >>> - STATUS_IO_REPARSE_TAG_NOT_HANDLED >> >>...so this shouldn't happen to us, right? > >I think so. > >I will continue with the implementation/testing of reparse points and >report back when I have more. I have finally completed the implementation of reparse points for WinFsp. I have also implemented NFS reparse point support for WinFsp-FUSE. I tested this implementation with Cygwin although my testing was general command line use and not rigorous. For this purpose I cobbled together a patch for Cygwin; the patch is low quality and I hesitate to post it here, but I can if so desired. Effectively I changed the implementation of Cygwin’s mknod_worker, symlink_info::check_reparse_point and readdir_check_reparse_point to understand and use NFS reparse points. I am unclear on how to proceed from here. Although I do not understand the Cygwin internals well enough and have rather limited time at the moment I could try cleaning up the patch and officially submitting. I am also thinking that using reparse points for POSIX special files on other file systems that support it (i.e. NTFS) may be something that the list should consider. Bill
Re: FUSE, symbolic links and special files
On 8/26/16, 11:05 AM, Corinna Vinschen wrote: >On Aug 25 19:04, Bill Zissimopoulos wrote: >>- The first case is during the processing of NtCreateFile (without the >> FILE_OPEN_REPARSE_POINT flag set). > >This case doesn't matter to us. Cygwin always opens the file with >FILE_OPEN_REPARSE_POINT set... > >> - The second case is through direct manipulation of the reparse point >> using FSCTL_GET_REPARSE_POINT, FSCTL_SET_REPARSE_POINT and >> FSCTL_DELETE_REPARSE_POINT. >> >> Let us consider the expected behavior of an NFS_SPECFILE_LNK reparse >>point >> (this is speculation) during NtCreateFile: >> >> - On NTFS prior to Win8: >> - STATUS_IO_REPARSE_TAG_NOT_HANDLED > >...so this shouldn't happen to us, right? I think so. I will continue with the implementation/testing of reparse points and report back when I have more. Bill
Re: FUSE, symbolic links and special files
On 8/25/16, 3:45 PM, Corinna Vinschen wrote:
>On Aug 25 11:46, Bill Zissimopoulos wrote:
>>In the following OP is the originating process, CW is the Cygwin
>> layer, WL is the WinFsp layer and FL is the FUSE layer.
>>
>> OP: mkfifo("myfifo")
>> CW: NtCreateFile, NtDeviceIoControlFile(FSCTL_SET_REPARSE_POINT)
>> [NFS_SPECFILE_FIFO]
>> WL: IRP_MJ_FILE_SYSTEM_CONTROL/FSCTL_SET_REPARSE_POINT
>>[NFS_SPECFILE_FIFO]
>> FL: fuse_operations::mknod("myfifo", S_IFIFO)
>>
>> Regarding symbolic link support specifically I am still undecided on
>> whether the right thing to do is to use NTFS symbolic links
>> (IO_REPARSE_TAG_SYMLINK) or use NFS_SPECFILE_LNK for the FUSE layer. My
>> inclination is to support both and let the FUSE file system developer
>> decide on their preference.
>>
>> Any comments welcome.
>
>...it needs thorough testing(*). There's a good chance that the NFS RP
>buffer is not exposed to user space, but instead only handled by the NFS
>driver. *If* the RP method works fine in user space, I'm inclined to do
>as outlined above and get rid of the EA stuff in symlink_info::check
>since it could be transparently shared between NFS and WinFSP.
I agree. FYI I have not tested the use of NFS reparse points yet, although
I intend to.
My expectation is that there should not be any issue accessing such
reparse points from user mode. My understanding of the reparse point
mechanism is that it comes into play in a couple of cases:
- The first case is during the processing of NtCreateFile (without the
FILE_OPEN_REPARSE_POINT flag set). In this case the file system driver
will return STATUS_REPARSE and either:
- Return a path that NTOS will resend to the file system driver for
further processing (useful for symlink-style processing), or...
- Return a reparse data buffer that some higher-level component (e.g. a
filter driver) may interpret specially. For example, a hierarchical
storage management system may download a file pointed to by a reparse
point locally. If there is no higher-level component that knows about the
reparse point, I believe the NTOS IO manager will return
STATUS_IO_REPARSE_TAG_NOT_HANDLED.
- The second case is through direct manipulation of the reparse point
using FSCTL_GET_REPARSE_POINT, FSCTL_SET_REPARSE_POINT and
FSCTL_DELETE_REPARSE_POINT.
Let us consider the expected behavior of an NFS_SPECFILE_LNK reparse point
(this is speculation) during NtCreateFile:
- On NTFS prior to Win8:
- STATUS_IO_REPARSE_TAG_NOT_HANDLED
- On NFS prior to Win8:
- STATUS_IO_REPARSE_TAG_NOT_HANDLED
- On FUSE for Cygwin prior to Win8:
- STATUS_SUCCESS with file pointed by symlink opened (because the WinFsp
FSD would know how to handle NFS reparse points).
- On NTFS after Win8:
- Likely STATUS_IO_REPARSE_TAG_NOT_HANDLED, although there is a
possibility that Microsoft has built knowledge of the NFS_SPECFILE_LNK
reparse point into the NTFS FSD or another filter driver, in which case
the file would be opened and the return would be STATUS_SUCCESS.
- On NFS after Win8:
- STATUS_SUCCESS with file pointed by symlink opened.
- On FUSE for Cygwin prior to Win8:
- STATUS_SUCCESS with file pointed by symlink opened.
This is not ideal, because sometimes an NtCreateFile of an
NFS_SPECFILE_LNK reparse point will result in STATUS_SUCCESS and sometimes
it will result in STATUS_IO_REPARSE_TAG_NOT_HANDLED. I can imagine of
course that the NtCreateFile call could be retried with
FILE_OPEN_REPARSE_POINT followed by FSCTL_GET_REPARSE_POINT and symlink
resolution could happen in user mode.
Bill
Re: FUSE, symbolic links and special files
On 8/25/16, 5:46 PM, Jeffrey Altman wrote: >The only file system for which this tag is known to be interpreted is >the Microsoft NFS provider that will report its > > FILE_REMOTE_PROTOCOL_INFORMATION.Protocol > >value as > > #define WNNC_NET_MS_NFS 0x0042 I missed this. Jeffrey do you have a pointer for this information? Bill
Re: FUSE, symbolic links and special files
On 8/25/16, 7:14 PM, Jeffrey Altman wrote: >On 8/25/2016 11:21 AM, Corinna Vinschen wrote: >>Granted, it *could* be used by Cygwin on NTFS to indicate Cygwin's own >> implementations of AF_LOCAL sockets or fifos. Or even for symlinks. >> But that would only introduce YA symlink type which would be unusable >> from non-Cygwin applications. > >Correct. > >With its own reparse tag Cygwin could store exactly the same metadata it >stores today in the data stream of the .lnk file as reparse tag data. >The benefit of applying a reparse tag is that the .lnk will no longer be >confused for a regular file. On file systems that do not support >reparse points it can continue to store the data in the data stream. I agree with Jeffrey, and I did think of this as a potential solution for FUSE for Cygwin (and perhaps Cygwin itself). But since I learned about the NFS reparse points I find them a very good solution, especially because they indicate that someone in Microsoft has already thought about this problem. Bill
FUSE, symbolic links and special files
While on vacation I have been (slowly) working to add reparse point and
symbolic link support for WinFsp and FUSE for Cygwin. This work is mostly
complete and is currently being tested.
I am writing to the Cygwin list because I want to resolve a problem that
Herbert Stocker originally brought up:
>F) Pipes:
>> [Quick experiment:
>>
>> $ mkfifo foo; cmd /c dir 'foo*' | grep foo; rm foo
>> 06/16/2016 11:02 PM 130 foo.lnk
>> ]
>>
>> Ok, so they are shortcuts. Naturally they are supported.
>
>I think they are not.
>The mkfifo system call will have Cygwin create a .lnk file and
>WinFsp will forward it as such to the file system process. The
>sytem calls readdir or open will then have the file system
>process tell WinFsp that there is a .lnk file and Cygwin will
>translate this back to a fifo, so in this sense it does work.
>
>But the file system will see a file (with name *.lnk) where it
>should see a pipe (mknod call with 'mode' set to S_IFIFO).
>IMHO one could say this is a break of the FUSE API.
>
>Practically it will break:
> - file systems that special-treat pipe files (or .lnk files).
>
> - If one uses sshfs to connect to a Linux based server and
> issues the command mkfifo foo from Cygwin, the server will
> end up with a .lnk file instead of a pipe special file.
>
> - Imagine something like mysqlfs, which stores the stuff in a
> database. When you run SQL statements to analyze the data
> in the file system, you won't see the pipes as such. Or if
> you open the file system from Linux you'll see the .lnk
> files.
Herbert is of course right. A .lnk file is not a FIFO to any system other
than Cygwin. I have been thinking about how to solve this problem and I
believe I have an answer in the form of reparse points.
Reparse points can be viewed as a form of special metadata that can be
attached to a file or directory. The interesting thing about reparse
points is that they can have special meaning to a file system driver
(NTFS/WinFsp), a filter driver (e.g. a hierarchical storage system) or
even an application (Cygwin).
NTFS uses reparse points to implement symbolic links and places severe
limitations on them. For one, it differentiates between file and directory
symlinks. It also requires the SE_CREATE_SYMBOLIC_LINK_PRIVILEGE privilege
to be held to create a symbolic link account.
Interestingly it does not perform any extraneous access checks on other
reparse points. IMO this makes reparse points very interesting because it
offers a path for WinFsp and FUSE for Cygwin to provide special file
support.
Turns out that Microsoft already has a solution for special files on NFS:
https://msdn.microsoft.com/en-us/library/dn617178.aspx
I see no reason that the same solution cannot be used for FUSE for Cygwin
as well. In the following OP is the originating process, CW is the Cygwin
layer, WL is the WinFsp layer and FL is the FUSE layer.
OP: mkfifo("myfifo")
CW: NtCreateFile, NtDeviceIoControlFile(FSCTL_SET_REPARSE_POINT)
[NFS_SPECFILE_FIFO]
WL: IRP_MJ_FILE_SYSTEM_CONTROL/FSCTL_SET_REPARSE_POINT [NFS_SPECFILE_FIFO]
FL: fuse_operations::mknod("myfifo", S_IFIFO)
Regarding symbolic link support specifically I am still undecided on
whether the right thing to do is to use NTFS symbolic links
(IO_REPARSE_TAG_SYMLINK) or use NFS_SPECFILE_LNK for the FUSE layer. My
inclination is to support both and let the FUSE file system developer
decide on their preference.
Any comments welcome.
Bill
Re: POSIX permission mapping and NULL SIDs
On 6/29/16, 1:21 AM, "Corinna Vinschen" wrote: >If that's the case, then why do you explain all these things to me? I'm >a bit at a loss to see the difference between me explaining things to >you you already know vs. you explaing things to me I already know. >Aren't we kind of on par here? Yes, we are. Perhaps I spoke “out of turn” as the Americans say. I am sorry if it also felt like I was explaining things that you know. >>In any case I will use your mapping of S-1-0-65534 <-> 65534. > >Thanks. Do you want to add handling for this mapping to >pwdgrp::fetch_account_from_windows yourself or shall I do it? I could >come up with a patch in the next couple of days. I will prepare a >developer's snapshot then, so you can immediately test if it works as >desired. I have already added the mapping to WinFsp-FUSE. I can look into what is required to patch Cygwin. >>How do we avoid name collisions? I can easily see admins creating an >>AD account called "nobody". >>Shall we fake a "WinFSP" domain such that the name is "WinFSP+nobody"? > >Preliminary patch attached. Ok, looks like you have patched it already. Thanks for this. BTW, if the name is case-sensitive (strcmp) I usually use the “WinFsp” capitalization, if it makes a difference. Also do you foresee any situation where the “nobody” mapping might be useful outside of WinFsp? Perhaps it would make more sense to name it “nodomain+nobody”? Just a suggestion. Many thanks. Bill
Re: POSIX permission mapping and NULL SIDs
On 6/28/16, 3:27 AM, "Corinna Vinschen" wrote: >>Ok. Please keep in mind that > >a) there can't be a bijective mapping between arbitrary length SIDs > and a 32 bit uid/gid. > >b) The mapping used in Cygwin is not self-created but (mostly, except > for a single deviation) identical to the Interix mapping. The code > basically follows how this mapping has been defined by Microsoft. Corinna, please stop explaining things to me that I already know. >> BTW, I have here a partitioning of the UID namespace that may help >>choose >> the right mapping: >> >> /* >> * UID namespace partitioning (from [IDMAP] rules): >> * >> * 0x00 + RID S-1-5-RID,S-1-5-32-RID >> * 0x000ffeOtherSession >> * 0x000fffCurrentSession >> * 0x001000 * X + RID S-1-5-X-RID ([WKSID]: >> X=1-15,17-21,32,64,80,83) >> * 0x01 + 0x100 * X + YS-1-X-Y ([WKSID]: X=1,2,3,4,5,9,16) >> * 0x03 + RID S-1-5-21-X-Y-Z-RID >> * 0x06 + RID S-1-16-RID >> * 0x10 + RID S-1-5-21-X-Y-Z-RID >> */ > >You're aware that I wrote the code for this mapping as well as its >documentation? :) Corinna, of course I am aware of that. I have found your original post to this list about it. Why would you think otherwise? And why would it change anything? >>With all that and to help conclude this thread I gather here all the >> proposed mappings. Corinna, I will use the one which you prefer the >>most: >> >> S-1-0-65534<-> 65534 > >This one is still my favorite. Again, the range from 0x1000 up to >0x is unused. Right now any incoming uid/gid value in this range >for a reverse SID lookup is treated as invalid SID. I disagree. You are saying that it is unused, but a (perhaps erroneous) SID would map into that space. In any case I will use your mapping of S-1-0-65534 <-> 65534. Bill
Re: POSIX permission mapping and NULL SIDs
>Why don't we just follow Fedora Linux here and use a mapping to either
>99 (nobody) or 65534 (nfsnobody)? Both uid values are ununsed in the
>mapping and 65534 aka 0xfffe has the additional advantage that it's not
>mapped at all (all values between 0x1000 and 0x are invalid).
>
>Also, since 65534 is -2 in a 16 bit uid it seems like a natural choice
>to me.
>
>So, what about S-1-0-65534 <-> 65534, name of "{nfs}nobody"?
I am happy with the S-1-0-65534 *SID*, but I note that the 65534 *UID* is
perhaps *not* a good choice. It is actually already mapped to
S-1-5-15-4095, according to your own [IDMAP] document:
S-1-5-X-RID <=> uid/gid: 0x1000 * X + RID
With X=15 and RID=4095, we get uid==65534. Unfortunately S-1-5-15 is the
SID for "This Organization” according to the “Well-known security
identifiers in Windows operating systems” document [WKSID]. OTOH, because
S-1-5-15 is a “leaf” SID and not a “namespace” it may be possible to
assume that the S-1-5-15-4095 SID cannot appear (I am not sure about that).
BTW, I have here a partitioning of the UID namespace that may help choose
the right mapping:
/*
* UID namespace partitioning (from [IDMAP] rules):
*
* 0x00 + RID S-1-5-RID,S-1-5-32-RID
* 0x000ffeOtherSession
* 0x000fffCurrentSession
* 0x001000 * X + RID S-1-5-X-RID ([WKSID]:
X=1-15,17-21,32,64,80,83)
* 0x01 + 0x100 * X + YS-1-X-Y ([WKSID]: X=1,2,3,4,5,9,16)
* 0x03 + RID S-1-5-21-X-Y-Z-RID
* 0x06 + RID S-1-16-RID
* 0x10 + RID S-1-5-21-X-Y-Z-RID
*/
Clearly the namespace is very busy with multiple overlapping ranges.
With all that and to help conclude this thread I gather here all the
proposed mappings. Corinna, I will use the one which you prefer the most:
S-1-0-65534<-> 65534
S-1-0-65534<-> -1==0x
S-1-0-65534<-> -2==0xfffe
S-1-0-99 <-> -1==0x
S-1-0-99 <-> -2==0xfffe
Bill
[IDMAP] https://cygwin.com/cygwin-ug-net/ntsec.html
[WKSID] https://support.microsoft.com/en-us/kb/243330
Re: POSIX permission mapping and NULL SIDs
On 6/24/16, 2:59 PM, "Corinna Vinschen" wrote: >>>If you want some specific mapping we can arrange that, but it must not >> >be the NULL SID. If you know you're communicating with a Cygwin >>process, >> >what about using an arbitrary, unused SID like S-1-0-42? >> >> I am inclined to try S-1-5-7 (Anonymous). But I do not know if that is a >> bad choice for some reason or other. > >I thought about Anonymous myself when I wrote my reply to your OP. I >refrained from mentioning it because it might have some unexpected side >effect we're not aware about. I ended up implementing this a couple of days ago. I was just spending a lazy Sunday morning and then it hit me: this is an exceptionally bad idea. The problem is that Windows uses the Anonymous identity for accounts who have not logged in using a password (as per Erik Soderquist’s email regarding IIS behavior). Files in FUSE file systems that have a UID that cannot be mapped to a SID, will suddenly be owned by that Anonymous user! Obviously this is a huge security hole. I intend to fix this ASAP, but I am now back to where we started. The obvious SID to use is the NULL SID, but that is already used by Cygwin for other purposes. >> The main reason that I am weary of using an unused SID is that Microsoft >> may decide to assign some special powers to it in a future release (e.g. >> GodMode SID). But I agree that this is rather unlikely in the S-1-0-X >> namespace. > >I think it's very unlikely. We could chose any RID value we like and >the chance for collision is nil. When I created the new implementation >for POSIX ACLs, I toyed around with this already and used a special >Cygwin SID within the NULL SID AUTHORITY. I'm not entirely sure why I >changed this to the NULL SID deny ACE. I think I disliked the fact that >almost every Cygwin ACL would contain a mysterious "unknown SID". Ideally we should choose a SID that: (1) Is very unlikely to be used by Microsoft at any point in the future. (2) Cannot be associated to a user logon for any reason (see problem with Anonymous SID) above. (3) Maps to a reasonable UID in Cygwin. I propose the following SID/UID mapping: S-1-0-99 <=> UID 0x (32-bit -1) This is a SID in the S-1-0 (Null Authority) namespace (same one that contains the NULL SID), which is unlikely to be used by Microsoft. So it likely satisfies (1). For the same reason (that it is a new/unused SID in the S-1-0) namespace, I think it also satisfies (2). If we follow the rules from Cygwin’s "POSIX accounts, permission, and security” document [IDMAP], the SID S-1-0-99 maps to 0x10063. But we can make a special rule for this SID to map it to a different UID. Mapping it to -1 may be the easiest option, but perhaps we can also consider mapping it to 0xfffe (-2). Bill [IDMAP] https://cygwin.com/cygwin-ug-net/ntsec.html
Re: POSIX permission mapping and NULL SIDs
On 6/24/16, 3:53 PM, "cygwin-ow...@cygwin.com on behalf of Bill Zissimopoulos" wrote: >One caveat is that Cygwin already maps S-1-5-7 to uid 7. So does that mean >that 7==nobody in Cygwin’s case? Here is output from Cygwin/SSHFS after mapping “nobody/nogroup” to S-1-5-7: << billziss@windows:~$ cd /cygdrive/y billziss@windows:/cygdrive/y$ ls -la total 8 drwxr-xr-x 1 billziss ANONYMOUS LOGON 0 Jun 23 23:57 . dr-xr-xr-x 1 billziss None 0 Jun 24 15:56 .. -rw-r--r-- 1 billziss ANONYMOUS LOGON 15 Jun 23 23:57 Foo.txt billziss@windows:/cygdrive/y$ cacls Foo.txt /S Y:\Foo.txt "D:P(A;;0x1f019f;;;S-1-5-21-383059176-2062642591-2866287538-1001)(A;;FR;;;A N)(A;;FR;;;WD)" billziss@windows:/cygdrive/y$ ls -lna total 8 drwxr-xr-x 1 197609 7 0 Jun 23 23:57 . dr-xr-xr-x 1 197609 197121 0 Jun 24 15:58 .. -rw-r--r-- 1 197609 7 15 Jun 23 23:57 Foo.txt billziss@windows:/cygdrive/y$ >> Opinions? Bill
Re: POSIX permission mapping and NULL SIDs
On 6/24/16, 3:06 PM, "cygwin-ow...@cygwin.com on behalf of Erik Soderquist" wrote: >On Fri, Jun 24, 2016 at 5:59 PM, Corinna Vinschen wrote: >>> I am inclined to try S-1-5-7 (Anonymous). But I do not know if that is >>>a >>> bad choice for some reason or other. >> >> I thought about Anonymous myself when I wrote my reply to your OP. I >> refrained from mentioning it because it might have some unexpected side >> effect we're not aware about. > >I know in at least some versions of IIS, the Anonymous SID is the one >explicitly used for all not logged in www/ftp/etc clients connecting >to IIS -- not sure if that would affect this discussion. It is my understanding that this is the intended use of the Anonymous SID. I have not thought enough to know whether it would fit properly in this use case. One caveat is that Cygwin already maps S-1-5-7 to uid 7. So does that mean that 7==nobody in Cygwin’s case? Bill
Re: POSIX permission mapping and NULL SIDs
On 6/24/16, 2:59 PM, "Corinna Vinschen" wrote: >>>If you want some specific mapping we can arrange that, but it must not >> >be the NULL SID. If you know you're communicating with a Cygwin >>process, >> >what about using an arbitrary, unused SID like S-1-0-42? >> >> I am inclined to try S-1-5-7 (Anonymous). But I do not know if that is a >> bad choice for some reason or other. > >I thought about Anonymous myself when I wrote my reply to your OP. I >refrained from mentioning it because it might have some unexpected side >effect we're not aware about. Let me try this with the Anonymous SID and see what happens. In the meantime I am going to ping some contacts from my MS days to see if there are indeed unintended consequences. >Keep in mind that Interix only supported standard POSIX permission bits. >Cygwin strives to support POSIX ACLs per POSIX 1003.1e draft 17. That's >a bit more extensive. Yes, that is beginning to dawn on me now that I have taken a better look at sec_acl.cc. Bill
Re: POSIX permission mapping and NULL SIDs
On 6/24/16, 12:51 PM, "Corinna Vinschen" wrote: >>Could my mapping of the NULL SID somehow interfere with Cygwin’s ACL >> mapping? No way right? Turns out that: yes! >>File:winsup/cygwin/sec_acl.cc, >> line:787 > >Read the comment at the beginning of the file explaining how new-style >ACLs look like. Thank you for the pointers and the historical information. >>I am also seeking an alternative to using the NULL SID for >> “nobody”/“nogroup”. Is there a Cygwin suggested one? > >Not yet. We're coming from the other side. We always have *some* SID. >pwdgrp::fetch_account_from_windows() in uinfo.cc tries to convert the SID >to a passwd or group entry. If everything fails, the SID is used in this >passwd/group entry verbatim, but mapped to uid/gid -1. I also noticed that there is no uid mapping for nobody. On my OSX box it is -2. On many other POSIX systems it appears to be the 32-bit or 16-bit equivalent of -2. For the time being I am mapping unknown SID’s to -1 as per Cygwin. >If you want some specific mapping we can arrange that, but it must not >be the NULL SID. If you know you're communicating with a Cygwin process, >what about using an arbitrary, unused SID like S-1-0-42? I am inclined to try S-1-5-7 (Anonymous). But I do not know if that is a bad choice for some reason or other. The main reason that I am weary of using an unused SID is that Microsoft may decide to assign some special powers to it in a future release (e.g. GodMode SID). But I agree that this is rather unlikely in the S-1-0-X namespace. >How do you differ nobody from nogroup if you use the same SID for both, >btw.? I use the same SID for both nobody and nogroup. This should work as long as you use the permission mapping from the [PERMS] document. Bill
POSIX permission mapping and NULL SIDs
EXEUTIVE EDITION
I am seeking information on how exactly Cygwin uses NULL SID ACE’s in
Windows ACL’s. Cygwin’s use of NULL SID ACE’s interferes with my use of
the NULL SID to represent “nobody”/“nogroup”.
AN EXPERIMENT
Working through some remaining warts in my WinFsp-FUSE for Cygwin layer I
stumbled upon an interesting one last night. In one Cygwin window I ran
sshfs -d -o idmap=user @ Y:
Then in another:
billziss@windows:~$ cd /cygdrive/y
billziss@windows:/cygdrive/y$ ls -la
total 8
drwxr-xr-t 1 billziss Unknown+Group 0 Jun 23 23:57 .
dr-xr-xr-x 1 billziss None 0 Jun 24 10:13 ..
-rw-r--r-T 1 billziss Unknown+Group 15 Jun 23 23:57 Foo.txt
What are those sticky bits doing there?!
billziss@windows:/cygdrive/y$ cacls Foo.txt /S
Y:\Foo.txt
"D:P(A;;0x1f019f;;;S-1-5-21--1001)(A;;FR;;;S-1-0-0)(A;;FR;;;WD)"
Ok, so WinFsp-FUSE presents an ACL with the proper rights for the owner
(billziss), read rights for the NULL SID and read rights for the WORLD
SID. We see the NULL SID because WinFsp-FUSE maps unknown users to it
(nobody/nogroup).
This looks correct. How does Cygwin decide that the sticky bit is somehow
set?!
PERMISSION MAPPING IN WINFSP-FUSE
WinFsp-FUSE implements permissions as described in the "Permissions In
Microsoft Services for UNIX v3.0” [PERMS]. This document contains a
discussion of mapping the sticky and setuid/setgid bits. They use
combinations of the FILE_DELETE_CHILD access right to implement the sticky
bit and NTFS extended attributes to implement the setuid/setgid bits.
WinFsp-FUSE follows the same methodology for the sticky bit and ignores
the setuid/setgid bits.
WinFsp-FUSE also has a UID/SID mapping that is (mostly) compatible with
Cygwin’s. I do not implement the trustPosixOffset, because I am lazy and
because I do not understand it. Turns out that there is another
incompatibility: I map unknown UID’s to the NULL SID (S-1-0-0).
Could my mapping of the NULL SID somehow interfere with Cygwin’s ACL
mapping? No way right? Turns out that: yes! File:winsup/cygwin/sec_acl.cc,
line:787
if (ace_sid == well_known_null_sid)
{
/* Fetch special bits. */
attr |= CYG_ACE_ISBITS_TO_POSIX (ace->Mask);
if (ace->Header.AceType == ACCESS_DENIED_ACE_TYPE
&& ace->Mask & CYG_ACE_NEW_STYLE)
{
/* New-style ACL. Note the fact that a mask value is present
since that changes how getace fetches the information.
That's
fine, because the NULL deny ACE is supposed to precede all
USER, GROUP and GROUP_OBJ entries. Any ACL not created that
way has been rearranged by the Windows functionality to
create
the brain-dead "canonical" ACL order and is broken anyway.
*/
new_style = true;
attr |= CYG_ACE_ISBITS_TO_POSIX (ace->Mask);
So Cygwin appears to use NULL SID ACE’s to store the special bits. In
addition if the ACE is a special ACCESS_DENIED one it assigns even more
semantics to it.
Allow me to say that I find this a *gross* hack. You are subverting the
Windows ACL mechanism to store information that it was not designed to
store. I would love to hear a good rationale for this decision.
BTW, this also appears to break BashOnWindows: see [BASHW]
In any case I am seeking more information regarding Cygwin’s use of NULL
SID’s. I have found an old post that sheds some light [OPOST].
I am also seeking an alternative to using the NULL SID for
“nobody”/“nogroup”. Is there a Cygwin suggested one?
Bill
[PERMS] https://technet.microsoft.com/en-us/library/bb463216.aspx
[BASHW] https://github.com/Microsoft/BashOnWindows/issues/51
[OPOST] https://www.cygwin.com/ml/cygwin/2015-02/msg00197.html
Re: FUSE for Cygwin
On 6/22/16, 1:39 PM, "Jeffrey Altman" wrote: >On 6/22/2016 3:43 PM, Bill Zissimopoulos wrote: >> >> The bigger question is whether the Cygwin community would want a package >> like this. The obvious answer might be yes (I hope), but there is a >>large >> caveat. WinFsp includes a kernel-mode driver that needs to be built >>using >> Microsoft tools and signed using an EV certificate. In fact it looks >>like >> those requirements will only get harder as time passes -- soon we may >>need >> a sysdev account just to sign drivers. This means that the familiar >>model >> of getting the source and compiling everything using Cygwin tools cannot >> work here. > >I believe that as of Windows 10 Anniversary Edition and Server 2016 >Secure Boot becomes mandatory for new installations and with Secure Boot >comes the requirement that all device drivers (including file system >drivers) be signed by Microsoft. I agree. That is my understanding as well. Bill
Re: FUSE for Cygwin
Hi, Herbert: On 6/19/16, 1:32 PM, "cygwin-ow...@cygwin.com on behalf of Herbert Stocker" wrote: >>>G) Case sensitivity. >> >> WinFsp (and Windows) allows for both case-sensitive and case-insensitive >> file systems. > > >> FUSE file systems are marked as case-sensitive by default. > >If WinFsp marks FUSE file systems as case-sensitive then there is no >issue with case sensitiity. > >However we should have a look into OSXFUSE, it also has to deal with >case sensitivity. Let's see how they solved it. There's also MacFUSE >and Fuse4X according to Wikipedia. I am familiar with OSXFUSE as my primary development platform is OSX. OSXFUSE has various extensions to FUSE. In this case a file system can advertise that it is case insensitive by setting the case_insensitive bit in fuse_conn_info (passed to fuse_operations::init). As far as WinFsp-FUSE is concerned I have made it a command line option. I am happy to change how this is done though and perhaps conform to the OSXFUSE convention. >>you would now make those Win32 processes 2nd class citizens, >> as they would have to go through that intermediate process (which means >>4 >> context switches per file system request for them). > >You are right. I did not think about this extra process. Performance >wise Win32 processes would be 2nd class. i was just thinking about >feature wise, where they don't lose. > > >>> >Now this argument of yours makes me more biased towards mode (3). Let's >pay with some design unprettiness (remember pipes) and have the better >performance for both worlds. > >And besides, the translations are necessary anyway for mode (2). >Except for the pipes and hardlinks, they need not be supported in >mode (2). > >best regards, >as before, i am looking forward to use WinFsp and have FUSE available >for my Cygwin scripts and stuff. And even for windows processes. Thanks, Herbert. In the last few days I have also worked in a mode (3b): a Cygwin package that contains a cygfuse-2.8.dll that knows how to interface with the rest of the WinFsp machinery (FSD/DLL). This makes WinFsp-FUSE usable for a variety of FUSE file systems and libraries, including language bindings for Python, Perl, etc. For example, I have a small patch for FUSEPY (Python bindings for FUSE), which basically defines the Cygwin stat and statvfs structs using Python ctypes. Other than that FUSEPY just runs. [There are some small issues to be resolved, but the basis is there.] The bigger question is whether the Cygwin community would want a package like this. The obvious answer might be yes (I hope), but there is a large caveat. WinFsp includes a kernel-mode driver that needs to be built using Microsoft tools and signed using an EV certificate. In fact it looks like those requirements will only get harder as time passes -- soon we may need a sysdev account just to sign drivers. This means that the familiar model of getting the source and compiling everything using Cygwin tools cannot work here. The source code for the cygfuse package can be found below. You will notice that it is a rather thin layer on top of WinFsp-FUSE implemented in the WinFsp DLL. https://github.com/billziss-gh/winfsp/tree/master/opt/cygfuse Bill
N00b question regarding Cygwin and delay loading
TLDR: I am trying to use delay loading with Cygwin. Is it supported? I have found information that I can do something along the following lines: gendef NATIVE.dll dlltool --input-def NATIVE.def --output-delaylib NATIVE.dll.a gcc -shared -o cygNAME.dll -Wl,--out-implib=libNAME.a NAME.c NATIVE.dll.a However the messages that I get back say that I am missing the symbol __delayLoadHelper2. Which actually makes sense. But where is Cygwin’s delayimp.lib? Is there even one? Or am I supposed to somehow use this with MSVC’s delayimp.lib? Or perhaps this is just a MinGW feature and not supported on Cygwin? Thank for any help. Full (failing) make output below: << billziss@windows:~/Projects/winfsp/src/cygwin [libfuse]$ make gendef "/cygdrive/c/Program Files (x86)/WinFsp/bin/winfsp-x64.dll" * [/cygdrive/c/Program Files (x86)/WinFsp/bin/winfsp-x64.dll] Found PE+ image dlltool --input-def winfsp-x64.def --output-delaylib winfsp-x64.dll.a gcc -shared -o cygfuse.dll -Wl,--out-implib=libfuse.a -I../../inc/fuse cygfuse.c winfsp-x64.dll.a winfsp-x64.dll.a(dwjdh.o):fake:(.text+0x15): undefined reference to `__delayLoadHelper2' winfsp-x64.dll.a(dwjdh.o):fake:(.text+0x15): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__delayLoadHelper2' collect2: error: ld returned 1 exit status make: *** [Makefile:7: cygfuse.dll] Error 1 >> Bill
Re: FUSE for Cygwin
On 6/19/16, 4:20 AM, "cygwin-ow...@cygwin.com on behalf of Herbert Stocker" wrote: >What i don't like on (3) is that when a Cygwin process accesses the >FUSE file system there are two Cygwin processes whose communication >is translated from Posix to Win32 and then back (which is done again >for the response). [snip] >A) Besides the double double translations of every request, there >is also the need for four Kernel/Usermode transitions, which take >their time. By the way Herbert in re-reading your email, you may *not* have meant to say that in the current WinFsp/FUSE scheme there are 4 *context switches* (instead of 2), in which case my apologies for trying to explain something you already were aware of. Bill
Re: FUSE for Cygwin
Hi, Herbert:
On 6/19/16, 4:20 AM, "cygwin-ow...@cygwin.com on behalf of Herbert
Stocker" wrote:
>this is now my proposal of an alternative mode for WinFsp to support
>Cygwin based FUSE file systems. I'll call it mode (4)...
>
>To repeat your 3 modes of operation (in my words):...
>
>(3) The file system is a Cygwin process and thus can use all POSIX
> features of Cygwin. It needs no porting or only little.
>
>What i don't like on (3) is that when a Cygwin process accesses the
>FUSE file system there are two Cygwin processes whose communication
>is translated from Posix to Win32 and then back (which is done again
>for the response).
Actually I do not believe this is the case. Let me show again the complete
path of a creat() request, this time with context switches.
As before in the following OP is the “originating process", CW is the
"Cygwin runtime", NT is NTOS, WD is the "WinFsp FSD”, WL is the "WinFsp
DLL", FL is the "FUSE layer", and FS is the "user mode FUSE file system".
RUNNING in OP context
OP: call creat("/cygdrive/z/foo*bar")
CW: call NtCreateFile(L"\\foo\xf02abar”)
NT: call IRP_MJ_CREATE L"\\foo\xf02abar"
WD: post FspFsctlTransactCreateKind L"\\foo\xf02abar”
SWITCH to FS context
WL: recv FspFsctlTransactCreateKind,
call FSP_FILE_SYSTEM_INTERFACE::Create L"\\foo\xf02abar"
FL: call fuse_operations::create "/foo*bar"
FS: somehow satisfies fuse_operations::create
FS: return errno=0,
fuse_file_info::fh
FL: return STATUS_SUCCESS,
FSP_FSCTL_TRANSACT_RSP::Rsp.Create.Opened.UserContext2
WL: return STATUS_SUCCESS,
FSP_FSCTL_TRANSACT_RSP::Rsp.Create.Opened.UserContext2
WD: complete IRP_MJ_CREATE with
STATUS_SUCCESS, NTOS FILE_OBJECT properly initialized
SWITCH to OP context
NT: return STATUS_SUCCESS,
HANDLE opened by NTOS for FILE_OBJECT
CW: return fd for HANDLE just opened, (maybe?) set errno=0
As you can see there are only two context switches. The only two processes
involved are OP (the originating process) and FS (the user mode file
system).
[BTW, a WinFsp file system process is able to process multiple requests in
the same thread without a context switch, but from the perspective of an
OP there will be (at least) 2 context switches for a creat() response to
complete.]
>So my proposal is basically this:
>
>(4) The file system is a Cygwin process and Cygwin is extended to
> pass file system requests to that process if they fall into the
> respective path (where the file system is mounted.)
>
> This way the file system is exported to Cygwin only. But another
> tool exports this to the Win32 world.
It is my understanding from my brief foray into the Cygwin source that
Cygwin already has the fhandler mechanism, which looks like a natural
place to implement a Cygwin-only FUSE for Windows.
>Mode (4) would avoid the following issues coming from the double
>translation of mode (3):
>
>A) Besides the double double translations of every request, there
>is also the need for four Kernel/Usermode transitions, which take
>their time.
As explained, there are 2 context switches and not 4. If you had an
fhandler that somehow calls out into a FUSE file system, it would still
need to do 2 context switches (one to get the request to the FUSE file
system and one to get the response back).
The POSIX/Windows translations are actually not that expensive, although
they are not trivial (creating a security descriptor requires memory
allocations). But of course the dominant factor here is the context switch.
>C) i guess this way we could support fuse with all other languages
>besides C/C++.
BTW, I already had some luck last night getting FUSEPY to work with
WinFsp. I should have something out on this soon.
>E) Same for uid/gid to SID mapping.
>No need to implement or follow Cygwin.
>And how about the case where a uid/gid has no correspoinding SID?
>Can this happen?
Yes, it can, but this is an issue that Cygwin faces itself.
>F) Pipes:
> > [Quick experiment:
> >
> > $ mkfifo foo; cmd /c dir 'foo*' | grep foo; rm foo
> > 06/16/2016 11:02 PM 130 foo.lnk
> > ]
> >
> > Ok, so they are shortcuts. Naturally they are supported.
>
>I think they are not.
>The mkfifo system call will have Cygwin create a .lnk file and
>WinFsp will forward it as such to the file system process. The
>sytem calls readdir or open will then have the file system
>process tell WinFsp that there is a .lnk file and Cygwin will
>translate this back to a fifo, so in this sense it does work.
>
>But the file system will see a file (with name *.lnk) where it
>should see a pipe (mknod call with 'mode' set to S_IFIFO).
>IMHO one could say this is a break of the FUSE API.
Good catch this one!
Of course I can try to fix this in the WinFsp/FUSE layer implementation,
but the fix would probably be a gross and Cygwin-specific hack. But
definitely good catch!
>To fix this with mode (3) you'd have to recognize these .lnk
>
Re: FUSE for Cygwin - was: Re: Fork and Windows Heap
Hello, Jeffrey: On 6/18/16, 1:19 PM, "Jeffrey Altman" wrote: >On 6/18/2016 4:03 PM, Bill Zissimopoulos wrote: >> * A directory cannot be renamed if it or any of its subdirectories >> contains a file that has open handles (except in the batch-oplock case >> described earlier). >> >> >> In particular the third bullet point mandates that the FSD keeps >> information not only about files that are open, but also of their >> hierarchical relationships. The easiest way to do this on Windows is to >> maintain a mapping of file names to open files. > >This is not how my file system redirector enforces the rule. The file >control block (representing the handle) for an open file maintains a >reference to the directory object through which it was opened. As long >as the directory object has outstanding references the redirector fails >the rename request. Thank you for your very useful pointer. I have a question. In your file system redirector if the file "\comp1\comp2\name.ext” gets opened, do you also open (internally) the directories "\comp1" and "\comp1\comp2”? It sounds like you do. I understand that this is what UNIX VFS does, but it is my understanding that Windows does not require this. WinFsp avoids it so that it will not send extraneous requests to the user mode file system. WinFsp only knows about files that have been explicitly opened using IRP_MJ_CREATE. But perhaps I misunderstand what you are suggesting. Apologies for being thick, but can you please elaborate? >File path maps to specific files in fact do not work because the file >can be hard linked into more than one directory. Only the directory >that was used to create a file handle is restricted from renaming. I agree. But my FSD does not currently support hard links. If/when it does I will have to revisit this design. Many thanks for your insight. BTW, if your redirector is public/opensource I would love to take a look. Bill
Re: FUSE for Cygwin - was: Re: Fork and Windows Heap
On 6/18/16, 1:02 AM, "Corinna Vinschen" wrote: >>but I eventually had to change it for a number of issues (notably Rename >> support). > >For rename support you can use the index number as well, usually, >since you can open a file by index number. At least on NTFS. Unfortunately it is not as simple as that. A proper Rename implementation needs to conform to rules that effectively require the FSD to maintain filename/path information about open files. From "FILE_RENAME_INFORMATION structure” [1]: << Special rules for renaming open files: * A file cannot be renamed if it has any open handles, unless it is only open because of a batch opportunistic lock (oplock) and the batch oplock can be broken immediately. * A file cannot be renamed if a file with the same name exists and has open handles (except in the batch-oplock case described earlier). * A directory cannot be renamed if it or any of its subdirectories contains a file that has open handles (except in the batch-oplock case described earlier). >> In particular the third bullet point mandates that the FSD keeps information not only about files that are open, but also of their hierarchical relationships. The easiest way to do this on Windows is to maintain a mapping of file names to open files. >>I am not saying that it would not be possible to change this >> part of WinFsp, I just believe that it is a non-trivial change at this >> moment. > >Ok. I have thought more about this. As mentioned, originally the FSD maintained a mapping of IndexNumber to “FileNode” somewhat similar to UNIX. This is because it is important for an FSD to be able to identify when the same file is being reopened; for example, to properly implement file deletion (a file gets deleted only when its last handle is closed) or sharing or caching. When I had to implement Rename, bullet point (3) above complicated matters. So I ended up maintaining two tables for a while, one for IndexNumbers and one for file names. Then I ended up scrapping the IndexNumber table when I decided that I would not implement hard links (for a variety of reasons). Perhaps what I should have done instead is to maintain IndexNumber relationships (parent/child) and do my rename tests based on that. That’s what a VFS system would do, although I am not sure that it is the cleanest solution within a Windows FSD. Compounding this the user mode file system may not even send me back real IndexNumber’s, so then I would have to fake them (e.g. file name hash). In any case I am going to shelve this for a while and come back to it at a later time as there are lots of higher priority (for me) things to do on WinFsp. -- In other news I made a new release yesterday and I am happy to say that this release supports Cygwin. I am able to compile SSHFS with a minimal patch and it runs correctly under Cygwin. The SSHFS-Win repo is here [2]. I may actually go the extra mile and create a libfuse.dll so that things like FUSEPY can work out of the box. Bill [1] https://msdn.microsoft.com/en-us/library/windows/hardware/ff540344(v=vs.85) .aspx [2] https://bitbucket.org/billziss/sshfs-win
Re: FUSE for Cygwin
Hi, Herbert:
> > WinFsp provides three (3) different modes of integration:
[snip]
> i'm planning to make a suggestion of mode (4). It will be in addition or
> instead of (3) and will avoid those issues we touched.
I think (based on your earlier ask re: bindings to Python, Perl, etc.) I
may see what you mean with mode (4). Perhaps a libfuse Cygwin DLL that
presents an API *exactly* like the one from FUSE?
Currently WinFsp goes through a number of hoops to support the Win32 and
Cygwin environments with a single DLL. As a result a call to fuse_new is
really a call to fsp_fuse_new through a static inline function.
static inline struct fuse *fuse_new(...)
{
return fsp_fuse_new(fsp_fuse_env(), ...);
}
Fsp_fuse_env “captures” the environment (e.g. the local malloc/free) so
that the WinFsp DLL can remain environment independent. But this scheme
will not (and cannot) work for e.g. Python bindings that need to dlopen
the “fuse” library. Here is FUSEPY, as example:
https://github.com/terencehonles/fusepy/blob/master/fuse.py#L66
If that is indeed your plan I would support it.
Creating a libfuse Cygwin DLL like that should be easy. Perhaps I can help
by making the “static inline” part a macro such as FSP_FUSE_STATIC_INLINE,
which one could redefine to __attribute__ ((visibility("default")))
Bill
Re: FUSE for Cygwin - was: Re: Fork and Windows Heap
Hi, Corinna:
On Jun 17 07:25, Bill Zissimopoulos wrote:
> > Windows hard links are rather un-POSIX like and rarely used on Windows.
> > After considering the required changes on the FSD for a feature that is
> > so rarely used I opted against supporting them.
>
> I disagree here. Windows hardlinks work fine and pretty much as on
> POSIX with the exception of DOS mode bits. Those are not per file but
> per file entry as far as my experiecen goes. One of the reasons we try
> to ignore them as much as possible.
I no longer remember all the details now, because it was a few months ago
that I looked into this and determined that hard links are "un-POSIX like".
As far as I recall there was the FileAttributes issue (which I now think
may have been my incorrect understanding of the documentation), but there
was also the issue of FindFirstFileName/FindNextFileName, which is not
something that POSIX supports out of the box (AFAIK).
Regarding the FileAttributes issue. The Windows documentation seems to
suggest that they are stored with the file and not the directory entry.
I have not experimented though to confirm whether this is true.
From MSDN "Hard Links and Junctions" [1]:
<<
Note that the attributes on the file are reflected in every hard link
to that file, and changes to that file's attributes propagate to all
the hard links. For example if you reset the READONLY attribute on a
hard link to delete that particular hard link, and there are multiple
hard links to the actual file, then you will need to reset the READONLY
bit on the file from one of the remaining hard links to bring the file
and all remaining hard links back to the READONLY state.
>>
And then of course there is this ambiguous bit:
<<
However, the directory entry size and attribute information is updated
only for the link through which the change was made.
>>
I think that attribute information in this case means "NTFS attribute"
and not FILE_ATTRIBUTE_*. NTFS attributes are simply the method that
NTFS uses to store information about files.
Taken together this information suggests that NTFS hard links are more
"POSIX like" than I thought. In fact they may be "POSIX like with extra
capabilities".
The bigger issue is that the FSD now maintains an internal table of open
files that is indexed by file name. It actually started as a table of open
files indexed by IndexNumber (inode number Windows equivalent), but I
eventually had to change it for a number of issues (notably Rename
support). I am not saying that it would not be possible to change this
part of WinFsp, I just believe that it is a non-trivial change at this
moment.
> > > You write that you are mapping
> > > - characters not supported by Win32 but by POSIX via the Unicode
> > >private use page
> > > - Security apspects (SID vs. uid/gid, ACL)
> > > between POSIX and Windows and that you do it like Cygwin/SFU/SFM is
> > > doing it.
>
> Uhm... I don't understand. Cygwin is using the private unicode range as
> well to map characters invalid in a Windows filename (e.g. colon 0x3a is
> mapped to 0xf03a). This is how such filenames are given to the OS
>functions.
>
> This means, WinFsp would get these filenames from a Cygwin process
> already in the transposed form, with invalid FS chars transposed into the
> private area.
>
> If WinFsp would support that, it would be transparent to applications
> with very minor effort.
I think I may have inadequately explained what WinFsp does with this
mapping. It basically does the opposite of what Cygwin does on any given
code path.
Let's examine the lifetime of a call to creat(). Suppose a Cygwin process
does creat("/cygdrive/z/foo*bar"). In the following OP is the "originating
process", CW is the "Cygwin runtime", NT is NTOS, WD is the "WinFsp FSD",
WL is the "WinFsp DLL", FL is the "FUSE layer", and FS is the "user mode
FUSE file system".
OP: creat("/cygdrive/z/foo*bar")
CW: NtCreateFile(L"\\foo\xf02abar") <--- Cygwin translation
NT: IRP_MJ_CREATE L"\\foo\xf02abar"
WD: FspFsctlTransactCreateKind L"\\foo\xf02abar"
WL: FSP_FILE_SYSTEM_INTERFACE::Create L"\\foo\xf02abar"
FL: fuse_operations::create "/foo*bar" <--- WinFsp/FUSE
translation
FS: somehow satisfies fuse_operations::create
[snip return path]
The opposite happens when the file system communicates a file name back
to the originating process, as in readdir().
OP: readdir("/cygdrive/z/")
[snip call path]
FS: fuse_operations::readdir response: "foo*bar"
FL: FSP_FILE_SYSTEM_INTERFACE::ReadDirectory response: L"foo\xf02abar"
WL: FspFsctlTransactQueryDirectoryKind response: L"foo\xf02abar"
FUSE for Cygwin - was: Re: Fork and Windows Heap
[I apologize if my responses to the list appear to break the mailing list's threading model. I am not actually subscribed to the list and I respond to individual messages using my mail app.] Hello, Herbert: Herbert Stocker wrote: > > On 16.06.2016 08:37, Bill Zissimopoulos wrote: > > > > I have a Windows solution developed using Visual Studio > > and the Windows Driver Kit that I am building a FUSE > > compatibility layer for. > > this is a GREAT piece of work. > Although i did not dive into kernel development, i do understand that > writing a file system driver for Windows is surely lots of painful work, > with its undocumented and grown API. > > You should write a book documenting all the knowledge you gathered > through your research, if the only one available is from '97 as you say. > > Really, great work. > Also that you add the FUSE API. Thank you for your very kind words. > So, how is your architecture of the FUSE part of WinFsp? > Are you porting libfuse to Cygwin, so that a Cygwin process can link to > it and instead of receiving requests via /dev/fuse fro the Linux kernel, > your libfuse will receive IRPs through your user mode DLL from your > Windows kernel driver FSD? WinFsp consists of a kernel-mode file system driver (FSD) and a user mode dynamic link library (DLL). The FSD registers two devices in the NTOS (Windows kernel) namespace with names: \Device\WinFsp.Disk and \Device\WinFsp.Net. The DLL interacts with these devices using CreateFile and DeviceIoControl. The FSD receives IRP's by the kernel and if it cannot handle them itself, it posts them to an I/O queue. At a later time the user mode file system process receives the IRP's from the queue using DeviceIoControl. The user mode file system handles the IRP's and then send the responses back to the FSD, which then completes the IRP's. All this is of course nicely wrapped by an API so that the user-mode file system developer never sees IRP's. Instead IRP's are translated to callback calls with familiar names and semantics such as Create, Open, Read, Write, etc. The FUSE layer is a thin layer on top of this api. For example, the WinFsp Create callback maps to FUSE mkdir or create, the Open callback maps to FUSE opendir or open, etc. I am not porting libfuse to Windows. This is a complete reimplementation. For the full details on the WinFsp design please see this document: http://www.secfs.net/winfsp/develop/design/ For details on the FUSE port please see the WinFsp blog: http://www.secfs.net/winfsp/blog/ > Are you planning to have FUSE file systems be ported to native Windows > apps, i.e. don't link with cygwin1.dll, or will they run in a Cygwin > environment? WinFsp provides three (3) different modes of integration: (1) Its own native API, which allows a user mode file system to do almost anything NTFS can do (with a few exceptions). It also allows for the Read, Write and ReadDirectory callbacks to be asynchronous (return STATUS_PENDING). I recommend this API for file systems that want maximum performance and integration with Windows. (2) A FUSE (high-level) API for Win32 (i.e. non-Cygwin) applications. This is useful if one has a FUSE file system that they can easily port to Windows or they do not want any Cygwin dependencies. (3) A FUSE (high-level) API for Cygwin. As you correctly note many FUSE file systems are too POSIX specific (as they were born on Linux/OSX) and they only make sense in a Cygwin environment. I expect that most FUSE file systems will probably go for option (3) initially. Then they may want to consider going to (2) if they can easily port their core file system code to Windows. Then they may even consider (1) if they have needs that the FUSE API cannot support (e.g. full support for Windows ACL's). > If porting to native app, this would be easier for the user to install > them, but: > - FUSE has bindings to many languages like Perl, PHP, etc. Will these >work then? I expect that parts of the bindings will have to be rewritten. WinFsp does not pretend to be FUSE, it only has a FUSE API for someone that writes C or C++ and has a compiler that understands #include > How are you dealing with limitations of the Windows file system as seen > from a POSIX perspective? You say you can't support hard links. Windows hard links are rather un-POSIX like and rarely used on Windows. After considering the required changes on the FSD for a feature that is so rarely used I opted against supporting them. I expect that POSIX file systems with hard link support need to take some care when exposed through WinFsp. But I admit that I have not thought too hard about this problem and I do not have any solid recommendations or solutions at this point. > How will a Cygwin program see symlinks exported by the FUSE file system? WinFsp will presen
Re: Fork and Windows Heap
Hi, Corinna: > You are correct. Cygwin fork only clones the datastructures explicitely > set up by Cygwin and stuff allocated using Cygwin's POSIX API. > > You can't simply clone a Windows heap for various reasons... Thank you for your detailed response and explanation. Bill
Re: Fork and Windows Heap
Renà Berber wrote: > On 6/15/2016 7:42 PM, Bill Zissimopoulos wrote: > > (1) Is my assumption that Windows heaps are not properly cloned after a > > fork correct? A 2011 post [2] seems to suggest so. > > (2) Is there any workaround that the WinFsp DLL can use to get around >this > > limitation and work properly after a fork? The obvious answer would be >to > > have the DLL use Cygwin's malloc/free and this is indeed possible >within > > the DLL's FUSE layer, but not workable elsewhere. > > Those are the wrong questions: you shouldn't be mixing Windows and Unix > (SuSv4, Posix) APIs, and by your description you are trying a Windows > port, mixing it with probably the base, and expecting it to work on > Cygwin, that's a no-go from the start. Thank you for your response. With all due respect however I think you have given me the wrong answers/pointers. I am not porting anything from UNIX. I have a Windows solution developed using Visual Studio and the Windows Driver Kit that I am building a FUSE compatibility layer for. My DLL is not a Cygwin DLL. It is a native Windows DLL that also has a FUSE compatibility layer. I am taking pains to make that FUSE compatibility layer available to both Win32 and Cygwin apps. The answers to my questions are: (1) Cygwin does not clone Windows heaps after a fork. [At least my brief perusal of winsup/cygwin/fork.cc seems to indicate so.] (2) The workaround is to avoid allocating resources that Cygwin cannot account for (e.g. from the Windows heap) prior to daemon/fork. Luckily this is possible in a FUSE file system design, because in general it looks like this: fuse_new fuse_daemonize // do not allocate any non-Cygwin resources prior to this fuse_loop/fuse_loop_mt // safe to allocate non-Cygwin resources fuse_destroy I have now modified the WinFsp FUSE layer accordingly and tested it against SSHFS. I am able to daemonize SSHFS and everything works correctly. I therefore consider this problem solved for me. Bill
Fork and Windows Heap
I am the creator of WinFsp [1], a user mode file system solution for Windows (i.e. FUSE for Windows). WinFsp has its own API, but I have been working on adding a FUSE (high-level) API using SSHFS as my primary test case. This has proven to work very well, except for one important problem which is the subject of my post. FUSE provides an API fuse_daemonize() that allows a FUSE file system to become a daemon, which SSHFS uses. I have provided a simple implementation using daemon(3), which of course uses fork. Unfortunately SSHFS crashes whenever it is daemonized and I have traced the failure to an invalid Windows heap pointer. WinFsp consists of an FSD (file system driver) and a DLL component. The WinFsp DLL uses the Windows heap functions to manage memory (HeapAlloc/HeapFree). It seems that the Windows heap is not properly cloned after a fork, which leads to the crash I am experiencing. I have the following questions: (1) Is my assumption that Windows heaps are not properly cloned after a fork correct? A 2011 post [2] seems to suggest so. (2) Is there any workaround that the WinFsp DLL can use to get around this limitation and work properly after a fork? The obvious answer would be to have the DLL use Cygwin's malloc/free and this is indeed possible within the DLL's FUSE layer, but not workable elsewhere. Any insights welcome. Bill Zissimopoulos [1] http://www.secfs.net/winfsp/ [2] https://www.cygwin.com/ml/cygwin-developers/2011-04/msg00035.html
