Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation
Hi Adam, On 2015-08-09 19:05, Adam Dinwoodie wrote: > On 09/08/2015 10:01, Johannes Schindelin wrote: >> On 2015-08-09 04:01, Adam Dinwoodie wrote: >>> We've gotten a lot of users on the list who ask why their Git directories on shared drives aren't working (or are broken in some way). Since I don't use Windows, let me ask: does the Cygwin DLL handle link(2) properly on shared drives, and if not, would this patch help it do so? I can imagine that perhaps SMB doesn't support the necessary operations to make a POSIX link(2) work properly. >>> >>> I'd need to go back to the Cygwin list to get a definite answer, but as >>> I understand it, yes, this is is exactly the problem -- quoting Corinna, >>> one of the Cygwin project leads, "The MS NFS is not very reliable in >>> keeping up with changes to metadata." >>> >>> We have verified that setting `core.createobject rename` resolves the >>> problem for people who are seeing it, which very strongly implies that >>> this build option would solve the problem similarly, but would fix it >>> for all users, not just those who spend enough time investigating the >>> problem to find that setting. >> >> From my experience, it appears that providing Corinna Vinschen (or better >> put: the Cygwin developers in general) with a sound patch gets things fixed >> pretty timely. >> >> And since `core.createObject = rename` seems to work around the problem, it >> should be possible to patch the Cygwin runtime accordingly. Sure, it will >> take a little investigation *what* code should be changed, and how, but the >> obvious benefit to *all* Cygwin applications should make that effort worth >> your while. > > Hmm. I'm not sure what a Cygwin fix would look like here. As I > understand it, using link(2) will fail if the target exists, while > using rename(2) will just clobber a target file. Thanks to you and Junio for explaining why my idea was bad. Sorry! Ciao, Johannes -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation
On Mon, Aug 10, 2015 at 12:08:29PM -0700, Junio C Hamano wrote: > But in the end, I'd prefer the choice of the compiled-in default up > to the port maintainers. You earlier said: > > This problem was reported on the Cygwin mailing list at > https://cygwin.com/ml/cygwin/2015-08/msg00102.html (amongst others) and > is being applied as a manual patch to the Cygwin builds until the patch > is taken here. > > so my preference is to see Cygwin continue to do so for a couple > release cycles of ours to make sure all Cygwin end-users are happy > and consider the flip of the default a good change for them, and > then the official Cygwin packager of Git sends a patch our way. Wait a few releases then resubmit assuming we've not had complaints from Cygwin user. Okay! > https://cygwin.com/ml/cygwin/2015-08/msg00102.html seems to indicate > that somebody called Adam Dinwoodie is wearing Git maintainer hat, > so perhaps you may be that "official Cygwin packager of Git" ;-) That would indeed be yours truly :) > I agree with everything you said in that message to Peter---the > patch should be included when you hear reports of `git config > core.createObject rename` helping more people. After versions of > Cygwin Git package with such a change proves good, let's reduce the > workload of Cygwin Git maintainer by upstreaming that change to my > tree. But not before. Cool. FWIW, the reason we've started applying this patch to the downstream Cygwin builds is that I've now seen a number of reports (primarily on Stack Overflow) of this problem where that config change has fixed things. I'd been holding off until I had those extra reports, but now I have them I made the patch and figured I'd submit it upstream at the same time. As above, I'll wait a while until we're happy it's stable, and resubmit at that point. (On which note, I should probably submit the patch to the git-gui Makefile we've had in our builds since back in v1.7.9, before I took over the maintainership, since that one clearly is pretty stable...) Thanks! -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation
Adam Dinwoodie writes: > If the desired goal is that Cygwin's link(2) acts like POSIX link(2) > on network drives, I'm not convinced that's possible, at least not by > emulating `core.createObject = rename` in the Cygwin library > layer. The way core.createObject=rename makes things work is by avoiding link(2) in the first place and using rename(2) instead. You might be able to emulate rename(2) of A to B by doing a link(2) of A to B and then unlink(2) of A, but I do not think it is reasonable for the system call emulation layer to detect a sequence of link followed by unlink and use rename, i.e. emulationg the other way around. So I suspect "fix in Cygwin" is a non-starter. But in the end, I'd prefer the choice of the compiled-in default up to the port maintainers. You earlier said: This problem was reported on the Cygwin mailing list at https://cygwin.com/ml/cygwin/2015-08/msg00102.html (amongst others) and is being applied as a manual patch to the Cygwin builds until the patch is taken here. so my preference is to see Cygwin continue to do so for a couple release cycles of ours to make sure all Cygwin end-users are happy and consider the flip of the default a good change for them, and then the official Cygwin packager of Git sends a patch our way. https://cygwin.com/ml/cygwin/2015-08/msg00102.html seems to indicate that somebody called Adam Dinwoodie is wearing Git maintainer hat, so perhaps you may be that "official Cygwin packager of Git" ;-) I agree with everything you said in that message to Peter---the patch should be included when you hear reports of `git config core.createObject rename` helping more people. After versions of Cygwin Git package with such a change proves good, let's reduce the workload of Cygwin Git maintainer by upstreaming that change to my tree. But not before. Thanks. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation
On 09/08/2015 10:01, Johannes Schindelin wrote: On 2015-08-09 04:01, Adam Dinwoodie wrote: I do not see any difference between the situation here and the situation for MinGW, which is fundamentally a Cygwin fork, but which already has this build option set for it in config.mak.uname. This is incorrect. MinGW is distinctly *not* a Cygwin fork. MinGW means "Minimal GNU on Windows" and that in turn means that it provides an environment to build executables that purely use the Win32 API. Read: no POSIX emulation whatsoever. Most notably, MinGW programs cannot use fork(2); It is simply unavailable. What you *probably* meant is that Git for Windows relies on MSys2 for its shell and Perl scripts, and that MSys2 in turn is a fork of Cygwin. That affects *only* the scripts, though; Git itself (as in `git.exe`) is still a pure MinGW program (and as a consequence, is quite a bit faster than Cygwin Git, at the price of certain quirks that Cygwin Git does not suffer). You're quite right; I'd misread the MinGW page. Thank you for pointing that out :) We've gotten a lot of users on the list who ask why their Git directories on shared drives aren't working (or are broken in some way). Since I don't use Windows, let me ask: does the Cygwin DLL handle link(2) properly on shared drives, and if not, would this patch help it do so? I can imagine that perhaps SMB doesn't support the necessary operations to make a POSIX link(2) work properly. I'd need to go back to the Cygwin list to get a definite answer, but as I understand it, yes, this is is exactly the problem -- quoting Corinna, one of the Cygwin project leads, "The MS NFS is not very reliable in keeping up with changes to metadata." We have verified that setting `core.createobject rename` resolves the problem for people who are seeing it, which very strongly implies that this build option would solve the problem similarly, but would fix it for all users, not just those who spend enough time investigating the problem to find that setting. From my experience, it appears that providing Corinna Vinschen (or better put: the Cygwin developers in general) with a sound patch gets things fixed pretty timely. And since `core.createObject = rename` seems to work around the problem, it should be possible to patch the Cygwin runtime accordingly. Sure, it will take a little investigation *what* code should be changed, and how, but the obvious benefit to *all* Cygwin applications should make that effort worth your while. Hmm. I'm not sure what a Cygwin fix would look like here. As I understand it, using link(2) will fail if the target exists, while using rename(2) will just clobber a target file. If the desired goal is that Cygwin's link(2) acts like POSIX link(2) on network drives, I'm not convinced that's possible, at least not by emulating `core.createObject = rename` in the Cygwin library layer. The only implementations I can think of would introduce a window condition between checking for the target file's existence and then clobbering it. That'd mean the vast majority of the time the Cygwin command would work as expected, but there'd be race conditions where occasionally a file gets silently clobbered. Intermittent silent bugs seem much worse than the command reliably failing. I think, given that, the current behaviour is preferable: calling link just errors out in this scenario, because there's no way to map it down to something that reliably acts like the POSIX call. It's then up to whatever program is trying to use the call to find an alternative way to achieve its desired result, and the consequences of that workaround are the responsibility of the calling application. In short, as I understand it, this is exactly the scenario OBJECT_CREATION_USES_RENAMES is intended for. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation
Hi Adam, On 2015-08-09 04:01, Adam Dinwoodie wrote: > I do not see any difference between the situation here and the situation > for MinGW, which is fundamentally a Cygwin fork, but which already has > this build option set for it in config.mak.uname. This is incorrect. MinGW is distinctly *not* a Cygwin fork. MinGW means "Minimal GNU on Windows" and that in turn means that it provides an environment to build executables that purely use the Win32 API. Read: no POSIX emulation whatsoever. Most notably, MinGW programs cannot use fork(2); It is simply unavailable. What you *probably* meant is that Git for Windows relies on MSys2 for its shell and Perl scripts, and that MSys2 in turn is a fork of Cygwin. That affects *only* the scripts, though; Git itself (as in `git.exe`) is still a pure MinGW program (and as a consequence, is quite a bit faster than Cygwin Git, at the price of certain quirks that Cygwin Git does not suffer). >> We've gotten a lot of users on the list who ask why their Git >> directories on shared drives aren't working (or are broken in some way). >> Since I don't use Windows, let me ask: does the Cygwin DLL handle >> link(2) properly on shared drives, and if not, would this patch help it >> do so? I can imagine that perhaps SMB doesn't support the necessary >> operations to make a POSIX link(2) work properly. > > I'd need to go back to the Cygwin list to get a definite answer, but as > I understand it, yes, this is is exactly the problem -- quoting Corinna, > one of the Cygwin project leads, "The MS NFS is not very reliable in > keeping up with changes to metadata." > > We have verified that setting `core.createobject rename` resolves the > problem for people who are seeing it, which very strongly implies that > this build option would solve the problem similarly, but would fix it > for all users, not just those who spend enough time investigating the > problem to find that setting. >From my experience, it appears that providing Corinna Vinschen (or better put: >the Cygwin developers in general) with a sound patch gets things fixed pretty >timely. And since `core.createObject = rename` seems to work around the problem, it should be possible to patch the Cygwin runtime accordingly. Sure, it will take a little investigation *what* code should be changed, and how, but the obvious benefit to *all* Cygwin applications should make that effort worth your while. Please note that Cygwin's source code itself is in Git now, too: https://cygwin.com/git.html Ciao, Johannes -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation
On Sat, Aug 08, 2015 at 09:06:28PM +, brian m. carlson wrote: > On Sat, Aug 08, 2015 at 04:47:46PM -0400, Mark Levedahl wrote: > > On 08/07/2015 04:30 PM, Adam Dinwoodie wrote: > > >When generating build options for Cygwin, enable > > >OBJECT_CREATION_USES_RENAMES. This is necessary to use Git on Windows > > >shared directories, and is already enabled for the MinGW and plain > > >Windows builds. > > > > I've been supporting use of git on cygwin since about 2008, this issue has > > never risen that I know. Whatever issue is being patched around here, if > > truly repeatable, should be handled by the cygwin dll as that code is > > focused on providing full linux compatibility. If git on linux does need > > this patch, git on cygwin should not, either. So, I vote against this. There has been recent and historical discussion on the Cygwin mailing list about this problem, as well as in other places like Stack Overflow. I've put a link to one of those discussions on the Cygwin mailing list in the original patch email. I can also see some discussiions on this list that seem related (search archives for "failed to read delta-pack base object" and "Cygwin"). It may be the technically correct approach that the Cygwin library ought to fix this, and indeed some improvements have been made in this area. However given the limited interfaces that Windows offers here, a final fix is very unlikely to come any time soon, so this patch is the pragmatic solution. I do not see any difference between the situation here and the situation for MinGW, which is fundamentally a Cygwin fork, but which already has this build option set for it in config.mak.uname. > We've gotten a lot of users on the list who ask why their Git > directories on shared drives aren't working (or are broken in some way). > Since I don't use Windows, let me ask: does the Cygwin DLL handle > link(2) properly on shared drives, and if not, would this patch help it > do so? I can imagine that perhaps SMB doesn't support the necessary > operations to make a POSIX link(2) work properly. I'd need to go back to the Cygwin list to get a definite answer, but as I understand it, yes, this is is exactly the problem -- quoting Corinna, one of the Cygwin project leads, "The MS NFS is not very reliable in keeping up with changes to metadata." We have verified that setting `core.createobject rename` resolves the problem for people who are seeing it, which very strongly implies that this build option would solve the problem similarly, but would fix it for all users, not just those who spend enough time investigating the problem to find that setting. Adam -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation
On Sat, Aug 08, 2015 at 04:47:46PM -0400, Mark Levedahl wrote: > On 08/07/2015 04:30 PM, Adam Dinwoodie wrote: > >When generating build options for Cygwin, enable > >OBJECT_CREATION_USES_RENAMES. This is necessary to use Git on Windows > >shared directories, and is already enabled for the MinGW and plain > >Windows builds. > I've been supporting use of git on cygwin since about 2008, this issue has > never risen that I know. Whatever issue is being patched around here, if > truly repeatable, should be handled by the cygwin dll as that code is > focused on providing full linux compatibility. If git on linux does need > this patch, git on cygwin should not, either. So, I vote against this. We've gotten a lot of users on the list who ask why their Git directories on shared drives aren't working (or are broken in some way). Since I don't use Windows, let me ask: does the Cygwin DLL handle link(2) properly on shared drives, and if not, would this patch help it do so? I can imagine that perhaps SMB doesn't support the necessary operations to make a POSIX link(2) work properly. -- brian m. carlson / brian with sandals: Houston, Texas, US +1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187 signature.asc Description: Digital signature
Re: [PATCH] config.mak.uname: Cygwin: Use renames for creation
On 08/07/2015 04:30 PM, Adam Dinwoodie wrote: When generating build options for Cygwin, enable OBJECT_CREATION_USES_RENAMES. This is necessary to use Git on Windows shared directories, and is already enabled for the MinGW and plain Windows builds. This problem was reported on the Cygwin mailing list at https://cygwin.com/ml/cygwin/2015-08/msg00102.html (amongst others) and is being applied as a manual patch to the Cygwin builds until the patch is taken here. Reported-by: Peter Rosin Signed-off-by: Adam Dinwoodie --- config.mak.uname | 1 + 1 file changed, 1 insertion(+) diff --git a/config.mak.uname b/config.mak.uname index 943c439..be5cbec 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -187,6 +187,7 @@ ifeq ($(uname_O),Cygwin) X = .exe UNRELIABLE_FSTAT = UnfortunatelyYes SPARSE_FLAGS = -isystem /usr/include/w32api -Wno-one-bit-signed-bitfield + OBJECT_CREATION_USES_RENAMES = UnfortunatelyNeedsTo endif ifeq ($(uname_S),FreeBSD) NEEDS_LIBICONV = YesPlease I've been supporting use of git on cygwin since about 2008, this issue has never risen that I know. Whatever issue is being patched around here, if truly repeatable, should be handled by the cygwin dll as that code is focused on providing full linux compatibility. If git on linux does need this patch, git on cygwin should not, either. So, I vote against this. Mark -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] config.mak.uname: Cygwin: Use renames for creation
When generating build options for Cygwin, enable OBJECT_CREATION_USES_RENAMES. This is necessary to use Git on Windows shared directories, and is already enabled for the MinGW and plain Windows builds. This problem was reported on the Cygwin mailing list at https://cygwin.com/ml/cygwin/2015-08/msg00102.html (amongst others) and is being applied as a manual patch to the Cygwin builds until the patch is taken here. Reported-by: Peter Rosin Signed-off-by: Adam Dinwoodie --- config.mak.uname | 1 + 1 file changed, 1 insertion(+) diff --git a/config.mak.uname b/config.mak.uname index 943c439..be5cbec 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -187,6 +187,7 @@ ifeq ($(uname_O),Cygwin) X = .exe UNRELIABLE_FSTAT = UnfortunatelyYes SPARSE_FLAGS = -isystem /usr/include/w32api -Wno-one-bit-signed-bitfield + OBJECT_CREATION_USES_RENAMES = UnfortunatelyNeedsTo endif ifeq ($(uname_S),FreeBSD) NEEDS_LIBICONV = YesPlease -- 2.4.5 -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
