Re: FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"
On Wed, Dec 03, 2014 at 06:31:18PM -0500, Jason Pyeron wrote: > I remember hitting this a while ago, but just gave up. > > It seems to be a problem for others too. > > Any ideas on how to debug this so it can be patched? > > -Original Message- > From: Dave Lindbergh > Sent: Wednesday, December 03, 2014 18:07 > To: cygwin > > Aha - you're right. > > It works fine on a local NTFS volume. > > I get the error when I do it on Z:, which is mapped to a network drive > (on another Windows box). > > Is there a workaround? Why does this happen? I don't think anyone is sure. My wild guess is that there's something about the way that Cygwin wraps Windows calls that makes it fail in this case. It might be interesting to run the Windows and Cygwin versions under an strace equivalent and see where things differ. It's an interesting problem, but I don't have any Windows systems, so I can't look into it. -- brian m. carlson / brian with sandals: Houston, Texas, US +1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187 signature.asc Description: Digital signature
RE: FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"
> -Original Message- > From: brian m. carlson > Sent: Wednesday, December 03, 2014 19:55 > > On Wed, Dec 03, 2014 at 06:31:18PM -0500, Jason Pyeron wrote: > > I remember hitting this a while ago, but just gave up. > > > > It seems to be a problem for others too. > > > > Any ideas on how to debug this so it can be patched? > > > > -Original Message- > > From: Dave Lindbergh > > Sent: Wednesday, December 03, 2014 18:07 > > To: cygwin > > > > Aha - you're right. > > > > It works fine on a local NTFS volume. > > > > I get the error when I do it on Z:, which is mapped to a > network drive > > (on another Windows box). > > > > Is there a workaround? Why does this happen? > > I don't think anyone is sure. My wild guess is that there's something > about the way that Cygwin wraps Windows calls that makes it > fail in this > case. It might be interesting to run the Windows and Cygwin versions > under an strace equivalent and see where things differ. [this was posted to the cygwin list] http://nerdfever.com/files/strace.txt > > It's an interesting problem, but I don't have any Windows > systems, so I > can't look into it. If it becomes a little less magic, I will chomp on the problem, but I cannot make a predictable test case. -Jason -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100- - +1 (443) 269-1555 x333Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"
> -Original Message-
> From: Jason Pyeron
> Sent: Thursday, December 04, 2014 16:34
> To: git@vger.kernel.org
> Cc: 'brian m. carlson'
> Subject: RE: FW: [cygwin] Cygwin's git says "error:
> failed to read delta-pack base object"
>
> > -Original Message-
> > From: brian m. carlson
> > Sent: Wednesday, December 03, 2014 19:55
> >
> > On Wed, Dec 03, 2014 at 06:31:18PM -0500, Jason Pyeron wrote:
> > > I remember hitting this a while ago, but just gave up.
> > >
> > > It seems to be a problem for others too.
> > >
> > > Any ideas on how to debug this so it can be patched?
> > >
> > > -Original Message-
> > > From: Dave Lindbergh
> > > Sent: Wednesday, December 03, 2014 18:07
> > > To: cygwin
> > >
> > > Aha - you're right.
> > >
> > > It works fine on a local NTFS volume.
> > >
> > > I get the error when I do it on Z:, which is mapped to a
> > network drive
> > > (on another Windows box).
> > >
> > > Is there a workaround? Why does this happen?
> >
> > I don't think anyone is sure. My wild guess is that
> there's something
> > about the way that Cygwin wraps Windows calls that makes it
> > fail in this
> > case. It might be interesting to run the Windows and
> Cygwin versions
> > under an strace equivalent and see where things differ.
>
> [this was posted to the cygwin list]
>
> http://nerdfever.com/files/strace.txt
>
> >
> > It's an interesting problem, but I don't have any Windows
> > systems, so I
> > can't look into it.
>
> If it becomes a little less magic, I will chomp on the
> problem, but I cannot make a predictable test case.
Corrina at Cygwin devined the strace (see below) and I am working on a test
cases and hacks.
Pseudo code and observations
./sha1_file.c:write_loose_object(sha1)
{
filename=sha1_file_name(sha1)
(fd,tmp_file)=create_tmpfile(filename)
write_buffer(fd)
close_sha1_file(fd)
return move_temp_to_file(tmp_file, filename)
}
move_temp_to_file(tmpfile, filename)
{
// I am thinking about forcing renames to see if the problem exists then as
well
// if that "works" then a per repo config option allowing for forced renames
if (OBJECT_CREATION_USES_RENAMES) goto try_rename
else if link(tmpfile,filename)
if (failed except exist failures)
{
try_rename:
rename(tmpfile, filename)
if (ok) goto out
}
unlink_or_warn(tmpfile)
if (failed except exist failures) return error
out:
}
write_sha1_file(sha1)
{
return write_loose_object(sha1)
}
> -Original Message-
> From: Corinna Vinschen
> Sent: Friday, December 05, 2014 6:35
> To: cyg...@cygwin.com
> What I found in the strace is this:
>
> - Create file Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ
>
> - open file, write something, close file.
>
> - link (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ,
>
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4)
> succeeds.
>
> - unlink (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ) succeeds
>
> - Trying to open
>
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4
> but the file doesn't exist and NtCreateFile fails with status
> 0xC034, STATUS_OBJECT_NAME_NOT_FOUND --> ENOENT.
>
> - Subsequent unlink (Z:\pic32mx-bmf\.git\objects\30) fails with a
> STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY.
>
> - git seems to be prepared for such a case, the parent process calls
> opendir/readdir on the directory. Enumerating the files in
> Z:\pic32mx-bmf\.git\objects\30 shows the entries ".", ".." and
> "0bdeb2fd209d24afb865584da10b78aa8fefc4".
>
> - Then git calls lstat on the file, which results in NtOpenFile
> returning status STATUS_OBJECT_NAME_NOT_FOUND again.
>
> - From a POSIX POV this means "somebody else" deleted the file,
> so the dir is empty now. Git tries to delete the directory again,
> which again results in STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY
> and, internally, a sharing violation which disallows to move the
> directory out of the way.
>
> This looks suspiciously like a bug in the remote filesystem. Link
> succeeded, so there are two links to the same file in the directory.
> Unlinking link 1 succeeds, so there's still one link to the
> file in the
> directory, but link 2 is inaccessible as if the file has been deleted
> completely. Thus, a full POSIX git on this drive is broken.
>
> Can you please run
>
> /usr/lib/csih/getVolInfo /cygdriv
RE: FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"
TLDR = Cygwin remote filesystem sometimes has strange failures -> workaround is
to use rename, not link/unlink;
see
https://github.com/pdinc-oss/git/commit/5a36824ed01d4335148ca3846e75cc99c11650e2
> -Original Message-
> From: Jason Pyeron
> Sent: Friday, December 05, 2014 10:30
>
> > -Original Message-
> > From: Jason Pyeron
> > Sent: Thursday, December 04, 2014 16:34
> >
> > > -Original Message-
> > > From: brian m. carlson
> > > Sent: Wednesday, December 03, 2014 19:55
> > >
> > > On Wed, Dec 03, 2014 at 06:31:18PM -0500, Jason Pyeron wrote:
> > > > I remember hitting this a while ago, but just gave up.
> > > >
> > > > It seems to be a problem for others too.
> > > >
> > > > Any ideas on how to debug this so it can be patched?
> > > >
> > > > -Original Message-
> > > > From: Dave Lindbergh
> > > > Sent: Wednesday, December 03, 2014 18:07
> > > > To: cygwin
> > > >
> > > > Aha - you're right.
> > > >
> > > > It works fine on a local NTFS volume.
> > > >
> > > > I get the error when I do it on Z:, which is mapped to a
> > > network drive
> > > > (on another Windows box).
> > > >
> > > > Is there a workaround? Why does this happen?
I have a really hacky workaround, commit
5a36824ed01d4335148ca3846e75cc99c11650e2 comments out the logic and forces a
rename instead of link and unlink.
https://github.com/pdinc-oss/git/tree/cygwin-issue-remoteCIFS-rename
> Pseudo code and observations
> ./sha1_file.c:write_loose_object(sha1)
> {
> filename=sha1_file_name(sha1)
> (fd,tmp_file)=create_tmpfile(filename)
> write_buffer(fd)
> close_sha1_file(fd)
> return move_temp_to_file(tmp_file, filename)
> }
>
> move_temp_to_file(tmpfile, filename)
> {
> // I am thinking about forcing renames to see if the problem
> exists then as well
> // if that "works" then a per repo config option allowing
> for forced renames
> if (OBJECT_CREATION_USES_RENAMES) goto try_rename
> else if link(tmpfile,filename)
>
Dave has tested a build I made for him on 64 bit Cygwin and it works. I no
longer have access to the environment I was having this problem in last
February. I will try to investigate this further, but I am not hopeful, maybe
Corinna will have luck on the issue. But I was in a secure corporate
environment and I thought the host based security system (AV), coupled with the
remote file system was causing the problem, namely files created are not
available instantly.
I do think that we should have a config option for this, as most users who
could encounter such a problem are not likely to be able (or allowed) to
rebuild the git executable themselves.
> > -Original Message-
> > From: Corinna Vinschen
> > Sent: Friday, December 05, 2014 6:35
> > To: cyg...@cygwin.com
>
> > What I found in the strace is this:
> >
> > - Create file Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ
> >
> > - open file, write something, close file.
> >
> > - link (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ,
> >
> >
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4)
> > succeeds.
> >
> > - unlink (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ) succeeds
> >
> > - Trying to open
> >
> >
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4
> > but the file doesn't exist and NtCreateFile fails with status
> > 0xC034, STATUS_OBJECT_NAME_NOT_FOUND --> ENOENT.
> >
> > - Subsequent unlink (Z:\pic32mx-bmf\.git\objects\30) fails with a
> > STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY.
> >
> > - git seems to be prepared for such a case, the parent process calls
> > opendir/readdir on the directory. Enumerating the files in
> > Z:\pic32mx-bmf\.git\objects\30 shows the entries ".", ".." and
> > "0bdeb2fd209d24afb865584da10b78aa8fefc4".
> >
> > - Then git calls lstat on the file, which results in NtOpenFile
> > returning status STATUS_OBJECT_NAME_NOT_FOUND again.
> >
> > - From a POSIX POV this means "somebody else" deleted the file,
> > so the dir is empty now. Git tries to delete the directory again,
> > which again results in STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY
> > and, internally, a sharing violation which disallows to move the
> > directory out of the way.
> >
> > This looks suspiciously like a bug in the remote filesystem. Link
> > succeeded, so there are two links to the same file in the directory.
> > Unlinking link 1 succeeds, so there's still one link to the
> > file in the
> > directory, but link 2 is inaccessible as if the file has
> been deleted
> > completely. Thus, a full POSIX git on this drive is broken.
> >
> > Can you please run
> >
> > /usr/lib/csih/getVolInfo /cygdrive/z
> >
> > and paste the output here? Maybe I can workaround this in the next
> > Cygwin version.
Dave's response: https://www.cygwin.com/ml/cygwin/2014-12/msg00066.html
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
- -
RE: SPAM: RE: FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"
> -Original Message-
> From: git-ow...@vger.kernel.org
> [mailto:git-ow...@vger.kernel.org] On Behalf Of Jason Pyeron
> Sent: Thursday, December 04, 2014 16:34
> To: git@vger.kernel.org
> Cc: 'brian m. carlson'
> Subject: SPAM: RE: FW: [cygwin] Cygwin's git says "error:
> failed to read delta-pack base object"
>
> > -Original Message-
> > From: brian m. carlson
> > Sent: Wednesday, December 03, 2014 19:55
> >
> > On Wed, Dec 03, 2014 at 06:31:18PM -0500, Jason Pyeron wrote:
> > > I remember hitting this a while ago, but just gave up.
> > >
> > > It seems to be a problem for others too.
> > >
> > > Any ideas on how to debug this so it can be patched?
> > >
> > > -Original Message-
> > > From: Dave Lindbergh
> > > Sent: Wednesday, December 03, 2014 18:07
> > > To: cygwin
> > >
> > > Aha - you're right.
> > >
> > > It works fine on a local NTFS volume.
> > >
> > > I get the error when I do it on Z:, which is mapped to a
> > network drive
> > > (on another Windows box).
> > >
> > > Is there a workaround? Why does this happen?
> >
> > I don't think anyone is sure. My wild guess is that
> there's something
> > about the way that Cygwin wraps Windows calls that makes it
> > fail in this
> > case. It might be interesting to run the Windows and
> Cygwin versions
> > under an strace equivalent and see where things differ.
>
> [this was posted to the cygwin list]
>
> http://nerdfever.com/files/strace.txt
>
> >
> > It's an interesting problem, but I don't have any Windows
> > systems, so I
> > can't look into it.
>
> If it becomes a little less magic, I will chomp on the
> problem, but I cannot make a predictable test case.
Corrina at Cygwin devined the strace (see below) and I am working on a test
cases and hacks.
Pseudo code and observations
./sha1_file.c:write_loose_object(sha1)
{
filename=sha1_file_name(sha1)
(fd,tmp_file)=create_tmpfile(filename)
write_buffer(fd)
close_sha1_file(fd)
return move_temp_to_file(tmp_file, filename)
}
move_temp_to_file(tmpfile, filename)
{
// I am thinking about forcing renames to see if the problem exists then as
well
// if that "works" then a per repo config option allowing for forced renames
if (OBJECT_CREATION_USES_RENAMES) goto try_rename
else if link(tmpfile,filename)
if (failed except exist failures)
{
try_rename:
rename(tmpfile, filename)
if (ok) goto out
}
unlink_or_warn(tmpfile)
if (failed except exist failures) return error
out:
}
write_sha1_file(sha1)
{
return write_loose_object(sha1)
}
> -Original Message-
> From: Corinna Vinschen
> Sent: Friday, December 05, 2014 6:35
> To: cyg...@cygwin.com
> What I found in the strace is this:
>
> - Create file Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ
>
> - open file, write something, close file.
>
> - link (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ,
>
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4)
> succeeds.
>
> - unlink (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ) succeeds
>
> - Trying to open
>
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4
> but the file doesn't exist and NtCreateFile fails with status
> 0xC034, STATUS_OBJECT_NAME_NOT_FOUND --> ENOENT.
>
> - Subsequent unlink (Z:\pic32mx-bmf\.git\objects\30) fails with a
> STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY.
>
> - git seems to be prepared for such a case, the parent process calls
> opendir/readdir on the directory. Enumerating the files in
> Z:\pic32mx-bmf\.git\objects\30 shows the entries ".", ".." and
> "0bdeb2fd209d24afb865584da10b78aa8fefc4".
>
> - Then git calls lstat on the file, which results in NtOpenFile
> returning status STATUS_OBJECT_NAME_NOT_FOUND again.
>
> - From a POSIX POV this means "somebody else" deleted the file,
> so the dir is empty now. Git tries to delete the directory again,
> which again results in STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY
> and, internally, a sharing violation which disallows to move the
> directory out of the way.
>
> This looks suspiciously like a bug in the remote filesystem. Link
> succeeded, so there are two links to the same file in the directory.
> Unlinking link 1 succeeds, so there's still one link to the
> file in the
> directory, but link 2 is inaccessible as if the file has
