RE: Bug in gzip's stdout handling
On Mon, 17 Nov 2003, Thomas Hammer wrote:
-Original Message-
From: Igor Pechtchanski [mailto:[EMAIL PROTECTED]
Sent: 14. november 2003 17:22
To: Thomas Hammer
Cc: [EMAIL PROTECTED]
Subject: RE: Bug in gzip's stdout handling
...
Hi Igor,
This turned out to be a very long mail, so here's the summary:
- I figured it out. I had forgotten to delete the original mount table from
the registry before reinstalling cygwin. All my drives were still mounted in
textmode, and that's why cat and redirection failed.
- I don't know why mount and cygcheck -svr gives different reports regarding
mount mode.
- I have one question it would be nice if you answered. It's at the end of
the email.
And here's the rest of the email:
(2) shows that you used to have text mounts. Your cygcheck.out shows
binary mounts, so that's probably not the problem.
I forgot to mention that I reverted to my original version of cygwin (with
DOS-style newlines) when I discovered that using UNIX-style newlines didn't
fix the problem.
I did the following:
1) Renamed c:\cygwin to c:\cygwin_old
2) Reinstalled cygwin, specifying UNIX-style newlines when asked
3) Test if cat file.bin | gzip -c filecopy.gz produced a valig gz-file.
It didn't
4) Deleted c:\cygwin and renamed c:\cygwin_old to c:\cygwin
I did not delete the registry keys or the local cygwin package directory. I
don't know is this matters or not.
I didn't know about text vs binary mounts until you mentioned it, and did
some reading up. One weird thing I came accross is that if I run mount, the
information seems to conflict with the information from cygcheck -svr.
$ mount
C:\cygwin\bin on /usr/bin type system (textmode)
C:\cygwin\lib on /usr/lib type system (textmode)
C:\cygwin on / type system (textmode)
c: on /cygdrive/c type user (textmode,noumount)
s: on /cygdrive/s type user (textmode,noumount)
The relevant lines from cygcheck -svr:
a: fd N/AN/A
c: hd NTFS 57231Mb 47% CP CS UN PA FC
d: cd N/AN/A
o: net N/AN/A
s: net NTFS 76308Mb 76% CP CS UN PA FC
C:\cygwin / system binmode
C:\cygwin/bin /usr/bin system binmode
C:\cygwin/lib /usr/lib system binmode
. /cygdrive system binmode,cygdrive
It looks to me as if mount claims all mounts are text mounts (which would
explain my problems, I guess). Whereas cygcheck -svr claims all my mounts
are binmode.
I'm beginning to wonder if I did something wrong when reinstalling cygwin
and specifying binary mode. I'm giving it another try now.
1) Renamed c:\cygwin to c:\cygwin_old
2) Renamed the HKCU\Software\Cygnus Solutions registry key
3) Renamed the local cygwin package dir.
4) Reinstalled cygwin, specifying UNIX-style newlines when asked
5) Test if cat file.bin | gzip -c filecopy.gz produced a valig gz-file.
It didn't.
Running mount gives the same result as before (shows textmode on all
mounts).
I tried installing cygwin (downloaded setup.exe from www.cygwin.com) from
scratch on another computer in my office. That computer also runs WindowsXP.
$HOME wasn't defined here, but it was on my primary computer. Don't know if
that mattered.
On that computer, mount shows all mountpoints as being of type binmode, and
cat file.bin | gzip -c out.bin works as it should.
I have no Idea why my primary computer insists on mounting everything in
textmode :-(.
I did try to remove my .bashrc-file on my primary computer. It didn't help -
and I couldn't find anything in there related to mounting.
I wonder where the mount table is stored. Maybe it for some reason survived
a reinstall of cygwin...
Please read on for some more discoveries.
Please try the following: sed '' binaryfile.bin acopy.bin and compare
the files. Also echo | sed '' test.out; od -c test.out and perl -e
'print q{ }x2560' | gzip -c test.gz; od -c test.gz.
[EMAIL PROTECTED] /cygdrive/c/temp/temp
$ echo | sed '' test.out
[EMAIL PROTECTED] /cygdrive/c/temp/temp
$ od -c test.out
000 \r \n
002
[EMAIL PROTECTED] /cygdrive/c/temp/temp
$ perl -e 'print q{ }x2560' | gzip -c test.gz
[EMAIL PROTECTED] /cygdrive/c/temp/temp
$ od -c test.gz
000 037 213 \b \0 = . 271 ? \0 003 355 301 201 \0 \0 \0
020 \0 303 225 371 S 036 344 U 001 \0 360 d 6 271 357
040 252 \0 \r \n \0 \0
046
Back to the mounting thread of thought.
I tried to mount a directory as binary:
$ mount -b c:\\temp\\temp\\t /mytest
mount: warning - /mytest does not exist.
$ cd /mytest
[EMAIL PROTECTED] /mytest
$ echo | sed '' test.out; od -c test.out
000 \n
001
Hey, I'm getting somewhere :-).
[EMAIL PROTECTED] /mytest
$ ls -l
total 109
-rwx--1 thammer mkgroup110755 Nov 17 21:51 bin.jpg
[EMAIL PROTECTED] /mytest
$ cat bin.jpg | gzip -c another.jpg.gz
[EMAIL PROTECTED] /mytest
$ gunzip another.jpg.gz
[EMAIL
RE: Bug in gzip's stdout handling
-Original Message-
From: Igor Pechtchanski [mailto:[EMAIL PROTECTED]
Sent: 14. november 2003 17:22
To: Thomas Hammer
Cc: [EMAIL PROTECTED]
Subject: RE: Bug in gzip's stdout handling
...
Hi Igor,
This turned out to be a very long mail, so here's the summary:
- I figured it out. I had forgotten to delete the original mount table from
the registry before reinstalling cygwin. All my drives were still mounted in
textmode, and that's why cat and redirection failed.
- I don't know why mount and cygcheck -svr gives different reports regarding
mount mode.
- I have one question it would be nice if you answered. It's at the end of
the email.
And here's the rest of the email:
(2) shows that you used to have text mounts. Your cygcheck.out shows
binary mounts, so that's probably not the problem.
I forgot to mention that I reverted to my original version of cygwin (with
DOS-style newlines) when I discovered that using UNIX-style newlines didn't
fix the problem.
I did the following:
1) Renamed c:\cygwin to c:\cygwin_old
2) Reinstalled cygwin, specifying UNIX-style newlines when asked
3) Test if cat file.bin | gzip -c filecopy.gz produced a valig gz-file.
It didn't
4) Deleted c:\cygwin and renamed c:\cygwin_old to c:\cygwin
I did not delete the registry keys or the local cygwin package directory. I
don't know is this matters or not.
I didn't know about text vs binary mounts until you mentioned it, and did
some reading up. One weird thing I came accross is that if I run mount, the
information seems to conflict with the information from cygcheck -svr.
$ mount
C:\cygwin\bin on /usr/bin type system (textmode)
C:\cygwin\lib on /usr/lib type system (textmode)
C:\cygwin on / type system (textmode)
c: on /cygdrive/c type user (textmode,noumount)
s: on /cygdrive/s type user (textmode,noumount)
The relevant lines from cygcheck -svr:
a: fd N/AN/A
c: hd NTFS 57231Mb 47% CP CS UN PA FC
d: cd N/AN/A
o: net N/AN/A
s: net NTFS 76308Mb 76% CP CS UN PA FC
C:\cygwin / system binmode
C:\cygwin/bin /usr/bin system binmode
C:\cygwin/lib /usr/lib system binmode
. /cygdrive system binmode,cygdrive
It looks to me as if mount claims all mounts are text mounts (which would
explain my problems, I guess). Whereas cygcheck -svr claims all my mounts
are binmode.
I'm beginning to wonder if I did something wrong when reinstalling cygwin
and specifying binary mode. I'm giving it another try now.
1) Renamed c:\cygwin to c:\cygwin_old
2) Renamed the HKCU\Software\Cygnus Solutions registry key
3) Renamed the local cygwin package dir.
4) Reinstalled cygwin, specifying UNIX-style newlines when asked
5) Test if cat file.bin | gzip -c filecopy.gz produced a valig gz-file.
It didn't.
Running mount gives the same result as before (shows textmode on all
mounts).
I tried installing cygwin (downloaded setup.exe from www.cygwin.com) from
scratch on another computer in my office. That computer also runs WindowsXP.
$HOME wasn't defined here, but it was on my primary computer. Don't know if
that mattered.
On that computer, mount shows all mountpoints as being of type binmode, and
cat file.bin | gzip -c out.bin works as it should.
I have no Idea why my primary computer insists on mounting everything in
textmode :-(.
I did try to remove my .bashrc-file on my primary computer. It didn't help -
and I couldn't find anything in there related to mounting.
I wonder where the mount table is stored. Maybe it for some reason survived
a reinstall of cygwin...
Please read on for some more discoveries.
Please try the following: sed '' binaryfile.bin acopy.bin and compare
the files. Also echo | sed '' test.out; od -c test.out and perl -e
'print q{ }x2560' | gzip -c test.gz; od -c test.gz.
[EMAIL PROTECTED] /cygdrive/c/temp/temp
$ echo | sed '' test.out
[EMAIL PROTECTED] /cygdrive/c/temp/temp
$ od -c test.out
000 \r \n
002
[EMAIL PROTECTED] /cygdrive/c/temp/temp
$ perl -e 'print q{ }x2560' | gzip -c test.gz
[EMAIL PROTECTED] /cygdrive/c/temp/temp
$ od -c test.gz
000 037 213 \b \0 = . 271 ? \0 003 355 301 201 \0 \0 \0
020 \0 303 225 371 S 036 344 U 001 \0 360 d 6 271 357
040 252 \0 \r \n \0 \0
046
Back to the mounting thread of thought.
I tried to mount a directory as binary:
$ mount -b c:\\temp\\temp\\t /mytest
mount: warning - /mytest does not exist.
$ cd /mytest
[EMAIL PROTECTED] /mytest
$ echo | sed '' test.out; od -c test.out
000 \n
001
Hey, I'm getting somewhere :-).
[EMAIL PROTECTED] /mytest
$ ls -l
total 109
-rwx--1 thammer mkgroup110755 Nov 17 21:51 bin.jpg
[EMAIL PROTECTED] /mytest
$ cat bin.jpg | gzip -c another.jpg.gz
[EMAIL PROTECTED] /mytest
$ gunzip another.jpg.gz
[EMAIL PROTECTED] /mytest
$ ls -l
total 218
-rw-r--r--1 thammer mkgroup110755 Nov 17 22:08 another.jpg
-rwx--1 thammer mkgroup110755 Nov 17 21:51
Re: Bug in gzip's stdout handling
On Fri, 14 Nov 2003, Thomas Hammer wrote: Hi. I run the latest version of cygwin, with gzip, on WindowsXP SP1 and have a problem with gzip. Please see http://cygwin.com/problems.html for guidelines on how to report Cygwin problems. The command $ cat binaryfile.bin | gzip -c bin.gz produces an invalid .gz file: $ gunzip bin.gz gunzip: bin.gz: invalid compressed data--crc error gunzip: bin.gz: invalid compressed data--length error If I do gzip binaryfile.bin everything works as expected - i.e. the resulting .gz file is valid. If I do this instead: cat binaryfile.bin acopy.bin The two .bin files are identical. My gzip version: $ gzip --version gzip 1.3.5 (2002-09-30) Copyright 2002 Free Software Foundation Copyright 1992-1993 Jean-loup Gailly This program comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of this program under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING. Compilation options: DIRENT UTIME STDC_HEADERS HAVE_UNISTD_H HAVE_MEMORY_H HAVE_STRING_H HAVE_LSTAT ASMV Written by Jean-loup Gailly. My conclusion is that gzip's handling of stdout is broken in some way. I bet you have a text mount. Please *attach* the output of cygcheck -svr, as per http://cygwin.com/problems.html - this will confirm the guess. The reason I report this problem, instead of just avoiding it by not using the -c option for gzip, is that I discovered the problem while running make dist on a Makefile.in created by automake. And it's automake that creates the gzip command line. I'll send a patch to the automake people and ask them to fix this - but since it's really gzip that is broken, I think it should be fixed here too. I think you could fix this by setting the mode for the filehandle for stdout to O_BINARY, i.e. something like this: #if defined (_WIN32) setmode(FileHandle, O_BINARY); /* Make sure it is in binary mode. */ #endif FYI, Cygwin doesn't define _WIN32. Besides, the above is not a proper patch. Igor It would be nice if you could look into it. Best Regards, Thomas Hammer [EMAIL PROTECTED] -- http://cs.nyu.edu/~pechtcha/ |\ _,,,---,,_[EMAIL PROTECTED] ZZZzz /,`.-'`'-. ;-;;,_[EMAIL PROTECTED] |,4- ) )-,_. ,\ ( `'-' Igor Pechtchanski, Ph.D. '---''(_/--' `-'\_) fL a.k.a JaguaR-R-R-r-r-r-.-.-. Meow! I have since come to realize that being between your mentor and his route to the bathroom is a major career booster. -- Patrick Naughton -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: Bug in gzip's stdout handling
-Original Message- From: Igor Pechtchanski [mailto:[EMAIL PROTECTED] Sent: 14. november 2003 16:28 To: Thomas Hammer Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: Bug in gzip's stdout handling Hi Igor. Thanks for responding so promptly :). ... I bet you have a text mount. Please *attach* the output of cygcheck -svr, as per http://cygwin.com/problems.html - this will confirm the guess. cygcheck.out attached. I didn't find anything about any drives being mounted as text, though. Two small bits of info that might or might not have some value: 1) I believe this has worked earlier (i.e. 6 months ago) for a colleague of mine, running a recent version of cygwin. Which means that something could have been broken in cygwin after that, or (more likely) that something for some reason is configured differently on my system. 2) When I installed cygwin, I was asked if I wanted the tools to support Unix-style newlines or DOS-style newlines. I answered DOS-style. I did reinstall cygwin, specifying Unix-style newlines this time, and had the same problem though. .Thomas cygcheck.out Description: Binary data -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: Bug in gzip's stdout handling
I've removed support at gzip dot org from this discussion. So far this
is Cygwin-related, and you can update them later on the results if it
turns out to be a generic gzip issue.
On Fri, 14 Nov 2003, Thomas Hammer wrote:
-Original Message-
From: Igor Pechtchanski [mailto:[EMAIL PROTECTED]
Sent: 14. november 2003 16:28
To: Thomas Hammer
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: Bug in gzip's stdout handling
Hi Igor.
Thanks for responding so promptly :).
...
I bet you have a text mount. Please *attach* the output of cygcheck
-svr, as per http://cygwin.com/problems.html - this will confirm the
guess.
cygcheck.out attached.
I didn't find anything about any drives being mounted as text, though.
Two small bits of info that might or might not have some value:
1) I believe this has worked earlier (i.e. 6 months ago) for a colleague of
mine, running a recent version of cygwin. Which means that something could
have been broken in cygwin after that, or (more likely) that something for
some reason is configured differently on my system.
2) When I installed cygwin, I was asked if I wanted the tools to support
Unix-style newlines or DOS-style newlines. I answered DOS-style. I did
reinstall cygwin, specifying Unix-style newlines this time, and had the
same problem though.
.Thomas
(2) shows that you used to have text mounts. Your cygcheck.out shows
binary mounts, so that's probably not the problem.
Please try the following: sed '' binaryfile.bin acopy.bin and compare
the files. Also echo | sed '' test.out; od -c test.out and perl -e
'print q{ }x2560' | gzip -c test.gz; od -c test.gz.
Igor
--
http://cs.nyu.edu/~pechtcha/
|\ _,,,---,,_[EMAIL PROTECTED]
ZZZzz /,`.-'`'-. ;-;;,_[EMAIL PROTECTED]
|,4- ) )-,_. ,\ ( `'-' Igor Pechtchanski, Ph.D.
'---''(_/--' `-'\_) fL a.k.a JaguaR-R-R-r-r-r-.-.-. Meow!
I have since come to realize that being between your mentor and his route
to the bathroom is a major career booster. -- Patrick Naughton
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
