Re: 1.5.24: incorrect default behavior of dd in popen context on text-mounted filesystem
No problem. The responsiveness of the Cygwin maintainers is phenomenal. Keep up the good work. -Hugh Christopher Faylor wrote: > >So AFAICT that particular issue was quietly fixed. > > No, actually, it was a typo in the announcement. Somehow I managed to > keep a URL from my previous announcement in that message. > > I did mention that I was going to update gzip here, however: > > http://cygwin.com/ml/cygwin/2007-07/msg00611.html > > Sorry for the confusion. > > cgf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: 1.5.24: incorrect default behavior of dd in popen context on text-mounted filesystem
Thanks for the careful discussion. It almost all makes sense to me.
> > 1) In the Cygwin User's Guide, page 33:
> >
> > c. Pipes and non-file devices are opened in binary mode, except if
> > the CYGWIN environment variable contains nobinmode.
> >
> > Warning!
> > In b20.1 of 12/98, a file will be opened in binary mode if any
> > of the following conditions hold:
>
> This documentation is rather old, so it must be read with a grain of salt.
>
> > 1. binary mode is specified in the open call
> > 2. the filename is a MS-DOS filename
> > 3. the file resides on a binary mounted partition
> > 4. CYGWIN contains binmode
>
> In particular, CYGWIN defaults to binmode, but binmode/nobinmode only affects
> non-disk files (ie. pipes, special devices) - it has no bearing on disk
> files,
> since that is what mount is for.
>
> > 5. the file is not a disk file
>
> In other words, 4 and 5 should be merged into a single condition.
As you've surmised, the part I'm stumbling over is Warning number 4.
The overall intention of the set of warnings seems clear, that any one
of the conditions will give rise to binary mode files. My reading of
4 is thus: "a file will be opened in binary mode if .. CYGWIN contains
binmode". My interpretation is thus that when a shell is redirecting
to a file it will open the in binary mode (because I have
CYGWIN=binmode). Hence my expectation that dd's stdout would be in
binmode in the context of popen("gzip|dd>file","w"), and, originally,
that gzip's stdout would be binmode in popen("gzip>file","w").
It seems to me that if the shell(s) were to honor Warning 4 when
redirecting, then maintainers of individual programs would not have to
fuss over whether to change the modes of the stdin/stdout fds that
they've been given. This puts (or leaves) control in the hands of the
caller of the executable. And this supports a desireable uniformity
of behavior. OTOH, this would appear to require that every use of
fopen() or open() should worry about the value of CYGWIN.
Your suggestion of merging warnings 4 and 5 appears to be an attempt
to change the documentation to match the current implementation. What
would a merged 4 and 5 be saying that isn't already covered by what
(c) itself is saying? Could 4 and 5 be removed? This then puts the
onus on each implementor to decide what to do with their stdin/stdout
fds. This would lead to less uniformity of behavior. It might lead
to finer control in the form of a proliferation of [io]flag= options,
but there would very likely be less consistency. A benefit might be
that CYGWIN=binmode goes away as having any meaning, and only
CYGWIN=nobinmode matters.
In either case, the documentation should be reworded, and that is
something that I'd be willing to help with (once I understand more
about the intent and/or practices associated with
CYGWIN=[no]binmode).
-Hugh
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
Re: 1.5.24: incorrect default behavior of dd in popen context on text-mounted filesystem
> >Yes, in my rush to try out the modified dd I did not notice that > >cgf had quietly fixed the binary issue in gzip. So, thank you to > >cgf for so quickly fixing the original problem. > > It wasn't "quietly": > > http://cygwin.com/ml/cygwin-announce/2007-07/msg00028.html > http://cygwin.com/ml/cygwin/2007-07/msg00615.html > > You just weren't paying attention. I had read both of those. Neither of those, nor the referenced http://cygwin.com/ml/cygwin/2007-04/msg00680.html or http://lists.gnu.org/archive/html/bug-gnu-utils/2007-04/msg00041.html mentions a fix for text/binary mode of files. So AFAICT that particular issue was quietly fixed. -Hugh -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
1.5.24: incorrect default behavior of dd in popen context on text-mounted filesystem
Firstly, a thank you to Eric Blake for his quick response to the dd
problem we uncovered yesterday. As of coreutils 6.9-4, dd honors the
oflag=binary option in the popen() context in which I've been having
trouble. This test case now works as expected:
$ cat popenbug.c
#include
int main() {
FILE * const out = popen("gzip | dd oflag=binary > popenbug.out.gz", "w");
int i;
for( i = 0; i < 25; ++i )
fprintf(out, "line %d: * ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING,
BUT NOT LIMITED TO \n", i);
fclose(out);
return 0;
}
$
$ gcc popenbug.c -o popenbug.exe && rm -f popenbug.out.gz && ./popenbug.exe &&
sleep 1 && gzip -d -t popenbug.out.gz
0+1 records in
0+1 records out
172 bytes (172 B) copied, 0 s, Infinity B/s
$
However, in the discussion and the updated coreutils announcement
http://cygwin.com/ml/cygwin/2007-07/msg00610.html
http://cygwin.com/ml/cygwin/2007-07/msg00617.html
the behavior of dd is specified as defaulting to binary in the absence
of an [io]flag=text option. This spec is in agreement with the Cygwin
documenation. Bug, as of coreutils 6.9-4, this default behavior is
not honored in the popen() context:
$ cat popenbug.c
#include
int main() {
FILE * const out = popen("gzip | dd > popenbug.out.gz", "w");
int i;
for( i = 0; i < 25; ++i )
fprintf(out, "line %d: * ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING,
BUT NOT LIMITED TO \n", i);
fclose(out);
return 0;
}
$
$ set|fgrep -i cygwin
BASH_VERSINFO=([0]="3" [1]="2" [2]="17" [3]="15" [4]="release"
[5]="i686-pc-cygwin")
CYGWIN=binmode
MACHTYPE=i686-pc-cygwin
OSTYPE=cygwin
$
$ gcc popenbug.c -o popenbug.exe && rm -f popenbug.out.gz && ./popenbug.exe &&
sleep 1 && gzip -d -t popenbug.out.gz
0+1 records in
0+1 records out
172 bytes (172 B) copied, 0.015 s, 11.5 kB/s
gzip: popenbug.out.gz: invalid compressed data--crc error
gzip: popenbug.out.gz: invalid compressed data--length error
$
Notice that dd processes the same number of bytes in both cases, so
the problem is somewhere in the stdout handling when the oflag=binary
argument isn't provided
Regards,
-Hugh
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
Re: 1.5.24: data corruption problem with popen and gzip on a text mounted filesystem
Eric Blake byu.net> writes:
> Hugh Secker-Walker merl.com> writes:
>
> > I'm having trouble getting correct behavior on a third-party OpenSource
> > project that I'm building using Cygwin. The problem involves the
> > writing of corrupt data to a file. The output file is created and
> > written via popen("gzip > outputfile", "wb"). The data is fine if the
> > filesystem is mounted in binary mode. The data is corrupted if the
> > filesystem is mounted in text mode.
>
> As a workaround, you can force gzip to see stdout in binary mode by
using an
> intermediary pipe, as in popen("gzip | cat > outputfile", "wb").
Thank you. That simple workaround appears to work.
Interestingly, I had tried popen("gzip | dd of=outputfile", "wb") as a
lower-level way to avoid what I guessed was a problem with gzip using
the stdout. The dd attempt didn't work. This failure suggests to me
that dd and gzip experience a similar problem in this popen() context,
a problem that cat somehow avoids.
-Hugh
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
1.5.24: data corruption problem with popen and gzip on a text mounted filesystem
Hi Folks,
I'm having trouble getting correct behavior on a third-party OpenSource
project that I'm building using Cygwin. The problem involves the
writing of corrupt data to a file. The output file is created and
written via popen("gzip > outputfile", "wb"). The data is fine if the
filesystem is mounted in binary mode. The data is corrupted if the
filesystem is mounted in text mode.
My understanding of the documentation is that pipes and shells' stdin
and stdout will be in binary mode if the CYGWIN env variable contains
the 'binmode' setting, regardless of the binary/text mount flag of the
filesystem the file is mounted on. From page 33 of the users guide:
...
c. Pipes and non-file devices are opened in binary mode, except if
the CYGWIN environment variable
contains nobinmode.
Warning!
In b20.1 of 12/98, a file will be opened in binary mode if any
of the
following conditions hold:
1. binary mode is specified in the open call
2. the filename is a MS-DOS filename
3. the file resides on a binary mounted partition
4. CYGWIN contains binmode
5. the file is not a disk file
d. When redirecting, the Cygwin shells uses rules (a-e). [sic,
should be (a-d) -HSW]
For these shells the relevant value of
CYGWIN is that at the time the shell was launched and not that at
the time the program is executed.
Non-Cygwin shells always pipe and redirect with binary mode. With
non-Cygwin shells the
commands cat filename | program and program < filename are not
equivalent when filename
is on a text-mounted partition.
I've generated a small example of the problem. Attached is an annotated
log showing the code and the behaviors. One thing that is not clear
from the logs is that I generally start all executables from within a
bash shell within emacs. However, the problem appears in rxvt shell
windows and in XP cmd.exe windows in which I have run cygwin.bat to get
a bash shell.
I've tried several workarounds of this variety: " sh -c 'CYGWIN=binmode
blah.exe' " both to start the program and within the popen() command,
but none of them has succeeded. The use of "wb" or "w" in the second
argument to popen() has no visible effect on the problem. And the
setting or not of CYGWIN=binmode in the XP environment has no visible
effect.
So far I have no workaround to the problem. I cannot require users to
switch their filesystem mounts to binary.
Any insight or workarounds would be appreciate.
Thanks,
-Hugh
# 1.5.24: data corruption problem with popen and gzip on a textmode mounted
filesystem
# verify CYGWIN=binmode, inherited from XP environment
$ set|fgrep -i cygwin
BASH_VERSINFO=([0]="3" [1]="2" [2]="17" [3]="15" [4]="release"
[5]="i686-pc-cygwin")
CYGWIN=binmode
MACHTYPE=i686-pc-cygwin
OSTYPE=cygwin
# the test program, pipes some data through gzip
$ cat popenbug.c
#include
int main() {
FILE * const out = popen("gzip > popenbug.out.gz", "wb");
int i;
for( i = 0; i < 25; ++i )
fprintf(out, "line %d: * ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING,
BUT NOT LIMITED TO \n", i);
fclose(out);
return 0;
}
# we're on the cygdrive-mapped C drive
$ pwd
/cygdrive/c/home/hugh
# binary mode usage
$ mount -s -b --change-cygdrive-prefix /cygdrive
# there is no problem when popen pipes into gzip, the 'gzip -d -t' check
succeeds
$ gcc popenbug.c -o popenbug.exe && ./popenbug.exe && sleep 1 && od -t x1
popenbug.out.gz && gzip -d -t popenbug.out.gz
000 1f 8b 08 00 8e a8 a0 46 00 03 ad d5 3d 0e c2 30
020 0c 40 e1 9d 53 78 46 1d 70 f8 67 0b 34 42 96 d2
040 a4 4a 52 01 07 60 40 aa b8 ff 48 b9 c3 db 3c 58
060 6f f2 27 cf 9f ef 5b 36 17 59 8b 4f 2f 09 cf b1
100 84 5a 43 2f b9 88 0d 63 b4 65 7c f8 52 7c 6a 16
120 6a 27 96 6e 71 ea 2d dd 3b b9 4e 4d 52 6e 12 6d
140 b0 b6 ac b5 2c b2 9a ff 39 65 73 8e cd 6d d9 dc
160 8e cd ed d9 dc 81 cd 1d d9 dc 89 cd 9d e1 33 a6
200 59 c0 2e 14 86 a1 b0 0c 85 69 28 6c 43 61 1c 0a
220 eb 50 98 87 c2 3e 1c ec c3 d1 7f 03 f6 e1 60 1f
240 8e f3 f1 03 cc 55 f2 25 c6 07 00 00
254
# switch to text mode usage
$ mount -s -t --change-cygdrive-prefix /cygdrive
# piping into gzip works from the shell, the 'gzip -d -t' check succeeds
$ gzip -d -c popenbug.out.gz | gzip > popenbug.out2.gz
$ gzip -d -t popenbug.out2.gz
# but, there is a problem when popen pipes into gzip, the 'gzip -d -t' check
fails
$ gcc popenbug.c -o popenbug.exe && ./popenbug.exe && sleep 1 && od -t x1
popenbug.out.gz && gzip -d -t popenbug.out.gz
000 1f 8b 08 00 9a a8 a0 46 00 03 ad d5 3d 0e c2 30
020 0c 40 e1 9d 53 78 46 1d 70 f8 67 0b 34 42 96 d2
040 a4 4a 52 01 07 60 40 aa b8 ff 48 b9 c3 db 3c 58
060 6f f2 27 cf 9f ef 5b 36 17 59 8b 4f 2f 09 cf b1
100 84 5a 43 2f b9 88 0d 63 b4 65 7c f8 52 7c 6a 16
120 6a 27 96 6e 71 ea 2d dd 3b b9 4e 4d 52 6e 12 6d
140 b0 b6 ac b5 2c b2 9a ff 39 65 73 8e
