Re: Error reported dd'ing close of end of block device with skip
On Dec 14 16:04, Eric Blake wrote: > > > > Btw., I just applied a patch which implements O_DIRECT, which will be > > available in the next snapshot. The same restriction as on Linux > > applies, which is, the buffers used in read/write operations must be > > sector-aligned. Note that using O_DIRECT requires to rebuild the > > application, so you'll be unable to experiment with this using the > > current versions of dd(1). > > I get the hint. coreutils-5.93-2 will be out as soon as I can rebuild it > against a new snapshot. Heh, cool. Just FYI, I tested my changes using a rebuilt version of dd from 5.90 and it behaved fine using `iflags=direct' and `oflags=direct'. If you notice any problems, though, kick me immediately. Just keep the alignment restriction in mind. MSDN says that it's not necessarily enforced, depending on the device, but dd should obviously take this into account. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat, Inc. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: Error reported dd'ing close of end of block device with skip
> > Btw., I just applied a patch which implements O_DIRECT, which will be > available in the next snapshot. The same restriction as on Linux > applies, which is, the buffers used in read/write operations must be > sector-aligned. Note that using O_DIRECT requires to rebuild the > application, so you'll be unable to experiment with this using the > current versions of dd(1). I get the hint. coreutils-5.93-2 will be out as soon as I can rebuild it against a new snapshot. -- Eric Blake volunteer cygwin corerutils maintainer -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: Error reported dd'ing close of end of block device with skip
On Dec 13 19:05, Loh, Joe wrote: > We downloaded cygwin-inst-20051213.tar.bz2 and implemented the solution > recommended below. So far, our cursory testing indicates that we are > able to do unbuffered read. However, we are not seeing any "bug" that > was referenced here. What is the "bug" that is being referenced here? The RDIOCDOP/RDSETBLK was incorrectly implemented when a request was made to set the buffer size to 0 or 1 (aka "non-buffered"). But since that's fixed in the above snapshot, you can't trip over it. Btw., I just applied a patch which implements O_DIRECT, which will be available in the next snapshot. The same restriction as on Linux applies, which is, the buffers used in read/write operations must be sector-aligned. Note that using O_DIRECT requires to rebuild the application, so you'll be unable to experiment with this using the current versions of dd(1). Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat, Inc. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: Error reported dd'ing close of end of block device with skip
We downloaded cygwin-inst-20051213.tar.bz2 and implemented the solution
recommended below. So far, our cursory testing indicates that we are
able to do unbuffered read. However, we are not seeing any "bug" that
was referenced here. What is the "bug" that is being referenced here?
Here's the read from the strace showing that the read did what we ask
for, which is 512 bytes.
57 222458 [unknown (0x940)] lightning 3480 readv: readv (3,
0x10EED60, 1) blocking, sigcatchers 7
51 222509 [unknown (0x940)] lightning 3480 readv: no need to call
ready_for_read
132 222641 [unknown (0x940)] lightning 3480
fhandler_dev_floppy::read_file: 1 (err 0) = ReadFile (1580, 268575744,
to_read 512, read 512, 0)
59 222700 [unknown (0x940)] lightning 3480 fhandler_base::read:
returning 512, binary mode
52 222752 [unknown (0x940)] lightning 3480 readv: 512 = readv (3,
0x10EED60, 1), errno 0
Thanks,
Joe
-Original Message-
From: Corinna Vinschen [mailto:[EMAIL PROTECTED]
Sent: Thursday, December 08, 2005 4:57 AM
To: cygwin@cygwin.com
Subject: Re: Error reported dd'ing close of end of block device with
skip
On Dec 7 19:35, Corinna Vinschen wrote:
> On Dec 7 11:55, Loh, Joe wrote:
> > QUESTION:
> >
> > Is there a way in Cygwin to do a read of a block device using "C"
> > that does not do a read-ahead? We needed to develop an application
> > that will issue the exact transfer size to the target device as
requested.
> > Looking at the strace, it appears that read() less than 61440-bytes
> > gets translated to reading 61440 bytes into buffer and then a subset
> > of that read is returned to the caller.
>
> There's a non-portable (only Cygwin) way to set the buffer size after
> opening a device:
>
> #include
>
> struct rdop rd;
>
> fd = open ("dev/sda", ...);
>
> rd.rd_op = RDSETBLK;
> rd.rd_parm = 0; /* Unbuffered reading */
> rd.rd_parm = 1; /* Unbuffered reading */
> rd.rd_parm = n; /* Buffered reading with buffer size n */
>
> ioctl (fd, RDIOCDOP, &rd);
>
> Note that the ioctl fails if the buffer already contains data, so
> ioctl should be called before the first read. Also note that in
> unbuffered mode the usual Windows blocking rule applies, the length
> given to read must be a multiple of 512.
>
> Sigh, it seems that I introduced a bug into this ioctl also not long
ago.
> I fixed this in CVS. Recent code will probably not work very well
> when setting it to unbuffered mode. Please wait for the next
> developers snapshot.
I just added another requirement, that rd.rd_parm must be either 0, 1,
or a multiple of 512. Other sizes for buffers don't make much sense.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Project Co-Leader cygwin AT cygwin DOT com
Red Hat, Inc.
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
Re: Error reported dd'ing close of end of block device with skip
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to Corinna Vinschen on 12/8/2005 3:30 AM: > > Hey, cool idea! There's no definition of O_DIRECT in Cygwin so far, but > this sounds like a worthwhile extension, at least for disk and tape > devices, but maybe also for files. This won't be in the next Cygwin > version 1.5.19, probably, but I'll certainly implement that for 1.5.20. Another cool flag is newer Linux's O_NOATIME, which inhibits updating the access time. In particular, coreutils CVS has already added a -noatime option to dd (a nop unless the system supports it), and is considering moving to using O_NOATIME on directories for ls and other traversal routines, since POSIX allows ls to not change directory atime during traversal, and since listing directories is effectively a read-only operation that doesn't really need to write to disk just to update the directory atimes. If Windows FILE_FLAG_BACKUP_SEMANTICS can be used to avoid touching the atime, then this flag seems useful to implement for the 1.5.20 timeframe. While we are talking about wishes, Solaris and newer glibc provide the *at family of functions (such as openat); the cool thing about these functions is that they provide the ability to make directory traversal reentrant since with fd-relative semantics you no longer need to call chdir(). See for example http://lists.gnu.org/archive/html/bug-gnulib/2005-11/msg00042.html. - -- Life is short - so eat dessert first! Eric Blake [EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDmEEm84KuGfSFAYARAsd0AJ43Lun2yfiMdFHKXWt4Rq9bHwZV/ACfSn8O XXKWh5FBRn/V2BbLl+82NsU= =CzeF -END PGP SIGNATURE- -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: Error reported dd'ing close of end of block device with skip
On Dec 7 19:35, Corinna Vinschen wrote:
> On Dec 7 11:55, Loh, Joe wrote:
> > QUESTION:
> >
> > Is there a way in Cygwin to do a read of a block device using "C" that
> > does not do a read-ahead? We needed to develop an application that will
> > issue the exact transfer size to the target device as requested.
> > Looking at the strace, it appears that read() less than 61440-bytes gets
> > translated to reading 61440 bytes into buffer and then a subset of that
> > read is returned to the caller.
>
> There's a non-portable (only Cygwin) way to set the buffer size after
> opening a device:
>
> #include
>
> struct rdop rd;
>
> fd = open ("dev/sda", ...);
>
> rd.rd_op = RDSETBLK;
> rd.rd_parm = 0; /* Unbuffered reading */
> rd.rd_parm = 1; /* Unbuffered reading */
> rd.rd_parm = n; /* Buffered reading with buffer size n */
>
> ioctl (fd, RDIOCDOP, &rd);
>
> Note that the ioctl fails if the buffer already contains data, so ioctl
> should be called before the first read. Also note that in unbuffered
> mode the usual Windows blocking rule applies, the length given to read
> must be a multiple of 512.
>
> Sigh, it seems that I introduced a bug into this ioctl also not long ago.
> I fixed this in CVS. Recent code will probably not work very well when
> setting it to unbuffered mode. Please wait for the next developers
> snapshot.
I just added another requirement, that rd.rd_parm must be either 0, 1,
or a multiple of 512. Other sizes for buffers don't make much sense.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Project Co-Leader cygwin AT cygwin DOT com
Red Hat, Inc.
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
Re: Error reported dd'ing close of end of block device with skip
On Dec 7 16:47, Loh, Joe wrote: > Thank you ... This will work. Any plans on supporting O_DIRECT flag in > open()? We believe this effectively gives unbuffered I/O. Please > correct if we are making the wrong assumption. Hey, cool idea! There's no definition of O_DIRECT in Cygwin so far, but this sounds like a worthwhile extension, at least for disk and tape devices, but maybe also for files. This won't be in the next Cygwin version 1.5.19, probably, but I'll certainly implement that for 1.5.20. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat, Inc. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: Error reported dd'ing close of end of block device with skip
>> On Dec 7 11:55, Loh, Joe wrote:
>> QUESTION:
>>
>> Is there a way in Cygwin to do a read of a block device using "C"
that
>> does not do a read-ahead? We needed to develop an application that
>> will issue the exact transfer size to the target device as requested.
>> Looking at the strace, it appears that read() less than 61440-bytes
>> gets translated to reading 61440 bytes into buffer and then a subset
>> of that read is returned to the caller.
>
> There's a non-portable (only Cygwin) way to set the buffer size after
opening a device:
>
> #include
>
> struct rdop rd;
>
> fd = open ("dev/sda", ...);
>
> rd.rd_op = RDSETBLK;
> rd.rd_parm = 0; /* Unbuffered reading */
> rd.rd_parm = 1; /* Unbuffered reading */
> rd.rd_parm = n; /* Buffered reading with buffer size n */
>
> ioctl (fd, RDIOCDOP, &rd);
>
> Note that the ioctl fails if the buffer already contains data, so
ioctl should be called before the first > read. Also note that in
unbuffered mode the usual Windows blocking rule applies, the length
given to read > must be a multiple of 512.
>
> Sigh, it seems that I introduced a bug into this ioctl also not long
ago.
> I fixed this in CVS. Recent code will probably not work very well
when setting it to unbuffered mode. Please wait for the next developers
snapshot.
>
>
> Corinna
Thank you ... This will work. Any plans on supporting O_DIRECT flag in
open()? We believe this effectively gives unbuffered I/O. Please
correct if we are making the wrong assumption.
Again. Thank you for the recommendation and we will wait for the next
Cygwin Kernel before porting our codes over.
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
Re: Error reported dd'ing close of end of block device with skip
On Dec 7 11:55, Loh, Joe wrote:
> QUESTION:
>
> Is there a way in Cygwin to do a read of a block device using "C" that
> does not do a read-ahead? We needed to develop an application that will
> issue the exact transfer size to the target device as requested.
> Looking at the strace, it appears that read() less than 61440-bytes gets
> translated to reading 61440 bytes into buffer and then a subset of that
> read is returned to the caller.
There's a non-portable (only Cygwin) way to set the buffer size after
opening a device:
#include
struct rdop rd;
fd = open ("dev/sda", ...);
rd.rd_op = RDSETBLK;
rd.rd_parm = 0; /* Unbuffered reading */
rd.rd_parm = 1; /* Unbuffered reading */
rd.rd_parm = n; /* Buffered reading with buffer size n */
ioctl (fd, RDIOCDOP, &rd);
Note that the ioctl fails if the buffer already contains data, so ioctl
should be called before the first read. Also note that in unbuffered
mode the usual Windows blocking rule applies, the length given to read
must be a multiple of 512.
Sigh, it seems that I introduced a bug into this ioctl also not long ago.
I fixed this in CVS. Recent code will probably not work very well when
setting it to unbuffered mode. Please wait for the next developers
snapshot.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Project Co-Leader cygwin AT cygwin DOT com
Red Hat, Inc.
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
RE: Error reported dd'ing close of end of block device with skip
> Searching the archive > > http://cygwin.com/cgi-bin/htsearch?config=htdig&words=dd+eom > > reveals a thread "Bug in dd ?? at EOM" which starts here: > > http://sourceware.org/ml/cygwin/2005-09/msg00878.html > > Btw, have you tried the latest snapshot from http://cygwin.com/snapshots/ ? > > > Corinna We tried with the cygwin-inst-20051207.tar.bz2 snapshot. Using the same "dd" commands we no longer observed the problem. QUESTION: Is there a way in Cygwin to do a read of a block device using "C" that does not do a read-ahead? We needed to develop an application that will issue the exact transfer size to the target device as requested. Looking at the strace, it appears that read() less than 61440-bytes gets translated to reading 61440 bytes into buffer and then a subset of that read is returned to the caller. Here're the outputs: dd if=/dev/sda count=1 bs=1024 skip=`expr 78125000 - 60` > /dev/null 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0 seconds, Infinity B/s dd if=/dev/sda count=1 bs=1024 skip=`expr 78125000 - 59` > /dev/null 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0 seconds, Infinity B/s Looking at the strace for the command that used to failed, we noticed that the read ahead transfer size is adjusted to not overshoot the end of device: 3260 25940 [main] dd 3660 lseek64: 7939584 = lseek (0, 7939584, 1) 61 26001 [main] dd 3660 readv: readv (0, 0x22ED80, 1) blocking, sigcatchers 4 45 26046 [main] dd 3660 readv: no need to call ready_for_read 49 26095 [main] dd 3660 fhandler_dev_floppy::raw_read: read 60416 bytes into buffer 567 26662 [main] dd 3660 fhandler_dev_floppy::read_file: 1 (err 0) = ReadFile (1640, 268507896, to_read 60416, read 60416, 0) 47 26709 [main] dd 3660 fhandler_dev_floppy::raw_read: read 1024 bytes from buffer (rest 59392) -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: Error reported dd'ing close of end of block device with skip
On Dec 6 15:56, Loh, Joe wrote: > Hello Folks, > Here's the second part of the problem. The follow on from a previous > issue observed as reported in "Size difference reported by > /proc/partitions and lseek(SEEK_END) on block device." Searching the archive http://cygwin.com/cgi-bin/htsearch?config=htdig&words=dd+eom reveals a thread "Bug in dd ?? at EOM" which starts here: http://sourceware.org/ml/cygwin/2005-09/msg00878.html Btw, have you tried the latest snapshot from http://cygwin.com/snapshots/ ? Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat, Inc. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
