RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> From: Hashim Aziz > Sent: Monday, January 4, 2021 3:16 PM > > > From: Jason Pyeron > > Sent: 30 December 2020 1:35 AM > > To: mailto:cygwin@cygwin.com <mailto:cygwin@cygwin.com> > > Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk > > > > > -Original Message- > > > From: Brian Inglis > > > Sent: Tuesday, December 29, 2020 12:55 PM > > > > > > On 2020-12-28 19:41, Jason Pyeron wrote: > > > > On Monday, December 28, 2020 7:46 PM, Hashim Aziz wrote: > > > >> On 23 June 2020 8:33 PM, Brian Inglis wrote: > > > >>> I don't have the facilities to test, and there appear to be *NO* > > > >>> Windows > > > >>> documentation details on error condition handling, but my suspicion is > > > >>> that Unix reads and writes fail only *AFTER* reading or writing at the > > > >>> end of the device, but Windows reads and writes extents may be checked > > > >>> and failed *BEFORE* reading or writing any data near the end of the > > > >>> device. > > > >>> If the actual Windows error code returned is generic, Cygwin would > > > >>> need > > > >>> to pre-check the device size as Windows does, and reduce read and > > > >>> write > > > >>> sizes to the allowed maximum at the end of the device. > > > > > > >> That's very helpful, thank you. Do you know if any more work has been > > > >> done > > > >> to attempt to fix this bug, and whether it's likely to be fixed anytime > > > >> soon? It's crazy that such a commonly used command leaves so much data > > > >> unwiped unbeknown to so many users, it's a very serious security hole > > > >> and > > > >> the sooner it can be fixed the better. > > > > > > > Have you tried iflag=fullblock ? This causes special handling. > > > > > > >> I didn't previously see this email, but the point is that this is a > > > >> bug - > > > >> dd should not require first making calculations based on the size of > > > >> each > > > >> drive or using the smallest possible block size (and hence taking a > > > >> ridiculous amount of time) in order to do what > > > > > > > Do you have any metrics that it is faster, by any meaningful amount? If > > > > so I > > > > would be very interested in mitigating it, but I suspect not the actual > > > > case. > > That dd is faster with a block size of something large like 4M vs the default > of 512 or 1024 has > long been attested to by anyone who has used these tools on large drives - > here's just > one https://superuser.com/a/234241/323079 from SuperUser. But I just copied 1TB in 2 hours on Cygwin dd using a GCD calculated block size. No one is arguing that 512 is slow, the issue seems to be you do not like using a whole multiple of the actual drive size, and are insisting on using a non-divisible block size. > > > > > Could you please state explicitly, how many > > > bytes/sectors/blocks/pages/clusters > > > of what size you expect to get written, and how many > > > bytes/sectors/blocks/pages/clusters of what size are actually written? > > > > As mentioned in the original email and in > > https://superuser.com/a/1509903/323079, I have confirmed that everything > > but the last 48 sectors of the drive are wiped on both a 1TB HDD and a > > 128GB SSD, by inspecting the disks themselves with a hex editor (opening > > the raw disks in HxD). As someone else said earlier, on both occasions this > > points to the very last block failing to write properly. > > BUT, I want a clear test plan first, with clearly articulated issues > > BECAUSE I do not believe there are any issues actually existing. > > > > The best I can glean from the thread is > > > > 1. Cygwin is agedly breaking dd, but is also broken in the windows native > > dd [1] > > 2. Using "correct" (as I have previously defined it e.g. [4]) values is both > > 2.1. too slow on the write [2] > > 2.2. inappropriately complicated of a process for the user > > 3. It has been well investigated > > 3.1. with a root cause was identified as a windows OS issue [3,5,6] > > 3.2. with a mitigation [4,5] > > > > I am happy to spend time and money on 2.1. > > I will not spend my time on dealing with 2.2 - except providing an e
RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> -Original Message-
> From: Brian Inglis
> Sent: Tuesday, December 29, 2020 12:55 PM
>
> On 2020-12-28 19:41, Jason Pyeron wrote:
> > On Monday, December 28, 2020 7:46 PM, Hashim Aziz wrote:
> >> On 23 June 2020 8:33 PM, Brian Inglis wrote:
> >>> I don't have the facilities to test, and there appear to be *NO* Windows
> >>> documentation details on error condition handling, but my suspicion is
> >>> that Unix reads and writes fail only *AFTER* reading or writing at the
> >>> end of the device, but Windows reads and writes extents may be checked
> >>> and failed *BEFORE* reading or writing any data near the end of the
> >>> device.
> >>> If the actual Windows error code returned is generic, Cygwin would need
> >>> to pre-check the device size as Windows does, and reduce read and write
> >>> sizes to the allowed maximum at the end of the device.
>
> >> That's very helpful, thank you. Do you know if any more work has been done
> >> to attempt to fix this bug, and whether it's likely to be fixed anytime
> >> soon? It's crazy that such a commonly used command leaves so much data
> >> unwiped unbeknown to so many users, it's a very serious security hole and
> >> the sooner it can be fixed the better.
>
> > Have you tried iflag=fullblock ? This causes special handling.
>
> >> I didn't previously see this email, but the point is that this is a bug -
> >> dd should not require first making calculations based on the size of each
> >> drive or using the smallest possible block size (and hence taking a
> >> ridiculous amount of time) in order to do what
>
> > Do you have any metrics that it is faster, by any meaningful amount? If so I
> > would be very interested in mitigating it, but I suspect not the actual
> > case.
>
> >> it's meant to do. It should always wipe the last sector of the drive
> >> regardless, just as it does on other UNIX-like systems. This is why this
> >> behaviour is a bug that needs to be fixed.
>
> > This does not appear to be a bug, but user error. Per the DD source "Some
> > devices require alignment on a sector or page boundary"
> > DD has never "dealt with error handling" except when conversion were in
> > play.
> > When no conversions are in play it
>
> > {
> >/* Write any partial block. */
> >exit_status = EXIT_FAILURE;
> >break;
> > }
>
> > On windows the block devices require respecting block device boundaries, any
> > change would be an upstream patch - not a Cygwin patch.
>
> Your dd output appears to be ambiguous, relative to your claim that the last
> 48
> sectors are not written, and may appear to indicate that all sectors of the
> drive may have been written, assuming that you mean 512 byte sectors.
>
> > 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s
>
> 1000182120448 == 238462*4*1024^2
>
> > dd: error writing '/dev/sda': No space left on device
> > 238468+0 records in
>
> 1000207286272 == 238468*4*1024^2
>
> > 238467+0 records out
>
> 1000203091968 == 238467*4*1024^2
>
> > 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s
>
> 1000204861440 == 238467*4*1024^2 + 27*64*1024
>
> None of these numbers +/-48*512 bytes, which have odd factors, make a lot of
> sense as a disk size.
>
> Could you please state explicitly, how many
> bytes/sectors/blocks/pages/clusters
> of what size you expect to get written, and how many
> bytes/sectors/blocks/pages/clusters of what size are actually written?
>
> If anyone has access to a Linux system which has write access to a Windows
> drive
> over the network (e.g. Samba, NFS) where this can be reproduced, we can try to
> take this upstream, get their take, suggest an incremental reseek and write
> half
> buffer size patch, if they agree this is an issue and could be tackled in this
> manner.
I do, and I am even willing to spend a few dollars in testing this too.
BUT, I want a clear test plan first, with clearly articulated issues BECAUSE I
do not believe there are any issues actually existing.
The best I can glean from the thread is
1. Cygwin is agedly breaking dd, but is also broken in the windows native dd [1]
2. Using "correct" (as I have previously defined it e.g. [4]) values is both
2.1. too slow on the write [2]
2.2. inappropriately complicated of a process for the user
3. It has been well investigated
3.1. with a root cause was identifie
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
On 2020-12-28 19:41, Jason Pyeron wrote:
On Monday, December 28, 2020 7:46 PM, Hashim Aziz wrote:
On 23 June 2020 8:33 PM, Brian Inglis wrote:
I don't have the facilities to test, and there appear to be *NO* Windows
documentation details on error condition handling, but my suspicion is
that Unix reads and writes fail only *AFTER* reading or writing at the
end of the device, but Windows reads and writes extents may be checked
and failed *BEFORE* reading or writing any data near the end of the
device.
If the actual Windows error code returned is generic, Cygwin would need
to pre-check the device size as Windows does, and reduce read and write
sizes to the allowed maximum at the end of the device.
That's very helpful, thank you. Do you know if any more work has been done
to attempt to fix this bug, and whether it's likely to be fixed anytime
soon? It's crazy that such a commonly used command leaves so much data
unwiped unbeknown to so many users, it's a very serious security hole and
the sooner it can be fixed the better.
Have you tried iflag=fullblock ? This causes special handling.
I didn't previously see this email, but the point is that this is a bug -
dd should not require first making calculations based on the size of each
drive or using the smallest possible block size (and hence taking a
ridiculous amount of time) in order to do what
Do you have any metrics that it is faster, by any meaningful amount? If so I
would be very interested in mitigating it, but I suspect not the actual
case.
it's meant to do. It should always wipe the last sector of the drive
regardless, just as it does on other UNIX-like systems. This is why this
behaviour is a bug that needs to be fixed.
This does not appear to be a bug, but user error. Per the DD source "Some
devices require alignment on a sector or page boundary"
DD has never "dealt with error handling" except when conversion were in play.
When no conversions are in play it
{
/* Write any partial block. */
exit_status = EXIT_FAILURE;
break;
}
On windows the block devices require respecting block device boundaries, any
change would be an upstream patch - not a Cygwin patch.
Your dd output appears to be ambiguous, relative to your claim that the last 48
sectors are not written, and may appear to indicate that all sectors of the
drive may have been written, assuming that you mean 512 byte sectors.
1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s
1000182120448 == 238462*4*1024^2
dd: error writing '/dev/sda': No space left on device
238468+0 records in
1000207286272 == 238468*4*1024^2
238467+0 records out
1000203091968 == 238467*4*1024^2
1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s
1000204861440 == 238467*4*1024^2 + 27*64*1024
None of these numbers +/-48*512 bytes, which have odd factors, make a lot of
sense as a disk size.
Could you please state explicitly, how many bytes/sectors/blocks/pages/clusters
of what size you expect to get written, and how many
bytes/sectors/blocks/pages/clusters of what size are actually written?
If anyone has access to a Linux system which has write access to a Windows drive
over the network (e.g. Samba, NFS) where this can be reproduced, we can try to
take this upstream, get their take, suggest an incremental reseek and write half
buffer size patch, if they agree this is an issue and could be tackled in this
manner.
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> -Original Message-
> From: Hashim Aziz via Cygwin
> Sent: Monday, December 28, 2020 7:46 PM
>
>> From: Cygwin on behalf of Brian Inglis
>>
>> Sent: 23 June 2020 8:33 PM
>> To: cygwin@cygwin.com
>> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
>>
>> I don't have the facilities to test, and there appear to be *NO* Windows
>> documentation details on error condition handling, but my suspicion is that
>> Unix
>> reads and writes fail only *AFTER* reading or writing at the end of the
>> device,
>> but Windows reads and writes extents may be checked and failed *BEFORE*
>> reading
>> or writing any data near the end of the device.
>> If the actual Windows error code returned is generic, Cygwin would need to
>> pre-check the device size as Windows does, and reduce read and write sizes to
>> the allowed maximum at the end of the device.
>
> That's very helpful, thank you. Do you know if any more work has been done to
> attempt to
> fix this bug, and whether it's likely to be fixed anytime soon? It's crazy
> that such a
> commonly used command leaves so much data unwiped unbeknown to so many users,
> it's a very
> serious security hole and the sooner it can be fixed the better.
Have you tried iflag=fullblock ? This causes special handling.
> Sent: Monday, December 28, 2020 7:25 PM
> To: Jason Pyeron
>
> I didn't previously see this email, but the point is that this is a bug - dd
> should not
> require first making calculations based on the size of each drive or using
> the smallest
> possible block size (and hence taking a ridiculous amount of time) in order
> to do what
Do you have any metrics that it is faster, by any meaningful amount? If so I
would be very interested in mitigating it, but I suspect not the actual case.
> it's meant to do. It should always wipe the last sector of the drive
> regardless, just
> as it does on other UNIX-like systems. This is why this behaviour is a bug
> that
> needs to be fixed.
This does not appear to be a bug, but user error. Per the DD source "Some
devices require alignment
on a sector or page boundary"
DD has never "dealt with error handling" except when conversion were in play.
When no conversions are in play it
{
/* Write any partial block. */
exit_status = EXIT_FAILURE;
break;
}
On windows the block devices require respecting block device boundaries, any
change would be an upstream patch - not a Cygwin patch.
Sorry if this sounds rough.
Respectfully,
Jason Pyeron
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
From: Cygwin on behalf of Brian Inglis Sent: 23 June 2020 8:33 PM To: cygwin@cygwin.com Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk I don't have the facilities to test, and there appear to be *NO* Windows documentation details on error condition handling, but my suspicion is that Unix reads and writes fail only *AFTER* reading or writing at the end of the device, but Windows reads and writes extents may be checked and failed *BEFORE* reading or writing any data near the end of the device. If the actual Windows error code returned is generic, Cygwin would need to pre-check the device size as Windows does, and reduce read and write sizes to the allowed maximum at the end of the device. That's very helpful, thank you. Do you know if any more work has been done to attempt to fix this bug, and whether it's likely to be fixed anytime soon? It's crazy that such a commonly used command leaves so much data unwiped unbeknown to so many users, it's a very serious security hole and the sooner it can be fixed the better. -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
On 2020-06-28 11:50, Jason Pyeron wrote:
> On Sunday, June 28, 2020 10:35 AM, Christian Franke wrote:
>> Andrey Repin wrote:
dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress
>> The root of the problem is that the Windows WriteFile() function
>> apparently does not support truncated writes at EOM. If seek_position +
>> write_size > disk_size, then WriteFile() does nothing and returns an error.
>>> oflag=direct
>>> Although I'm unsure how Cygwin/Windows handles it. But without this flag,
>>> the
>>> write is cached, and the problem may be outside dd, or even Cygwin.
>> If 'oflag=direct' is used, dd passes O_DIRECT flag to open() call of
>> output file. Cygwin's open() function then passes
>> FILE_NO_INTERMEDIATE_BUFFERING to NtCreateFile() and the write()
>> function calls WriteFile() directly with original address and size.
>> Without O_DIRECT, Cygwin ensures that address and size passed to
>> WriteFile() are both aligned to sector size. All writes are then done
>> through a 64KiB internal buffer.
>> As a consequence, oflag=direct in the above dd command may increase
>> speed but would also let the final 4MiB WriteFile() fail. Without
>> oflag=direct, only the last 64KiB WriteFile() fails.
>> To clear the last sectors of the disk, use an appropriate small block
>> size. I did this several times with Cygwin 'dd seek=... bs=512 ...' to
>> get rid of Intel RST RAID metadata.
It appears that write(2) allows:
"An error return value while performing write() using direct I/O does not mean
the entire write has failed. Partial data may be written and the data at the
file offset on which the write() was attempted should be considered
inconsistent."
and write(3p) can return -1 errno EAGAIN to suggest a retry with a smaller
buffer size, in regards to pipes and FIFOs, so this could be allowed on devices:
"Deferred: ret=−1, errno=[EAGAIN]
This error indicates that a later request may succeed. It does not indicate
that it shall succeed, even if nbyte≤{PIPE_BUF}, because if no process reads
from the pipe or FIFO, the write never succeeds. An application could usefully
count the number of times [EAGAIN] is caused by a particular value of
nbyte>{PIPE_BUF} and perhaps do later writes with a smaller value, on the
assumption that the effective size of the pipe may have decreased.
Partial and deferred writes are only possible with O_NONBLOCK set."
"There is no exception regarding partial writes when O_NONBLOCK is set. With the
exception of writing to an empty pipe, this volume of POSIX.1‐2008 does not
specify exactly when a partial write is performed since that would require
specifying internal details of the implementation. Every application should be
prepared to handle partial writes when O_NONBLOCK is set and the requested
amount is greater than {PIPE_BUF}, just as every application should be prepared
to handle partial writes on other kinds of file descriptors."
and Cygwin write() could return -1 errno EAGAIN when a write at the end of a
device fails, and the application (dd) should be prepared to retry with a
smaller blocksize (half while even, then round up to even, as I/Os are
often/always? required to be even or binary multiple sizes) e.g. if the last
block below returned an error 39072726/2 -> 19536363 so +1 -> 19536364, or a
larger binary multiple such as 512, 64K, 1M using integer arithmetic: ((n - m +
1)/m + 1)*m, e.g in the trivial case above: (19536363 - 2 + 1)/2 + 1)*2 ->
19536364, either until a write succeeded, or the size was small enough e.g. 512,
that the result should be returned as a failure.
> The reason I have never encountered this is because I use a block size which
> is the largest practical GCD of the drive size and 512 bytes (typically
> between 32 MB and 64 MB).
>
> E.g. I have a drive that is 160,041,885,696 bytes, which divides 312,581,808
> times evenly into 512. I would use a block size of 39,072,726 bytes, which
> gives 4,096 blocks to write.
I am somewhat surprised that Unix and Cygwin support non-blocksize aligned
writes, but it seems you have used this successfully, correct?
A useful way to approach this uses coreutils bc and factor:
$ bc <<< 160041885696/512 # blocks
312581808
$ factor 312581808 # prime factors
312581808: 2 2 2 2 3 3 3 7 11 9397
checking and splitting off the non-binary factors, we have to pick some binary
multiple of the non-binary factor product in a reasonable range:
$ bc <<< 9397*11*7*3^3*2^4\;9397*11*7*3^3\;2^4\;9397*11*7*3^3*2
312581808
19536363
16
39072726
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/do
RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> -Original Message- > From: Christian Franke > Sent: Sunday, June 28, 2020 10:35 AM > > Andrey Repin wrote: > > > >> dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress > > The root of the problem is that the Windows WriteFile() function > apparently does not support truncated writes at EOM. If seek_position + > write_size > disk_size, then WriteFile() does nothing and returns an error. > > > > oflag=direct > > > > Although I'm unsure how Cygwin/Windows handles it. But without this flag, > > the > > write is cached, and the problem may be outside dd, or even Cygwin. > > If 'oflag=direct' is used, dd passes O_DIRECT flag to open() call of > output file. Cygwin's open() function then passes > FILE_NO_INTERMEDIATE_BUFFERING to NtCreateFile() and the write() > function calls WriteFile() directly with original address and size. > > Without O_DIRECT, Cygwin ensures that address and size passed to > WriteFile() are both aligned to sector size. All writes are then done > through a 64KiB internal buffer. > > As a consequence, oflag=direct in the above dd command may increase > speed but would also let the final 4MiB WriteFile() fail. Without > oflag=direct, only the last 64KiB WriteFile() fails. > > To clear the last sectors of the disk, use an appropriate small block > size. I did this several times with Cygwin 'dd seek=... bs=512 ...' to > get rid of Intel RST RAID metadata. The reason I have never encountered this is because I use a block size which is the largest practical GCD of the drive size and 512 bytes (typically between 32 MB and 64 MB). E.g. I have a drive that is 160,041,885,696 bytes, which divides 312,581,808 times evenly into 512. I would use a block size of 39,072,726 bytes, which gives 4,096 blocks to write. -Jason -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
Andrey Repin wrote: dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress The root of the problem is that the Windows WriteFile() function apparently does not support truncated writes at EOM. If seek_position + write_size > disk_size, then WriteFile() does nothing and returns an error. oflag=direct Although I'm unsure how Cygwin/Windows handles it. But without this flag, the write is cached, and the problem may be outside dd, or even Cygwin. If 'oflag=direct' is used, dd passes O_DIRECT flag to open() call of output file. Cygwin's open() function then passes FILE_NO_INTERMEDIATE_BUFFERING to NtCreateFile() and the write() function calls WriteFile() directly with original address and size. Without O_DIRECT, Cygwin ensures that address and size passed to WriteFile() are both aligned to sector size. All writes are then done through a 64KiB internal buffer. As a consequence, oflag=direct in the above dd command may increase speed but would also let the final 4MiB WriteFile() fail. Without oflag=direct, only the last 64KiB WriteFile() fails. To clear the last sectors of the disk, use an appropriate small block size. I did this several times with Cygwin 'dd seek=... bs=512 ...' to get rid of Intel RST RAID metadata. -- Regards, Christian -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
On 2020-06-25 06:08, Hashim Aziz via Cygwin wrote: > On 24 June 2020 11:21 AM, Andrey Repin wrote: >> Bottom posting in this channel, please. >> >>> I've just tested with the command: >> >>> dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress >> >> oflag=direct >> >> Although I'm unsure how Cygwin/Windows handles it. But without this flag, the >> write is cached, and the problem may be outside dd, or even Cygwin. >> >>> ...and can confirm that the last 48 sectors remain unwiped in this case >>> also. >> >>> Here is the output: >> >>> 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s >>> dd: error writing '/dev/sda': No space left on device >>> 238468+0 records in >>> 238467+0 records out >>> 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s > I have no idea what bottom posting is, mailing lists are new to me and I find > them a very archaic and noisy way to communicate development changes. I > really wish the Cygwin dev team would enter the 21st century and move to a > modern development process involving a repository like Github, I'm convinced > it would increase developer engagement with the project massively while > cutting down on so much of the noise. > I ran:> dd if=/dev/zero of=/dev/SDA bs=4m oflag=direct status=progress > ...and can confirm the bug still presents - the last 48 sectors fail to be > wiped. Please learn to use web search and/or information sites to educate yourself on topics new to you: https://en.wikipedia.org/wiki/Web_search_query https://en.wikipedia.org/wiki/Mailing_list https://en.wikipedia.org/wiki/Posting_style and please trim others' email addresses so you do not quote them verbatim in public posts which may be harvested by spammers. The Cygwin project is all volunteer with zero budget and little free time, on free shared infrastructure provided for more mainline projects. Fixing code, updating packages, and responding to posts are the priorities, in that order, if people have any free time to spare. When days, weeks, or months may pass between checking, active mailing lists can be more effectively filtered, to see what posts may be relevant for maintainers to look at. Other users with more experience and free time may try to help posters with information or suggestions as first or second level type support, until maintainers have time to look and perhaps respond. Please also remember that presently maintainers may be dealing with other issues and priorities e.g. health, home, family, work, and may have no free time for weeks to come. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in IEC units and prefixes, physical quantities in SI.] -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
I have no idea what bottom posting is, mailing lists are new to me and I find them a very archaic and noisy way to communicate development changes. I really wish the Cygwin dev team would enter the 21st century and move to a modern development process involving a repository like Github, I'm convinced it would increase developer engagement with the project massively while cutting down on so much of the noise. I ran: dd if=/dev/zero of=/dev/SDA bs=4m oflag=direct status=progress ...and can confirm the bug still presents - the last 48 sectors fail to be wiped. From: Andrey Repin Sent: 24 June 2020 11:21 AM To: Hashim Aziz ; cygwin@cygwin.com Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk Greetings, Hashim Aziz! Bottom posting in this channel, please. > I've just tested with the command: > dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress oflag=direct Although I'm unsure how Cygwin/Windows handles it. But without this flag, the write is cached, and the problem may be outside dd, or even Cygwin. > ...and can confirm that the last 48 sectors remain unwiped in this case also. > Here is the output: > 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s > dd: error writing '/dev/sda': No space left on device > 238468+0 records in > 238467+0 records out > 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s -- With best regards, Andrey Repin Wednesday, June 24, 2020 13:19:42 Sorry for my terrible english... -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
Greetings, Hashim Aziz! Bottom posting in this channel, please. > I've just tested with the command: > dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress oflag=direct Although I'm unsure how Cygwin/Windows handles it. But without this flag, the write is cached, and the problem may be outside dd, or even Cygwin. > ...and can confirm that the last 48 sectors remain unwiped in this case also. > Here is the output: > 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s > dd: error writing '/dev/sda': No space left on device > 238468+0 records in > 238467+0 records out > 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s -- With best regards, Andrey Repin Wednesday, June 24, 2020 13:19:42 Sorry for my terrible english... -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
I've just tested with the command: dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress ...and can confirm that the last 48 sectors remain unwiped in this case also. Here is the output: 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s dd: error writing '/dev/sda': No space left on device 238468+0 records in 238467+0 records out 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s From: Cygwin on behalf of Nicholas Clark via Cygwin Sent: 23 June 2020 5:06 PM To: cygwin Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk Do you get the same behavior when you run dd with iflag=fullblock? Without that flag, it's possible dd's input to read a smaller block than requested (which would correspond with a smaller write to the output). On Tue, Jun 23, 2020 at 9:28 AM Hashim Aziz via Cygwin wrote: > I hadn't checked with 512 byte block sizes because of the amount of time > it would have taken, but sure enough have just finished trying with bs=512 > and no block size at all (so dd's default block size, which is either 512 > or 1024) and although each wipe took over 24 hours, they did indeed wipe > all of the sectors. So it seems that there's a bug with regards to how > Cygwin handles the last block when a large (i.e. sane) block size is > selected, and that this bug doesn't occur on actual UNIX-based systems. > > > From: Cygwin on behalf of Eliot Moss < > m...@cs.umass.edu> > Sent: 20 June 2020 9:26 PM > To: The Cygwin Mailing List > Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk > > On 6/20/2020 1:31 PM, Hashim Aziz via Cygwin wrote: > > To reproduce simply run the following command on a drive (obviously, > this will irreversibly wipe all data): > > > > dd if=/dev/zero of=/dev/sdX bs=4M status=progress > > > > Both drives were attached via internal SATA (by way of a PCIE to SATA > Host Bus Adapter). > > > > Cygwin was running in an elevated window as dd cannot run in Cygwin > without administrator access, at least not on Windows 10 and not when > dealing with raw disks. I was running Avast the first time I discovered > this, and am currently running Windows Defender, so doubt that the AV is > the cause of this. > > > > The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial: > WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK). > > > > I just ran DD for Windows 0.6beta3 with variations of the following > command: > > > > dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M > > > > ...and can confirm that the bug also manifests here, but in a slightly > different way - irrespective of the disk or block size, it fails to wipe > the last 176 sectors of the drive. > > I was going to ask: even with block size 512 bytes? But I guess you > checked that ... > > Regards - Eliot > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple > -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
On 2020-06-23 09:28, Hashim Aziz via Cygwin wrote: > I hadn't checked with 512 byte block sizes because of the amount of time it > would have taken, but sure enough have just finished trying with bs=512 and > no block size at all (so dd's default block size, which is either 512 or > 1024) and although each wipe took over 24 hours, they did indeed wipe all of > the sectors. So it seems that there's a bug with regards to how Cygwin > handles the last block when a large (i.e. sane) block size is selected, and > that this bug doesn't occur on actual UNIX-based systems. > > > From: Cygwin on behalf of Eliot Moss > > Sent: 20 June 2020 9:26 PM > To: The Cygwin Mailing List > Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk > > On 6/20/2020 1:31 PM, Hashim Aziz via Cygwin wrote: >> To reproduce simply run the following command on a drive (obviously, this >> will irreversibly wipe all data): >> >> dd if=/dev/zero of=/dev/sdX bs=4M status=progress >> >> Both drives were attached via internal SATA (by way of a PCIE to SATA Host >> Bus Adapter). >> >> Cygwin was running in an elevated window as dd cannot run in Cygwin without >> administrator access, at least not on Windows 10 and not when dealing with >> raw disks. I was running Avast the first time I discovered this, and am >> currently running Windows Defender, so doubt that the AV is the cause of >> this. >> >> The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial: >> WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK). >> >> I just ran DD for Windows 0.6beta3 with variations of the following command: >> >> dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M >> >> ...and can confirm that the bug also manifests here, but in a slightly >> different way - irrespective of the disk or block size, it fails to wipe the >> last 176 sectors of the drive. > > I was going to ask: even with block size 512 bytes? But I guess you checked > that ... I don't have the facilities to test, and there appear to be *NO* Windows documentation details on error condition handling, but my suspicion is that Unix reads and writes fail only *AFTER* reading or writing at the end of the device, but Windows reads and writes extents may be checked and failed *BEFORE* reading or writing any data near the end of the device. If the actual Windows error code returned is generic, Cygwin would need to pre-check the device size as Windows does, and reduce read and write sizes to the allowed maximum at the end of the device. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in IEC units and prefixes, physical quantities in SI.] -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
Do you get the same behavior when you run dd with iflag=fullblock? Without that flag, it's possible dd's input to read a smaller block than requested (which would correspond with a smaller write to the output). On Tue, Jun 23, 2020 at 9:28 AM Hashim Aziz via Cygwin wrote: > I hadn't checked with 512 byte block sizes because of the amount of time > it would have taken, but sure enough have just finished trying with bs=512 > and no block size at all (so dd's default block size, which is either 512 > or 1024) and although each wipe took over 24 hours, they did indeed wipe > all of the sectors. So it seems that there's a bug with regards to how > Cygwin handles the last block when a large (i.e. sane) block size is > selected, and that this bug doesn't occur on actual UNIX-based systems. > > > From: Cygwin on behalf of Eliot Moss < > m...@cs.umass.edu> > Sent: 20 June 2020 9:26 PM > To: The Cygwin Mailing List > Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk > > On 6/20/2020 1:31 PM, Hashim Aziz via Cygwin wrote: > > To reproduce simply run the following command on a drive (obviously, > this will irreversibly wipe all data): > > > > dd if=/dev/zero of=/dev/sdX bs=4M status=progress > > > > Both drives were attached via internal SATA (by way of a PCIE to SATA > Host Bus Adapter). > > > > Cygwin was running in an elevated window as dd cannot run in Cygwin > without administrator access, at least not on Windows 10 and not when > dealing with raw disks. I was running Avast the first time I discovered > this, and am currently running Windows Defender, so doubt that the AV is > the cause of this. > > > > The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial: > WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK). > > > > I just ran DD for Windows 0.6beta3 with variations of the following > command: > > > > dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M > > > > ...and can confirm that the bug also manifests here, but in a slightly > different way - irrespective of the disk or block size, it fails to wipe > the last 176 sectors of the drive. > > I was going to ask: even with block size 512 bytes? But I guess you > checked that ... > > Regards - Eliot > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple > -- > Problem reports: https://cygwin.com/problems.html > FAQ: https://cygwin.com/faq/ > Documentation:https://cygwin.com/docs.html > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple > -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
On 6/23/2020 11:28 AM, Hashim Aziz via Cygwin wrote: I hadn't checked with 512 byte block sizes because of the amount of time it would have taken, but sure enough have just finished trying with bs=512 and no block size at all (so dd's default block size, which is either 512 or 1024) and although each wipe took over 24 hours, they did indeed wipe all of the sectors. So it seems that there's a bug with regards to how Cygwin handles the last block when a large (i.e. sane) block size is selected, and that this bug doesn't occur on actual UNIX-based systems. Well, that certainly points the finger at last-block handling ... EM -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
I hadn't checked with 512 byte block sizes because of the amount of time it would have taken, but sure enough have just finished trying with bs=512 and no block size at all (so dd's default block size, which is either 512 or 1024) and although each wipe took over 24 hours, they did indeed wipe all of the sectors. So it seems that there's a bug with regards to how Cygwin handles the last block when a large (i.e. sane) block size is selected, and that this bug doesn't occur on actual UNIX-based systems. From: Cygwin on behalf of Eliot Moss Sent: 20 June 2020 9:26 PM To: The Cygwin Mailing List Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk On 6/20/2020 1:31 PM, Hashim Aziz via Cygwin wrote: > To reproduce simply run the following command on a drive (obviously, this > will irreversibly wipe all data): > > dd if=/dev/zero of=/dev/sdX bs=4M status=progress > > Both drives were attached via internal SATA (by way of a PCIE to SATA Host > Bus Adapter). > > Cygwin was running in an elevated window as dd cannot run in Cygwin without > administrator access, at least not on Windows 10 and not when dealing with > raw disks. I was running Avast the first time I discovered this, and am > currently running Windows Defender, so doubt that the AV is the cause of this. > > The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial: > WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK). > > I just ran DD for Windows 0.6beta3 with variations of the following command: > > dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M > > ...and can confirm that the bug also manifests here, but in a slightly > different way - irrespective of the disk or block size, it fails to wipe the > last 176 sectors of the drive. I was going to ask: even with block size 512 bytes? But I guess you checked that ... Regards - Eliot -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
On 6/20/2020 1:31 PM, Hashim Aziz via Cygwin wrote: To reproduce simply run the following command on a drive (obviously, this will irreversibly wipe all data): dd if=/dev/zero of=/dev/sdX bs=4M status=progress Both drives were attached via internal SATA (by way of a PCIE to SATA Host Bus Adapter). Cygwin was running in an elevated window as dd cannot run in Cygwin without administrator access, at least not on Windows 10 and not when dealing with raw disks. I was running Avast the first time I discovered this, and am currently running Windows Defender, so doubt that the AV is the cause of this. The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial: WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK). I just ran DD for Windows 0.6beta3 with variations of the following command: dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M ...and can confirm that the bug also manifests here, but in a slightly different way - irrespective of the disk or block size, it fails to wipe the last 176 sectors of the drive. I was going to ask: even with block size 512 bytes? But I guess you checked that ... Regards - Eliot -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
To reproduce simply run the following command on a drive (obviously, this will irreversibly wipe all data): dd if=/dev/zero of=/dev/sdX bs=4M status=progress Both drives were attached via internal SATA (by way of a PCIE to SATA Host Bus Adapter). Cygwin was running in an elevated window as dd cannot run in Cygwin without administrator access, at least not on Windows 10 and not when dealing with raw disks. I was running Avast the first time I discovered this, and am currently running Windows Defender, so doubt that the AV is the cause of this. The hard drives are a Western Digital WD10PURX-64E5EY0 (Serial: WD-WCC4J6HX189U) and a Kingston SV200S3128G (Serial: 12BA315PKAWK). I just ran DD for Windows 0.6beta3 with variations of the following command: dd.exe if=/dev/zero of=\\.\PHYSICALDRIVEX --progress bs=4M ...and can confirm that the bug also manifests here, but in a slightly different way - irrespective of the disk or block size, it fails to wipe the last 176 sectors of the drive. Hashim From: Jason Pyeron Sent: 19 June 2020 9:25 PM To: 'Hashim Aziz' Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk > -Original Message- > From: Hashim Aziz via Cygwin > Sent: Friday, June 19, 2020 2:19 PM > > I first came across this bug last year, and wrote up my findings in a > detailed answer to the > StackExchange network: > > https://superuser.com/questions/1505715/why-does-dd-give-me-an-error-at-the-end-of-zero-writing-a-disk > > I didn’t then have the time to report it to this mailing list, but have just > tested it using the > latest version of Cygwin on the same two disks as I did then, and can confirm > that the problem still > persists. It’s also of note that I was running Windows 7 back then and am > currently running 10, so > this is now confirmed to manifest on both of the last two Windows OSes. I’m > happy to work with the > team to get this fixed, as I frequently need to make use of Cygwin dd to zero > out drives for both > security and regulatory compliance reasons, and failing to wipe the last 48 > sectors of all drives is a > pretty significant security risk. Hmm, never had that issue. Steps to reproduce please. Running as Admin? Antivirus? How is the drive attached (USB, SATA, other)? What are the drive models (is there a known firmware issue on windows)? Does the non-cygwin http://www.chrysocome.net/dd work? -Jason -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
