Re: How long will this take?
On Fri 26 Jun 2020 at 19:50:21 (-0700), David Christensen wrote: > On 2020-06-26 18:25, David Wright wrote: > > On Fri 26 Jun 2020 at 15:06:31 (-0700), David Christensen wrote: > > > On 2020-06-26 06:07, David Wright wrote: > > > > > On this slow machine with an oldish PATA disk, > > > > I can get about 75% speed from urandom, 15MB/s vs 20MB/s on a 29GiB > > > > partition (no encryption). There's a noticeable slowdown because, > > > > I presume, the machine runs a bit short of entropy after a while. > > > > > > I think you are noticing a slowdown when the Linux write buffer fills. > > > > I'm not sure where these write buffers might be hiding: the > > 2000-vintage PC has 512MB memory, and the same size swap partition, > > though the latter is on a disk constructed one month earlier than the > > target disk (Feb/Mar 2008). The target disk has 8MB of cache. > > With a leisurely determination of dd's PID, my first USR1 poke > > occurred no earlier than after 4GB of copying, over three minutes in. > > I seem to recall that most of my EIDE interfaces and drives were 100 > MB/s. (A few were 133 MB/s.) So, bulk reads or writes can completely > use an 8 MB cache in a fraction of a second. This is IDE. The buses run at 100MHz, but I don't know where the bottlenecks are. The idea was only to compare writing zeros and random data. The machine, a 650MHz SE440BX-2 (Seattle 2), was selected on the basis that it's presently housing a secondary drive with two spare "root filesystem partitions" (which was in the Dell Optiplex that died last month). It was doing nothing but running two ssh sessions, one for dd and one for kill. > top(1) reports memory statistics on line 4. I believe "buff/cache" is > the amount of memory being used for I/O write buffering and read > caching. Line 5 has statics for swap. I do not know if memory write > buffer / read cache usage interacts with swap usage, but it would not > surprise me. top(1) should be able to show you. > > Perhaps I misinterpreted your "slowdown" statement. I assumed you ran > a command similar to: > > # dd if=/dev/urandom of=/dev/sdxn bs=1M status=progress Close: I was running within a script command, so I just poked the dd occasionally with kill -USR1 to record its progress in the typescript file. > dd(1) is copying PRN data from the CPU to the kernel write buffer (in > memory) and the kernel input/ output stack is copying from the write > buffer to the HDD (likely via direct memory access, DMA). The > 'status=progress' option will cause dd(1) to display the rate at which > the write buffer is being filled. I am not sure how to monitor the > rate at which the write buffer is being drained. Assuming the write > buffer is initially empty, the filling process is "fast", and the > draining process is "slow" when the above command is started, dd(1) > should show fast throughput until the write buffer fills and then show > slow throughput for the remainder of the transfer. And, without a > 'sync' option to dd(1), dd(1) will exit and the shell will display the > next prompt as the final write buffer contents are being written to > the HDD (e.g. the HDD will be busy for a short while after dd(1) is > finished). > > Another possibility -- magnetic disk drives have more sectors in outer > tracks (lower sector number) than they have in inner tracks (higher > sector number). When filling an entire drive, I have seen the > transfer rate drop by 40~50% over the duration of the transfer. This > is normal. Is this what you are referring to? I tried to take account of these possibilities by using 29GB partitions, much larger than the buffer sizes, and writing two different partitions. But replicating the run a few times didn't give me consistent enough timings to have confidence in any conclusions. When I tried using sync to reduce the effect of buffering, things slowed so much that I suspect there would be no shortage of entropy anyway. Regardless, the loss of speed is not serious enough for me to change my strategy from: urandom before cryptsetup, zero before encrypting swap, zero to erase disk at end of life/possession. I *have* given up running badblocks. (The disk layout is: Device Start End Sectors Size Type /dev/sdb1 2048 8191 6144 3M BIOS boot /dev/sdb2 8192 1023999 1015808 496M EFI System /dev/sdb31024000 2047999 1024000 500M Linux swap /dev/sdb42048000 63487999 6144 29.3G Linux filesystem /dev/sdb5 63488000 124927999 6144 29.3G Linux filesystem /dev/sdb6 124928000 976773119 851845120 406.2G Linux filesystem ) Cheers, David.
Re: How long will this take?
On Friday, June 26, 2020 09:41:26 PM Seeds Notoneofmy wrote: > On 6/27/20 3:20 AM, rhkra...@gmail.com wrote: > > This (the above) subject line is not very good, but at least it gives a > > hint that it probably is, or at least could be, computer related. > > Would you please explain the computer related "hint" in "How long will > this take?" I'm reallly not inclined to do so. It really doesn't matter if you used a poor subject line and somebody else did also -- no point arguing over which is worse. I will say apparently others also were able to see a hint in "how long will this take"? To make a feeble attempt at explaining the hint I saw -- computers require time to perform tasks, so that was a hint to me. "Have you seen this inside" provides no hint to me. I would suggest that you take the criticism in stride and try to make better subjects in the future. And, if you can suggest better subjects to someone else, that can be acceptable as long as you do it with good will. > For starters, I could end that sentence in so many ways, just use your > imagination. > > In much the same way I could end, 'have you seen this inside...' > > We can drop the subjective statements and look objectively at things.
Re: On Subject drift [was: How long will this take?]
On Sat, Jun 27, 2020 at 04:36:03PM +0200, Seeds Notoneofmy wrote: [...] Thanks for your kind words. > To learn is a personal effort. I've learnt from this and you have too. I > hope others have as well. The community will be better of it. Let's hope so, yes :-) Cheers -- t signature.asc Description: Digital signature
Re: On Subject drift [was: How long will this take?]
On 6/27/20 10:51 AM, to...@tuxteam.de wrote: I'll take your question face value. While there are big similarities between both, I still see at least two (small!) differences. 1) The subject "how long will this take" at least gives a hint at something performance-related. The subject "have you seen this inside..." doesn't give any hint. 2) In the current example, the subjet was (somewhat, see (1) for the limitations) appropriate in the initial mail and slowly lost relevance as the thread drifted away (it still has a bit of relevance, since the speed of /dev/urandom is somewhere around there). As to why they merit different treatment -- I don't know. I don't think so. Human behaviour is messy, more so in big groups. If you still feel you've been treated unjustly, my apologies are still up for the part I took in that. You can always mail me personally if you think there's something I can/should learn from all that. Cheers -- t With such a well considered response, I will be foolish, if not stupid, to press this point further. And I want to thank you, very much, Tomas, for it. And you're right, I do feel treated unjustly. But we can drop all that now. I'm responding primarily to share with you how much of a man I think you're, based on your response. Thank you. And you're right too, where ever humans meet, you can expect some mess. This is not a perfect system here. And I do not wish it so. My hope is that your handling of it, my bringing it up, would contribute towards making this 'community' a welcoming place for all. To learn is a personal effort. I've learnt from this and you have too. I hope others have as well. The community will be better of it. Cheers
Re: On Subject drift [was: How long will this take?]
Hi, 27 juin 2020 à 10:51 de to...@tuxteam.de: > I'll take your question face value. While there are big similarities > between both, I still see at least two (small!) differences. > > 1) The subject "how long will this take" at least gives a hint > at something performance-related. The subject "have you seen > this inside..." doesn't give any hint. > > 2) In the current example, the subjet was (somewhat, see (1) for > the limitations) appropriate in the initial mail and slowly > lost relevance as the thread drifted away (it still has a bit > of relevance, since the speed of /dev/urandom is somewhere around > there). > People can try to find any justification they want, "How long will this take?" was not a "meaningful, specific subject header" either (see http://www.catb.org/~esr/faqs/smart-questions.html#bespecific that has been recommended multiple times on this list). > As to why they merit different treatment -- I don't know. I don't > think so. Human behaviour is messy, more so in big groups. > I think this is definitely the real reason ;) Best regards, l0f4r0
On Subject drift [was: How long will this take?]
On Sat, Jun 27, 2020 at 01:37:57AM +0200, Seeds Notoneofmy wrote: [...] > Recently I posted here with the subject line: "have you seen this inside..." > > And I was lectured by no fewer than three people. As someone of those who did the "lecturing", I somehow feel urged to respond ;-) > The subject line in this thread is: "How long will this take?" Yes. The original subject wasn't very informative. Then, the thread drifted away (not the subject, as I promised above in the Subject, alas). This happens, often one small nudge at a time (you may notice that you just contributed to that yourself ;-) Sometimes people try to straighten the mess a bit (I tried in this mail) by adjusting the subject. As a human endeavour, this, too, is bound to fail alarmingly often. > I struggle to understand the difference between the two subject lines > that merits their different treatment. I'll take your question face value. While there are big similarities between both, I still see at least two (small!) differences. 1) The subject "how long will this take" at least gives a hint at something performance-related. The subject "have you seen this inside..." doesn't give any hint. 2) In the current example, the subjet was (somewhat, see (1) for the limitations) appropriate in the initial mail and slowly lost relevance as the thread drifted away (it still has a bit of relevance, since the speed of /dev/urandom is somewhere around there). As to why they merit different treatment -- I don't know. I don't think so. Human behaviour is messy, more so in big groups. If you still feel you've been treated unjustly, my apologies are still up for the part I took in that. You can always mail me personally if you think there's something I can/should learn from all that. Cheers -- t signature.asc Description: Digital signature
Re: How long will this take?
On Sb, 27 iun 20, 01:37:57, Seeds Notoneofmy wrote: > > Recently I posted here with the subject line: "have you seen this inside..." > > And I was lectured by no fewer than three people. > > The subject line in this thread is: "How long will this take?" > > I struggle to understand the difference between the two subject lines > that merits their different treatment. It seems to me you are complaining that the OP of this thread was not lectured as well. Was this your intention? Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser signature.asc Description: PGP signature
Re: How long will this take?
On 2020-06-26 18:25, David Wright wrote: On Fri 26 Jun 2020 at 15:06:31 (-0700), David Christensen wrote: On 2020-06-26 06:07, David Wright wrote: On this slow machine with an oldish PATA disk, I can get about 75% speed from urandom, 15MB/s vs 20MB/s on a 29GiB partition (no encryption). There's a noticeable slowdown because, I presume, the machine runs a bit short of entropy after a while. I think you are noticing a slowdown when the Linux write buffer fills. I'm not sure where these write buffers might be hiding: the 2000-vintage PC has 512MB memory, and the same size swap partition, though the latter is on a disk constructed one month earlier than the target disk (Feb/Mar 2008). The target disk has 8MB of cache. With a leisurely determination of dd's PID, my first USR1 poke occurred no earlier than after 4GB of copying, over three minutes in. I seem to recall that most of my EIDE interfaces and drives were 100 MB/s. (A few were 133 MB/s.) So, bulk reads or writes can completely use an 8 MB cache in a fraction of a second. top(1) reports memory statistics on line 4. I believe "buff/cache" is the amount of memory being used for I/O write buffering and read caching. Line 5 has statics for swap. I do not know if memory write buffer / read cache usage interacts with swap usage, but it would not surprise me. top(1) should be able to show you. Perhaps I misinterpreted your "slowdown" statement. I assumed you ran a command similar to: # dd if=/dev/urandom of=/dev/sdxn bs=1M status=progress dd(1) is copying PRN data from the CPU to the kernel write buffer (in memory) and the kernel input/ output stack is copying from the write buffer to the HDD (likely via direct memory access, DMA). The 'status=progress' option will cause dd(1) to display the rate at which the write buffer is being filled. I am not sure how to monitor the rate at which the write buffer is being drained. Assuming the write buffer is initially empty, the filling process is "fast", and the draining process is "slow" when the above command is started, dd(1) should show fast throughput until the write buffer fills and then show slow throughput for the remainder of the transfer. And, without a 'sync' option to dd(1), dd(1) will exit and the shell will display the next prompt as the final write buffer contents are being written to the HDD (e.g. the HDD will be busy for a short while after dd(1) is finished). Another possibility -- magnetic disk drives have more sectors in outer tracks (lower sector number) than they have in inner tracks (higher sector number). When filling an entire drive, I have seen the transfer rate drop by 40~50% over the duration of the transfer. This is normal. Is this what you are referring to? David
Re: How long will this take?
On 6/27/20 3:20 AM, rhkra...@gmail.com wrote: This (the above) subject line is not very good, but at least it gives a hint that it probably is, or at least could be, computer related. Would you please explain the computer related "hint" in "How long will this take?" For starters, I could end that sentence in so many ways, just use your imagination. In much the same way I could end, 'have you seen this inside...' We can drop the subjective statements and look objectively at things.
Re: How long will this take?
On Fri 26 Jun 2020 at 15:06:31 (-0700), David Christensen wrote: > On 2020-06-26 06:07, David Wright wrote: > > On Fri 19 Jun 2020 at 14:52:11 (-0700), David Christensen wrote: > > > > Benchmark is one thing. But, from a security viewpoint, writing zeros > > > to an encrypted volume amounts to providing blocks of plaintext for > > > corresponding blocks of cyphertext, thereby facilitating > > > cryptanalysis. > > > > So in view of the unlikelihood of badblocks actually logging something > > more useful than SMART (where available) or normal disk write errors, > > perhaps a compromise (for my use case) is to just write /dev/urandom > > rather than /dev/zero. > > Copying random data to a partition while creating an encrypted > filesystem provides a high-entropy backdrop to conceal ciphertext > blocks. This is a form of steganography. The Debian Installer manual > partitioning page has an option to do this. I presume you meet this option when you select "Configure encrypted volumes", something that I've never done. Because currently I only encrypt /home and swap, I set these up after installation, if they're not already there. I must admit that I prefer to partition disks and set up encryption outside the d-i, usually capturing the process with script. > As the storage is used, the initial random blocks will be overwritten > by ciphertext blocks. Depending upon filesystem, encryption, volume > management, and/or device details, the steganography degrades and may > eventually disappear. > > Copying random data to storage will add fresh nearly-random blocks on > the device, improving the steganography. (The canonical example is to > copy /dev/urandom to a file until the filesystem fills up, and then > delete the file. But, this takes time and adds wear to the device.) Yes, SSD caveat taken on board. > > On this slow machine with an oldish PATA disk, > > I can get about 75% speed from urandom, 15MB/s vs 20MB/s on a 29GiB > > partition (no encryption). There's a noticeable slowdown because, > > I presume, the machine runs a bit short of entropy after a while. > > I think you are noticing a slowdown when the Linux write buffer fills. I'm not sure where these write buffers might be hiding: the 2000-vintage PC has 512MB memory, and the same size swap partition, though the latter is on a disk constructed one month earlier than the target disk (Feb/Mar 2008). The target disk has 8MB of cache. With a leisurely determination of dd's PID, my first USR1 poke occurred no earlier than after 4GB of copying, over three minutes in. Cheers, David.
Re: How long will this take?
Quoted lines resequenced for my convenience in responding. On Friday, June 26, 2020 07:37:57 PM Seeds Notoneofmy wrote: > I struggle to understand the difference between the two subject lines > that merits their different treatment. > The subject line in this thread is: "How long will this take?" This (the above) subject line is not very good, but at least it gives a hint that it probably is, or at least could be, computer related. > Recently I posted here with the subject line: "have you seen this > inside..." This gives no hint that it is computer related, and sounds very spammy, like the subject lines that many of us run into, trying to get us to open an email that has subject matter we are absolutely not interested in. > And I was lectured by no fewer than three people. Not I, said the spider to the fly
Re: How long will this take?
On 6/8/20 10:22 PM, Matthew Campbell wrote: I bought a new 4 terrabyte hard drive that is connected with a USB cable using USB2. It took about 32 hours to read every sector on the drive to look for bad sectors. I started blanking the sectors using /dev/zero last Friday night. It still isn't done. Is there a way I can find out how much data a particular process has written to the disk? I'm using Debian 10.4. dd if=/dev/zero of=/dev/sdb ibs=4096 count=976754646 name=Matthew%20Campbell=trenix25%40pm.me Recently I posted here with the subject line: "have you seen this inside..." And I was lectured by no fewer than three people. The subject line in this thread is: "How long will this take?" I struggle to understand the difference between the two subject lines that merits their different treatment. Thanks.
Re: How long will this take?
On 2020-06-26 06:07, David Wright wrote: On Fri 19 Jun 2020 at 14:52:11 (-0700), David Christensen wrote: Benchmark is one thing. But, from a security viewpoint, writing zeros to an encrypted volume amounts to providing blocks of plaintext for corresponding blocks of cyphertext, thereby facilitating cryptanalysis. So in view of the unlikelihood of badblocks actually logging something more useful than SMART (where available) or normal disk write errors, perhaps a compromise (for my use case) is to just write /dev/urandom rather than /dev/zero. Copying random data to a partition while creating an encrypted filesystem provides a high-entropy backdrop to conceal ciphertext blocks. This is a form of steganography. The Debian Installer manual partitioning page has an option to do this. As the storage is used, the initial random blocks will be overwritten by ciphertext blocks. Depending upon filesystem, encryption, volume management, and/or device details, the steganography degrades and may eventually disappear. Copying random data to storage will add fresh nearly-random blocks on the device, improving the steganography. (The canonical example is to copy /dev/urandom to a file until the filesystem fills up, and then delete the file. But, this takes time and adds wear to the device.) On this slow machine with an oldish PATA disk, I can get about 75% speed from urandom, 15MB/s vs 20MB/s on a 29GiB partition (no encryption). There's a noticeable slowdown because, I presume, the machine runs a bit short of entropy after a while. I think you are noticing a slowdown when the Linux write buffer fills. David
Re: How long will this take?
On Fri 19 Jun 2020 at 14:52:11 (-0700), David Christensen wrote: > On 2020-06-18 19:13, David Wright wrote: > > On Fri 12 Jun 2020 at 07:51:30 (-0400), Michael Stone wrote: > > > On Thu, Jun 11, 2020 at 08:52:10PM -0500, David Wright wrote: > > > > > > The only unaddressed point in my use case is the prevention of a > > > > high-water mark, because zeroing the drive achieves precisely the > > > > opposite. What ought I to be running, instead of badblocks -w -t random, > > > > to achieve that goal? > > > > > > Create the encrypted volume first, then write zeros to it. :) > > > > Duh! That should work a treat. My posting that example bore me fruit. > > Benchmark is one thing. But, from a security viewpoint, writing zeros > to an encrypted volume amounts to providing blocks of plaintext for > corresponding blocks of cyphertext, thereby facilitating > cryptanalysis. So in view of the unlikelihood of badblocks actually logging something more useful than SMART (where available) or normal disk write errors, perhaps a compromise (for my use case) is to just write /dev/urandom rather than /dev/zero. On this slow machine with an oldish PATA disk, I can get about 75% speed from urandom, 15MB/s vs 20MB/s on a 29GiB partition (no encryption). There's a noticeable slowdown because, I presume, the machine runs a bit short of entropy after a while. Cheers, David.
Re: How long will this take?
On 2020-06-18 19:13, David Wright wrote: On Fri 12 Jun 2020 at 07:51:30 (-0400), Michael Stone wrote: On Thu, Jun 11, 2020 at 08:52:10PM -0500, David Wright wrote: The only unaddressed point in my use case is the prevention of a high-water mark, because zeroing the drive achieves precisely the opposite. What ought I to be running, instead of badblocks -w -t random, to achieve that goal? Create the encrypted volume first, then write zeros to it. :) Duh! That should work a treat. My posting that example bore me fruit. Cheers, David. Benchmark is one thing. But, from a security viewpoint, writing zeros to an encrypted volume amounts to providing blocks of plaintext for corresponding blocks of cyphertext, thereby facilitating cryptanalysis. David
Re: How long will this take?
On Fri 12 Jun 2020 at 07:51:30 (-0400), Michael Stone wrote: > On Thu, Jun 11, 2020 at 08:52:10PM -0500, David Wright wrote: > > The only unaddressed point in my use case is the prevention of a > > high-water mark, because zeroing the drive achieves precisely the > > opposite. What ought I to be running, instead of badblocks -w -t random, > > to achieve that goal? > > Create the encrypted volume first, then write zeros to it. :) Duh! That should work a treat. My posting that example bore me fruit. Cheers, David.
Re: Fw: Fw: How long will this take?
On Thu, Jun 11, 2020, at 11:05 AM, Dan Ritter wrote: > Matthew Campbell wrote: > > The process is complete. The 4 TB drive has been successfully blanked in > > less than 40 hours using dd. It got done between 11 pm last night and 12 am > > this morning. dd showed an overall average write speed of 28.4 MB/s. It was > > never my intention to start a war. My sincere apologies if I have said or > > done anything to offend anyone. I prefer to test out new hardware after I > > buy it and I prefer to blank new hard drives before partitioning and > > writing out new file systems. Thank you to each of you for your assistance. > > > > I think you stated your issue well, responded to queries, and > reported back with the results. Thanks for being part of the > community. +1 You didn't start the war, Mathew. It was there long before you stumbled into the battlefield. Sadly, any list with more than one participant will have issues about which people will disagree vociferously. There's probably somebody's law that states that observation... Something like, "Whenever the words 'stupid' or 'idiot' occur in a thread, there's nothing to be gained in following it beyond that point." Enjoy! Rick
Re: How long will this take?
On Thu, Jun 11, 2020 at 08:52:10PM -0500, David Wright wrote: If you were preserving the disk contents (imagine there were proprietary encryption software on it), and performed a "read test" or ran badblocks on it, would that be sufficient to test the disk's performance, as it's merely reading the sectors. Or do you have to actually write, with badblocks -r for example? That really depends on whether you want to test read or write performance. If you mean that you want to test for correct operation rather than performance then you need a write test to fully exercise the disk. Some errors will only be found on write, and conversely some read errors will be corrected (remapped) on write. The only unaddressed point in my use case is the prevention of a high-water mark, because zeroing the drive achieves precisely the opposite. What ought I to be running, instead of badblocks -w -t random, to achieve that goal? Create the encrypted volume first, then write zeros to it. :)
Re: [OT] How to help the OP? (was: How long will this take?)
On Wed 10 Jun 2020 at 15:48:01 (-0400), Michael Stone wrote: > On Wed, Jun 10, 2020 at 09:20:50PM +0200, l0f...@tuta.io wrote: > > 10 juin 2020 à 20:51 de mst...@debian.org: > > > On Wed, Jun 10, 2020 at 12:02:13PM -0500, David Wright wrote: > > > > > >> Both l0f4r0 and I have asked why the OP > > >> is zeroing the drive, but no reply yet. Perhaps you can suggest an > > >> answer. > > >> > > > I don't really care why the OP is doing it. I can think of several > > > possibilities, but I don't see any need to argue with him over it. At > > > some point it's reasonable to simply accept that someone is trying to do > > > something and either help or not. I haven't seen anyone arguing with the OP, even though there are several arguments elsewhere in the thread. > > It's the usual dilemma on mailing-lists/forums: > > * Are the respondents supposed to answer the OP question directly without > > hindsight (potential XY problem sometimes - http://xyproblem.info/)? > > * Or are they expected to put the issue into perspective and challenge it? > > I know it can be frustrating because this is not always the answer we as OP > > want to read but it can be really instructive as we understand we were on a > > wrong track. > > > > I'm still divided on this subject, maybe we need both depending on the > > context/OP... I don't see it as a dilemma; it's been opined many a time here that people answer the question however they want. Sometimes that might involve asking questions, and on occasions a thread might end up like a round of Twenty Questions. > In general, I think that once someone's been asked a question multiple > times, especially when that question isn't really germane, it's just > rude to keep insisting on an answer. JMO. Who insisted on an answer? I didn't actually ask any direct question of the OP, and l0f4r0 merely asked out of curiosity. You seem to imply that the OP was being hectored. All my direct questions have been asked of you. Thanks for the answers. Cheers, David.
Re: How long will this take?
On Wed 10 Jun 2020 at 14:51:32 (-0400), Michael Stone wrote: > On Wed, Jun 10, 2020 at 12:02:13PM -0500, David Wright wrote: [snipped the first part as it's covered elsewhere] > > My use case for badblocks was closer to that of the OP, but still > > different. Firstly, the disk contained personal data from unencrypted > > use in the past. Secondly, I was intending to use it encrypted (as > > mentioned) and prefer no high-watermark. Thirdly, because of its > > age (2011), I was interested in seeing how well it performed. I have > > no idea whether the disk is "modern" in the sense you used, as I don't > > follow the technology like some people on this list evidently do. > > Fourthly, I don't make a habit of throwing away 2TB disks. > > badblocks isn't particularly useful for achieving any of those goals > vs just writing zeros. "modern" in this context means anything since > probably the mid 90s but my memory is a bit fuzzy on the exact dates. > certainly anything since the turn of the century. > > > But, as you know about these things, a few questions: > > > > . How does badblocks do its job in readonly mode, given that it > > doesn't know what any block's content ought to be. > > you have to write the test data ahead of time > > > . Why might the OP run badblocks, particularly non-destructively > > (as if to preserve something), and *then* zero the drive. > > the only person I saw mention badblocks in this thread was you, but I > guess I might have missed it No, you're right, I brought it up, and I *am* conflating two things: the OP running an unspecified "read test", reading every sector looking for errors, and a hypothetical person running badblocks. If you were preserving the disk contents (imagine there were proprietary encryption software on it), and performed a "read test" or ran badblocks on it, would that be sufficient to test the disk's performance, as it's merely reading the sectors. Or do you have to actually write, with badblocks -r for example? > > . What's the easiest way of finding out about "consistent bad > > (not remappable) sectors" on a drive, as I soon will have to > > repeat this result (if not by this exact process) with a 3TB > > disk of 2013 vintage. (The good news: it has a USB3 connection.) > > you'll get a bunch of errors while writing, and probably the drive > will drop offline. you can use smartctl in the smartmontools package > to see the status of retries & remapped sectors and get a health > report on the drive, which you can use to decide whether to keep the > drive in service even if it is currently working. (as a drive ages it > will often record an increasing number of correctable errors, which > typically will result in failure in the not-distant future.) OK, so as far as the 2TB disk is concerned, writing anything over the entire disk will provoke the reporting and/or remapping of any bad sectors by SMART, so you can then check the statistics. The only unaddressed point in my use case is the prevention of a high-water mark, because zeroing the drive achieves precisely the opposite. What ought I to be running, instead of badblocks -w -t random, to achieve that goal? > a confounding factor is that you might also get write errors and > dropped disk if there's a USB issue, separate from whether the drive > is working properly. smartctl may help you understand whether there's > a physical drive issue, and you can try different USB adapters, ports, > and cables. Actually, one of the difficulties I have with the 3TB disk is reading its SMART information. The disk claims to collect and retain it, but the ?protocol/?device interface (in the container?) prevents my reading it successfully. But I'll ask about that in a separate post sometime. Cheers, David.
Re: Fw: Fw: How long will this take?
On Thu, Jun 11, 2020, 2:05 PM Dan Ritter wrote: > Matthew Campbell wrote: > > The process is complete. The 4 TB drive has been successfully blanked in > less than 40 hours using dd. It got done between 11 pm last night and 12 am > this morning. dd showed an overall average write speed of 28.4 MB/s. It was > never my intention to start a war. My sincere apologies if I have said or > done anything to offend anyone. I prefer to test out new hardware after I > buy it and I prefer to blank new hard drives before partitioning and > writing out new file systems. Thank you to each of you for your assistance. > > > > I think you stated your issue well, responded to queries, and > reported back with the results. Thanks for being part of the > community. > +1 I concur. I do things that some people think of as strange myself. Good luck, Matthew! Kenneth Parker >
Re: Fw: Fw: How long will this take?
Matthew Campbell wrote: > The process is complete. The 4 TB drive has been successfully blanked in less > than 40 hours using dd. It got done between 11 pm last night and 12 am this > morning. dd showed an overall average write speed of 28.4 MB/s. It was never > my intention to start a war. My sincere apologies if I have said or done > anything to offend anyone. I prefer to test out new hardware after I buy it > and I prefer to blank new hard drives before partitioning and writing out new > file systems. Thank you to each of you for your assistance. > I think you stated your issue well, responded to queries, and reported back with the results. Thanks for being part of the community. -dsr-
Re: [OT] How to help the OP? (was: How long will this take?)
Michael Stone (12020-06-11): > I wasn't referring to that statement, I was referring to several other > emails which were apparently being justified by that statement. I have not seen anything like this. Do you have references? > So the rest of your strawman is, I guess, irrelevant. You could have made your statement more explicit. Regards, -- Nicolas George signature.asc Description: PGP signature
Re: [OT] How to help the OP? (was: How long will this take?)
On Thu, Jun 11, 2020 at 10:58:39AM +0200, deloptes wrote: There is nothing unpolite in the mentioned statement: "There is no dilemma at all: we are not paid help-desk technicians." I wasn't referring to that statement, I was referring to several other emails which were apparently being justified by that statement. So the rest of your strawman is, I guess, irrelevant.
Re: [OT] How to help the OP? (was: How long will this take?)
On Thu, Jun 11, 2020 at 10:58:39AM +0200, deloptes wrote: > to...@tuxteam.de wrote: > > >> Now the superior moral leftists came here to judge what is polite and > >> what > > ^^^ > > > > Again, your idea of polite differs from mine by a wide margin. > > With all respect to you tomas this is perfectly OK. Deviations should be > acceptable both right and left, agreed > however I see more often finger pointing > from the left you ever stopped to consider how much of that is due to your own perspective? >and what actually bothers me most is that some people think > they are on this earth to judge over others. > > There is nothing unpolite in the mentioned statement: "There is no dilemma > at all: we are not paid help-desk technicians." You changed the topic mid-flight: that's not the things I quoted from you: "superior moral leftists" is heavy artillery, in my eyes. > I simply refuse to accept this attempt of censoring - it is gone too far (by > wide margin). You can have all the free speech you want; by the same token I am free to say that I sometimes perceive your way to interact as rude. Plus, I am free to choose the point where I'm fed up and quit, what I am doing now. This thread ends for me here. -- tomás signature.asc Description: Digital signature
Re: [OT] How to help the OP? (was: How long will this take?)
to...@tuxteam.de wrote: >> Now the superior moral leftists came here to judge what is polite and >> what > ^^^ > > Again, your idea of polite differs from mine by a wide margin. With all respect to you tomas this is perfectly OK. Deviations should be acceptable both right and left, however I see more often finger pointing from the left and what actually bothers me most is that some people think they are on this earth to judge over others. There is nothing unpolite in the mentioned statement: "There is no dilemma at all: we are not paid help-desk technicians." This is simple truth. There is a big problem in the way it is being criticized and the critics accepted and/or tolerated. The main problem I see is that people do not care how language works, insufficient funding of schools etc. Simply listen at Jordan Peterson. I simply refuse to accept this attempt of censoring - it is gone too far (by wide margin).
Re: [OT] How to help the OP? (was: How long will this take?)
On Thu, Jun 11, 2020 at 07:27:08AM +0200, deloptes wrote: > Michael Stone wrote: > > > It helps create a positive community. Constantly attacking people > > because you think they're in some way wrong does not. > > Who is giving you the right to judge others? This beeing asked politely! OK > I am honestly fed of this. When I read following: ^^^ It seems that the definition of "polite" differ strongly among people :-/ > There is no dilemma at all: we are not paid help-desk technicians. > > I do not think or have a perseption it is unpolite. It is a true statement. > Period. agreed, so far. > Now the superior moral leftists came here to judge what is polite and what ^^^ Again, your idea of polite differs from mine by a wide margin. > not. Please give me a break. g'bye. Cheers -- t signature.asc Description: Digital signature
Re: [OT] How to help the OP? (was: How long will this take?)
Michael Stone wrote: > It helps create a positive community. Constantly attacking people > because you think they're in some way wrong does not. Who is giving you the right to judge others? This beeing asked politely! I am honestly fed of this. When I read following: There is no dilemma at all: we are not paid help-desk technicians. I do not think or have a perseption it is unpolite. It is a true statement. Period. Now the superior moral leftists came here to judge what is polite and what not. Please give me a break. Thank you
Re: Fw: How long will this take?
On 2020-06-10 10:13, Matthew Campbell wrote: I ran a read test first to look for bad sectors on the new hard drive to see if it was damaged by the shipping process. I use the manufacture toolkit to test and verify drives. Now I am writing to each sector, after looking around on the drive to inspect any data written by the manufacturer, to clear everything for the aforementioned reasons. I use the manufacturer toolkit to zero fill and/or secure erase drives. Two files had been placed on the second partition by the manufacturer. The first partition did not have a file system. The manufacturer really screwed up the partitioning process. They started the first partition inside a physical 4096 byte sector. fdisk pointed this out in red text. The drive uses a logical sector size of 512 bytes, but uses a physical sector size of 4096 bytes. I still don't know the rest of the disk geometry. If you have any console sessions from before destroying the factory layout, please post any commands and output that are of interest. David
Re: How long will this take?
On 2020-06-10 07:00, Michael Stone wrote: On Mon, Jun 08, 2020 at 08:22:39PM +, Matthew Campbell wrote: dd if=/dev/zero of=/dev/sdb ibs=4096 count=976754646 This command line gets data in 4k chunks from /dev/zero and then writes them to the disk in 512 byte chunks. That's pretty much the worst possible case for writing to a disk. > # dd if=/dev/zero of=/dev/sdh ibs=4096 count=1 conv=fdatasync > 1+0 records in > 8+0 records out > 4096 bytes (41 MB, 39 MiB) copied, 3.15622 s, 13.0 MB/s Good catch. You want "bs", not "ibs". I'd suggest dd if=/dev/zero of=/dev/sdb bs=64k +1 (I do not recall having a need for 'ibs' or 'obs'.) and I wouldn't bother trying to calculate a count if you're trying to overwrite the entire disk +1 IME performance peaks at 16-64k. Beyond that things don't improve, and can potentially get worse or cause other issues. I've run benchmarks over the years, usually on Linux. I forget where the performance knees are, but do recall that bs=1M has always been in between. I've been getting sustained USB2 disk writes in the low 40MB/s range for more than 15 years. I'd suggest either checking that you're using a reasonable block size or getting a better USB2 adapter. 25MB/s is definitely low. # dd if=/dev/zero of=/dev/sdh bs=64k count=1 conv=fdatasync 1+0 records in 1+0 records out 65536 bytes (655 MB, 625 MiB) copied, 15.1168 s, 43.4 MB/s I have Intel desktop and server motherboards, and Dell laptops and one server. I believe they all have Intel USB chips. Looking at a recent imaging script run of dd(1) with bs=1M over USB 2.0 to a modern USB 3.0 external enclosure with a vintage SATA I HDD, the numbers were better than I was remembering: 13997441024 bytes (14 GB, 13 GiB) copied, 387 s, 36.2 MB/s David
Re: [OT] How to help the OP? (was: How long will this take?)
Michael Stone (12020-06-10): > It helps create a positive community. Constantly attacking people because > you think they're in some way wrong does not. Yes. That is true. What is the link with the current sub-discussion? Regards, -- Nicolas George signature.asc Description: PGP signature
Re: How long will this take?
On Wed, Jun 10, 2020 at 4:00 PM Nicolas George wrote: > > Anders Andersson (12020-06-10): > > Because the police raiding my house for dealing drugs is not a > > realistic threat. Looking at my drives for running Tor could be. > > I have tried to explain that your threat assessment is inadequate, you > do not want to listen. Fine, keep wasting your time on your own private > security theater. It would only take a quick google to show that this is something that actually happens. Maybe you want to believe it doesn't, but that doesn't make it less true.
Re: [OT] How to help the OP? (was: How long will this take?)
On Wed, Jun 10, 2020 at 09:51:37PM +0200, Nicolas George wrote: Michael Stone (12020-06-10): Interacting in a polite fashion shouldn't require a paycheck. What does politeness have to do with it? It helps create a positive community. Constantly attacking people because you think they're in some way wrong does not.
Re: [OT] How to help the OP? (was: How long will this take?)
Michael Stone (12020-06-10): > Interacting in a polite fashion shouldn't require a paycheck. What does politeness have to do with it? We can very politely explain to somebody that their question makes no sense. Regards, -- Nicolas George signature.asc Description: PGP signature
Re: [OT] How to help the OP? (was: How long will this take?)
On Wed, Jun 10, 2020 at 09:24:08PM +0200, Nicolas George wrote: l0f...@tuta.io (12020-06-10): It's the usual dilemma on mailing-lists/forums: * Are the respondents supposed to answer the OP question directly without hindsight (potential XY problem sometimes - http://xyproblem.info/)? * Or are they expected to put the issue into perspective and challenge it? I know it can be frustrating because this is not always the answer we as OP want to read but it can be really instructive as we understand we were on a wrong track. There is no dilemma at all: we are not paid help-desk technicians. Interacting in a polite fashion shouldn't require a paycheck.
Re: [OT] How to help the OP? (was: How long will this take?)
On Wed, Jun 10, 2020 at 09:20:50PM +0200, l0f...@tuta.io wrote: * Are the respondents supposed to answer the OP question directly without hindsight (potential XY problem sometimes - http://xyproblem.info/)? * Or are they expected to put the issue into perspective and challenge it? I know it can be frustrating because this is not always the answer we as OP want to read but it can be really instructive as we understand we were on a wrong track. I'm still divided on this subject, maybe we need both depending on the context/OP... In general, I think that once someone's been asked a question multiple times, especially when that question isn't really germane, it's just rude to keep insisting on an answer. JMO.
Re: [OT] How to help the OP? (was: How long will this take?)
l0f...@tuta.io (12020-06-10): > It's the usual dilemma on mailing-lists/forums: > * Are the respondents supposed to answer the OP question directly without > hindsight (potential XY problem sometimes - http://xyproblem.info/)? > * Or are they expected to put the issue into perspective and challenge it? I > know it can be frustrating because this is not always the answer we as OP > want to read but it can be really instructive as we understand we were on a > wrong track. There is no dilemma at all: we are not paid help-desk technicians. Regards, -- Nicolas George signature.asc Description: PGP signature
[OT] How to help the OP? (was: How long will this take?)
Hi, 10 juin 2020 à 20:51 de mst...@debian.org: > On Wed, Jun 10, 2020 at 12:02:13PM -0500, David Wright wrote: > >> Both l0f4r0 and I have asked why the OP >> is zeroing the drive, but no reply yet. Perhaps you can suggest an >> answer. >> > I don't really care why the OP is doing it. I can think of several > possibilities, but I don't see any need to argue with him over it. At some > point it's reasonable to simply accept that someone is trying to do something > and either help or not. > It's the usual dilemma on mailing-lists/forums: * Are the respondents supposed to answer the OP question directly without hindsight (potential XY problem sometimes - http://xyproblem.info/)? * Or are they expected to put the issue into perspective and challenge it? I know it can be frustrating because this is not always the answer we as OP want to read but it can be really instructive as we understand we were on a wrong track. I'm still divided on this subject, maybe we need both depending on the context/OP... Best regards, l0f4r0
Re: How long will this take?
On Wed, Jun 10, 2020 at 12:02:13PM -0500, David Wright wrote: I tried to make clear that my use case differed from that of the OP, in case you missed that. Just before lockdown (=lockout). I borrowed an AIO computer and, to make room, returned a 2006 vintage tower that would no longer pass its POST. I used /dev/zero to erase all the information from the disk as there was little point in trying to put Windows XP (licensed to a dead computer) back onto it. Quick, easy, and quick to check with od. Both l0f4r0 and I have asked why the OP is zeroing the drive, but no reply yet. Perhaps you can suggest an answer. I don't really care why the OP is doing it. I can think of several possibilities, but I don't see any need to argue with him over it. At some point it's reasonable to simply accept that someone is trying to do something and either help or not. My use case for badblocks was closer to that of the OP, but still different. Firstly, the disk contained personal data from unencrypted use in the past. Secondly, I was intending to use it encrypted (as mentioned) and prefer no high-watermark. Thirdly, because of its age (2011), I was interested in seeing how well it performed. I have no idea whether the disk is "modern" in the sense you used, as I don't follow the technology like some people on this list evidently do. Fourthly, I don't make a habit of throwing away 2TB disks. badblocks isn't particularly useful for achieving any of those goals vs just writing zeros. "modern" in this context means anything since probably the mid 90s but my memory is a bit fuzzy on the exact dates. certainly anything since the turn of the century. But, as you know about these things, a few questions: . How does badblocks do its job in readonly mode, given that it doesn't know what any block's content ought to be. you have to write the test data ahead of time . Why might the OP run badblocks, particularly non-destructively (as if to preserve something), and *then* zero the drive. the only person I saw mention badblocks in this thread was you, but I guess I might have missed it . What's the easiest way of finding out about "consistent bad (not remappable) sectors" on a drive, as I soon will have to repeat this result (if not by this exact process) with a 3TB disk of 2013 vintage. (The good news: it has a USB3 connection.) you'll get a bunch of errors while writing, and probably the drive will drop offline. you can use smartctl in the smartmontools package to see the status of retries & remapped sectors and get a health report on the drive, which you can use to decide whether to keep the drive in service even if it is currently working. (as a drive ages it will often record an increasing number of correctable errors, which typically will result in failure in the not-distant future.) a confounding factor is that you might also get write errors and dropped disk if there's a USB issue, separate from whether the drive is working properly. smartctl may help you understand whether there's a physical drive issue, and you can try different USB adapters, ports, and cables.
Re: How long will this take?
On Wed 10 Jun 2020 at 10:14:02 (-0400), Michael Stone wrote: > On Mon, Jun 08, 2020 at 10:01:13PM -0500, David Wright wrote: > > On Mon 08 Jun 2020 at 20:22:39 (+), Matthew Campbell wrote: > > > I bought a new 4 terrabyte hard drive that is connected with a > > > USB cable using USB2. It took about 32 hours to read every > > > sector on the drive to look for bad sectors. > > > > I recently ran > > > > # badblocks -c 1024 -s -w -t random -v /dev/sdz > > > > on a 2TB disk with a USB2 connection. The whole process, writing and > > checking, took 33⅓ hours. (The disk now holds an encrypted ext4 filesystem.) > > Yes, it's a slower process than just writing zeros. A modern drive > will verify writes as they're made. badblocks is basically a relic of > another age primarily intended to give a list of bad sectors to avoid > when making a filesystem. Once upon a time, hard drives actually had a > handwritten label on the top listing any bad sectors identified at the > factory so you could avoid them. They don't have that anymore. If any > modern hard drive has consistent bad (not remappable) sectors it > should just be thrown away, because that means it is so far gone that > it no longer has the ability to internally map bad sectors to reserved > good sectors. > > > > I started blanking the sectors using /dev/zero last Friday > > > night. It still isn't done. Is there a way I can find out how > > > much data a particular process has written to the disk? I'm > > > using Debian 10.4. > > > I'm not sure why you'd do that. I've only zeroed disks to erase them > > before I return them to the owner. (They're inside loaned computers.) > > Because it accomplishes what your badblocks run does, in less than > half the time. :) I tried to make clear that my use case differed from that of the OP, in case you missed that. Just before lockdown (=lockout). I borrowed an AIO computer and, to make room, returned a 2006 vintage tower that would no longer pass its POST. I used /dev/zero to erase all the information from the disk as there was little point in trying to put Windows XP (licensed to a dead computer) back onto it. Quick, easy, and quick to check with od. Both l0f4r0 and I have asked why the OP is zeroing the drive, but no reply yet. Perhaps you can suggest an answer. My use case for badblocks was closer to that of the OP, but still different. Firstly, the disk contained personal data from unencrypted use in the past. Secondly, I was intending to use it encrypted (as mentioned) and prefer no high-watermark. Thirdly, because of its age (2011), I was interested in seeing how well it performed. I have no idea whether the disk is "modern" in the sense you used, as I don't follow the technology like some people on this list evidently do. Fourthly, I don't make a habit of throwing away 2TB disks. But, as you know about these things, a few questions: . How does badblocks do its job in readonly mode, given that it doesn't know what any block's content ought to be. . Why might the OP run badblocks, particularly non-destructively (as if to preserve something), and *then* zero the drive. . What's the easiest way of finding out about "consistent bad (not remappable) sectors" on a drive, as I soon will have to repeat this result (if not by this exact process) with a 3TB disk of 2013 vintage. (The good news: it has a USB3 connection.) Cheers, David.
Fw: How long will this take?
I wipe a new drive for two reasons. 1) I like to make sure it's nice and clean which makes me comfortable. It bothers me if I don't. 2) I can make a reasonable determination about which sectors were used when setting up the partition table(s) and any boot loaders so I know which sectors were used and need to be backed up. Yes, I'm actually going to use dd to back up that space outside of any partition and store it elsewhere so I can restore it later if it is needed. I prefer to be safe rather than sorry. In today's cyberspace you just can't run without a backup. It's not a question of if you will be attacked, but when and how. I refuse to pay a ransom. If you get a used drive and you are curious then sure, you could go snooping around. I remember when we used a VAX running VMS back in college. It would allocate some huge amount of disk space when compiling our C programs, even though it wasn't used or cleared. I found old deleted emails from other users in that slack space. Yes, they were in plain text. We later found an option to use when compiling our programs to prevent that so our programs didn't have to use as much space which was limited by a disk quota. name=Matthew%20Campbell=trenix25%40pm.me Original Message On Jun 10, 2020, 6:58 AM, Anders Andersson wrote: > On Wed, Jun 10, 2020 at 3:33 PM Nicolas George wrote: >> >> Anders Andersson (12020-06-10): >> > Except wiping a disk is trivial. Just start the job and come back >> > later to a clean disk. It's not like you have to wipe it by hand. I do >> > it routinely before I put a disk to use that's going to be used for a >> > couple of years. >> >> There is no "except" about: define your threat model; if it requires >> wiping, wipe. If it does not, wiping is just a waste of time, little or >> lots, still a waste. And it is a waste of power too. >> >> There are many things that are trivial to do with a hard drive and could >> benefit security in far-fetched scenarios. Did you wipe the possible >> traces of cocaine? Did you weight it to check it matches the specs? Did >> you take pictures of all angles? All these and many others are trivial. >> Why one but not the others? > > Because the police raiding my house for dealing drugs is not a > realistic threat. Looking at my drives for running Tor could be.
Re: How long will this take?
On Wed, Jun 10, 2020 at 05:53:17PM +0300, Andrei POPESCU wrote: On Mi, 10 iun 20, 10:00:48, Michael Stone wrote: IME performance peaks at 16-64k. Beyond that things don't improve, and can potentially get worse or cause other issues. Even so, bs=1M is easy to remember and type ;) I don't find it all that hard to remember 64k but YMMV.
Re: How long will this take?
On Mi, 10 iun 20, 10:00:48, Michael Stone wrote: > > IME performance peaks at 16-64k. Beyond that things don't improve, and can > potentially get worse or cause other issues. Even so, bs=1M is easy to remember and type ;) Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser signature.asc Description: PGP signature
Re: How long will this take?
On Mon, Jun 08, 2020 at 10:01:13PM -0500, David Wright wrote: On Mon 08 Jun 2020 at 20:22:39 (+), Matthew Campbell wrote: I bought a new 4 terrabyte hard drive that is connected with a USB cable using USB2. It took about 32 hours to read every sector on the drive to look for bad sectors. I recently ran # badblocks -c 1024 -s -w -t random -v /dev/sdz on a 2TB disk with a USB2 connection. The whole process, writing and checking, took 33⅓ hours. (The disk now holds an encrypted ext4 filesystem.) Yes, it's a slower process than just writing zeros. A modern drive will verify writes as they're made. badblocks is basically a relic of another age primarily intended to give a list of bad sectors to avoid when making a filesystem. Once upon a time, hard drives actually had a handwritten label on the top listing any bad sectors identified at the factory so you could avoid them. They don't have that anymore. If any modern hard drive has consistent bad (not remappable) sectors it should just be thrown away, because that means it is so far gone that it no longer has the ability to internally map bad sectors to reserved good sectors. I started blanking the sectors using /dev/zero last Friday night. It still isn't done. Is there a way I can find out how much data a particular process has written to the disk? I'm using Debian 10.4. I'm not sure why you'd do that. I've only zeroed disks to erase them before I return them to the owner. (They're inside loaned computers.) Because it accomplishes what your badblocks run does, in less than half the time. :)
Re: How long will this take?
On Mon, Jun 08, 2020 at 08:22:39PM +, Matthew Campbell wrote: I bought a new 4 terrabyte hard drive that is connected with a USB cable using USB2. It took about 32 hours to read every sector on the drive to look for bad sectors. I started blanking the sectors using /dev/zero last Friday night. It still isn't done. Is there a way I can find out how much data a particular process has written to the disk? I'm using Debian 10.4. dd if=/dev/zero of=/dev/sdb ibs=4096 count=976754646 This command line gets data in 4k chunks from /dev/zero and then writes them to the disk in 512 byte chunks. That's pretty much the worst possible case for writing to a disk. You want "bs", not "ibs". I'd suggest dd if=/dev/zero of=/dev/sdb bs=64k and I wouldn't bother trying to calculate a count if you're trying to overwrite the entire disk (any human is likely to screw up the math and there's no actual benefit). IME performance peaks at 16-64k. Beyond that things don't improve, and can potentially get worse or cause other issues. On Mon, Jun 08, 2020 at 04:33:29PM -0400, Dan Ritter wrote: Matthew Campbell wrote: I bought a new 4 terrabyte hard drive that is connected with a USB cable using USB2. It took about 32 hours to read every sector on the drive to look for bad sectors. I started blanking the sectors using /dev/zero last Friday night. It still isn't done. Is there a way I can find out how much data a particular process has written to the disk? I'm using Debian 10.4. dd if=/dev/zero of=/dev/sdb ibs=4096 count=976754646 USB2 disks are good for about 25MB/s. I've been getting sustained USB2 disk writes in the low 40MB/s range for more than 15 years. I'd suggest either checking that you're using a reasonable block size or getting a better USB2 adapter. 25MB/s is definitely low. That said, these days you'd be much better off with USB3 because any modern disk is going to bottleneck on USB2. On Mon, Jun 08, 2020 at 06:02:46PM -0400, Dan Ritter wrote: deloptes wrote: Dan Ritter wrote: > USB2 disks are good for about 25MB/s. > Where do you have those numbers? USB 2.0 standard can theoretically transfer data at a very high 480 megabits per second (mbps), or 60 megabytes per second (MBps) [for example in wikipedia). Yes, that's the theory. In years of running several USB 2.0 attached disks, I found that they were actually good for about 25MB/s long-term. Bursts to 37MB/s were not uncommon. # dd if=/dev/zero of=/dev/sdh bs=64k count=1 conv=fdatasync 1+0 records in 1+0 records out 65536 bytes (655 MB, 625 MiB) copied, 15.1168 s, 43.4 MB/s The OP's 4K writes will be particularly badly performing. OP was doing 512 byte writes. (same adapter/disk) # dd if=/dev/zero of=/dev/sdh ibs=4096 count=1 conv=fdatasync 1+0 records in 8+0 records out 4096 bytes (41 MB, 39 MiB) copied, 3.15622 s, 13.0 MB/s I didn't have the patience to write the same amount of data. :D
Re: How long will this take?
Anders Andersson (12020-06-10): > Because the police raiding my house for dealing drugs is not a > realistic threat. Looking at my drives for running Tor could be. I have tried to explain that your threat assessment is inadequate, you do not want to listen. Fine, keep wasting your time on your own private security theater. Regards, -- Nicolas George signature.asc Description: PGP signature
Re: How long will this take?
On Wed, Jun 10, 2020 at 3:33 PM Nicolas George wrote: > > Anders Andersson (12020-06-10): > > Except wiping a disk is trivial. Just start the job and come back > > later to a clean disk. It's not like you have to wipe it by hand. I do > > it routinely before I put a disk to use that's going to be used for a > > couple of years. > > There is no "except" about: define your threat model; if it requires > wiping, wipe. If it does not, wiping is just a waste of time, little or > lots, still a waste. And it is a waste of power too. > > There are many things that are trivial to do with a hard drive and could > benefit security in far-fetched scenarios. Did you wipe the possible > traces of cocaine? Did you weight it to check it matches the specs? Did > you take pictures of all angles? All these and many others are trivial. > Why one but not the others? Because the police raiding my house for dealing drugs is not a realistic threat. Looking at my drives for running Tor could be.
Re: How long will this take?
Anders Andersson (12020-06-10): > Except wiping a disk is trivial. Just start the job and come back > later to a clean disk. It's not like you have to wipe it by hand. I do > it routinely before I put a disk to use that's going to be used for a > couple of years. There is no "except" about: define your threat model; if it requires wiping, wipe. If it does not, wiping is just a waste of time, little or lots, still a waste. And it is a waste of power too. There are many things that are trivial to do with a hard drive and could benefit security in far-fetched scenarios. Did you wipe the possible traces of cocaine? Did you weight it to check it matches the specs? Did you take pictures of all angles? All these and many others are trivial. Why one but not the others? Regards, -- Nicolas George signature.asc Description: PGP signature
Re: How long will this take?
On Wed, Jun 10, 2020 at 1:14 PM Nicolas George wrote: > > Anders Andersson (12020-06-10): > > Too bad if you end up in a routine police investigation and they find > > child pornography when scanning the disks for deleted files. > > > > "Must have been the previous owner" is a valid defense, but I'd rather > > not end up having to use it. > > Ah, but maybe the previous owner had discovered a cheap cure for > covid-19 and big pharma had them silenced. You would be wiping the last > traces of their research! > > Seriously, first we were talking about hard drives straight from the > factory in China, making the thread… industrial espionage, I suppose? > And now we are talking about child pornography found in an unrelated > seizure. > > So, for that to be relevant, you would need that all the following > conditions to be met: > > - the previous owner had child pornography on this disk; > > - unencrypted; > > - they gave it away their disk in a way that makes it reusable; > > - without wiping it themselves; > > - cops show up at your door and take the drive to examine it; > > - they do it before regular use has wiped it. > > That is a fine Drake equation you got here, but maybe not a rational > justification for spending days wiping a drive. > > For any security measure, it is easy to find afterwards a far-fetched > scenario where it makes a difference. But that is how TV writers work, > not security. For security, we must first define the attack model, and > then search for defense. Otherwise we end up barricading the back door > while the key to the front door is still under the mat. Except wiping a disk is trivial. Just start the job and come back later to a clean disk. It's not like you have to wipe it by hand. I do it routinely before I put a disk to use that's going to be used for a couple of years.
Re: How long will this take?
Anders Andersson (12020-06-10): > Too bad if you end up in a routine police investigation and they find > child pornography when scanning the disks for deleted files. > > "Must have been the previous owner" is a valid defense, but I'd rather > not end up having to use it. Ah, but maybe the previous owner had discovered a cheap cure for covid-19 and big pharma had them silenced. You would be wiping the last traces of their research! Seriously, first we were talking about hard drives straight from the factory in China, making the thread… industrial espionage, I suppose? And now we are talking about child pornography found in an unrelated seizure. So, for that to be relevant, you would need that all the following conditions to be met: - the previous owner had child pornography on this disk; - unencrypted; - they gave it away their disk in a way that makes it reusable; - without wiping it themselves; - cops show up at your door and take the drive to examine it; - they do it before regular use has wiped it. That is a fine Drake equation you got here, but maybe not a rational justification for spending days wiping a drive. For any security measure, it is easy to find afterwards a far-fetched scenario where it makes a difference. But that is how TV writers work, not security. For security, we must first define the attack model, and then search for defense. Otherwise we end up barricading the back door while the key to the front door is still under the mat. Regards, -- Nicolas George signature.asc Description: PGP signature
Re: How long will this take?
On Tue, Jun 9, 2020 at 8:28 PM Nicolas George wrote: > > Jude DaShiell (12020-06-09): > > High security operations do this routinely. They properly don't trust > > parts are as labeled from manufacturers especially manufacturers that > > send any of their stuff or get any of their stuff from China. > > There is no trust to have. The previous contents would be overwritten on > the first actual write of a file. > > And if the filesystem reads a sector that has never been written, that's > a serious bug in the operating system. Too bad if you end up in a routine police investigation and they find child pornography when scanning the disks for deleted files. "Must have been the previous owner" is a valid defense, but I'd rather not end up having to use it.
Re: How long will this take?
On 6/9/2020 5:39 AM, Nicolas George wrote: How do you add "status=progress" to a process that has already been running for three days? You can't, of course. I was merely suggesting using this in future invocations. -- Chris Howie http://www.chrishowie.com http://en.wikipedia.org/wiki/User:Crazycomputers If you correspond with me on a regular basis, please read this document: http://www.chrishowie.com/email-preferences/ PGP fingerprint: 2B7A B280 8B12 21CC 260A DF65 6FCE 505A CF83 38F5 IMPORTANT INFORMATION/DISCLAIMER This document should be read only by those persons to whom it is addressed. If you have received this message it was obviously addressed to you and therefore you can read it. Additionally, by sending an email to ANY of my addresses or to ANY mailing lists to which I am subscribed, whether intentionally or accidentally, you are agreeing that I am "the intended recipient," and that I may do whatever I wish with the contents of any message received from you, unless a pre-existing agreement prohibits me from so doing. This overrides any disclaimer or statement of confidentiality that may be included on your message.
Re: How long will this take?
Jude DaShiell (12020-06-09): > To search disk drives. Are you still talking about binary search? To search disk drives for what? Binary search is for sorted data. There is nothing sorted on a hard drive. And binary search is when random access is fast; on mechanical hard drives, random access is much slower than sequential access. Regards, -- Nicolas George signature.asc Description: PGP signature
Re: How long will this take?
To search disk drives. On Tue, 9 Jun 2020, Nicolas George wrote: > Date: Tue, 9 Jun 2020 14:27:46 > From: Nicolas George > Reply-To: debian-user@lists.debian.org > To: Jude DaShiell > Cc: l0f...@tuta.io, Debian User > Subject: Re: How long will this take? > > Jude DaShiell (12020-06-09): > > High security operations do this routinely. They properly don't trust > > parts are as labeled from manufacturers especially manufacturers that > > send any of their stuff or get any of their stuff from China. > > There is no trust to have. The previous contents would be overwritten on > the first actual write of a file. > > And if the filesystem reads a sector that has never been written, that's > a serious bug in the operating system. > > > I'm thinking of the binary search method and am wondering if disk > > operations of all sorts could be speeded up using it rather than > > sequential searches. Or is binary already used now? > > To search what? > > Regards, > > --
Re: How long will this take?
Jude DaShiell (12020-06-09): > High security operations do this routinely. They properly don't trust > parts are as labeled from manufacturers especially manufacturers that > send any of their stuff or get any of their stuff from China. There is no trust to have. The previous contents would be overwritten on the first actual write of a file. And if the filesystem reads a sector that has never been written, that's a serious bug in the operating system. > I'm thinking of the binary search method and am wondering if disk > operations of all sorts could be speeded up using it rather than > sequential searches. Or is binary already used now? To search what? Regards, -- Nicolas George signature.asc Description: PGP signature
Re: How long will this take?
High security operations do this routinely. They properly don't trust parts are as labeled from manufacturers especially manufacturers that send any of their stuff or get any of their stuff from China. I'm thinking of the binary search method and am wondering if disk operations of all sorts could be speeded up using it rather than sequential searches. Or is binary already used now? On Tue, 9 Jun 2020, l0f...@tuta.io wrote: > Date: Tue, 9 Jun 2020 14:08:34 > From: l0f...@tuta.io > To: Debian User > Subject: Re: How long will this take? > Resent-Date: Tue, 9 Jun 2020 18:08:47 + (UTC) > Resent-From: debian-user@lists.debian.org > > Hi, > > 8 juin 2020 ? 22:22 de treni...@pm.me: > > > I bought a new 4 terrabyte hard drive that is connected with a USB cable > > using USB2. It took about 32 hours to read every sector on the drive to > > look for bad sectors. I started blanking the sectors using /dev/zero last > > Friday night. > > > Out of curiosity, what is the purpose to wipe a brand new HDD? > Wouldn't formatting (or GPT overwrite) be sufficient? > > 9 juin 2020 ? 08:59 de dpchr...@holgerdanske.com: > > > Also as others have stated, writing zeros to an SSD may wear it out > > prematurely (depends upon internals of SSD). The best approach is to do a > > "secure erase". > > > It seems to be a hard drive here ;) > > Rather than wiping storage devices with GNU/Linux userland tools, your best > > bet is to use the manufacturer's diagnostic utility. In the ideal case, > > the utility sends a command to the drive controller and everything gets > > done internally at maximum speed. I prefer the bootable "Live" tools, if > > available. Each manufacturer has their own toolkit. Get the one for your > > drive brand. For example, SeaTools Bootable: > > > > https://www.seagate.com/support/downloads/seatools/ > > > Even more true for an SSD (and yet, I'm not sure we can say "secure" for sure > as those utilities are generally proprietary so we cannot verify what they do > exactly). > > Best regards, > l0f4r0 > > --
Re: How long will this take?
Hi, 8 juin 2020 à 22:22 de treni...@pm.me: > I bought a new 4 terrabyte hard drive that is connected with a USB cable > using USB2. It took about 32 hours to read every sector on the drive to look > for bad sectors. I started blanking the sectors using /dev/zero last Friday > night. > Out of curiosity, what is the purpose to wipe a brand new HDD? Wouldn't formatting (or GPT overwrite) be sufficient? 9 juin 2020 à 08:59 de dpchr...@holgerdanske.com: > Also as others have stated, writing zeros to an SSD may wear it out > prematurely (depends upon internals of SSD). The best approach is to do a > "secure erase". > It seems to be a hard drive here ;) > Rather than wiping storage devices with GNU/Linux userland tools, your best > bet is to use the manufacturer's diagnostic utility. In the ideal case, the > utility sends a command to the drive controller and everything gets done > internally at maximum speed. I prefer the bootable "Live" tools, if > available. Each manufacturer has their own toolkit. Get the one for your > drive brand. For example, SeaTools Bootable: > > https://www.seagate.com/support/downloads/seatools/ > Even more true for an SSD (and yet, I'm not sure we can say "secure" for sure as those utilities are generally proprietary so we cannot verify what they do exactly). Best regards, l0f4r0
Fw: How long will this take?
I have started the process over from the beginning. It seems to respond well to obs=4M as the write speed has gone from 3.7 MB/s to 28.2 MB/s, which is about the same as where it was with obs=1M. It should take a couple of days to complete this write process. The WD drive that I have, which is connected to a separate USB2 port, showed a write speed of 27.5 Mb/s. I am using: dd if=/dev/zero of=/dev/sdb obs=4M status=progress The read speed was faster, of course. name=Matthew%20Campbell=trenix25%40pm.me Original Message On Jun 9, 2020, 2:39 AM, Nicolas George wrote: > Christopher David Howie (12020-06-08): > I'd suggest simply adding > "status=progress" which gives you a summary every > second including bytes > written, elapsed time, and average transfer rate. How do you add > "status=progress" to a process that has already been running for three days? > Regards, -- Nicolas George
Re: How long will this take?
Christopher David Howie (12020-06-08): > I'd suggest simply adding "status=progress" which gives you a summary every > second including bytes written, elapsed time, and average transfer rate. How do you add "status=progress" to a process that has already been running for three days? Regards, -- Nicolas George signature.asc Description: PGP signature
Re: How long will this take?
On Lu, 08 iun 20, 20:09:54, Jude DaShiell wrote: > > From: Dan Ritter > People are suing Western Digital for sneaking those SMR disks into their > supply chain. They're supposed to be red in color if what I read in the > news is correct. > > > > https://www.ixsystems.com/community/resources/list-of-known-smr-drives.141/ According to that list it's only specific model numbers of WD Red and Blue. Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser signature.asc Description: PGP signature
Re: How long will this take?
On 2020-06-08 13:22, Matthew Campbell wrote: I bought a new 4 terrabyte hard drive that is connected with a USB cable using USB2. It took about 32 hours to read every sector on the drive to look for bad sectors. I started blanking the sectors using /dev/zero last Friday night. It still isn't done. Is there a way I can find out how much data a particular process has written to the disk? I'm using Debian 10.4. dd if=/dev/zero of=/dev/sdb ibs=4096 count=976754646 name=Matthew%20Campbell=trenix25%40pm.me Install 'nmon'. Then start a terminal and run 'nmon'. Press 'd' to display the disk monitoring screen. This will show read throughput, write throughput, and percent utilization. Alternatively, if you are using the Xfce desktop, add a Disk Performance Monitor applet to the panel and configure it for the correct device node /dev/sdX: Device /dev/sdX unchecked Label sdX Update interval(s) 1.000 Monitor Busy time checked Combine Read/Write data Then hover your mouse pointer over the applet and it will show you read, write, and total statistics for both throughput and for busy time. USB 2.0 ports have a maximum write speed around 25 MB/s. eSATA ports (version 1) get much closer to their theoretical maximum of 150 MB/s. USB 3.0 beats them both. Of course, you must have a fast drive and a fast program. When using dd(1) to write blocks to a raw drive, use a block size of 1M (e.g. 1 Mibabyte). As others have stated, a small block size of 4K will significantly reduce throughput due to I/O overhead. Also as others have stated, writing zeros to an SSD may wear it out prematurely (depends upon internals of SSD). The best approach is to do a "secure erase". Rather than wiping storage devices with GNU/Linux userland tools, your best bet is to use the manufacturer's diagnostic utility. In the ideal case, the utility sends a command to the drive controller and everything gets done internally at maximum speed. I prefer the bootable "Live" tools, if available. Each manufacturer has their own toolkit. Get the one for your drive brand. For example, SeaTools Bootable: https://www.seagate.com/support/downloads/seatools/ David
Re: Fw: How long will this take?
This means that reads and writes should be on 4KiB boundaries, and writes should be multiples of 4KiB, for optimal performance. As long as those criteria are met, there's no harm and some real benefits of reading and writing larger blocks than the minimum. One example benefit, among several possible, is that the OS overhead of one single 1MiB write will be much less than 256 individual 4KiB writes. (number of system calls performed; overhead inside the OS of merging successive writes to optimize the size of actual disk transfers, etc...) Hope that helps! Rick On Mon, Jun 8, 2020, at 9:47 PM, Matthew Campbell wrote: > fdisk said the minimum and optimal access size for my hard drive was 4096 > bytes. > > name=Matthew%20Campbell=trenix25%40pm.me > > > > > > Original Message > On Jun 8, 2020, 7:42 PM, Dan Ritter < d...@randomstring.org> wrote: >> >> Jude DaShiell wrote: >> > Does any optimal formula exist based on hard drive size which minimizes >> > time needed for checking and blanking hard drives in connection with the >> > block size value? >> If the disk firmware offers it, a SMART long read/verify test >> should be close to optimal. Consult smartctl and the disk manufacturer >> for details. >> For conventional spinning hard disks, the optimal write size would be >> a complete cylinder at a time. That varies across the radius of the disk, >> and may not be made available to the OS. >> In lieue of knowing that, writes which are reasonable integer >> multiples of the sector size are very good. 1 MB is probably >> good for most drives. >> For SMR spinning disks,the optimal write size is one complete >> write zone. I've heard that this is standardizing at 256MB, but >> I would want to confirm with the manufacturer. There are a lot >> of interactions with PMR caches. >> For SSD, writing wears out the storage mechanism. A write-all >> test won't test reliability; flaws will be detected and remapped >> without letting the host know. >> -dsr-
Re: How long will this take?
On 6/8/2020 11:01 PM, David Wright wrote: I, too, determine progress with # kill -USR1 I'd suggest simply adding "status=progress" which gives you a summary every second including bytes written, elapsed time, and average transfer rate. -- Chris Howie http://www.chrishowie.com http://en.wikipedia.org/wiki/User:Crazycomputers If you correspond with me on a regular basis, please read this document: http://www.chrishowie.com/email-preferences/ PGP fingerprint: 2B7A B280 8B12 21CC 260A DF65 6FCE 505A CF83 38F5 IMPORTANT INFORMATION/DISCLAIMER This document should be read only by those persons to whom it is addressed. If you have received this message it was obviously addressed to you and therefore you can read it. Additionally, by sending an email to ANY of my addresses or to ANY mailing lists to which I am subscribed, whether intentionally or accidentally, you are agreeing that I am "the intended recipient," and that I may do whatever I wish with the contents of any message received from you, unless a pre-existing agreement prohibits me from so doing. This overrides any disclaimer or statement of confidentiality that may be included on your message.
Re: How long will this take?
On Mon 08 Jun 2020 at 20:22:39 (+), Matthew Campbell wrote: > I bought a new 4 terrabyte hard drive that is connected with a USB cable > using USB2. It took about 32 hours to read every sector on the drive to look > for bad sectors. I recently ran # badblocks -c 1024 -s -w -t random -v /dev/sdz on a 2TB disk with a USB2 connection. The whole process, writing and checking, took 33⅓ hours. (The disk now holds an encrypted ext4 filesystem.) > I started blanking the sectors using /dev/zero last Friday night. It still > isn't done. Is there a way I can find out how much data a particular process > has written to the disk? I'm using Debian 10.4. I'm not sure why you'd do that. I've only zeroed disks to erase them before I return them to the owner. (They're inside loaned computers.) > dd if=/dev/zero of=/dev/sdb ibs=4096 count=976754646 … And I'd be using bs=1M and no count. I, too, determine progress with # kill -USR1 Cheers, David.
Re: How long will this take?
Jude DaShiell wrote: > Does any optimal formula exist based on hard drive size which minimizes > time needed for checking and blanking hard drives in connection with the > block size value? If the disk firmware offers it, a SMART long read/verify test should be close to optimal. Consult smartctl and the disk manufacturer for details. For conventional spinning hard disks, the optimal write size would be a complete cylinder at a time. That varies across the radius of the disk, and may not be made available to the OS. In lieue of knowing that, writes which are reasonable integer multiples of the sector size are very good. 1 MB is probably good for most drives. For SMR spinning disks,the optimal write size is one complete write zone. I've heard that this is standardizing at 256MB, but I would want to confirm with the manufacturer. There are a lot of interactions with PMR caches. For SSD, writing wears out the storage mechanism. A write-all test won't test reliability; flaws will be detected and remapped without letting the host know. -dsr-
re: How long will this take?
Does any optimal formula exist based on hard drive size which minimizes time needed for checking and blanking hard drives in connection with the block size value? --
Re: How long will this take?
People are suing Western Digital for sneaking those SMR disks into their supply chain. They're supposed to be red in color if what I read in the news is correct. On Mon, 8 Jun 2020, Dan Ritter wrote: > Date: Mon, 8 Jun 2020 16:33:29 > From: Dan Ritter > To: Matthew Campbell > Cc: Debian User Support > Subject: Re: How long will this take? > Resent-Date: Mon, 8 Jun 2020 20:33:43 + (UTC) > Resent-From: debian-user@lists.debian.org > > Matthew Campbell wrote: > > I bought a new 4 terrabyte hard drive that is connected with a USB cable > > using USB2. It took about 32 hours to read every sector on the drive to > > look for bad sectors. I started blanking the sectors using /dev/zero last > > Friday night. It still isn't done. Is there a way I can find out how much > > data a particular process has written to the disk? I'm using Debian 10.4. > > > > dd if=/dev/zero of=/dev/sdb ibs=4096 count=976754646 > > USB2 disks are good for about 25MB/s. > > 4 seconds gets you 100MB. > > 40 seconds gets you 1000MB. > > 4000 * 40 seconds is 16 seconds, so that's not quite two > days. > > Is something wrong? Based on current news reports, I would say > you accidentally purchased an SMR disk. (By accidentally, I mean > that the box didn't say, the ad didn't say, and the manufacturer > might even have lied to you for a while.) > > https://www.ixsystems.com/community/resources/list-of-known-smr-drives.141/ > > Is it one of those? > > If so, return it. Tell the store that it's an unlabelled SMR > drive. They'll take it back. > > -dsr- > > --
Re: How long will this take?
deloptes wrote: > Dan Ritter wrote: > > > USB2 disks are good for about 25MB/s. > > > > Where do you have those numbers? > > USB 2.0 standard can theoretically transfer data at a very high 480 megabits > per second (mbps), or 60 megabytes per second (MBps) [for example in > wikipedia). Yes, that's the theory. In years of running several USB 2.0 attached disks, I found that they were actually good for about 25MB/s long-term. Bursts to 37MB/s were not uncommon. The USB mass storage protocol forces a queue depth of 1: one request, one response, nothing else until it's done. The OP's 4K writes will be particularly badly performing. -dsr-
Re: How long will this take?
Matthew Campbell (12020-06-08): > Is that in bytes? You can compare with the stats presented by USR1 to be sure. > stdin and stderr both show a position of zero. You can look in /proc/$PID/fd to see where the various fd points. I guess 0 will point to /dev/zero, 1 to the hard drive and 2 to the tty. Regards, -- Nicolas George signature.asc Description: PGP signature
Re: How long will this take?
# cat /proc/24283/fdinfo/1 pos: 877106917376 flags: 011 mnt_id: 21 # Is that in bytes? stdin and stderr both show a position of zero. name=Matthew%20Campbell=trenix25%40pm.me Original Message On Jun 8, 2020, 1:32 PM, Nicolas George wrote: > Matthew Campbell (12020-06-08): > I bought a new 4 terrabyte hard drive that > is connected with a USB > cable using USB2. It took about 32 hours to read > every sector on the > drive to look for bad sectors. I started blanking the > sectors using > /dev/zero last Friday night. It still isn't done. Is there a > way I can > find out how much data a particular process has written to the > disk? > I'm using Debian 10.4. Sending a USR1 signal to a running 'dd' > process makes it print I/O sta‐ tistics to standard error and then resume > copying. Fron dd(1). Also, you can go read /proc/$(pidof dd)/fdinfo, it > contains the information too. Note that it becomes much slower as it nears > the center of the disk. Regards, -- Nicolas George
Re: How long will this take?
Dan Ritter wrote: > USB2 disks are good for about 25MB/s. > Where do you have those numbers? USB 2.0 standard can theoretically transfer data at a very high 480 megabits per second (mbps), or 60 megabytes per second (MBps) [for example in wikipedia). but as you say it is slowing down at some point. It is not working at max anyway - some people say it will not increase above 20MB/s anyway. I suggest add to dd status=progress The LEVEL of information to print to stderr; 'none' suppresses everything but error messages, 'noxfer' suppresses the final transfer sta- tistics, 'progress' shows periodic transfer statistics And honestly I do not think 4TB disks were meant to be used via USB2 - think of using eSATA.
Re: How long will this take?
Matthew Campbell wrote: > I bought a new 4 terrabyte hard drive that is connected with a USB cable > using USB2. It took about 32 hours to read every sector on the drive to look > for bad sectors. I started blanking the sectors using /dev/zero last Friday > night. It still isn't done. Is there a way I can find out how much data a > particular process has written to the disk? I'm using Debian 10.4. > > dd if=/dev/zero of=/dev/sdb ibs=4096 count=976754646 USB2 disks are good for about 25MB/s. 4 seconds gets you 100MB. 40 seconds gets you 1000MB. 4000 * 40 seconds is 16 seconds, so that's not quite two days. Is something wrong? Based on current news reports, I would say you accidentally purchased an SMR disk. (By accidentally, I mean that the box didn't say, the ad didn't say, and the manufacturer might even have lied to you for a while.) https://www.ixsystems.com/community/resources/list-of-known-smr-drives.141/ Is it one of those? If so, return it. Tell the store that it's an unlabelled SMR drive. They'll take it back. -dsr-
Re: How long will this take?
Matthew Campbell (12020-06-08): > I bought a new 4 terrabyte hard drive that is connected with a USB > cable using USB2. It took about 32 hours to read every sector on the > drive to look for bad sectors. I started blanking the sectors using > /dev/zero last Friday night. It still isn't done. Is there a way I can > find out how much data a particular process has written to the disk? > I'm using Debian 10.4. Sending a USR1 signal to a running 'dd' process makes it print I/O sta‐ tistics to standard error and then resume copying. Fron dd(1). Also, you can go read /proc/$(pidof dd)/fdinfo, it contains the information too. Note that it becomes much slower as it nears the center of the disk. Regards, -- Nicolas George signature.asc Description: PGP signature
How long will this take?
I bought a new 4 terrabyte hard drive that is connected with a USB cable using USB2. It took about 32 hours to read every sector on the drive to look for bad sectors. I started blanking the sectors using /dev/zero last Friday night. It still isn't done. Is there a way I can find out how much data a particular process has written to the disk? I'm using Debian 10.4. dd if=/dev/zero of=/dev/sdb ibs=4096 count=976754646 name=Matthew%20Campbell=trenix25%40pm.me
Re: How long should it take to extract 1GIG off Tape?
My Wangtek QIC 150 tape (an old 5110 ES drive) runs at a rate of 5MB/min, or 300MB/h, which should give 1GB in something near/below 4h (if the 250MB-tapes are changed fast enough). The streaming throughput should be limited from the tape, not the host. : I'm just checking but whenever I extract the contents of my mirrored : tapes it seems to take nearly all night to extract. At least 6 hours. : That doesn't seem right to me. Is the kernel configured to work with all : SCSI tape drives in an optimal manner? Is there the same kind of QIC : interpretation problem like under Solaris here? ie. stconf.c? -- Uni Wuppertal, FB Elektrotechnik, Tel/Fax: (0202) 439 - 3009 Dr. Andreas Wehler; [EMAIL PROTECTED] -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word unsubscribe to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED]
