[gentoo-user] Re: [OT] Tools for putting HDD back to new state

[Kai Krakow]

Am Fri, 14 Apr 2017 09:37:09 +0200
schrieb Marc Joliet :

> (Sorry for the late reply, I hope it's still useful to you.)

NP. The links below were interesting.

> On Dienstag, 4. April 2017 00:46:54 CEST Kai Krakow wrote:
> > Am Mon, 3 Apr 2017 16:15:24 -0400
> > 
> > schrieb Rich Freeman :  
> > > On Mon, Apr 3, 2017 at 2:34 PM, Kai Krakow 
> > > 
> > > wrote:  
>  [...]  
> > > 
> > > If it contains data you'd prefer not be recoverable you might
> > > want to use shred or ATA secure erase.  
> > 
> > I wonder if shredding adds any value with the high density of modern
> > drives... Each bit is down to a "few" (*) atoms. It should be pretty
> > difficult, if not impossible, to infer the previous data from it. I
> > think most of the ability to infer the previous data comes from
> > magnetic leakage from the written bit to the neighbor bits. And
> > this is why clever mathematicians created series of alternating bit
> > patterns to distribute this leakage evenly, which is the different
> > algorithms the shredder programs use.
> > 
> > Do you have any insights on that matter? Just curious.  
> 
> For the record, there was some discussion on this on this not too
> long ago [edit: oops, looks like it was almost two years ago now]:
> see the thread "Securely Securely deletion of an HDD" (yes, I
> including my spelling mistake), which you can find online at https://
> archives.gentoo.org/gentoo-user/message/a01e0ad7b07855647a528f1e0324631a
> and
> https://archives.gentoo.org/gentoo-user/message/582fe3c66c7e13de979b656e9db33325.

So you suggest shooting a bullet at the disks? ;-)

You could also use the hammer method:
https://youtu.be/oNcaIQMjbM8?t=2m55s

> > > Shred overwrites the drive with random data using a few passes to
> > > make recovery more difficult.  Some debate whether it actually
> > > adds value.  
> > 
> > For a mere mortal it is already impossible to recover data after
> > writing zeros to it. Shredding is very time consuming and probably
> > not worth the effort if you just want a blank drive and have no
> > critical or security relevant data on it, i.e. you used it for
> > testing.
> > 
> > But while you are at it: Shredding tools should usually do a read
> > check to compare that the data that ought to have been written
> > actually was written, otherwise the whole procedure is pretty
> > pointless. As a side effect, this exposes sector defects.
> > 
> > If you want to do this to pretend data has never been written to the
> > drive, you're probably out of luck anyways: If you'd be able to
> > recover data after a single write of zeros, it should be easily
> > possible to see that the data was shredded with different bit
> > patterns. The S.M.A.R.T counters will add the rest and tell you the
> > power-on hours, maybe even amount of data written, head moves etc.
> > 
> > (*): On an atomic scale, that's still 1 million atoms...  
> 
> I don't think using zeros is enough, certainly not on SSDs that do
> their own compression, I would think.

Well, I don't think that compression and its overhead to be effective
is worth the effort to implement it. I don't think drives do this.
Especially that the bus speed is becoming the bottleneck. Thus to be
effective, data would have to be compressed before transferring over
the bus and uncompressed after. Also deduplication is very unlikely to
be done in firmware. I wouldn't take that as an argument why you want
use random data.

But I think the point here is sector remapping (as pointed out in the
references threads): SSDs do that through the FTL constantly, HDDs do
that upon encountering physical problems on the platter. It absolutely
makes no difference if you put random data or zero data to the disk:
You won't reach the previously mapped sector locations. Secure erase is
probably the only thing you can do here, hoping that it covers all
sectors (also the spare sectors and unmapped sectors).

> And AFAIK using random data
> can still fill the drive at native write speed, so I don't see what
> you gain by avoiding that.  But really, if you haven't already, check
> the primary sources in the thread I mentioned above.

Depends on what's your random source: /dev/random won't generate
entropy fast enough to do this. /dev/urandom could, but actually it's
not that very random because it's generated mathematically. That
somehow defeats the purpose for using as overwrite source. A mixture of
both could do good enough, that's probably where special wiping software
comes in.

Conclusion: If you don't store state secrets, overwriting with zeros
should be enough. If you store data in a high security environment,
you're probably required to physically destroy the disks anyway. If you
mind remapped sectors, you could use secure erase but you don't know
how thoroughly that really works, thus the only other option would be
to physically destroy the disk.

Then there's the option to use full disk encryption right from 

Re: [gentoo-user] Re: [OT] Tools for putting HDD back to new state

[Marc Joliet]

(Sorry for the late reply, I hope it's still useful to you.)

On Dienstag, 4. April 2017 00:46:54 CEST Kai Krakow wrote:
> Am Mon, 3 Apr 2017 16:15:24 -0400
> 
> schrieb Rich Freeman :
> > On Mon, Apr 3, 2017 at 2:34 PM, Kai Krakow 
> > 
> > wrote:
> > > Just dd /dev/zero to the complete device. That purges everything you
> > > need: partition tables, boot sectors, contents:
> > > 
> > > # dd if=/dev/zero of=/dev/sdX
> > 
> > If it contains data you'd prefer not be recoverable you might want to
> > use shred or ATA secure erase.
> 
> I wonder if shredding adds any value with the high density of modern
> drives... Each bit is down to a "few" (*) atoms. It should be pretty
> difficult, if not impossible, to infer the previous data from it. I
> think most of the ability to infer the previous data comes from
> magnetic leakage from the written bit to the neighbor bits. And this is
> why clever mathematicians created series of alternating bit patterns to
> distribute this leakage evenly, which is the different algorithms the
> shredder programs use.
> 
> Do you have any insights on that matter? Just curious.

For the record, there was some discussion on this on this not too long ago 
[edit: oops, looks 
like it was almost two years ago now]: see the thread "Securely Securely 
deletion of an HDD" 
(yes, I including my spelling mistake), which you can find online at https://
archives.gentoo.org/gentoo-user/message/a01e0ad7b07855647a528f1e0324631a and 
https://archives.gentoo.org/gentoo-user/message/582fe3c66c7e13de979b656e9db33325.

> > Shred overwrites the drive with random data using a few passes to make
> > recovery more difficult.  Some debate whether it actually adds value.
> 
> For a mere mortal it is already impossible to recover data after
> writing zeros to it. Shredding is very time consuming and probably not
> worth the effort if you just want a blank drive and have no critical or
> security relevant data on it, i.e. you used it for testing.
> 
> But while you are at it: Shredding tools should usually do a read check
> to compare that the data that ought to have been written actually was
> written, otherwise the whole procedure is pretty pointless. As a side
> effect, this exposes sector defects.
> 
> If you want to do this to pretend data has never been written to the
> drive, you're probably out of luck anyways: If you'd be able to recover
> data after a single write of zeros, it should be easily possible to see
> that the data was shredded with different bit patterns. The S.M.A.R.T
> counters will add the rest and tell you the power-on hours, maybe even
> amount of data written, head moves etc.
> 
> (*): On an atomic scale, that's still 1 million atoms...

I don't think using zeros is enough, certainly not on SSDs that do their own 
compression, I 
would think.  And AFAIK using random data can still fill the drive at native 
write speed, so I 
don't see what you gain by avoiding that.  But really, if you haven't already, 
check the 
primary sources in the thread I mentioned above.

HTH
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup



signature.asc
Description: This is a digitally signed message part.

[gentoo-user] Re: [OT] Tools for putting HDD back to new state

[Nikos Chantziaras]

On 04/04/2017 04:07 PM, Harry Putnam wrote:

I've googled fairly extensively on the subject and did not find a way
described anywhere to return a disk to what is called its raw state.


There's not such thing. When shipping, the disk might contain all 
zero-bytes, or random bytes.




There may even be legal ramifications I suppose along the line of
selling used discs as new after some kind of processing.


Wiping the disk does not change the internal book-keeping data of the 
device. It's stored in its SMART memory, which lists how many hours the 
disk has been used, and whether there's any errors that have been detected.


That data cannot be wiped since it's not on the disk. It's on a chip. 
That data can be viewed with any SMART viewer (like sys-apps/gsmartcontrol).

[gentoo-user] Re: [OT] Tools for putting HDD back to new state

[Harry Putnam]

Mike Gilbert  writes:

[...]

> If you are not worried about securely removing all data and simply
> want to fool fdisk into thinking your drive is empty, use the wipefs
> utility. This will zero-out key bytes like the MBR, partition table,
> filesystem magic numbers, etc.
>
> You'll want to run it once for each partition, and then once for the
> whole device.
>
> wipefs -a /dev/sdx1
> wipefs -a /dev/sdx2
> wipefs -a /dev/sdx

This sounds like more what I had in mind... there is no worry about
making data irrecoverable.  I'll check this out... booting the
hardware with a liveCD of some sort that I know has that tool on it.
SystemrescueCD probabably has it.

Nikos Chantziaras  writes:

[...]

> You can use cfdisk (or another partitioning tool) and delete all partitions.
>
> Then, delete the MBR (Master Boot Record), which is where boot
> managers put themselves. You do that with:
>
>   dd if=/dev/zero of=/dev/your_hard_disk bs=446 count=1

[...]

This may be all I really need. I had considered it to start but had
the notion that it might not be that hard to return a disk to its new
condition ... apparently that is not really all that easy or in this
case ... even necessary.

I've googled fairly extensively on the subject and did not find a way
described anywhere to return a disk to what is called its raw state.
Or, put another way, the state a disk is in why you buy one new.

There may even be legal ramifications I suppose along the line of
selling used discs as new after some kind of processing.

[gentoo-user] Re: [OT] Tools for putting HDD back to new state

[Nikos Chantziaras]

On 04/03/2017 09:11 PM, Harry Putnam wrote:

I probably should know this, but off the top of my head I don't
remember ever running into anything like this.

I'd like to do what ever is done to set a used  disk back to the
state it was in when new... Not sure what that state is, but at least
no evidence of boot manager or fs having been installed.


You can use cfdisk (or another partitioning tool) and delete all partitions.

Then, delete the MBR (Master Boot Record), which is where boot managers 
put themselves. You do that with:


  dd if=/dev/zero of=/dev/your_hard_disk bs=446 count=1

It's not necessary to write zeroes all over the disk. You only need to 
delete the partitions and the boot manager, unless you also want to make 
the old data on the disk irrecoverable instead of it just appearing 
empty out of the box. In that case, following the advise of the other 
posters here and write zeroes all over the disk with dd is a good idea.

[gentoo-user] Re: [OT] Tools for putting HDD back to new state

[Kai Krakow]

Am Mon, 3 Apr 2017 16:15:24 -0400
schrieb Rich Freeman :

> On Mon, Apr 3, 2017 at 2:34 PM, Kai Krakow 
> wrote:
> >
> > Just dd /dev/zero to the complete device. That purges everything you
> > need: partition tables, boot sectors, contents:
> >
> > # dd if=/dev/zero of=/dev/sdX
> >  
> 
> If it contains data you'd prefer not be recoverable you might want to
> use shred or ATA secure erase.

I wonder if shredding adds any value with the high density of modern
drives... Each bit is down to a "few" (*) atoms. It should be pretty
difficult, if not impossible, to infer the previous data from it. I
think most of the ability to infer the previous data comes from
magnetic leakage from the written bit to the neighbor bits. And this is
why clever mathematicians created series of alternating bit patterns to
distribute this leakage evenly, which is the different algorithms the
shredder programs use.

Do you have any insights on that matter? Just curious.

> Shred overwrites the drive with random data using a few passes to make
> recovery more difficult.  Some debate whether it actually adds value.

For a mere mortal it is already impossible to recover data after
writing zeros to it. Shredding is very time consuming and probably not
worth the effort if you just want a blank drive and have no critical or
security relevant data on it, i.e. you used it for testing.

But while you are at it: Shredding tools should usually do a read check
to compare that the data that ought to have been written actually was
written, otherwise the whole procedure is pretty pointless. As a side
effect, this exposes sector defects.

If you want to do this to pretend data has never been written to the
drive, you're probably out of luck anyways: If you'd be able to recover
data after a single write of zeros, it should be easily possible to see
that the data was shredded with different bit patterns. The S.M.A.R.T
counters will add the rest and tell you the power-on hours, maybe even
amount of data written, head moves etc.

(*): On an atomic scale, that's still 1 million atoms...

-- 
Regards,
Kai

Replies to list-only preferred.

Re: [gentoo-user] Re: [OT] Tools for putting HDD back to new state

[Rich Freeman]

On Mon, Apr 3, 2017 at 2:34 PM, Kai Krakow  wrote:
>
> Just dd /dev/zero to the complete device. That purges everything you
> need: partition tables, boot sectors, contents:
>
> # dd if=/dev/zero of=/dev/sdX
>

If it contains data you'd prefer not be recoverable you might want to
use shred or ATA secure erase.

Shred overwrites the drive with random data using a few passes to make
recovery more difficult.  Some debate whether it actually adds value.

Secure erase is a standard command supported by most drives.  It has
the advantage of being MUCH faster, and it also should take care of
things like relocated blocks and such which might not be seen by the
OS.  It has the disadvantage of being a black box that might not
actually work or which might have some kind of NSA back door.
Typically it is implemented by the drive controller encrypting all
your data transparently using a random key in normal operation, and
then the secure erase command tells it to forget the key and generate
a new one.  I suspect that secure erase would probably be the closest
thing to restoring "factory" condition for a drive.

Instructions can be found at:
https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase

Unless I'm in a hurry I tend to do the best of both worlds.  I run
shred, and then I do a secure erase.

And of course another option is to always encrypt your drives all the
time anyway, which means that even if the drive fails and you can't
erase it that your data is secure anyway.

-- 
Rich

[gentoo-user] Re: [OT] Tools for putting HDD back to new state

[Kai Krakow]

Am Mon, 03 Apr 2017 14:11:40 -0400
schrieb Harry Putnam :

> I probably should know this, but off the top of my head I don't
> remember ever running into anything like this.
> 
> I'd like to do what ever is done to set a used  disk back to the
> state it was in when new... Not sure what that state is, but at least
> no evidence of boot manager or fs having been installed.
> 
> This if for something I'm doing on OS openindiana (a solaris offshoot)
> and the disks are for that OS.
> 
> The solaris milieu is somewhat behind linux in development of tools at
> least in my opinion.  That is why I'm asking here.
> 
> I am a gentoo user as well, but expect I may have to boot the solaris
> host with one or another linux boot ISO in order to have the tools
> required. 

Just dd /dev/zero to the complete device. That purges everything you
need: partition tables, boot sectors, contents:

# dd if=/dev/zero of=/dev/sdX

where sdX is the disk to "kill". Be sure not to purge just on partition
by saying sdX1 or so but the complete disk (without number). Then,
before running any partitioning software, reboot, so the partition
cache of the kernel becomes cleared, or run part-probe if it is
installed.

dd should also be available in solaris but I guess the whole-disk
devices are named differently.

A faster alternative may be ddrescue. You could also give dd bigger
block sizes:

# dd ... bs=1M

-- 
Regards,
Kai

Replies to list-only preferred.