On Sun, Nov 01, 2026 at 11:03:49AM -0500, Nick Holland wrote: > On 11/1/26 09:45, Richard Bostrom wrote: > > Hello Friends! > > > > Is there a way of protecting against data-corruption? +10.000 files stored > > for the long term. > * Mounting static file systems read-only, except when changes are needed > * Multiple copies on physically different media > * some kind of hash (checksum) to verify file integrity. Hash stored with > all backup copies. > * periodically verifying the hash > * periodically verifying read integrity of the disks -- ALL the disks, > including your extra copies (dd can do that). >
Excellent. I would add to this Solomon-Reed checksums. A program like par2 allows me to specify level of protection (i.e. should I be able to recover up to 5% damage of said files? 10%, you name it). I was able to recover some minor damage to backups with par2. In one case, damage was too big (exceeded 5% of blocks that was specified by me) and so, no luck, par2 could not undo this. Block size is fluid, as far as I have observed - the bigger the data, the bigger the block. For example, with 1GB of data I specify 64k block size, but when I have +2GB, par2 protests and I have to specify 128k block size. If I do not specify, it will choose something by itself, but I think I prefer to have it standardized a bit. Alternative: rar with recovery volumes. Note: par2 expects that all files to be protected reside on same directory, no recursion. Or it used to expect that in a past and I did not try otherwise since. Rar can process subdirectories, but I had not tried it yet for backup. Rar can also do multivolume, while right now I use tar and split to "multivolume" and then compute Solomon-Reed on this. > The physical dd read test is pretty slick... > # dd if=/dev/rsdXc of=/dev/null bs=1m > (where "X" is your physical device you are checking). > (...) > > Sometimes, deleting the damaged files, writing a "zero file" to the > entire file system > # dd if=/dev/zero of=/some/file/system/zerofile bs=1m > will force bad spots to be swapped out by the disk, and "repair" the > problem, maybe short-term, maybe until some unrelated failure, but > the non-cheapskate solution is to replace the disk, of course. Rotating hard drives nowadays have a pool of spare blocks. When one block goes bad, firmware replaces it (remaps) with spare from the pool upon write - this is why dd does the job, as you described above. It shows in SMART parameters in a drive (as shown by smartctl). The folk wisdom says, when "Reallocated_Sector_Ct" starts counting, it is time to find a new disk. This parameter should be 0 in normal case. HTH -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home ** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:[email protected] **
