Re: Ensuring data integrity
On 2/17/2023 5:30 AM, Stuart Henderson wrote: They're often slower (especially in failure conditions) and more complex. Reconstructing RAID5/6 after a drive failure is pretty intensive on the other disks. Not only that but your other (or spare) drives may have a bad sector that won't be detected until it's read. One of the drives in the array goes bad, you rebuild it, and the rebuild fails because now two (or more) drives are bad. At least if you do backups you're periodically verifying all the data can be read.
Re: Ensuring data integrity
> We write our own software. Kudos, appointed somewhere.. However, when you are under data recovery a ready system utility eventually could bring some appreciable goodness to the most.. -- Daniele Bonini
Re: Ensuring data integrity
On Fri, Feb 17, 2023 at 04:24:19PM +0100, Daniele B. wrote: > Feb 17, 2023 11:51:52 Crystal Kolipe : > > > Then you perform your weekly backup, overwriting an older backup which had > > a good copy of the file in question. But this time when the system reads > > the file in from the _bad_ disk, and corrupt data gets written to your > > backup. > > > > Verifying the backup against the on-disk copy of the data may or may not > > reveal the issue, depending on which on-disk copy is read during the > > verify operation. > > I just would ask what software do you use for file system comparison. We write our own software.
Re: Ensuring data integrity
Feb 17, 2023 11:51:52 Crystal Kolipe : > Then you perform your weekly backup, overwriting an older backup which had > a good copy of the file in question. But this time when the system reads > the file in from the _bad_ disk, and corrupt data gets written to your > backup. > > Verifying the backup against the on-disk copy of the data may or may not > reveal the issue, depending on which on-disk copy is read during the > verify operation. I just would ask what software do you use for file system comparison. I know only xxdiff and I'm used to that but it has evident limits: when the file structure is too complex it crashes miserably. -- Daniele Bonini Feb 17, 2023 11:51:52 Crystal Kolipe : > Then you perform your weekly backup, overwriting an older backup which had > a good copy of the file in question. But this time when the system reads > the file in from the _bad_ disk, and corrupt data gets written to your > backup. > > Verifying the backup against the on-disk copy of the data may or may not > reveal the issue, depending on which on-disk copy is read during the > verify operation.
Re: Ensuring data integrity
On Fri, Feb 17, 2023 at 10:30:42AM -, Stuart Henderson wrote: > On 2023-02-17, Eric Johnson <726960+openbsd0...@pm.me> wrote: > > Ask yourself what happens when someone writes a file to a mirror? > > Answer: It means that both drives in the mirror will then contain the > > file. If you make a mistake in the file, it means that you have the > > issue on both drives and no copy of the original file. > > Also it means that e.g. any filesystem errors are just replicated to two > drives instead of one. > > There's also an issue if the drives don't agree about the stored data, > either due to media errors, or partial writes following a crash. You can't > tell which is correct. And you might not even realise that one of the disks contains bad data, being lulled in to a false sense of data integrity: OpenBSD reads in a round-robin style from RAID-1 mirrors implemented with softraid. Imagine that you have a file that is stored in 1024 blocks, and also have a known good checksum for that particular file. Unknown to you, block 10 is corrupted on _one_ of the disks that makes up the mirror. If you check the checksum against the file, the system will read that file block-by-block. Maybe block 10 will be read from the disk with the good copy, and the checksum will be reported as correct. Then you perform your weekly backup, overwriting an older backup which had a good copy of the file in question. But this time when the system reads the file in from the _bad_ disk, and corrupt data gets written to your backup. Verifying the backup against the on-disk copy of the data may or may not reveal the issue, depending on which on-disk copy is read during the verify operation.
Re: Ensuring data integrity
On 2023-02-17, Eric Johnson <726960+openbsd0...@pm.me> wrote: > Ask yourself what happens when someone writes a file to a mirror? > Answer: It means that both drives in the mirror will then contain the > file. If you make a mistake in the file, it means that you have the > issue on both drives and no copy of the original file. Also it means that e.g. any filesystem errors are just replicated to two drives instead of one. There's also an issue if the drives don't agree about the stored data, either due to media errors, or partial writes following a crash. You can't tell which is correct. (there are mirror-like filesystems on some OS which use checksums/hashes to improve chances to identify this, but a standard mirror doesn't). > Create a mirror if you must (why not go to higher Raid levels instead?) They're often slower (especially in failure conditions) and more complex. Reconstructing RAID5/6 after a drive failure is pretty intensive on the other disks. > Do not bet the business on a mirror instead of a backup. Yep. The various forms of RAID allow keeping running in some conditions where hardware has failed but it's something to do *in addition* to backups. Ideally offsite, or at least stored offline - physically disconnected not just unmounted.
Re: Ensuring data integrity
> The operating system has other tools for ensuring data integrity and > compactness." I can confirm you this statement: 1) With OpenBSD and FFS you can remain enough quite and there is no way of comparison with any other OS. 2) You need to set your supporting backups almost at the important checkpoints. I.e: yesterday I just discovered tracker3 browsing and I found myself in the midde of a data recovery of just the storage device containing my work (50gb of plain data)... DUP over DUP sectors, omg so serious, my first time in decades.. I passed by a manual ffs_chk fixing everything and deleting one file I neither lost the latter one.. Fragmentation stable at 0.01%. I guess other guys can quote to us almost the techical features names that are involved to arrive to this given (AS-IS) *data reissurance* of OpenBSD.
Re: Ensuring data integrity
Raid Mirror? I assume you mean Raid-1. One of my brothers used to be a big fan of mirrors. He somehow thought it was some kind of substitute for backing up his data. Guess what? He was wrong. It is generally far better to put the effort into producing and maintaining proper backups. Ask yourself what happens when someone writes a file to a mirror? Answer: It means that both drives in the mirror will then contain the file. If you make a mistake in the file, it means that you have the issue on both drives and no copy of the original file. What do you do if someone steals the computer? Or the building is wiped out in a fire or other disaster? Or if you drop the computer down an escalator (I saw someone do that once). Not long ago, I meant to delete one directory containing less than 1 MB that was only intended to be temporary, but accidentally deleted another directory containing about 35 GB which was easily the most important data on my workstation. Fortunately, I have very good backups and recovered all but the most recent file within two hours. If I was depending on a mirror, I would have lost ten to fifteen years of work. Create a mirror if you must (why not go to higher Raid levels instead?), but you still need to do backups unless the data is meaningless and/or unnecessary. If you can lose your data without having any impact on your business at all, why even bother with a mirror? Do not bet the business on a mirror instead of a backup. Eric --- Original Message --- On Wednesday, February 15th, 2023 at 18:48, i...@tutanota.com wrote: > In the latest book by Michael Lucas, OpenBSD Mastery: Filesystems, Michael > writes, "A filesystem should put data on disk. That data should be safely > stored and reliably read. That's it. Error checking? Deduplication? No. > The operating system has other tools for ensuring data integrity and > compactness." > > If I setup a couple of drives in a RAID mirror on OpenBSD to serve as > a NAS box, what is the best way to ensure data integrity? > > -- > Sent with Tutanota, enjoy secure & ad-free emails.
Re: Ensuring data integrity
On Thu, Feb 16, 2023 at 01:48:49AM +0100, i...@tutanota.com wrote: > If I setup a couple of drives in a RAID mirror on OpenBSD to serve as > a NAS box, what is the best way to ensure data integrity? This is a very broad question and the answer is always going to depend on your specific use case. However, in general, it's best to do this kind of thing at the application level, or rather not to add excessive features at the filesystem or block device level to try to ensure 'data integrity'. This could be as simple as a script run from cron to check the hashes of files which are not expected to change, or as complicated as modifying the applications you are using to store ECC in their data files, or even write multiple copies of the same data to different files on the disks, (and check those as deemed to be required). But the point is to do this _in the application_ or at least as a user-land task, and not try to build a fancy 'one size fits all' solution to the (perceived), problem with a complex combination of RAID arrays, funky filesystems, and other trickery. And the other key to ensuring data integrity over a long period of time is to separate out data which is not expected to change, and archive it separately. Don't just create one massive RAID-5 dumping ground for everything and keep adding data and disks to it in an unorganised fashion until it breaks.
Ensuring data integrity
In the latest book by Michael Lucas, OpenBSD Mastery: Filesystems, Michael writes, "A filesystem should put data on disk. That data should be safely stored and reliably read. That's it. Error checking? Deduplication? No. The operating system has other tools for ensuring data integrity and compactness." If I setup a couple of drives in a RAID mirror on OpenBSD to serve as a NAS box, what is the best way to ensure data integrity? -- Sent with Tutanota, enjoy secure & ad-free emails.
