On 2017-12-19 09:46, Tomasz Pala wrote:
On Tue, Dec 19, 2017 at 07:25:49 -0500, Austin S. Hemmelgarn wrote:
Well, the RAID1+ is all about the failing hardware.
About catastrophically failing hardware, not intermittent failure.
It shouldn't matter - as long as disk failing once is kicked out of the
array *if possible*. Or reattached in write-only mode as a best effort,
meaning "will try to keep your *redundancy* copy, but won't trust it to
be read from".
As you see, the "failure level handled" is not by definition, but by
implementation.
*if possible* == when there are other volume members having the same
data /or/ there are spare members that could take over the failing ones.
Actually, it very much does matter, at least with hardware RAID. The
exact failure mode that causes issues for BTRFS (intermittent
disconnects at the bus level) causes just as many issues with most
hardware RAID controllers (though the exact issues are not quite the
same), and is in and of itself an indicator that something else is wrong.
I never said the hardware needed to not fail, just that it needed to
fail in a consistent manner. BTRFS handles catastrophic failures of
storage devices just fine right now. It has issues with intermittent
failures, but so does hardware RAID, and so do MD and LVM to a lesser
degree.
When planning hardware failovers/backups I can't predict the failing
pattern. So first of all - every *known* shortcoming should be
documented somehow. Secondly - permanent failures are not handled "just
fine", as there is (1) no automatic mount as degraded, so the machine
won't reboot properly and (2) the r/w degraded mount is[*] one-timer.
Again, this should be:
1. documented in manpage, as a comment to profiles, not wiki page or
linux-btrfs archives,
Agreed, our documentation needs consolidated in general (I would
absolutely love to see it just be the man pages, and have those up on
the wiki like some other software does).
2. printed on screen when creating/converting "RAID1" profile (by btrfs tools),
I don't agree on this one. It is in no way unreasonable to expect that
someone has read the documentation _before_ trying to use something.
3. blown into one's face when doing r/w degraded mount (by kernel).
Agreed here though.
[*] yes, I know the recent kernels handle this, but the last LTS (4.14)
is just too young.
4.14 should have gotten that patch last I checked.
I'm now aware of issues with MD you're referring to - I got drives
kicked off many times and they were *never* causing any problems despite
being visible in the system. Moreover, since 4.10 there is FAILFAST
which would do this even faster. There is also no problem with mounting
degraded MD array automatically, so telling that btrfs is doing "just
fine" is, well... not even theoretically close. And in my practice it
never saved the day, but already ruined a few ones... It's not right for
the protection to make more problems than it solves.
Regarding handling of degraded mounts, BTRFS _is_ working just fine, we
just chose a different default behavior from MD and LVM (we make certain
the user knows about the issue without having to look through syslog).
No, classical RAID (other than RAID0) is supposed to handle catastrophic
failure of component devices. That is the entirety of the original
design purpose, and that is the entirety of what you should be using it
for in production.
1. no, it's not: https://www.cs.cmu.edu/~garth/RAIDpaper/Patterson88.pdf
OK, so I see here performance as a motivation, but listed secondarily to
reliability, and all the discussion of reliability assumes that either:
1. Disks fail catastrophically.
or:
2. Disks return read or write errors when there is a problem.
Following just those constraints, RAID is not designed to handle devices
that randomly drop off the bus and reappear or exhibit silent data
corruption, so my original statement largely was accurate, the primary
design intent was handling of catastrophic failures.
2. even if there was, the single I/O failure (e.g. one bad block) might
be interpreted as "catastrophic" and the entire drive should be kicked off
then.
This I will agree with, given that it's common behavior in many RAID
implementations. As people are quick to point out BTRFS _IS NOT_ RAID,
the devs just made a poor choice in the original naming of the 2-way
replication implementation, and it stuck.
3. if sysadmin doesn't request any kind of device autobinding, the
device that were already failed doesn't matter anymore - regardless of
it's current state or reappearences.
You have to explicitly disable automatic binding of drivers to
hot-plugged devices though, so that's rather irrelevant. Yes, you can
do so yourself if you want, and it will mitigate one of the issues with
BTRFS to a limited degree (we still don't 'kick-out' old devices, even
if we should).
The point at which you are getting random corruption
on a disk and you're using anything but BTRFS for replication, you
_NEED_ to replace that disk, and if you don't you risk it causing
corruption on the other disk.
Not only BTRFS, there are hardware solutions like T10 PI/DIF.
Guess what should RAID controller do in such situation? Fail
drive immediately after the first CRC mismatch?
If it's more than single errors, yes, it should fail the drive. If
you're getting any kind of recurring corruption, it's time to replace
the drive, whether the error gets corrected or not.
BTW do you consider "random corruption" as a catastrophic failure?
No, catastrophic failure in reference to hard drives is (usually)
mechanical failure rendering the drive unusable (such as a head crash
for example), or a complete controller failure (for example, the drive
won't enumerate at all).
To use a (possibly strained) analogy: Catastrophic failure is like a
handgun blowing up when you try to fire it, you won't be able to use it
ever again. Random corruption is equivalent to not consistently feeding
new rounds from the magazine properly, it still technically works, and
can (theoretically) be fixed, but it's usually just simpler (and
significantly safer) to replace the gun than it is to try and jury rig
things so that it works reliably.
As of right now, BTRFS is no different in
that respect, but I agree that it _should_ be able to handle such a
situation eventually.
The first step should be to realize, that there are some tunables
required if you want to handle many different situation.
Having said that, let's back to reallity:
The classical RAID is about keeping the system functional - trashing a
single drive from RAID1 should be fully-ignorable by sysadmin. The
system must reboot properly, work properly and there MUST NOT by ANY
functional differences compared to non-degraded mode except for slower
read rate (and having no more redundancy obviously).
'No functional differences' isn't even a standard that MD or LVM
achieve, and it's definitely not one that most hardware RAID controllers
have.
- not having this == not having RAID1.
Again, BTRFS _IS NOT_ RAID.
It shouldn't have been called RAID in the first place, that we can agree
on (even if for different reasons).
The misnaming would be much less of a problem if it were documented
properly (man page, btrfs-progs and finally kernel screaming).
Yes, our documentation could be significantly better.
- I got one "RAID1" stuck in r/o after degraded mount, not nice... Not
_expected_ to happen after single disk failure (without any reappearing).
And that's a known bug on older kernels (not to mention that you should
not be mounting writable and degraded for any purpose other than fixing
the volume).
Yes, ...but:
1. "known" only to the people that already stepped into it, meaning too
late - it should be "COMMONLY known", i.e. documented,
And also known to people who have done proper research.
2. "older kernels" are not so old, the newest mature LTS (4.9) is still
affected,
I really don't see this as a valid excuse. It's pretty well documented
that you absolutely should be running the most recent kernel if you're
using BTRFS.
3. I was about to fix the volume, accidentally the machine has rebooted.
Which should do no harm if I had a RAID1.
Agreed.
4. As already said before, using r/w degraded RAID1 is FULLY ACCEPTABLE,
as long as you accept "no more redundancy"...
This is a matter of opinion. I still contend that running half a two
device array for an extended period of time without reshaping it to be a
single device is a bad idea for cases other than BTRFS. The fewer
layers of code you're going through, the safer you are.
4a. ...or had an N-way mirror and there is still some redundancy if N>2.
N-way mirroring is still on the list of things to implement, believe me,
many people want it.
Since we agree, that btrfs RAID != common RAID, as there are/were
different design principles and some features are in WIP state at best,
the current behaviour should be better documented. That's it.
Patches would be gratefully accepted. It's really not hard to update
the documentation, it's just that nobody has had the time to do it.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
