On 2018-03-21 16:02, Christoph Anton Mitterer wrote:
On the note of maintenance specifically:
- Maintenance tools
- How to get the status of the RAID? (Querying kernel logs is IMO
rather a bad way for this)
This includes:
- Is the raid degraded or not?
Check for the 'degraded' flag in the mount options. Assuming you're
doing things sensibly and not specifying it on mount, it gets added when
the array goes degraded.
- Are scrubs/repairs/rebuilds/reshapes in progress and how far are
they? (Reshape would be: if the raid level is changed or the raid
grown/shrinked: has all data been replicated enough to be
"complete" for the desired raid lvl/number of devices/size?
A bit trickier, but still not hard, just check the the output of `btrfs
scrub status`, `btrfs balance status`, and `btrfs replace status` for
the volume. It won't check automatic spot-repairs (that is, repairing
individual blocks that fail checksums), but most people really don't care
- What should one regularly do? scrubs? balance? How often?
Do we get any automatic (but configurable) tools for this?
There aren't any such tools that I know of currently. storaged might
have some, but I've never really looked at it so i can't comment (I'm
kind of adverse to having hundreds of background services running to do
stuff that can just as easily be done in a polling manner from cron
without compromising their utility). Right now though, it's _trivial_
to automate things with cron, or systemd timers, or even third-party
tools like monit (which has the bonus that if the maintenance fails, you
get an e-mail about it).
- There should be support in commonly used tools, e.g. Icinga/Nagios
check_raid
Agreed. I think there might already be a Nagios plugin for the basic
checks, not sure about anything else though.
Netdata has had basic monitoring support for a while now, but it only
looks at allocations, not error counters, so while it will help catch
impending ENOSPC issues, it can't really help much with data corruption
issues.
- Ideally there should also be some desktop notification tool, which
tells about raid (and btrfs errors in general) as small
installations with raids typically run no Icinga/Nagios but rely
on e.g. email or gui notifications.
Desktop notifications would be nice, but are out of scope for the main
btrfs-progs. Not even LVM, MDADM, or ZFS ship desktop notification
support from upstream. You don't need Icinga or Nagios for monitoring
either. Netdata works pretty well for covering the allocation checks
(and I'm planning to have something soon, and it's trivial to set up
e-mail notifications with cron or systemd timers or even tools like monit.
On the note of generic monitoring though, I've been working on a Python
3 script (with no dependencies beyond the Python standard library) to do
the same checks that Netdata does regarding allocations, as well as
checking device error counters and mount options that should be
reasonable as a simple warning tool run from cron or a systemd timer.
I'm hoping to get it included in the upstream btrfs-progs, but I don't
have it in a state yet that it's ready to be posted (the checks are
working, but I'm still having issues reliably mapping between mount
points and filesystem UUID's).
I think especially for such tools it's important that these are
maintained by upstream (and yes I know you guys are rather fs
developers not)... but since these tools are so vital, having them done
3rd party can easily lead to the situation where something changes in
btrfs, the tools don't notice and errors remain undetected.
It depends on what they look at. All the stuff under /sys/fs/btrfs
should never change (new things might get added, but none of the old
stuff is likely to ever change because /sys is classified as part of the
userspace ABI, and any changes would get shot down by Linus), so
anything that just uses those will likely have no issues (Netdata falls
into this category for example). Same goes for anything using ioctls
directly, as those are also userspace ABI.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
