On 2017-10-31 16:06, ST wrote:
Thank you very much for such an informative response!
On Tue, 2017-10-31 at 13:45 -0400, Austin S. Hemmelgarn wrote:
On 2017-10-31 12:23, ST wrote:
Hello,
I've recently learned about btrfs and consider to utilize for my needs.
I have several questions in this regard:
I manage a dedicated server remotely and have some sort of script that
installs an OS from several images. There I can define partitions and
their FSs.
1. By default the script provides a small separate partition for /boot
with ext3. Does it have any advantages or can I simply have /boot
within / all on btrfs? (Note: the OS is Debian9)
It depends on the boot loader. I think Debian 9's version of GRUB has
no issue with BTRFS, but see the response below to your question on
subvolumes for the one caveat.
2. as for the / I get ca. following written to /etc/fstab:
UUID=blah_blah /dev/sda3 / btrfs ...
So top-level volume is populated after initial installation with the
main filesystem dir-structure (/bin /usr /home, etc..). As per btrfs
wiki I would like top-level volume to have only subvolumes (at least,
the one mounted as /) and snapshots. I can make a snapshot of the
top-level volume with / structure, but how can get rid of all the
directories within top-lvl volume and keep only the subvolume
containing / (and later snapshots), unmount it and then mount the
snapshot that I took? rm -rf / - is not a good idea...
There are three approaches to doing this, from a live environment, from
single user mode running with init=/bin/bash, or from systemd emergency
mode. Doing it from a live environment is much safer overall, even if
it does take a bit longer. I'm listing the last two methods here only
for completeness, and I very much suggest that you use the first (do it
from a live environment).
Regardless of which method you use, if you don't have a separate boot
partition, you will have to create a symlink called /boot outside the
subvolume, pointing at the boot directory inside the subvolume, or
change the boot loader to look at the new location for /boot.
From a live environment, it's pretty simple overall, though it's much
easier if your live environment matches your distribution:
1. Create the snapshot of the root, naming it what you want the
subvolume to be called (I usually just call it root, SUSE and Ubuntu
call it @, others may have different conventions).
2. Delete everything except the snapshot you just created. The safest
way to do this is to explicitly list each individual top-level directory
to delete.
3. Use `btrfs subvolume list` to figure out the subvolume ID for the
subvolume you just created, and then set that as the default subvolume
with `btrfs subvolume set-default /path SUBVOLID`.
Do I need to chroot into old_root before doing set-default? Otherwise it
will attempt to set in the live environment, will it not?
The `subvolume set-default` command operates on a filesystem, not an
environment, since the default subvolume is stored in the filesystem
itself (it would be kind of pointless otherwise). The `/path` above
should be replaced with where you have the filesystem mounted, but it
doesn't matter what your environment is when you call it (as long as the
filesystem is mounted of course).
Also another questions in this regard - I tried to "set-default" and
then reboot and it worked nice - I landed indeed in the snapshot, not
top-level volume. However /etc/fstab didn't change and actually showed
that top-level volume should have been mounted instead. It seems that
"set-default" has higher precedence than fstab...
1. is it true?
2. how do they actually interact?
3. such a discrepancy disturbs me, so how should I tune fstab to reflect
the change? Or maybe I should not?
The default subvolume is what gets mounted if you don't specify a
subvolume to mount. On a newly created filesystem, it's subvolume ID 5,
which is the top-level of the filesystem itself. Debian does not
specify a subvo9lume in /etc/fstab during the installation, so setting
the default subvolume will control what gets mounted. If you were to
add a 'subvolume=' or 'subvolid=' mount option to /etc/fstab for that
filesystem, that would override the default subvolume.
The reason I say to set the default subvolume instead of editing
/etc/fstab is a pretty simple one though. If you edit /etc/fstab and
don't set the default subvolume, you will need to mess around with the
bootloader configuration (and possibly rebuild the initramfs) to make
the system bootable again, whereas by setting the default subvolume, the
system will just boot as-is without needing any other configuration changes.
Once you do this,
you will need to specify subvolid=5 in the mount options to get the real
top-level subvolume.
4. Reboot.
For single user mode (check further down for what to do with systemd,
also note that this may brick your system if you get it wrong):
1. When booting up the system, stop the bootloader and add
'init=/bin/bash' to the kernel command line before booting.
2. When you get a shell prompt, create the snapshot, just like above.
3. Run the following:
'cd /path ; mkdir old_root ; pivot_root . old_root ; chroot . /bin/bash'
3. You're now running inside the new subvolume, and the old root
filesystem is mounted at /old_root. From here, just follow steps 2 to 4
from the live environment method.
For doing it from emergency mode, things are a bit more complicated:
1. Create the snapshot of the root, just like above.
2. Make sure the only services running are udev and systemd-journald.
3. Run `systemctl switch-root` with the path to the subvolume you just
created.
4. You're now running inside the new root, systemd _may_ try to go all
the way to a full boot now.
5. Mount the root filesystem somewhere, and follow steps 2 through 4 of
the live environment method.
3. in my current ext4-based setup I have two servers while one syncs
files of certain dir to the other using lsyncd (which launches rsync on
inotify events). As far as I have understood it is more efficient to use
btrfs send/receive (over ssh) than rsync (over ssh) to sync two boxes.
Do you think it would be possible to make lsyncd to use btrfs for
syncing instead of rsync? I.e. can btrfs work with inotify events? Did
somebody try it already?
BTRFS send/receive needs a read-only snapshot to send from. This means
that triggering it on inotify events is liable to cause performance
issues and possibly lose changes
Actually triggering doesn't happen on each and every inotify event.
lsyncd has an option to define a time interval within which all inotify
events are accumulated and only then rsync is launched. It could be 5-10
seconds or more. Which is quasi real time sync. Do you still hold that
it will not work with BTRFS send/receive (i.e. keeping previous snapshot
around and creating a new one)?
Okay, I actually didn't know that. Depending on how lsyncd invokes
rsync though (does it call out rsync with the exact paths or just on the
whole directory?), it may still be less efficient to use BTRFS send/receive.
(contrary to popular belief, snapshot
creation is neither atomic nor free). It also means that if you want to
match rsync performance in terms of network usage, you're going to have
to keep the previous snapshot around so you can do an incremental send
(which is also less efficient than rsync's file comparison, unless rsync
is checksumming files).
Indeed? From what I've read so far I got an impression that rsync is
slower... but I might be wrong... Is this by design so, or can BTRFS
beat rsync in future (even without checksumming)?
It really depends. BTRFS send/receive transfers _everything_, period.
Any xattrs, any ACL's, any other metadata, everything. Rsync can
optionally not transfer some of that data (and by default doesn't), so
if you don't need all of that (and most people don't need xattrs or
ACL's transferred), rsync is usually going to be faster. When you
actually are transferring everything, send/receive is probably faster,
and it's definitely faster than rsync with checksumming.
There's one other issue at hand though that I had forgot to mention.
The current implementation of send/receive doesn't properly validate
sources for reflinks, which means it's possible to create an information
leak with a carefully crafted send stream and some pretty minimal
knowledge of the destination filesystem. Whether or not this matters is
of course specific to your use case.
Because of this, it would be pretty complicated right now to get lsyncd
reliable integration.
Otherwise I can sync using btrfs send/receive from within cron every
10-15 minutes, but it seems less elegant.When it comes to stuff like this, it's
usually best to go for the
simplest solution that meets your requirements. Unless you need
real-time synchronization, inotify is overkill,
I actually got inotify-based lsyncd working and I like it... however
real-time syncing is not a must, and several years everything worked
well with a simple rsync within a cron every 15 minutes. Could you
please elaborate on the disadvantages of lsyncd, so maybe I should
switch back? For example, in which of two cases the life of the hard
drive is negatively impacted? On one side the data doesn't change too
often, so 98% of rsync's from cron are wasted, on the other triggering a
rsync on inotify might be too intensive task for a hard drive? What do
you think? What other considerations could be?
The biggest one is largely irrelevant if lsyncd batches transfers, and
arises from the possibility of events firing faster than you can handle
them (which runs the risk of events getting lost, and in turn things
getting out of sync). The other big one (for me at least) is
determinism. With a cron job, you know exactly when things will get
copied, and in turn exactly when the system will potentially be under
increased load (which makes it a lot easier to quickly explain to users
why whatever they were doing unexpectedly took longer than normal).
and unless you need to
copy reflinks (you probably don't, as almost nothing uses them yet, and
absolutely nothing I know of depends on them) send/receive is overkill.
I saw in a post that rsync would create a separate copy of a cloned file
(consuming double space and maybe traffic?)
That's correct, but you technically need to have that extra space in
most cases anyway, since you can't assume nothing will write to that
file and double the space usage.
As a pretty simple example, we've got a couple of systems that have
near-line active backups set up. The data is stored on BTRFS, but we
just use a handful of parallel rsync invocations every 15 minutes to
keep the backup system in sync (because of what we do, we can afford to
lose 15 minutes of data). It's not 'elegant', but it's immediately
obvious to any seasoned sysadmin what it's doing, and it gets the job
done easily syncing the data in question in at most a few minutes. Back
when I switched to using BTRFS, I considered using send/receive, but
even using incremental send/receive still performed worse than rsync.
4. In a case when compression is used - what quota is based on - (a)
amount of GBs the data actually consumes on the hard drive while in
compressed state or (b) amount of GBs the data naturally is in
uncompressed form. I need to set quotas as in (b). Is it possible? If
not - should I file a feature request?
I can't directly answer this as I don't know myself (I don't use
quotas), but have two comments I would suggest you consider:
1. qgroups (the BTRFS quota implementation) cause scaling and
performance issues. Unless you absolutely need quotas (unless you're a
hosting company, or are dealing with users who don't listen and don't
pay attention to disk usage, you usually do not need quotas), you're
almost certainly better off disabling them for now, especially for a
production system.
Ok. I'll use more standard approaches. Which of following commands will
work with BTRFS:
https://debian-handbook.info/browse/stable/sect.quotas.html
None, qgroups are the only option right now with BTRFS, and it's pretty
likely to stay that way since the internals of the filesystem don't fit
well within the semantics of the regular VFS quota API. However,
provided you're not using huge numbers of reflinks and subvolumes, you
should be fine using qgroups.
However, it's important to know that if your users have shell access,
they can bypass qgroups. Normal users can create subvolumes, and new
subvolumes aren't added to an existing qgroup by default (and unless I'm
mistaken, aren't constrained by the qgroup set on the parent subvolume),
so simple shell access is enough to bypass quotas.
2. Compression and quotas cause issues regardless of how they interact.
In case (a), the user has no way of knowing if a given file will fit
under their quota until they try to create it. In case (b), actual disk
usage (as reported by du) will not match up with what the quota says the
user is using, which makes it harder for them to figure out what to
delete to free up space. It's debatable which is a less objectionable
situation for users, though most people I know tend to think in a way
that the issue with (a) doesn't matter, but the issue with (b) does.
I think both (a) and (b) should be possible and it should be up to
sysadmin to choose what he prefers. The concerns of the (b) scenario
probably could be dealt with some sort of --real-size to the du command,
while by default it could have behavior (which might be emphasized with
--compressed-size).
Reporting anything but the compressed size by default in du would mean
it doesn't behave as existing software expect it to. It's supposed to
report actual disk usage (in contrast to the sum of file sizes), which
means for example that a 1G sparse file with only 64k of data is
supposed to be reported as being 64k by du.
Two more question came to my mind: as I've mentioned above - I have two
boxes one syncs to another. No RAID involved. I want to scrub (or scan -
don't know yet, what is the difference...) the whole filesystem once in
a month to look for bitrot. Questions:
1. is it a stable setup for production? Let's say I'll sync with rsync -
either in cron or in lsyncd?
Reasonably, though depending on how much data and other environmental
constraints, you may want to scrub a bit more frequently.
2. should any data corruption be discovered - is there any way to heal
it using the copy from the other box over SSH?
Provided you know which file is affected, yes, you can fix it by just
copying the file back from the other system.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
