On 2021/3/10 上午7:08, Pete wrote:
I am unsure where I am going wrong here. This might be a btrfs device
adding issue, or more likely this is a complex file system stack which I
don't touch much and I'm doing something wrong. I'm posting here as at
the moment my logical conclusion is that it is a btrfs issue, but in all
likelihood it is probably me.
Kernel 5.10.19
btrfs-progs 5.10.1
It's a known regression in v5.10.1 btrfs-progs, which did wrong path
normalization for device map.
It's fixed in v5.11 btrfs-progs.
Thanks,
Qu
Slackware current (but with my own kernel).
The basic difference between the stock slackware kernel the one I build
is I usually building in the common file system drivers rather than have
them as modules to avoid creating an initrd.
I have a nvme drive which I have encrypted with dm-crypt / luks format.
The stack looks like:
Hardware: nvme0
-> dm-crypt
--> lvm
--> nvme_vg-lxc-20200226 - existing btrfs
--> nvme_vg-lxc_db - xfs for mariadb data
--> nvme_vg_virt - for qemu images etc. (xfs I think)
--> nvme_vg-lxc-20210309 - new device to add to existing btrfs
This is opened (with small tweaks for privacy) with:
/usr/sbin/cryptsetup luksOpen UUID=<UUID> nvme0
I have created a physical volume group on this device:
root@phoenix:/mnt# pvscan
PV /dev/mapper/nvme0 VG nvme0_vg lvm2 [<429.00 GiB / <19.00
GiB free]
Total: 1 [<429.00 GiB] / in use: 1 [<429.00 GiB] / in no VG: 0 [0 ]
Looking in /dev/mapper I have
ls -al /dev/mapper/nvme*
lrwxrwxrwx 1 root root 7 Mar 9 22:01 /dev/mapper/nvme0 -> ../dm-1
lrwxrwxrwx 1 root root 7 Mar 9 22:01 /dev/mapper/nvme0_vg-lxc_20200626
-> ../dm-4
lrwxrwxrwx 1 root root 7 Mar 9 22:15 /dev/mapper/nvme0_vg-lxc_20210309
-> ../dm-5
lrwxrwxrwx 1 root root 7 Mar 9 22:01 /dev/mapper/nvme0_vg-lxc_db -> ../dm-2
lrwxrwxrwx 1 root root 7 Mar 9 22:01 /dev/mapper/nvme0_vg-virt -> ../dm-3
The logical volumes are:
root@phoenix:/mnt# lvscan
ACTIVE '/dev/nvme0_vg/lxc_db' [50.00 GiB] inherit
ACTIVE '/dev/nvme0_vg/virt' [30.00 GiB] inherit
ACTIVE '/dev/nvme0_vg/lxc_20200626' [80.00 GiB] inherit
ACTIVE '/dev/nvme0_vg/lxc_20210309' [250.00 GiB] inherit
All good. The logical volume /dev/nvme0_vg/lxc_20200626 is mounted on
/var/lib/lxc as a btrfs with no issue except for being a bit on the
small side. I therefore created a larger physical volume
/dev/nvme0_vg/lxc_20210309 with the aim of adding it to the btrfs
mounted on /var/lib/lxc. I can later remove the smaller lxc logical
volume from the file system. That seems a neater solution than resizing
a live file system. However, when I try adding the logical volume to
the file system I get:
cd /var/lib/lxc
root@phoenix:/var/lib/lxc# btrfs dev add /dev/nvme0_vg/lxc_20210309 .
ERROR: error adding device 'dm-5': No such file or directory. I can see
no messages in /var/log/messages, /var/log/syslog or with dmesg.
If I zero out the first 10M I get no difference.
root@phoenix:/var/lib/lxc# dd if=/dev/zero of=/dev/nvme0_vg/lxc_20210309
bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0216118 s, 485 MB/s
root@phoenix:/var/lib/lxc# btrfs dev add /dev/nvme0_vg/lxc_20210309 .
ERROR: error adding device 'dm-5': No such file or directory
root@phoenix:/var/lib/lxc#
Note that /dev/nvme0_vg/lxc_20210309 is a symlink to /dev/dm-5.
For whatever reason /dev/nvme0_vg/lxc_20210309 looks to be an issue.
However, I can create an ext4 file system on it or a new btrfs on it and
mount it:
root@phoenix:/var/lib/lxc# mkfs.btrfs /dev/nvme0_vg/lxc_20210309
btrfs-progs v5.10.1
See http://btrfs.wiki.kernel.org for more information.
Detected a SSD, turning off metadata duplication. Mkfs with -m dup if
you want to force metadata duplication.
Label: (null)
UUID: b6c37a47-6029-441c-8092-dbe3a1bdd549
Node size: 16384
Sector size: 4096
Filesystem size: 250.00GiB
Block group profiles:
Data: single 8.00MiB
Metadata: single 8.00MiB
System: single 4.00MiB
SSD detected: yes
Incompat features: extref, skinny-metadata
Runtime features:
Checksum: crc32c
Number of devices: 1
Devices:
ID SIZE PATH
1 250.00GiB /dev/nvme0_vg/lxc_20210309
root@phoenix:/var/lib/lxc# mount /dev/nvme0_vg/lxc_20210309 /mnt/zip
root@phoenix:/var/lib/lxc# mount | grep zip
/dev/mapper/nvme0_vg-lxc_20210309 on /mnt/zip type btrfs
(rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
root@phoenix:/var/lib/lxc#
root@phoenix:/var/lib/lxc# cd /mnt/zip
root@phoenix:/mnt/zip# ls
root@phoenix:/mnt/zip# touch wibble
root@phoenix:/mnt/zip# ls -al
total 16
drwxr-xr-x 1 root root 12 Mar 9 22:39 ./
drwxr-xr-x 1 root root 276 Sep 26 2006 ../
-rw-r--r-- 1 root root 0 Mar 9 22:39 wibble
root@phoenix:/mnt/zip#
I'm at a loss to understand what is going on. If this problematic
logical volume cannot be added to an existing btrfs why can I create a
fresh file system on it? Ext4 works as well, but I did not think the
example useful over the one above.
I could just create a new btrfs on the new logical volume for lxc and
copy the data across. Slightly wary if the error I am seeing when I try
to add it to an existing file system is occurring that this will not
bite me later.
For completeness the old btrfs looks like this:
root@phoenix:/var/lib/lxc# btrfs fi u .
Overall:
Device size: 80.00GiB
Device allocated: 80.00GiB
Device unallocated: 1.00MiB
Device missing: 0.00B
Used: 69.33GiB
Free (estimated): 3.91GiB (min: 3.91GiB)
Free (statfs, df): 3.91GiB
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data,single: Size:62.01GiB, Used:58.10GiB (93.70%)
/dev/mapper/nvme0_vg-lxc_20200626 62.01GiB
Metadata,single: Size:17.99GiB, Used:11.22GiB (62.41%)
/dev/mapper/nvme0_vg-lxc_20200626 17.99GiB
System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
/dev/mapper/nvme0_vg-lxc_20200626 4.00MiB
Unallocated:
/dev/mapper/nvme0_vg-lxc_20200626 1.00MiB
root@phoenix:/var/lib/lxc#
Any thoughts?
I realise that this filesystem stack is complex. The reason is that I
am running mariadb and I have used lvm to create a logical volume with a
xfs partition to hold the database as that is supposed to work better,
at least historically. Similar concerns with the qemu images. I'm
unsure of whether it gives me a sufficient performance difference to be
worth the extra complexity, and if I can't tell subjectively it is not
clear that the extra complexity is worthwhile.
regards,
Pete
