Re: Creating btrfs RAID on LUKS devs makes devices disappear

[Andrei Borzenkov]

13.05.2017 18:28, Ochi пишет:
> Hello,
> 
> okay, I think I now have a repro that is stupidly simple, I'm not even
> sure if I overlook something here. No multi-device btrfs involved, but
> notably it does happen with btrfs, but not with e.g. ext4.
> 

I could not reproduce it with single device but I finally was able to
reliably reproduce your problem with multiple devices. It looks a bit
different, I think difference is due to encrypted root in your case. Anyway

https://github.com/systemd/systemd/issues/5955

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Creating btrfs RAID on LUKS devs makes devices disappear

[Ochi]

Hello,

okay, I think I now have a repro that is stupidly simple, I'm not even 
sure if I overlook something here. No multi-device btrfs involved, but 
notably it does happen with btrfs, but not with e.g. ext4.


[Sidenote: At first I thought it had to do with systemd-cryptsetup 
opening multiple devices with the same key that makes a difference. 
Rationale: I think the whole systemd machinery for opening crypt devices 
is able to try the same password on multiple devices when manual 
keyphrase input is used, and I thought maybe the same is true for 
keyfiles which may cause race conditions, but after all it doesn't seem 
to matter much. Also it seemed to relate to multi-device btrfs volumes, 
but now it appears to be simpler than that. That said, I can't be sure 
whether there are more problems hidden when actually using RAID.]


I have tried to repro the issue on a completely fresh Arch Linux in a 
VirtualBox VM. No custom systemd magic involved whatsoever, all stock 
services, generators, etc. In addition to the root volume (no crypto), 
there is another virtual HDD with one partition. This is a LUKS 
partition with a keyfile added to open it automatically on boot. I added 
a corresponding /etc/crypttab line as follows:


storage0/dev/sdb1/etc/crypto/keyfile

Let's suppose we open the crypt device manually the first time and 
perform mkfs.btrfs on the /dev/mapper/storage0 device. Reboot the system 
such that systemd-cryptsetup can do its magic to open the dm device.


After reboot, log in. /dev/mapper/storage0 should be there, and of 
course the corresponding /dev/dm-*. Perform another mkfs.btrfs on 
/dev/mapper/storage0. What I observe is (possibly try multiple times, 
but it has been pretty reliable in my testing):


- /dev/mapper/storage0 and the /dev/dm-* device are gone.

- A process systemd-cryptsetup is using 100% CPU (haven't noticed 
before, but now on my laptop I can actually hear it)


- The dm-device was eliminated by systemd, see the logs below.

- Logging out and in again (as root in my case) solves the issue, the 
device is back.


I have prepared outputs of journalctl and udevadm info --export-db 
produced after the last step (logging out and back in). Since the logs 
are quite large, I link them here, I hope that is okay:


https://pastebin.com/1r6j1Par
https://pastebin.com/vXLGFQ0Z

In the journal, the interesting spots are after the two "ROOT LOGIN ON 
tty1". A few seconds after the first one, I performed the mkfs.


Notably, it doesn't seem to happen when using e.g. ext4 instead of 
btrfs. Also, it doesn't happen when opening the crypt device manually, 
without crypttab and thus without systemd-cryptsetup, 
systemd-cryptsetup-generator, etc. which parses crypttab.


So after all, I suspect the systemd-cryptsetup to be the culprit in 
combination with btrfs volumes. Maybe someone can repro that.


Versions used in the VM:
- Current Arch Linux
- Kernel 4.10.13
- btrfs-progs 4.10.2
- systemd v232 (also tested v233 from testing repo with same results)

Hope this helps
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Creating btrfs RAID on LUKS devs makes devices disappear

[Andrei Borzenkov]

12.05.2017 20:07, Chris Murphy пишет:
> On Thu, May 11, 2017 at 5:24 PM, Ochi  wrote:
>> Hello,
>>
>> here is the journal.log (I hope). It's quite interesting. I rebooted the
>> machine, performed a mkfs.btrfs on dm-{2,3,4} and dm-3 was missing
>> afterwards (around timestamp 66.*). However, I then logged into the machine
>> from another terminal (around timestamp 118.*) which triggered something to
>> make the device appear again :O Indeed, dm-3 was once again there after
>> logging in. Does systemd mix something up?
> 

Yes :) Did you doubt?

Please, try to reproduce it and provide both journalctl and "udevadm
info --export-db" output. I have my theory what happens here.

> I don't see any Btrfs complaints. If dm-3 is the device Btrfs is
> expecting and it vanishes, then Btrfs would mention it on a read or
> write. So either nothing is happening and Btrfs isn't yet aware that
> dm-3 is gone, or it's looking at some other instance of that encrypted
> volume, maybe it's using /dev/mapper/storage1. You can find out with
> btrfs fi show.
> 

/dev/mapper/xxx should be link to /dev/dm-NN (although you are never
sure with Linux). dm-NN is *the* device. /dev/mapper/storage1 cannot
exist without /dev/dm-3, irrespectively of what btrfs shows.

> Whenever I use Btrfs on LUKS I invariably see fi show, show me one
> device using /dev/dm-0 notation and the other device is
> /dev/mapper/blah notation. I think this is just an artifact of all the
> weird behind the scenes shit with symlinks and such, and that is a
> systemd thing as far as I know.
> 

Not quite.

As implemented today, when device appears (i.e. udev gets ADD uevent)
and it is detected as part of btrfs, udev rule scans device (with
equivalent of "btrfs device ready"). At the time event is being
processed the only name that is available is canonical kernel /dev/dm-0;
all convenience syminks are created later, after all rules have been
processed. btrfs remembers device name that was passed to it.

What makes it even more confusing - some btrfs utilities seem to resolve
/dev/dm-0 to /dev/mapper/blah by itself, and some not. I am not sure
which ones.

Recently btrfsprogs got extra rule that repeats "btrfs device ready" but
now *after* symlinks have been created (using RUN directive with
/dev/mapper/blah). It updates kernel with new name. So my guess is that
for this device this rule is missing (probably it gets created in initrd
and rule is not added to it).

> Anyway, more information is needed. First see if the device is really
> missing per Btrfs (read or write something and also check with 'btrfs
> fi show') You can add systemd.log_level=debug as a boot parameter to
> get more information in the journal, although it's rather verbose. You
> could combine it with rd.udev.debug but that gets really crazy verbose
> so I tend not to use them at the same time.
> 

@ochi: Please, before running with debug repeat your test as you did and
provide udevadm info --export-db. This may be enough. In my experience
debug output while being extremely verbose contains very little
additional useful information, but has tendency of skewing relative
timing thus changing behavior.

> The other possibility is there's a conflict with dracut which may be
> doing some things, and the debug switch for that is rd.debug and is
> likewise extremely verbose, producing huge logs, so I would start out
> with just the systemd debug and see if that reveals anything
> *assuming* Btrfs is complaining. If Btrfs doesn't complain about a
> device going away then I wouldn't worry about it.
> 
> 

This is not btrfs issue in any case (except btrfs folks should really
work together with systemd folks and finally come to common
implementation of multi-device filesystems).

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Creating btrfs RAID on LUKS devs makes devices disappear

[Chris Murphy]

On Thu, May 11, 2017 at 5:24 PM, Ochi  wrote:
> Hello,
>
> here is the journal.log (I hope). It's quite interesting. I rebooted the
> machine, performed a mkfs.btrfs on dm-{2,3,4} and dm-3 was missing
> afterwards (around timestamp 66.*). However, I then logged into the machine
> from another terminal (around timestamp 118.*) which triggered something to
> make the device appear again :O Indeed, dm-3 was once again there after
> logging in. Does systemd mix something up?

I don't see any Btrfs complaints. If dm-3 is the device Btrfs is
expecting and it vanishes, then Btrfs would mention it on a read or
write. So either nothing is happening and Btrfs isn't yet aware that
dm-3 is gone, or it's looking at some other instance of that encrypted
volume, maybe it's using /dev/mapper/storage1. You can find out with
btrfs fi show.

Whenever I use Btrfs on LUKS I invariably see fi show, show me one
device using /dev/dm-0 notation and the other device is
/dev/mapper/blah notation. I think this is just an artifact of all the
weird behind the scenes shit with symlinks and such, and that is a
systemd thing as far as I know.

Anyway, more information is needed. First see if the device is really
missing per Btrfs (read or write something and also check with 'btrfs
fi show') You can add systemd.log_level=debug as a boot parameter to
get more information in the journal, although it's rather verbose. You
could combine it with rd.udev.debug but that gets really crazy verbose
so I tend not to use them at the same time.

The other possibility is there's a conflict with dracut which may be
doing some things, and the debug switch for that is rd.debug and is
likewise extremely verbose, producing huge logs, so I would start out
with just the systemd debug and see if that reveals anything
*assuming* Btrfs is complaining. If Btrfs doesn't complain about a
device going away then I wouldn't worry about it.


-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Creating btrfs RAID on LUKS devs makes devices disappear

[Austin S. Hemmelgarn]

On 2017-05-12 09:54, Ochi wrote:

On 12.05.2017 13:25, Austin S. Hemmelgarn wrote:

On 2017-05-11 19:24, Ochi wrote:

Hello,

here is the journal.log (I hope). It's quite interesting. I rebooted the
machine, performed a mkfs.btrfs on dm-{2,3,4} and dm-3 was missing
afterwards (around timestamp 66.*). However, I then logged into the
machine from another terminal (around timestamp 118.*) which triggered
something to make the device appear again :O Indeed, dm-3 was once again
there after logging in. Does systemd mix something up?

Hmm, I just did another mkfs once the devices where back, devices were
missing, but they re-appeared a few seconds later, without logging into
a terminal. After another mkfs, they were gone again and are now still
gone after waiting a few minutes. It's really weird, I can't really tell
what triggers this yet. Will test more tomorrow, let me know if you have
any more ideas what to try.


It looks like something made systemd think that it should tear down
the LUKS volumes, but it somehow only got /dev/dm-3 and not the
others, and then you logging in and triggering the creation of the
associated user slice somehow made it regain it's senses.  Next time
you see it disappear, try checking `systemctl status` for the unit you
have set up for the LUKS volume and see what that says.  I doubt it
will give much more info, but I suspect it will say it's stopped,
which will solidify that systemd is either misconfigured or doing
something stupid (and based on past experience, I'm willing to bet
it's the latter).


I will take a closer look at systemd when I get home. I would like to
point out that this sounds very related to these fairly recent systemd
issues:

https://github.com/systemd/systemd/issues/5781

https://github.com/systemd/systemd/issues/5866

So my best guess is that systemd is indeed doing weird stuff with
multi-device btrfs volumes.
Oh, it's known that systemd does weird stuff with multi-device setups, 
even if they aren't BTRFS, but this is odd behavior that I've not seen 
anything about before (I don't use systemd myself except on the 
reference VM's I have for the various distros that use it by default, so 
I end up hearing about this stuff through word of mouth).


That said, some of the discussion in that first issue you mentioned 
actually provides a possible explanation for some of the issues existing 
at all, namely that there appears to be a disconnect between how BTRFS 
actually behaves and how the systemd developers thinks it behaves (the 
question from Pottering about what's doing the 'assembly' is 
particularly interesting, given that there is nothing other than the 
kernel that does the equivalent of the 'assembly' step for an MD array 
or scanning for an LVM array).

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Creating btrfs RAID on LUKS devs makes devices disappear

[Ochi]

On 12.05.2017 13:25, Austin S. Hemmelgarn wrote:

On 2017-05-11 19:24, Ochi wrote:

Hello,

here is the journal.log (I hope). It's quite interesting. I rebooted the
machine, performed a mkfs.btrfs on dm-{2,3,4} and dm-3 was missing
afterwards (around timestamp 66.*). However, I then logged into the
machine from another terminal (around timestamp 118.*) which triggered
something to make the device appear again :O Indeed, dm-3 was once again
there after logging in. Does systemd mix something up?

Hmm, I just did another mkfs once the devices where back, devices were
missing, but they re-appeared a few seconds later, without logging into
a terminal. After another mkfs, they were gone again and are now still
gone after waiting a few minutes. It's really weird, I can't really tell
what triggers this yet. Will test more tomorrow, let me know if you have
any more ideas what to try.


It looks like something made systemd think that it should tear down the 
LUKS volumes, but it somehow only got /dev/dm-3 and not the others, and 
then you logging in and triggering the creation of the associated user 
slice somehow made it regain it's senses.  Next time you see it 
disappear, try checking `systemctl status` for the unit you have set up 
for the LUKS volume and see what that says.  I doubt it will give much 
more info, but I suspect it will say it's stopped, which will solidify 
that systemd is either misconfigured or doing something stupid (and 
based on past experience, I'm willing to bet it's the latter).


I will take a closer look at systemd when I get home. I would like to 
point out that this sounds very related to these fairly recent systemd 
issues:


https://github.com/systemd/systemd/issues/5781

https://github.com/systemd/systemd/issues/5866

So my best guess is that systemd is indeed doing weird stuff with 
multi-device btrfs volumes.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Creating btrfs RAID on LUKS devs makes devices disappear

[Austin S. Hemmelgarn]

On 2017-05-11 19:24, Ochi wrote:

Hello,

here is the journal.log (I hope). It's quite interesting. I rebooted the
machine, performed a mkfs.btrfs on dm-{2,3,4} and dm-3 was missing
afterwards (around timestamp 66.*). However, I then logged into the
machine from another terminal (around timestamp 118.*) which triggered
something to make the device appear again :O Indeed, dm-3 was once again
there after logging in. Does systemd mix something up?

Hmm, I just did another mkfs once the devices where back, devices were
missing, but they re-appeared a few seconds later, without logging into
a terminal. After another mkfs, they were gone again and are now still
gone after waiting a few minutes. It's really weird, I can't really tell
what triggers this yet. Will test more tomorrow, let me know if you have
any more ideas what to try.


It looks like something made systemd think that it should tear down the 
LUKS volumes, but it somehow only got /dev/dm-3 and not the others, and 
then you logging in and triggering the creation of the associated user 
slice somehow made it regain it's senses.  Next time you see it 
disappear, try checking `systemctl status` for the unit you have set up 
for the LUKS volume and see what that says.  I doubt it will give much 
more info, but I suspect it will say it's stopped, which will solidify 
that systemd is either misconfigured or doing something stupid (and 
based on past experience, I'm willing to bet it's the latter).


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Creating btrfs RAID on LUKS devs makes devices disappear

[Ochi]

Hello,

here is the journal.log (I hope). It's quite interesting. I rebooted the 
machine, performed a mkfs.btrfs on dm-{2,3,4} and dm-3 was missing 
afterwards (around timestamp 66.*). However, I then logged into the 
machine from another terminal (around timestamp 118.*) which triggered 
something to make the device appear again :O Indeed, dm-3 was once again 
there after logging in. Does systemd mix something up?


Hmm, I just did another mkfs once the devices where back, devices were 
missing, but they re-appeared a few seconds later, without logging into 
a terminal. After another mkfs, they were gone again and are now still 
gone after waiting a few minutes. It's really weird, I can't really tell 
what triggers this yet. Will test more tomorrow, let me know if you have 
any more ideas what to try.


Best regards
Sebastian
-- Logs begin at Sun 2017-03-26 20:36:24 CEST, end at Fri 2017-05-12 01:00:45 CEST. --
[0.00] nas kernel: Linux version 4.9.27-1-lts (builduser@andyrtr) (gcc version 6.3.1 20170306 (GCC) ) #1 SMP Mon May 8 13:37:42 CEST 2017
[0.00] nas kernel: Command line: BOOT_IMAGE=/default/vmlinuz-linux-lts root=UUID=4ac09b56-3e02-40c0-bf64-02a4cf9344fc rw rootflags=subvol=default ip=192.168.0.3:eth0:none cryptdevice=/dev/sda2:root:allow-discards
[0.00] nas kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[0.00] nas kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[0.00] nas kernel: x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[0.00] nas kernel: x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[0.00] nas kernel: x86/fpu: xstate_offset[3]:  576, xstate_sizes[3]:   64
[0.00] nas kernel: x86/fpu: xstate_offset[4]:  640, xstate_sizes[4]:   64
[0.00] nas kernel: x86/fpu: Enabled xstate features 0x1b, context size is 704 bytes, using 'compacted' format.
[0.00] nas kernel: x86/fpu: Using 'eager' FPU context switches.
[0.00] nas kernel: e820: BIOS-provided physical RAM map:
[0.00] nas kernel: BIOS-e820: [mem 0x-0x0009c3ff] usable
[0.00] nas kernel: BIOS-e820: [mem 0x0009c400-0x0009] reserved
[0.00] nas kernel: BIOS-e820: [mem 0x000e-0x000f] reserved
[0.00] nas kernel: BIOS-e820: [mem 0x0010-0x78770fff] usable
[0.00] nas kernel: BIOS-e820: [mem 0x78771000-0x78771fff] ACPI NVS
[0.00] nas kernel: BIOS-e820: [mem 0x78772000-0x78772fff] reserved
[0.00] nas kernel: BIOS-e820: [mem 0x78773000-0x7e137fff] usable
[0.00] nas kernel: BIOS-e820: [mem 0x7e138000-0x7e5bafff] reserved
[0.00] nas kernel: BIOS-e820: [mem 0x7e5bb000-0x7e667fff] usable
[0.00] nas kernel: BIOS-e820: [mem 0x7e668000-0x7ea06fff] ACPI NVS
[0.00] nas kernel: BIOS-e820: [mem 0x7ea07000-0x7effefff] reserved
[0.00] nas kernel: BIOS-e820: [mem 0x7efff000-0x7eff] usable
[0.00] nas kernel: BIOS-e820: [mem 0x7f00-0x8fff] reserved
[0.00] nas kernel: BIOS-e820: [mem 0xe000-0xefff] reserved
[0.00] nas kernel: BIOS-e820: [mem 0xfe00-0xfe010fff] reserved
[0.00] nas kernel: BIOS-e820: [mem 0xfec0-0xfec00fff] reserved
[0.00] nas kernel: BIOS-e820: [mem 0xfed0-0xfed00fff] reserved
[0.00] nas kernel: BIOS-e820: [mem 0xfee0-0xfee00fff] reserved
[0.00] nas kernel: BIOS-e820: [mem 0xff00-0x] reserved
[0.00] nas kernel: BIOS-e820: [mem 0x0001-0x00046dff] usable
[0.00] nas kernel: NX (Execute Disable) protection: active
[0.00] nas kernel: SMBIOS 3.0 present.
[0.00] nas kernel: DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./C236 WSI, BIOS P2.10 04/18/2017
[0.00] nas kernel: e820: update [mem 0x-0x0fff] usable ==> reserved
[0.00] nas kernel: e820: remove [mem 0x000a-0x000f] usable
[0.00] nas kernel: e820: last_pfn = 0x46e000 max_arch_pfn = 0x4
[0.00] nas kernel: MTRR default type: write-back
[0.00] nas kernel: MTRR fixed ranges enabled:
[0.00] nas kernel:   0-9 write-back
[0.00] nas kernel:   A-B uncachable
[0.00] nas kernel:   C-F write-protect
[0.00] nas kernel: MTRR variable ranges enabled:
[0.00] nas kernel:   0 base 008000 mask 7F8000 uncachable
[0.00] nas kernel:   1 base 007F80 mask 7FFF80 uncachable
[0.00] nas kernel:   2 disabled
[0.00] nas kernel:   3 disabled
[0.00] nas kernel:   4 disabled
[0.00] nas kernel:   5 disabled
[0.00] 

Re: Creating btrfs RAID on LUKS devs makes devices disappear

[Chris Murphy]

journalctl -b -o short-monotonic > journal.log

And then attached the log, hopefully it's small enough to be accepted
by the list server (should be). If that's not revealing it might be
necessary to reboot with rd.udev.debug but start with the simple case
first and see if that reveals what's going on.



Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html