Re: [linux-lvm] " Failed to find physical volume
A possibility I just debugged for a non-booting system. If there is a partition table on the underlying device then that device is not detected as an LVM1/2 member in at least one version of udevd, and won't be seen nor turned on automatically by the systemd-udev code. lvm vgchange -ay worked to enable it (emergency mode, it was the root pv--no udevd involvement) and eventually I found the partition table and removed it and the machine would then boot without needing a manual intervention. dd if=/dev/zero of=device bs=512 count=1 was used once we determined there was a partition signature still left (after partition deletion with fdisk, still had a header), examined with dd if=/dev/device bs=512 count=1 | xxd found 4 non-zero bytes in the block. On Sun, Sep 26, 2021 at 9:22 AM alessandro macuz wrote: > > Thanks Roger, Zdenek, > > I have my ZVOL on my NAS exposed as LUNs. The initiator were switched off and > for unknown reason I found my NAS switched off as well. > It had run for long and I feared the worst (CPU/motherboard/etc). Instead > once powered up everything started to work again but the LUNs that seemed to > jeopardized. > I have many ZVOLs used by ESXi in which I have EVE-NG who uses LVM and such > ZVOLs have the same size so I wanted to inspect them to check the hostname. > > Now some LUNs started to work normally, some others still behave weirdly. I > will run pvs on them with extra debugs to see what's going on. > > Many thanks, > > Alex > > Le jeu. 23 sept. 2021 à 23:48, Roger Heflin a écrit : >> >> If you have lvmetad running and in use then the lvm commands ask it >> what the system has on it. >> >> I have seen on random boots fairly separated systems (rhel7 versions, >> and many years newer fedora systems) at random fail to find one or >> more pv.s >> >> I have disabled it at home, and in my day job we have also disabled >> (across 20k+ systems) as we confirmed it had inconsistency issues >> several times on a variety of our newest installs. >> >> Stopping lvmetad and/or restarting it would generally fix it.But >> it was a source of enough random issues(often failure to mount on a >> boot, so often issues that resulted in page-outs to debug) and did >> not speed things up much enough to be worth it even on devices with >> >2000 SAN volumes. >> >> On Thu, Sep 23, 2021 at 8:52 AM Zdenek Kabelac wrote: >> > >> > Dne 22. 09. 21 v 18:48 alessandro macuz napsal(a): >> > > fdisk correctly identifies the extended partition as 8e. >> > > I wonder which kind of data lvmdiskscan and pvs use in order to list LVM >> > > physical volumes. >> > > Does PVS check some specific metadata within the partition without just >> > > relying on the type of partition displayed by fdisk? >> > > >> > > >> > >> > Hi >> > >> > Yes - PVs do have header signature keeping information about PV attributes >> > and also has the storage area to keep lvm2 metadata. >> > >> > Partition flags known to fdisk are irrelevant. >> > >> > >> > Regards >> > >> > Zdenek >> > >> > ___ >> > linux-lvm mailing list >> > linux-lvm@redhat.com >> > https://listman.redhat.com/mailman/listinfo/linux-lvm >> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >> > ___ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Discussion: performance issue on event activation mode
On Tue, Sep 28, 2021 at 03:16:08PM +, Martin Wilck wrote: > Hm. This would mean that the switch to event-based PV detection could > happen before "udev settle" ends. A coldplug storm of uevents could > create 1000s of PVs in a blink after event-based detection was enabled. > Wouldn't that resurrect the performance issues that you are trying to > fix with this patch set? Possibly, I'm unsure how this looks in practice, so I need to try it. When the device node exists will make a difference, not only when the uevent occurs. > > Otherwise, when the devices file is not used, > > md: from reading the md headers from the disk > > mpath: from reading sysfs links and /etc/multipath/wwids > > Ugh. Reading sysfs links means that you're indirectly depending on > udev, because udev creates those. It's *more* fragile than calling into > libudev directly, IMO. I meant /sys/dev/block/... (some of those files are links). We don't look at /dev symlinks created by udev. > Using /etc/multipath/wwids is plain wrong in > general. It works only on distros that use "find_multipaths strict", > like RHEL. Not to mention that the path can be customized in > multipath.conf. Right, it's not great and I held off for a couple years adding that. As a practical matter it can at least help. There's a constant stream of problems with mpath component detection, so anything that can help I'm interested in. I expect we could be more intelligent understanding multipath config to handle more cases. > multipathd does listen to uevents (only "udev" events, not "kernel"). > But that doesn't help us on startup. Currently we try hard to start up > after coldplug is finished. multipathd doesn't have a concurrency issue > like LVM2 (at least I hope so; it handles events with just two threads, > a producer and a consumer). The problem is rather that dm devices > survive the initramfs->rootfs switch, while member devices don't (see > above). The other day I suggested that multipath devices not be set up in the initramfs at all. If the root fs requires mpath, then handle that as a special one-off setup. Then the transition problems go away. But, I know basically nothing about this, so I won't be surprised if there are reasons it's done this way. Dave ___ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Discussion: performance issue on event activation mode
On Tue, Sep 28, 2021 at 06:34:06AM +, Martin Wilck wrote:
> Hello David and Peter,
>
> On Mon, 2021-09-27 at 10:38 -0500, David Teigland wrote:
> > On Mon, Sep 27, 2021 at 12:00:32PM +0200, Peter Rajnoha wrote:
> > > > - We could use the new lvm-activate-* services to replace the
> > > > activation
> > > > generator when lvm.conf event_activation=0. This would be done by
> > > > simply
> > > > not creating the event-activation-on file when event_activation=0.
> > >
> > > ...the issue I see here is around the systemd-udev-settle:
> >
> > Thanks, I have a couple questions about the udev-settle to understand
> > that
> > better, although it seems we may not need it.
> >
> > > - the setup where lvm-activate-vgs*.service are always there (not
> > > generated only on event_activation=0 as it was before with the
> > > original lvm2-activation-*.service) practically means we always
> > > make a dependency on systemd-udev-settle.service, which we
> > > shouldn't
> > > do in case we have event_activation=1.
> >
> > Why wouldn't the event_activation=1 case want a dependency on udev-
> > settle?
>
> You said it should wait for multipathd, which in turn waits for udev
> settle. And indeed it makes some sense. After all: the idea was to
> avoid locking issues or general resource starvation during uevent
> storms, which typically occur in the coldplug phase, and for which the
> completion of "udev settle" is the best available indicator.
Hi Martin, thanks, you have some interesting details here.
Right, the idea is for lvm-activate-vgs-last to wait for other services
like multipath (or anything else that a PV would typically sit on), so
that it will be able to activate as many VGs as it can that are present at
startup. And we avoid responding to individual coldplug events for PVs,
saving time/effort/etc.
> I'm arguing against it (perhaps you want to join in :-), but odds are
> that it'll disappear sooner or later. Fot the time being, I don't see a
> good alternative.
multipath has more complex udev dependencies, I'll be interested to see
how you manage to reduce those, since I've been reducing/isolating our
udev usage also.
> The dependency type you have to use depends on what you need. Do you
> really only depend on udev settle because of multipathd? I don't think
> so; even without multipath, thousands of PVs being probed
> simultaneously can bring the performance of parallel pvscans down. That
> was the original motivation for this discussion, after all. If this is
> so, you should use both "Wants" and "After". Otherwise, using only
> "After" might be sufficient.
I don't think we really need the settle. If device nodes for PVs are
present, then vgchange -aay from lvm-activate-vgs* will see them and
activate VGs from them, regardless of what udev has or hasn't done with
them yet.
> > - Reading the udev db: with the default
> > external_device_info_source=none
> > we no longer ask the udev db for any info about devs. (We now
> > follow that setting strictly, and only ask udev when source=udev.)
>
> This is a different discussion, but if you don't ask udev, how do you
> determine (reliably, and consistently with other services) whether a
> given device will be part of a multipath device or a MD Raid member?
Firstly, with the new devices file, only the actual md/mpath device will
be in the devices file, the components will not be, so lvm will never
attempt to look at an md or mpath component device.
Otherwise, when the devices file is not used,
md: from reading the md headers from the disk
mpath: from reading sysfs links and /etc/multipath/wwids
> In the past, there were issues with either pvscan or blkid (or
> multipath) failing to open a device while another process had opened it
> exclusively. I've never understood all the subtleties. See systemd
> commit 3ebdb81 ("udev: serialize/synchronize block device event
> handling with file locks").
Those locks look like a fine solution if a problem comes up like that.
I suspect the old issues may have been caused by a program using an
exclusive open when it shouldn't.
> After=udev-settle will make sure that you're past a coldplug uevent
> storm during boot. IMO this is the most important part of the equation.
> I'd be happy to find a solution for this that doesn't rely on udev
> settle, but I don't see any.
I don't think multipathd is listening to uevents directly? If it were,
you might use a heuristic to detect a change in uevents (e.g. the volume)
and conclude coldplug is finished.
Dave
___
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
