[linux-lvm] Unusually long boot times with LVM Snapshots
Hi, I use LVM with full disk encryption. I also use LVM snapshots, currently I have a 70GB snapshot. Now the only problem is that the boot times when LVM snapshots are present are extremely long. I boot my machine, enter my encryption passphrase and then wait for about 3 minutes. It is only after that long wait does my machine boot. This only happens if snapshots are present, if no snapshots are present, boot is almost instantaneous after I enter my passphrase. Will adding --skip-mappings help ? Is there any way this can be solved ? -- Regards, Sreyan Chakravarty ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Unusually long boot times with LVM Snapshots
On Thu, Nov 19, 2020 at 07:46:21PM +0530, Sreyan Chakravarty wrote: > I use LVM with full disk encryption. I also use LVM snapshots, currently I > have a 70GB snapshot. What type of snapshot are you using? LVM2 allows either "classic" CoW snaps, or the newer thin provisioned snapshots using the dm-thinp target. Classic snapshots are known to have very poor IO performance when multiple snapshots of the same volume exist simultaneously (especially for write- heavy workloads). Thin provisioned snapshots are not normally activated at boot time unless they are explicitly requested (via dracut's rd.lvm.lv options) since they have the skip activation flag set by default. > I boot my machine, enter my encryption passphrase and then wait for about 3 > minutes. It is only after that long wait does my machine boot. > > This only happens if snapshots are present, if no snapshots are present, > boot is almost instantaneous after I enter my passphrase. Is there any indication in the log of what's happening during the delay? Look through the journalctl output to see if there are any messages logged while the delay happens. Another option is to use systemd-analyze to look into where the time is going during boot. It has various commands including "plot" which will generate an SVG plot of the boot timings on stdout. You can then compare that with a regular boot to try to understand the difference. Regards, Bryn. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Unusually long boot times with LVM Snapshots
On Fri, Nov 20, 2020 at 4:45 PM Bryn M. Reeves wrote: > What type of snapshot are you using? LVM2 allows either "classic" CoW > snaps, > or the newer thin provisioned snapshots using the dm-thinp target. > > Classic snapshots are known to have very poor IO performance when multiple > snapshots of the same volume exist simultaneously (especially for write- > heavy workloads). > > Thin provisioned snapshots are not normally activated at boot time unless > they are explicitly requested (via dracut's rd.lvm.lv options) since they > have the skip activation flag set by default. > > How can I check which type of snapshot I am using ? I am very interested in knowing more about these newer snapshots. I think I am using the older COW snapshots. I created it using: sudo lvcreate -L 70GB -s -n test_snapshot /dev/mapper/vgfedora-fedora Is there any indication in the log of what's happening during the delay? > Look through the journalctl output to see if there are any messages logged > while the delay happens. > > Yes, I have found the reason for the 3 minute delay: Nov 20 21:14:15 dracut-initqueue[902]: Scanning devices dm-0 for LVM logical volumes vgfedora/fedora Nov 20 21:14:16 dracut-initqueue[927]: inactive Original '/dev/vgfedora/fedora' [700.00 GiB] inherit Nov 20 21:14:16 dracut-initqueue[927]: inactive Snapshot '/dev/vgfedora/pre_kde_Nov_9' [70.00 GiB] inherit Nov 20 21:17:27 systemd[1]: Found device /dev/mapper/vgfedora-fedora. Nov 20 21:17:27 systemd[1]: Found device /dev/disk/by-uuid/03aef3ba-dca1-4cba-a3f5-36c5c0fe948e. Nov 20 21:17:27 systemd[1]: Reached target Initrd Root Device. As you can see it initializing the snapshots. This is my entire boot log if you need to delve deeper. https://pastebin.com/raw/275JPvZB > Another option is to use systemd-analyze to look into where the time is > going during boot. It has various commands including "plot" which will > generate an SVG plot of the boot timings on stdout. You can then compare > that with a regular boot to try to understand the difference. > Well I don't know about "plot" but this is the output from "systemd-analyze blame" 3min 30.737s dracut-initqueue.service 31.399s udisks2.service 26.650s lvm2-monitor.service 25.500s systemd-journal-flush.service 20.289s ModemManager.service 18.242s abrtd.service 18.233s avahi-daemon.service 18.227s bluetooth.service 17.725s systemd-cryptsetup@luks\x2d2ec7f1ae\x2d6f9b\x2d4896\x2da7b2\x2dbe7809e9d2f4.service 17.604s firewalld.service As you can see the 3min delay matches with the above LVM entry from the boot log. Again, I am very interested in knowing more about these newer snapshots. -- Regards, Sreyan Chakravarty ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Unusually long boot times with LVM Snapshots
On Fri, Nov 20, 2020 at 4:45 PM Bryn M. Reeves wrote: > What type of snapshot are you using? LVM2 allows either "classic" CoW > snaps, > or the newer thin provisioned snapshots using the dm-thinp target. > > Classic snapshots are known to have very poor IO performance when multiple > snapshots of the same volume exist simultaneously (especially for write- > heavy workloads). > > Thin provisioned snapshots are not normally activated at boot time unless > they are explicitly requested (via dracut's rd.lvm.lv options) since they > have the skip activation flag set by default. > I tried thin snapshots, but what use are they if I can't use my system all together. Once my root logical volume was marked read only I could not revert it back to writable, after the snapshot was created. -- Regards, Sreyan Chakravarty ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Unusually long boot times with LVM Snapshots
On Sat, Nov 21, 2020 at 01:51:42PM +0530, Sreyan Chakravarty wrote: > On Fri, Nov 20, 2020 at 4:45 PM Bryn M. Reeves wrote: > > > What type of snapshot are you using? LVM2 allows either "classic" CoW > > snaps, > > or the newer thin provisioned snapshots using the dm-thinp target. > > > > Classic snapshots are known to have very poor IO performance when multiple > > snapshots of the same volume exist simultaneously (especially for write- > > heavy workloads). > > > > Thin provisioned snapshots are not normally activated at boot time unless > > they are explicitly requested (via dracut's rd.lvm.lv options) since they > > have the skip activation flag set by default. > > > > I tried thin snapshots, but what use are they if I can't use my system all > together. I'm not following this statement. Can you explain what happened when you tried thin snapshots? > Once my root logical volume was marked read only I could not revert it back > to writable, after the snapshot was created. That's not a normal outcome of creating a snapshot - it sounds like you ran into some problem with the snaphot or the volume group, but without more details of the situation and what actually happened it's very hard to give you guidance to help avoid it in future. Regards, Bryn. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Unusually long boot times with LVM Snapshots
On Fri, Nov 20, 2020 at 09:48:10PM +0530, Sreyan Chakravarty wrote: > How can I check which type of snapshot I am using ? > > I am very interested in knowing more about these newer snapshots. > > I think I am using the older COW snapshots. > > I created it using: > > sudo lvcreate -L 70GB -s -n test_snapshot > /dev/mapper/vgfedora-fedora This is a classic (aka CoW/copy-on-write or "thick") snapshot since it was created with a size (-L 70GB). When creating a thin snapshot you do not specify a size for the snapshot LV - it consumes space from the associated thin pool as needed. Generally you can find information about the LVs on your system using the "lvs" command. The "Attr" field displays lv attributes including the volume type - see the NOTES section of 'man lvs' for the meaning of the values. The lvs report can also show you whether a snapshot is using CoW or thinp and the amount of data it is consuming. Regards, Bryn. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Unusually long boot times with LVM Snapshots
On Mon, Nov 23, 2020 at 6:43 PM Bryn M. Reeves wrote: > This is a classic (aka CoW/copy-on-write or "thick") snapshot since it > was created with a size (-L 70GB). When creating a thin snapshot you do > not specify a size for the snapshot LV - it consumes space from the > associated thin pool as needed. > > Generally you can find information about the LVs on your system using > the "lvs" command. The "Attr" field displays lv attributes including the > volume type - see the NOTES section of 'man lvs' for the meaning of the > values. The lvs report can also show you whether a snapshot is using CoW > or thinp and the amount of data it is consuming. > I have now migrated to thin snapshots: These are my logical volumes: # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert fedora vgfedora Vwi-a-tz-- 700.00g pool0016.61 pool00 vgfedora twi-aotz-- 929.75g 12.50 16.61 But I am unable to boot now due to these errors: Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning devices dm-0 for LVM logical volumes vgfedora/fedora Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive '/dev/vgfedora/pool00' [929.75 GiB] inherit Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive '/dev/vgfedora/fedora' [700.00 GiB] inherit Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe: FATAL: Module dm-thin-pool not found in directory /lib/modules/5.8.18-200.fc32.x86_64 Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: /usr/sbin/modprobe failed: 1 Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process LV vgfedora/fedora: thin target support missing from kernel? I have tried to mark the volumes as active by : lvchange -kn vgfedora/pool00 lvchange -kn vgfedora/fedora But no change. How do I add the dm-thin-pool module ? I can't even boot. -- Regards, Sreyan Chakravarty ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Unusually long boot times with LVM Snapshots
On Tue, Nov 24, 2020 at 05:27:21PM +0530, Sreyan Chakravarty wrote: > But I am unable to boot now due to these errors: > > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning > devices dm-0 for LVM logical volumes vgfedora/fedora > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive > '/dev/vgfedora/pool00' [929.75 GiB] inherit > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive > '/dev/vgfedora/fedora' [700.00 GiB] inherit > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe: > FATAL: Module dm-thin-pool not found in directory > /lib/modules/5.8.18-200.fc32.x86_64 > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: > /usr/sbin/modprobe failed: 1 > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process > LV vgfedora/fedora: thin target support missing from kernel? > > I have tried to mark the volumes as active by : > > lvchange -kn vgfedora/pool00 > lvchange -kn vgfedora/fedora > > But no change. > > > > How do I add the dm-thin-pool module ? I can't even boot. Most likely you are just missing the dm-thin-pool module in the dracut initramfs image. You can re-build it with the following command: # dracut -f /boot/initramfs-$(uname -r).img $(uname -r) If the running kernel matches the initramfs you want to rebuild then you can leave out the file and kernel version: # dracut -f To build an initramfs with only the drivers that the running host uses add the -H switch (the dm-thin-pool module must be loaded at the time you run dracut for this to work): # dracut -fH See the dracut man page for more details. Regards, Bryn. ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Unusually long boot times with LVM Snapshots
On Tue, Nov 24, 2020 at 5:27 PM Sreyan Chakravarty wrote: > > But I am unable to boot now due to these errors: > > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning > devices dm-0 for LVM logical volumes vgfedora/fedora > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive > '/dev/vgfedora/pool00' [929.75 GiB] inherit > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive > '/dev/vgfedora/fedora' [700.00 GiB] inherit > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe: > FATAL: Module dm-thin-pool not found in directory > /lib/modules/5.8.18-200.fc32.x86_64 > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: > /usr/sbin/modprobe failed: 1 > Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process > LV vgfedora/fedora: thin target support missing from kernel? > > I have tried to mark the volumes as active by : > > lvchange -kn vgfedora/pool00 > lvchange -kn vgfedora/fedora > > But no change. > > > > How do I add the dm-thin-pool module ? I can't even boot. > > Never mind, I have solved it by regenerating my initramfs. -- Regards, Sreyan Chakravarty ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] Unusually long boot times with LVM Snapshots
On Wed, Nov 25, 2020 at 8:50 PM Bryn M. Reeves wrote: > Most likely you are just missing the dm-thin-pool module in the dracut > initramfs image. > > You can re-build it with the following command: > > # dracut -f /boot/initramfs-$(uname -r).img $(uname -r) > > If the running kernel matches the initramfs you want to rebuild then > you can leave out the file and kernel version: > > # dracut -f > > To build an initramfs with only the drivers that the running host > uses add the -H switch (the dm-thin-pool module must be loaded at > the time you run dracut for this to work): > > # dracut -fH > > See the dracut man page for more details. > Yes, I did do the above thanks. -- Regards, Sreyan Chakravarty ___ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/