So, the hidden disk was introduced by hch as commit 8ddcd653257c18a669fcb75ee42c37054908e0d6 just to support nvme multipath. This thing prevented lsblk from working when there were slaves under the multipath nvme, as lsblk would postpone processing those, and lsblk could not process the paths themselves, because they have no dev_t.
Some discussion about reverting that slaves/holders patch is under the revert submission, and they decided to go on with the revert and think about how to fix that later. A link for one of the discussion messages is below. https://www.spinics.net/lists/stable/msg222846.html Going forward, we could either use nvme cli, embed the assumption that kernel won't change and some of the numbers are kept (like the subsys instance is embedded in both the multipath device and the underlying devices), or introduce this new paths subdir in the kernel. Any of those solutions would require changes to initramfs-tools. The latter would require a kernel change. The only way of not changing initramfs-tools is going back to using slaves relationships. That means we would need to get rid of the hidden disks thing. This always worked for dm-multipath, but I believe this required some use of blkdev and that is one thing hch wanted to avoid, to have a very fast path for doing multipath. I don't like the idea of using nvme cli that much, as it would be a hard dependency on initramfs-tools. I don't like assuming some of the implementation details on the kernel numbering/naming of those devices. So, I will take a stab at providing the paths interface. Cascardo. ** Also affects: initramfs-tools (Ubuntu Cosmic) Importance: Critical Assignee: Canonical Kernel Team (canonical-kernel-team) Status: New ** Also affects: linux (Ubuntu Cosmic) Importance: Critical Assignee: Canonical Kernel Team (canonical-kernel-team) Status: New ** Also affects: makedumpfile (Ubuntu Cosmic) Importance: Critical Assignee: Canonical Kernel Team (canonical-kernel-team) Status: New ** Changed in: makedumpfile (Ubuntu Cosmic) Status: New => Invalid ** Changed in: makedumpfile (Ubuntu Bionic) Status: New => Invalid ** Changed in: linux (Ubuntu Bionic) Status: New => Confirmed ** Changed in: linux (Ubuntu Cosmic) Status: New => Confirmed ** Changed in: initramfs-tools (Ubuntu Cosmic) Status: New => Confirmed ** Changed in: initramfs-tools (Ubuntu Bionic) Status: New => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: nvme multipath does not report path relationships Status in The Ubuntu-power-systems project: Triaged Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Status in makedumpfile package in Ubuntu: Invalid Status in initramfs-tools source package in Bionic: Confirmed Status in linux source package in Bionic: Confirmed Status in makedumpfile source package in Bionic: Invalid Status in initramfs-tools source package in Cosmic: Confirmed Status in linux source package in Cosmic: Confirmed Status in makedumpfile source package in Cosmic: Invalid Bug description: Problem Description: =================== After triggering crash ,kdump is not working & system enters into initramfs state Steps to re-create: ================== >. woo is installed ubuntu180401 kernel root@woo:~# uname -a Linux woo 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 17:59:00 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux root@woo:~# >. Crashkernel value as below root@woo:~# free -h total used free shared buff/cache available Mem: 503G 2.0G 501G 13M 279M 499G Swap: 2.0G 0B 2.0G root@woo:~# cat /proc/cmdline root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet crashkernel=8192M > kdump status root@woo:~# kdump-config status current state : ready to kdump root@woo:~# kdump-config show DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_SYSCTL: kernel.panic_on_oops=1 KDUMP_COREDIR: /var/crash crashkernel addr: /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.15.0-23-generic kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.15.0-23-generic current state: ready to kdump kexec command: /sbin/kexec -p --command-line="root=UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb ro splash quiet nr_cpus=1 systemd.unit=kdump-tools.service irqpoll noirqdistrib nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz root@woo:~# dmesg | grep Reser [ 0.000000] Reserving 8192MB of memory at 128MB for crashkernel (System RAM: 524288MB) [ 0.000000] cma: Reserved 26224 MiB at 0x0000203995000000 [ 3.545490] Copyright (C) 2017-2018 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries. > Triggered crash root@woo:~# echo 1 > /proc/sys/kernel/sysrq root@woo:~# echo c > /proc/sysrq-trigger [ 73.056308] sysrq: SysRq : Trigger a crash [ 73.056357] Unable to handle kernel paging request for data at address 0x00000000 [ 73.056459] Faulting instruction address: 0xc0000000007f24c8 [ 73.056543] Oops: Kernel access of bad area, sig: 11 [#1] [ 73.056609] LE SMP NR_CPUS=2048 NUMA PowerNV [ 73.056668] Modules linked in: rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) rpcsec_gss_krb5 nfsv4 nfs fscache binfmt_misc idt_89hpesx vmx_crypto crct10dif_vpmsum ofpart cmdlinepart ipmi_powernv ipmi_devintf at24 powernv_flash ipmi_msghandler ibmpowernv mtd opal_prd uio_pdrv_genirq uio nfsd auth_rpcgss nfs_acl lockd sch_fq_codel grace sunrpc knem(OE) ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq mlx5_ib(OE) ib_core(OE) nouveau lpfc ast i2c_algo_bit ttm mlx5_core(OE) drm_kms_helper mlxfw(OE) nvmet_fc devlink syscopyarea nvmet mlx_compat(OE) sysfillrect cxl nvme_fc sysimgblt fb_sys_fops nvme_fabrics nvme ahci crc32c_vpmsum drm scsi_transport_fc [ 73.057601] tg3 libahci nvme_core pnv_php [ 73.057652] CPU: 44 PID: 4626 Comm: bash Tainted: G OE 4.15.0-23-generic #25-Ubuntu [ 73.057767] NIP: c0000000007f24c8 LR: c0000000007f3568 CTR: c0000000007f24a0 [ 73.057868] REGS: c000003f8269f9f0 TRAP: 0300 Tainted: G OE (4.15.0-23-generic) [ 73.057986] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28222222 XER: 20040000 [ 73.058099] CFAR: c0000000007f3564 DAR: 0000000000000000 DSISR: 42000000 SOFTE: 1 [ 73.058099] GPR00: c0000000007f3568 c000003f8269fc70 c0000000016eaf00 0000000000000063 [ 73.058099] GPR04: c000003fef47ce18 c000003fef494368 9000000000009033 0000000031da0058 [ 73.058099] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003 [ 73.058099] GPR12: c0000000007f24a0 c000000007a2e400 00000e4fa497c900 0000000000000000 [ 73.058099] GPR16: 00000e4f79cc94b0 00000e4f79d567e0 00000e4f79d88204 00000e4f79d56818 [ 73.058099] GPR20: 00000e4f79d8d5d8 0000000000000001 0000000000000000 00007ffffefce644 [ 73.058099] GPR24: 00007ffffefce640 00000e4f79d8afb4 c0000000015e9aa8 0000000000000002 [ 73.058099] GPR28: 0000000000000063 0000000000000004 c000000001572b1c c0000000015e9e68 [ 73.059060] NIP [c0000000007f24c8] sysrq_handle_crash+0x28/0x30 [ 73.059142] LR [c0000000007f3568] __handle_sysrq+0xf8/0x2c0 [ 73.059215] Call Trace: [ 73.059254] [c000003f8269fc70] [c0000000007f3548] __handle_sysrq+0xd8/0x2c0 (unreliable) [ 73.059358] [c000003f8269fd10] [c0000000007f3d74] write_sysrq_trigger+0x64/0x90 [ 73.059456] [c000003f8269fd40] [c000000000481248] proc_reg_write+0x88/0xd0 [ 73.059543] [c000003f8269fd70] [c0000000003d43fc] __vfs_write+0x3c/0x70 [ 73.059627] [c000003f8269fd90] [c0000000003d4658] vfs_write+0xd8/0x220 [ 73.059716] [c000003f8269fde0] [c0000000003d4978] SyS_write+0x68/0x110 [ 73.059809] [c000003f8269fe30] [c00000000000b284] system_call+0x58/0x6c [ 73.059896] Instruction dump: [ 73.059940] 4bfff9f1 4bfffe50 3c4c00f0 38428a60 7c0802a6 60000000 39200001 3d42001c [ 73.060040] 394a6db0 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f0 38428a30 [ 73.060159] ---[ end trace e116d2421d2f59a5 ]--- [ 74.067059] [ 74.067172] Sending IPI to other CPUs [ 75.851509[ 202.275317797,5] OPAL: Switch to big-endian OS ] IPI complet[ 207.151277658,5] OPAL: Switch to little-endian OS [ 232.159296542,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting... e [ 78.164472] kexec: Starting switchover sequence. [ 1.412463] integrity: Unable to open file: /etc/keys/x509_ima.der (-2) [ 1.412468] integrity: Unable to open file: /etc/keys/x509_evm.der (-2) [ 1.481335] vio vio: uevent: failed to send synthetic uevent [ 2.534732] nouveau 0004:04:00.0: unknown chipset (140000a1) [ 2.534847] nouveau 0004:05:00.0: unknown chipset (140000a1) [ 2.534967] nouveau 0035:03:00.0: unknown chipset (140000a1) [ 2.535144] nouveau 0035:04:00.0: unknown chipset (140000a1) Gave up waiting for root file system device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Missing modules (cat /proc/modules; ls /dev) ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell! BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs) (initramfs) == Comment: #1 - INDIRA P. JOGA <> - 2018-06-21 01:19:41 == Attached woo console logs for kdump issue == Comment: #4 - INDIRA P. JOGA <> - 2018-06-26 01:51:47 == I have triggered crash & sits here [ 1032.259696471,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting... omplete [ 823.882048] kexec: Starting switchover sequence. [ 1.154056] integrity: Unable to open file: /etc/keys/x509_ima.der (-2) [ 1.154060] integrity: Unable to open file: /etc/keys/x509_evm.der (-2) [ 1.222719] vio vio: uevent: failed to send synthetic uevent [ 2.212065] nouveau 0004:04:00.0: unknown chipset (140000a1) [ 2.214995] nouveau 0004:05:00.0: unknown chipset (140000a1) [ 2.215259] nouveau 0035:03:00.0: unknown chipset (140000a1) [ 2.215408] nouveau 0035:04:00.0: unknown chipset (140000a1) Gave up waiting for root file system device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Missing modules (cat /proc/modules; ls /dev) ALERT! UUID=45bb7eb2-4c61-425d-8bf9-4e6f16829ddb does not exist. Dropping to a shell! BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs) == Comment: #5 - INDIRA P. JOGA <> - 2018-06-26 02:40:56 == NOTE: Used nvme disk as root disk here. == Comment: #8 - Hari Krishna Bathini <> - 2018-06-26 06:06:13 == The dump target (/var/crash) is on NVMe device (also, the root disk). But the kdump initrd is not being built with nvme driver modules. Eventually, nvme disk is not found and kdump kernel is hitting the initramfs shell. Using the default initrd, which has the nvme driver modules included, dump was captured successfully. Can someone from Canonical take a look at this and comment on why nvme modules are not included in kdump initrd despite it being the root disk.. Thanks Hari To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp