On Fri, Jul 8, 2022 at 10:28 AM G.R. <firemet...@users.sourceforge.net> wrote: > > On Fri, Jul 8, 2022 at 12:38 AM Jan Beulich <jbeul...@suse.com> wrote: > > > But the 'xl pci-assignable-remove' will lead to xl segmentation fault... > > >> [ 655.041442] xl[975]: segfault at 0 ip 00007f2cccdaf71f sp > > >> 00007ffd73a3d4d0 error 4 in libxenlight.so.4.16.0[7f2cccd92000+7c000] > > >> [ 655.041460] Code: 61 06 00 eb 13 66 0f 1f 44 00 00 83 c3 01 39 5c 24 > > >> 2c 0f 86 1b 01 00 00 48 8b 34 24 89 d8 4d 89 f9 4d 89 f0 4c 89 e9 4c 89 > > >> e2 <48> 8b 3c c6 31 c0 48 89 ee e8 53 44 fe ff 83 f8 04 75 ce 48 8b 44 > > > > That'll need debugging. Cc-ing Anthony for awareness, but I'm sure > > he'll need more data to actually stand a chance of doing something > > about it. > > > > Is there any chance you could be doing some debugging work yourself, > > at the very least to figure out where this (apparent) NULL deref is > > happening? > Yep, I can collect the call-stack for sure.
The call-stack of the segfault is like this: 0x00007ffff7f0971f in name2bdf () from /usr/lib/libxenlight.so.4.16 (gdb) bt #0 0x00007ffff7f0971f in name2bdf () from /usr/lib/libxenlight.so.4.16 #1 0x00007ffff7f0a75e in libxl_device_pci_assignable_remove () from /usr/lib/libxenlight.so.4.16 #2 0x00005555555725bf in main_pciassignable_remove () #3 0x00005555555610ab in main () It's with a release version of libxenlight. Once I switch it to a debug version, the segment fault just goes away... This allows me to move on and test the behavior on 4.16.1 -- unfortunately no change observed at all. Once I get the SSD assigned to the FreeeBSD 12 domU, the domU still sees the device but fails to operate. This time I also built the debug version of 4.16.1 hypervisor. But unfortunately it shares the same reboot on the first pci-assignable-add problem. I cannot follow the suggestion of attaching a serial console yet. The motherboard does have a serial port connector, but I do not have a cable at the moment. Maybe I can grab one, but it takes some time... What I was able to do is to dump the 'xl dmesg' output from the dom0 boot with a debug hypervisor (see attached). It does give a few extra lines and hope they could be helpful. Thanks, G.R.
Xen 4.16.1 (XEN) Xen version 4.16.1 (firemeteor@) (gcc (Debian 11.2.0-13) 11.2.0) debug=y Fri Jul 8 21:09:41 HKT 2022 (XEN) Latest ChangeSet: Wed Jul 6 16:22:55 2022 +0800 git:514aba9623 (XEN) build-id: 3e07b621cf5201a82b867a44ef1ad58a233e4ec8 (XEN) Bootloader: GRUB 2.04-20 (XEN) Command line: placeholder dom0_mem=2G,max:3G,min:1G dom0_max_vcpus=4 loglvl=all guest_loglvl=all iommu=verbose (XEN) Xen image load base address: 0x87a00000 (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds (XEN) Disc information: (XEN) Found 5 MBR signatures (XEN) Found 5 EDD information structures (XEN) CPU Vendor: Intel, Family 6 (0x6), Model 158 (0x9e), Stepping 10 (raw 000906ea) (XEN) Xen-e820 RAM map: (XEN) [0000000000000000, 000000000009d3ff] (usable) (XEN) [000000000009d400, 000000000009ffff] (reserved) (XEN) [00000000000e0000, 00000000000fffff] (reserved) (XEN) [0000000000100000, 00000000835bffff] (usable) (XEN) [00000000835c0000, 00000000835c0fff] (ACPI NVS) (XEN) [00000000835c1000, 00000000835c1fff] (reserved) (XEN) [00000000835c2000, 0000000088c0bfff] (usable) (XEN) [0000000088c0c000, 000000008907dfff] (reserved) (XEN) [000000008907e000, 00000000891f4fff] (usable) (XEN) [00000000891f5000, 00000000895dcfff] (ACPI NVS) (XEN) [00000000895dd000, 0000000089efefff] (reserved) (XEN) [0000000089eff000, 0000000089efffff] (usable) (XEN) [0000000089f00000, 000000008f7fffff] (reserved) (XEN) [00000000e0000000, 00000000efffffff] (reserved) (XEN) [00000000fe000000, 00000000fe010fff] (reserved) (XEN) [00000000fec00000, 00000000fec00fff] (reserved) (XEN) [00000000fee00000, 00000000fee00fff] (reserved) (XEN) [00000000ff000000, 00000000ffffffff] (reserved) (XEN) [0000000100000000, 000000086c7fffff] (usable) (XEN) ACPI: RSDP 000F05B0, 0024 (r2 ALASKA) (XEN) ACPI: XSDT 895120A8, 00D4 (r1 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: FACP 895509C0, 0114 (r6 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: DSDT 89512218, 3E7A6 (r2 ALASKA A M I 1072009 INTL 20160527) (XEN) ACPI: FACS 895DC080, 0040 (XEN) ACPI: APIC 89550AD8, 00F4 (r4 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: FPDT 89550BD0, 0044 (r1 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: FIDT 89550C18, 009C (r1 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: MCFG 89550CB8, 003C (r1 ALASKA A M I 1072009 MSFT 97) (XEN) ACPI: SSDT 89550CF8, 0204 (r1 ALASKA A M I 1000 INTL 20160527) (XEN) ACPI: SSDT 89550F00, 17D5 (r2 ALASKA A M I 3000 INTL 20160527) (XEN) ACPI: SSDT 895526D8, 933D (r1 ALASKA A M I 1 INTL 20160527) (XEN) ACPI: SSDT 8955BA18, 31C7 (r2 ALASKA A M I 3000 INTL 20160527) (XEN) ACPI: SSDT 8955EBE0, 2358 (r2 ALASKA A M I 1000 INTL 20160527) (XEN) ACPI: HPET 89560F38, 0038 (r1 ALASKA A M I 2 1000013) (XEN) ACPI: SSDT 89560F70, 1BE1 (r2 ALASKA A M I 1000 INTL 20160527) (XEN) ACPI: SSDT 89562B58, 0F9E (r2 ALASKA A M I 1000 INTL 20160527) (XEN) ACPI: SSDT 89563AF8, 2D1B (r2 ALASKA A M I 0 INTL 20160527) (XEN) ACPI: UEFI 89566818, 0042 (r1 ALASKA A M I 2 1000013) (XEN) ACPI: LPIT 89566860, 005C (r1 ALASKA A M I 2 1000013) (XEN) ACPI: SSDT 895668C0, 27DE (r2 ALASKA A M I 1000 INTL 20160527) (XEN) ACPI: SSDT 895690A0, 0FFE (r2 ALASKA A M I 0 INTL 20160527) (XEN) ACPI: DBGP 8956A0A0, 0034 (r1 ALASKA A M I 2 1000013) (XEN) ACPI: DBG2 8956A0D8, 0054 (r0 ALASKA A M I 2 1000013) (XEN) ACPI: DMAR 8956A130, 00A8 (r1 ALASKA A M I 2 1000013) (XEN) ACPI: WSMT 8956A1D8, 0028 (r1 ALASKA A M I 1072009 AMI 10013) (XEN) System RAM: 32597MB (33379452kB) (XEN) No NUMA configuration found (XEN) Faking a node at 0000000000000000-000000086c800000 (XEN) Domain heap initialised (XEN) found SMP MP-table at 000fce30 (XEN) SMBIOS 3.1 present. (XEN) Using APIC driver default (XEN) ACPI: PM-Timer IO Port: 0x1808 (24 bits) (XEN) ACPI: v5 SLEEP INFO: control[1:1804], status[1:1800] (XEN) ACPI: Invalid sleep control/status register data: 0:0x8:0x3 0:0x8:0x3 (XEN) ACPI: SLEEP INFO: pm1x_cnt[1:1804,1:0], pm1x_evt[1:1800,1:0] (XEN) ACPI: 32/64X FACS address mismatch in FADT - 895dc080/0000000000000000, using 32 (XEN) ACPI: wakeup_vec[895dc08c], vec_size[20] (XEN) ACPI: Local APIC address 0xfee00000 (XEN) Overriding APIC driver with bigsmp (XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) (XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119 (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) (XEN) ACPI: IRQ0 used by override. (XEN) ACPI: IRQ2 used by override. (XEN) ACPI: IRQ9 used by override. (XEN) Enabling APIC mode: Phys. Using 1 I/O APICs (XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000 (XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff (XEN) PCI: MCFG area at e0000000 reserved in E820 (XEN) PCI: Using MCFG for segment 0000 bus 00-ff (XEN) [VT-D]Host address width 39 (XEN) [VT-D]found ACPI_DMAR_DRHD: (XEN) [VT-D] dmaru->address = fed90000 (XEN) [VT-D]drhd->address = fed90000 iommu->reg = ffff82c00021d000 (XEN) [VT-D]cap = 1c0000c40660462 ecap = 19e2ff0505e (XEN) [VT-D] endpoint: 0000:00:02.0 (XEN) [VT-D]found ACPI_DMAR_DRHD: (XEN) [VT-D] dmaru->address = fed91000 (XEN) [VT-D]drhd->address = fed91000 iommu->reg = ffff82c00021f000 (XEN) [VT-D]cap = d2008c40660462 ecap = f050da (XEN) [VT-D] IOAPIC: 0000:00:1e.7 (XEN) [VT-D] MSI HPET: 0000:00:1e.6 (XEN) [VT-D] flags: INCLUDE_ALL (XEN) [VT-D]found ACPI_DMAR_RMRR: (XEN) [VT-D] endpoint: 0000:00:14.0 (XEN) [VT-D]dmar.c:617: RMRR: [899e0000,89c29fff] (XEN) [VT-D]found ACPI_DMAR_RMRR: (XEN) [VT-D] endpoint: 0000:00:02.0 (XEN) [VT-D]dmar.c:617: RMRR: [8b000000,8f7fffff] (XEN) Using ACPI (MADT) for SMP configuration information (XEN) SMP: Allowing 12 CPUs (0 hotplug CPUs) (XEN) IRQ limits: 120 GSI, 2376 MSI/MSI-X (XEN) [VT-D]qinval.c:421: QI: using 256-entry ring(s) (XEN) Switched to APIC driver x2apic_cluster (XEN) CPU0: TSC: ratio: 292 / 2 (XEN) CPU0: bus: 100 MHz base: 3500 MHz max: 4500 MHz (XEN) CPU0: 800 ... 3500 MHz (XEN) xstate: size: 0x440 and states: 0x1f (XEN) mce_intel.c:773: MCA Capability: firstbank 0, extended MCE MSR 0, BCAST, CMCI (XEN) CPU0: Intel machine check reporting enabled (XEN) Speculative mitigation facilities: (XEN) Hardware hints: (XEN) Hardware features: IBPB IBRS STIBP SSBD L1D_FLUSH MD_CLEAR (XEN) Compiled-in support: INDIRECT_THUNK SHADOW_PAGING (XEN) Xen settings: BTI-Thunk JMP, SPEC_CTRL: IBRS+ STIBP- SSBD-, Other: IBPB L1D_FLUSH VERW BRANCH_HARDEN (XEN) L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 39, Safe address 8000000000 (XEN) Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU MD_CLEAR (XEN) Support for PV VMs: MSR_SPEC_CTRL EAGER_FPU MD_CLEAR (XEN) XPTI (64-bit PV only): Dom0 enabled, DomU enabled (with PCID) (XEN) PV L1TF shadowing: Dom0 disabled, DomU enabled (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2) (XEN) Initializing Credit2 scheduler (XEN) load_precision_shift: 18 (XEN) load_window_shift: 30 (XEN) underload_balance_tolerance: 0 (XEN) overload_balance_tolerance: -3 (XEN) runqueues arrangement: socket (XEN) cap enforcement granularity: 10ms (XEN) load tracking window length 1073741824 ns (XEN) Disabling HPET for being unreliable (XEN) Platform timer is 3.580MHz ACPI PM Timer (XEN) Detected 3504.012 MHz processor. (XEN) Freed 1024kB unused BSS memory (XEN) alt table ffff82d04048b5b0 -> ffff82d040497d06 (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB (XEN) Intel VT-d Snoop Control not enabled. (XEN) Intel VT-d Dom0 DMA Passthrough not enabled. (XEN) Intel VT-d Queued Invalidation enabled. (XEN) Intel VT-d Interrupt Remapping enabled. (XEN) Intel VT-d Posted Interrupt not enabled. (XEN) Intel VT-d Shared EPT tables enabled. (XEN) I/O virtualisation enabled (XEN) - Dom0 mode: Relaxed (XEN) Interrupt remapping enabled (XEN) nr_sockets: 1 (XEN) Enabled directed EOI with ioapic_ack_old on! (XEN) ENABLING IO-APIC IRQs (XEN) -> Using old ACK method (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 (XEN) TSC deadline timer enabled (XEN) Allocated console ring of 128 KiB. (XEN) mwait-idle: MWAIT substates: 0x11142120 (XEN) mwait-idle: v0.4.1 model 0x9e (XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff (XEN) VMX: Supported advanced features: (XEN) - APIC MMIO access virtualisation (XEN) - APIC TPR shadow (XEN) - Extended Page Tables (EPT) (XEN) - Virtual-Processor Identifiers (VPID) (XEN) - Virtual NMI (XEN) - MSR direct-access bitmap (XEN) - Unrestricted Guest (XEN) - VMCS shadowing (XEN) - VM Functions (XEN) - Virtualisation Exceptions (XEN) - Page Modification Logging (XEN) HVM: ASIDs enabled. (XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207 (XEN) HVM: VMX enabled (XEN) HVM: Hardware Assisted Paging (HAP) detected (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB (XEN) alt table ffff82d04048b5b0 -> ffff82d040497d06 (XEN) Brought up 12 CPUs (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource (XEN) Adding cpu 0 to runqueue 0 (XEN) First cpu on runqueue, activating (XEN) Adding cpu 1 to runqueue 0 (XEN) Adding cpu 2 to runqueue 0 (XEN) Adding cpu 3 to runqueue 0 (XEN) Adding cpu 4 to runqueue 0 (XEN) Adding cpu 5 to runqueue 0 (XEN) Adding cpu 6 to runqueue 0 (XEN) Adding cpu 7 to runqueue 0 (XEN) Adding cpu 8 to runqueue 0 (XEN) Adding cpu 9 to runqueue 0 (XEN) Adding cpu 10 to runqueue 0 (XEN) Adding cpu 11 to runqueue 0 (XEN) mcheck_poll: Machine check polling timer started. (XEN) Running stub recovery selftests... (XEN) Fixup #UD[0000]: ffff82d07fffe044 [ffff82d07fffe044] -> ffff82d040386809 (XEN) Fixup #GP[0000]: ffff82d07fffe045 [ffff82d07fffe045] -> ffff82d040386809 (XEN) Fixup #SS[0000]: ffff82d07fffe044 [ffff82d07fffe044] -> ffff82d040386809 (XEN) Fixup #BP[0000]: ffff82d07fffe045 [ffff82d07fffe045] -> ffff82d040386809 (XEN) NX (Execute Disable) protection active (XEN) Dom0 has maximum 952 PIRQs (XEN) *** Building a PV Dom0 *** (XEN) ELF: phdr: paddr=0x1000000 memsz=0x1395d08 (XEN) ELF: phdr: paddr=0x2400000 memsz=0x685000 (XEN) ELF: phdr: paddr=0x2a85000 memsz=0x30d98 (XEN) ELF: phdr: paddr=0x2ab6000 memsz=0x376000 (XEN) ELF: memory: 0x1000000 -> 0x2e2c000 (XEN) ELF: note: GUEST_OS = "linux" (XEN) ELF: note: GUEST_VERSION = "2.6" (XEN) ELF: note: XEN_VERSION = "xen-3.0" (XEN) ELF: note: VIRT_BASE = 0xffffffff80000000 (XEN) ELF: note: INIT_P2M = 0x8000000000 (XEN) ELF: note: ENTRY = 0xffffffff82ab6160 (XEN) ELF: note: HYPERCALL_PAGE = 0xffffffff81002000 (XEN) ELF: note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb" (XEN) ELF: note: SUPPORTED_FEATURES = 0x8801 (XEN) ELF: note: PAE_MODE = "yes" (XEN) ELF: note: LOADER = "generic" (XEN) ELF: note: unknown (0xd) (XEN) ELF: note: SUSPEND_CANCEL = 0x1 (XEN) ELF: note: MOD_START_PFN = 0x1 (XEN) ELF: note: HV_START_LOW = 0xffff800000000000 (XEN) ELF: note: PADDR_OFFSET = 0 (XEN) ELF: note: PHYS32_ENTRY = 0x10004b0 (XEN) ELF: addresses: (XEN) virt_base = 0xffffffff80000000 (XEN) elf_paddr_offset = 0x0 (XEN) virt_offset = 0xffffffff80000000 (XEN) virt_kstart = 0xffffffff81000000 (XEN) virt_kend = 0xffffffff82e2c000 (XEN) virt_entry = 0xffffffff82ab6160 (XEN) p2m_base = 0x8000000000 (XEN) Xen kernel: 64-bit, lsb (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x2e2c000 (XEN) PHYSICAL MEMORY ARRANGEMENT: (XEN) Dom0 alloc.: 0000000850000000->0000000854000000 (504314 pages to be allocated) (XEN) Init. ramdisk: 000000086b9fa000->000000086c7ff3f6 (XEN) VIRTUAL MEMORY ARRANGEMENT: (XEN) Loaded kernel: ffffffff81000000->ffffffff82e2c000 (XEN) Phys-Mach map: 0000008000000000->0000008000400000 (XEN) Start info: ffffffff82e2c000->ffffffff82e2c4b8 (XEN) Page tables: ffffffff82e2d000->ffffffff82e48000 (XEN) Boot stack: ffffffff82e48000->ffffffff82e49000 (XEN) TOTAL: ffffffff80000000->ffffffff83000000 (XEN) ENTRY ADDRESS: ffffffff82ab6160 (XEN) Dom0 has maximum 4 VCPUs (XEN) ELF: phdr 0 at 0xffffffff81000000 -> 0xffffffff82395d08 (XEN) ELF: phdr 1 at 0xffffffff82400000 -> 0xffffffff82a85000 (XEN) ELF: phdr 2 at 0xffffffff82a85000 -> 0xffffffff82ab5d98 (XEN) ELF: phdr 3 at 0xffffffff82ab6000 -> 0xffffffff82be0000 (XEN) [VT-D]iommu_enable_translation: iommu->reg = ffff82c00021d000 (XEN) [VT-D]iommu_enable_translation: iommu->reg = ffff82c00021f000 (XEN) Initial low memory virq threshold set at 0x4000 pages. (XEN) Scrubbing Free RAM in background (XEN) Std. Loglevel: All (XEN) Guest Loglevel: All (XEN) *************************************************** (XEN) Booted on L1TF-vulnerable hardware with SMT/Hyperthreading (XEN) enabled. Please assess your configuration and choose an (XEN) explicit 'smt=<bool>' setting. See XSA-273. (XEN) *************************************************** (XEN) Booted on MLPDS/MFBDS-vulnerable hardware with SMT/Hyperthreading (XEN) enabled. Mitigations will not be fully effective. Please (XEN) choose an explicit smt=<bool> setting. See XSA-297. (XEN) *************************************************** (XEN) 3... 2... 1... (XEN) Xen is relinquishing VGA console. (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input) (XEN) Freed 620kB init memory (XEN) d0: Forcing write emulation on MFNs e0000-effff (XEN) PCI add device 0000:00:00.0 (XEN) PCI add device 0000:00:01.0 (XEN) PCI add device 0000:00:02.0 (XEN) PCI add device 0000:00:12.0 (XEN) PCI add device 0000:00:14.0 (XEN) PCI add device 0000:00:14.2 (XEN) PCI add device 0000:00:16.0 (XEN) PCI add device 0000:00:16.3 (XEN) PCI add device 0000:00:17.0 (XEN) PCI add device 0000:00:1b.0 (XEN) PCI add device 0000:00:1c.0 (XEN) PCI add device 0000:00:1c.5 (XEN) PCI add device 0000:00:1d.0 (XEN) PCI add device 0000:00:1f.0 (XEN) PCI add device 0000:00:1f.3 (XEN) PCI add device 0000:00:1f.4 (XEN) PCI add device 0000:00:1f.5 (XEN) PCI add device 0000:01:00.0 (XEN) PCI add device 0000:04:00.0 (XEN) PCI add device 0000:05:00.0 (XEN) emul-priv-op.c:1025:d0v2 RDMSR 0x00000639 unimplemented (XEN) emul-priv-op.c:1025:d0v2 RDMSR 0x00000611 unimplemented (XEN) emul-priv-op.c:1025:d0v2 RDMSR 0x00000619 unimplemented (XEN) emul-priv-op.c:1025:d0v2 RDMSR 0x00000641 unimplemented (XEN) emul-priv-op.c:1025:d0v2 RDMSR 0x0000064d unimplemented (XEN) emul-priv-op.c:1025:d0v2 RDMSR 0x00000606 unimplemented (XEN) emul-priv-op.c:1025:d0v2 RDMSR 0x0000064e unimplemented (XEN) emul-priv-op.c:1025:d0v2 RDMSR 0x00000034 unimplemented (XEN) d0: Forcing read-only access to MFN fed00 (XEN) emul-priv-op.c:1025:d0v3 RDMSR 0xc0011020 unimplemented
