> On Feb 11, 2019, at 6:32 AM, Christian Kratzer <ck-li...@cksoft.de> wrote: > > I am running freebsd vm on debian 10 buster with libvirt/kvm/qemu. > > I have several kvm hosts in the cluster. Some with various intel xeon and > others with AMD EPYC 7301 cpu. > > FreeBSD vms upto 11.2-RELEASE-p9 boo fine on all systems when passing through > the host cpu using following libvirt xml > > <cpu mode='host-model' check='partial'> > <model fallback='allow'/> > </cpu>
Probably not the same issue, but this sounds similar to this bug I reported a few years ago: https://bugs.launchpad.net/qemu/+bug/1329956 It's just as likely to be a bug in Qemu or KVM as it is in FreeBSD IMO. Maybe you can start by determining which CPU feature or features trigger(s) the issue. You'll have to hand-roll either some libvirt XML or qemu command lines to do it. Assuming you want to stick with XML, first grab the CPU model and features list from `virsh capabilities`. Then start with just the model without any extra features (using AMD hardware I have access to as an example, replace "Opteron_G3" as appropriate): <cpu mode='custom' match='exact' check='partial'> <model fallback='forbid'>Opteron_G3</model> <topology sockets='1' cores='2' threads='1'/> </cpu> If that works, then add the other features a few at a time until you break it. Here's an example feature list from my same hardware. <cpu mode='custom' match='exact' check='partial'> <model fallback='forbid'>Opteron_G3</model> <topology sockets='1' cores='2' threads='1'/> <feature name='vme'/> <feature name='ht'/> <feature name='mmxext'/> <feature name='fxsr_opt'/> <feature name='pdpe1gb'/> <feature name='3dnowext'/> <feature name='3dnow'/> <feature name='cmp_legacy'/> <feature name='extapic'/> <feature name='cr8legacy'/> <feature name='3dnowprefetch'/> <feature name='osvw'/> <feature name='ibs'/> <feature name='skinit'/> <feature name='wdt'/> <feature name='nodeid_msr'/> <feature name='invtsc'/> </cpu> Once you identify the feature or features that cause things to break, you can report back here, look for open bugs in Qemu or KVM regarding those features, and/or open new bugs. > FreeBSD 12.0-RELEASE and later hang after boot when swithcing to usermode in > start_init: trying /sbin/init > > Following is dmesg from a succesfull boot of 12.0-RELEASE using host-model on > Intel CPU > > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights > reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-RELEASE-p3 GENERIC amd64 > FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on > LLVM 6.0.1) > VT(vga): text 80x25 > CPU: QEMU Virtual CPU version 2.1.0 (2400.13-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x663 Family=0x6 Model=0x6 Stepping=3 > > Features=0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2> > Features2=0x80a02001<SSE3,CX16,x2APIC,POPCNT,HV> > AMD Features=0x20100800<SYSCALL,NX,LM> > AMD Features2=0x1<LAHF> > Hypervisor: Origin = "KVMKVMKVM" > real memory = 1073741824 (1024 MB) > avail memory = 1000058880 (953 MB) > Event timer "LAPIC" quality 100 > ACPI APIC Table: <BOCHS BXPCAPIC> > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 4 package(s) x 1 core(s) > ... > > Following is dmesg from a succesfull boot of 12.0-RELEASE using host-model on > the qemu virtual cpu > > > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights > reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-RELEASE-p3 GENERIC amd64 > FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on > LLVM 6.0.1) > VT(vga): text 80x25 > CPU: QEMU Virtual CPU version 2.1.0 (2200.06-MHz K8-class CPU) > Origin="AuthenticAMD" Id=0x663 Family=0x6 Model=0x6 Stepping=3 > > Features=0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2> > Features2=0x80a02001<SSE3,CX16,x2APIC,POPCNT,HV> > AMD Features=0x20100800<SYSCALL,NX,LM> > AMD Features2=0x65<LAHF,SVM,ABM,SSE4A> > SVM: NAsids=16 > Hypervisor: Origin = "KVMKVMKVM" > real memory = 4294967296 (4096 MB) > avail memory = 4099080192 (3909 MB) > Event timer "LAPIC" quality 100 > ACPI APIC Table: <BOCHS BXPCAPIC> > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 4 package(s) x 1 core(s) > > Following is dmesg from a succesfull boot of 11.2-RELEASE using host-model on > AMD EPYC > > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights > reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 11.2-RELEASE-p9 #0: Tue Feb 5 15:30:36 UTC 2019 > r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC > amd64 > FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on > LLVM 6.0.0) > VT(vga): text 80x25 > CPU: AMD EPYC Processor (with IBPB) (2200.05-MHz K8-class CPU) > Origin="AuthenticAMD" Id=0x800f12 Family=0x17 Model=0x1 Stepping=2 > > Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2> > > Features2=0xfff83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> > AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> > AMD > Features2=0x8003f7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,PCXC> > Structured Extended > Features=0x201c01ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,SHA> > XSAVE Features=0x7<XSAVEOPT,XSAVEC,XINUSE> > AMD Extended Feature Extensions ID EBX=0x2001000 > SVM: NAsids=16 > Hypervisor: Origin = "KVMKVMKVM" > real memory = 4294967296 (4096 MB) > avail memory = 4088360960 (3898 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: <BOCHS BXPCAPIC> > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 4 package(s) > ... > > > And finally following is the failing verbose boot of 12.0-RELEASE on AMD-EPYC > > Table 'FACP' at 0xbffe104e > Table 'APIC' at 0xbffe10c2 > APIC: Found table at 0xbffe10c2 > APIC: Using the MADT enumerator. > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights > reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-RELEASE-p3 GENERIC amd64 > FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on > LLVM 6.0.1) > Table 'FACP' at 0xbffe104e > Table 'APIC' at 0xbffe10c2 > Table 'HPET' at 0xbffe1152 > ACPI: No SRAT table found > PPIM 0: PA=0xb8000, VA=0xffffffff82810000, size=0x8000, mode=0 > pmap: large map 8 PML4 slots (4096 Gb) > VT(vga): text 80x25 > Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff8263e000. > Preloaded boot_entropy_cache "/boot/entropy" at 0xffffffff826471d0. > Table 'FACP' at 0xbffe104e > FACP: Found table at 0xbffe104e > Calibrating TSC clock ... TSC clock: 2200048246 Hz > CPU: AMD EPYC Processor (with IBPB) (2200.05-MHz K8-class CPU) > Origin="AuthenticAMD" Id=0x800f12 Family=0x17 Model=0x1 Stepping=2 > > Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2> > > Features2=0xfff83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> > AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> > AMD > Features2=0x8003f7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,PCXC> > Structured Extended > Features=0x209c01ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA> > XSAVE Features=0x7<XSAVEOPT,XSAVEC,XINUSE> > AMD Extended Feature Extensions ID EBX=0x2001000 > SVM: Features=0x0 > Revision=1, ASIDs=16 > L1 2MB data TLB: 255 entries, 1-way associative > L1 2MB instruction TLB: 255 entries, 1-way associative > L1 4KB data TLB: 255 entries, 1-way associative > L1 4KB instruction TLB: 255 entries, 1-way associative > L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative > L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way > associative > L2 2MB unified TLB: 0 entries, disabled/not present > L2 4KB data TLB: 512 entries, 4-way associative > L2 4KB instruction TLB: 512 entries, 4-way associative > L2 unified cache: 512 kbytes, 64 bytes/line, 1 lines/tag, 16-way > associative > Hypervisor: Origin = "KVMKVMKVM" > real memory = 4294967296 (4096 MB) > Physical memory chunk(s): > 0x0000000000001000 - 0x000000000009efff, 647168 bytes (158 pages) > 0x0000000000103000 - 0x00000000001fffff, 1036288 bytes (253 pages) > 0x0000000002800000 - 0x00000000b7d44fff, 3042201600 bytes (742725 pages) > 0x0000000100000000 - 0x000000013ffe7fff, 1073643520 bytes (262120 pages) > avail memory = 4099080192 (3909 MB) > Table 'FACP' at 0xbffe104e > Table 'APIC' at 0xbffe10c2 > Table 'HPET' at 0xbffe1152 > ACPI: No DMAR table found > ... > ... > ... > ata0: stat0=0x00 err=0x00 lsb=0x00 msb=0x00 > ata0: stat1=0x00 err=0x00 lsb=0x00 msb=0x00 > ata0: reset tp2 stat0=00 stat1=00 devices=0x0 > ata1: reset tp1 mask=03 ostat0=00 ostat1=00 > ata1: stat0=0x00 err=0x00 lsb=0x00 msb=0x00 > ata1: stat1=0x00 err=0x00 lsb=0x00 msb=0x00 > ata1: reset tp2 stat0=00 stat1=00 devices=0x0 > Trying to mount root from ufs:/dev/vtbd0p2 [rw]... > atrtc0: providing initial system time > start_init: trying /sbin/init > uhub0: 2 ports with 2 removable, self powered > > After that the system freezes. > > I can provide more complete information if somebody tells me what would be > helpfull. > > How should I go about debugging this ??? > > Greetings > Christian > > -- > Christian Kratzer CK Software GmbH > Email: c...@cksoft.de Wildberger Weg 24/2 > Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden > Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart > Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer > Web: http://www.cksoft.de/ > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" > _______________________________________________ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
