On Wed, Oct 16, 2019 at 06:14:55PM +0200, Mischa wrote: > Hi Stuart, > > > > On 16 Oct 2019, at 18:07, Stuart Henderson <s...@spacehopper.org> wrote: > > > > On 2019/10/16 18:00, Mischa wrote: > >> Hi All, > >> > >> One of the OpenBSD VMs running on 6.6-beta #313 is rebooting or panicing > >> in different ways. > >> Not sure if they are all relevant or useful but here are the ones we > >> managed to capture. > > > > There's not a lot of information in your mail... for starters, what are > > you running the VM in, and is there any difference in the config for that > > VM compared to other working ones? > > Fair point. > > There are 10 VMs running on this host, the host is running: > $ sysctl kern.version > kern.version=OpenBSD 6.6-beta (GENERIC.MP) #313: Tue Sep 10 23:30:52 MDT 2019 > dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP > > I know of one other VM which is rebooting every once in a while, but haven’t > seen any panics. > As for the other 8 I can see every once in a while a VM shutdown. But no > capture of the console. > > > Do you have any other VMs running the same OpenBSD snapshot successfully? > > The rest of the VMs are on -stable as far as I am aware. Other people are > operating these VMS. > > > Can you boot an old kernel and get a dmesg? > > Here is a dmesg which we manage to capture after one of the panics: >
Are you in swap at all on that host? -ml > fd0# panic: mtx 0xffffffff81f353f0: locking against myself > Using drive 0, partition 3. > Loading...... > probing: pc0 com0 mem[638K 510M a20=on] > disk: hd0+ hd1+ > >> OpenBSD/amd64 BOOT 3.45 > / > com0: 115200 baud > switching console to com0 > >> OpenBSD/amd64 BOOT 3.45 > boot> > booting hd0a:/bsd: 12666184+2937864+332896+0+704512 > [987630+128+1010256+738953]=0x127d750 > entry point at 0xffffffff81001000 > [ using 2738000 bytes of bsd ELF symbol table ] > Copyright (c) 1982, 1986, 1989, 1991, 1993 > The Regents of the University of California. All rights reserved. > Copyright (c) 1995-2019 OpenBSD. All rights reserved. https://www.OpenBSD.org > > OpenBSD 6.6 (GENERIC) #325: Wed Oct 2 11:38:13 MDT 2019 > dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC > real mem = 520077312 (495MB) > avail mem = 491753472 (468MB) > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf3f40 (10 entries) > bios0: vendor SeaBIOS version "1.11.0p2-OpenBSD-vmm" date 01/01/2011 > bios0: OpenBSD VMM > acpi at bios0 not configured > cpu0 at mainbus0: (uniprocessor) > cpu0: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3101.63 MHz, 06-3a-09 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,LONG,LAHF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,MELTDOWN > cpu0: 256KB 64b/line 8-way L2 cache > tsc_timecounter_init: TSC skew=0 observed drift=0 > cpu0: smt 0, core 0, package 0 > cpu0: using VERW MDS workaround > pvbus0 at mainbus0: OpenBSD > pvclock0 at pvbus0 > pci0 at mainbus0 bus 0 > pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00 > virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00 > viornd0 at virtio0 > virtio0: irq 3 > virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00 > vio0 at virtio1: address fe:e1:bb:d1:24:36 > virtio1: irq 5 > virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00 > vioblk0 at virtio2 > scsibus1 at vioblk0: 2 targets > sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > > sd0: 51200MB, 512 bytes/sector, 104857600 sectors > virtio2: irq 6 > virtio3 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00 > vioblk1 at virtio3 > scsibus2 at vioblk1: 2 targets > sd1 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > > sd1: 51200MB, 512 bytes/sector, 104857600 sectors > virtio3: irq 7 > virtio4 at pci0 dev 5 function 0 "OpenBSD VMM Control" rev 0x00 > vmmci0 at virtio4 > virtio4: irq 9 > isa0 at mainbus0 > isadma0 at isa0 > com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo > com0: console > vscsi0 at root > scsibus3 at vscsi0: 256 targets > softraid0 at root > scsibus4 at softraid0: 256 targets > root on sd0a (d4c2875ac610c324.a) swap on sd0b dump on sd0b > WARNING: / was not properly unmounted > Automatic boot in progress: starting file system checks. > /dev/sd0a (d4c2875ac610c324.a): 2323 files, 47973 used, 466466 free (178 > frags, 58286 blocks, 0.0% fragmentation) > /dev/sd0a (d4c2875ac610c324.a): MARKING FILE SYSTEM CLEAN > /dev/rsd1a: file system is clean; not checking > /dev/rsd1k: file system is clean; not checking > /dev/rsd1d: file system is clean; not checking > /dev/rsd1f: file system is clean; not checking > /dev/rsd1g: file system is clean; not checking > /dev/rsd1h: file system is clean; not checking > /dev/rsd1j: file system is clean; not checking > /dev/rsd1i: file system is clean; not checking > /dev/rsd1e: file system is clean; not checking > /dev/sd0k (d4c2875ac610c324.k): 3992 files, 548865 used, 10831142 free (78 > frags, 1353883 blocks, 0.0% fragmentation) > /dev/sd0k (d4c2875ac610c324.k): MARKING FILE SYSTEM CLEAN > /dev/sd0d (d4c2875ac610c324.d): 10 files, 7 used, 1697224 free (56 frags, > 212146 blocks, 0.0% fragmentation) > /dev/sd0d (d4c2875ac610c324.d): MARKING FILE SYSTEM CLEAN > /dev/sd0f (d4c2875ac610c324.f): 185257 files, 743839 used, 286688 free (32896 > frags, 31724 blocks, 3.2% fragmentation) > /dev/sd0f (d4c2875ac610c324.f): MARKING FILE SYSTEM CLEAN > /dev/sd0g (d4c2875ac610c324.g): 9215 files, 107450 used, 406989 free (6373 > frags, 50077 blocks, 1.2% fragmentation) > /dev/sd0g (d4c2875ac610c324.g): MARKING FILE SYSTEM CLEAN > /dev/sd0h (d4c2875ac610c324.h): 1474 files, 18825 used, 3565814 free (70 > frags, 445718 blocks, 0.0% fragmentation) > /dev/sd0h (d4c2875ac610c324.h): MARKING FILE SYSTEM CLEAN > /dev/sd0j (d4c2875ac610c324.j): 1 files, 1 used, 3094878 free (14 frags, > 386858 blocks, 0.0% fragmentation) > /dev/sd0j (d4c2875ac610c324.j): MARKING FILE SYSTEM CLEAN > /dev/sd0i (d4c2875ac610c324.i): 1 files, 1 used, 1030526 free (14 frags, > 128814 blocks, 0.0% fragmentation) > /dev/sd0i (d4c2875ac610c324.i): MARKING FILE SYSTEM CLEAN > /dev/sd0e (d4c2875ac610c324.e): 471 files, 6955 used, 2555452 free (252 > frags, 319400 blocks, 0.0% fragmentation) > /dev/sd0e (d4c2875ac610c324.e): MARKING FILE SYSTEM CLEAN > pf enabled > starting network > reordering libraries: done. > starting early daemons: syslogd pflogd ntpd. > starting RPC daemons:. > savecore: no core dump > checking quotas: done. > clearing /tmp > kern.securelevel: 0 -> 1 > creating runtime link editor directory cache. > preserving editor files. > starting network daemons: sshd smtpd httpd sndiod. > starting package daemons: haproxy. > starting local daemons: cron. > Sun Oct 6 09:05:17 CEST 2019 > > OpenBSD/amd64 (fd0.openbsd.amsterdam) (tty00) > > login: > > Mischa > > > > > > >> Mischa > >> > >> ### > >> > >> fd0$ uvm_fault(0xffffffff81f9def0, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Stopped at splassert_check+0x1: cmpb %cl,0(%rax) > >> Using drive 0, partition 3. > >> Loading...... > >> probing: pc0 com0 mem[638K 510M a20=on] > >> disk: hd0+ hd1+ > >>>> OpenBSD/amd64 BOOT 3.45 > >> / > >> com0: 115200 baud > >> switching console to com0 > >>>> OpenBSD/amd64 BOOT 3.45 > >> boot> > >> booting hd0a:/bsd: 12641608+2937864+331376+0+704512 > >> [980486+128+1009392+738445]=0x1274608 > >> entry point at 0xffffffff81001000 > >> [ using 2729480 bytes of bsd ELF symbol table ] > >> Copyright (c) 1982, 1986, 1989, 1991, 1993 > >> The Regents of the University of California. All rights reserved. > >> Copyright (c) 1995-2019 OpenBSD. All rights reserved. > >> https://www.OpenBSD.org > >> > >> OpenBSD 6.6-beta (GENERIC) #301: Tue Sep 24 15:28:24 MDT 2019 > >> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC > >> > >> ### > >> > >> • d0# panic: mtx 0xffffffff81f353f0: locking against myself > >> Using drive 0, partition 3. > >> Loading...... > >> probing: pc0 com0 mem[638K 510M a20=on] > >> disk: hd0+ hd1+ > >>>> OpenBSD/amd64 BOOT 3.45 > >> / > >> com0: 115200 baud > >> switching console to com0 > >>>> OpenBSD/amd64 BOOT 3.45 > >> boot> > >> booting hd0a:/bsd: 12666184+2937864+332896+0+704512 > >> [987630+128+1010256+738953]=0x127d750 > >> entry point at 0xffffffff81001000 > >> [ using 2738000 bytes of bsd ELF symbol table ] > >> Copyright (c) 1982, 1986, 1989, 1991, 1993 > >> The Regents of the University of California. All rights reserved. > >> Copyright (c) 1995-2019 OpenBSD. All rights reserved. > >> https://www.OpenBSD.org > >> > >> OpenBSD 6.6 (GENERIC) #325: Wed Oct 2 11:38:13 MDT 2019 > >> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC > >> > >> ### > >> > >> fd0# sysupgrade -s > >> SHA256.sig 100% |*************************************| 2141 > >> 00:00 > >> Signature Verified > >> Verifying old sets. > >> base66.tgz 2% | | 5109 KB > >> 00:01 > >> fd0# uvm_fault(0xffffffff81fb5808, 0x3ff, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Stopped at amdgpu_atombios_encoder_setup_dig_transmitter+0x63: > >> testb % > >> r9b,0xffffffffffffffff(%r9,%rcx,4) > >> uvm_fault(0xffffffff81fb5808, 0x3fd, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e > >> Using drive 0, partition 3. > >> Loading...... > >> probing: pc0 com0 mem[638K 510M a20=on] > >> disk: hd0+ hd1+ > >>>> OpenBSD/amd64 BOOT 3.45 > >> / > >> com0: 115200 baud > >> switching console to com0 > >>>> OpenBSD/amd64 BOOT 3.45 > >> boot> > >> booting hd0a:/bsd: 12666184+2937872+333664+0+704512 > >> [983130+128+1010256+738953]=0x127c5c0 > >> entry point at 0xffffffff81001000 > >> [ using 2733504 bytes of bsd ELF symbol table ] > >> Copyright (c) 1982, 1986, 1989, 1991, 1993 > >> The Regents of the University of California. All rights reserved. > >> Copyright (c) 1995-2019 OpenBSD. All rights reserved. > >> https://www.OpenBSD.org > >> > >> OpenBSD 6.6 (GENERIC) #334: Sat Oct 5 12:16:54 MDT 2019 > >> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC > >> > >> ### > >> > >> OpenBSD/amd64 (fd0.openbsd.amsterdam) (tty00) > >> > >> login: uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Stopped at Xintr_uvuvm_fault(0xffffffff81fdf560, 0x43a, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Stopped at Xintr_legacy5_untramp: fcompl 0x42(%rdx) > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulteuvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> kernel: page fault trap, code=0 > >> Faulted in DDB; continuing... > >> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e > >> > >> >
