> On 16 Oct 2019, at 21:35, Mike Larkin <mlar...@nested.page> wrote:
>
> On Wed, Oct 16, 2019 at 06:14:55PM +0200, Mischa wrote:
>> Hi Stuart,
>>
>>
>>>> On 16 Oct 2019, at 18:07, Stuart Henderson <s...@spacehopper.org> wrote:
>>>
>>> On 2019/10/16 18:00, Mischa wrote:
>>>> Hi All,
>>>>
>>>> One of the OpenBSD VMs running on 6.6-beta #313 is rebooting or panicing
>>>> in different ways.
>>>> Not sure if they are all relevant or useful but here are the ones we
>>>> managed to capture.
>>>
>>> There's not a lot of information in your mail... for starters, what are
>>> you running the VM in, and is there any difference in the config for that
>>> VM compared to other working ones?
>>
>> Fair point.
>>
>> There are 10 VMs running on this host, the host is running:
>> $ sysctl kern.version
>> kern.version=OpenBSD 6.6-beta (GENERIC.MP) #313: Tue Sep 10 23:30:52 MDT 2019
>> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>>
>> I know of one other VM which is rebooting every once in a while, but haven’t
>> seen any panics.
>> As for the other 8 I can see every once in a while a VM shutdown. But no
>> capture of the console.
>>
>>> Do you have any other VMs running the same OpenBSD snapshot successfully?
>>
>> The rest of the VMs are on -stable as far as I am aware. Other people are
>> operating these VMS.
>>
>>> Can you boot an old kernel and get a dmesg?
>>
>> Here is a dmesg which we manage to capture after one of the panics:
>>
>
> Are you in swap at all on that host?
Yes. :/
load averages: 0.02, 0.06, 0.13 server1.openbsd.amsterdam
21:41:11
71 processes: 70 idle, 1 on processor up 34 days,
2:15
CPU0: 0.7% user, 0.0% nice, 2.6% sys, 0.5% spin, 0.1% intr, 96.2% idle
CPU1: 0.7% user, 0.0% nice, 3.4% sys, 0.4% spin, 0.0% intr, 95.6% idle
CPU2: 6.2% user, 0.0% nice, 31.0% sys, 11.0% spin, 0.0% intr, 51.8% idle
CPU3: 0.7% user, 0.0% nice, 2.8% sys, 0.3% spin, 0.0% intr, 96.1% idle
Memory: Real: 5427M/7623M act/tot Free: 275M Cache: 2001M Swap: 41M/8405M
Mischa
>
> -ml
>
>> fd0# panic: mtx 0xffffffff81f353f0: locking against myself
>> Using drive 0, partition 3.
>> Loading......
>> probing: pc0 com0 mem[638K 510M a20=on]
>> disk: hd0+ hd1+
>>>> OpenBSD/amd64 BOOT 3.45
>> /
>> com0: 115200 baud
>> switching console to com0
>>>> OpenBSD/amd64 BOOT 3.45
>> boot>
>> booting hd0a:/bsd: 12666184+2937864+332896+0+704512
>> [987630+128+1010256+738953]=0x127d750
>> entry point at 0xffffffff81001000
>> [ using 2738000 bytes of bsd ELF symbol table ]
>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>> The Regents of the University of California. All rights reserved.
>> Copyright (c) 1995-2019 OpenBSD. All rights reserved.
>> https://www.OpenBSD.org
>>
>> OpenBSD 6.6 (GENERIC) #325: Wed Oct 2 11:38:13 MDT 2019
>> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
>> real mem = 520077312 (495MB)
>> avail mem = 491753472 (468MB)
>> mpath0 at root
>> scsibus0 at mpath0: 256 targets
>> mainbus0 at root
>> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf3f40 (10 entries)
>> bios0: vendor SeaBIOS version "1.11.0p2-OpenBSD-vmm" date 01/01/2011
>> bios0: OpenBSD VMM
>> acpi at bios0 not configured
>> cpu0 at mainbus0: (uniprocessor)
>> cpu0: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3101.63 MHz, 06-3a-09
>> cpu0:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,LONG,LAHF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,MELTDOWN
>> cpu0: 256KB 64b/line 8-way L2 cache
>> tsc_timecounter_init: TSC skew=0 observed drift=0
>> cpu0: smt 0, core 0, package 0
>> cpu0: using VERW MDS workaround
>> pvbus0 at mainbus0: OpenBSD
>> pvclock0 at pvbus0
>> pci0 at mainbus0 bus 0
>> pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
>> virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
>> viornd0 at virtio0
>> virtio0: irq 3
>> virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
>> vio0 at virtio1: address fe:e1:bb:d1:24:36
>> virtio1: irq 5
>> virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
>> vioblk0 at virtio2
>> scsibus1 at vioblk0: 2 targets
>> sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
>> sd0: 51200MB, 512 bytes/sector, 104857600 sectors
>> virtio2: irq 6
>> virtio3 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
>> vioblk1 at virtio3
>> scsibus2 at vioblk1: 2 targets
>> sd1 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
>> sd1: 51200MB, 512 bytes/sector, 104857600 sectors
>> virtio3: irq 7
>> virtio4 at pci0 dev 5 function 0 "OpenBSD VMM Control" rev 0x00
>> vmmci0 at virtio4
>> virtio4: irq 9
>> isa0 at mainbus0
>> isadma0 at isa0
>> com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
>> com0: console
>> vscsi0 at root
>> scsibus3 at vscsi0: 256 targets
>> softraid0 at root
>> scsibus4 at softraid0: 256 targets
>> root on sd0a (d4c2875ac610c324.a) swap on sd0b dump on sd0b
>> WARNING: / was not properly unmounted
>> Automatic boot in progress: starting file system checks.
>> /dev/sd0a (d4c2875ac610c324.a): 2323 files, 47973 used, 466466 free (178
>> frags, 58286 blocks, 0.0% fragmentation)
>> /dev/sd0a (d4c2875ac610c324.a): MARKING FILE SYSTEM CLEAN
>> /dev/rsd1a: file system is clean; not checking
>> /dev/rsd1k: file system is clean; not checking
>> /dev/rsd1d: file system is clean; not checking
>> /dev/rsd1f: file system is clean; not checking
>> /dev/rsd1g: file system is clean; not checking
>> /dev/rsd1h: file system is clean; not checking
>> /dev/rsd1j: file system is clean; not checking
>> /dev/rsd1i: file system is clean; not checking
>> /dev/rsd1e: file system is clean; not checking
>> /dev/sd0k (d4c2875ac610c324.k): 3992 files, 548865 used, 10831142 free (78
>> frags, 1353883 blocks, 0.0% fragmentation)
>> /dev/sd0k (d4c2875ac610c324.k): MARKING FILE SYSTEM CLEAN
>> /dev/sd0d (d4c2875ac610c324.d): 10 files, 7 used, 1697224 free (56 frags,
>> 212146 blocks, 0.0% fragmentation)
>> /dev/sd0d (d4c2875ac610c324.d): MARKING FILE SYSTEM CLEAN
>> /dev/sd0f (d4c2875ac610c324.f): 185257 files, 743839 used, 286688 free
>> (32896 frags, 31724 blocks, 3.2% fragmentation)
>> /dev/sd0f (d4c2875ac610c324.f): MARKING FILE SYSTEM CLEAN
>> /dev/sd0g (d4c2875ac610c324.g): 9215 files, 107450 used, 406989 free (6373
>> frags, 50077 blocks, 1.2% fragmentation)
>> /dev/sd0g (d4c2875ac610c324.g): MARKING FILE SYSTEM CLEAN
>> /dev/sd0h (d4c2875ac610c324.h): 1474 files, 18825 used, 3565814 free (70
>> frags, 445718 blocks, 0.0% fragmentation)
>> /dev/sd0h (d4c2875ac610c324.h): MARKING FILE SYSTEM CLEAN
>> /dev/sd0j (d4c2875ac610c324.j): 1 files, 1 used, 3094878 free (14 frags,
>> 386858 blocks, 0.0% fragmentation)
>> /dev/sd0j (d4c2875ac610c324.j): MARKING FILE SYSTEM CLEAN
>> /dev/sd0i (d4c2875ac610c324.i): 1 files, 1 used, 1030526 free (14 frags,
>> 128814 blocks, 0.0% fragmentation)
>> /dev/sd0i (d4c2875ac610c324.i): MARKING FILE SYSTEM CLEAN
>> /dev/sd0e (d4c2875ac610c324.e): 471 files, 6955 used, 2555452 free (252
>> frags, 319400 blocks, 0.0% fragmentation)
>> /dev/sd0e (d4c2875ac610c324.e): MARKING FILE SYSTEM CLEAN
>> pf enabled
>> starting network
>> reordering libraries: done.
>> starting early daemons: syslogd pflogd ntpd.
>> starting RPC daemons:.
>> savecore: no core dump
>> checking quotas: done.
>> clearing /tmp
>> kern.securelevel: 0 -> 1
>> creating runtime link editor directory cache.
>> preserving editor files.
>> starting network daemons: sshd smtpd httpd sndiod.
>> starting package daemons: haproxy.
>> starting local daemons: cron.
>> Sun Oct 6 09:05:17 CEST 2019
>>
>> OpenBSD/amd64 (fd0.openbsd.amsterdam) (tty00)
>>
>> login:
>>
>> Mischa
>>
>>>
>>>
>>>> Mischa
>>>>
>>>> ###
>>>>
>>>> fd0$ uvm_fault(0xffffffff81f9def0, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Stopped at splassert_check+0x1: cmpb %cl,0(%rax)
>>>> Using drive 0, partition 3.
>>>> Loading......
>>>> probing: pc0 com0 mem[638K 510M a20=on]
>>>> disk: hd0+ hd1+
>>>>>> OpenBSD/amd64 BOOT 3.45
>>>> /
>>>> com0: 115200 baud
>>>> switching console to com0
>>>>>> OpenBSD/amd64 BOOT 3.45
>>>> boot>
>>>> booting hd0a:/bsd: 12641608+2937864+331376+0+704512
>>>> [980486+128+1009392+738445]=0x1274608
>>>> entry point at 0xffffffff81001000
>>>> [ using 2729480 bytes of bsd ELF symbol table ]
>>>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>>>> The Regents of the University of California. All rights reserved.
>>>> Copyright (c) 1995-2019 OpenBSD. All rights reserved.
>>>> https://www.OpenBSD.org
>>>>
>>>> OpenBSD 6.6-beta (GENERIC) #301: Tue Sep 24 15:28:24 MDT 2019
>>>> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
>>>>
>>>> ###
>>>>
>>>> • d0# panic: mtx 0xffffffff81f353f0: locking against myself
>>>> Using drive 0, partition 3.
>>>> Loading......
>>>> probing: pc0 com0 mem[638K 510M a20=on]
>>>> disk: hd0+ hd1+
>>>>>> OpenBSD/amd64 BOOT 3.45
>>>> /
>>>> com0: 115200 baud
>>>> switching console to com0
>>>>>> OpenBSD/amd64 BOOT 3.45
>>>> boot>
>>>> booting hd0a:/bsd: 12666184+2937864+332896+0+704512
>>>> [987630+128+1010256+738953]=0x127d750
>>>> entry point at 0xffffffff81001000
>>>> [ using 2738000 bytes of bsd ELF symbol table ]
>>>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>>>> The Regents of the University of California. All rights reserved.
>>>> Copyright (c) 1995-2019 OpenBSD. All rights reserved.
>>>> https://www.OpenBSD.org
>>>>
>>>> OpenBSD 6.6 (GENERIC) #325: Wed Oct 2 11:38:13 MDT 2019
>>>> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
>>>>
>>>> ###
>>>>
>>>> fd0# sysupgrade -s
>>>> SHA256.sig 100% |*************************************| 2141
>>>> 00:00
>>>> Signature Verified
>>>> Verifying old sets.
>>>> base66.tgz 2% | | 5109 KB
>>>> 00:01
>>>> fd0# uvm_fault(0xffffffff81fb5808, 0x3ff, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Stopped at amdgpu_atombios_encoder_setup_dig_transmitter+0x63:
>>>> testb %
>>>> r9b,0xffffffffffffffff(%r9,%rcx,4)
>>>> uvm_fault(0xffffffff81fb5808, 0x3fd, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>>>> Using drive 0, partition 3.
>>>> Loading......
>>>> probing: pc0 com0 mem[638K 510M a20=on]
>>>> disk: hd0+ hd1+
>>>>>> OpenBSD/amd64 BOOT 3.45
>>>> /
>>>> com0: 115200 baud
>>>> switching console to com0
>>>>>> OpenBSD/amd64 BOOT 3.45
>>>> boot>
>>>> booting hd0a:/bsd: 12666184+2937872+333664+0+704512
>>>> [983130+128+1010256+738953]=0x127c5c0
>>>> entry point at 0xffffffff81001000
>>>> [ using 2733504 bytes of bsd ELF symbol table ]
>>>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>>>> The Regents of the University of California. All rights reserved.
>>>> Copyright (c) 1995-2019 OpenBSD. All rights reserved.
>>>> https://www.OpenBSD.org
>>>>
>>>> OpenBSD 6.6 (GENERIC) #334: Sat Oct 5 12:16:54 MDT 2019
>>>> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
>>>>
>>>> ###
>>>>
>>>> OpenBSD/amd64 (fd0.openbsd.amsterdam) (tty00)
>>>>
>>>> login: uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Stopped at Xintr_uvuvm_fault(0xffffffff81fdf560, 0x43a, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Stopped at Xintr_legacy5_untramp: fcompl 0x42(%rdx)
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulteuvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>> kernel: page fault trap, code=0
>>>> Faulted in DDB; continuing...
>>>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>>>
>>>>
>>
