Hi Stuart,
> On 16 Oct 2019, at 18:07, Stuart Henderson <s...@spacehopper.org> wrote:
>
> On 2019/10/16 18:00, Mischa wrote:
>> Hi All,
>>
>> One of the OpenBSD VMs running on 6.6-beta #313 is rebooting or panicing in
>> different ways.
>> Not sure if they are all relevant or useful but here are the ones we managed
>> to capture.
>
> There's not a lot of information in your mail... for starters, what are
> you running the VM in, and is there any difference in the config for that
> VM compared to other working ones?
Fair point.
There are 10 VMs running on this host, the host is running:
$ sysctl kern.version
kern.version=OpenBSD 6.6-beta (GENERIC.MP) #313: Tue Sep 10 23:30:52 MDT 2019
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
I know of one other VM which is rebooting every once in a while, but haven’t
seen any panics.
As for the other 8 I can see every once in a while a VM shutdown. But no
capture of the console.
> Do you have any other VMs running the same OpenBSD snapshot successfully?
The rest of the VMs are on -stable as far as I am aware. Other people are
operating these VMS.
> Can you boot an old kernel and get a dmesg?
Here is a dmesg which we manage to capture after one of the panics:
fd0# panic: mtx 0xffffffff81f353f0: locking against myself
Using drive 0, partition 3.
Loading......
probing: pc0 com0 mem[638K 510M a20=on]
disk: hd0+ hd1+
>> OpenBSD/amd64 BOOT 3.45
/
com0: 115200 baud
switching console to com0
>> OpenBSD/amd64 BOOT 3.45
boot>
booting hd0a:/bsd: 12666184+2937864+332896+0+704512
[987630+128+1010256+738953]=0x127d750
entry point at 0xffffffff81001000
[ using 2738000 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2019 OpenBSD. All rights reserved. https://www.OpenBSD.org
OpenBSD 6.6 (GENERIC) #325: Wed Oct 2 11:38:13 MDT 2019
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 520077312 (495MB)
avail mem = 491753472 (468MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf3f40 (10 entries)
bios0: vendor SeaBIOS version "1.11.0p2-OpenBSD-vmm" date 01/01/2011
bios0: OpenBSD VMM
acpi at bios0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3101.63 MHz, 06-3a-09
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,LONG,LAHF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
tsc_timecounter_init: TSC skew=0 observed drift=0
cpu0: smt 0, core 0, package 0
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: OpenBSD
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio0
virtio0: irq 3
virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address fe:e1:bb:d1:24:36
virtio1: irq 5
virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
scsibus1 at vioblk0: 2 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 51200MB, 512 bytes/sector, 104857600 sectors
virtio2: irq 6
virtio3 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk1 at virtio3
scsibus2 at vioblk1: 2 targets
sd1 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd1: 51200MB, 512 bytes/sector, 104857600 sectors
virtio3: irq 7
virtio4 at pci0 dev 5 function 0 "OpenBSD VMM Control" rev 0x00
vmmci0 at virtio4
virtio4: irq 9
isa0 at mainbus0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
com0: console
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (d4c2875ac610c324.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
Automatic boot in progress: starting file system checks.
/dev/sd0a (d4c2875ac610c324.a): 2323 files, 47973 used, 466466 free (178 frags,
58286 blocks, 0.0% fragmentation)
/dev/sd0a (d4c2875ac610c324.a): MARKING FILE SYSTEM CLEAN
/dev/rsd1a: file system is clean; not checking
/dev/rsd1k: file system is clean; not checking
/dev/rsd1d: file system is clean; not checking
/dev/rsd1f: file system is clean; not checking
/dev/rsd1g: file system is clean; not checking
/dev/rsd1h: file system is clean; not checking
/dev/rsd1j: file system is clean; not checking
/dev/rsd1i: file system is clean; not checking
/dev/rsd1e: file system is clean; not checking
/dev/sd0k (d4c2875ac610c324.k): 3992 files, 548865 used, 10831142 free (78
frags, 1353883 blocks, 0.0% fragmentation)
/dev/sd0k (d4c2875ac610c324.k): MARKING FILE SYSTEM CLEAN
/dev/sd0d (d4c2875ac610c324.d): 10 files, 7 used, 1697224 free (56 frags,
212146 blocks, 0.0% fragmentation)
/dev/sd0d (d4c2875ac610c324.d): MARKING FILE SYSTEM CLEAN
/dev/sd0f (d4c2875ac610c324.f): 185257 files, 743839 used, 286688 free (32896
frags, 31724 blocks, 3.2% fragmentation)
/dev/sd0f (d4c2875ac610c324.f): MARKING FILE SYSTEM CLEAN
/dev/sd0g (d4c2875ac610c324.g): 9215 files, 107450 used, 406989 free (6373
frags, 50077 blocks, 1.2% fragmentation)
/dev/sd0g (d4c2875ac610c324.g): MARKING FILE SYSTEM CLEAN
/dev/sd0h (d4c2875ac610c324.h): 1474 files, 18825 used, 3565814 free (70 frags,
445718 blocks, 0.0% fragmentation)
/dev/sd0h (d4c2875ac610c324.h): MARKING FILE SYSTEM CLEAN
/dev/sd0j (d4c2875ac610c324.j): 1 files, 1 used, 3094878 free (14 frags, 386858
blocks, 0.0% fragmentation)
/dev/sd0j (d4c2875ac610c324.j): MARKING FILE SYSTEM CLEAN
/dev/sd0i (d4c2875ac610c324.i): 1 files, 1 used, 1030526 free (14 frags, 128814
blocks, 0.0% fragmentation)
/dev/sd0i (d4c2875ac610c324.i): MARKING FILE SYSTEM CLEAN
/dev/sd0e (d4c2875ac610c324.e): 471 files, 6955 used, 2555452 free (252 frags,
319400 blocks, 0.0% fragmentation)
/dev/sd0e (d4c2875ac610c324.e): MARKING FILE SYSTEM CLEAN
pf enabled
starting network
reordering libraries: done.
starting early daemons: syslogd pflogd ntpd.
starting RPC daemons:.
savecore: no core dump
checking quotas: done.
clearing /tmp
kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd smtpd httpd sndiod.
starting package daemons: haproxy.
starting local daemons: cron.
Sun Oct 6 09:05:17 CEST 2019
OpenBSD/amd64 (fd0.openbsd.amsterdam) (tty00)
login:
Mischa
>
>
>> Mischa
>>
>> ###
>>
>> fd0$ uvm_fault(0xffffffff81f9def0, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Stopped at splassert_check+0x1: cmpb %cl,0(%rax)
>> Using drive 0, partition 3.
>> Loading......
>> probing: pc0 com0 mem[638K 510M a20=on]
>> disk: hd0+ hd1+
>>>> OpenBSD/amd64 BOOT 3.45
>> /
>> com0: 115200 baud
>> switching console to com0
>>>> OpenBSD/amd64 BOOT 3.45
>> boot>
>> booting hd0a:/bsd: 12641608+2937864+331376+0+704512
>> [980486+128+1009392+738445]=0x1274608
>> entry point at 0xffffffff81001000
>> [ using 2729480 bytes of bsd ELF symbol table ]
>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>> The Regents of the University of California. All rights reserved.
>> Copyright (c) 1995-2019 OpenBSD. All rights reserved.
>> https://www.OpenBSD.org
>>
>> OpenBSD 6.6-beta (GENERIC) #301: Tue Sep 24 15:28:24 MDT 2019
>> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
>>
>> ###
>>
>> • d0# panic: mtx 0xffffffff81f353f0: locking against myself
>> Using drive 0, partition 3.
>> Loading......
>> probing: pc0 com0 mem[638K 510M a20=on]
>> disk: hd0+ hd1+
>>>> OpenBSD/amd64 BOOT 3.45
>> /
>> com0: 115200 baud
>> switching console to com0
>>>> OpenBSD/amd64 BOOT 3.45
>> boot>
>> booting hd0a:/bsd: 12666184+2937864+332896+0+704512
>> [987630+128+1010256+738953]=0x127d750
>> entry point at 0xffffffff81001000
>> [ using 2738000 bytes of bsd ELF symbol table ]
>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>> The Regents of the University of California. All rights reserved.
>> Copyright (c) 1995-2019 OpenBSD. All rights reserved.
>> https://www.OpenBSD.org
>>
>> OpenBSD 6.6 (GENERIC) #325: Wed Oct 2 11:38:13 MDT 2019
>> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
>>
>> ###
>>
>> fd0# sysupgrade -s
>> SHA256.sig 100% |*************************************| 2141 00:00
>>
>> Signature Verified
>> Verifying old sets.
>> base66.tgz 2% | | 5109 KB 00:01
>>
>> fd0# uvm_fault(0xffffffff81fb5808, 0x3ff, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Stopped at amdgpu_atombios_encoder_setup_dig_transmitter+0x63:
>> testb %
>> r9b,0xffffffffffffffff(%r9,%rcx,4)
>> uvm_fault(0xffffffff81fb5808, 0x3fd, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> uvm_fault(0xffffffff81f88c90, 0xffffffff894c1488, 0, 2) -> e
>> Using drive 0, partition 3.
>> Loading......
>> probing: pc0 com0 mem[638K 510M a20=on]
>> disk: hd0+ hd1+
>>>> OpenBSD/amd64 BOOT 3.45
>> /
>> com0: 115200 baud
>> switching console to com0
>>>> OpenBSD/amd64 BOOT 3.45
>> boot>
>> booting hd0a:/bsd: 12666184+2937872+333664+0+704512
>> [983130+128+1010256+738953]=0x127c5c0
>> entry point at 0xffffffff81001000
>> [ using 2733504 bytes of bsd ELF symbol table ]
>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>> The Regents of the University of California. All rights reserved.
>> Copyright (c) 1995-2019 OpenBSD. All rights reserved.
>> https://www.OpenBSD.org
>>
>> OpenBSD 6.6 (GENERIC) #334: Sat Oct 5 12:16:54 MDT 2019
>> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
>>
>> ###
>>
>> OpenBSD/amd64 (fd0.openbsd.amsterdam) (tty00)
>>
>> login: uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Stopped at Xintr_uvuvm_fault(0xffffffff81fdf560, 0x43a, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Stopped at Xintr_legacy5_untramp: fcompl 0x42(%rdx)
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulteuvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x43f, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>> kernel: page fault trap, code=0
>> Faulted in DDB; continuing...
>> ddb> uvm_fault(0xffffffff81fdf560, 0x0, 0, 1) -> e
>>
>>
