I am running a VM under qemu 2.5.0/Ubuntu 16.04. The guest VM is also
Ubuntu 16.04, with ZFS and LXD, and I am hitting it hard with I/O,
concurrently building multiple LXD containers. Previously it was
serialized to build one container at a time, and it was fine.
With the concurrent builds I got to a point where the VM livelocked:
- CPU utilisation on the host is 100%:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20321 libvirt+ 20 0 18.864g 0.014t 23032 S 100.3 45.7 603:33.30
qemu-system-x86
(host has 32GB of RAM, plenty of headroom there)
- VM does not respond to ping
- I can connect to VM's VNC console, and it shows the login prompt, but
keystrokes are ignored
I attached gdb and took backtraces (attached)
I imagine I could have hit a problem either with qemu, or with the guest
kernel itself. So next I'm going to try to replicate this with qemu
2.7.0. But I thought I'd post the backtraces here first to see if
anyone has any comments/thoughts about what's going on.
Thanks,
Brian.
P.S. I'm not subscribed to qemu-devel so please CC me directly.
# gdb -p 20321 /usr/bin/qemu-system-x86_64
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/qemu-system-x86_64...(no debugging symbols
found)...done.
Attaching to program: /usr/bin/qemu-system-x86_64, process 20321
[New LWP 20322]
[New LWP 20325]
[New LWP 20326]
[New LWP 20327]
[New LWP 20328]
[New LWP 20329]
[New LWP 20330]
[New LWP 20331]
[New LWP 20332]
[New LWP 20334]
warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/librbd.so.1" is not at
the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f9b6957fc21 in __GI_ppoll (fds=0x559d5388e080, nfds=10,
timeout=<optimised out>, sigmask=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:50
50 ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
(gdb) bt
#0 0x00007f9b6957fc21 in __GI_ppoll (fds=0x559d5388e080, nfds=10,
timeout=<optimised out>,
sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:50
#1 0x0000559d516ec8d9 in qemu_poll_ns ()
#2 0x0000559d516ec36a in main_loop_wait ()
#3 0x0000559d5146f2d4 in main ()
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f9b6e86bc00 (LWP 20321) "qemu-system-x86" 0x00007f9b6957fc21
in __GI_ppoll (
fds=0x559d5388e080, nfds=10, timeout=<optimised out>, sigmask=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:50
2 Thread 0x7f9b63727700 (LWP 20322) "qemu-system-x86" syscall ()
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
3 Thread 0x7f9b4933d700 (LWP 20325) "qemu-system-x86" 0x00007f9b69581357
in ioctl ()
at ../sysdeps/unix/syscall-template.S:84
4 Thread 0x7f9b48b3c700 (LWP 20326) "qemu-system-x86" 0x00007f9b69581357
in ioctl ()
at ../sysdeps/unix/syscall-template.S:84
5 Thread 0x7f9b3bfff700 (LWP 20327) "qemu-system-x86" 0x00007f9b69581357
in ioctl ()
at ../sysdeps/unix/syscall-template.S:84
6 Thread 0x7f9b3b7fe700 (LWP 20328) "qemu-system-x86" 0x00007f9b69581357
in ioctl ()
at ../sysdeps/unix/syscall-template.S:84
7 Thread 0x7f9b3affd700 (LWP 20329) "qemu-system-x86" 0x00007f9b69581357
in ioctl ()
at ../sysdeps/unix/syscall-template.S:84
8 Thread 0x7f9b3a7fc700 (LWP 20330) "qemu-system-x86" 0x00007f9b69581357
in ioctl ()
at ../sysdeps/unix/syscall-template.S:84
9 Thread 0x7f9b39ffb700 (LWP 20331) "qemu-system-x86" 0x00007f9b69581357
in ioctl ()
at ../sysdeps/unix/syscall-template.S:84
10 Thread 0x7f9b397fa700 (LWP 20332) "qemu-system-x86" 0x00007f9b69581357
in ioctl ()
at ../sysdeps/unix/syscall-template.S:84
11 Thread 0x7f979bbff700 (LWP 20334) "qemu-system-x86"
pthread_cond_wait@@GLIBC_2.3.2 ()
at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f9b63727700 (LWP 20322))]
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38 ../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory.
(gdb) bt
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x0000559d5175d101 in qemu_event_wait ()
#2 0x0000559d5176b5be in ?? ()
#3 0x00007f9b6985570a in start_thread (arg=0x7f9b63727700) at
pthread_create.c:333
#4 0x00007f9b6958b82d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f9b4933d700 (LWP 20325))]
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000559d514b0e44 in kvm_vcpu_ioctl ()
#2 0x0000559d514b0efd in kvm_cpu_exec ()
#3 0x0000559d5149f8d2 in ?? ()
#4 0x00007f9b6985570a in start_thread (arg=0x7f9b4933d700) at
pthread_create.c:333
#5 0x00007f9b6958b82d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) thread 4
[Switching to thread 4 (Thread 0x7f9b48b3c700 (LWP 20326))]
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000559d514b0e44 in kvm_vcpu_ioctl ()
#2 0x0000559d514b0efd in kvm_cpu_exec ()
#3 0x0000559d5149f8d2 in ?? ()
#4 0x00007f9b6985570a in start_thread (arg=0x7f9b48b3c700) at
pthread_create.c:333
#5 0x00007f9b6958b82d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) thread 5
[Switching to thread 5 (Thread 0x7f9b3bfff700 (LWP 20327))]
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000559d514b0e44 in kvm_vcpu_ioctl ()
#2 0x0000559d514b0efd in kvm_cpu_exec ()
#3 0x0000559d5149f8d2 in ?? ()
#4 0x00007f9b6985570a in start_thread (arg=0x7f9b3bfff700) at
pthread_create.c:333
#5 0x00007f9b6958b82d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) thread 6
[Switching to thread 6 (Thread 0x7f9b3b7fe700 (LWP 20328))]
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000559d514b0e44 in kvm_vcpu_ioctl ()
#2 0x0000559d514b0efd in kvm_cpu_exec ()
#3 0x0000559d5149f8d2 in ?? ()
#4 0x00007f9b6985570a in start_thread (arg=0x7f9b3b7fe700) at
pthread_create.c:333
#5 0x00007f9b6958b82d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) thread 7
[Switching to thread 7 (Thread 0x7f9b3affd700 (LWP 20329))]
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000559d514b0e44 in kvm_vcpu_ioctl ()
#2 0x0000559d514b0efd in kvm_cpu_exec ()
#3 0x0000559d5149f8d2 in ?? ()
#4 0x00007f9b6985570a in start_thread (arg=0x7f9b3affd700) at
pthread_create.c:333
#5 0x00007f9b6958b82d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) thread 8
[Switching to thread 8 (Thread 0x7f9b3a7fc700 (LWP 20330))]
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000559d514b0e44 in kvm_vcpu_ioctl ()
#2 0x0000559d514b0efd in kvm_cpu_exec ()
#3 0x0000559d5149f8d2 in ?? ()
#4 0x00007f9b6985570a in start_thread (arg=0x7f9b3a7fc700) at
pthread_create.c:333
#5 0x00007f9b6958b82d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) thread 9
[Switching to thread 9 (Thread 0x7f9b39ffb700 (LWP 20331))]
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000559d514b0e44 in kvm_vcpu_ioctl ()
#2 0x0000559d514b0efd in kvm_cpu_exec ()
#3 0x0000559d5149f8d2 in ?? ()
#4 0x00007f9b6985570a in start_thread (arg=0x7f9b39ffb700) at
pthread_create.c:333
#5 0x00007f9b6958b82d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) thread 10
[Switching to thread 10 (Thread 0x7f9b397fa700 (LWP 20332))]
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f9b69581357 in ioctl () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000559d514b0e44 in kvm_vcpu_ioctl ()
#2 0x0000559d514b0efd in kvm_cpu_exec ()
#3 0x0000559d5149f8d2 in ?? ()
#4 0x00007f9b6985570a in start_thread (arg=0x7f9b397fa700) at
pthread_create.c:333
#5 0x00007f9b6958b82d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) thread 11
[Switching to thread 11 (Thread 0x7f979bbff700 (LWP 20334))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or
directory.
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000559d5175ce09 in qemu_cond_wait ()
#2 0x0000559d516dc32b in ?? ()
#3 0x0000559d516dc848 in ?? ()
#4 0x00007f9b6985570a in start_thread (arg=0x7f979bbff700) at
pthread_create.c:333
#5 0x00007f9b6958b82d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) cont
Continuing.
