[Qemu-discuss] qemu hangs
Hello, qemu 2.7.0-0.1.rc2.fc24, but the same happened with 2.6.0. I start systemctl start libvirtd, and systemctl status libvirtd shows: Active: active (running) since Pr 2016-08-15 22:41:43 EEST; 30s ago CGroup: /system.slice/libvirtd.service ├─2907 /usr/sbin/libvirtd ├─2932 /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib └─2934 /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib These processes (/usr/bin/qemu-system-alpha -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize) hang indefinitely. The same happens if I run it manually. Disabling selinux does not help. If I kill qemu-system-alpha, qemu-system-arm is started and so on. I cannot connect to libvirt with virt-manager. Any ideas why qemu hangs? Regards, Nerijus
Re: [Qemu-discuss] qemu hangs
I tried to debug with gdb: # gdb /usr/bin/qemu-system-alpha Reading symbols from /usr/bin/qemu-system-alpha...Reading symbols from /usr/lib/debug/usr/bin/qemu-system-alpha.debug...done. done. (gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize Starting program: /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7fffc9708700 (LWP 12596)] Detaching after fork from child process 12597. ^C Thread 1 "qemu-system-alp" received signal SIGINT, Interrupt. 0x7fffdabc2dad in read () at ../sysdeps/unix/syscall-template.S:84 84 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) (gdb) bt #0 0x7fffdabc2dad in read () at ../sysdeps/unix/syscall-template.S:84 #1 0x557d63d3 in os_daemonize (__nbytes=1, __buf=0x7fffdfdf, __fd=) at /usr/include/bits/unistd.h:44 #2 0x557d63d3 in os_daemonize () at /usr/src/debug/qemu-2.7.0-rc2/os-posix.c:224 #3 0x5573ac9e in main (argc=, argv=, envp=) at /usr/src/debug/qemu-2.7.0-rc2/vl.c:3989 2016-08-16 12:45, Nerijus Baliūnas rašė: Hello, qemu 2.7.0-0.1.rc2.fc24, but the same happened with 2.6.0. I start systemctl start libvirtd, and systemctl status libvirtd shows: Active: active (running) since Pr 2016-08-15 22:41:43 EEST; 30s ago CGroup: /system.slice/libvirtd.service ├─2907 /usr/sbin/libvirtd ├─2932 /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib └─2934 /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib These processes (/usr/bin/qemu-system-alpha -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize) hang indefinitely. The same happens if I run it manually. Disabling selinux does not help. If I kill qemu-system-alpha, qemu-system-arm is started and so on. I cannot connect to libvirt with virt-manager. Any ideas why qemu hangs? Regards, Nerijus
Re: [Qemu-discuss] qemu hangs
On 16 August 2016 at 17:50, Nerijus Baliūnas wrote: > I tried to debug with gdb: > > # gdb /usr/bin/qemu-system-alpha > Reading symbols from /usr/bin/qemu-system-alpha...Reading symbols from > /usr/lib/debug/usr/bin/qemu-system-alpha.debug...done. > done. > (gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp > unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile > /var/lib/libvirt/qemu/capabilities.pidfile -daemonize > Starting program: /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults > -nographic -M none -qmp > unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile > /var/lib/libvirt/qemu/capabilities.pidfile -daemonize > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > [New Thread 0x7fffc9708700 (LWP 12596)] > Detaching after fork from child process 12597. > ^C > Thread 1 "qemu-system-alp" received signal SIGINT, Interrupt. > 0x7fffdabc2dad in read () at ../sysdeps/unix/syscall-template.S:84 > 84 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) > > (gdb) bt > #0 0x7fffdabc2dad in read () at ../sysdeps/unix/syscall-template.S:84 > #1 0x557d63d3 in os_daemonize (__nbytes=1, __buf=0x7fffdfdf, > __fd=) > at /usr/include/bits/unistd.h:44 > #2 0x557d63d3 in os_daemonize () at > /usr/src/debug/qemu-2.7.0-rc2/os-posix.c:224 > #3 0x5573ac9e in main (argc=, argv=, > envp=) > at /usr/src/debug/qemu-2.7.0-rc2/vl.c:3989 This is the outer process waiting for the child process to tell it it's got started in os_daemonize(). Does QEMU still hang if you don't pass it -daemonize ? If so, then it will be easier to debug that. If the hang is actually caused by the daemonize code then you'll need to get gdb to debug into the child process of the fork(). thanks -- PMM
Re: [Qemu-discuss] qemu hangs
2016-08-16 19:58, Peter Maydell rašė: This is the outer process waiting for the child process to tell it it's got started in os_daemonize(). Does QEMU still hang if you don't pass it -daemonize ? If so, then it will be easier to debug that. Yes, it still hangs: # gdb /usr/bin/qemu-system-alpha (gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile Starting program: /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7fffc9708700 (LWP 13010)] ^C Thread 1 "qemu-system-alp" received signal SIGINT, Interrupt. 0x7fffda8e83f1 in __GI_ppoll (fds=0x56cfb000, nfds=5, timeout=, timeout@entry=0x0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:50 50result = SYSCALL_CANCEL (ppoll, fds, nfds, timeout, sigmask, _NSIG / 8); (gdb) bt #0 0x7fffda8e83f1 in __GI_ppoll (fds=0x56cfb000, nfds=5, timeout=, timeout@entry=0x0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:50 #1 0x559455bb in qemu_poll_ns (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 0x559455bb in qemu_poll_ns (fds=, nfds=out>, timeout=timeout@entry=-1) at /usr/src/debug/qemu-2.7.0-rc2/qemu-timer.c:313 #3 0x55944fba in main_loop_wait (timeout=-1) at /usr/src/debug/qemu-2.7.0-rc2/main-loop.c:252 #4 0x55944fba in main_loop_wait (nonblocking=) at /usr/src/debug/qemu-2.7.0-rc2/main-loop.c:506 #5 0x5573be65 in main () at /usr/src/debug/qemu-2.7.0-rc2/vl.c:1908 #6 0x5573be65 in main (argc=, argv=, envp=) at /usr/src/debug/qemu-2.7.0-rc2/vl.c:4603
Re: [Qemu-discuss] qemu hangs
On 16 August 2016 at 18:13, Nerijus Baliūnas wrote: > 2016-08-16 19:58, Peter Maydell rašė: >> >> This is the outer process waiting for the child process to tell >> it it's got started in os_daemonize(). >> >> Does QEMU still hang if you don't pass it -daemonize ? If so, >> then it will be easier to debug that. > > > Yes, it still hangs: > > # gdb /usr/bin/qemu-system-alpha > (gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp > unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile > /var/lib/libvirt/qemu/capabilities.pidfile > Starting program: /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults > -nographic -M none -qmp > unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile > /var/lib/libvirt/qemu/capabilities.pidfile > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > [New Thread 0x7fffc9708700 (LWP 13010)] > ^C Has it actually hung, or is it just waiting for you to talk QMP protocol to it on the socket ? I think what is happening here is that libvirt starts these QEMU processes to query them for what they support. What should happen is that QEMU is started, libvirt talks QMP protocol to them and then stops the QEMU processes when it's found out the answers to its questions. I think we need to somehow get a log of what libvirt is doing on the QMP socket to find out whether this bug is in QEMU (eg libvirt asks it to do something and it hangs) or in libvirt (eg libvirt doesn't tell QEMU to quit). Perhaps the libvirt mailing list might be able to help? I'm afraid I don't use it myself. thanks -- PMM
Re: [Qemu-discuss] qemu hangs
2016-08-16 20:26, Peter Maydell rašė: Yes, it still hangs: # gdb /usr/bin/qemu-system-alpha (gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile Starting program: /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7fffc9708700 (LWP 13010)] ^C Has it actually hung, or is it just waiting for you to talk QMP protocol to it on the socket ? Probably hung, because the same command launches on another PC, process is running and the shell returns. I think what is happening here is that libvirt starts these QEMU processes to query them for what they support. What should happen is that QEMU is started, libvirt talks QMP protocol to them and then stops the QEMU processes when it's found out the answers to its questions. I think we need to somehow get a log of what libvirt is doing on the QMP socket to find out whether this bug is in QEMU (eg libvirt asks it to do something and it hangs) or in libvirt (eg libvirt doesn't tell QEMU to quit). Yes, but as I wrote above, the command runs on another PC even without libvirt. So most probably something wrong with qemu. The PC with a problem is quite old AMD Athlon 64 X2 Dual-Core Processor TK-55.
Re: [Qemu-discuss] qemu hangs
On 16 August 2016 at 18:41, Nerijus Baliūnas wrote: > 2016-08-16 20:26, Peter Maydell rašė: >>> >>> Yes, it still hangs: >>> >>> # gdb /usr/bin/qemu-system-alpha >>> (gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp >>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait >>> -pidfile >>> /var/lib/libvirt/qemu/capabilities.pidfile >>> Starting program: /usr/bin/qemu-system-alpha -S -no-user-config >>> -nodefaults >>> -nographic -M none -qmp >>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait >>> -pidfile >>> /var/lib/libvirt/qemu/capabilities.pidfile >>> [Thread debugging using libthread_db enabled] >>> Using host libthread_db library "/lib64/libthread_db.so.1". >>> [New Thread 0x7fffc9708700 (LWP 13010)] >>> ^C >> >> >> Has it actually hung, or is it just waiting for you to talk >> QMP protocol to it on the socket ? > > > Probably hung, because the same command launches on another PC, > process is running and the shell returns. Even with -daemonize not supplied? It shouldn't return you to the shell prompt if you don't pass that option! >> I think what is happening here is that libvirt starts these >> QEMU processes to query them for what they support. What should >> happen is that QEMU is started, libvirt talks QMP protocol to >> them and then stops the QEMU processes when it's found out the >> answers to its questions. I think we need to somehow get a >> log of what libvirt is doing on the QMP socket to find out >> whether this bug is in QEMU (eg libvirt asks it to do something >> and it hangs) or in libvirt (eg libvirt doesn't tell >> QEMU to quit). > > > Yes, but as I wrote above, the command runs on another PC > even without libvirt. So most probably something wrong with qemu. Does the same binary work on one PC and not on the other? (not just the same command). thanks -- PMM
Re: [Qemu-discuss] qemu hangs
2016-08-16 20:53, Peter Maydell rašė: Has it actually hung, or is it just waiting for you to talk QMP protocol to it on the socket ? Probably hung, because the same command launches on another PC, process is running and the shell returns. Even with -daemonize not supplied? It shouldn't return you to the shell prompt if you don't pass that option! With -daemonize. It does not return to the shell without -daemonize of course. Yes, but as I wrote above, the command runs on another PC even without libvirt. So most probably something wrong with qemu. Does the same binary work on one PC and not on the other? (not just the same command). Yes, both OS (Fedora 24) and binaries are identical. On one PC it works, on another - does not.
Re: [Qemu-discuss] qemu hangs
2016-08-16 20:58, Nerijus Baliūnas rašė: Does the same binary work on one PC and not on the other? (not just the same command). Yes, both OS (Fedora 24) and binaries are identical. On one PC it works, on another - does not. I've tried gdb without daemonize on a working PC and got the same backtrace. So it seems it is daemonize code which has a problem.
Re: [Qemu-discuss] qemu hangs
2016-08-16 21:21, Nerijus Baliūnas rašė: I've tried gdb without daemonize on a working PC and got the same backtrace. So it seems it is daemonize code which has a problem. Debugging the child revealed the problem: # gdb /usr/bin/qemu-system-alpha (gdb) set follow-fork-mode child (gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize [New Thread 0x7fffc9708700 (LWP 16179)] [New process 16180] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". ^C Thread 2.1 "qemu-system-alp" received signal SIGINT, Interrupt. [Switching to Thread 0x77eb9fc0 (LWP 16180)] base::internal::SpinLockDelay (w=0x7fffdbf1f280 , value=2, loop=) at src/base/spinlock_linux-inl.h:88 88 errno = save_errno; (gdb) bt #0 0x7fffdbce9e47 in base::internal::SpinLockDelay(int volatile*, int, int) (w=0x7fffdbf1f280 , value=2, loop=) at src/base/spinlock_linux-inl.h:88 #1 0x7fffdbce9d64 in SpinLock::SlowLock() (this=0x7fffdbf1f280 ) at src/base/spinlock.cc:119 #2 0x7fffdbcdacba in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) (this=0x7fffdbf1f280 ) at src/base/spinlock.h:71 #3 0x7fffdbcdacba in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) (this=0x7fffdbf1f280 , start=start@entry=0x7fffdd38, end=end@entry=0x7fffdd40, N=2) at src/central_freelist.cc:248 #4 0x7fffdbcddcda in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) (this=this@entry=0x564ae0c0, cl=, byte_size=byte_size@entry=253952) at src/thread_cache.cc:126 #5 0x7fffdbcedf03 in tc_calloc(size_t, size_t) (cl=, size=, this=) at src/thread_cache.h:368 #6 0x7fffdbcedf03 in tc_calloc(size_t, size_t) (size=, heap=) at src/tcmalloc.cc:1193 #7 0x7fffdbcedf03 in tc_calloc(size_t, size_t) (size=) at src/tcmalloc.cc:1208 #8 0x7fffdbcedf03 in tc_calloc(size_t, size_t) (size=) at src/tcmalloc.cc:1219 #9 0x7fffdbcedf03 in tc_calloc(size_t, size_t) (elem_size=out>, n=) at src/tcmalloc.cc:1232 #10 0x7fffdbcedf03 in tc_calloc(size_t, size_t) (n=, elem_size=) at src/tcmalloc.cc:1645 #11 0x7fffd162778c in () at /usr/lib64/nvidia-304xx/libGL.so.1 #12 0x7fffcb4440cf in () at /usr/lib64/nvidia-304xx/libnvidia-glcore.so.304.131 #13 0x7fffd160220d in () at /usr/lib64/nvidia-304xx/libGL.so.1 #14 0x7fffd160958f in () at /usr/lib64/nvidia-304xx/libGL.so.1 #15 0x7fffd16096c4 in () at /usr/lib64/nvidia-304xx/libGL.so.1 #16 0x7fffd1609e4f in () at /usr/lib64/nvidia-304xx/libGL.so.1 #17 0x7fffda8b9a0e in __libc_fork () at ../sysdeps/nptl/fork.c:211 #18 0x557d6324 in os_daemonize () at /usr/src/debug/qemu-2.7.0-rc2/os-posix.c:216 #19 0x5573ac9e in main (argc=, argv=, envp=) at /usr/src/debug/qemu-2.7.0-rc2/vl.c:3989 It is nvidia binary driver. Running LD_PRELOAD=/usr/lib64/libGL.so.1 /usr/bin/qemu-system-alpha -S... works OK. Is there something that can be done/workarounded in qemu or should I just patch libvirtd systemd unit file to include LD_PRELOAD=/usr/lib64/libGL.so.1?
Re: [Qemu-discuss] qemu hangs
On 16 August 2016 at 19:45, Nerijus Baliūnas wrote: > 2016-08-16 21:21, Nerijus Baliūnas rašė: >> >> I've tried gdb without daemonize on a working PC and got the same >> backtrace. So it seems it is daemonize code which has a problem. > > > Debugging the child revealed the problem: > #11 0x7fffd162778c in () at /usr/lib64/nvidia-304xx/libGL.so.1 > #12 0x7fffcb4440cf in () at > /usr/lib64/nvidia-304xx/libnvidia-glcore.so.304.131 > #13 0x7fffd160220d in () at /usr/lib64/nvidia-304xx/libGL.so.1 > #14 0x7fffd160958f in () at /usr/lib64/nvidia-304xx/libGL.so.1 > #15 0x7fffd16096c4 in () at /usr/lib64/nvidia-304xx/libGL.so.1 > #16 0x7fffd1609e4f in () at /usr/lib64/nvidia-304xx/libGL.so.1 > #17 0x7fffda8b9a0e in __libc_fork () at ../sysdeps/nptl/fork.c:211 > #18 0x557d6324 in os_daemonize () at > /usr/src/debug/qemu-2.7.0-rc2/os-posix.c:216 > #19 0x5573ac9e in main (argc=, argv=, > envp=) > at /usr/src/debug/qemu-2.7.0-rc2/vl.c:3989 > > It is nvidia binary driver. Running LD_PRELOAD=/usr/lib64/libGL.so.1 > /usr/bin/qemu-system-alpha -S... > works OK. Aha, thanks for tracking down the cause. > Is there something that can be done/workarounded in qemu or should I just > patch libvirtd systemd unit file to include > LD_PRELOAD=/usr/lib64/libGL.so.1? QEMU's just doing fork() and fork is supposed to work. If this driver is supplied by Fedora I would suggest reporting the bug to them and they can decide whether they want to get the driver fixed or work around it in libvirt. Otherwise if I were you I would uninstall the binary blob :-) thanks -- PMM
Re: [Qemu-discuss] qemu hangs
2016-08-16 21:49, Peter Maydell rašė: It is nvidia binary driver. Running LD_PRELOAD=/usr/lib64/libGL.so.1 Is there something that can be done/workarounded in qemu or should I just patch libvirtd systemd unit file to include LD_PRELOAD=/usr/lib64/libGL.so.1? QEMU's just doing fork() and fork is supposed to work. If this driver is supplied by Fedora I would suggest reporting the bug to them and they can decide whether they want to get the driver fixed or work around it in libvirt. Otherwise if I were you I would uninstall the binary blob :-) It is from rpmfusion, not Fedora. Unfortunately I cannot uninstall it because nouveau does not work on this notebook and I didn't have time to investigate.
[Qemu-discuss] qemu hangs on loading Fedora 20 guest
Hello, I launch Qemu in test console: % qemu-system-x86_64 -cpu host -boot c -hda fedora.qcow2 -snapshot -m 1024 --enable-kvm -name vm0 -curses -pidfile /var/run/vm0.pid -net none -netdev type=tap,id=net0,script=no,downscript=no,ifname=vhost0,vhost=on -device virtio-net-pci,netdev=net0 And it starts booting the kernel, I see messages, however it hangs shortly and all I see on the screen is "1280x1024 Graphic mode" in the center. I don't want any graphics, so I even updated grub.cfg on the guest and replaced 'rhgb' with 'text'. Also in the guest: % ls -la /etc/systemd/system/default.target lrwxrwxrwx. 1 root root 37 Sep 22 17:17 /etc/systemd/system/default.target -> /lib/systemd/system/multi-user.target % What else should I do to run it in purely text mode? Thanks. -- Roman Mashak
