Re: [Qemu-discuss] qemu hangs

[Nerijus Baliūnas]

2016-08-16 21:21, Nerijus Baliūnas rašė:

I've tried gdb without daemonize on a working PC and got the same
backtrace. So it seems it is daemonize code which has a problem.


Debugging the child revealed the problem:
# gdb /usr/bin/qemu-system-alpha
(gdb) set follow-fork-mode child
(gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp 
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile 
/var/lib/libvirt/qemu/capabilities.pidfile -daemonize

[New Thread 0x7fffc9708700 (LWP 16179)]
[New process 16180]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
^C
Thread 2.1 "qemu-system-alp" received signal SIGINT, Interrupt.
[Switching to Thread 0x77eb9fc0 (LWP 16180)]
base::internal::SpinLockDelay (w=0x7fffdbf1f280 
, value=2,

loop=) at src/base/spinlock_linux-inl.h:88
88  errno = save_errno;
(gdb) bt
#0  0x7fffdbce9e47 in base::internal::SpinLockDelay(int volatile*, int, 
int) (w=0x7fffdbf1f280 , value=2, 
loop=) at src/base/spinlock_linux-inl.h:88
#1  0x7fffdbce9d64 in SpinLock::SlowLock() (this=0x7fffdbf1f280 
)

at src/base/spinlock.cc:119
#2  0x7fffdbcdacba in tcmalloc::CentralFreeList::RemoveRange(void**, 
void**, int) (this=0x7fffdbf1f280 ) 
at src/base/spinlock.h:71
#3  0x7fffdbcdacba in tcmalloc::CentralFreeList::RemoveRange(void**, 
void**, int) (this=0x7fffdbf1f280 , 
start=start@entry=0x7fffdd38, end=end@entry=0x7fffdd40, N=2)

at src/central_freelist.cc:248
#4  0x7fffdbcddcda in 
tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) 
(this=this@entry=0x564ae0c0, cl=, 
byte_size=byte_size@entry=253952) at src/thread_cache.cc:126
#5  0x7fffdbcedf03 in tc_calloc(size_t, size_t) (cl=, 
size=, this=) at src/thread_cache.h:368
#6  0x7fffdbcedf03 in tc_calloc(size_t, size_t) (size=, 
heap=)

at src/tcmalloc.cc:1193
#7  0x7fffdbcedf03 in tc_calloc(size_t, size_t) (size=) at 
src/tcmalloc.cc:1208
#8  0x7fffdbcedf03 in tc_calloc(size_t, size_t) (size=) at 
src/tcmalloc.cc:1219
#9  0x7fffdbcedf03 in tc_calloc(size_t, size_t) (elem_size=out>, n=)

at src/tcmalloc.cc:1232
#10 0x7fffdbcedf03 in tc_calloc(size_t, size_t) (n=, 
elem_size=)

at src/tcmalloc.cc:1645
#11 0x7fffd162778c in  () at /usr/lib64/nvidia-304xx/libGL.so.1
#12 0x7fffcb4440cf in  () at 
/usr/lib64/nvidia-304xx/libnvidia-glcore.so.304.131

#13 0x7fffd160220d in  () at /usr/lib64/nvidia-304xx/libGL.so.1
#14 0x7fffd160958f in  () at /usr/lib64/nvidia-304xx/libGL.so.1
#15 0x7fffd16096c4 in  () at /usr/lib64/nvidia-304xx/libGL.so.1
#16 0x7fffd1609e4f in  () at /usr/lib64/nvidia-304xx/libGL.so.1
#17 0x7fffda8b9a0e in __libc_fork () at ../sysdeps/nptl/fork.c:211
#18 0x557d6324 in os_daemonize () at 
/usr/src/debug/qemu-2.7.0-rc2/os-posix.c:216
#19 0x5573ac9e in main (argc=, argv=, 
envp=)

at /usr/src/debug/qemu-2.7.0-rc2/vl.c:3989

It is nvidia binary driver. Running LD_PRELOAD=/usr/lib64/libGL.so.1 
/usr/bin/qemu-system-alpha -S...

works OK.

Is there something that can be done/workarounded in qemu or should I just
patch libvirtd systemd unit file to include LD_PRELOAD=/usr/lib64/libGL.so.1?

Re: [Qemu-discuss] qemu hangs

[Peter Maydell]

On 16 August 2016 at 18:41, Nerijus Baliūnas
 wrote:
> 2016-08-16 20:26, Peter Maydell rašė:
>>>
>>> Yes, it still hangs:
>>>
>>> # gdb /usr/bin/qemu-system-alpha
>>> (gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp
>>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
>>> -pidfile
>>> /var/lib/libvirt/qemu/capabilities.pidfile
>>> Starting program: /usr/bin/qemu-system-alpha -S -no-user-config
>>> -nodefaults
>>> -nographic -M none -qmp
>>> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
>>> -pidfile
>>> /var/lib/libvirt/qemu/capabilities.pidfile
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>> [New Thread 0x7fffc9708700 (LWP 13010)]
>>> ^C
>>
>>
>> Has it actually hung, or is it just waiting for you to talk
>> QMP protocol to it on the socket ?
>
>
> Probably hung, because the same command launches on another PC,
> process is running and the shell returns.

Even with -daemonize not supplied? It shouldn't return
you to the shell prompt if you don't pass that option!

>> I think what is happening here is that libvirt starts these
>> QEMU processes to query them for what they support. What should
>> happen is that QEMU is started, libvirt talks QMP protocol to
>> them and then stops the QEMU processes when it's found out the
>> answers to its questions. I think we need to somehow get a
>> log of what libvirt is doing on the QMP socket to find out
>> whether this bug is in QEMU (eg libvirt asks it to do something
>> and it hangs) or in libvirt (eg libvirt doesn't tell
>> QEMU to quit).
>
>
> Yes, but as I wrote above, the command runs on another PC
> even without libvirt. So most probably something wrong with qemu.

Does the same binary work on one PC and not on the other?
(not just the same command).

thanks
-- PMM

Re: [Qemu-discuss] qemu hangs

[Nerijus Baliūnas]

2016-08-16 20:26, Peter Maydell rašė:

Yes, it still hangs:

# gdb /usr/bin/qemu-system-alpha
(gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile
/var/lib/libvirt/qemu/capabilities.pidfile
Starting program: /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults
-nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile
/var/lib/libvirt/qemu/capabilities.pidfile
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffc9708700 (LWP 13010)]
^C


Has it actually hung, or is it just waiting for you to talk
QMP protocol to it on the socket ?


Probably hung, because the same command launches on another PC,
process is running and the shell returns.


I think what is happening here is that libvirt starts these
QEMU processes to query them for what they support. What should
happen is that QEMU is started, libvirt talks QMP protocol to
them and then stops the QEMU processes when it's found out the
answers to its questions. I think we need to somehow get a
log of what libvirt is doing on the QMP socket to find out
whether this bug is in QEMU (eg libvirt asks it to do something
and it hangs) or in libvirt (eg libvirt doesn't tell
QEMU to quit).


Yes, but as I wrote above, the command runs on another PC
even without libvirt. So most probably something wrong with qemu.

The PC with a problem is quite old AMD Athlon 64 X2 Dual-Core
Processor TK-55.

Re: [Qemu-discuss] qemu hangs

[Peter Maydell]

On 16 August 2016 at 17:50, Nerijus Baliūnas
 wrote:
> I tried to debug with gdb:
>
> # gdb /usr/bin/qemu-system-alpha
> Reading symbols from /usr/bin/qemu-system-alpha...Reading symbols from
> /usr/lib/debug/usr/bin/qemu-system-alpha.debug...done.
> done.
> (gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp
> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile
> /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
> Starting program: /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults
> -nographic -M none -qmp
> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile
> /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> [New Thread 0x7fffc9708700 (LWP 12596)]
> Detaching after fork from child process 12597.
> ^C
> Thread 1 "qemu-system-alp" received signal SIGINT, Interrupt.
> 0x7fffdabc2dad in read () at ../sysdeps/unix/syscall-template.S:84
> 84  T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
>
> (gdb) bt
> #0  0x7fffdabc2dad in read () at ../sysdeps/unix/syscall-template.S:84
> #1  0x557d63d3 in os_daemonize (__nbytes=1, __buf=0x7fffdfdf,
> __fd=)
> at /usr/include/bits/unistd.h:44
> #2  0x557d63d3 in os_daemonize () at
> /usr/src/debug/qemu-2.7.0-rc2/os-posix.c:224
> #3  0x5573ac9e in main (argc=, argv=,
> envp=)
> at /usr/src/debug/qemu-2.7.0-rc2/vl.c:3989

This is the outer process waiting for the child process to tell
it it's got started in os_daemonize().

Does QEMU still hang if you don't pass it -daemonize ? If so,
then it will be easier to debug that. If the hang is actually
caused by the daemonize code then you'll need to get gdb to
debug into the child process of the fork().

thanks
-- PMM

Re: [Qemu-discuss] qemu hangs

[Nerijus Baliūnas]

I tried to debug with gdb:

# gdb /usr/bin/qemu-system-alpha
Reading symbols from /usr/bin/qemu-system-alpha...Reading symbols from 
/usr/lib/debug/usr/bin/qemu-system-alpha.debug...done.

done.
(gdb) r -S -no-user-config -nodefaults -nographic -M none -qmp 
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile 
/var/lib/libvirt/qemu/capabilities.pidfile -daemonize
Starting program: /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults 
-nographic -M none -qmp 
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile 
/var/lib/libvirt/qemu/capabilities.pidfile -daemonize

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffc9708700 (LWP 12596)]
Detaching after fork from child process 12597.
^C
Thread 1 "qemu-system-alp" received signal SIGINT, Interrupt.
0x7fffdabc2dad in read () at ../sysdeps/unix/syscall-template.S:84
84  T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)

(gdb) bt
#0  0x7fffdabc2dad in read () at ../sysdeps/unix/syscall-template.S:84
#1  0x557d63d3 in os_daemonize (__nbytes=1, __buf=0x7fffdfdf, 
__fd=)

at /usr/include/bits/unistd.h:44
#2  0x557d63d3 in os_daemonize () at 
/usr/src/debug/qemu-2.7.0-rc2/os-posix.c:224
#3  0x5573ac9e in main (argc=, argv=, 
envp=)

at /usr/src/debug/qemu-2.7.0-rc2/vl.c:3989

2016-08-16 12:45, Nerijus Baliūnas rašė:

Hello,

qemu 2.7.0-0.1.rc2.fc24, but the same happened with 2.6.0.
I start systemctl start libvirtd, and systemctl status libvirtd
shows:

   Active: active (running) since Pr 2016-08-15 22:41:43 EEST; 30s ago
   CGroup: /system.slice/libvirtd.service
   ├─2907 /usr/sbin/libvirtd
   ├─2932 /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults 
-nographic -M none -qmp

unix:/var/lib
   └─2934 /usr/bin/qemu-system-alpha -S -no-user-config -nodefaults 
-nographic -M none -qmp

unix:/var/lib

These processes (/usr/bin/qemu-system-alpha -S -no-user-config -nodefaults 
-nographic -M none -qmp

unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile
/var/lib/libvirt/qemu/capabilities.pidfile -daemonize) hang indefinitely.
The same happens if I run it manually. Disabling selinux does not help.
If I kill qemu-system-alpha, qemu-system-arm is started and so on.
I cannot connect to libvirt with virt-manager.
Any ideas why qemu hangs?

Regards,
Nerijus