Public bug reported:
I have 10 vms, run nova reboot the 10 vms. After reboot 285 times,can
not nova reboot again.The detail information:
nova list:
+--+---+++-++
| ID | Name
| Status | Task State | Power State | Networks |
+--+---+++-++
| 26d74bba-2e9a-4594-b6a1-1b27f337759b |
test-26d74bba-2e9a-4594-b6a1-1b27f337759b | REBOOT | reboot_started | Running
| net01=192.168.0.8 |
| 7d129f4d-0af7-4ea8-9de5-3483cdcb7a68 |
test-7d129f4d-0af7-4ea8-9de5-3483cdcb7a68 | REBOOT | reboot_started | Running
| net01=192.168.0.13 |
| 8d2e097b-e478-4a7f-b06d-7b578dfaf7c0 |
test-8d2e097b-e478-4a7f-b06d-7b578dfaf7c0 | REBOOT | reboot_started | Running
| net01=192.168.0.9 |
| a2d264f0-c1d0-449c-9ba3-32041247490c |
test-a2d264f0-c1d0-449c-9ba3-32041247490c | REBOOT | reboot_started | Running
| net01=192.168.0.12 |
| a3d9448d-25d8-4c01-925c-825a19164970 |
test-a3d9448d-25d8-4c01-925c-825a19164970 | REBOOT | reboot_started | Running
| net01=192.168.0.11 |
| a993222f-4e41-483e-8e91-300be0d525a4 |
test-a993222f-4e41-483e-8e91-300be0d525a4 | REBOOT | reboot_started | Running
| net01=192.168.0.5 |
| ace0736a-0371-4a3d-8a7f-525e299a924b |
test-ace0736a-0371-4a3d-8a7f-525e299a924b | REBOOT | reboot_started | Running
| net01=192.168.0.6 |
| e5515b31-8edb-4558-bb9f-f6f8b1142db2 |
test-e5515b31-8edb-4558-bb9f-f6f8b1142db2 | REBOOT | reboot_started | Running
| net01=192.168.0.4 |
| f380346a-bc24-4d63-a6e0-e36b5a508a59 |
test-f380346a-bc24-4d63-a6e0-e36b5a508a59 | REBOOT | reboot_started | Running
| net01=192.168.0.10 |
| fc396066-7ec6-4fe9-b7bf-70f162e293fe |
test-fc396066-7ec6-4fe9-b7bf-70f162e293fe | REBOOT | reboot_started | Running
| net01=192.168.0.7 |
+--+---+++-++
The nova-compute.log:
2015-07-22 14:51:43.808 15940 WARNING nova.openstack.common.deadlock_monitor
[-] service function nova.servicegroup.drivers.db._report_state is deadlocked!
2015-07-22 14:51:43.984 15940 WARNING nova.openstack.common.deadlock_monitor
[-] deadlock traceback is [' File
/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py, line
78, in _inner\nself.f(*self.args, **self.kw)\n', ' File
/usr/lib/python2.7/site-packages/nova/openstack/common/deadlock_monitor.py,
line 58, in __deco\nret = func(*args, **kwargs)\n', ' File
/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py, line 116,
in _report_state\nservice.service_ref, state_catalog)\n', ' File
/usr/lib/python2.7/site-packages/nova/conductor/api.py, line 218, in
service_update\nreturn self._manager.service_update(context, service,
values)\n', ' File
/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py, line 330, in
service_update\nservice=service_p, values=values)\n', ' File
/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py, line 150, in
call\nwait_for_reply=True, timeout=timeout)\n', ' File
/usr/lib/python2.7/site-packages/oslo/messaging/transport.py, line 90, in
_send\ntimeout=timeout)\n', ' File
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py, line
412, in send\nreturn self._send(target, ctxt, message, wait_for_reply,
timeout)\n', ' File
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py, line
400, in _send\nconn.topic_send(topic, msg, timeout=timeout)\n', ' File
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py, line 147,
in __exit__\nself._done()\n', ' File
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py, line 136,
in _done\nself.connection.reset()\n', ' File
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py, line
656, in reset\nself.channel.close()\n', ' File
/usr/lib/python2.7/site-packages/amqp/channel.py, line 163, in close\n
(20, 41), # Channel.close_ok\n', ' File
/usr/lib/python2.7/site-packages/amqp/abstract_channel.py, line 69, in wait\n
self.channel_id, allowed_methods)\n', ' File
/usr/lib/python2.7/site-packages/amqp/connection.py, line 204, in
_wait_method\nself.method_reader.read_method()\n', ' File
/usr/lib/python2.7/site-packages/amqp/method_framing.py, line 189, in
read_method\nself._next_method()\n', ' File
/usr/lib/python2.7/site-packages/amqp/method_framing.py, line 112, in
_next_method\nframe_type, channel, payload = read_frame()\n', ' File
/usr/lib/python2.7/site-packages/amqp/transport.py, line 147, in read_frame\n
frame_type, channel, size = unpack(\'BHI\', read(7,