Public bug reported: Description =========== nova-compute process gets stuck after running for a while
Steps to reproduce ================== Sorry, I haven't specified how to reproduce,but I know this problem has occurred at least three times in the arm environment, but not on x86. Expected result =============== nova-compute run normally Actual result ============= 2022-04-28 15:07:15.150 5967 DEBUG oslo_concurrency.lockutils [-] Lock "ca12badc-42a2-43f2-8a28-cb2afa3a377c" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "53988d8e-75bc-40de-b9af-ce7509f94918" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "bc2b5ad1-a7a4-45fc-8d95-e986476c3129" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "f2068deb-c12f-4c31-998c-cf48af4440ff" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "48577b65-28e2-4858-abed-db00f90c0dc6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "5e6a68fc-b711-4115-bc77-179a224ed5c6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "4f47003a-c056-4aa6-9be2-e5bec0cb4a07" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "a1416265-68cb-4e06-a1b9-80c0bd138a09" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "96408c82-5951-46e6-b105-bcb21118ed99" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 17:11:03.055 5967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers timer() File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__ cb(*args, **kw) File "/usr/local/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire waiter.switch() greenlet.error: cannot switch to a different thread Environment =========== Python 3.6.8 eventlet 0.30.2 oslo.concurrency 4.4.0 oslo.messaging 12.7.1 nova-compute Stack list ====================== Stack for thread 281471352828416 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281473394471424 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281471310557696 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281471838974464 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281471847428608 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281471855882752 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281471864336896 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281471872791040 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281471881245184 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281471889699328 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472375845376 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472384299520 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472392753664 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472401207808 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472409661952 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472418116096 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472426570240 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472979825152 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472988279296 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472996733440 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473005187584 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473013641728 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473022095872 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473030550016 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473111552512 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473120006656 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/nova/virt/libvirt/host.py", line 205, in _native_thread libvirt.virEventRunDefaultImpl() File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 442, in virEventRunDefaultImpl ret = libvirtmod.virEventRunDefaultImpl() Stack for thread 281473614563696 File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 42, in __thread_body func(*args, **kwargs) File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 63, in wrap_bootstrap_inner bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "<string>", line 167, in run File "/usr/lib64/python3.6/code.py", line 233, in interact more = self.push(line) File "/usr/lib64/python3.6/code.py", line 259, in push more = self.runsource(source, self.filename) File "/usr/lib64/python3.6/code.py", line 75, in runsource self.runcode(code) File "/usr/lib64/python3.6/code.py", line 91, in runcode exec(code, self.locals) File "<console>", line 3, in <module> ======================================================== I tried to analyze why I got stuck, but it took me a long time to figure it out, so looking for community help, thanks. ** Affects: nova Importance: Undecided Status: New ** Description changed: Description =========== nova-compute process gets stuck after running for a while - Steps to reproduce ================== Sorry, I haven't specified how to reproduce,but I know this problem has occurred at least three times in the arm environment, but not on x86. - Expected result =============== nova-compute run normally - Actual result ============= 2022-04-28 15:07:15.150 5967 DEBUG oslo_concurrency.lockutils [-] Lock "ca12badc-42a2-43f2-8a28-cb2afa3a377c" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "53988d8e-75bc-40de-b9af-ce7509f94918" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "bc2b5ad1-a7a4-45fc-8d95-e986476c3129" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "f2068deb-c12f-4c31-998c-cf48af4440ff" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "48577b65-28e2-4858-abed-db00f90c0dc6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "5e6a68fc-b711-4115-bc77-179a224ed5c6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "4f47003a-c056-4aa6-9be2-e5bec0cb4a07" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "a1416265-68cb-4e06-a1b9-80c0bd138a09" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "96408c82-5951-46e6-b105-bcb21118ed99" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 17:11:03.055 5967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Traceback (most recent call last): - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers - timer() - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__ - cb(*args, **kw) - File "/usr/local/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire - waiter.switch() + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers + timer() + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__ + cb(*args, **kw) + File "/usr/local/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire + waiter.switch() greenlet.error: cannot switch to a different thread - Environment =========== Python 3.6.8 eventlet 0.30.2 oslo.concurrency 4.4.0 oslo.messaging 12.7.1 - nova-compute Stack list ====================== - i + Stack for thread 281471352828416 - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run - self.wait(sleep_time) - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait - time.sleep(seconds) + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run + self.wait(sleep_time) + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait + time.sleep(seconds) Stack for thread 281473394471424 - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run - self.wait(sleep_time) - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait - time.sleep(seconds) + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run + self.wait(sleep_time) + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait + time.sleep(seconds) Stack for thread 281471310557696 - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run - self.wait(sleep_time) - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait - time.sleep(seconds) + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run + self.wait(sleep_time) + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait + time.sleep(seconds) Stack for thread 281471838974464 - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run - self.wait(sleep_time) - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait - time.sleep(seconds) + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run + self.wait(sleep_time) + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait + time.sleep(seconds) Stack for thread 281471847428608 - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run - self.wait(sleep_time) - File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait - time.sleep(seconds) + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run + self.wait(sleep_time) + File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait + time.sleep(seconds) Stack for thread 281471855882752 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281471864336896 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281471872791040 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281471881245184 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281471889699328 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281472375845376 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281472384299520 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281472392753664 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281472401207808 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281472409661952 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281472418116096 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281472426570240 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281472979825152 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281472988279296 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281472996733440 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281473005187584 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281473013641728 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281473022095872 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281473030550016 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281473111552512 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker - msg = _reqq.get() - File "/usr/lib64/python3.6/queue.py", line 164, in get - self.not_empty.wait() - File "/usr/lib64/python3.6/threading.py", line 295, in wait - waiter.acquire() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker + msg = _reqq.get() + File "/usr/lib64/python3.6/queue.py", line 164, in get + self.not_empty.wait() + File "/usr/lib64/python3.6/threading.py", line 295, in wait + waiter.acquire() Stack for thread 281473120006656 - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "/usr/lib64/python3.6/threading.py", line 864, in run - self._target(*self._args, **self._kwargs) - File "/usr/local/lib/python3.6/site-packages/nova/virt/libvirt/host.py", line 205, in _native_thread - libvirt.virEventRunDefaultImpl() - File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 442, in virEventRunDefaultImpl - ret = libvirtmod.virEventRunDefaultImpl() + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "/usr/lib64/python3.6/threading.py", line 864, in run + self._target(*self._args, **self._kwargs) + File "/usr/local/lib/python3.6/site-packages/nova/virt/libvirt/host.py", line 205, in _native_thread + libvirt.virEventRunDefaultImpl() + File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 442, in virEventRunDefaultImpl + ret = libvirtmod.virEventRunDefaultImpl() Stack for thread 281473614563696 - File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 42, in __thread_body - func(*args, **kwargs) - File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap - self._bootstrap_inner() - File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 63, in wrap_bootstrap_inner - bootstrap_inner() - File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner - self.run() - File "<string>", line 167, in run - File "/usr/lib64/python3.6/code.py", line 233, in interact - more = self.push(line) - File "/usr/lib64/python3.6/code.py", line 259, in push - more = self.runsource(source, self.filename) - File "/usr/lib64/python3.6/code.py", line 75, in runsource - self.runcode(code) - File "/usr/lib64/python3.6/code.py", line 91, in runcode - exec(code, self.locals) - File "<console>", line 3, in <module> - + File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 42, in __thread_body + func(*args, **kwargs) + File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap + self._bootstrap_inner() + File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 63, in wrap_bootstrap_inner + bootstrap_inner() + File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner + self.run() + File "<string>", line 167, in run + File "/usr/lib64/python3.6/code.py", line 233, in interact + more = self.push(line) + File "/usr/lib64/python3.6/code.py", line 259, in push + more = self.runsource(source, self.filename) + File "/usr/lib64/python3.6/code.py", line 75, in runsource + self.runcode(code) + File "/usr/lib64/python3.6/code.py", line 91, in runcode + exec(code, self.locals) + File "<console>", line 3, in <module> ======================================================== I tried to analyze why I got stuck, but it took me a long time to figure it out, so looking for community help, thanks. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1971663 Title: nova-compute process stuck without any actions Status in OpenStack Compute (nova): New Bug description: Description =========== nova-compute process gets stuck after running for a while Steps to reproduce ================== Sorry, I haven't specified how to reproduce,but I know this problem has occurred at least three times in the arm environment, but not on x86. Expected result =============== nova-compute run normally Actual result ============= 2022-04-28 15:07:15.150 5967 DEBUG oslo_concurrency.lockutils [-] Lock "ca12badc-42a2-43f2-8a28-cb2afa3a377c" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "53988d8e-75bc-40de-b9af-ce7509f94918" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "bc2b5ad1-a7a4-45fc-8d95-e986476c3129" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "f2068deb-c12f-4c31-998c-cf48af4440ff" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "48577b65-28e2-4858-abed-db00f90c0dc6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "5e6a68fc-b711-4115-bc77-179a224ed5c6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "4f47003a-c056-4aa6-9be2-e5bec0cb4a07" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "a1416265-68cb-4e06-a1b9-80c0bd138a09" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "96408c82-5951-46e6-b105-bcb21118ed99" acquired by "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 17:11:03.055 5967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers timer() File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__ cb(*args, **kw) File "/usr/local/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire waiter.switch() greenlet.error: cannot switch to a different thread Environment =========== Python 3.6.8 eventlet 0.30.2 oslo.concurrency 4.4.0 oslo.messaging 12.7.1 nova-compute Stack list ====================== Stack for thread 281471352828416 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281473394471424 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281471310557696 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281471838974464 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281471847428608 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281471855882752 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281471864336896 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281471872791040 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281471881245184 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281471889699328 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472375845376 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472384299520 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472392753664 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472401207808 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472409661952 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472418116096 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472426570240 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472979825152 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472988279296 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281472996733440 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473005187584 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473013641728 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473022095872 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473030550016 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473111552512 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/eventlet/tpool.py", line 75, in tworker msg = _reqq.get() File "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() Stack for thread 281473120006656 File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/site-packages/nova/virt/libvirt/host.py", line 205, in _native_thread libvirt.virEventRunDefaultImpl() File "/usr/local/lib64/python3.6/site-packages/libvirt.py", line 442, in virEventRunDefaultImpl ret = libvirtmod.virEventRunDefaultImpl() Stack for thread 281473614563696 File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 42, in __thread_body func(*args, **kwargs) File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/local/lib/python3.6/site-packages/eventlet/green/thread.py", line 63, in wrap_bootstrap_inner bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "<string>", line 167, in run File "/usr/lib64/python3.6/code.py", line 233, in interact more = self.push(line) File "/usr/lib64/python3.6/code.py", line 259, in push more = self.runsource(source, self.filename) File "/usr/lib64/python3.6/code.py", line 75, in runsource self.runcode(code) File "/usr/lib64/python3.6/code.py", line 91, in runcode exec(code, self.locals) File "<console>", line 3, in <module> ======================================================== I tried to analyze why I got stuck, but it took me a long time to figure it out, so looking for community help, thanks. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1971663/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp