On Wed, Nov 15, 2017 at 9:59 AM, Song Sam <samsong8...@gmail.com> wrote:
Hi, forks:
I have a openstack o version cluster with two compute nodes. The
resources are almost used up.
So, I add a new compute node to a running cluster and try to cold
migrate a vm to the new compute node. At the first time, the
migration failed with 'No valid host found' for my mistake. I forgot
to mount the data disk to extend local available storage.
After I extend the storage, I confirmed the hypervisor have
recognized the free disk space, as shown in the output of 'openstack
hypervisor show'. But when I try to cold migrate again, It failed
with 'No valid host found' again. After some dig in the scheduler
log, I find the RetryFilter reject the new added host.
It seems to me that you hit this bug:
https://bugs.launchpad.net/nova/+bug/1718512
My question are:
1. Where does the RetryFilter store the host availability status? In
memory or in DB? Which table?
The RequestSpec persists the already tried hosts and RetryFilter checks
that.
2. How to convince the RetryFilter that the host is available again?
I tried to restart nova-api, nova-scheduler, memcached services, but
it doesn't work.
There is a request_spec table in the nova_api db where the RequestSpec
objects are stored. Unfortunately the content of the object is json
serialized into the spec field of that table so I don't suggest to try
to fix it manually.
cheers,
gibi
The related log lines as following:
2017-11-15 14:24:37.906 7 DEBUG oslo_messaging._drivers.amqpdriver
[-] received message msg_id: fa38b97af9504ebaa1d3118d91157f45 reply
to reply_2c3ce56f29774ed19c504407625c632c __call__
/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:194
2017-11-15 14:24:37.999 7 DEBUG nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Starting with 0 host(s) get_filtered_objects
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:70
2017-11-15 14:24:38.000 7 INFO nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Filter RetryFilter returned 0 hosts
2017-11-15 14:24:38.001 7 DEBUG nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Filtering removed all hosts for the request with instance ID
'14139777-351e-4fe9-9554-3c5729bc88cf'. Filter results:
[('RetryFilter', None)] get_filtered_objects
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:129
2017-11-15 14:24:38.001 7 INFO nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Filtering removed all hosts for the request with instance ID
'14139777-351e-4fe9-9554-3c5729bc88cf'. Filter results:
['RetryFilter: (start: 0, end: 0)']
2017-11-15 14:24:38.002 7 DEBUG nova.scheduler.filter_scheduler
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] There are 0 hosts available but 1 instances requested to build.
select_destinations
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:76
Any help are appreciated.
Thanks
Sam Song
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
