Re: [Openstack] [nova]How to convince the RetryFilter that a host is recovered from a failure?
On Wed, Nov 15, 2017 at 9:59 AM, Song Sam wrote:
Hi, forks:
I have a openstack o version cluster with two compute nodes. The
resources are almost used up.
So, I add a new compute node to a running cluster and try to cold
migrate a vm to the new compute node. At the first time, the
migration failed with 'No valid host found' for my mistake. I forgot
to mount the data disk to extend local available storage.
After I extend the storage, I confirmed the hypervisor have
recognized the free disk space, as shown in the output of 'openstack
hypervisor show'. But when I try to cold migrate again, It failed
with 'No valid host found' again. After some dig in the scheduler
log, I find the RetryFilter reject the new added host.
It seems to me that you hit this bug:
https://bugs.launchpad.net/nova/+bug/1718512
My question are:
1. Where does the RetryFilter store the host availability status? In
memory or in DB? Which table?
The RequestSpec persists the already tried hosts and RetryFilter checks
that.
2. How to convince the RetryFilter that the host is available again?
I tried to restart nova-api, nova-scheduler, memcached services, but
it doesn't work.
There is a request_spec table in the nova_api db where the RequestSpec
objects are stored. Unfortunately the content of the object is json
serialized into the spec field of that table so I don't suggest to try
to fix it manually.
cheers,
gibi
The related log lines as following:
2017-11-15 14:24:37.906 7 DEBUG oslo_messaging._drivers.amqpdriver
[-] received message msg_id: fa38b97af9504ebaa1d3118d91157f45 reply
to reply_2c3ce56f29774ed19c504407625c632c __call__
/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:194
2017-11-15 14:24:37.999 7 DEBUG nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Starting with 0 host(s) get_filtered_objects
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:70
2017-11-15 14:24:38.000 7 INFO nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Filter RetryFilter returned 0 hosts
2017-11-15 14:24:38.001 7 DEBUG nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Filtering removed all hosts for the request with instance ID
'14139777-351e-4fe9-9554-3c5729bc88cf'. Filter results:
[('RetryFilter', None)] get_filtered_objects
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/filters.py:129
2017-11-15 14:24:38.001 7 INFO nova.filters
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] Filtering removed all hosts for the request with instance ID
'14139777-351e-4fe9-9554-3c5729bc88cf'. Filter results:
['RetryFilter: (start: 0, end: 0)']
2017-11-15 14:24:38.002 7 DEBUG nova.scheduler.filter_scheduler
[req-25d6619e-c611-4ecd-b696-f5cc8a39e93d
a5efca4a04654a829ee19229df707e08 2e13a26ead3c4da493c5c234233308df - -
-] There are 0 hosts available but 1 instances requested to build.
select_destinations
/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:76
Any help are appreciated.
Thanks
Sam Song
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [nova] Accessing instance.flavor.projects fails due to orphaned Flavor
On Thu, Jan 12, 2017 at 4:56 PM, Jay Pipes wrote: On 01/12/2017 05:31 AM, Balazs Gibizer wrote: Hi, The flavor field of the Instance object is a lazy-loaded field and the projects field of the Flavor object is also lazy-loaded. Now it seems to me that when the Instance object lazy loads instance.flavor then the created Flavor object is orphaned [1] therefore instance.flavor.projects will never work and result in an exceptuion: OrphanedObjectError: Cannot call _load_projects on orphaned Flavor object. Is the Flavor left orphaned by intention or it is a bug? Depends :) I would say it is intentional for the most part. Is there a reason why the Flavor *notification* payload needs to contain a list of projects associated with the flavor? My gut says that information isn't particularly germane to the relationship of the Instance to the Flavor? The whole thing came up as part of the https://blueprints.launchpad.net/nova/+spec/flavor-notifications where the FlavorPayload was extended with flavor.projects. As the same FlavorPayload is used in the instance. notifications the instance notification code path also needs the flavor.projects field. The payload of instance. notifications contains the flavor related data of the instance in question and to have the flavor.projects in the payload as well the code would need to access the projects field via instance.flavor.projects. Sure, I understand it would ease the access to the projects field in the notification payload packing, but is there really a reason to bother retrieving and sending that data each time an Instance notification event is made (which is quite often)? So it is mainly there to have a single, consistent FlavorPayload used across notifications. Sure we could include only just the flavor_id in the instance. notifications. However there was a similar discussions how to handle delete notification [1]. There we decided to include the whole entity to the delete not just the uuid of the deleted entity. There the main reasoning (besides consistency) was that a notification consumer might want to listen only to certain notification but and still want to get enough information to avoid the need of a subsequent REST query. I think the same reasoning could be applied here. Cheers, gibi [1] http://lists.openstack.org/pipermail/openstack-dev/2017-January/109508.html Best, -jay ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [nova] Accessing instance.flavor.projects fails due to orphaned Flavor
On Thu, Jan 12, 2017 at 2:31 PM, Balazs Gibizer wrote: Hi, The flavor field of the Instance object is a lazy-loaded field and the projects field of the Flavor object is also lazy-loaded. Now it seems to me that when the Instance object lazy loads instance.flavor then the created Flavor object is orphaned [1] therefore instance.flavor.projects will never work and result in an exceptuion: OrphanedObjectError: Cannot call _load_projects on orphaned Flavor object. Is the Flavor left orphaned by intention or it is a bug? The payload of instance. notifications contains the flavor related data of the instance in question and to have the flavor.projects in the payload as well the code would need to access the projects field via instance.flavor.projects. Cheers, gibi [1] https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L315 Forget to add [nova] to the subject. Cheers, gibi ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] Accessing instance.flavor.projects fails due to orphaned Flavor
Hi, The flavor field of the Instance object is a lazy-loaded field and the projects field of the Flavor object is also lazy-loaded. Now it seems to me that when the Instance object lazy loads instance.flavor then the created Flavor object is orphaned [1] therefore instance.flavor.projects will never work and result in an exceptuion: OrphanedObjectError: Cannot call _load_projects on orphaned Flavor object. Is the Flavor left orphaned by intention or it is a bug? The payload of instance. notifications contains the flavor related data of the instance in question and to have the flavor.projects in the payload as well the code would need to access the projects field via instance.flavor.projects. Cheers, gibi [1] https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L315 ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
