[Openstack] [Ocata-Xenserver] Nova-compute wouldn't start
Hi all,
I have openstack RDO ocata using xenserver as Hypervisor, setup manually.
It works fine about 2 weeks, today my nova-compute wouldn't start and when
I check the nova-compute log :
2017-05-18 16:29:24.769 10764 INFO nova.service [-] Starting compute node
(version 15.0.0-1.el7)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
[req-f29142ce-c68a-4d0a-b6ff-ba5b0a67ec6a - - - - -] Error starting thread.
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service Traceback (most
recent call last):
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/oslo_service/service.py", line 722, in
run_service
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service service.start()
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/service.py", line 144, in start
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
self.manager.init_host()
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1152, in
init_host
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
self._init_instance(context, instance)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 949, in
_init_instance
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
self.driver.plug_vifs(instance, net_info)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 309, in
plug_vifs
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
self._vmops.plug_vifs(instance, network_info)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/virt/xenapi/vmops.py", line 1959, in
plug_vifs
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
self.vif_driver.plug(instance, vif, device=device)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/virt/xenapi/vif.py", line 250, in
plug
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service vif_ref =
self._get_vif_ref(vif, vm_ref)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/virt/xenapi/vif.py", line 42, in
_get_vif_ref
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service vif_refs =
self._session.call_xenapi("VM.get_VIFs", vm_ref)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/os_xenapi/client/session.py", line 200,
in call_xenapi
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service return
session.xenapi_request(method, args)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/os_xenapi/client/XenAPI.py", line 130, in
xenapi_request
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service result =
_parse_result(getattr(self, methodname)(*full_params))
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service return
self.__send(self.__name, args)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib64/python2.7/xmlrpclib.py", line 1581, in __request
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
allow_none=self.__allow_none)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib64/python2.7/xmlrpclib.py", line 1086, in dumps
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service data =
m.dumps(params)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib64/python2.7/xmlrpclib.py", line 633, in dumps
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service dump(v, write)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib64/python2.7/xmlrpclib.py", line 655, in __dump
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service f(self, value,
write)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib64/python2.7/xmlrpclib.py", line 659, in dump_nil
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service raise
TypeError, "cannot marshal None unless allow_none is enabled"
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service TypeError: cannot
marshal None unless allow_none is enabled
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
Anyone can help me with this problem ?
--
Cheers,
[image: --]
Adhi Priharmanto
[image: http://]about.me/a_dhi
+62-812-82121584
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start
Hi Adhi,
Very interesting. I suspect that self._get_vif_ref, below, is returning None.
This appears to be when the VM does not exist on the host - perhaps the VM was
renamed, deleted, or it has been migrated to a different host and Nova’s
records didn’t get updated?
I believe the issue is most likely occurring at
http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/xenapi/vif.py?h=15.0.4#n246
where we attempt to detect the VM based on the instance’s name in Nova’s
database.
Do you have VMs on this host? They don’t have to be running (for example if the
host rebooted then the VMs may exist but in a shutdown state)
Could you check that the VMs that are on the host still have the name
associated with them according to Nova?
For example:
root@host~/openrc# openstack server show 3662f9b8-1303-496a-8c21-97bba312432c
…
| OS-EXT-SRV-ATTR:instance_name| instance-914e
|
…
| name |
dsvm-devstack-citrix-mia-nodepool-683420
|
In this case, the server has the name dsvm-devstack-citrix-mia-nodepool-683420
but on the hypervisor the VM itself has the name instance-914e
Thanks,
Bob
From: Adhi Priharmanto [mailto:adhi@gmail.com]
Sent: 18 May 2017 10:34
To: openstack
Subject: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start
Hi all,
I have openstack RDO ocata using xenserver as Hypervisor, setup manually. It
works fine about 2 weeks, today my nova-compute wouldn't start and when I check
the nova-compute log :
2017-05-18 16:29:24.769 10764 INFO nova.service [-] Starting compute node
(version 15.0.0-1.el7)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
[req-f29142ce-c68a-4d0a-b6ff-ba5b0a67ec6a - - - - -] Error starting thread.
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service Traceback (most recent
call last):
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/oslo_service/service.py", line 722, in
run_service
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service service.start()
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/service.py", line 144, in start
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
self.manager.init_host()
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1152, in
init_host
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
self._init_instance(context, instance)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 949, in
_init_instance
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
self.driver.plug_vifs(instance, net_info)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 309, in
plug_vifs
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
self._vmops.plug_vifs(instance, network_info)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/virt/xenapi/vmops.py", line 1959, in
plug_vifs
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
self.vif_driver.plug(instance, vif, device=device)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/virt/xenapi/vif.py", line 250, in plug
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service vif_ref =
self._get_vif_ref(vif, vm_ref)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/virt/xenapi/vif.py", line 42, in
_get_vif_ref
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service vif_refs =
self._session.call_xenapi("VM.get_VIFs", vm_ref)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/os_xenapi/client/session.py", line 200, in
call_xenapi
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service return
session.xenapi_request(method, args)
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/os_xenapi/client/XenAPI.py", line 130, in
xenapi_request
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service result =
_parse_result(getattr(self, methodname)(*full_params))
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
"/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
2017-05-18 16:29:24.847 10764 ERROR oslo_service.service return
Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start
hi Bob,
I don't have any vm/instance on that compute/xenserver , I'm deleted all vm
since last week.
On Thu, May 18, 2017 at 6:03 PM, Bob Ball wrote:
> Hi Adhi,
>
>
>
> Very interesting. I suspect that self._get_vif_ref, below, is returning
> None. This appears to be when the VM does not exist on the host - perhaps
> the VM was renamed, deleted, or it has been migrated to a different host
> and Nova’s records didn’t get updated?
>
>
>
> I believe the issue is most likely occurring at
> http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/
> xenapi/vif.py?h=15.0.4#n246 where we attempt to detect the VM based on
> the instance’s name in Nova’s database.
>
> Do you have VMs on this host? They don’t have to be running (for example
> if the host rebooted then the VMs may exist but in a shutdown state)
>
>
>
> Could you check that the VMs that are on the host still have the name
> associated with them according to Nova?
>
> For example:
>
> root@host~/openrc# openstack server show 3662f9b8-1303-496a-8c21-
> 97bba312432c
>
> …
>
> | OS-EXT-SRV-ATTR:instance_name| instance-914e
>
>
> |
>
> …
>
> | name | dsvm-devstack-citrix-mia-
> nodepool-683420
>
> |
>
>
>
> In this case, the server has the name dsvm-devstack-citrix-mia-nodepool-683420
> but on the hypervisor the VM itself has the name instance-914e
>
>
>
> Thanks,
>
>
>
> Bob
>
>
>
>
>
> *From:* Adhi Priharmanto [mailto:adhi@gmail.com]
> *Sent:* 18 May 2017 10:34
> *To:* openstack
> *Subject:* [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start
>
>
>
> Hi all,
>
> I have openstack RDO ocata using xenserver as Hypervisor, setup manually.
> It works fine about 2 weeks, today my nova-compute wouldn't start and when
> I check the nova-compute log :
>
> 2017-05-18 16:29:24.769 10764 INFO nova.service [-] Starting compute node
> (version 15.0.0-1.el7)
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
> [req-f29142ce-c68a-4d0a-b6ff-ba5b0a67ec6a - - - - -] Error starting
> thread.
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service Traceback (most
> recent call last):
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
> "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 722, in
> run_service
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
> service.start()
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
> "/usr/lib/python2.7/site-packages/nova/service.py", line 144, in start
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
> self.manager.init_host()
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1152, in
> init_host
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
> self._init_instance(context, instance)
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 949, in
> _init_instance
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
> self.driver.plug_vifs(instance, net_info)
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
> "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 309,
> in plug_vifs
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
> self._vmops.plug_vifs(instance, network_info)
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
> "/usr/lib/python2.7/site-packages/nova/virt/xenapi/vmops.py", line 1959,
> in plug_vifs
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
> self.vif_driver.plug(instance, vif, device=device)
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
> "/usr/lib/python2.7/site-packages/nova/virt/xenapi/vif.py", line 250, in
> plug
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service vif_ref =
> self._get_vif_ref(vif, vm_ref)
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
> "/usr/lib/python2.7/site-packages/nova/virt/xenapi/vif.py", line 42, in
> _get_vif_ref
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service vif_refs =
> self._session.call_xenapi("VM.get_VIFs", vm_ref)
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File
> "/usr/lib/python2.7/site-packages/os_xenapi/client/session.py", line 200,
> in call_xenapi
> 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service return
> session.xenapi_request(method, args)
> 2017-05-18 16:29:24.847 10764 ERROR oslo_s
Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start
Hi Adhi, The point is that there is a mismatch between what Nova believes exists, and what really exists on the hypervisor. How were the instances deleted? Check the hypervisors that each server Nova knows about - I suspect at least one is expected to exist on the XenServer for which the compute will not start. Bob From: Adhi Priharmanto [mailto:adhi@gmail.com] Sent: 18 May 2017 16:40 To: Bob Ball Cc: openstack ; #OpenStack External Email Subject: Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start hi Bob, I don't have any vm/instance on that compute/xenserver , I'm deleted all vm since last week. On Thu, May 18, 2017 at 6:03 PM, Bob Ball mailto:bob.b...@citrix.com>> wrote: Hi Adhi, Very interesting. I suspect that self._get_vif_ref, below, is returning None. This appears to be when the VM does not exist on the host - perhaps the VM was renamed, deleted, or it has been migrated to a different host and Nova’s records didn’t get updated? I believe the issue is most likely occurring at http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/xenapi/vif.py?h=15.0.4#n246 where we attempt to detect the VM based on the instance’s name in Nova’s database. Do you have VMs on this host? They don’t have to be running (for example if the host rebooted then the VMs may exist but in a shutdown state) Could you check that the VMs that are on the host still have the name associated with them according to Nova? For example: root@host~/openrc# openstack server show 3662f9b8-1303-496a-8c21-97bba312432c … | OS-EXT-SRV-ATTR:instance_name| instance-914e | … | name | dsvm-devstack-citrix-mia-nodepool-683420 | In this case, the server has the name dsvm-devstack-citrix-mia-nodepool-683420 but on the hypervisor the VM itself has the name instance-914e Thanks, Bob From: Adhi Priharmanto [mailto:adhi@gmail.com<mailto:adhi@gmail.com>] Sent: 18 May 2017 10:34 To: openstack mailto:openstack@lists.openstack.org>> Subject: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start Hi all, I have openstack RDO ocata using xenserver as Hypervisor, setup manually. It works fine about 2 weeks, today my nova-compute wouldn't start and when I check the nova-compute log : 2017-05-18 16:29:24.769 10764 INFO nova.service [-] Starting compute node (version 15.0.0-1.el7) 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service [req-f29142ce-c68a-4d0a-b6ff-ba5b0a67ec6a - - - - -] Error starting thread. 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service Traceback (most recent call last): 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 722, in run_service 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service service.start() 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 144, in start 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service self.manager.init_host() 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1152, in init_host 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service self._init_instance(context, instance) 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 949, in _init_instance 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service self.driver.plug_vifs(instance, net_info) 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 309, in plug_vifs 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service self._vmops.plug_vifs(instance, network_info) 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/vmops.py", line 1959, in plug_vifs 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service self.vif_driver.plug(instance, vif, device=device) 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/vif.py", line 250, in plug 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service vif_ref = self._get_vif_ref(vif, vm_ref) 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/vif.py", line 42, in _get_vif_ref 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service vif_refs = self._session.c
Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start
Hi bob, Thanks for your suggest, because it's non production use, I decide to hardcoded clean up instance record from nova table, and finally nova-compute service was running again. :) On Thu, May 18, 2017 at 10:53 PM, Bob Ball wrote: > Hi Adhi, > > > > The point is that there is a mismatch between what Nova believes exists, > and what really exists on the hypervisor. How were the instances deleted? > > > > Check the hypervisors that each server Nova knows about - I suspect at > least one is expected to exist on the XenServer for which the compute will > not start. > > > > Bob > > > > *From:* Adhi Priharmanto [mailto:adhi@gmail.com] > *Sent:* 18 May 2017 16:40 > *To:* Bob Ball > *Cc:* openstack ; #OpenStack External > Email > *Subject:* Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start > > > > hi Bob, > > I don't have any vm/instance on that compute/xenserver , I'm deleted all > vm since last week. > > > > On Thu, May 18, 2017 at 6:03 PM, Bob Ball wrote: > > Hi Adhi, > > > > Very interesting. I suspect that self._get_vif_ref, below, is returning > None. This appears to be when the VM does not exist on the host - perhaps > the VM was renamed, deleted, or it has been migrated to a different host > and Nova’s records didn’t get updated? > > > > I believe the issue is most likely occurring at > http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/ > xenapi/vif.py?h=15.0.4#n246 where we attempt to detect the VM based on > the instance’s name in Nova’s database. > > Do you have VMs on this host? They don’t have to be running (for example > if the host rebooted then the VMs may exist but in a shutdown state) > > > > Could you check that the VMs that are on the host still have the name > associated with them according to Nova? > > For example: > > root@host~/openrc# openstack server show 3662f9b8-1303-496a-8c21- > 97bba312432c > > … > > | OS-EXT-SRV-ATTR:instance_name| instance-914e > > > | > > … > > | name | dsvm-devstack-citrix-mia- > nodepool-683420 > > | > > > > In this case, the server has the name dsvm-devstack-citrix-mia-nodepool-683420 > but on the hypervisor the VM itself has the name instance-914e > > > > Thanks, > > > > Bob > > > > > > *From:* Adhi Priharmanto [mailto:adhi@gmail.com] > *Sent:* 18 May 2017 10:34 > *To:* openstack > *Subject:* [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start > > > > Hi all, > > I have openstack RDO ocata using xenserver as Hypervisor, setup manually. > It works fine about 2 weeks, today my nova-compute wouldn't start and when > I check the nova-compute log : > > 2017-05-18 16:29:24.769 10764 INFO nova.service [-] Starting compute node > (version 15.0.0-1.el7) > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > [req-f29142ce-c68a-4d0a-b6ff-ba5b0a67ec6a - - - - -] Error starting > thread. > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service Traceback (most > recent call last): > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File > "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 722, in > run_service > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > service.start() > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File > "/usr/lib/python2.7/site-packages/nova/service.py", line 144, in start > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > self.manager.init_host() > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File > "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1152, in > init_host > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > self._init_instance(context, instance) > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File > "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 949, in > _init_instance > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > self.driver.plug_vifs(instance, net_info) > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File > "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 309, > in plug_vifs > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > self._vmops.plug_vifs(instance, network_info) > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File > "/usr/lib/python2.7/site-packages/nova/virt/xenapi/vmops.py", line 1959, > in plug_vifs > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > self.vif_dri
Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start
Hi Adhi, Please see bug report at https://bugs.launchpad.net/nova/+bug/1693147 which I believe covers your issue. Huan is working on a fix at https://review.openstack.org/#/c/467926 Thanks, Bob From: Adhi Priharmanto [mailto:adhi@gmail.com] Sent: 18 May 2017 17:52 To: Bob Ball Cc: openstack ; #OpenStack External Email Subject: Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start Hi bob, Thanks for your suggest, because it's non production use, I decide to hardcoded clean up instance record from nova table, and finally nova-compute service was running again. :) On Thu, May 18, 2017 at 10:53 PM, Bob Ball mailto:bob.b...@citrix.com>> wrote: Hi Adhi, The point is that there is a mismatch between what Nova believes exists, and what really exists on the hypervisor. How were the instances deleted? Check the hypervisors that each server Nova knows about - I suspect at least one is expected to exist on the XenServer for which the compute will not start. Bob From: Adhi Priharmanto [mailto:adhi@gmail.com<mailto:adhi@gmail.com>] Sent: 18 May 2017 16:40 To: Bob Ball mailto:bob.b...@citrix.com>> Cc: openstack mailto:openstack@lists.openstack.org>>; #OpenStack External Email mailto:openst...@citrix.com>> Subject: Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start hi Bob, I don't have any vm/instance on that compute/xenserver , I'm deleted all vm since last week. On Thu, May 18, 2017 at 6:03 PM, Bob Ball mailto:bob.b...@citrix.com>> wrote: Hi Adhi, Very interesting. I suspect that self._get_vif_ref, below, is returning None. This appears to be when the VM does not exist on the host - perhaps the VM was renamed, deleted, or it has been migrated to a different host and Nova’s records didn’t get updated? I believe the issue is most likely occurring at http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/xenapi/vif.py?h=15.0.4#n246 where we attempt to detect the VM based on the instance’s name in Nova’s database. Do you have VMs on this host? They don’t have to be running (for example if the host rebooted then the VMs may exist but in a shutdown state) Could you check that the VMs that are on the host still have the name associated with them according to Nova? For example: root@host~/openrc# openstack server show 3662f9b8-1303-496a-8c21-97bba312432c … | OS-EXT-SRV-ATTR:instance_name| instance-914e | … | name | dsvm-devstack-citrix-mia-nodepool-683420 | In this case, the server has the name dsvm-devstack-citrix-mia-nodepool-683420 but on the hypervisor the VM itself has the name instance-914e Thanks, Bob From: Adhi Priharmanto [mailto:adhi@gmail.com<mailto:adhi@gmail.com>] Sent: 18 May 2017 10:34 To: openstack mailto:openstack@lists.openstack.org>> Subject: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start Hi all, I have openstack RDO ocata using xenserver as Hypervisor, setup manually. It works fine about 2 weeks, today my nova-compute wouldn't start and when I check the nova-compute log : 2017-05-18 16:29:24.769 10764 INFO nova.service [-] Starting compute node (version 15.0.0-1.el7) 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service [req-f29142ce-c68a-4d0a-b6ff-ba5b0a67ec6a - - - - -] Error starting thread. 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service Traceback (most recent call last): 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 722, in run_service 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service service.start() 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 144, in start 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service self.manager.init_host() 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1152, in init_host 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service self._init_instance(context, instance) 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 949, in _init_instance 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service self.driver.plug_vifs(instance, net_info) 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 309, in plug_vifs 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service
Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start
Hi Bob, Thanks for share this update, I'll try to patch my nova and reproduce the problem again. On Tue, May 30, 2017 at 5:25 PM, Bob Ball wrote: > Hi Adhi, > > > > Please see bug report at https://bugs.launchpad.net/nova/+bug/1693147 > which I believe covers your issue. > > > > Huan is working on a fix at https://review.openstack.org/#/c/467926 > > > > Thanks, > > > > Bob > > > > *From:* Adhi Priharmanto [mailto:adhi@gmail.com] > *Sent:* 18 May 2017 17:52 > > *To:* Bob Ball > *Cc:* openstack ; #OpenStack External > Email > *Subject:* Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start > > > > Hi bob, > > Thanks for your suggest, because it's non production use, I decide to > hardcoded clean up instance record from nova table, and finally > nova-compute service was running again. :) > > > > On Thu, May 18, 2017 at 10:53 PM, Bob Ball wrote: > > Hi Adhi, > > > > The point is that there is a mismatch between what Nova believes exists, > and what really exists on the hypervisor. How were the instances deleted? > > > > Check the hypervisors that each server Nova knows about - I suspect at > least one is expected to exist on the XenServer for which the compute will > not start. > > > > Bob > > > > *From:* Adhi Priharmanto [mailto:adhi@gmail.com] > *Sent:* 18 May 2017 16:40 > *To:* Bob Ball > *Cc:* openstack ; #OpenStack External > Email > *Subject:* Re: [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start > > > > hi Bob, > > I don't have any vm/instance on that compute/xenserver , I'm deleted all > vm since last week. > > > > On Thu, May 18, 2017 at 6:03 PM, Bob Ball wrote: > > Hi Adhi, > > > > Very interesting. I suspect that self._get_vif_ref, below, is returning > None. This appears to be when the VM does not exist on the host - perhaps > the VM was renamed, deleted, or it has been migrated to a different host > and Nova’s records didn’t get updated? > > > > I believe the issue is most likely occurring at > http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/ > xenapi/vif.py?h=15.0.4#n246 where we attempt to detect the VM based on > the instance’s name in Nova’s database. > > Do you have VMs on this host? They don’t have to be running (for example > if the host rebooted then the VMs may exist but in a shutdown state) > > > > Could you check that the VMs that are on the host still have the name > associated with them according to Nova? > > For example: > > root@host~/openrc# openstack server show 3662f9b8-1303-496a-8c21- > 97bba312432c > > … > > | OS-EXT-SRV-ATTR:instance_name| instance-914e > > > | > > … > > | name | dsvm-devstack-citrix-mia- > nodepool-683420 > > | > > > > In this case, the server has the name dsvm-devstack-citrix-mia-nodepool-683420 > but on the hypervisor the VM itself has the name instance-914e > > > > Thanks, > > > > Bob > > > > > > *From:* Adhi Priharmanto [mailto:adhi@gmail.com] > *Sent:* 18 May 2017 10:34 > *To:* openstack > *Subject:* [Openstack] [Ocata-Xenserver] Nova-compute wouldn't start > > > > Hi all, > > I have openstack RDO ocata using xenserver as Hypervisor, setup manually. > It works fine about 2 weeks, today my nova-compute wouldn't start and when > I check the nova-compute log : > > 2017-05-18 16:29:24.769 10764 INFO nova.service [-] Starting compute node > (version 15.0.0-1.el7) > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > [req-f29142ce-c68a-4d0a-b6ff-ba5b0a67ec6a - - - - -] Error starting > thread. > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service Traceback (most > recent call last): > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File > "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 722, in > run_service > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > service.start() > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File > "/usr/lib/python2.7/site-packages/nova/service.py", line 144, in start > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > self.manager.init_host() > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service File > "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1152, in > init_host > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service > self._init_instance(context, instance) > 2017-05-18 16:29:24.847 10764 ERROR oslo_service.service F
