[Yahoo-eng-team] [Bug 1331537] Re: nova service-list shows nova-compute as down and is required to be restarted frequently in order to provision new vms
This is an automated cleanup. This bug report has been closed because it is older than 18 months and there is no open code change to fix this. After this time it is unlikely that the circumstances which lead to the observed issue can be reproduced. If you can reproduce the bug, please: * reopen the bug report (set to status "New") * AND add the detailed steps to reproduce the issue (if applicable) * AND leave a comment "CONFIRMED FOR: " Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON). Valid example: CONFIRMED FOR: LIBERTY ** Changed in: nova Importance: Medium => Undecided ** Changed in: nova Status: Confirmed => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1331537 Title: nova service-list shows nova-compute as down and is required to be restarted frequently in order to provision new vms Status in OpenStack Compute (nova): Expired Bug description: Nova compute services in Openstack Havana go down frequently as listed by "nova service-list" and requires to be restarted very frequently, multiple times every day. All the compute nodes have the ntp times in sync. When a node shows down, it is not able to use those compute nodes for launching new VMs and we quickly run out of compute resources. Hence our workaround is to restart the Compute nodes on those servers hourly. In the nova-compute node I've found the following error and they did match with the "Updated_at" field from nova service-list. 2014-06-07 00:21:15.690 511340 ERROR nova.servicegroup.drivers.db [-] model server went away 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db Traceback (most recent call last): 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/servicegroup/drivers/db.py", l ine 92, in _report_state 5804 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db report_count = service.service_ref['report_count'] + 1 5805 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db TypeError: 'NoneType' object has no attribute '__getitem__' 5806 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db It looks like the ones that are shown as down haven't been able to update the database with the latest status and they did match with the Traceback seen above (2014-06-07 00:21:15.690) on at least two compute nodes that I have seen. +--++--+--+---++-+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +--++--+--+---++-+ | nova-compute | nova1| blabla | enabled | up | 2014-06-07T00:37:42.00 | None | | nova-compute | nova2 | blabla | enabled | down | 2014-06-07T00:21:05.00 | None | To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1331537/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1331537] Re: nova service-list shows nova-compute as down and is required to be restarted frequently in order to provision new vms
** Changed in: nova Status: Won't Fix => New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1331537 Title: nova service-list shows nova-compute as down and is required to be restarted frequently in order to provision new vms Status in OpenStack Compute (nova): Incomplete Bug description: Nova compute services in Openstack Havana go down frequently as listed by "nova service-list" and requires to be restarted very frequently, multiple times every day. All the compute nodes have the ntp times in sync. When a node shows down, it is not able to use those compute nodes for launching new VMs and we quickly run out of compute resources. Hence our workaround is to restart the Compute nodes on those servers hourly. In the nova-compute node I've found the following error and they did match with the "Updated_at" field from nova service-list. 2014-06-07 00:21:15.690 511340 ERROR nova.servicegroup.drivers.db [-] model server went away 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db Traceback (most recent call last): 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/servicegroup/drivers/db.py", l ine 92, in _report_state 5804 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db report_count = service.service_ref['report_count'] + 1 5805 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db TypeError: 'NoneType' object has no attribute '__getitem__' 5806 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db It looks like the ones that are shown as down haven't been able to update the database with the latest status and they did match with the Traceback seen above (2014-06-07 00:21:15.690) on at least two compute nodes that I have seen. +--++--+--+---++-+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +--++--+--+---++-+ | nova-compute | nova1| blabla | enabled | up | 2014-06-07T00:37:42.00 | None | | nova-compute | nova2 | blabla | enabled | down | 2014-06-07T00:21:05.00 | None | To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1331537/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1331537] Re: nova service-list shows nova-compute as down and is required to be restarted frequently in order to provision new vms
Cleanup === This bug report has the status "Incomplete" since more than 30 days and it looks like that there are no open reviews for it. To keep the bug list sane, I close this bug with "won't fix". This does not mean that it is not a valid bug report, it's more to acknowledge that no progress can be expected here anymore. You are still free to push a new patch for this bug. If you could reproduce it on the current master code or on a maintained stable branch, please switch it to "Confirmed". ** Changed in: nova Status: Incomplete => Won't Fix ** Changed in: nova Assignee: Roman Podoliaka (rpodolyaka) => (unassigned) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1331537 Title: nova service-list shows nova-compute as down and is required to be restarted frequently in order to provision new vms Status in OpenStack Compute (nova): Won't Fix Bug description: Nova compute services in Openstack Havana go down frequently as listed by "nova service-list" and requires to be restarted very frequently, multiple times every day. All the compute nodes have the ntp times in sync. When a node shows down, it is not able to use those compute nodes for launching new VMs and we quickly run out of compute resources. Hence our workaround is to restart the Compute nodes on those servers hourly. In the nova-compute node I've found the following error and they did match with the "Updated_at" field from nova service-list. 2014-06-07 00:21:15.690 511340 ERROR nova.servicegroup.drivers.db [-] model server went away 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db Traceback (most recent call last): 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/servicegroup/drivers/db.py", l ine 92, in _report_state 5804 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db report_count = service.service_ref['report_count'] + 1 5805 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db TypeError: 'NoneType' object has no attribute '__getitem__' 5806 2014-06-07 00:21:15.690 511340 TRACE nova.servicegroup.drivers.db It looks like the ones that are shown as down haven't been able to update the database with the latest status and they did match with the Traceback seen above (2014-06-07 00:21:15.690) on at least two compute nodes that I have seen. +--++--+--+---++-+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +--++--+--+---++-+ | nova-compute | nova1| blabla | enabled | up | 2014-06-07T00:37:42.00 | None | | nova-compute | nova2 | blabla | enabled | down | 2014-06-07T00:21:05.00 | None | To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1331537/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
