Re: [Openstack-operators] [nova] nova-compute automatically disabling itself?

2018-06-07 Thread Matt Riedemann
On 2/6/2018 6:44 PM, Matt Riedemann wrote: On 2/6/2018 2:14 PM, Chris Apsey wrote: but we would rather have intermittent build failures rather than compute nodes falling over in the future. Note that once a compute has a successful build, the consecutive build failures counter is reset. So if

Re: [Openstack-operators] [nova] nova-compute automatically disabling itself?

2018-02-06 Thread Matt Riedemann
On 2/6/2018 2:14 PM, Chris Apsey wrote: but we would rather have intermittent build failures rather than compute nodes falling over in the future. Note that once a compute has a successful build, the consecutive build failures counter is reset. So if your limit is the default (10) and you hav

Re: [Openstack-operators] [nova] nova-compute automatically disabling itself?

2018-02-06 Thread Chris Apsey
All, This was the core issue - setting consecutive_build_service_disable_threshold = 0 in nova.conf (on controllers and compute nodes) solved this. It was being triggered by neutron dropping requests (and/or responses) for vif-plugging due to cpu usage on the neutron endpoints being pegged a

Re: [Openstack-operators] [nova] nova-compute automatically disabling itself?

2018-01-31 Thread Chris Apsey
That looks promising. I'll report back to confirm the solution. Thanks! --- v/r Chris Apsey bitskr...@bitskrieg.net https://www.bitskrieg.net On 2018-01-31 04:40 PM, Matt Riedemann wrote: On 1/31/2018 3:16 PM, Chris Apsey wrote: All, Running in to a strange issue I haven't seen before. Ra

Re: [Openstack-operators] [nova] nova-compute automatically disabling itself?

2018-01-31 Thread Eric Fried
There's [1], but I would have expected you to see error logs like [2] if that's what you're hitting. [1] https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L627-L645 [2] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1714-L1716 efried On 01/31/2018 03:16 P

Re: [Openstack-operators] [nova] nova-compute automatically disabling itself?

2018-01-31 Thread Matt Riedemann
On 1/31/2018 3:16 PM, Chris Apsey wrote: All, Running in to a strange issue I haven't seen before. Randomly, the nova-compute services on compute nodes are disabling themselves (as if someone ran openstack compute service set --disable hostX nova-compute.  When this happens, the node continue

[Openstack-operators] [nova] nova-compute automatically disabling itself?

2018-01-31 Thread Chris Apsey
All, Running in to a strange issue I haven't seen before. Randomly, the nova-compute services on compute nodes are disabling themselves (as if someone ran openstack compute service set --disable hostX nova-compute. When this happens, the node continues to report itself as 'up' - the service