Re: [Openstack] "There are not enough hosts available."
+1 on Mikhail's suggestion. You will want to enable debug logging for the [nova]-scheduler (restart it of course after modifying the config) and then look for errors related to Filter. You can also correlate the req-XXX... ID with logs for nova-compute on your compute node(s) to see if that helps. On Thu, Jul 28, 2016 at 4:50 PM, Mikhail Medvedev wrote: > On Thu, Jul 28, 2016 at 2:34 PM, Ken D'Ambrosio wrote: > > Trying to orchestrate a bunch of VMs on our ~500 core cloud, and we're > > getting the "not enough hosts" problem. Only issue is that the dashboard > > seems to imply we've got the elbow room needed to instantiate. > Furthermore, > > it would be *really handy* if I knew what the bottleneck it thought was: > > disk, RAM, CPU, whatever. Here's a sample of the error: > > > > --- begin --- > > 2016-07-28 18:40:47.961 1080 WARNING nova.scheduler.utils > > [req-977badf4-a4a2-4766-87fd-c4e3ceb2e293 > a4c895a6fe294ae3a81e3c6bb9209f52 > > 33c2cf3cb61c425abcb0e0d35fdf21b7 - - -] [instance: > > ee175a40-11d9-4a43-b6e1-c4d7935acff1] Setting instance to ERROR state. > > 2016-07-28 18:40:51.892 1023 WARNING nova.scheduler.utils > > [req-e2a65200-1662-4cd7-aa9e-db5e225adfa1 > a4c895a6fe294ae3a81e3c6bb9209f52 > > 33c2cf3cb61c425abcb0e0d35fdf21b7 - - -] Failed to > > compute_task_build_instances: No valid host was found. There are not > enough > > hosts available. > > Traceback (most recent call last): > > > > File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", > line > > 142, in inner > > return func(*args, **kwargs) > > > > File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line > > 84, in select_destinations > > filter_properties) > > > > File > > "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", > line > > 90, in select_destinations > > raise exception.NoValidHost(reason=reason) > > > > NoValidHost: No valid host was found. There are not enough hosts > available. > > --- begin --- > > > > Any suggestions on how to track down exactly what's going wrong? > > Try taking a look at nova-scheduler log on your controller (usually in > /var/log/nova/nova-scheduler.log). I do not remember any time I have > seen the "not enough hosts" when the reason was not obvious from the > scheduler log. > > > > > Thanks! > > > > -Ken > > > > ___ > > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > --- > Mikhail Medvedev > IBM > > ___ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] "There are not enough hosts available."
On Thu, Jul 28, 2016 at 2:34 PM, Ken D'Ambrosio wrote: > Trying to orchestrate a bunch of VMs on our ~500 core cloud, and we're > getting the "not enough hosts" problem. Only issue is that the dashboard > seems to imply we've got the elbow room needed to instantiate. Furthermore, > it would be *really handy* if I knew what the bottleneck it thought was: > disk, RAM, CPU, whatever. Here's a sample of the error: > > --- begin --- > 2016-07-28 18:40:47.961 1080 WARNING nova.scheduler.utils > [req-977badf4-a4a2-4766-87fd-c4e3ceb2e293 a4c895a6fe294ae3a81e3c6bb9209f52 > 33c2cf3cb61c425abcb0e0d35fdf21b7 - - -] [instance: > ee175a40-11d9-4a43-b6e1-c4d7935acff1] Setting instance to ERROR state. > 2016-07-28 18:40:51.892 1023 WARNING nova.scheduler.utils > [req-e2a65200-1662-4cd7-aa9e-db5e225adfa1 a4c895a6fe294ae3a81e3c6bb9209f52 > 33c2cf3cb61c425abcb0e0d35fdf21b7 - - -] Failed to > compute_task_build_instances: No valid host was found. There are not enough > hosts available. > Traceback (most recent call last): > > File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line > 142, in inner > return func(*args, **kwargs) > > File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line > 84, in select_destinations > filter_properties) > > File > "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line > 90, in select_destinations > raise exception.NoValidHost(reason=reason) > > NoValidHost: No valid host was found. There are not enough hosts available. > --- begin --- > > Any suggestions on how to track down exactly what's going wrong? Try taking a look at nova-scheduler log on your controller (usually in /var/log/nova/nova-scheduler.log). I do not remember any time I have seen the "not enough hosts" when the reason was not obvious from the scheduler log. > > Thanks! > > -Ken > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack --- Mikhail Medvedev IBM ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] "There are not enough hosts available."
Ken, Can you take a look on the logs generated in nova-compute, maybe you’ll find a detail information about the root cause of the problem Victor Morales On 7/28/16, 2:34 PM, "Ken D'Ambrosio" wrote: >Trying to orchestrate a bunch of VMs on our ~500 core cloud, and we're >getting the "not enough hosts" problem. Only issue is that the >dashboard seems to imply we've got the elbow room needed to instantiate. > Furthermore, it would be *really handy* if I knew what the bottleneck >it thought was: disk, RAM, CPU, whatever. Here's a sample of the error: > >--- begin --- >2016-07-28 18:40:47.961 1080 WARNING nova.scheduler.utils >[req-977badf4-a4a2-4766-87fd-c4e3ceb2e293 >a4c895a6fe294ae3a81e3c6bb9209f52 33c2cf3cb61c425abcb0e0d35fdf21b7 - - -] >[instance: ee175a40-11d9-4a43-b6e1-c4d7935acff1] Setting instance to >ERROR state. >2016-07-28 18:40:51.892 1023 WARNING nova.scheduler.utils >[req-e2a65200-1662-4cd7-aa9e-db5e225adfa1 >a4c895a6fe294ae3a81e3c6bb9209f52 33c2cf3cb61c425abcb0e0d35fdf21b7 - - -] >Failed to compute_task_build_instances: No valid host was found. There >are not enough hosts available. >Traceback (most recent call last): > > File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", >line 142, in inner > return func(*args, **kwargs) > > File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", >line 84, in select_destinations > filter_properties) > > File >"/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", >line 90, in select_destinations > raise exception.NoValidHost(reason=reason) > >NoValidHost: No valid host was found. There are not enough hosts >available. >--- begin --- > >Any suggestions on how to track down exactly what's going wrong? > >Thanks! > >-Ken > >___ >Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >Post to : openstack@lists.openstack.org >Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] "There are not enough hosts available."
On Jul 28, 2016, at 8:34 PM, Ken D'Ambrosio wrote: > Any suggestions on how to track down exactly what's going wrong? Enable debugging. I've been plagued with this for a couple of weeks now when playing with a "new" Cinder volume driver.. Just before you get that message (if you enable debugging), there's a bunch of Filter XXX returned Y hosts Find the one that "returned 0 hosts" and you have a clue of why it's rejected. -- Imagine you're an idiot and then imagine you're in the government. Oh, sorry. Now I'm repeating myself - Mark Twain ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] "There are not enough hosts available."
Can you respond with flavor details ? Nova boot arguments ? nova service-list Thanks! On Thu, Jul 28, 2016 at 2:34 PM, Ken D'Ambrosio wrote: > Trying to orchestrate a bunch of VMs on our ~500 core cloud, and we're > getting the "not enough hosts" problem. Only issue is that the dashboard > seems to imply we've got the elbow room needed to instantiate. > Furthermore, it would be *really handy* if I knew what the bottleneck it > thought was: disk, RAM, CPU, whatever. Here's a sample of the error: > > --- begin --- > 2016-07-28 18:40:47.961 1080 WARNING nova.scheduler.utils > [req-977badf4-a4a2-4766-87fd-c4e3ceb2e293 a4c895a6fe294ae3a81e3c6bb9209f52 > 33c2cf3cb61c425abcb0e0d35fdf21b7 - - -] [instance: > ee175a40-11d9-4a43-b6e1-c4d7935acff1] Setting instance to ERROR state. > 2016-07-28 18:40:51.892 1023 WARNING nova.scheduler.utils > [req-e2a65200-1662-4cd7-aa9e-db5e225adfa1 a4c895a6fe294ae3a81e3c6bb9209f52 > 33c2cf3cb61c425abcb0e0d35fdf21b7 - - -] Failed to > compute_task_build_instances: No valid host was found. There are not enough > hosts available. > Traceback (most recent call last): > > File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", > line 142, in inner > return func(*args, **kwargs) > > File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line > 84, in select_destinations > filter_properties) > > File > "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line > 90, in select_destinations > raise exception.NoValidHost(reason=reason) > > NoValidHost: No valid host was found. There are not enough hosts available. > --- begin --- > > Any suggestions on how to track down exactly what's going wrong? > > Thanks! > > -Ken > > ___ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
