+1 on Mikhail's suggestion. You will want to enable debug logging for the [nova]-scheduler (restart it of course after modifying the config) and then look for errors related to Filter. You can also correlate the req-XXX... ID with logs for nova-compute on your compute node(s) to see if that helps.
On Thu, Jul 28, 2016 at 4:50 PM, Mikhail Medvedev <mihail...@gmail.com> wrote: > On Thu, Jul 28, 2016 at 2:34 PM, Ken D'Ambrosio <k...@jots.org> wrote: > > Trying to orchestrate a bunch of VMs on our ~500 core cloud, and we're > > getting the "not enough hosts" problem. Only issue is that the dashboard > > seems to imply we've got the elbow room needed to instantiate. > Furthermore, > > it would be *really handy* if I knew what the bottleneck it thought was: > > disk, RAM, CPU, whatever. Here's a sample of the error: > > > > ----------------------- begin ----------------------- > > 2016-07-28 18:40:47.961 1080 WARNING nova.scheduler.utils > > [req-977badf4-a4a2-4766-87fd-c4e3ceb2e293 > a4c895a6fe294ae3a81e3c6bb9209f52 > > 33c2cf3cb61c425abcb0e0d35fdf21b7 - - -] [instance: > > ee175a40-11d9-4a43-b6e1-c4d7935acff1] Setting instance to ERROR state. > > 2016-07-28 18:40:51.892 1023 WARNING nova.scheduler.utils > > [req-e2a65200-1662-4cd7-aa9e-db5e225adfa1 > a4c895a6fe294ae3a81e3c6bb9209f52 > > 33c2cf3cb61c425abcb0e0d35fdf21b7 - - -] Failed to > > compute_task_build_instances: No valid host was found. There are not > enough > > hosts available. > > Traceback (most recent call last): > > > > File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", > line > > 142, in inner > > return func(*args, **kwargs) > > > > File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line > > 84, in select_destinations > > filter_properties) > > > > File > > "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", > line > > 90, in select_destinations > > raise exception.NoValidHost(reason=reason) > > > > NoValidHost: No valid host was found. There are not enough hosts > available. > > ----------------------- begin ----------------------- > > > > Any suggestions on how to track down exactly what's going wrong? > > Try taking a look at nova-scheduler log on your controller (usually in > /var/log/nova/nova-scheduler.log). I do not remember any time I have > seen the "not enough hosts" when the reason was not obvious from the > scheduler log. > > > > > Thanks! > > > > -Ken > > > > _______________________________________________ > > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > --- > Mikhail Medvedev > IBM > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack