Hello everyone and thanks in advance. I have Openstack Pike (KVM,FC-SAN/Cinder) installed in our lab for testing before upgrade and am seeing a possible issue with disabling a host and live migrating the instances off via the horizon interface. I can migrate the instances individually via the Openstack client without issue. It looks like I might be missing something relating to concurrent jobs in my nova config? Interestingly enough when a migrate host is attempted via horizon they all fail. Migrating a single instance through the horizon interface does function. Below is what I am seeing in my scheduler log on the controller when trying to live migrate all instances from a disabled host. I believe the last line to be the obvious issue but I can not find a nova variable that seems to relate to this. Can anyone point me in the right direction?
----------------------------------- 2017-09-19 19:02:30.529 19741 INFO nova.scheduler.host_manager [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Host filter ignoring hosts: kvm02 2017-09-19 19:02:30.530 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Starting with 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:70 2017-09-19 19:02:30.530 19741 DEBUG nova.scheduler.filters.retry_filter [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Re-scheduling is disabled host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:34 2017-09-19 19:02:30.530 19741 DEBUG nova.scheduler.filters.retry_filter [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Re-scheduling is disabled host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:34 2017-09-19 19:02:30.531 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter RetryFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104 2017-09-19 19:02:30.531 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter AvailabilityZoneFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104 2017-09-19 19:02:30.532 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ComputeFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104 2017-09-19 19:02:30.532 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ComputeCapabilitiesFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104 2017-09-19 19:02:30.532 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ImagePropertiesFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104 2017-09-19 19:02:30.533 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ServerGroupAntiAffinityFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104 2017-09-19 19:02:30.533 19741 DEBUG nova.filters [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ServerGroupAffinityFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104 2017-09-19 19:02:30.533 19741 DEBUG nova.scheduler.filter_scheduler [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Filtered [(kvm01, kvm01.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk: 199680MB io_ops: 0 instances: 0, (kvm03, kvm03.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk: 199680MB io_ops: 0 instances: 0] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:298 2017-09-19 19:02:30.534 19741 DEBUG nova.scheduler.filter_scheduler [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Weighed [(kvm01, kvm01.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk: 199680MB io_ops: 0 instances: 0, (kvm03, kvm03.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk: 199680MB io_ops: 0 instances: 0] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:308 2017-09-19 19:02:30.534 19741 DEBUG nova.scheduler.filter_scheduler [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Attempting to claim resources in the placement API for instance a16f866b-9d82-44b7-bc66-414f919ba067 _claim_resources /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:270 2017-09-19 19:02:30.547 19741 DEBUG nova.scheduler.client.report [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Doubling-up allocation request for move operation. _move_operation_alloc_request /usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:162 2017-09-19 19:02:30.547 19741 DEBUG nova.scheduler.client.report [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'fb1b6ab4-df2a-4b0c-9578-6f7dee019e36'}, 'resources': {u'VCPU': 4, u'MEMORY_MB': 8192, u'DISK_GB': 80}}, {'resource_provider': {'uuid': u'04fe593a-5af9-4878-a635-e9981a7d4dcd'}, 'resources': {u'VCPU': 2, u'MEMORY_MB': 4096, u'DISK_GB': 40}}]} _move_operation_alloc_request /usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:202 2017-09-19 19:02:30.583 19741 DEBUG nova.scheduler.filter_scheduler [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Selected host: (kvm01, kvm01.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk: 199680MB io_ops: 0 instances: 0 _schedule /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:227 2017-09-19 19:02:30.584 19741 DEBUG oslo_concurrency.lockutils [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Lock "(u'kvm01', u'kvm01.c1.us-east-dtw.os.libertycenterone.com')" acquired by "nova.scheduler.host_manager._locked" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273 2017-09-19 19:02:30.585 19741 DEBUG nova.virt.hardware [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Require both a host and instance NUMA topology to fit instance on host. numa_fit_instance_to_host /usr/lib/python2.7/dist-packages/nova/virt/hardware.py:1467 2017-09-19 19:02:30.588 19741 DEBUG oslo_concurrency.lockutils [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Lock "(u'kvm01', u'kvm01.c1.us-east-dtw.os.libertycenterone.com')" released by "nova.scheduler.host_manager._locked" :: held 0.004s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285 2017-09-19 19:02:30.588 19741 DEBUG nova.scheduler.filter_scheduler [req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] There are 1 hosts available but 10 instances requested to build. select_destinations /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:101 root@controller01:/var/log/nova# --------------------------------------------- Migrating from the openstack console. root@controller01:~# openstack server list --long |grep Linux-Test | 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.24 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.19 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.17 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.23 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.11 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.9 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.15 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.18 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.7 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.14 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | root@controller01:~# openstack server migrate --live kvm03 Linux-Test-1 root@controller01:~# openstack server migrate --live kvm03 Linux-Test-2 root@controller01:~# openstack server migrate --live kvm03 Linux-Test-3 root@controller01:~# openstack server migrate --live kvm03 Linux-Test-4 root@controller01:~# openstack server migrate --live kvm03 Linux-Test-5 root@controller01:~# openstack server migrate --live kvm03 Linux-Test-6 root@controller01:~# openstack server migrate --live kvm03 Linux-Test-7 root@controller01:~# openstack server migrate --live kvm03 Linux-Test-8 root@controller01:~# openstack server migrate --live kvm03 Linux-Test-9 root@controller01:~# openstack server migrate --live kvm03 Linux-Test-10 root@controller01:~# openstack server list --long |grep Linux-Test | 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.24 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.19 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.17 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.23 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.11 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.9 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | MIGRATING | migrating | Running | Admin-RFC1918=10.0.0.15 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | | | 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.18 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.7 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.14 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | root@controller01:~# root@controller01:~# openstack server list --long |grep Linux-Test | 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.24 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.19 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.17 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.23 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.11 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.9 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.15 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.18 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.7 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | | f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None | Running | Admin-RFC1918=10.0.0.14 | | | m1.small | 14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | | root@controller01:~# Thanks! Steven Searles
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack