Hello everyone and thanks in advance. I have Openstack Pike
(KVM,FC-SAN/Cinder) installed in our lab for testing before upgrade and am
seeing a possible issue with disabling a host and live migrating the instances
off via the horizon interface. I can migrate the instances individually via the
Openstack client without issue. It looks like I might be missing something
relating to concurrent jobs in my nova config? Interestingly enough when a
migrate host is attempted via horizon they all fail. Migrating a single
instance through the horizon interface does function. Below is what I am
seeing in my scheduler log on the controller when trying to live migrate all
instances from a disabled host. I believe the last line to be the obvious issue
but I can not find a nova variable that seems to relate to this. Can anyone
point me in the right direction?
-----------------------------------
2017-09-19 19:02:30.529 19741 INFO nova.scheduler.host_manager
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Host filter ignoring hosts:
kvm02
2017-09-19 19:02:30.530 19741 DEBUG nova.filters
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Starting with 2 host(s)
get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:70
2017-09-19 19:02:30.530 19741 DEBUG nova.scheduler.filters.retry_filter
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Re-scheduling is disabled
host_passes
/usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:34
2017-09-19 19:02:30.530 19741 DEBUG nova.scheduler.filters.retry_filter
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Re-scheduling is disabled
host_passes
/usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:34
2017-09-19 19:02:30.531 19741 DEBUG nova.filters
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Filter RetryFilter returned
2 host(s) get_filtered_objects
/usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.531 19741 DEBUG nova.filters
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Filter
AvailabilityZoneFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.532 19741 DEBUG nova.filters
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Filter ComputeFilter
returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.532 19741 DEBUG nova.filters
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Filter
ComputeCapabilitiesFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.532 19741 DEBUG nova.filters
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Filter
ImagePropertiesFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.533 19741 DEBUG nova.filters
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Filter
ServerGroupAntiAffinityFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.533 19741 DEBUG nova.filters
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Filter
ServerGroupAffinityFilter returned 2 host(s) get_filtered_objects
/usr/lib/python2.7/dist-packages/nova/filters.py:104
2017-09-19 19:02:30.533 19741 DEBUG nova.scheduler.filter_scheduler
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Filtered [(kvm01,
kvm01.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk: 199680MB
io_ops: 0 instances: 0, (kvm03, kvm03.c1.us-east-dtw.os.libertycenterone.com)
ram: 257403MB disk: 199680MB io_ops: 0 instances: 0] _get_sorted_hosts
/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:298
2017-09-19 19:02:30.534 19741 DEBUG nova.scheduler.filter_scheduler
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Weighed [(kvm01,
kvm01.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk: 199680MB
io_ops: 0 instances: 0, (kvm03, kvm03.c1.us-east-dtw.os.libertycenterone.com)
ram: 257403MB disk: 199680MB io_ops: 0 instances: 0] _get_sorted_hosts
/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:308
2017-09-19 19:02:30.534 19741 DEBUG nova.scheduler.filter_scheduler
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Attempting to claim
resources in the placement API for instance
a16f866b-9d82-44b7-bc66-414f919ba067 _claim_resources
/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:270
2017-09-19 19:02:30.547 19741 DEBUG nova.scheduler.client.report
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Doubling-up allocation
request for move operation. _move_operation_alloc_request
/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:162
2017-09-19 19:02:30.547 19741 DEBUG nova.scheduler.client.report
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] New allocation request
containing both source and destination hosts in move operation: {'allocations':
[{'resource_provider': {'uuid': u'fb1b6ab4-df2a-4b0c-9578-6f7dee019e36'},
'resources': {u'VCPU': 4, u'MEMORY_MB': 8192, u'DISK_GB': 80}},
{'resource_provider': {'uuid': u'04fe593a-5af9-4878-a635-e9981a7d4dcd'},
'resources': {u'VCPU': 2, u'MEMORY_MB': 4096, u'DISK_GB': 40}}]}
_move_operation_alloc_request
/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py:202
2017-09-19 19:02:30.583 19741 DEBUG nova.scheduler.filter_scheduler
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Selected host: (kvm01,
kvm01.c1.us-east-dtw.os.libertycenterone.com) ram: 257403MB disk: 199680MB
io_ops: 0 instances: 0 _schedule
/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:227
2017-09-19 19:02:30.584 19741 DEBUG oslo_concurrency.lockutils
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Lock "(u'kvm01',
u'kvm01.c1.us-east-dtw.os.libertycenterone.com')" acquired by
"nova.scheduler.host_manager._locked" :: waited 0.000s inner
/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273
2017-09-19 19:02:30.585 19741 DEBUG nova.virt.hardware
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Require both a host and
instance NUMA topology to fit instance on host. numa_fit_instance_to_host
/usr/lib/python2.7/dist-packages/nova/virt/hardware.py:1467
2017-09-19 19:02:30.588 19741 DEBUG oslo_concurrency.lockutils
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] Lock "(u'kvm01',
u'kvm01.c1.us-east-dtw.os.libertycenterone.com')" released by
"nova.scheduler.host_manager._locked" :: held 0.004s inner
/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2017-09-19 19:02:30.588 19741 DEBUG nova.scheduler.filter_scheduler
[req-4268ea83-0657-40cc-961b-f0ae9fb3019e 385c60230b3f49da930dda4d089eda6b
723aa12337a44f818b6d1e1a59f16e49 - default default] There are 1 hosts available
but 10 instances requested to build. select_destinations
/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:101
root@controller01:/var/log/nova#
---------------------------------------------
Migrating from the openstack console.
root@controller01:~# openstack server list --long |grep Linux-Test
| 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.24 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.19 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.17 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.23 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.11 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.9 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.15 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.18 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.7 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.14 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
root@controller01:~# openstack server migrate --live kvm03 Linux-Test-1
root@controller01:~# openstack server migrate --live kvm03 Linux-Test-2
root@controller01:~# openstack server migrate --live kvm03 Linux-Test-3
root@controller01:~# openstack server migrate --live kvm03 Linux-Test-4
root@controller01:~# openstack server migrate --live kvm03 Linux-Test-5
root@controller01:~# openstack server migrate --live kvm03 Linux-Test-6
root@controller01:~# openstack server migrate --live kvm03 Linux-Test-7
root@controller01:~# openstack server migrate --live kvm03 Linux-Test-8
root@controller01:~# openstack server migrate --live kvm03 Linux-Test-9
root@controller01:~# openstack server migrate --live kvm03 Linux-Test-10
root@controller01:~# openstack server list --long |grep Linux-Test
| 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | MIGRATING | migrating
| Running | Admin-RFC1918=10.0.0.24 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | MIGRATING | migrating
| Running | Admin-RFC1918=10.0.0.19 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | MIGRATING | migrating
| Running | Admin-RFC1918=10.0.0.17 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | MIGRATING | migrating
| Running | Admin-RFC1918=10.0.0.23 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | MIGRATING | migrating
| Running | Admin-RFC1918=10.0.0.11 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | MIGRATING | migrating
| Running | Admin-RFC1918=10.0.0.9 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | MIGRATING | migrating
| Running | Admin-RFC1918=10.0.0.15 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm02 | |
| 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None
| Running | Admin-RFC1918=10.0.0.18 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None
| Running | Admin-RFC1918=10.0.0.7 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None
| Running | Admin-RFC1918=10.0.0.14 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
root@controller01:~#
root@controller01:~# openstack server list --long |grep Linux-Test
| 1d2866c3-ffc1-43a1-82d7-8bb92f396836 | Linux-Test-10 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.24 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| f5e17cb0-9679-4322-a480-7a9bd21e4d5b | Linux-Test-9 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.19 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| 38a84ba2-a410-4903-983b-6d1631ad1f0b | Linux-Test-8 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.17 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| 2598ed75-9c35-4871-864d-924215eeb5d7 | Linux-Test-7 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.23 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| 04218d15-6bbd-4687-900d-00ad8260c867 | Linux-Test-6 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.11 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| 5e28d8ef-a577-4982-8799-adb87ec3359a | Linux-Test-5 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.9 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| daee8c1b-5c69-4600-86d6-b10adbbe6e21 | Linux-Test-4 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.15 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| 33c348a7-60a1-4887-8761-e5ae17ea9357 | Linux-Test-3 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.18 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| f7d50f42-8f55-47a5-8701-fe8c47b62bcc | Linux-Test-2 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.7 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
| f9242792-80c3-4dcf-bdb9-f820156925f7 | Linux-Test-1 | ACTIVE | None |
Running | Admin-RFC1918=10.0.0.14 | | | m1.small |
14d14e99-4bc9-4ead-9a56-d05f574ccaa5 | nova | kvm03 | |
root@controller01:~#
Thanks!
Steven Searles
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
