Public bug reported:
this issue reproduce steps:
1. nova boot vm1 base fcsan volume, this instance bdm table connection_info as
following:
{"driver_volume_type": "fibre_channel", "serial":
"63013f85-34fd-4040-a234-6ee57e1a8eeb", "data": {"device_path": "/dev/dm-11",
"target_discovered": false, "devices": [{"device": "/dev/sdv", "host": "7",
"id": "0", "channel": "0", "lun": "8"}, {"device": "/dev/sdx", "host": "8",
"id": "0", "channel": "0", "lun": "8"}], "qos_specs": null, "volume_id":
"63013f85-34fd-4040-a234-6ee57e1a8eeb", "target_lun": 8, "access_mode": "rw",
"target_wwn": ["20014c09b4b04d0e", "20024c09b4b04d0e", "20034c09b4b04d0e",
"20044c09b4b04d0e", "20014c09b4b04d18", "20024c09b4b04d18", "20034c09b4b04d18",
"20044c09b4b04d18"], "multipath_id": "3500422790776c4d6"}}
2. run nova live-migration vm1 fininshed, vm1 was migrated another host, but
this instance bdm table connection_info as following:
{"driver_volume_type": "fibre_channel", "serial":
"63013f85-34fd-4040-a234-6ee57e1a8eeb", "data": { "target_discovered": false,
"qos_specs": null, "volume_id": "63013f85-34fd-4040-a234-6ee57e1a8eeb",
"target_lun": 9, "access_mode": "rw", "target_wwn": ["20014c09b4b04d0e",
"20024c09b4b04d0e", "20034c09b4b04d0e", "20044c09b4b04d0e", "20014c09b4b04d18",
"20024c09b4b04d18", "20034c09b4b04d18", "20044c09b4b04d18"], "multipath_id":
"3500422790776c4d6"}}
we can find the key "device_path" and "devices" was lost
3.if we live-migration this vm1 again,will fail . Call chain are as follows:
2014-12-16 05:19:12.039 9678 INFO nova.compute.manager [-] [instance:
5901a64a-32a6-448d-a535-5b03bb4df533] _post_live_migration() is started..
2014-12-16 05:19:12.068 9678 INFO urllib3.connectionpool [-] Starting new HTTP
connection (1): 10.43.177.244
2014-12-16 05:19:12.139 9678 INFO nova.compute.manager [-] [instance:
5901a64a-32a6-448d-a535-5b03bb4df533] During sync_power_state the instance has
a pending task. Skip.
2014-12-16 05:19:12.512 9678 ERROR nova.openstack.common.loopingcall [-] in
fixed duration looping call
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall Traceback
(most recent call last):
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py", line
78, in _inner
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
self.f(*self.args, **self.kw)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4987, in
wait_for_live_migration
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
migrate_data)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
payload)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
six.reraise(self.type_, self.value, self.tb)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall return
f(self, context, *args, **kw)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in
decorated_function
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall e,
sys.exc_info())
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
six.reraise(self.type_, self.value, self.tb)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 356, in
decorated_function
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall return
function(self, context, *args, **kwargs)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4824, in
_post_live_migration
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
migrate_data)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5188, in
post_live_migration
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
disk_dev)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1213, in
volume_driver_method
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall return
method(connection_info, *args, **kwargs)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line
249, in inner
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall return
f(*args, **kwargs)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 1034, in
disconnect_volume
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
devicespath = connection_info['data']['device_path']
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall KeyError:
'device_path'
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
2014-12-16 05:19:31.629 9678 AUDIT nova.compute.resource_tracker
[req-30cd2379-2ece-4577-a118-7b9daf4ef350 None None] Auditing locally available
compute resources
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1406161
Title:
boot from fcsan volume live migration caused bdm table lost
connection_info value
Status in OpenStack Compute (Nova):
New
Bug description:
this issue reproduce steps:
1. nova boot vm1 base fcsan volume, this instance bdm table connection_info
as following:
{"driver_volume_type": "fibre_channel", "serial":
"63013f85-34fd-4040-a234-6ee57e1a8eeb", "data": {"device_path": "/dev/dm-11",
"target_discovered": false, "devices": [{"device": "/dev/sdv", "host": "7",
"id": "0", "channel": "0", "lun": "8"}, {"device": "/dev/sdx", "host": "8",
"id": "0", "channel": "0", "lun": "8"}], "qos_specs": null, "volume_id":
"63013f85-34fd-4040-a234-6ee57e1a8eeb", "target_lun": 8, "access_mode": "rw",
"target_wwn": ["20014c09b4b04d0e", "20024c09b4b04d0e", "20034c09b4b04d0e",
"20044c09b4b04d0e", "20014c09b4b04d18", "20024c09b4b04d18", "20034c09b4b04d18",
"20044c09b4b04d18"], "multipath_id": "3500422790776c4d6"}}
2. run nova live-migration vm1 fininshed, vm1 was migrated another host, but
this instance bdm table connection_info as following:
{"driver_volume_type": "fibre_channel", "serial":
"63013f85-34fd-4040-a234-6ee57e1a8eeb", "data": { "target_discovered": false,
"qos_specs": null, "volume_id": "63013f85-34fd-4040-a234-6ee57e1a8eeb",
"target_lun": 9, "access_mode": "rw", "target_wwn": ["20014c09b4b04d0e",
"20024c09b4b04d0e", "20034c09b4b04d0e", "20044c09b4b04d0e", "20014c09b4b04d18",
"20024c09b4b04d18", "20034c09b4b04d18", "20044c09b4b04d18"], "multipath_id":
"3500422790776c4d6"}}
we can find the key "device_path" and "devices" was lost
3.if we live-migration this vm1 again,will fail . Call chain are as follows:
2014-12-16 05:19:12.039 9678 INFO nova.compute.manager [-] [instance:
5901a64a-32a6-448d-a535-5b03bb4df533] _post_live_migration() is started..
2014-12-16 05:19:12.068 9678 INFO urllib3.connectionpool [-] Starting new
HTTP connection (1): 10.43.177.244
2014-12-16 05:19:12.139 9678 INFO nova.compute.manager [-] [instance:
5901a64a-32a6-448d-a535-5b03bb4df533] During sync_power_state the instance has
a pending task. Skip.
2014-12-16 05:19:12.512 9678 ERROR nova.openstack.common.loopingcall [-] in
fixed duration looping call
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
Traceback (most recent call last):
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py", line
78, in _inner
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
self.f(*self.args, **self.kw)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4987, in
wait_for_live_migration
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
migrate_data)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
payload)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
six.reraise(self.type_, self.value, self.tb)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
return f(self, context, *args, **kw)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in
decorated_function
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall e,
sys.exc_info())
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
six.reraise(self.type_, self.value, self.tb)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 356, in
decorated_function
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
return function(self, context, *args, **kwargs)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4824, in
_post_live_migration
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
migrate_data)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5188, in
post_live_migration
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
disk_dev)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1213, in
volume_driver_method
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
return method(connection_info, *args, **kwargs)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line
249, in inner
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
return f(*args, **kwargs)
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 1034, in
disconnect_volume
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
devicespath = connection_info['data']['device_path']
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
KeyError: 'device_path'
2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall
2014-12-16 05:19:31.629 9678 AUDIT nova.compute.resource_tracker
[req-30cd2379-2ece-4577-a118-7b9daf4ef350 None None] Auditing locally available
compute resources
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1406161/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
