** Changed in: nova/grizzly
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1207422
Title:
Spawning multiple instances can cause race conditions with nbd
Status in OpenStack Compute (Nova):
Fix Released
Status in OpenStack Compute (nova) folsom series:
New
Status in OpenStack Compute (nova) grizzly series:
Fix Released
Bug description:
Spawning a number of instances using a single request (or number of
requests close enough to each other) can result in the same nbd device
chosen for more than one instance. For example, see the log of a
collision below. The python threads switch after the device is
selected and qemu-nbd is spawned, but before the qemu-nbd gets a
chance to connect and create a /sys/block/nbd*/pid file.
Solution is most likely to put a lock around the _inner_get_dev() part
of NbdMount.
2013-08-01 12:18:02.893 23130 DEBUG nova.virt.disk.mount.nbd
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Get
nbd device /dev/nbd14 for
/var/lib/nova/instances/0fde8031-0152-49ca-bd59-fd7a9353afc0/disk
_inner_get_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:87
2013-08-01 12:18:02.893 23130 DEBUG nova.openstack.common.processutils
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811]
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd
-c /dev/nbd14 /var/lib/nova/instances/0fde8031-0152-49ca-bd59-fd7a9353afc0/disk
execute
/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
2013-08-01 12:18:02.924 23130 DEBUG nova.virt.disk.mount.nbd
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Get
nbd device /dev/nbd14 for
/var/lib/nova/instances/10474a10-4bfa-41f2-8bd6-9add1e6aff1f/disk
_inner_get_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:87
2013-08-01 12:18:02.924 23130 DEBUG nova.openstack.common.processutils
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811]
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd
-c /dev/nbd14 /var/lib/nova/instances/10474a10-4bfa-41f2-8bd6-9add1e6aff1f/disk
execute
/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
2013-08-01 12:18:02.968 23130 DEBUG nova.virt.disk.mount.api
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Map
dev /dev/nbd14 map_dev
/usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:135
2013-08-01 12:18:02.968 23130 DEBUG nova.virt.disk.mount.api
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Mount
/dev/nbd14p1 on /tmp/openstack-vfs-localfsnW0rfa mnt_dev
/usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:188
2013-08-01 12:18:02.968 23130 DEBUG nova.openstack.common.processutils
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811]
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf mount
/dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa execute
/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
2013-08-01 12:18:03.000 23130 DEBUG nova.virt.disk.mount.api
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Map
dev /dev/nbd14 map_dev
/usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:135
2013-08-01 12:18:03.000 23130 DEBUG nova.openstack.common.processutils
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811]
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf kpartx -a
/dev/nbd14 execute
/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1
/tmp/openstack-vfs-localfsnW0rfa
Stderr: 'mount: special device /dev/nbd14p1 does not exist\n' mnt_dev
/usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:193
2013-08-01 12:18:03.044 23130 DEBUG nova.virt.disk.mount.api
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Unmap
dev /dev/nbd14 unmap_dev
/usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:179
2013-08-01 12:18:03.045 23130 DEBUG nova.virt.disk.mount.nbd
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811]
Release nbd device /dev/nbd14 unget_dev
/usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:126
2013-08-01 12:18:03.045 23130 DEBUG nova.openstack.common.processutils
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811]
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd
-d /dev/nbd14 execute
/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1
/tmp/openstack-vfs-localfsnW0rfa
Stderr: 'mount: special device /dev/nbd14p1 does not exist\n') setup
/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py:81
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1
/tmp/openstack-vfs-localfsnW0rfa
Stderr: 'mount: special device /dev/nbd14p1 does not exist\n')
2013-08-01 12:18:04.117 23130 DEBUG nova.virt.disk.mount.nbd
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811]
Release nbd device /dev/nbd14 unget_dev
/usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:126
2013-08-01 12:18:04.118 23130 DEBUG nova.openstack.common.processutils
[req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811]
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd
-d /dev/nbd14 execute
/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1207422/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
