** Changed in: nova Status: Fix Committed => Fix Released ** Changed in: nova Milestone: None => havana-3
-- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1207422 Title: Spawning multiple instances can cause race conditions with nbd Status in OpenStack Compute (Nova): Fix Released Status in OpenStack Compute (nova) folsom series: New Status in OpenStack Compute (nova) grizzly series: In Progress Bug description: Spawning a number of instances using a single request (or number of requests close enough to each other) can result in the same nbd device chosen for more than one instance. For example, see the log of a collision below. The python threads switch after the device is selected and qemu-nbd is spawned, but before the qemu-nbd gets a chance to connect and create a /sys/block/nbd*/pid file. Solution is most likely to put a lock around the _inner_get_dev() part of NbdMount. 2013-08-01 12:18:02.893 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Get nbd device /dev/nbd14 for /var/lib/nova/instances/0fde8031-0152-49ca-bd59-fd7a9353afc0/disk _inner_get_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:87 2013-08-01 12:18:02.893 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -c /dev/nbd14 /var/lib/nova/instances/0fde8031-0152-49ca-bd59-fd7a9353afc0/disk execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142 2013-08-01 12:18:02.924 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Get nbd device /dev/nbd14 for /var/lib/nova/instances/10474a10-4bfa-41f2-8bd6-9add1e6aff1f/disk _inner_get_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:87 2013-08-01 12:18:02.924 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -c /dev/nbd14 /var/lib/nova/instances/10474a10-4bfa-41f2-8bd6-9add1e6aff1f/disk execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142 2013-08-01 12:18:02.968 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Map dev /dev/nbd14 map_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:135 2013-08-01 12:18:02.968 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Mount /dev/nbd14p1 on /tmp/openstack-vfs-localfsnW0rfa mnt_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:188 2013-08-01 12:18:02.968 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142 2013-08-01 12:18:03.000 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Map dev /dev/nbd14 map_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:135 2013-08-01 12:18:03.000 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf kpartx -a /dev/nbd14 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142 Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa Stderr: 'mount: special device /dev/nbd14p1 does not exist\n' mnt_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:193 2013-08-01 12:18:03.044 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Unmap dev /dev/nbd14 unmap_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:179 2013-08-01 12:18:03.045 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Release nbd device /dev/nbd14 unget_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:126 2013-08-01 12:18:03.045 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -d /dev/nbd14 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142 Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa Stderr: 'mount: special device /dev/nbd14p1 does not exist\n') setup /usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py:81 Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa Stderr: 'mount: special device /dev/nbd14p1 does not exist\n') 2013-08-01 12:18:04.117 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Release nbd device /dev/nbd14 unget_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:126 2013-08-01 12:18:04.118 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -d /dev/nbd14 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1207422/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp