[Yahoo-eng-team] [Bug 1531212] [NEW] "Cannot find device" during _create_veth_pair
Public bug reported:
While running Kilo in a scale test, my colleague observed a stack trace
on a compute host which boils down to:
Cannot find device "qvbcdc5cf8d-ad"
Checking the log, I found:
CMD "sudo nova-rootwrap
/opt/stack/service/nova-compute/etc/nova/rootwrap.conf ip link add
qvb59fd2beb-0c type veth peer name qvo59fd2beb-0c" returned: 0 in 0.091s
CMD "sudo nova-rootwrap
/opt/stack/service/nova-compute/etc/nova/rootwrap.conf ip link set
qvb59fd2beb-0c up" returned: 1 in 0.092s
There was also a stack trace pointing to line 1357 of
nova/network/linux_net.py, which is in _create_veth_pair. The code here
is very simple: it deletes the devices if they exist, runs `ip link add
...` then `ip link set ...`. The code has not changed since Kilo.
So it seems like either `ip link add ...` should have returned nonzero
and didn't; or another thread (maybe of another process) managed to
delete the devices in between the two commands. Neither seem
particularly likely, 'ip' is presumably pretty robust and the device
names are randomly generated so a collision seems curious.
Full stack trace:
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup Traceback
(most recent call last):
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/openstack/common/threadgroup.py", line 145, in wait
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
x.wait()
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/openstack/common/threadgroup.py", line 47, in wait
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup return
self.thread.wait()
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"eventlet/greenthread.py", line 175, in wait
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup return
self._exit_event.wait()
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"eventlet/event.py", line 121, in wait
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup return
hubs.get_hub().switch()
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"eventlet/hubs/hub.py", line 294, in switch
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup return
self.greenlet.switch()
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"eventlet/greenthread.py", line 214, in main
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup result
= function(*args, **kwargs)
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/openstack/common/service.py", line 502, in run_service
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
service.start()
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/service.py", line 164, in start
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
self.manager.init_host()
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/compute/manager.py", line 1298, in init_host
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
self._init_instance(context, instance)
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/compute/manager.py", line 1133, in _init_instance
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
self.driver.plug_vifs(instance, net_info)
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/virt/libvirt/driver.py", line 604, in plug_vifs
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
self.vif_driver.plug(instance, vif)
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/virt/libvirt/vif.py", line 609, in plug
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
func(instance, vif)
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/virt/libvirt/vif.py", line 447, in plug_ovs
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
self.plug_ovs_hybrid(instance, vif)
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/virt/libvirt/vif.py", line 443, in plug_ovs_hybrid
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
self._plug_bridge_with_port(instance, vif, port='ovs')
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/virt/libvirt/vif.py", line 424, in _plug_bridge_with_port
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
linux_net._create_veth_pair(v1_name, v2_name)
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup File
"nova/network/linux_net.py", line 1352, in _create_veth_pair
2015-12-13 18:34:58.876 1069 TRACE nova.openstack.common.threadgroup
utils.execute('ip', 'link', 'set', dev, 'up', run
[Yahoo-eng-team] [Bug 1521464] Re: create an instance with ephemeral disk or swap disk error
Sorry, newbie bug triager here. After double-checking with markus_z, he recommends setting incomplete and asking for DEBUG level logs. ** Changed in: nova Status: Invalid => Incomplete -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1521464 Title: create an instance with ephemeral disk or swap disk error Status in OpenStack Compute (nova): Incomplete Bug description: I boot an instance with ephemeral disk(or swap disk) in kilo, but return http request error: eg： nova boot --flavor 2 --image 5905bd7e-a87f-4856-8401-b8eb7211c84d --nic net-id=12ace164-d996-4261-9228-23ca0680f7a8 --ephemeral size=5,format=ext3 test_vm1 ERROR (BadRequest): Block Device Mapping is Invalid: Boot sequence for the instance and image/block device mapping combination is not valid. (HTTP 400) (Request-ID: req-b571662f-e554-49a7-979f-763f34b4b162) I think the instance with ephemeral disk should be able to boot from the image, instead of invalid boot sequence. the log in: http://paste.openstack.org/show/480452/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1521464/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1521464] Re: create an instance with ephemeral disk or swap disk error
Marking this invalid because it sounds like a feature to me. You may want to create a backlog spec, see http://specs.openstack.org/openstack /nova-specs/specs/backlog/index.html ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1521464 Title: create an instance with ephemeral disk or swap disk error Status in OpenStack Compute (nova): Invalid Bug description: I boot an instance with ephemeral disk(or swap disk) in kilo, but return http request error: eg： nova boot --flavor 2 --image 5905bd7e-a87f-4856-8401-b8eb7211c84d --nic net-id=12ace164-d996-4261-9228-23ca0680f7a8 --ephemeral size=5,format=ext3 test_vm1 ERROR (BadRequest): Block Device Mapping is Invalid: Boot sequence for the instance and image/block device mapping combination is not valid. (HTTP 400) (Request-ID: req-b571662f-e554-49a7-979f-763f34b4b162) I think the instance with ephemeral disk should be able to boot from the image, instead of invalid boot sequence. the log in: http://paste.openstack.org/show/480452/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1521464/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1525074] Re: test for submit
** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1525074 Title: test for submit Status in OpenStack Compute (nova): Invalid Bug description: just test for submit a bug report To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1525074/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1483613] [NEW] It may be possible to request (un)pinning of CPUs not in the NUMA cpuset
Public bug reported: There's already a check to ensure pinned CPUs are unpinned and vice versa, but none to ensure the CPUs are in the known set. This could lead to an invalid system state and emergent bugs. I noticed this via code inspection during Liberty. I don't know if it's possible to hit externally but it seems like a potential bug. John Garbutt encouraged me to open this for advertising. ** Affects: nova Importance: Undecided Assignee: Alexis Lee (alexisl) Status: In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1483613 Title: It may be possible to request (un)pinning of CPUs not in the NUMA cpuset Status in OpenStack Compute (nova): In Progress Bug description: There's already a check to ensure pinned CPUs are unpinned and vice versa, but none to ensure the CPUs are in the known set. This could lead to an invalid system state and emergent bugs. I noticed this via code inspection during Liberty. I don't know if it's possible to hit externally but it seems like a potential bug. John Garbutt encouraged me to open this for advertising. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1483613/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1427745] [NEW] Tests fail in non-en_US locale
Public bug reported: When I run `tox -epy27`, generally I get 13 test failures. One of these is a proxy/WSGI interaction I don't care about. The remaining 12 are due to tests which compare translated strings to untranslated message IDs. My locale is: $ locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en LC_CTYPE="en_GB.UTF-8" LC_NUMERIC="en_GB.UTF-8" LC_TIME="en_GB.UTF-8" LC_COLLATE="en_GB.UTF-8" LC_MONETARY="en_GB.UTF-8" LC_MESSAGES="en_GB.UTF-8" LC_PAPER="en_GB.UTF-8" LC_NAME="en_GB.UTF-8" LC_ADDRESS="en_GB.UTF-8" LC_TELEPHONE="en_GB.UTF-8" LC_MEASUREMENT="en_GB.UTF-8" LC_IDENTIFICATION="en_GB.UTF-8" LC_ALL= See this patch for the tests which fail for me: I352cd37d79401866e3116bcf0a62031bfe1d5d93 This patch removes the TranslationFixture which was supposed to prevent translations during tests but didn't: Idcc4409edae5ddfa0a1c2052a746d6412dda24ac Suggested fix: enforce an en_US locale or prevent translations from occurring during tests. ** Affects: nova Importance: Undecided Assignee: Alexis Lee (alexisl) Status: In Progress ** Tags: i18n testing -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1427745 Title: Tests fail in non-en_US locale Status in OpenStack Compute (Nova): In Progress Bug description: When I run `tox -epy27`, generally I get 13 test failures. One of these is a proxy/WSGI interaction I don't care about. The remaining 12 are due to tests which compare translated strings to untranslated message IDs. My locale is: $ locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en LC_CTYPE="en_GB.UTF-8" LC_NUMERIC="en_GB.UTF-8" LC_TIME="en_GB.UTF-8" LC_COLLATE="en_GB.UTF-8" LC_MONETARY="en_GB.UTF-8" LC_MESSAGES="en_GB.UTF-8" LC_PAPER="en_GB.UTF-8" LC_NAME="en_GB.UTF-8" LC_ADDRESS="en_GB.UTF-8" LC_TELEPHONE="en_GB.UTF-8" LC_MEASUREMENT="en_GB.UTF-8" LC_IDENTIFICATION="en_GB.UTF-8" LC_ALL= See this patch for the tests which fail for me: I352cd37d79401866e3116bcf0a62031bfe1d5d93 This patch removes the TranslationFixture which was supposed to prevent translations during tests but didn't: Idcc4409edae5ddfa0a1c2052a746d6412dda24ac Suggested fix: enforce an en_US locale or prevent translations from occurring during tests. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1427745/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
