[Yahoo-eng-team] [Bug 1868125] Re: [ovn] Metadata agent spawns haproxy quickly twice on a single new port binding
Reviewed: https://review.opendev.org/713956
Committed:
https://git.openstack.org/cgit/openstack/neutron/commit/?id=06fde66b0b05dc987a4280275813f0cf4bf54c88
Submitter: Zuul
Branch:master
commit 06fde66b0b05dc987a4280275813f0cf4bf54c88
Author: Jakub Libosvar
Date: Thu Mar 19 17:57:25 2020 +
[ovn] Stricter matching on metadata port binding event
Previously the Port Binding event spawned haproxy process even if the
change wasn't done in chassis column, it attempted to spawn haproxy on
all port bindings changes.
This patch spawns haproxy only if new chassis is the one managed by the
agent and old chassis was empty. Similarly it destroys haproxy if new
chassis is empty and old chassis was the one managed by the agent.
Closes-bug: #1868125
Co-Authored-By: Daniel Alvarez
Signed-off-by: Jakub Libosvar
Change-Id: I5b87726eafa71d717ae22f48d1c9c6343b680c7f
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1868125
Title:
[ovn] Metadata agent spawns haproxy quickly twice on a single new port
binding
Status in neutron:
Fix Released
Bug description:
When the veth pair for metadata namespace is created, OVSDB emits
multiple Port Binding events. The agent re-acts on two and attempts to
spawn haproxy process twice in a small timeframe.
This is an example of port 26d98ec8-ce82-45d7-8043-f00f42b1f15c and
datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea
2020-03-16 19:50:13.052 29393 DEBUG ovsdbapp.backend.ovs_idl.event [-]
Matched UPDATE: PortBindingChassisEvent(events=('update',),
table='Port_Binding', conditions=None, old_conditions=None) to
row=Port_Binding(tunnel_key=3, parent_port=[],
logical_port=26d98ec8-ce82-45d7-8043-f00f42b1f15c, mac=['fa:16:3e:07:36:c9
10.100.0.7'], chassis=[],
ha_chassis_group=[], options={'requested-chassis': 'compute-0.redhat.local'},
external_ids={'neutron:cidrs': '10.100.0.7/28', 'neutron:device_id':
'27584016-6cf4-4d2a-9300-098fb9dc6686', 'neutron:device_owner': 'compute:nova',
'neutron:network_name': 'neutron-c4c60426-9c40-4746-8193-2354be1fa436',
'neutron:port_name': '', 'neutron:project_id':
'27c427e573c34229a2651f14da1bf15f', 'neutron:revision_number': '2',
'neutron:security_group_ids': 'c289479c-ee0e-4e52-a475-ec51d04d67d8'},
encap=[], gateway_chassis=[], type=, nat_addresses=[], virtual_parent=[],
datapath=1d59fe65-eca7-4341-95a1-8fd2dad007ea, tag=[])
old=Port_Binding(chassis=[]) matches
/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/event.py:44
2020-03-16 19:50:13.053 29393 INFO networking_ovn.agent.metadata.agent [-]
Port 26d98ec8-ce82-45d7-8043-f00f42b1f15c in datapath
1d59fe65-eca7-4341-95a1-8fd2dad007ea bound to our chassis
2020-03-16 19:50:13.054 29393 DEBUG networking_ovn.agent.metadata.agent [-]
Provisioning datapath 1d59fe65-eca7-4341-95a1-8fd2dad007ea provision_datapath
/usr/lib/python3.6/site-packages/networking_ovn/agent/metadata/agent.py:318
2020-03-16 19:50:13.744 29393 DEBUG neutron.agent.linux.utils [-]
Running command: ['sudo', 'neutron-rootwrap',
'/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'ovnmeta-
1d59fe65-eca7-4341-95a1-8fd2dad007ea', 'haproxy', '-f',
'/var/lib/neutron/ovn-metadata-
proxy/1d59fe65-eca7-4341-95a1-8fd2dad007ea.conf'] create_process
/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:87
2020-03-16 19:50:14.501 29393 DEBUG ovsdbapp.backend.ovs_idl.event [-]
Matched UPDATE: PortBindingChassisEvent(events=('update',),
table='Port_Binding', conditions=None, old_conditions=None) to
row=Port_Binding(tunnel_key=3, parent_port=[],
logical_port=26d98ec8-ce82-45d7-8043-f00f42b1f15c, mac=['fa:16:3e:07:36:c9
10.100.0.7'], chassis=[],
ha_chassis_group=[], options={'requested-chassis': 'compute-0.redhat.local'},
external_ids={'neutron:cidrs': '10.100.0.7/28', 'neutron:device_id':
'27584016-6cf4-4d2a-9300-098fb9dc6686', 'neutron:device_owner': 'compute:nova',
'neutron:network_name': 'neutron-c4c60426-9c40-4746-8193-2354be1fa436',
'neutron:port_name': '', 'neutron:project_id':
'27c427e573c34229a2651f14da1bf15f', 'neutron:revision_number': '4',
'neutron:security_group_ids': 'c289479c-ee0e-4e52-a475-ec51d04d67d8'},
encap=[], gateway_chassis=[], type=, nat_addresses=[], virtual_parent=[],
datapath=1d59fe65-eca7-4341-95a1-8fd2dad007ea, tag=[])
old=Port_Binding(external_ids={'neutron:cidrs': '10.100.0.7/28',
'neutron:device_id': '27584016-6cf4-4d2a-9300-098fb9dc6686',
'neutron:device_owner': 'compute:nova', 'neutron:network_name':
'neutron-c4c60426-9c40-4746-8193-2354be1fa436', 'neutron:port_name': '',
'neutron:project_id': '27c427e573c34229a2651f14da1bf15f',
'neutron:revision_number': '2', 'neutron:security_group_ids':
'c289479c-ee0e-4e52-a475-ec51d04d67d8'}) matches
[Yahoo-eng-team] [Bug 1868327] [NEW] RuntimeError: dictionary keys changed during iteration
Public bug reported: Forwarded from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=954276 This appears to be a python 3.8 incompatibility. | 2020-03-19 14:31:48,840 - util.py[DEBUG]: Running module disk_setup () failed | Traceback (most recent call last): | File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 848, in _run_modules | ran, _r = cc.run(run_name, mod.handle, func_args, | File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run | return self._runners.run(name, functor, args, freq, clear_on_fail) | File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 185, in run | results = functor(*args) | File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 129, in handle | update_disk_setup_devices(disk_setup, cloud.device_name_to_device) | File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 166, in update_disk_setup_devices | for origname in disk_setup.keys(): | RuntimeError: dictionary keys changed during iteration I've attached a small patch that implements a minimal fix for this issue. ** Affects: cloud-init Importance: Undecided Status: New ** Patch added: "cloud-init-python-3.8.patch" https://bugs.launchpad.net/bugs/1868327/+attachment/5339495/+files/cloud-init-python-3.8.patch -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1868327 Title: RuntimeError: dictionary keys changed during iteration Status in cloud-init: New Bug description: Forwarded from https://bugs.debian.org/cgi- bin/bugreport.cgi?bug=954276 This appears to be a python 3.8 incompatibility. | 2020-03-19 14:31:48,840 - util.py[DEBUG]: Running module disk_setup () failed | Traceback (most recent call last): | File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 848, in _run_modules | ran, _r = cc.run(run_name, mod.handle, func_args, | File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run | return self._runners.run(name, functor, args, freq, clear_on_fail) | File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 185, in run | results = functor(*args) | File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 129, in handle | update_disk_setup_devices(disk_setup, cloud.device_name_to_device) | File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 166, in update_disk_setup_devices | for origname in disk_setup.keys(): | RuntimeError: dictionary keys changed during iteration I've attached a small patch that implements a minimal fix for this issue. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1868327/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1868100] Re: ncat process isn't always spawned on guest vm in scenario tests
Reviewed: https://review.opendev.org/713208 Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=fd4141f2015d25f1b009d7cf2ebdd2907cd8e81a Submitter: Zuul Branch:master commit fd4141f2015d25f1b009d7cf2ebdd2907cd8e81a Author: Slawek Kaplonski Date: Sat Mar 14 14:34:00 2020 +0100 Fix how nc is called in qos test We have already nc_listen method in base scenario tests class. It was since now used only in port_forwarding tests but we can reuse it in QoS tests also. There was also problem with spawning ncat process, that sometimes, without any clear reason for me, process wasn't spawned at all. That caused failure of test. So this patch adds new method ensure_nc_listen() which spawns ncat process on remote host and checkes if process is really spawned. That way we can avoid problems with not spawned ncat process. This patch also makes "server" attribute to be optional in nc_listen method. It is used only to log console output in case when ssh to the server wouldn't work as expected. And if server is not given, _log_console_output() method will list all servers which belongs to tenant and log console for each of them. Closes-Bug: #1868100 Change-Id: I54c9f041f2f971219c32005b3fa573c06f0110ef ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1868100 Title: ncat process isn't always spawned on guest vm in scenario tests Status in neutron: Fix Released Bug description: Some scenario tests are using ncat process spawned on guest vm to perform some checks. It's like that e.g. for port_forwarding and qos tests. Recently I found that for unknown for me reason, sometimes "nc" process isn't spawned properly in the guest vm. That causes test failure. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1868100/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1868246] Re: No network after subiquity LPAR installation on s390x with VLAN
Some additional information:
Early in the subiquity installation process (right after disk device
enablement) I can see two files in /etc/netplan/:
00-installer-config.yaml
50-cloud-init.yaml.dist-subiquity
I think both are not as they should be for this VLAN environment.
After replacing them with:
network:
version: 2
renderer: networkd
ethernets:
encc000:
dhcp4: no
dhcp6: no
vlans:
encc000.2653:
id: 2653
link: encc000
addresses: [ 10.245.236.15/24 ]
gateway4: 10.245.236.1
nameservers:
search: [ canonical.com ]
addresses:
- 10.245.236.1
I was able to bring up the network (in the subiquity shell) using netplan apply.
(I also disabled/enabled 0.0.c000 - but I think it was not needed).
Unfortunately there is still no network online after the post-install reboot:
$ ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group defaul
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: encc000: mtu 1500 qdisc mq state UP group d
efault qlen 1000
link/ether 16:9e:e9:36:c4:90 brd ff:ff:ff:ff:ff:ff
inet6 fe80::149e:e9ff:fe36:c490/64 scope link
valid_lft forever preferred_lft forever
3: encc000.2653@encc000: mtu 1500 qdisc noqueu
e state UP group default qlen 1000
link/ether 16:9e:e9:36:c4:90 brd ff:ff:ff:ff:ff:ff
inet6 fe80::149e:e9ff:fe36:c490/64 scope link
valid_lft forever preferred_lft forever
...since the following netplan yaml is in place - which is not correct:
$ cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
ethernets:
encc000: {}
version: 2
vlans:
encc000.2653:
id: 2653
link: encc000
nameservers:
addresses:
- 10.245.236.1
Replacing it again by the above (known to work) yaml allows to bring the
network up again (with the help of netplan).
I add cloud-init as affected package and let the maintainers decide if
this is a duplicate or not (see previous comment).
** Also affects: cloud-init
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team,
[Yahoo-eng-team] [Bug 1858019] Re: The flavor id is not limited when creating a flavor
Also agree. If we're going to do anything, it should be done on the client side. It should be possible to add a flag stating what field we wish to filter on (name or ID), if needed. Since there's nothing to do here from the server side, I'm going to close this as WONTFIX. ** Changed in: nova Status: Triaged => Won't Fix ** Changed in: nova Assignee: Choi-Sung-Hoon (knu-cse) => (unassigned) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1858019 Title: The flavor id is not limited when creating a flavor Status in OpenStack Compute (nova): Won't Fix Bug description: when creating a flavor by 'openstack flavor create --id --vcpus --ram --disk ', the parameter id is not limited. It can lead to ambiguities when id is set to an existed flavor name. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1858019/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1868203] Re: nova-compute error
This looks like your installation of QEMU is completely broken. As confirmed on IRC: /usr/bin/qemu-system-ppc64: relocation error: /usr/bin/qemu-system- ppc64: symbol fdt_check_full version LIBFDT_1.2 not defined in file libfdt.so.1 with link time reference Please consult Ubuntu's guidance to re-install virtualization packages properly. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1868203 Title: nova-compute error Status in OpenStack Compute (nova): Invalid Bug description: Hi All, I have a rocky setup. I am unable to launch instances on one of the compute nodes. I see the following error in the conductor logs and the libvirtd service on the compute. The host is ubuntu 18.04: -- File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations raise exception.NoValidHost(reason="") NoValidHost: No valid host was found. 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager Traceback (most recent call last): 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/conductor/manager.py", line 1237, in schedule_and_build_instances 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager instance_uuids, return_alternates=True) 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/conductor/manager.py", line 750, in _schedule_instances 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager return_alternates=return_alternates) 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 50, in select_destinations 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates) 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 35, in __run_method 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs) 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates) 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args) 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager retry=self.retry) 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager retry=retry) 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager call_monitor_timeout, retry=retry) 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager raise result 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager Traceback (most recent call last): 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager return func(*args, **kwargs) 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager File "/openstack/venvs/nova-18.1.9/lib/python2.7/site-packages/nova/scheduler/manager.py", line 154, in select_destinations 2020-03-20 05:31:09.006 35363 ERROR nova.conductor.manager
[Yahoo-eng-team] [Bug 1855776] Re: Aggregate ID validation
*** This bug is a duplicate of bug 1865040 *** https://bugs.launchpad.net/bugs/1865040 ** This bug has been marked a duplicate of bug 1865040 Able to show update and delete aggregate with invalid id -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1855776 Title: Aggregate ID validation Status in OpenStack Compute (nova): In Progress Bug description: Description === Nova API's aggregate ID lookup does not require an exact match. Alphanumeric strings can possibly be truncated and converted to integers incorrectly. Steps to reproduce == Determine the ID of an existing aggregate. Take the aggregate ID, and append junk data to it, ensuring that the digit following the actual ID is an alphabetic character.e.g. aggregate id = '13', the test string would be something like '13a2g152asdf'Send a PUT request to '/os_aggregates/,' modifying either the name or availability zone Check for whether or not the server returns an error (aggregate ID not found), or a 200 OK indicating the change was successful. Successful change indicates the issue still exists. Expected result === Nova should return error. Actual result = Nova returns 200. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1855776/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1792710] Re: glance backend is in crazy resize when an image is uploading
** Also affects: glance Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1792710 Title: glance backend is in crazy resize when an image is uploading Status in Cinder: In Progress Status in Glance: New Status in glance_store: New Bug description: When uploading a volume to glance as an image, the glance server don't know the image size, so the backend storage server(such as ceph) need to resize the image every time it received new chunk of data(by default 64K). So there will be huge times of resize operations that will impact the performance. - regarding cinder, it has not calculate the image size and pass the correct size to glance - regarding glance, it should allow the client to set image size. This is an known issue which can be found in driver files of all kinds of backend storage system: In file: glance_store/_drivers/rbd.py, function: add In file: glance_store/_drivers/cinder.py, function: add In file: glance_store/_drivers/sheepdog.py, function: add In all these files, there're comments like below: # If the image size provided is zero we need to do # a resize for the amount we are writing. This will # be slower so setting a higher chunk size may # speed things up a bit. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1792710/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1868234] Re: nova-live-migration evacuation fails if volumes created on subnode c-vol backend
** Changed in: nova Importance: Undecided => High ** Tags added: live-migration volumes ** Tags added: evacuate ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/ussuri Importance: High Assignee: Lee Yarwood (lyarwood) Status: In Progress ** Also affects: nova/train Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1868234 Title: nova-live-migration evacuation fails if volumes created on subnode c-vol backend Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: New Status in OpenStack Compute (nova) ussuri series: In Progress Bug description: Description === I8af2ad741ca08c3d88efb9aa817c4d1470491a23 has started to correctly fence the subnode during evacuation testing. However it missed that we deploy c-vol and g-api on these nodes. As a result during BFV evacuation testing we will fail if the volume has been created on the subnode c-vol. https://zuul.opendev.org/t/openstack/build/c78d3ab4e6a748b4a53c6ff6dc273106/log/logs/screen-n-cpu.txt#7060 Mar 19 19:43:26.844295 ubuntu-bionic-rax-ord-0015339373 nova- compute[9838]: ERROR nova.compute.manager [None req- 512a96c8-8b32-49c7-8d29-7ff300ed4482 demo admin] [instance: 702ff125-d947-4a28-853b-82dcd58b990e] Setting instance vm_state to ERROR: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) https://zuul.opendev.org/t/openstack/build/c78d3ab4e6a748b4a53c6ff6dc273106/log/logs/screen-c-api.txt#1936 Mar 19 19:43:26.262818 ubuntu-bionic-rax-ord-0015339373 devstack@c-api.service[27200]: ERROR cinder.api.middleware.fault [req-512a96c8-8b32-49c7-8d29-7ff300ed4482 req-826f7c01-3c02-4d9e-9046-8a15d7fa9b61 demo admin] Caught error: Timed out waiting for a reply to message ID 23fabce9b79441198fbe4fe71c0ac7ab: MessagingTimeout: Timed out waiting for a reply to message ID 23fabce9b79441198fbe4fe71c0ac7ab Mar 19 19:43:26.262818 ubuntu-bionic-rax-ord-0015339373 devstack@c-api.service[27200]: ERROR Ultimately we shouldn't run these services on the computes but for now we should limit the services we stop on the subnode to n-cpu and q-agt. Steps to reproduce == Run nova-live-migration, if volumes are created on the subnode evacuation testing will fail. Expected result === nova-live-migration passes. Actual result = nova-live-migration fails. Environment === 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ Master or stabe/train with I8af2ad741ca08c3d88efb9aa817c4d1470491a23 applied. 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? Libvirt + KVM 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? N/A 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1868234/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1868234] [NEW] nova-live-migration evacuation fails if volumes created on subnode c-vol backend
Public bug reported: Description === I8af2ad741ca08c3d88efb9aa817c4d1470491a23 has started to correctly fence the subnode during evacuation testing. However it missed that we deploy c-vol and g-api on these nodes. As a result during BFV evacuation testing we will fail if the volume has been created on the subnode c-vol. https://zuul.opendev.org/t/openstack/build/c78d3ab4e6a748b4a53c6ff6dc273106/log/logs/screen-n-cpu.txt#7060 Mar 19 19:43:26.844295 ubuntu-bionic-rax-ord-0015339373 nova- compute[9838]: ERROR nova.compute.manager [None req- 512a96c8-8b32-49c7-8d29-7ff300ed4482 demo admin] [instance: 702ff125-d947-4a28-853b-82dcd58b990e] Setting instance vm_state to ERROR: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) https://zuul.opendev.org/t/openstack/build/c78d3ab4e6a748b4a53c6ff6dc273106/log/logs/screen-c-api.txt#1936 Mar 19 19:43:26.262818 ubuntu-bionic-rax-ord-0015339373 devstack@c-api.service[27200]: ERROR cinder.api.middleware.fault [req-512a96c8-8b32-49c7-8d29-7ff300ed4482 req-826f7c01-3c02-4d9e-9046-8a15d7fa9b61 demo admin] Caught error: Timed out waiting for a reply to message ID 23fabce9b79441198fbe4fe71c0ac7ab: MessagingTimeout: Timed out waiting for a reply to message ID 23fabce9b79441198fbe4fe71c0ac7ab Mar 19 19:43:26.262818 ubuntu-bionic-rax-ord-0015339373 devstack@c-api.service[27200]: ERROR Ultimately we shouldn't run these services on the computes but for now we should limit the services we stop on the subnode to n-cpu and q-agt. Steps to reproduce == Run nova-live-migration, if volumes are created on the subnode evacuation testing will fail. Expected result === nova-live-migration passes. Actual result = nova-live-migration fails. Environment === 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ Master or stabe/train with I8af2ad741ca08c3d88efb9aa817c4d1470491a23 applied. 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? Libvirt + KVM 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? N/A 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A ** Affects: nova Importance: High Assignee: Lee Yarwood (lyarwood) Status: In Progress ** Affects: nova/stein Importance: Undecided Status: New ** Affects: nova/train Importance: Undecided Status: New ** Affects: nova/ussuri Importance: High Assignee: Lee Yarwood (lyarwood) Status: In Progress ** Tags: evacuate live-migration volumes -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1868234 Title: nova-live-migration evacuation fails if volumes created on subnode c-vol backend Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: New Status in OpenStack Compute (nova) ussuri series: In Progress Bug description: Description === I8af2ad741ca08c3d88efb9aa817c4d1470491a23 has started to correctly fence the subnode during evacuation testing. However it missed that we deploy c-vol and g-api on these nodes. As a result during BFV evacuation testing we will fail if the volume has been created on the subnode c-vol. https://zuul.opendev.org/t/openstack/build/c78d3ab4e6a748b4a53c6ff6dc273106/log/logs/screen-n-cpu.txt#7060 Mar 19 19:43:26.844295 ubuntu-bionic-rax-ord-0015339373 nova- compute[9838]: ERROR nova.compute.manager [None req- 512a96c8-8b32-49c7-8d29-7ff300ed4482 demo admin] [instance: 702ff125-d947-4a28-853b-82dcd58b990e] Setting instance vm_state to ERROR: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) https://zuul.opendev.org/t/openstack/build/c78d3ab4e6a748b4a53c6ff6dc273106/log/logs/screen-c-api.txt#1936 Mar 19 19:43:26.262818 ubuntu-bionic-rax-ord-0015339373 devstack@c-api.service[27200]: ERROR cinder.api.middleware.fault [req-512a96c8-8b32-49c7-8d29-7ff300ed4482 req-826f7c01-3c02-4d9e-9046-8a15d7fa9b61 demo admin] Caught error: Timed out waiting for a reply to message ID 23fabce9b79441198fbe4fe71c0ac7ab: MessagingTimeout: Timed out waiting for a reply to message ID 23fabce9b79441198fbe4fe71c0ac7ab Mar 19 19:43:26.262818 ubuntu-bionic-rax-ord-0015339373 devstack@c-api.service[27200]: ERROR Ultimately we shouldn't run these services on the computes but for now we should limit the services we stop on the subnode to n-cpu and q-agt. Steps to reproduce == Run nova-live-migration, if
[Yahoo-eng-team] [Bug 1868237] [NEW] [tempest] Error in "test_multicast_between_vms_on_same_network" test case
Public bug reported: Error during the execution of "test_multicast_between_vms_on_same_network" test case. Log: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_124/713208/6/check /neutron-tempest-plugin-scenario-openvswitch- iptables_hybrid/124431e/testr_results.html Log snippet: """ Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc return func(*args, **kwargs) File "/opt/stack/tempest/tempest/common/waiters.py", line 111, in wait_for_server_termination time.sleep(client.build_interval) File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler raise TimeoutException() fixtures._fixtures.timeout.TimeoutException """ ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1868237 Title: [tempest] Error in "test_multicast_between_vms_on_same_network" test case Status in neutron: New Bug description: Error during the execution of "test_multicast_between_vms_on_same_network" test case. Log: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_124/713208/6/check /neutron-tempest-plugin-scenario-openvswitch- iptables_hybrid/124431e/testr_results.html Log snippet: """ Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc return func(*args, **kwargs) File "/opt/stack/tempest/tempest/common/waiters.py", line 111, in wait_for_server_termination time.sleep(client.build_interval) File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler raise TimeoutException() fixtures._fixtures.timeout.TimeoutException """ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1868237/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1868232] [NEW] underscores should be stripped from hostnames generated for apt config
Public bug reported: In a ticket filed in the Ubuntu RT instance we were made aware of an issue where if a cloud is configured with an “_” in the region name, cloud-init will generate an apt configuration that also includes that “_” in the name. So for example if the region name is zone_01, apt will be configured to use zone_01.clouds.archive.ubuntu.com. On Friday March 13th we deployed some new archive servers on 18.04 using Apache 2.4.29-1ubuntu4.13. This version of apache has more strict protocol options than previous versions, per https://httpd.apache.org/docs/2.4/mod/core.html#httpprotocoloptions and the result is that a request to zone_01.clouds.archive.ubuntu.com returns a 400 Bad Request. Could cloud-init be updated to remove non-permitted characters including “_” per https://tools.ietf.org/html/rfc3986#section-3.2.2 ? ** Affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1868232 Title: underscores should be stripped from hostnames generated for apt config Status in cloud-init: New Bug description: In a ticket filed in the Ubuntu RT instance we were made aware of an issue where if a cloud is configured with an “_” in the region name, cloud-init will generate an apt configuration that also includes that “_” in the name. So for example if the region name is zone_01, apt will be configured to use zone_01.clouds.archive.ubuntu.com. On Friday March 13th we deployed some new archive servers on 18.04 using Apache 2.4.29-1ubuntu4.13. This version of apache has more strict protocol options than previous versions, per https://httpd.apache.org/docs/2.4/mod/core.html#httpprotocoloptions and the result is that a request to zone_01.clouds.archive.ubuntu.com returns a 400 Bad Request. Could cloud-init be updated to remove non-permitted characters including “_” per https://tools.ietf.org/html/rfc3986#section-3.2.2 ? To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1868232/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
