[Yahoo-eng-team] [Bug 1867075] [NEW] Arm64: Instance with Configure Drive attach volume failed failed
Public bug reported: Arm64. Image: cirros-0.5.1 hw_cdrom_bus='scsi', hw_disk_bus='scsi', hw_machine_type='virt', hw_rng_model='virtio', hw_scsi_model='virtio-scsi', os_command_line=''console=ttyAMA0'' Boot a vm. Create a volume: openstack volume create --size 1 test Attach: openstack server add volume cirros-test test Error: DEBUG nova.virt.libvirt.guest [None req-8dfbf677-50bb-42be-869f-52c9ac638d59 admin admin] attach device xml: b9abb789-1c55-4210-ab5c-78b0e3619405 ror: Requested operation is not valid: Domain already contains a disk with that address ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] Traceback (most recent call last): ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/opt/stack/nova/nova/virt/block_device.py", line 599, in _volume_attach ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] device_type=self['device_type'], encryption=encryption) ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1731, in attach_volume ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] conf = self._get_volume_config(connection_info, disk_info) ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] self.force_reraise() ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] six.reraise(self.type_, self.value, self.tb) ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] raise value ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1731, in attach_volume ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] conf = self._get_volume_config(connection_info, disk_info) ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 293, in attach_device ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] self._domain.attachDeviceFlags(device_xml, flags=flags) ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 190, in doit ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] result = proxy_call(self._autowrap, f, *args, **kwargs) ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 148, in proxy_call ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] rv = execute(f, *args, **kwargs) ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 129, in execute ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] six.reraise(c, e, tb) ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] raise value ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 83, in tworker ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] rv = meth(*args, **kwargs) ERROR nova.virt.block_device [instance: 22bdc0a6-1c0c-43fa-8c64-66735b6a6cb6] File
[Yahoo-eng-team] [Bug 1867077] [NEW] stein: when use "openstack server resize" command, there is an error: Unexpected API Error.
Public bug reported: kolla-ansible: 8.0.1 openstack_release: "stein" kolla_install_type: "source" kolla_base_distro: "centos" when I use "openstack server resize" command, there is an API error: # openstack server resize --flavor m1.large haproxyserver Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. (HTTP 500) (Request-ID: req-576f1e82-6d47-4c6a-87f1-1e552914f9d6) ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1867077 Title: stein: when use "openstack server resize" command, there is an error: Unexpected API Error. Status in OpenStack Compute (nova): New Bug description: kolla-ansible: 8.0.1 openstack_release: "stein" kolla_install_type: "source" kolla_base_distro: "centos" when I use "openstack server resize" command, there is an API error: # openstack server resize --flavor m1.large haproxyserver Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. (HTTP 500) (Request-ID: req-576f1e82-6d47-4c6a-87f1-1e552914f9d6) To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1867077/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1867043] Re: Daily builds failing due to missing pytest
** Changed in: cloud-init Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1867043 Title: Daily builds failing due to missing pytest Status in cloud-init: Fix Released Bug description: e.g. https://launchpadlibrarian.net/468666983/buildlog_ubuntu-xenial- amd64.cloud- init_20.1-2345-gd5dbbd1-0ubuntu1+1465~trunk~ubuntu16.04.1_BUILDING.txt.gz http_proxy= make PYVER=python3 check make[2]: Entering directory '/<>' python3 -m pytest -v tests/unittests cloudinit /usr/bin/python3: No module named pytest Makefile:52: recipe for target 'unittest3' failed make[2]: *** [unittest3] Error 1 make[2]: Leaving directory '/<>' debian/rules:11: recipe for target 'override_dh_auto_test' failed make[1]: *** [override_dh_auto_test] Error 2 make[1]: Leaving directory '/<>' debian/rules:7: recipe for target 'build' failed make: *** [build] Error 2 dpkg-buildpackage: error: debian/rules build gave error exit status 2 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1867043/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1867029] Re: Package ifupdown breaks network configuration by cloud-init
** Also affects: ifupdown (Ubuntu) Importance: Undecided Status: New ** Tags added: uec-images -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1867029 Title: Package ifupdown breaks network configuration by cloud-init Status in cloud-init: Confirmed Status in ifupdown package in Ubuntu: New Bug description: It appears the `ifupdown` package breaks networking configuration that cloud-init would normally handle (?) on both providers. The result, if you image an instance with the `ifupdown` package installed, is that the image is not bootable by Azure/AWS. I'm not familiar with how Azure/AWS/etc use cloud-init for network start up however. I'm filing here for now, in case `cloud-init` needs to be more defensive against `ifupdown`. To reproduce with Packer (I confirmed this method): * Add necessary Azure subscription env vars from `ifupdown-bug.json` * `packer build ifupdown-bug.json` * Try to boot an instance with resulting image * Instance will never boot -- more specifically, networking never comes up. See `boot.log` for an example boot To reproduce manually (only a guess, I did not confirm): * Boot 18.04 ubuntu instance * Install ifupdown * Image the instance * Try to boot instance using new image * Instance won't boot I hit this with Packer, creating new images for booting instances in scaling groups -- `ifupdown` is brought in by `salt`. If this is reproducible via manual installation, there could be some backup images/snapshots that are unable to boot though. This appears to be because cloud-init network start up operations are broken by whatever `ifupdown` sysv/systemd scripts perform on boot. With `ifupdown` installed on Azure, `eth0` does not get an IP address, the instance never fully boots according to Azure, and it will enter an error state. On AWS, the instance boots, but network operations like SSH fail. The `boot.log` is from Azure. Eventually `cloud-init` gives an exception (see `exception.log`), but this seems like a secondary issue related to networking being down. I did try to upgrade cloud-init using nightly PPAs (version 20.1 I believe), but no luck. This is on Ubuntu 18.04 instances from Azure marketplace and AWS Canonical AMI, cloud-init version `19.4-33-gbb4131a2-0ubuntu1~18.04.1`. Related: https://github.com/Azure/WALinuxAgent/issues/1612 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1867029/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1867043] [NEW] Daily builds failing due to missing pytest
Public bug reported: e.g. https://launchpadlibrarian.net/468666983/buildlog_ubuntu-xenial- amd64.cloud- init_20.1-2345-gd5dbbd1-0ubuntu1+1465~trunk~ubuntu16.04.1_BUILDING.txt.gz http_proxy= make PYVER=python3 check make[2]: Entering directory '/<>' python3 -m pytest -v tests/unittests cloudinit /usr/bin/python3: No module named pytest Makefile:52: recipe for target 'unittest3' failed make[2]: *** [unittest3] Error 1 make[2]: Leaving directory '/<>' debian/rules:11: recipe for target 'override_dh_auto_test' failed make[1]: *** [override_dh_auto_test] Error 2 make[1]: Leaving directory '/<>' debian/rules:7: recipe for target 'build' failed make: *** [build] Error 2 dpkg-buildpackage: error: debian/rules build gave error exit status 2 ** Affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1867043 Title: Daily builds failing due to missing pytest Status in cloud-init: New Bug description: e.g. https://launchpadlibrarian.net/468666983/buildlog_ubuntu-xenial- amd64.cloud- init_20.1-2345-gd5dbbd1-0ubuntu1+1465~trunk~ubuntu16.04.1_BUILDING.txt.gz http_proxy= make PYVER=python3 check make[2]: Entering directory '/<>' python3 -m pytest -v tests/unittests cloudinit /usr/bin/python3: No module named pytest Makefile:52: recipe for target 'unittest3' failed make[2]: *** [unittest3] Error 1 make[2]: Leaving directory '/<>' debian/rules:11: recipe for target 'override_dh_auto_test' failed make[1]: *** [override_dh_auto_test] Error 2 make[1]: Leaving directory '/<>' debian/rules:7: recipe for target 'build' failed make: *** [build] Error 2 dpkg-buildpackage: error: debian/rules build gave error exit status 2 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1867043/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1867029] [NEW] Package ifupdown breaks network configuration by cloud-init
Public bug reported: It appears the `ifupdown` package breaks networking configuration that cloud-init would normally handle (?) on both providers. The result, if you image an instance with the `ifupdown` package installed, is that the image is not bootable by Azure/AWS. I'm not familiar with how Azure/AWS/etc use cloud-init for network start up however. I'm filing here for now, in case `cloud-init` needs to be more defensive against `ifupdown`. To reproduce with Packer (I confirmed this method): * Add necessary Azure subscription env vars from `ifupdown-bug.json` * `packer build ifupdown-bug.json` * Try to boot an instance with resulting image * Instance will never boot -- more specifically, networking never comes up. See `boot.log` for an example boot To reproduce manually (only a guess, I did not confirm): * Boot 18.04 ubuntu instance * Install ifupdown * Image the instance * Try to boot instance using new image * Instance won't boot I hit this with Packer, creating new images for booting instances in scaling groups -- `ifupdown` is brought in by `salt`. If this is reproducible via manual installation, there could be some backup images/snapshots that are unable to boot though. This appears to be because cloud-init network start up operations are broken by whatever `ifupdown` sysv/systemd scripts perform on boot. With `ifupdown` installed on Azure, `eth0` does not get an IP address, the instance never fully boots according to Azure, and it will enter an error state. On AWS, the instance boots, but network operations like SSH fail. The `boot.log` is from Azure. Eventually `cloud-init` gives an exception (see `exception.log`), but this seems like a secondary issue related to networking being down. I did try to upgrade cloud-init using nightly PPAs (version 20.1 I believe), but no luck. This is on Ubuntu 18.04 instances from Azure marketplace and AWS Canonical AMI, cloud-init version `19.4-33-gbb4131a2-0ubuntu1~18.04.1`. Related: https://github.com/Azure/WALinuxAgent/issues/1612 ** Affects: cloud-init Importance: Undecided Status: New ** Attachment added: "cloud-init.tgz" https://bugs.launchpad.net/bugs/1867029/+attachment/5335794/+files/cloud-init.tgz -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1867029 Title: Package ifupdown breaks network configuration by cloud-init Status in cloud-init: New Bug description: It appears the `ifupdown` package breaks networking configuration that cloud-init would normally handle (?) on both providers. The result, if you image an instance with the `ifupdown` package installed, is that the image is not bootable by Azure/AWS. I'm not familiar with how Azure/AWS/etc use cloud-init for network start up however. I'm filing here for now, in case `cloud-init` needs to be more defensive against `ifupdown`. To reproduce with Packer (I confirmed this method): * Add necessary Azure subscription env vars from `ifupdown-bug.json` * `packer build ifupdown-bug.json` * Try to boot an instance with resulting image * Instance will never boot -- more specifically, networking never comes up. See `boot.log` for an example boot To reproduce manually (only a guess, I did not confirm): * Boot 18.04 ubuntu instance * Install ifupdown * Image the instance * Try to boot instance using new image * Instance won't boot I hit this with Packer, creating new images for booting instances in scaling groups -- `ifupdown` is brought in by `salt`. If this is reproducible via manual installation, there could be some backup images/snapshots that are unable to boot though. This appears to be because cloud-init network start up operations are broken by whatever `ifupdown` sysv/systemd scripts perform on boot. With `ifupdown` installed on Azure, `eth0` does not get an IP address, the instance never fully boots according to Azure, and it will enter an error state. On AWS, the instance boots, but network operations like SSH fail. The `boot.log` is from Azure. Eventually `cloud-init` gives an exception (see `exception.log`), but this seems like a secondary issue related to networking being down. I did try to upgrade cloud-init using nightly PPAs (version 20.1 I believe), but no luck. This is on Ubuntu 18.04 instances from Azure marketplace and AWS Canonical AMI, cloud-init version `19.4-33-gbb4131a2-0ubuntu1~18.04.1`. Related: https://github.com/Azure/WALinuxAgent/issues/1612 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1867029/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1863423] Re: Method "build_segment_queries_for_tenant_and_shared_ranges" returning empty query
Reviewed: https://review.opendev.org/710090
Committed:
https://git.openstack.org/cgit/openstack/neutron/commit/?id=046672247de56bad950e8267a57bd26205f354a0
Submitter: Zuul
Branch:master
commit 046672247de56bad950e8267a57bd26205f354a0
Author: Rodolfo Alonso Hernandez
Date: Wed Feb 26 10:39:19 2020 +
Fix queries to retrieve allocations with network_segment_range
Fixed the queries to retrieve the segment ID allocations when service
plugin network_segment_range is enabled. With the previous
implementation, a project user was able to allocate a segment ID
belonging to other project segment range.
The solution implemented was discussed in [1]:
- A project user will retrieve segments from the project ranges.
- When depleted, the segment IDs will be retrieved from the shared
range, never using another project segment ID.
[1]http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012736.html
Change-Id: I953062d9ee8ee5ee9a9f07aff4a8222ac63ed525
Closes-Bug: #1863423
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1863423
Title:
Method "build_segment_queries_for_tenant_and_shared_ranges" returning
empty query
Status in neutron:
Fix Released
Bug description:
Method "build_segment_queries_for_tenant_and_shared_ranges" returns two
queries:
- One for those network segment ranges matching the project_id
- One for those network segment ranges shared (no project_id, but available
for everyone)
The first one, if "project_id" is not present in the filter variable, returns
an empty list:
https://github.com/openstack/neutron/blob/6a8277d70ee28ae6fcb68a75634eb508d4e6952a/neutron/plugins/ml2/drivers/helpers.py#L117
The returned queries are used in
"allocate_partially_specified_segment":
https://github.com/openstack/neutron/blob/6a8277d70ee28ae6fcb68a75634eb508d4e6952a/neutron/plugins/ml2/drivers/helpers.py#L197-L200
If the first object is not a query but an empty list, the code will
fail.
UPDATE:
I've found some other issues related to this feature that should be addressed
in order to have a healthy functionality. Those issues were found during the
implementation of [1]
This service plugin creates, when the drivers are initialized (one per
segmentation type: VLAN, VXLAN, GRE or Geneve), a default segment
range not assigned to any project, with the min/max values defined
statically in the neutron plugin config ("network_vlan_ranges",
"vni_ranges", etc).
Then the administrator can create segment ranges for project. Those
segment ranges do not overlap among them but can overlap with the
default range.
When a network is created, the method
"SegmentTypeDriver.allocate_partially_specified_segment" selects a
segmentation ID from both the segment ranges assigned to the project
AND the shared range.
That means:
- When the the segment ranges are depleted, the project user can always
receive a segmentation from the default group. Why is then this feature needed?
- In this case, the user can have assigned a segmentation ID belonging to
other project (this segmentation can fall under the interval defined in other
segment range). There is no check for this.
- The tests implemented in [2] rely on the current buggy implementation of
this method.
Currently this new feature does not perform what is intended to do.
[1] https://review.opendev.org/708027
[2]
https://github.com/openstack/neutron-tempest-plugin/blob/b7e0eef8de92f6a70c16c879f6a9a20377e82882/neutron_tempest_plugin/api/admin/test_network_segment_range.py#L91
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1863423/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1866445] Re: br-int bridge in one compute can't learn MAC addresses of VMs in other compute nodes
** This bug is no longer a duplicate of bug 1732067 openvswitch firewall flows cause flooding on integration bridge -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1866445 Title: br-int bridge in one compute can't learn MAC addresses of VMs in other compute nodes Status in neutron: Incomplete Bug description: In Openstack Queens release, we noticed a very serious issue, br-int bridge in one compute node can't learn MAC addresses of VMs in other compute nodes, so after launched many VMs, VM-to-VM network performance will decrease linearly, especially ovs will broadcast packets because it doesn't learn target VM MAC address, other VMs in same subnet in same compute node can receive these broadcast packets, therefore, the corresponding vhost kernel threads are receiving these packets and wasting CPU resources. More VMs, more serious the issue, worse the performance, no matter UDP or TCP performance. We have checked several Queens deployments, they have same issues, but Openstack Rocky doesn't have this issue. Here is the flow I dumped: recirc_id(0),in_port(12),eth(src=fa:16:3e:49:26:51,dst=fa:16:3e:a7:0a:3a),eth_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:11012944, bytes:726983412, used:0.000s, flags:SP., actions:push_vlan(vid=1,pcp=0),2,set(tunnel(tun_id=0x49,src=10.3.2.17,dst=10.3.2.16,ttl=64,tp_dst=4789,flags(df|key))),pop_vlan,9,8,11,13,14,15,16,17,18,19 MAC address of target VM wasn't learnt by br-int $ sudo ovs-appctl fdb/show br-int | grep "fa:16:3e:a7:0a:3a" By the way, we used linuxbridge for security group. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1866445/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1866445] Re: br-int bridge in one compute can't learn MAC addresses of VMs in other compute nodes
*** This bug is a duplicate of bug 1732067 *** https://bugs.launchpad.net/bugs/1732067 bug #1732067 is same issue as this one. ** This bug has been marked a duplicate of bug 1732067 openvswitch firewall flows cause flooding on integration bridge -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1866445 Title: br-int bridge in one compute can't learn MAC addresses of VMs in other compute nodes Status in neutron: Incomplete Bug description: In Openstack Queens release, we noticed a very serious issue, br-int bridge in one compute node can't learn MAC addresses of VMs in other compute nodes, so after launched many VMs, VM-to-VM network performance will decrease linearly, especially ovs will broadcast packets because it doesn't learn target VM MAC address, other VMs in same subnet in same compute node can receive these broadcast packets, therefore, the corresponding vhost kernel threads are receiving these packets and wasting CPU resources. More VMs, more serious the issue, worse the performance, no matter UDP or TCP performance. We have checked several Queens deployments, they have same issues, but Openstack Rocky doesn't have this issue. Here is the flow I dumped: recirc_id(0),in_port(12),eth(src=fa:16:3e:49:26:51,dst=fa:16:3e:a7:0a:3a),eth_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:11012944, bytes:726983412, used:0.000s, flags:SP., actions:push_vlan(vid=1,pcp=0),2,set(tunnel(tun_id=0x49,src=10.3.2.17,dst=10.3.2.16,ttl=64,tp_dst=4789,flags(df|key))),pop_vlan,9,8,11,13,14,15,16,17,18,19 MAC address of target VM wasn't learnt by br-int $ sudo ovs-appctl fdb/show br-int | grep "fa:16:3e:a7:0a:3a" By the way, we used linuxbridge for security group. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1866445/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1866961] Re: ImportError: cannot import name 'Feature'
Problem still present in Kolla as setuptools 46 is installed by
virtualenv.
Will look at options.
** Changed in: kolla
Milestone: None => 10.0.0
** Also affects: kolla/train
Importance: Undecided
Status: New
** Also affects: kolla/ussuri
Importance: Critical
Status: Triaged
** Changed in: kolla/train
Milestone: None => 9.0.2
** Changed in: kolla/train
Status: New => Triaged
** Changed in: kolla/train
Importance: Undecided => Critical
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1866961
Title:
ImportError: cannot import name 'Feature'
Status in devstack:
Triaged
Status in OpenStack Dashboard (Horizon):
Confirmed
Status in kolla:
Triaged
Status in kolla train series:
Triaged
Status in kolla ussuri series:
Triaged
Bug description:
One of Horizon's requirements is pyscss package. Which had last
release over 4 years ago...
Two days ago setuptools v46 got released. One of changes was drop of
Features feature.
Now Kolla builds fail:
INFO:kolla.common.utils.horizon:Collecting pyScss===1.3.4
INFO:kolla.common.utils.horizon: Downloading
http://mirror.ord.rax.opendev.org:8080/pypifiles/packages/1d/4a/221ae7561c8f51c4f28b2b172366ccd0820b14bb947350df82428dfce381/pyScss-1.3.4.tar.gz
(120 kB)
INFO:kolla.common.utils.horizon:[91mERROR: Command errored out with exit
status 1:
INFO:kolla.common.utils.horizon: command: /var/lib/kolla/venv/bin/python
-c 'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';
__file__='"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
--egg-base /tmp/pip-install-rr0db3qs/pyScss/pip-egg-info
INFO:kolla.common.utils.horizon: cwd:
/tmp/pip-install-rr0db3qs/pyScss/
INFO:kolla.common.utils.horizon:Complete output (5 lines):
INFO:kolla.common.utils.horizon:Traceback (most recent call last):
INFO:kolla.common.utils.horizon: File "", line 1, in
INFO:kolla.common.utils.horizon: File
"/tmp/pip-install-rr0db3qs/pyScss/setup.py", line 9, in
INFO:kolla.common.utils.horizon:from setuptools import setup,
Extension, Feature
INFO:kolla.common.utils.horizon:ImportError: cannot import name 'Feature'
Devstack also has the same problem.
Are there any plans to fix it?
pyscss project got issue: https://github.com/Kronuz/pyScss/issues/385
What are plans of Horizon team?
To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1866961/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1866961] Re: ImportError: cannot import name 'Feature'
** Also affects: devstack
Importance: Undecided
Status: New
** Changed in: devstack
Status: New => Triaged
** Changed in: kolla
Status: Confirmed => Triaged
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1866961
Title:
ImportError: cannot import name 'Feature'
Status in devstack:
Triaged
Status in OpenStack Dashboard (Horizon):
Confirmed
Status in kolla:
Triaged
Bug description:
One of Horizon's requirements is pyscss package. Which had last
release over 4 years ago...
Two days ago setuptools v46 got released. One of changes was drop of
Features feature.
Now Kolla builds fail:
INFO:kolla.common.utils.horizon:Collecting pyScss===1.3.4
INFO:kolla.common.utils.horizon: Downloading
http://mirror.ord.rax.opendev.org:8080/pypifiles/packages/1d/4a/221ae7561c8f51c4f28b2b172366ccd0820b14bb947350df82428dfce381/pyScss-1.3.4.tar.gz
(120 kB)
INFO:kolla.common.utils.horizon:[91mERROR: Command errored out with exit
status 1:
INFO:kolla.common.utils.horizon: command: /var/lib/kolla/venv/bin/python
-c 'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';
__file__='"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
--egg-base /tmp/pip-install-rr0db3qs/pyScss/pip-egg-info
INFO:kolla.common.utils.horizon: cwd:
/tmp/pip-install-rr0db3qs/pyScss/
INFO:kolla.common.utils.horizon:Complete output (5 lines):
INFO:kolla.common.utils.horizon:Traceback (most recent call last):
INFO:kolla.common.utils.horizon: File "", line 1, in
INFO:kolla.common.utils.horizon: File
"/tmp/pip-install-rr0db3qs/pyScss/setup.py", line 9, in
INFO:kolla.common.utils.horizon:from setuptools import setup,
Extension, Feature
INFO:kolla.common.utils.horizon:ImportError: cannot import name 'Feature'
Devstack also has the same problem.
Are there any plans to fix it?
pyscss project got issue: https://github.com/Kronuz/pyScss/issues/385
What are plans of Horizon team?
To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1866961/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1866961] Re: ImportError: cannot import name 'Feature'
** Also affects: kolla
Importance: Undecided
Status: New
** Changed in: kolla
Status: New => Confirmed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1866961
Title:
ImportError: cannot import name 'Feature'
Status in devstack:
Triaged
Status in OpenStack Dashboard (Horizon):
Confirmed
Status in kolla:
Triaged
Bug description:
One of Horizon's requirements is pyscss package. Which had last
release over 4 years ago...
Two days ago setuptools v46 got released. One of changes was drop of
Features feature.
Now Kolla builds fail:
INFO:kolla.common.utils.horizon:Collecting pyScss===1.3.4
INFO:kolla.common.utils.horizon: Downloading
http://mirror.ord.rax.opendev.org:8080/pypifiles/packages/1d/4a/221ae7561c8f51c4f28b2b172366ccd0820b14bb947350df82428dfce381/pyScss-1.3.4.tar.gz
(120 kB)
INFO:kolla.common.utils.horizon:[91mERROR: Command errored out with exit
status 1:
INFO:kolla.common.utils.horizon: command: /var/lib/kolla/venv/bin/python
-c 'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';
__file__='"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
--egg-base /tmp/pip-install-rr0db3qs/pyScss/pip-egg-info
INFO:kolla.common.utils.horizon: cwd:
/tmp/pip-install-rr0db3qs/pyScss/
INFO:kolla.common.utils.horizon:Complete output (5 lines):
INFO:kolla.common.utils.horizon:Traceback (most recent call last):
INFO:kolla.common.utils.horizon: File "", line 1, in
INFO:kolla.common.utils.horizon: File
"/tmp/pip-install-rr0db3qs/pyScss/setup.py", line 9, in
INFO:kolla.common.utils.horizon:from setuptools import setup,
Extension, Feature
INFO:kolla.common.utils.horizon:ImportError: cannot import name 'Feature'
Devstack also has the same problem.
Are there any plans to fix it?
pyscss project got issue: https://github.com/Kronuz/pyScss/issues/385
What are plans of Horizon team?
To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1866961/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1866961] [NEW] ImportError: cannot import name 'Feature'
Public bug reported:
One of Horizon's requirements is pyscss package. Which had last release
over 4 years ago...
Two days ago setuptools v46 got released. One of changes was drop of
Features feature.
Now Kolla builds fail:
INFO:kolla.common.utils.horizon:Collecting pyScss===1.3.4
INFO:kolla.common.utils.horizon: Downloading
http://mirror.ord.rax.opendev.org:8080/pypifiles/packages/1d/4a/221ae7561c8f51c4f28b2b172366ccd0820b14bb947350df82428dfce381/pyScss-1.3.4.tar.gz
(120 kB)
INFO:kolla.common.utils.horizon:[91mERROR: Command errored out with exit
status 1:
INFO:kolla.common.utils.horizon: command: /var/lib/kolla/venv/bin/python -c
'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';
__file__='"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
--egg-base /tmp/pip-install-rr0db3qs/pyScss/pip-egg-info
INFO:kolla.common.utils.horizon: cwd: /tmp/pip-install-rr0db3qs/pyScss/
INFO:kolla.common.utils.horizon:Complete output (5 lines):
INFO:kolla.common.utils.horizon:Traceback (most recent call last):
INFO:kolla.common.utils.horizon: File "", line 1, in
INFO:kolla.common.utils.horizon: File
"/tmp/pip-install-rr0db3qs/pyScss/setup.py", line 9, in
INFO:kolla.common.utils.horizon:from setuptools import setup,
Extension, Feature
INFO:kolla.common.utils.horizon:ImportError: cannot import name 'Feature'
Devstack also has the same problem.
Are there any plans to fix it?
pyscss project got issue: https://github.com/Kronuz/pyScss/issues/385
What are plans of Horizon team?
** Affects: horizon
Importance: Undecided
Status: Confirmed
** Affects: kolla
Importance: Undecided
Status: Confirmed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1866961
Title:
ImportError: cannot import name 'Feature'
Status in OpenStack Dashboard (Horizon):
Confirmed
Status in kolla:
Confirmed
Bug description:
One of Horizon's requirements is pyscss package. Which had last
release over 4 years ago...
Two days ago setuptools v46 got released. One of changes was drop of
Features feature.
Now Kolla builds fail:
INFO:kolla.common.utils.horizon:Collecting pyScss===1.3.4
INFO:kolla.common.utils.horizon: Downloading
http://mirror.ord.rax.opendev.org:8080/pypifiles/packages/1d/4a/221ae7561c8f51c4f28b2b172366ccd0820b14bb947350df82428dfce381/pyScss-1.3.4.tar.gz
(120 kB)
INFO:kolla.common.utils.horizon:[91mERROR: Command errored out with exit
status 1:
INFO:kolla.common.utils.horizon: command: /var/lib/kolla/venv/bin/python
-c 'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';
__file__='"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
--egg-base /tmp/pip-install-rr0db3qs/pyScss/pip-egg-info
INFO:kolla.common.utils.horizon: cwd:
/tmp/pip-install-rr0db3qs/pyScss/
INFO:kolla.common.utils.horizon:Complete output (5 lines):
INFO:kolla.common.utils.horizon:Traceback (most recent call last):
INFO:kolla.common.utils.horizon: File "", line 1, in
INFO:kolla.common.utils.horizon: File
"/tmp/pip-install-rr0db3qs/pyScss/setup.py", line 9, in
INFO:kolla.common.utils.horizon:from setuptools import setup,
Extension, Feature
INFO:kolla.common.utils.horizon:ImportError: cannot import name 'Feature'
Devstack also has the same problem.
Are there any plans to fix it?
pyscss project got issue: https://github.com/Kronuz/pyScss/issues/385
What are plans of Horizon team?
To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1866961/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
