[Yahoo-eng-team] [Bug 1939623] [NEW] HTTP 410 Gone: Error in store configuration. Adding images to store is disabled.
Public bug reported: openstack release: Red Hat OpenStack Platform release 16.1.3 GA (Train) openstack glance --version : openstack 4.0.1 Error: HTTP 410 Gone: Error in store configuration. Adding images to store is disabled. While creating image from Opens tack CLI it is created first it was successfully created but next time by using same command to create another image it is giving below error: //--- command: openstack image create MME-Waltham-PXE-cxp9025898_4r113f01_1dot52 --disk-format qcow2 --file sgsn-mme_pxeboot_cxp9025898_4r113f01.qcow2 --property description='Ericsson MME1.52' --container-format bare Eroor: HTTP 410 Gone: Error in store configuration. Adding images to store is disabled. ** Affects: glance Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1939623 Title: HTTP 410 Gone: Error in store configuration. Adding images to store is disabled. Status in Glance: New Bug description: openstack release: Red Hat OpenStack Platform release 16.1.3 GA (Train) openstack glance --version : openstack 4.0.1 Error: HTTP 410 Gone: Error in store configuration. Adding images to store is disabled. While creating image from Opens tack CLI it is created first it was successfully created but next time by using same command to create another image it is giving below error: //--- command: openstack image create MME-Waltham-PXE-cxp9025898_4r113f01_1dot52 --disk-format qcow2 --file sgsn-mme_pxeboot_cxp9025898_4r113f01.qcow2 --property description='Ericsson MME1.52' --container-format bare Eroor: HTTP 410 Gone: Error in store configuration. Adding images to store is disabled. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1939623/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1308484] Re: glance member-create doesn't complain when tenant_id doesn't exist
AFAIK this by design, feel free to bring this as a topic in upcoming Yoga PTG. https://etherpad.opendev.org/p/yoga-ptg-glance-planning ** Changed in: glance Status: Confirmed => Opinion -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1308484 Title: glance member-create doesn't complain when tenant_id doesn't exist Status in Glance: Opinion Bug description: # glance image-create --name temporary_test --id 0bfbc34b-0ec8-4802-8f61-f45f0613fb54 # glance member-create --can-share 0bfbc34b-0ec8-4802-8f61-f45f0613fb54 'I do not exist!' # echo $? 0 # glance image-delete 0bfbc34b-0ec8-4802-8f61-f45f0613fb54 I expected it to return something greater than 0 To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1308484/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1939604] [NEW] Cannot create instance with multiqueue image, vif_type=tap (calico) and 1vcpu flavor
Public bug reported:
Tested on stable/wallaby
Fix for bug #1893263, in which it enabled vif_type=tap (calico use case)
devices to support multiqueue in nova, also caused a regression where
now when creating the instances with multiqueue, if using a flavor with
only VCPU, it fails with the error below in the logs.
This problem can easily be avoided by not using 1VCPUs flavors with
multiqueue images (because they wouldn't make sense anyway), and
therefore using non-multiqueue images when the flavor is 1VCPU, but
provides a bad user experience: Users shouldn't need to be concerned
about flavor+image combinations
Steps to reproduce are the same as #1893263 but using a 1VCPU flavor +
multiqueue metadata on images
21-08-11 17:36:44.317 376565 ERROR nova.compute.manager
[req-99e80890-6c99-4015-91b6-ef99e6be3fa7 ea7dfe225d48428c860321498e184739
8833157a5d244727a74017e5f8729312 - 0373963ccb0042da8306b35775521d60
0373963ccb0042da8306b35775521d60] [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] Failed to build and run instance:
libvirt.libvirtError: Unable to create tap device tap73a105b8-82: Invalid
argument
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] Traceback (most recent call last):
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2366, in
_build_and_run_instance
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] self.driver.spawn(context, instance,
image_meta,
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3885, in
spawn
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] self._create_guest_with_network(
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6961, in
_create_guest_with_network
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] self._cleanup_failed_start(
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] self.force_reraise()
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in
force_reraise
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] raise self.value
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6930, in
_create_guest_with_network
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] guest = self._create_guest(
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6863, in
_create_guest
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] guest.launch(pause=pause)
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 158, in launch
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] LOG.exception('Error launching a
defined domain with XML: %s',
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] self.force_reraise()
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in
force_reraise
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] raise self.value
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager [instance:
505b68b4-498c-4ea9-85ce-8be0c305ec4b] File
"/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 155, in launch
2021-08-11 17:36:44.317 376565 ERROR nova.compute.manager
[Yahoo-eng-team] [Bug 1939602] [NEW] [RFE] Add a memory profiler
Public bug reported: This RFE proposes to add a memory profiler plugin in Neutron. This plugin could be enabled to periodically log and report the Neutron server memory consumption. In https://review.opendev.org/c/openstack/neutron/+/803034 I present a draft of what this plugin could be: - It will periodically report the current and max memory consumption. - It will also report the modules with the biggest memory growth. - It is configurable: the reporting period, the max number of modules and the module patter (to filter only by a set of modules to track). By default, this plugin is disabled. This should be used only for debugging because it could affect to the performance. ** Affects: neutron Importance: Undecided Assignee: Rodolfo Alonso (rodolfo-alonso-hernandez) Status: New ** Tags: rfe ** Changed in: neutron Assignee: (unassigned) => Rodolfo Alonso (rodolfo-alonso-hernandez) ** Tags added: rfe -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1939602 Title: [RFE] Add a memory profiler Status in neutron: New Bug description: This RFE proposes to add a memory profiler plugin in Neutron. This plugin could be enabled to periodically log and report the Neutron server memory consumption. In https://review.opendev.org/c/openstack/neutron/+/803034 I present a draft of what this plugin could be: - It will periodically report the current and max memory consumption. - It will also report the modules with the biggest memory growth. - It is configurable: the reporting period, the max number of modules and the module patter (to filter only by a set of modules to track). By default, this plugin is disabled. This should be used only for debugging because it could affect to the performance. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1939602/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1939350] Re: Keystone protection tests are broken
reverted the devstack change fixed this https://review.opendev.org/c/openstack/devstack/+/804025 ** Also affects: devstack Importance: Undecided Status: New ** Changed in: devstack Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1939350 Title: Keystone protection tests are broken Status in devstack: Fix Released Status in OpenStack Identity (keystone): Triaged Bug description: The keystone functional protection tests that use tempest are failing during Neutron setup: + functions-common:is_service_enabled:1965 : return 1 + functions-common:run_process:1575: time_stop run_process + functions-common:time_stop:2309 : local name + functions-common:time_stop:2310 : local end_time + functions-common:time_stop:2311 : local elapsed_time + functions-common:time_stop:2312 : local total + functions-common:time_stop:2313 : local start_time + functions-common:time_stop:2315 : name=run_process + functions-common:time_stop:2316 : start_time=1628535788991 + functions-common:time_stop:2318 : [[ -z 1628535788991 ]] ++ functions-common:time_stop:2321 : date +%s%3N + functions-common:time_stop:2321 : end_time=1628535789042 + functions-common:time_stop:2322 : elapsed_time=51 + functions-common:time_stop:2323 : total=5353 + functions-common:time_stop:2325 : _TIME_START[$name]= + functions-common:time_stop:2326 : _TIME_TOTAL[$name]=5404 + ./stack.sh:main:1297 : is_service_enabled q-svc + functions-common:is_service_enabled:1965 : return 0 + ./stack.sh:main:1297 : [[ True == \T\r\u\e ]] + ./stack.sh:main:1298 : echo_summary 'Creating initial neutron network elements' + ./stack.sh:echo_summary:426 : [[ -t 3 ]] + ./stack.sh:echo_summary:432 : echo -e Creating initial neutron network elements + ./stack.sh:main:1301 : type -p neutron_plugin_create_initial_networks + ./stack.sh:main:1304 : create_neutron_initial_network + lib/neutron_plugins/services/l3:create_neutron_initial_network:164 : local project_id ++ lib/neutron_plugins/services/l3:create_neutron_initial_network:165 : oscwrap project list ++ lib/neutron_plugins/services/l3:create_neutron_initial_network:165 : grep ' demo ' ++ lib/neutron_plugins/services/l3:create_neutron_initial_network:165 : get_field 1 ++ functions-common:get_field:726 : local data field ++ functions-common:get_field:727 : read data ++ functions-common:oscwrap:2349: return 0 + lib/neutron_plugins/services/l3:create_neutron_initial_network:165 : project_id= + lib/neutron_plugins/services/l3:create_neutron_initial_network:166 : die_if_not_set 166 project_id 'Failure retrieving project_id for demo' + functions-common:die_if_not_set:216 : local exitcode=0 and /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead from cryptography.utils import int_from_bytes /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead from cryptography.utils import int_from_bytes You are not authorized to perform the requested action: identity:get_service. (HTTP 403) (Request-ID: req-1fcb0513-4511-48cf-ac80-34c241ddb211) ++functions-common:oscwrap:2349 return 1 +lib/glance:configure_glance_quotas:298iniset /etc/glance/glance-api.conf oslo_limit endpoint_id +lib/glance:configure_glance_quotas:302openstack role add --user glance --user-domain Default --system all reader /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead from cryptography.utils import int_from_bytes /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead from cryptography.utils import int_from_bytes You are not authorized to perform the requested action: identity:list_roles. (HTTP 403) (Request-ID: req-cab7850c-d24b-4bc7-9068-9153f2af0955) [1191190 Async create_glance_accounts:1239951]: finished create_glance_accounts with result 1 in 23 seconds There appears to be 403s when using keystone tokens to setup things in devstack, so the tests don't even run. I was able to reproduce this locally using: DEVSTACK_PARALLEL=True KEYSTONE_ENFORCE_SCOPE=True enable_plugin keystone https://opendev.org/opens
[Yahoo-eng-team] [Bug 1934917] Re: inconsistencies in OVS firewall on an agent restart
Rodolfo: Thanks for working on this one, and for the details. Based on your analysis, I'll also suggest that this probably does not meet the risk level necessary to warrant publishing a security advisory. I propose that we consider it class C1 (an impractical vulnerability) per our report taxonomy: https://security.openstack.org/vmt- process.html#report-taxonomy Accordingly, I'm switching it to a regular Public bug, but we can continue to discuss publicly whether this might warrant an advisory once (and if) fixes can be backported to maintained stable branches (so at least as far back as stable/ussuri currently). ** Description changed: - This issue is being treated as a potential security risk under - embargo. Please do not make any public mention of embargoed - (private) security vulnerabilities before their coordinated - publication by the OpenStack Vulnerability Management Team in the - form of an official OpenStack Security Advisory. This includes - discussion of the bug or associated fixes in public forums such as - mailing lists, code review systems and bug trackers. Please also - avoid private disclosure to other individuals not already approved - for access to this information, and provide this same reminder to - those who are made aware of the issue prior to publication. All - discussion should remain confined to this private bug report, and - any proposed fixes should be added to the bug as attachments. This - embargo shall not extend past 2021-10-05 and will be made - public by or on that date even if no fix is identified. - Summary On an pre-production OpenStack deployment, we observed he following during a restart of neutron-openvswitch-agent: some active flows that the OVS firewall was letting through based on SG rules before the restart, become marked as CT_MARK(CT_MARK_INVALID) ; their traffic is then dropped for a period of time that extends beyond the restart. The clearing of rules with the previous cookie does not resolve the issue. Digging this issue has led me to consider the hypothesis that during a restart, where neutron OVS agent is adding new rules with a new cookies and ultimately removing rules from the previous run not marked with newer cookies, the assumption that the new rules do not interfere with the old ones was broken. Looking at how conjunction IDs are used has led me to see that: A) the code offers no guarantee that, on a restart, a conjunction ID used for some SG rule in the previous run does not end up being used for some other SG rule on the next run B) in a case where there is an unfortunate collision (same conj_id used for two different SGs over a restart) the way OVS rules are replaced leaves room for race conditions resulting in either legitimate traffic to be dropped or illegitimate traffic to be accepted (B) with "legitimate traffic to be dropped" matches the issue as we saw it on the deployment, and the restricted conditions on which (B) would occur. This bug report first provides details on the operational issue, but independently of the analysis of this case the design issue in neutron agent described in the second part is what this bug report really is about. Slawek and Rodolfo have already been exposed to the details explained here. ## Details on the issue observed on our deployment # Context: - Queens (RH OSP13 containers) - we focus on two compute nodes where VMs run to form a cluster (one VM per compute) - SG rules are in places to allow traffic to said VM (more on this below) - TCP traffic # Reproduction attempt neutron OVS agent restarted at 11:41:35 on hyp1001 traffic is impacted (cluster healthcheck failure in the application that runs the VM) (We hadn't taken much traces for this step, we were only checking that reloading the ovs agent with debug logs was working as intended) neutron OVS agent restarted at 12:28:35 on hyp1001 no impact on cluster traffic neutron OVS agent restarted at 12:34:48 on hyp12003 VM impacted starting from "12:35:12" (24s after start of new agent) What follows is about this second occurence where traffic was impacted. extract from VM logs (redacted): 2021-04-28 12:35:22,769 WARN messages lost for 10.1s 2021-04-28 12:35:32,775 WARN messages lost for 20.0s When neutron OVS agent restarts, it is supposed to (a) redefine all the OVS rules, and then (b) remove the one that were existing before it's startup. The time at which the issue starts (12:35:12) is between (a) and (b). Time of (b) events for the different OVS bridges: 2021-04-28T12:35:29 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Cleaning stale br-int flows 2021-04-28T12:35:32 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Cleaning stale br-ovs-cp-bond2 flows 2021-04-28T12:35:32 INFO neutron.plugins.ml2.driv
[Yahoo-eng-team] [Bug 1939558] [NEW] Security group log entry remains in the database after its security group is deleted
Public bug reported: Issue originally reported by Alex Katz for OSP-16 but I can reproduce it also in master branch. Original Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1988793 Description of problem: After the security group is deleted its corresponding log entry is still presented in the database. Version-Release number of selected component (if applicable): How reproducible: Reproduced in ml2/OVS and ml2/OVN setups Steps to Reproduce: # openstack security group create sg_1 # openstack network log create --resource-type security_group --resource sg_1 --event ALL test_log # openstack security group delete sg_1 # openstack network log show test_log Actual results: there is sill entry for `test_log` Expected results: security group deletion should fail with the clear error message or the cascade deletion should happen ** Affects: neutron Importance: Medium Assignee: Slawek Kaplonski (slaweq) Status: Confirmed ** Tags: api logging -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1939558 Title: Security group log entry remains in the database after its security group is deleted Status in neutron: Confirmed Bug description: Issue originally reported by Alex Katz for OSP-16 but I can reproduce it also in master branch. Original Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1988793 Description of problem: After the security group is deleted its corresponding log entry is still presented in the database. Version-Release number of selected component (if applicable): How reproducible: Reproduced in ml2/OVS and ml2/OVN setups Steps to Reproduce: # openstack security group create sg_1 # openstack network log create --resource-type security_group --resource sg_1 --event ALL test_log # openstack security group delete sg_1 # openstack network log show test_log Actual results: there is sill entry for `test_log` Expected results: security group deletion should fail with the clear error message or the cascade deletion should happen To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1939558/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1939555] [NEW] Cache management allows queueing non-existing image-IDs
Public bug reported: Glance's cache management commands allows adding non-existing image-IDs into the queue for prefetching. As these IDs don't exists the prefetcher cannot process them and they will be just sitting in the queue until manually cleaned out wasting time and resources. ** Affects: glance Importance: High Assignee: Erno Kuvaja (jokke) Status: In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1939555 Title: Cache management allows queueing non-existing image-IDs Status in Glance: In Progress Bug description: Glance's cache management commands allows adding non-existing image- IDs into the queue for prefetching. As these IDs don't exists the prefetcher cannot process them and they will be just sitting in the queue until manually cleaned out wasting time and resources. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1939555/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1939545] [NEW] device_path not saved into bdm.connection_info during pre_live_migration
Public bug reported: Description === Various block based volume drivers attempt to save a device_path back into the stashed connection_info stored within Nova's block device mappings *after* connecting a volume to the underlying host. Thanks to the indirection caused by the various data structures used between the virt and compute layers this isn't actually saved into the underlying block device mapping database record during a typical attach flow until we get back into the compute and driver block device layer: https://github.com/openstack/nova/blob/84b61790763f91e12eebb96d955e2f83abc00d56/nova/virt/block_device.py#L613-L619 However when an instance is being live migrated these volumes are connected as part of pre_live_migration on the destination and no attempt is made to save the updates made to the connection_info of the volume into the database. This isn't a massive problem as os-brick can for the most part lookup the device during future operations but it is obviously inefficient. This was initially hit in bug #1936439 but that bug is now being used to track in a trivial DEBUG log change while this bug will track in the underlying fix for the above issue. Steps to reproduce == * Attach an iSCSI/FC/NVME etc volume to an instance * Live migrate the instance * Confirm that device_path isn't present in the connection_info stashed in the bdm Expected result === device_path is stashed in the connection_info of the bdm Actual result = device_path isn't stashed in the connection_info of the bdm Environment === 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ master 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? libvirt 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? LVM/iSCSI/FC/NVMe, any block based volume backends. 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs == See bug #1936439 ** Affects: nova Importance: Undecided Assignee: Lee Yarwood (lyarwood) Status: New ** Tags: volumes -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1939545 Title: device_path not saved into bdm.connection_info during pre_live_migration Status in OpenStack Compute (nova): New Bug description: Description === Various block based volume drivers attempt to save a device_path back into the stashed connection_info stored within Nova's block device mappings *after* connecting a volume to the underlying host. Thanks to the indirection caused by the various data structures used between the virt and compute layers this isn't actually saved into the underlying block device mapping database record during a typical attach flow until we get back into the compute and driver block device layer: https://github.com/openstack/nova/blob/84b61790763f91e12eebb96d955e2f83abc00d56/nova/virt/block_device.py#L613-L619 However when an instance is being live migrated these volumes are connected as part of pre_live_migration on the destination and no attempt is made to save the updates made to the connection_info of the volume into the database. This isn't a massive problem as os-brick can for the most part lookup the device during future operations but it is obviously inefficient. This was initially hit in bug #1936439 but that bug is now being used to track in a trivial DEBUG log change while this bug will track in the underlying fix for the above issue. Steps to reproduce == * Attach an iSCSI/FC/NVME etc volume to an instance * Live migrate the instance * Confirm that device_path isn't present in the connection_info stashed in the bdm Expected result === device_path is stashed in the connection_info of the bdm Actual result = device_path isn't stashed in the connection_info of the bdm Environment === 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ master 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? libvirt 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? LVM/iSCSI/FC/NVMe, any block based volume backends. 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs == See bug #1936439 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1
[Yahoo-eng-team] [Bug 1939507] [NEW] Timeout while waiting for router HA state transition
Public bug reported:
It happens in functional tests, like e.g. in
neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_ipv6_router_advts_and_fwd_after_router_state_change_backup:
https://a1fab4006c6a1daf82f2-bd8cbc347d913753596edf9ef5797d55.ssl.cf1.rackcdn.com/786478/17/check/neutron-
functional-with-uwsgi/7250dcf/testr_results.html
Error is like:
ft1.10:
neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_ipv6_router_advts_and_fwd_after_router_state_change_backuptesttools.testresult.real._StringException:
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py",
line 702, in wait_until_true
eventlet.sleep(sleep)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/greenthread.py",
line 36, in sleep
hub.switch()
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/hubs/hub.py",
line 313, in switch
return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py",
line 183, in func
return f(self, *args, **kwargs)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_ha_router.py",
line 148, in test_ipv6_router_advts_and_fwd_after_router_state_change_backup
self._test_ipv6_router_advts_and_fwd_helper('backup',
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_ha_router.py",
line 118, in _test_ipv6_router_advts_and_fwd_helper
common_utils.wait_until_true(lambda: router.ha_state == 'backup')
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py",
line 707, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds
** Affects: neutron
Importance: High
Status: Confirmed
** Tags: functional-tests gate-failure l3-ha
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1939507
Title:
Timeout while waiting for router HA state transition
Status in neutron:
Confirmed
Bug description:
It happens in functional tests, like e.g. in
neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_ipv6_router_advts_and_fwd_after_router_state_change_backup:
https://a1fab4006c6a1daf82f2-bd8cbc347d913753596edf9ef5797d55.ssl.cf1.rackcdn.com/786478/17/check/neutron-
functional-with-uwsgi/7250dcf/testr_results.html
Error is like:
ft1.10:
neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_ipv6_router_advts_and_fwd_after_router_state_change_backuptesttools.testresult.real._StringException:
Traceback (most recent call last):
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line
702, in wait_until_true
eventlet.sleep(sleep)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/greenthread.py",
line 36, in sleep
hub.switch()
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/hubs/hub.py",
line 313, in switch
return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py",
line 183, in func
return f(self, *args, **kwargs)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_ha_router.py",
line 148, in test_ipv6_router_advts_and_fwd_after_router_state_change_backup
self._test_ipv6_router_advts_and_fwd_helper('backup',
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_ha_router.py",
line 118, in _test_ipv6_router_advts_and_fwd_helper
common_utils.wait_until_true(lambda: router.ha_state == 'backup')
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line
707, in wait_until_true
raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1939507/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1939514] [NEW] test_dvr_fip_and_router_namespace_rules_with_address_scopes_match iptables failure
Public bug reported: The functional test neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR .test_dvr_fip_and_router_namespace_rules_with_address_scopes_match failed because of iptables failed to commit: ft1.1: neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_fip_and_router_namespace_rules_with_address_scopes_matchtesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/iptables_manager.py", line 432, in defer_apply self.defer_apply_off() File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/iptables_manager.py", line 446, in defer_apply_off self._apply() File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/iptables_manager.py", line 464, in _apply first = self._apply_synchronized() File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/iptables_manager.py", line 623, in _apply_synchronized raise err File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/iptables_manager.py", line 513, in _do_run_restore linux_utils.execute(args, process_input='\n'.join(commands), File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute raise exceptions.ProcessExecutionError(msg, neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'qrouter-c42abcb5-207c-43c1-9082-c97c7119066b', 'iptables-restore', '-n', '-w', '10', '-W', 20]; Stdin: # Generated by iptables_manager *filter -D run.py-scope 3 COMMIT # Completed by iptables_manager # Generated by iptables_manager *mangle -D run.py-scope 3 COMMIT # Completed by iptables_manager ; Stdout: ; Stderr: iptables-restore: line 3 failed https://bfef12d50a901fd19bfa-41761a29591b60104749264f30dd0204.ssl.cf5.rackcdn.com/804120/1/check/neutron- functional-with-uwsgi/c2fb7e2/testr_results.html ** Affects: neutron Importance: Undecided Status: New ** Tags: functional-tests gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1939514 Title: test_dvr_fip_and_router_namespace_rules_with_address_scopes_match iptables failure Status in neutron: New Bug description: The functional test neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR .test_dvr_fip_and_router_namespace_rules_with_address_scopes_match failed because of iptables failed to commit: ft1.1: neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_fip_and_router_namespace_rules_with_address_scopes_matchtesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/iptables_manager.py", line 432, in defer_apply self.defer_apply_off() File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/iptables_manager.py", line 446, in defer_apply_off self._apply() File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/iptables_manager.py", line 464, in _apply first = self._apply_synchronized() File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/iptables_manager.py", line 623, in _apply_synchronized raise err File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/iptables_manager.py", line 513, in _do_run_restore linux_utils.execute(args, process_input='\n'.join(commands), File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute raise exceptions.ProcessExecutionError(msg, neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'qrouter-c42abcb5-207c-43c1-9082-c97c7119066b', 'iptables-restore', '-n', '-w', '10', '-W', 20]; Stdin: # Generated by iptables_manager *filter -D run.py-scope 3 COMMIT # Completed by iptables_manager # Generated by iptables_manager *mangle -D run.py-scope 3 COMMIT # Completed by iptables_manager ; Stdout: ; Stderr: iptables-restore: line 3 failed https://bfef12d50a901fd19bfa-41761a29591b60104749264f30dd0204.ssl.cf5.rackcdn.com/804120/1/check/neutron- functional-with-uwsgi/c2fb7e2/testr_results.html To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1939514/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
