[Yahoo-eng-team] [Bug 2075150] Re: unable to use pdb for nova
Hey, I think this bug is actually belonging to the Nova project rather then OpenStack-Ansible, as that's basically a question regarding specific service source code rather then a deployment of this service. Next to that, I'd say that conductor part is likely to be catched not through nova-api but rather through nova-conductor service. But also - that's RPC which is likely run in async way which might be really tricky to catch with pdb. But I'll leave this to answer for Nova folks. ** Project changed: openstack-ansible => nova -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2075150 Title: unable to use pdb for nova Status in OpenStack Compute (nova): New Bug description: We have and openstack ansible antelope setup. I am trying to debug nova-api service. I have inserted "import pdb; pdb.set_trace()" at line 412 in the file "/openstack/venvs/nova-27.3.0/lib/python3.8/site- packages/nova/conductor/rpcapi.py" , then started the service manually from the prompt as shown below: --- root#: /openstack/venvs/uwsgi-27.3.0-python3/bin/uwsgi --autoload --ini /etc/uwsgi/nova-api-os-compute.ini After the launching the above command, I am trying to launch a vm using the nova command. It is dropping into the pdb shell but going to the next line after the prompt and waiting. when I input any pdb command, it is not responding as shown above. --- Nova API policies are introducing new default roles with scope_type capabilities. Old policies are deprecated and silently going to be ignored in nova 23.0.0 release. . Either ensure your deployment is ready for the new default or copy/paste the deprecated policy into your policy file and maintain it manually. warnings.warn(deprecated_msg) > /openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/conductor/rpcapi.py(413)schedule_and_build_instances() -> cctxt = self.client.prepare(version=version) (Pdb) I have attached a clip for reference. Please let me know why is pdb behaving like this To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2075150/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2063364] Re: forbidden traits in flavor not honored
Hey, Despite I do see this was deployed with OpenStack-Ansible, it feels this has very little to do with deployment method, but rather service itself. With that I will transfer this bug to Nova project, so that they could assist you. ** Project changed: openstack-ansible => nova -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2063364 Title: forbidden traits in flavor not honored Status in OpenStack Compute (nova): New Bug description: For compatibility reasons of some older VMs, I've used the following cpu_mode and cpu_models definition in one of the compute nodes nova.conf: [libvirt] inject_partition = -2 inject_password = False inject_key = False virt_type = kvm live_migration_with_native_tls = true live_migration_scheme = tls live_migration_inbound_addr = 172.29.236.114 hw_disk_discard = ignore disk_cachemodes = cpu_mode = custom cpu_models = IvyBridge In terms of flavor definition, I've set the following trait: openstack image set --property traits:HW_CPU_X86_AVX512BW=forbidden flavor.test I know for sure that HW_CPU_X86_AVX512BW is not supported in IvyBridge so my understanding is that this trait definition should suffice to select the specific compute node. In fact I have verified that placement service does NOT return the specific compute when quering for the specific trait (and instead returns all other nodes). Yet, when scheduling an instance using this flavor, I get the following error. Traceback (most recent call last): File "/openstack/venvs/nova-28.0.0/lib/python3.10/site- packages/nova/conductor/manager.py", line 1654, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/openstack/venvs/nova-28.0.0/lib/python3.10/site- packages/nova/conductor/manager.py", line 942, in _schedule_instances host_lists = self.query_client.select_destinations( File "/openstack/venvs/nova-28.0.0/lib/python3.10/site- packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/openstack/venvs/nova-28.0.0/lib/python3.10/site- packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/openstack/venvs/nova-28.0.0/lib/python3.10/site- packages/oslo_messaging/rpc/client.py", line 190, in call result = self.transport._send( File "/openstack/venvs/nova-28.0.0/lib/python3.10/site- packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/openstack/venvs/nova-28.0.0/lib/python3.10/site- packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/openstack/venvs/nova-28.0.0/lib/python3.10/site- packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. Traceback (most recent call last): File "/openstack/venvs/nova-28.0.0/lib/python3.10/site- packages/oslo_messaging/rpc/server.py", line 244, in inner return func(*args, **kwargs) File "/openstack/venvs/nova-28.0.0/lib/python3.10/site- packages/nova/scheduler/manager.py", line 243, in select_destinations raise exception.NoValidHost(reason="") nova.exception.NoValidHost: No valid host was found. This seems like a bug of the trait selection logic of nova. openstack ansible version is 2023.2 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2063364/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2056300] [NEW] Some Roboto fonts are looked in the wrong location
Public bug reported:
I do have a basic OpenStack-Ansible All-in-One deployment of the current
master branch.
Horizon SHA: a77818d745f25ed1e5b576760d3481f0f62ce042
Deployment was done following way:
1. git clone https://opendev.org/openstack/openstack-ansible
2. cd openstack-ansible
3. ./scripts/gate-check-commit.sh aio_lxc
Once it's done, horizon is working properly. However, from time to time
I do catch "429 Too many requests" which is potentially legit behavior
by LB configuration, that tries to implement bot protection:
http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 }:
https://opendev.org/openstack/openstack-ansible/src/commit/e72984ca956d44d8056ca4ea8ea7e263bc3c8881/inventory/group_vars/horizon_all/haproxy_service.yml#L24
Violation of this rule is triggered by invalid path to roboto fontface
which returns quite some 404 per each request:
[06/Mar/2024:08:58:13.851] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/1/1/2 404 340 - -
32/1/0/0/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/Roboto/Roboto-Regular.woff2
HTTP/2.0"
[06/Mar/2024:08:58:13.856] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/0/1/1 404 340 - -
32/1/0/0/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/Roboto/Roboto-Bold.woff2
HTTP/2.0"
On the horizon container this path is indeed not valid:
root@aio1-horizon-container-f7cace3d:/# ls -l
/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-packages/static/horizon/lib/roboto_fontface/fonts/Roboto/Roboto-Bold.woff2ls:
cannot access
'/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-packages/static/horizon/lib/roboto_fontface/fonts/Roboto/Roboto-Bold.woff2':
No such file or directory
root@aio1-horizon-container-f7cace3d:/#
However, expected path is slightly different:
root@aio1-horizon-container-f7cace3d:/# ls -l
/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-packages/static/horizon/lib/roboto_fontface/fonts/roboto/Roboto-Bold.woff2
-rwxr-xr-x 1 root root 63596 Mar 6 08:37
/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-packages/static/horizon/lib/roboto_fontface/fonts/roboto/Roboto-Bold.woff2
root@aio1-horizon-container-f7cace3d:/#
So doing smth like `cp -r
/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-
packages/static/horizon/lib/roboto_fontface/fonts/roboto/
/openstack/venvs/horizon-28.1.0.dev67/lib/python3.10/site-
packages/static/horizon/lib/roboto_fontface/fonts/Roboto/` does the
trick.
And the issue is basically in `roboto` vs `Roboto` in path.
If I would try to `mv roboto Roboto` - then some dashboard will fail,
like heat-dashboard, as they do expect `roboto`:
[06/Mar/2024:09:10:30.452] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/0/0/0 404 340 - -
38/1/2/2/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/roboto/Roboto-Regular.woff2
HTTP/2.0"
[06/Mar/2024:09:10:30.560] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/0/3/3 404 340 - -
38/1/2/2/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/roboto/Roboto-Regular.woff
HTTP/2.0"
[06/Mar/2024:09:10:30.604] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/0/1/1 404 340 - -
38/1/2/2/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/roboto/Roboto-Regular.ttf
HTTP/2.0"
The /roboto directory is being populated automatically during `horizon-
manage.py collectstatic --noinput` execution.
** Affects: horizon
Importance: Undecided
Status: New
** Description changed:
I do have a basic OpenStack-Ansible All-in-One deployment of the current
master branch.
Horizon SHA: a77818d745f25ed1e5b576760d3481f0f62ce042
Deployment was done following way:
1. git clone https://opendev.org/openstack/openstack-ansible
2. cd openstack-ansible
3. ./scripts/gate-check-commit.sh aio_lxc
Once it's done, horizon is working properly. However, from time to time
I do catch "429 Too many requests" which is potentially legit behavior
by LB configuration, that tries to implement bot protection:
http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 }:
https://opendev.org/openstack/openstack-ansible/src/commit/e72984ca956d44d8056ca4ea8ea7e263bc3c8881/inventory/group_vars/horizon_all/haproxy_service.yml#L24
-
- Violation of this rule is triggered by invalid path to roboto fontface which
returns quite some 404 per each request:
+ Violation of this rule is triggered by invalid path to roboto fontface
+ which returns quite some 404 per each request:
[06/Mar/2024:08:58:13.851] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/1/1/2 404 340 - -
32/1/0/0/0 0/0 "GET
https://ip.add.re.ss/static/horizon/lib/roboto_fontface/fonts/Roboto/Roboto-Regular.woff2
HTTP/2.0"
[06/Mar/2024:08:58:13.856] base-front-1~
horizon-back/aio1_horizon_container-f7cace3d 0/0/0/1/1 404
[Yahoo-eng-team] [Bug 1823023] Re: legacy router namespaces absence or error run exec
it's weird that routers were scheduled to compute nodes - we usually suggested using either standalone net nodes or controllers as destination for L3 agents Also we haven't seen issues like that really for a while now. Since Rocky is not supported and reach it's End Of Life, at this point there is no way to help you out with the issue as of today. But I suppose that it was fixed, as things work pretty much stable today. Please feel free to bump this bug report or open a new one in case of simmilar issues in the future. ** Changed in: openstack-ansible Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1823023 Title: legacy router namespaces absence or error run exec Status in neutron: Incomplete Status in openstack-ansible: Fix Released Bug description: Hi, We have a rocky OSA setup 18.1.4 version from git. Whenever we create a router it is getting created and showing a compute node as host with the command "l3-agent-list-hosting-router" . But when we log into the compute node and check, there is no namespace for that router and sometimes even though that namespace is created, when we do a ip netns exec qrouter-x ip a that throws an error "Unable to find router with name or id '7ec2fa3057374a1584418124d5b879ca': Also when we do a ip netns on the computes we see this : - Error: Peer netns reference is invalid. Error: Peer netns reference is invalid. - The neutron.conf file on the computes: -- # Ansible managed # General, applies to all host groups [DEFAULT] debug = True # Domain to use for building hostnames dns_domain = vbg.example.cloud ## Rpc all executor_thread_pool_size = 64 fatal_deprecations = False l3_ha = False log_file = /var/log/neutron/neutron.log rpc_response_timeout = 60 transport_url = rabbit://neutron:6a5c9d9634190b954f133f274d793be4d2@172.29.236.201:5671,neutron:6a5c9d9634190b954f133f274d793be4d2@172.29.239.27:5671,neutron:6a5c9d9634190b954f133f274d793be4d2@172.29.239.39:5671//neutron?ssl=1 # Disable stderr logging use_stderr = False # Agent [agent] polling_interval = 5 report_interval = 60 root_helper = sudo /openstack/venvs/neutron-18.1.4/bin/neutron-rootwrap /etc/neutron/rootwrap.conf # Concurrency (locking mechanisms) [oslo_concurrency] lock_path = /var/lock/neutron # Notifications [oslo_messaging_notifications] driver = messagingv2 notification_topics = notifications,notifications_designate transport_url = rabbit://neutron:6a5c9d9634190b954f133f274d793be4d2@172.29.236.201:5671,neutron:6a5c9d9634190b954f133f274d793be4d2@172.29.239.27:5671,neutron:6a5c9d9634190b954f133f274d793be4d2@172.29.239.39:5671//neutron?ssl=1 # Messaging [oslo_messaging_rabbit] rpc_conn_pool_size = 30 ssl = True -- l3_agent.ini file # Ansible managed # General [DEFAULT] debug = True # Drivers interface_driver = openvswitch agent_mode = legacy # Conventional failover allow_automatic_l3agent_failover = True # HA failover ha_confs_path = /var/lib/neutron/ha_confs ha_vrrp_advert_int = 2 ha_vrrp_auth_password = ec86ebf62a85f387569ed0251dc7c8dd9e484949ba320a6ee6bf483758a318 ha_vrrp_auth_type = PASS # Metadata enable_metadata_proxy = True # L3 plugins Please help us with this issue. Thanks Y.G Kumar To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1823023/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2040264] [NEW] VM rebuild fails after Zed->2023.1 upgrade
Public bug reported: Description === After upgrade of nova, including compute and conductor nodes, VM rebuild fails. All computes, that have service state UP, and all conductors are having version 66. Though, there was 1 compute during upgrade that is DOWN, which does have version 64. Due to it conductor negotiates minimal version to 64, which is still acceptable minimal RPC version though that leads to not passing another required argument. Steps to reproduce == * Setup env with Nova version 26.2.0 * Perform upgrade to 27.1.0 where 1 compute will be down or not upgraded (and thus can't update it's rpc version to latest 66) * Try to re-build the VM: openstack server rebuild --image Expected result === VM is rebuild Actual result = VM is stuck in rebuilding state with following trace in nova-compute Logs & Configs == Stack trace from nova-compute: https://paste.openstack.org/show/biUIcOzMCx0YlsFob2KK/ Nova-conductor does negotiation by minimal version: INFO nova.compute.rpcapi [None req-2670be51-8233-4269-ac6a-f49486e8893d - - - - - -] Automatically selected compute RPC version 6.1 from minimum service version 64 Potentially, there's another issue upgrading from Yoga to 2023.1 related to this: https://github.com/openstack/nova/commit/30aab9c234035b49c7e2cdc940f624a63eeffc1b#diff-47eb12598e353b9e0689707d7b477353200d0aa3ed13045ffd3d017ee7d9e753R3709 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2040264 Title: VM rebuild fails after Zed->2023.1 upgrade Status in OpenStack Compute (nova): New Bug description: Description === After upgrade of nova, including compute and conductor nodes, VM rebuild fails. All computes, that have service state UP, and all conductors are having version 66. Though, there was 1 compute during upgrade that is DOWN, which does have version 64. Due to it conductor negotiates minimal version to 64, which is still acceptable minimal RPC version though that leads to not passing another required argument. Steps to reproduce == * Setup env with Nova version 26.2.0 * Perform upgrade to 27.1.0 where 1 compute will be down or not upgraded (and thus can't update it's rpc version to latest 66) * Try to re-build the VM: openstack server rebuild --image Expected result === VM is rebuild Actual result = VM is stuck in rebuilding state with following trace in nova-compute Logs & Configs == Stack trace from nova-compute: https://paste.openstack.org/show/biUIcOzMCx0YlsFob2KK/ Nova-conductor does negotiation by minimal version: INFO nova.compute.rpcapi [None req-2670be51-8233-4269-ac6a-f49486e8893d - - - - - -] Automatically selected compute RPC version 6.1 from minimum service version 64 Potentially, there's another issue upgrading from Yoga to 2023.1 related to this: https://github.com/openstack/nova/commit/30aab9c234035b49c7e2cdc940f624a63eeffc1b#diff-47eb12598e353b9e0689707d7b477353200d0aa3ed13045ffd3d017ee7d9e753R3709 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2040264/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2039597] [NEW] nova-status upgrade check fails on Object ID linkage
Public bug reported: Description === With upgrade from 2023.1 to 2023.2 when running nova-status upgrade check it fails with exit code 2. According to the documentation [1], this command was run with the new codebase (2023.2) but before any service (api/conductor/scheduler/compute) was restarted, so they still run on 2023.1 codebase. With that all computes are UP and healthy: # openstack compute service list +--++--+--+-+---++ | ID | Binary | Host | Zone | Status | State | Updated At | +--++--+--+-+---++ | 001ea1ce-363f-41d1-9ce3-59ff966452a7 | nova-conductor | aio1 | internal | enabled | up| 2023-10-17T18:14:38.00 | | 8df25103-65c9-4892-be05-ebed7f3c1ad4 | nova-scheduler | aio1 | internal | enabled | up| 2023-10-17T18:14:40.00 | | d85b115a-cd8a-4ac9-82bc-f7a5f457cedc | nova-compute | aio1 | nova | enabled | up| 2023-10-17T18:14:39.00 | +--++--+--+-+---++ Steps to reproduce == * Run cluster on 2023.1 * Perform upgrade to 2023.2 but do not restart nova services (as assumed by the documentation) * Run nova-status upgrade check Expected result === Upgrade check passes Actual result = +-+ | Check: Object ID linkage| | Result: Failure | | Details: Compute node objects without service_id linkage were found | | in the database. Ensure all non-deleted compute services | | have started with upgraded code. | +-+ 1] https://docs.openstack.org/nova/latest/cli/nova-status.html#upgrade ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2039597 Title: nova-status upgrade check fails on Object ID linkage Status in OpenStack Compute (nova): New Bug description: Description === With upgrade from 2023.1 to 2023.2 when running nova-status upgrade check it fails with exit code 2. According to the documentation [1], this command was run with the new codebase (2023.2) but before any service (api/conductor/scheduler/compute) was restarted, so they still run on 2023.1 codebase. With that all computes are UP and healthy: # openstack compute service list +--++--+--+-+---++ | ID | Binary | Host | Zone | Status | State | Updated At | +--++--+--+-+---++ | 001ea1ce-363f-41d1-9ce3-59ff966452a7 | nova-conductor | aio1 | internal | enabled | up| 2023-10-17T18:14:38.00 | | 8df25103-65c9-4892-be05-ebed7f3c1ad4 | nova-scheduler | aio1 | internal | enabled | up| 2023-10-17T18:14:40.00 | | d85b115a-cd8a-4ac9-82bc-f7a5f457cedc | nova-compute | aio1 | nova | enabled | up| 2023-10-17T18:14:39.00 | +--++--+--+-+---++ Steps to reproduce == * Run cluster on 2023.1 * Perform upgrade to 2023.2 but do not restart nova services (as assumed by the documentation) * Run nova-status upgrade check Expected result === Upgrade check passes Actual result = +-+ | Check: Object ID linkage| | Result: Failure | | Details: Compute node objects without service_id linkage were found | | in the database. Ensure all non-deleted compute services | | have started with upgraded code. | +-+ 1] https://docs.openstack.org/nova/latest/cli/nova-status.html#upgrade To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2039597/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help
[Yahoo-eng-team] [Bug 2028809] Re: Upgrades from Zed to Antelope may fail due to the password truncation
** Changed in: openstack-ansible Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/2028809 Title: Upgrades from Zed to Antelope may fail due to the password truncation Status in OpenStack Identity (keystone): Fix Released Status in openstack-ansible: Fix Released Bug description: Since 2023.1, keystone truncates bcrypt passwords to 54 characters[1] while OSA generates passwords for openstack services with length between 16 and 64 characters[2]. It may cause issues with keystone authentication after upgrade because we recently disabled password updates by default.[3] Example scenario: 1. User1 was created during Zed release with password containing 64 characters. 2. Password was hashed using all 64 characters(In [4] it is only mentioned that bytes 55-72 are not fully mixed into the resulting hash, but it means they are still used to some extent). 3. Openstack is upgraded to 2023.1(where keystone truncates passwords to 54 chars when hashing). 4. User1 cannot authenticate to keystone because hash was originally created using 64 characters but now only 54 characters are used. As a solution I recommend: - Enable ``service_update_password`` during upgrade process to rehash service passwords again(and keep information about it in major upgrades guide until 2024.1). Please note that it will only fix passwords managed by openstack-ansible. User passwords containing more than 54 characters will stop working. Enabling ``service_update_password`` may also prolong API downtime due to the bug #2023370[5]. - Edit pw-token-gen.py script to generate passwords with length up to 54 characters. I do not suggest switching to scrypt because: - We cannot rehash bcrypt passwords anyway - Bcrypt is still default password_hash_algorithm in keystone [1] https://review.opendev.org/c/openstack/keystone/+/828595 [2] https://opendev.org/openstack/openstack-ansible/src/commit/d1e30257ae0c818780684fe77e1b34ba4dd0dc40/scripts/pw-token-gen.py#L85 [3] https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/888152 [4] https://passlib.readthedocs.io/en/stable/lib/passlib.hash.bcrypt.html#security-issues [5] https://bugs.launchpad.net/openstack-ansible/+bug/2023370 To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/2028809/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2033209] Re: changing openstack_domain does not change in nova DB
Hey, Sorry, I am inclined to say that the problem you are describing is specifically nova issue. I will change affected project to Nova, as I believe this bug report fits better there. As far as I know regarding the topic - this is pretty much intended behavior and a known dropback. The only workaround I'm aware of is to update the database. So in case a hostname for compute needs to change, we always ensure that no VM is running there before doing so. ** Project changed: openstack-ansible => nova -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2033209 Title: changing openstack_domain does not change in nova DB Status in OpenStack Compute (nova): New Bug description: In inventory/group_vars/all.yml file and in openstack_hosts/defaults/main.yml role openstack_domain variable defined as openstack.local openstack_domain: openstack.local However, during OSA deployment for some reason, we used the FQDN name. such as example.com after updating the openstack_domain. all existing instances has error while live-migration that they can not find the new compute. Unable to find record for source node compute-02.openstack.local on compute-02: nova.exception.ComputeHostNotFound: Compute host compute-02 could not be found. it searched for compute-02.openstack.local but it does not exist anymore. After I checked the Nova DB I found that the compute_nodes and instances table is not updated with the new openstack_domain name. now I want a way other than manually changing the DB fields To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2033209/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1973349] [NEW] Slow queries after upgrade to Xena
Public bug reported: After upgrading to Xena we started noticing slow queries that were written down in mysql slow log. Most of them were including next subquery: SELECT DISTINCT ports.id AS ports_id FROM ports, networks WHERE ports.project_id = '' OR ports.network_id = networks.id AND networks.project_id = ''. So for example, when issuing `openstack project list` this subquery appears several times: ``` SELECT allowedaddresspairs.port_id AS allowedaddresspairs_port_id, allowedaddresspairs.mac_address AS allowedaddresspairs_mac_address, allowedaddresspairs.ip_address AS allowedaddresspairs_ip_address, anon_1.ports_id AS anon_1_ports_id \nFROM (SELECT DISTINCT ports.id AS ports_id \nFROM ports, networks \nWHERE ports.project_id = '' OR ports.network_id = networks.id AND networks.project_id = '') AS anon_1 INNER JOIN allowedaddresspairs ON anon_1.ports_id = allowedaddresspairs.port_id SELECT extradhcpopts.id AS extradhcpopts_id, extradhcpopts.port_id AS extradhcpopts_port_id, extradhcpopts.opt_name AS extradhcpopts_opt_name, extradhcpopts.opt_value AS extradhcpopts_opt_value, extradhcpopts.ip_version AS extradhcpopts_ip_version, anon_1.ports_id AS anon_1_ports_id \nFROM (SELECT DISTINCT ports.id AS ports_id \nFROM ports, networks \nWHERE ports.project_id = '' OR ports.network_id = networks.id AND networks.project_id = '') AS anon_1 INNER JOIN extradhcpopts ON anon_1.ports_id = extradhcpopts.port_id0.000 SELECT ipallocations.port_id AS ipallocations_port_id, ipallocations.ip_address AS ipallocations_ip_address, ipallocations.subnet_id AS ipallocations_subnet_id, ipallocations.network_id AS ipallocations_network_id, anon_1.ports_id AS anon_1_ports_id \nFROM (SELECT DISTINCT ports.id AS ports_id \nFROM ports, networks \nWHERE ports.project_id = '' OR ports.network_id = networks.id AND networks.project_id = '') AS anon_1 INNER JOIN ipallocations ON anon_1.ports_id = ipallocations.port_id ORDER BY ipallocations.ip_address, ipallocations.subnet_id ``` Another interesting thing is difference in execution time between admin/non-admin call: (openstack) dmitriy@6BT6XT2:~$ . Documents/openrc/admin.rc (openstack) dmitriy@6BT6XT2:~$ time openstack port list --project | wc -l 2142 real0m5,401s user0m1,565s sys 0m0,086s (openstack) dmitriy@6BT6XT2:~$ . Documents/openrc/.rc (openstack) dmitriy@6BT6XT2:~$ time openstack port list | wc -l 2142 real2m38,101s user0m1,626s sys 0m0,083s (openstack) dmitriy@6BT6XT2:~$ Environment: Neutron SHA: 97180b01837638bd0476c28bdda2340eccd649af Backend: ovs OS: Ubuntu 20.04 Mariadb: 10.6.5 SQLalchemy: 1.4.23 Backend: openvswitch Plugins: router vpnaas metering neutron_dynamic_routing.services.bgp.bgp_plugin.BgpPlugin ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1973349 Title: Slow queries after upgrade to Xena Status in neutron: New Bug description: After upgrading to Xena we started noticing slow queries that were written down in mysql slow log. Most of them were including next subquery: SELECT DISTINCT ports.id AS ports_id FROM ports, networks WHERE ports.project_id = '' OR ports.network_id = networks.id AND networks.project_id = ''. So for example, when issuing `openstack project list` this subquery appears several times: ``` SELECT allowedaddresspairs.port_id AS allowedaddresspairs_port_id, allowedaddresspairs.mac_address AS allowedaddresspairs_mac_address, allowedaddresspairs.ip_address AS allowedaddresspairs_ip_address, anon_1.ports_id AS anon_1_ports_id \nFROM (SELECT DISTINCT ports.id AS ports_id \nFROM ports, networks \nWHERE ports.project_id = '' OR ports.network_id = networks.id AND networks.project_id = '') AS anon_1 INNER JOIN allowedaddresspairs ON anon_1.ports_id = allowedaddresspairs.port_id SELECT extradhcpopts.id AS extradhcpopts_id, extradhcpopts.port_id AS extradhcpopts_port_id, extradhcpopts.opt_name AS extradhcpopts_opt_name, extradhcpopts.opt_value AS extradhcpopts_opt_value, extradhcpopts.ip_version AS extradhcpopts_ip_version, anon_1.ports_id AS anon_1_ports_id \nFROM (SELECT DISTINCT ports.id AS ports_id \nFROM ports, networks \nWHERE ports.project_id = '' OR ports.network_id = networks.id AND networks.project_id = '') AS anon_1 INNER JOIN extradhcpopts ON anon_1.ports_id = extradhcpopts.port_id0.000 SELECT ipallocations.port_id AS ipallocations_port_id, ipallocations.ip_address AS ipallocations_ip_address, ipallocations.subnet_id AS ipallocations_subnet_id, ipallocations.network_id AS ipallocations_network_id, anon_1.ports_id AS anon_1_ports_id \nFROM (SELECT DISTINCT ports.id AS ports_id \nFROM ports, networks \nWHERE ports.project_id = '' OR ports.network_id = networks.id AND networks.project_id = '') AS anon_1 INNER JOIN ipallocations ON anon_1.ports_id =
[Yahoo-eng-team] [Bug 1966452] Re: Using resources:PCPU property does not result in CPU pinning
There's a typo in flavor specification I was copy-pasting several times ;( | properties | resouces:PCPU='4', trait:CUSTOM_HIGHCPU='required' | s/resouces/resources/ ** Changed in: nova Status: Triaged => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1966452 Title: Using resources:PCPU property does not result in CPU pinning Status in OpenStack Compute (nova): Invalid Bug description: When instance is being created from flavor with `--property resources:PCPU=4`, generated XML doesn't include vcpupin directives that basically does pinning. Instead only 4 is present in libvirt for flavor with PCPU resource. Nova version: Xena Sample flavor that's being used: +++ | Field | Value | +++ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | description| None | | disk | 50 | | id | 7d4455db-8d5e-4067-8c7d-2e0eb0d642f1 | | name | cpu-high-flavor_test | | os-flavor-access:is_public | True | | properties | resouces:PCPU='4', trait:CUSTOM_HIGHCPU='required' | | ram| 2048 | | rxtx_factor| 1.0 | | swap | | | vcpus | 4 | +++ Generated XML will be attached. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1966452/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1966452] [NEW] Using resources:PCPU property does not result in CPU pinning
Public bug reported: When instance is being created from flavor with `--property resources:PCPU=4`, generated XML doesn't include vcpupin directives that basically does pinning. Instead only 4 is present in libvirt for flavor with PCPU resource. Nova version: Xena Sample flavor that's being used: +++ | Field | Value | +++ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | description| None | | disk | 50 | | id | 7d4455db-8d5e-4067-8c7d-2e0eb0d642f1 | | name | cpu-high-flavor_test | | os-flavor-access:is_public | True | | properties | resouces:PCPU='4', trait:CUSTOM_HIGHCPU='required' | | ram| 2048 | | rxtx_factor| 1.0 | | swap | | | vcpus | 4 | +++ Generated XML will be attached. ** Affects: nova Importance: High Status: New ** Attachment added: "pcpu_domain.xml" https://bugs.launchpad.net/bugs/1966452/+attachment/5572936/+files/pcpu_domain.xml -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1966452 Title: Using resources:PCPU property does not result in CPU pinning Status in OpenStack Compute (nova): New Bug description: When instance is being created from flavor with `--property resources:PCPU=4`, generated XML doesn't include vcpupin directives that basically does pinning. Instead only 4 is present in libvirt for flavor with PCPU resource. Nova version: Xena Sample flavor that's being used: +++ | Field | Value | +++ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | description| None | | disk | 50 | | id | 7d4455db-8d5e-4067-8c7d-2e0eb0d642f1 | | name | cpu-high-flavor_test | | os-flavor-access:is_public | True | | properties | resouces:PCPU='4', trait:CUSTOM_HIGHCPU='required' | | ram| 2048 | | rxtx_factor| 1.0 | | swap | | | vcpus | 4 | +++ Generated XML will be attached. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1966452/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1945646] [NEW] Nova fails to live migrate instance with upper-case port MAC
Public bug reported:
Description
===
When neutron port has MAC address defined in upper case and libvirt stores MAC
in XML in lower case, migration is failed with KeyError:
```
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: 2021-09-30 10:31:38.028
3054313 ERROR nova.virt.libvirt.driver
[req-911a4b70-5448-48a1-afa4-1bbd0b38737b - - - - -] [instance: 75f7
9d85-6505-486c-bc34-e78fd6350a77] Live Migration failure: '00:50:56:af:e1:73'
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: Traceback (most recent
call last):
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/hubs/hub.py",
line 461, in fire_timers
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: timer()
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/hubs/timer.py",
line 59, in __call__
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: cb(*args, **kw)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/event.py",
line 175, in _do_send
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:
waiter.switch(result)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/greenthread.py",
line 221, in main
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: result =
function(*args, **kwargs)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/utils.py", line
661, in context_wrapper
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: return func(*args,
**kwargs)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 9196, in _live_migration_operation
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: LOG.error("Live
Migration failure: %s", e, instance=instance)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: self.force_reraise()
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]:
six.reraise(self.type_, self.value, self.tb)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/six.py", line 703, in
reraise
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: raise value
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 9152, in _live_migration_operation
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: new_xml_str =
libvirt_migrate.get_updated_guest_xml(
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/migration.py",
line 65, in get_updated_guest_xml
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: xml_doc =
_update_vif_xml(xml_doc, migrate_data, get_vif_config)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File
"/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/migration.py",
line 355, in _update_vif_xml
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: migrate_vif =
migrate_vif_by_mac[mac_addr]
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: KeyError:
'00:50:56:af:e1:73'
```
Environment
===
Ubuntu 20.04
Libvirt 6.0.0-0ubuntu8.14
Nova 22.2.3.dev2 (sha 4ce01d6c49f81b6b2438549b01a89ea1b5956320)
Neutron with OpenVSwitch
** Affects: nova
Importance: Undecided
Status: In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1945646
Title:
Nova fails to live migrate instance with upper-case port MAC
Status in OpenStack Compute (nova):
In Progress
Bug description:
Description
===
When neutron port has MAC address defined in upper case and libvirt stores
MAC in XML in lower case, migration is failed with KeyError:
```
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: 2021-09-30
10:31:38.028 3054313 ERROR nova.virt.libvirt.driver
[req-911a4b70-5448-48a1-afa4-1bbd0b38737b - - - - -] [instance: 75f7
9d85-6505-486c-bc34-e78fd6350a77] Live Migration failure: '00:50:56:af:e1:73'
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: Traceback (most
recent call last):
Sep 30 10:31:38
[Yahoo-eng-team] [Bug 1932326] [NEW] Compute process cast against instance it does not own
Public bug reported: Description === In case, when some request (ie instance shutdown) is sent to the compute that is not available, request continue hanging in queue. In the meanwhile, instance can be evacuated from this host to the new available compute. However once original compute host becomes available again it process all messages waiting for it in queue without extra verification, which results in super delayed request being processed (ie instance goes shut down on new compute) even if this host doesn't own resource anymore. Steps to reproduce == 1. compute1 goes down 2. Send shutdown request to the VM that is hosted on compute1 3. Evacuate VM (it would require state reset ATM - related bug #1932126) to compute2 4. boot compute1 5. compute1 will make instance that is currently working on compute2 to shutdown Expected result === Compute1 that does not own resource anymore should not be able to influence it Environment === SHA: c7d9d6d9dd25e21ec76ceea294cdf1690686a086 (Victoria) Hypervisor: KVM Networking: ovs Storage: Ceph ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1932326 Title: Compute process cast against instance it does not own Status in OpenStack Compute (nova): New Bug description: Description === In case, when some request (ie instance shutdown) is sent to the compute that is not available, request continue hanging in queue. In the meanwhile, instance can be evacuated from this host to the new available compute. However once original compute host becomes available again it process all messages waiting for it in queue without extra verification, which results in super delayed request being processed (ie instance goes shut down on new compute) even if this host doesn't own resource anymore. Steps to reproduce == 1. compute1 goes down 2. Send shutdown request to the VM that is hosted on compute1 3. Evacuate VM (it would require state reset ATM - related bug #1932126) to compute2 4. boot compute1 5. compute1 will make instance that is currently working on compute2 to shutdown Expected result === Compute1 that does not own resource anymore should not be able to influence it Environment === SHA: c7d9d6d9dd25e21ec76ceea294cdf1690686a086 (Victoria) Hypervisor: KVM Networking: ovs Storage: Ceph To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1932326/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1927521] [NEW] LDAP implementation is broken for py2.7 on T
Public bug reported: Issue descripton Train is supposed to be the last release that support python 2.7. In the meanwhile, https://review.opendev.org/c/openstack/keystone/+/755736 has been backported back to Train. It implements usage of the weakref.finalize which is valid attribute only since Python 3.4. That means, that Train py2.7 is broken. Expected behaviour -- LDAP can be used on Train with py27 Actual behaviour http://paste.openstack.org/show/804999/ Related-Bug: #1896125 ** Affects: keystone Importance: Undecided Status: New ** Summary changed: - LDAP implementation is broken for py2.7 + LDAP implementation is broken for py2.7 on T -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1927521 Title: LDAP implementation is broken for py2.7 on T Status in OpenStack Identity (keystone): New Bug description: Issue descripton Train is supposed to be the last release that support python 2.7. In the meanwhile, https://review.opendev.org/c/openstack/keystone/+/755736 has been backported back to Train. It implements usage of the weakref.finalize which is valid attribute only since Python 3.4. That means, that Train py2.7 is broken. Expected behaviour -- LDAP can be used on Train with py27 Actual behaviour http://paste.openstack.org/show/804999/ Related-Bug: #1896125 To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1927521/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1923899] [NEW] Nova create duplicate cells when templates are used
Public bug reported:
Description
===
When you use templates for cell creation and re-run create_cell command,
duplicate cell will be created
This happens because logic is broken here for templates:
https://opendev.org/openstack/nova/src/commit/9e5feb051c15166988e87b10132927135bd56884/nova/cmd/manage.py#L623-L624
As cell.database_connection and cell.transport_url are already resolved
into urls, while transport_url and database_connection are still
templates.
Steps to reproduce
==
Run 2 times: nova-manage cell_v2 create_cell --name cell1 --transport-
url {scheme}://{username}:{password}@{hostname}//{path}?{query}
--database_connection
{scheme}://{username}:{password}@{hostname}/{path}?{query} --verbose
This will result in creating 2 independent cells:
root@aio1:~# /openstack/venvs/nova-22.0.0.0rc2.dev149/bin/nova-manage cell_v2
create_cell --name cell1 --transport-url
{scheme}://{username}:{password}@{hostname}//{path}?{query}
--database_connection
{scheme}://{username}:{password}@{hostname}/{path}?{query} --verbose
046c88f0-097e-4a7c-9eee-45654355ff56
root@aio1:~# /openstack/venvs/nova-22.0.0.0rc2.dev149/bin/nova-manage cell_v2
create_cell --name cell1 --transport-url
{scheme}://{username}:{password}@{hostname}//{path}?{query}
--database_connection
{scheme}://{username}:{password}@{hostname}/{path}?{query} --verbose
b5f1672e-36c7-4d17-90b3-7433842cbbe0
root@aio1:~# /openstack/venvs/nova-22.0.0.0rc2.dev149/bin/nova-manage cell_v2
create_cell --name cell1 --transport-url
{scheme}://{username}:{password}@{hostname}//{path}?{query}
--database_connection
{scheme}://{username}:{password}@{hostname}/{path}?{query} --verbose
cf2efc1c-e091-4c93-96d4-84b3ecfa4647
root@aio1:~#
Expected result
===
Following error should be raised along with command exit code 2:
The specified transport_url and/or database_connection combination already
exists for another cell with uuid
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1923899
Title:
Nova create duplicate cells when templates are used
Status in OpenStack Compute (nova):
New
Bug description:
Description
===
When you use templates for cell creation and re-run create_cell
command, duplicate cell will be created
This happens because logic is broken here for templates:
https://opendev.org/openstack/nova/src/commit/9e5feb051c15166988e87b10132927135bd56884/nova/cmd/manage.py#L623-L624
As cell.database_connection and cell.transport_url are already
resolved into urls, while transport_url and database_connection are
still templates.
Steps to reproduce
==
Run 2 times: nova-manage cell_v2 create_cell --name cell1
--transport-url
{scheme}://{username}:{password}@{hostname}//{path}?{query}
--database_connection
{scheme}://{username}:{password}@{hostname}/{path}?{query} --verbose
This will result in creating 2 independent cells:
root@aio1:~# /openstack/venvs/nova-22.0.0.0rc2.dev149/bin/nova-manage cell_v2
create_cell --name cell1 --transport-url
{scheme}://{username}:{password}@{hostname}//{path}?{query}
--database_connection
{scheme}://{username}:{password}@{hostname}/{path}?{query} --verbose
046c88f0-097e-4a7c-9eee-45654355ff56
root@aio1:~# /openstack/venvs/nova-22.0.0.0rc2.dev149/bin/nova-manage cell_v2
create_cell --name cell1 --transport-url
{scheme}://{username}:{password}@{hostname}//{path}?{query}
--database_connection
{scheme}://{username}:{password}@{hostname}/{path}?{query} --verbose
b5f1672e-36c7-4d17-90b3-7433842cbbe0
root@aio1:~# /openstack/venvs/nova-22.0.0.0rc2.dev149/bin/nova-manage cell_v2
create_cell --name cell1 --transport-url
{scheme}://{username}:{password}@{hostname}//{path}?{query}
--database_connection
{scheme}://{username}:{password}@{hostname}/{path}?{query} --verbose
cf2efc1c-e091-4c93-96d4-84b3ecfa4647
root@aio1:~#
Expected result
===
Following error should be raised along with command exit code 2:
The specified transport_url and/or database_connection combination already
exists for another cell with uuid
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1923899/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1846198] Re: packet loss during active L3 HA agent restart
Fix has been released for Victoria ** Changed in: openstack-ansible Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1846198 Title: packet loss during active L3 HA agent restart Status in neutron: Invalid Status in openstack-ansible: Fix Released Bug description: Deployment: Openstack-ansible 19.0.3(Stein) with two dedicated network nodes(is_metal=True) + linuxbridge + vxlan. Ubuntu 16.04.6 4.15.0-62-generic neutron l3-agent-list-hosting-router R1 +--+---++---+--+ | id | host | admin_state_up | alive | ha_state | +--+---++---+--+ | 1b3b1b5d-08e7-48a1-ab8d-256d94099fb6 | test-network2 | True | :-) | standby | | fa402ada-7716-4ad4-a004-7f8114fb1edf | test-network1 | True | :-) | active | +--+---++---+--+ How to reproduce: Restart the active l3 agent. (systemctl restart neutron-l3-agent.service) test-network1 server side events: systemctl restart neutron-l3-agent: @02:58:56.135635630 ip monitor terminated (kill -9) @02:58:56.208922038 vip ips removed @02:58:56.268074480 keepalived terminated @02:58:57.318596743 l3-agent terminated @02:59:07.504366398 keepalived-state-change terminated @03:01:07.735281710 test-network1 journal: @02:58:56 test-network1 systemd[1]: Stopping neutron-l3-agent service... @02:58:56 test-network1 Keepalived_vrrp[24400]: VRRP_Instance(VR_217) sent 0 priority @02:58:56 test-network1 Keepalived_vrrp[24400]: VRRP_Instance(VR_217) removing protocol Virtual Routes @02:58:56 test-network1 Keepalived_vrrp[24400]: VRRP_Instance(VR_217) removing protocol VIPs. @02:58:56 test-network1 Keepalived_vrrp[24400]: VRRP_Instance(VR_217) removing protocol E-VIPs. @02:58:56 test-network1 Keepalived[24394]: Stopping @02:58:56 test-network1 neutron-keepalived-state-change[24278]: 2019-10-01 02:58:56.193 24278 DEBUG neutron.agent.linux.utils [-] enax_custom_log: pid: 24283, signal: 9 kill_process /openstack/venvs/neutron-19.0.4.dev1/lib/python2.7/site-packages/neutron/agent/linux/utils.py:243 @02:58:56 test-network1 audit[24089]: USER_END pid=24089 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_close acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success' @02:58:56 test-network1 sudo[24089]: pam_unix(sudo:session): session closed for user root @02:58:56 test-network1 audit[24089]: CRED_DISP pid=24089 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success' @02:58:57 test-network1 Keepalived_vrrp[24400]: Stopped @02:58:57 test-network1 Keepalived[24394]: Stopped Keepalived v1.3.9 (10/21,2017) TCPDUMP qrouter-24010932-a0a4-4454-9539-27c1535c5ed8 ha-57528491-1b: @02:58:53.130735 IP 169.254.195.168 > 224.0.0.18: VRRPv2, Advertisement, vrid 217, prio 50, authtype simple, intvl 2s, length 20 @02:58:55.131926 IP 169.254.195.168 > 224.0.0.18: VRRPv2, Advertisement, vrid 217, prio 50, authtype simple, intvl 2s, length 20 @02:58:56.188558 IP 169.254.195.168 > 224.0.0.18: VRRPv2, Advertisement, vrid 217, prio 0, authtype simple, intvl 2s, length 20 @02:58:56.215889 IP 169.254.195.168 > 224.0.0.22: igmp v3 report, 1 group record(s) @02:58:56.539804 IP 169.254.195.168 > 224.0.0.22: igmp v3 report, 1 group record(s) @02:58:56.995386 IP 169.254.194.242 > 224.0.0.18: VRRPv2, Advertisement, vrid 217, prio 50, authtype simple, intvl 2s, length 20 @02:58:58.998565 ARP, Request who-has 169.254.0.217 (ff:ff:ff:ff:ff:ff) tell 169.254.0.217, length 28 @02:58:59.000138 ARP, Request who-has 169.254.0.217 (ff:ff:ff:ff:ff:ff) tell 169.254.0.217, length 28 @02:58:59.001063 ARP, Request who-has 169.254.0.217 (ff:ff:ff:ff:ff:ff) tell 169.254.0.217, length 28 @02:58:59.002173 ARP, Request who-has 169.254.0.217 (ff:ff:ff:ff:ff:ff) tell 169.254.0.217, length 28 @02:58:59.003018 ARP, Request who-has 169.254.0.217 (ff:ff:ff:ff:ff:ff) tell 169.254.0.217, length 28 @02:58:59.003860 IP 169.254.194.242 > 224.0.0.18: VRRPv2, Advertisement, vrid 217, prio 50, authtype simple, intvl 2s, length 20 @02:59:01.004772 IP 169.254.194.242 > 224.0.0.18: VRRPv2, Advertisement, vrid 217, prio 50, authtype simple, intvl 2s, length 20 After l3-agent restart neutron l3-agent-list-hosting-router R1 +--+---++---+--+ | id | host | admin_state_up | alive |
[Yahoo-eng-team] [Bug 1718356] Re: Include default config files in python wheel
** Changed in: openstack-ansible Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1718356 Title: Include default config files in python wheel Status in Barbican: Fix Released Status in Cinder: Fix Released Status in Cyborg: Fix Released Status in Designate: Fix Released Status in Fuxi: New Status in Glance: Fix Released Status in OpenStack Heat: In Progress Status in Ironic: Fix Released Status in Karbor: Fix Released Status in OpenStack Identity (keystone): Fix Released Status in kuryr-libnetwork: New Status in Magnum: In Progress Status in neutron: Fix Released Status in OpenStack Compute (nova): In Progress Status in octavia: Invalid Status in openstack-ansible: Fix Released Status in Sahara: Fix Released Status in OpenStack DBaaS (Trove): Fix Released Status in Zun: Fix Released Bug description: The projects which deploy OpenStack from source or using python wheels currently have to either carry templates for api-paste, policy and rootwrap files or need to source them from git during deployment. This results in some rather complex mechanisms which could be radically simplified by simply ensuring that all the same files are included in the built wheel. A precedence for this has already been set in neutron [1], glance [2] and designate [3] through the use of the data_files option in the files section of setup.cfg. [1] https://github.com/openstack/neutron/blob/d3c393ff6b5fbd0bdaabc8ba678d755ebfba08f7/setup.cfg#L24-L39 [2] https://github.com/openstack/glance/blob/02cd5cba70a8465a951cb813a573d390887174b7/setup.cfg#L20-L21 [3] https://github.com/openstack/designate/blob/25eb143db04554d65efe2e5d60ad3afa6b51d73a/setup.cfg#L30-L37 This bug will be used for a cross-project implementation of patches to normalise the implementation across the OpenStack projects. Hopefully the result will be a consistent implementation across all the major projects. A mailing list thread corresponding to this standard setting was begun: http://lists.openstack.org/pipermail/openstack-dev/2017-September/122794.html To manage notifications about this bug go to: https://bugs.launchpad.net/barbican/+bug/1718356/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1632383] Re: The current Horizon settings indicate no valid image creation methods are available
This has been fixed for a while and IMAGES_ALLOW_LOCATION is present in local_settings.py ** Changed in: openstack-ansible Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1632383 Title: The current Horizon settings indicate no valid image creation methods are available Status in OpenStack Dashboard (Horizon): Fix Released Status in openstack-ansible: Fix Released Bug description: This is an all new install on CentOS 7. The error message is: "The current Horizon settings indicate no valid image creation methods are available. Providing an image location and/or uploading from the local file system must be allowed to support image creation." This link gives you some additional information about the file the message is associated with. https://github.com/openstack/horizon/blob/master/openstack_dashboard/dashboards/project/images/images/forms.py I am mostly new to OpenStack, so any initial help is much appreciated. Thanks. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1632383/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1853822] [NEW] Display only supported by glance disk_format
Public bug reported:
Glance can return list of supported disk formats in schema [1]. Now
disk_formats are configured staticly with OPENSTACK_IMAGE_BACKEND
variable in settings and do not take into account really supported
formats. So administrator needs to configure both horizon and glance.
I'd offer to add support into horizon to retrieve list of enabled
formats. This list can be retrieved in the following way:
>>> from glanceclient.v2 import client
>>> from openstack import connect
>>> conn = connect()
>>> glanceclient = client.Client("http://172.29.236.100:9292/v2/;,
>>> token=conn.auth_token, insecure=True)
>>> glanceclient.schemas.get('image').raw()['properties']['disk_format']['enum']
[None, 'raw']
>>>
[1] https://docs.openstack.org/api-ref/image/v2/index.html#image-schemas
** Affects: horizon
Importance: Undecided
Assignee: Dmitriy Rabotyagov (noonedeadpunk)
Status: In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1853822
Title:
Display only supported by glance disk_format
Status in OpenStack Dashboard (Horizon):
In Progress
Bug description:
Glance can return list of supported disk formats in schema [1]. Now
disk_formats are configured staticly with OPENSTACK_IMAGE_BACKEND
variable in settings and do not take into account really supported
formats. So administrator needs to configure both horizon and glance.
I'd offer to add support into horizon to retrieve list of enabled
formats. This list can be retrieved in the following way:
>>> from glanceclient.v2 import client
>>> from openstack import connect
>>> conn = connect()
>>> glanceclient = client.Client("http://172.29.236.100:9292/v2/;,
token=conn.auth_token, insecure=True)
>>>
glanceclient.schemas.get('image').raw()['properties']['disk_format']['enum']
[None, 'raw']
>>>
[1] https://docs.openstack.org/api-ref/image/v2/index.html#image-
schemas
To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1853822/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1848201] [NEW] [neutron-vpnaas] Neutron installed inside venv makes VPNaaS broken
Public bug reported: As location of NS_WRAPPER [1] is not aboslute and relies on $PATH in the situation when neutron and vpnaas is installed inside virtualenv, NS_WRAPPER won't be able to launch as it won't be found in $PATH which will cause a failure: 2019-10-08 17:37:11.619 17205 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-vpn-netns-wrapper" failed: No such file or directory 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec [-] Failed to enable vpn process on router 19514b2a-95bc-499f-9590-a5014ca04e7f: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-vpn-netns-wrapper" failed: No such file or directory 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec Traceback (most recent call last): 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/openstack/venvs/neutron-19.0.0.0b2.dev70/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 336, in enable 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec self.ensure_configs() 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/openstack/venvs/neutron-19.0.0.0b2.dev70/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py", line 91, in ensure_configs 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec self._ipsec_execute(['_stackmanager', 'start']) 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/openstack/venvs/neutron-19.0.0.0b2.dev70/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py", line 52, in _ipsec_execute 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec extra_ok_codes=extra_ok_codes) 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/openstack/venvs/neutron-19.0.0.0b2.dev70/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 788, in execute 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec run_as_root=run_as_root) 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/openstack/venvs/neutron-19.0.0.0b2.dev70/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 147, in execute 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec returncode=returncode) 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-vpn-netns-wrapper" failed: No such file or directory 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec [1] https://opendev.org/openstack/neutron- vpnaas/src/branch/master/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py#L22 ** Affects: neutron Importance: Undecided Assignee: Dmitriy Rabotyagov (noonedeadpunk) Status: In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1848201 Title: [neutron-vpnaas] Neutron installed inside venv makes VPNaaS broken Status in neutron: In Progress Bug description: As location of NS_WRAPPER [1] is not aboslute and relies on $PATH in the situation when neutron and vpnaas is installed inside virtualenv, NS_WRAPPER won't be able to launch as it won't be found in $PATH which will cause a failure: 2019-10-08 17:37:11.619 17205 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-vpn-netns- wrapper" failed: No such file or directory 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec [-] Failed to enable vpn process on router 19514b2a-95bc-499f-9590-a5014ca04e7f: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: exec of "neutron-vpn-netns-wrapper" failed: No such file or directory 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec Traceback (most recent call last): 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/openstack/venvs/neutron-19.0.0.0b2.dev70/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 336, in enable 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec self.ensure_configs() 2019-10-08 17:37:11.620 17205 ERROR neutron_vpnaas.services.vpn.devi
[Yahoo-eng-team] [Bug 1839360] [NEW] nova-compute fails with DBNotAllowed error
Public bug reported: Description === During routine operations or things like running regular tempest checks nova-compute tries to reach database and fails with DBNotAllowed error: https://logs.opendev.org/33/660333/10/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/97d8bc3/logs/host/nova-compute.service.journal-23-20-40.log.txt.gz#_Aug_06_22_51_25 Steps to reproduce == This might be reproduced with deploying all nova components (like api, scheduler, conductor, compute) on the same host (OSA all-in-one deployment). During such setup single configuration file is used (nova.conf). As a solution it's possible to log more helpful information why this happens and add some description into docs. ** Affects: nova Importance: Medium Assignee: Matt Riedemann (mriedem) Status: In Progress ** Affects: nova/rocky Importance: Medium Status: Confirmed ** Affects: nova/stein Importance: Medium Status: Confirmed ** Tags: docs serviceability -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1839360 Title: nova-compute fails with DBNotAllowed error Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: Confirmed Bug description: Description === During routine operations or things like running regular tempest checks nova-compute tries to reach database and fails with DBNotAllowed error: https://logs.opendev.org/33/660333/10/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/97d8bc3/logs/host/nova-compute.service.journal-23-20-40.log.txt.gz#_Aug_06_22_51_25 Steps to reproduce == This might be reproduced with deploying all nova components (like api, scheduler, conductor, compute) on the same host (OSA all-in-one deployment). During such setup single configuration file is used (nova.conf). As a solution it's possible to log more helpful information why this happens and add some description into docs. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1839360/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
