[Yahoo-eng-team] [Bug 1971760] [NEW] nova-compute leaks green threads
Public bug reported: At the moment, if the cloud sustain a large number of VIF plugging timeouts, it will lead into a ton of leaked green threads which can cause the nova-compute process to stop reporting/responding. The tracebacks that would occur would be: === 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] Traceback (most recent call last): 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7230, in _create_guest_with_network 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] guest = self._create_guest( 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] next(self.gen) 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 479, in wait_for_instance_event 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] actual_event = event.wait() 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] File "/var/lib/openstack/lib/python3.8/site-packages/eventlet/event.py", line 125, in wait 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] result = hub.switch() 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] File "/var/lib/openstack/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] return self.greenlet.switch() 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] eventlet.timeout.Timeout: 300 seconds 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] During handling of the above exception, another exception occurred: 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] Traceback (most recent call last): 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 2409, in _build_and_run_instance 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] self.driver.spawn(context, instance, image_meta, 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4193, in spawn 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] self._create_guest_with_network( 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7256, in _create_guest_with_network 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] raise exception.VirtualInterfaceCreateException() 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed 2022-04-17 00:21:28.651 877893 ERROR nova.compute.manager [instance: 0c0d2422-781c-4bd2-b6bd-e5e3c94b602b] === Eventually, with enough of these, the nova-compute process would hang. The output of GMR shows nearly 6094 threads, with around 3038 of them having the traceback below: === --Green Thread-- /var/lib/openstack/lib/python3.8/site-packages/eventlet/hubs/hub.py:355 in run `self.fire_timers(self.clock())` /var/lib/openstack/lib/python3.8/site-packages/eventlet/hubs/hub.py:476 in fire_timers `timer()` /var/lib/openstack/lib/python3.8/site-packages/eventlet/hubs/timer.py:59 in __call__ `cb(*args, **kw)` /var/lib/openstack/lib/python3.8/site-packages/eventlet/hubs/__init__.py:151 in _timeout `current.throw(exc)` === In addition, 3039 of
[Yahoo-eng-team] [Bug 1969347] Re: opesntack-dashboard doesn't show pie charts, doesn't show flavor in instance description, can't navigate to another section. Can't be unplugged. None of the dropdown
You have a wrong version of angular installed. Horizon requires 1.5.8.0
** Changed in: horizon
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1969347
Title:
opesntack-dashboard doesn't show pie charts, doesn't show flavor in
instance description, can't navigate to another section. Can't be
unplugged. None of the dropdowns work
Status in OpenStack Dashboard (Horizon):
Invalid
Bug description:
Good afternoon, I do not see functionality openstack wallaby
python-openstackclient-lang-5.5.0-1.noarch
python3-openstacksdk-0.55.0-1.noarch
openstack-keystone-19.0.0-2.noarch
python3-openstackclient-5.5.0-1.noarch
openstack-glance-22.0.0-2.noarch
openstack-placement-common-5.0.1-2.noarch
openstack-placement-api-5.0.1-2.noarch
openstack-neutron-common-18.1.1-3.noarch
openstack-neutron-18.1.1-3.noarch
openstack-neutron-ml2-18.1.1-3.noarch
openstack-neutron-openvswitch-18.1.1-3.noarch
openstack-dashboard-theme-19.2.1-6.noarch
openstack-dashboard-19.2.1-6.noarch
openstack-nova-common-23.2.0-1.noarch
openstack-nova-compute-23.2.0-1.noarch
openstack-nova-migration-23.2.0-1.noarch
openstack-nova-api-23.2.0-1.noarch
openstack-nova-conductor-23.2.0-1.noarch
openstack-nova-novncproxy-23.2.0-1.noarch
openstack-nova-scheduler-23.2.0-1.noarch
openstack-nova-23.2.0-1.noarch
[root@node1 share(keystone)]# python --version
Python 3.8.11
[root@node1 share(keystone)]# cat /etc/httpd/conf.d/op
openstack-dashboard.conf openstack-dashboard.conf.old
[root@node1 share(keystone)]# cat /etc/httpd/conf.d/openstack-dashboard.conf
WSGIDaemonProcess dashboard
WSGIProcessGroup dashboard
WSGISocketPrefix run/wsgi
WSGIApplicationGroup %{GLOBAL}
WSGIScriptAlias /dashboard
/usr/share/openstack-dashboard/openstack_dashboard/wsgi.py
Alias /dashboard/static /usr/share/openstack-dashboard/static
Options All
AllowOverride All
Require all granted
Options All
AllowOverride All
Require all granted
To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1969347/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1971592] Re: System scoped token: System->system information->compute service list results in 403
The compute service does not support system scope tokens fully yet.
** Changed in: horizon
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1971592
Title:
System scoped token: System->system information->compute service list
results in 403
Status in OpenStack Dashboard (Horizon):
Invalid
Bug description:
Hi all,
We have started to analyze Openstack Yoga a bit and there, one of the major
new feature is the activation of scope based token for regular use in nova.
While after some long lasting back and forth in configuring our role
assignments and policies we could make it work on one of our test environments
(Ubuntu) via Openstack SDK, however, we are still struggling with some system
scoped API calls to nova from horizon.
We have an admin user for the domain 'Default' who has set the role ‚admin'
for 'system all':
+---+---+---+-+++---+
| Role | User | Group | Project | Domain | System | Inherited |
+---+---+---+-+++---+
| admin | admin@Default | | || all| False |
+---+---+---+-+++---+
To make login in horizon successful, the user admin is also admin for the
domain 'default' and for the project 'admin' in the domain 'default'.
We have configured in local_settings.py:
SYSTEM_SCOPE_SERVICES = ['compute', 'image', 'volume', 'network‘]
(Note: this config line has been reverse engineered from horizon
source code as the syntax is nowhere possible to be found in the docs
yet … - so: not sure if it is correct)
Policy files are identical for horizon as for the services. Policy for this
api request is
"os_compute_api:os-services:list": "rule:context_is_admin"
For the user admin, we then get an additional field in the
domain/project top line menu adding a ‚system scope‘ switch (this is
what we understand how it should look like) and - when switching to
system scope - also a 'system' menu in the sidebar becomes visible
(also as expected).
If we then go to System->Systeminformation to see the nova service
list, we get an error ‚Unable to get nova services list‘, given reason
is an error: 'Policy doesn’t allow os_compute_api:os-services:list to
be performed (HTTP 403)‘. Informations for network and volume services
are shown normally (here scoped tokens are not activated yet).
apache2 error log in debug mode results for this call in:
[Wed May 04 13:40:45.314012 2022] [wsgi:error] [pid 3007771:tid
14063777288] [remote 192.168.1.70:59546]
DEBUG:keystoneauth.identity.v3.base:Making authentication request to
https://controller:5000/v3/auth/tokens
[Wed May 04 13:40:45.396814 2022] [wsgi:error] [pid 3007771:tid
14063777288] [remote 192.168.1.70:59546]
DEBUG:urllib3.connectionpool:https://controller:5000 "POST /v3/auth/tokens
HTTP/1.1" 201 12010
[Wed May 04 13:40:45.397786 2022] [wsgi:error] [pid 3007771:tid
14063777288] [remote 192.168.1.70:59546]
DEBUG:keystoneauth.identity.v3.base:{"token": {"methods": ["password",
"token"], "user": {"domain": {"id": "default", "name": "Default"}, "id":
"a837078533b64bac8be7f9fb0c57ead6", "name": "admin", "password_expires_at":
null}, "audit_ids": ["EAXc26zRR4KUEwDSLwKe4w", "iUCwAVveTAay4FfkHwZ1sA"],
"expires_at": "2022-05-04T12:40:30.00Z", "issued_at":
"2022-05-04T11:40:45.00Z", "project": {"domain": {"id": "default", "name":
"Default"}, "id": "6f79eb4041a142dfaffb5d1f8bff906c", "name": "admin"},
"is_domain": false, "roles": [{"id": "9109437956b544cebd1f5dfc87def5bd",
"name": "admin"}, {"id": "c98267f0aa4144d8b7e89d9b40b46c0e", "name": "member"},
{"id": "eea735edc3e14222bae6703c2c672c02", "name": "reader"}], "catalog":
[{"endpoints": [{"id": "7342dd7ff0ae438ba50ac669538feb4f", "interface": "
(...)
[Wed May 04 13:40:45.398554 2022] [wsgi:error] [pid 3007771:tid
14063777288] [remote 192.168.1.70:59546]
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:8774
[Wed May 04 13:40:45.726949 2022] [wsgi:error] [pid 3007771:tid
14063777288] [remote 192.168.1.70:59546]
DEBUG:urllib3.connectionpool:http://controller:8774 "GET /v2.1/os-services
HTTP/1.1" 403 112
[Wed May 04 13:40:45.727379 2022] [wsgi:error] [pid 3007771:tid
14063777288] [remote 192.168.1.70:59546] Recoverable error: Policy doesn't
allow os_compute_api:os-services:list to be performed. (HTTP 403) (Request-ID:
req-592af610-2daa-4318-8a2a-2f7fa89f83c6)
Logs indicate that horizon is using still a project-scoped token and
not a system-scoped one for these requests although ’system scope’ is
active.
Putting the same request from Openstack SDK with the same user 'admin'
results in
$ openstack compute
[Yahoo-eng-team] [Bug 1969496] Re: booting with PCI device fails: Attempt to consume PCI device xxx from empty pool
Reviewed: https://review.opendev.org/c/openstack/nova/+/838555 Committed: https://opendev.org/openstack/nova/commit/3af2ecc13fa9334de8418accaed4fffefefb41da Submitter: "Zuul (22348)" Branch:master commit 3af2ecc13fa9334de8418accaed4fffefefb41da Author: Balazs Gibizer Date: Tue Apr 19 18:36:50 2022 +0200 Allow claiming PCI PF if child VF is unavailable As If9ab424cc7375a1f0d41b03f01c4a823216b3eb8 stated there is a way for the pci_device table to become inconsistent. Parent PF can be in 'available' state while children VFs are still in 'unavailable' state. In this situation the PF is schedulable but the PCI claim will fail when try to mark the dependent VFs unavailable. This patch changes the PCI claim logic to allow claiming the parent PF in the inconsistent situation as we assume that it is safe to do so. This claim also fixed the inconsistency so that when the parent PF is freed the children VFs become available again. Closes-Bug: #1969496 Change-Id: I575ce06bcc913add7db0849f85728371da2032fc ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1969496 Title: booting with PCI device fails: Attempt to consume PCI device xxx from empty pool Status in OpenStack Compute (nova): Fix Released Bug description: We saw in the field that the pci_devices table can end up in inconsistent state after a compute node HW failure and re-deployment. There could be dependent devices where the parent PF is in available state while the children VFs are in unavailable state. (Before the HW fault the PF was allocated hence the VFs was marked unavailable). In this state this PF is still schedulable but during the PCI claim the handling of dependent devices in the PCI tracker will fail with the error: "Attempt to consume PCI device XXX from empty pool". The reason of the failure is that when the PF is claimed, all the children VFs are marked unavailable. But if the VF is already unavailable such step fails. There is no reproducer found so far that generates the inconsistent state. (We tried whitelist reconfiguration, evacuation, VM delete while the compute was down) But recovering from the inconsistency should be possible. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1969496/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1971697] [NEW] [Neutron][OpenContrail]Trunk ports doesn't have standard attributes.
Public bug reported: Environment: Openstack Victoria, OpenContrail (TungstenFabric) R2011. Problem: Trunk ports doesn't have standard attributes such as description, timestamp. I have an environment where core plugin is OpenContrail. OpenContrail has tf-neutron-plugin for proper work with neutron. There is TrunkPlugin, that proxies all trunk-related request to the OpenContrail backend. Here is the link for this plugin. https://gerrit.tungsten.io/r/gitweb?p=tungstenfabric/tf-neutron-plugin.git;a=blob;f=neutron_plugin_contrail/plugins/opencontrail/services/trunk/plugin.py;h=35fc3310911143fd3b4cf8997c23d0358d652dba;hb=refs/heads/master According to the openstack documentation: Resources that inherit from the HasStandardAttributes DB class can automatically have the extensions written for standard attributes (e.g. timestamps, revision number, etc) extend their resources by defining the ‘api_collections’ on their model. These are used by extensions for standard attr resources to generate the extended resources map. As I understood, it works only for plugins, which using Openstack database. For example, openstack native trunk plugin has models.py file where we can see this inheritance. https://github.com/openstack/neutron/blob/master/neutron/services/trunk/models.py#L26 In case of OpenContrail+OpenStack Trunk plugin only redirect requests. What can I do to make Contrail Trunk plugin works in the same way? Thanks in advance. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1971697 Title: [Neutron][OpenContrail]Trunk ports doesn't have standard attributes. Status in neutron: New Bug description: Environment: Openstack Victoria, OpenContrail (TungstenFabric) R2011. Problem: Trunk ports doesn't have standard attributes such as description, timestamp. I have an environment where core plugin is OpenContrail. OpenContrail has tf-neutron-plugin for proper work with neutron. There is TrunkPlugin, that proxies all trunk-related request to the OpenContrail backend. Here is the link for this plugin. https://gerrit.tungsten.io/r/gitweb?p=tungstenfabric/tf-neutron-plugin.git;a=blob;f=neutron_plugin_contrail/plugins/opencontrail/services/trunk/plugin.py;h=35fc3310911143fd3b4cf8997c23d0358d652dba;hb=refs/heads/master According to the openstack documentation: Resources that inherit from the HasStandardAttributes DB class can automatically have the extensions written for standard attributes (e.g. timestamps, revision number, etc) extend their resources by defining the ‘api_collections’ on their model. These are used by extensions for standard attr resources to generate the extended resources map. As I understood, it works only for plugins, which using Openstack database. For example, openstack native trunk plugin has models.py file where we can see this inheritance. https://github.com/openstack/neutron/blob/master/neutron/services/trunk/models.py#L26 In case of OpenContrail+OpenStack Trunk plugin only redirect requests. What can I do to make Contrail Trunk plugin works in the same way? Thanks in advance. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1971697/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1970383] Re: Segment-aware scheduling fails for non-admin users
Reviewed: https://review.opendev.org/c/openstack/nova/+/839361 Committed: https://opendev.org/openstack/nova/commit/ee32934f34afd8e6df467361e9d71788cd36f6ee Submitter: "Zuul (22348)" Branch:master commit ee32934f34afd8e6df467361e9d71788cd36f6ee Author: Andrew Bonney Date: Tue Apr 26 11:35:38 2022 +0100 Fix segment-aware scheduling permissions error Resolves a bug encountered when setting the Nova scheduler to be aware of Neutron routed provider network segments, by using 'query_placement_for_routed_network_aggregates'. Non-admin users attempting to access the 'segment_id' attribute of a subnet caused a traceback, resulting in instance creation failure. This patch ensures the Neutron client is initialised with an administrative context no matter what the requesting user's permissions are. Change-Id: Ic0f25e4d2395560fc2b68f3b469e266ac59abaa2 Closes-Bug: #1970383 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1970383 Title: Segment-aware scheduling fails for non-admin users Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) wallaby series: New Status in OpenStack Compute (nova) xena series: New Status in OpenStack Compute (nova) yoga series: New Bug description: This is a follow-up to https://bugs.launchpad.net/nova/+bug/1967314 Having deployed the Nova scheduler configuration for routed provider networks as follows (Xena deployment @ 7df9379d6661233174d49fb7be8eda0828a5e5ca), this was found to resolve issues around scheduling of instances to appropriate hypervisors, but it appears to have surfaced a side effect. [scheduler] query_placement_for_routed_network_aggregates = True When the above configuration is enabled, creation of new instances for admin users works correctly, but for non-admin users against the same networks results in the following error: 285768 ERROR oslo_messaging.rpc.server [req-79ca3cb3-eb52-4755-bba1-4c840c8ae5fc c35a1473225f422c90a6f75b25188bf2 d96f0cd70c6a4adbbbcf993502b264dc - default default] Exception during message handling: K> 285768 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 285768 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 285768 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 285768 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner 285768 ERROR oslo_messaging.rpc.server return func(*args, **kwargs) 285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/manager.py", line 154, in select_destinations 285768 ERROR oslo_messaging.rpc.server request_filter.process_reqspec(context, spec_obj) 285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 387, in process_reqspec 285768 ERROR oslo_messaging.rpc.server filter(ctxt, request_spec) 285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 41, in wrapper 285768 ERROR oslo_messaging.rpc.server ran = fn(ctxt, request_spec) 285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 348, in routed_networks_filter 285768 ERROR oslo_messaging.rpc.server aggregates = utils.get_aggregates_for_routed_network( 285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/utils.py", line 1406, in get_aggregates_for_routed_network 285768 ERROR oslo_messaging.rpc.server segment_ids = network_api.get_segment_ids_for_network( 285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/network/neutron.py", line 3721, in get_segment_ids_for_network 285768 ERROR oslo_messaging.rpc.server return
[Yahoo-eng-team] [Bug 1971691] [NEW] Support application credentials as a source for EC2 auth
Public bug reported:
Unfortunately EC2 credentials are not secure enough. EC2 credentials are
not protected by limited roles, expiration time, access rules and ec2
secret part is visible via get/list API calls. Leaked EC2 credentials
imply a big security risk in terms of access, because EC2 creds token
has the same power as a regular user/pass auth.
EC2 AUTH is actively used by Swift S3 emulation (not limited only to
Swift, btw.) it would be nice to use application credentials as an auth
source in keystone internals and issue a limited access token. With all
features application credentials provide, EC2 can get a second wind.
An example of EC2 auth request with application credentials:
$ openstack application credential list
+--+---+--+-++
| ID | Name | Project ID
| Description | Expires At |
+--+---+--+-++
| 3defd466f04646d094a1ee4b6afc53e8 | test |
f8d450e9cb7b4f1cbf664401d5bf1d29 | None| 2219-02-13T12:12:12.00 |
+--+---+--+-++
POST http://keystone:8080/v3/ec2tokens
{
"credentials": {
"access": "3defd466f04646d094a1ee4b6afc53e8",
"body_hash": "***",
"headers": {
"Accept-Encoding": "identity",
"Authorization": "AWS4-HMAC-SHA256
Credential=3defd466f04646d094a1ee4b6afc53e8/20220505/RegionOne/s3/aws4_request,
SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=appCredSecretBasedSignature",
"Host": "keystone:8080",
"User-Agent": "aws-cli/1.18.69 Python/3.8.10 Linux/5.4.0-109-generic
botocore/1.16.19",
"X-Amz-Content-Sha256":
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"X-Amz-Date": "20220505T101354Z",
"X-Amz-SignedHeaders": "host;x-amz-content-sha256;x-amz-date"
},
"host": "",
"params": {},
"path": "/",
"signature": "appCredSecretBasedSignature",
"verb": "GET"
}
}
An example of EC2 auth token response with application credentials:
{
"token": {
"application_credential": {
"access_rules": [
{
"id": "9416a34e7f3b45ecb029063d8a239463",
"method": "GET",
"path": "/v1/secrets/e8f07eae-3a6b-4c3c-a847-f14f6e348d8f**",
"service": "key-manager"
}
],
"id": "3defd466f04646d094a1ee4b6afc53e8",
"name": "test",
"restricted": true
},
"audit_ids": [
"m6C3NgSiQmqQnrBRySYW2A"
],
"catalog": [...],
"expires_at": "2022-05-05T18:24:48.00Z",
"is_admin_project": false,
"is_domain": false,
"issued_at": "2022-05-05T10:24:48.00Z",
"methods": [
"application_credential"
],
"project": {
"domain": {
"id": "default",
"name": "Default"
},
"id": "f8d450e9cb7b4f1cbf664401d5bf1d29",
"name": "test"
},
"roles": [
{
"id": "a66c3a324bc24c0da7259faa03f2704d",
"name": "limited_role"
}
],
"user": {
"domain": {
"id": "default",
"name": "Default"
},
"id": "0c4cac95c039441d9b8bb509fe836110",
"name": "appCredOwner",
"password_expires_at": "2022-09-07T18:13:38.126030"
}
}
}
** Affects: keystone
Importance: Undecided
Status: New
** Tags: application credentials ec2
** Description changed:
Unfortunately EC2 credentials are not secure enough. EC2 credentials are
not protected by limited roles, expiration time, access rules and ec2
secret part is visible via get/list API calls. Leaked EC2 credentials
imply a big security risk in terms of access, because EC2 creds token
has the same power as a regular user/pass auth.
Hence EC2 AUTH is actively used by Swift S3 emulation (not limited only
to Swift, btw.) it would be nice t
[Yahoo-eng-team] [Bug 1971672] [NEW] [FT] Error in "test_virtual_port_host_update"
Public bug reported: Error executing neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovsdb_monitor.TestNBDbMonitorOverSsl.test_virtual_port_host_update Log: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_207/840448/2/check/neutron- functional-with-uwsgi/2074ad0/testr_results.html Snippet: https://paste.opendev.org/show/bVvDh4svozwjBpLokYre/ ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1971672 Title: [FT] Error in "test_virtual_port_host_update" Status in neutron: New Bug description: Error executing neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovsdb_monitor.TestNBDbMonitorOverSsl.test_virtual_port_host_update Log: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_207/840448/2/check/neutron- functional-with-uwsgi/2074ad0/testr_results.html Snippet: https://paste.opendev.org/show/bVvDh4svozwjBpLokYre/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1971672/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1971663] [NEW] nova-compute process stuck without any actions
Public bug reported: Description === nova-compute process gets stuck after running for a while Steps to reproduce == Sorry, I haven't specified how to reproduce，but I know this problem has occurred at least three times in the arm environment, but not on x86. Expected result === nova-compute run normally Actual result = 2022-04-28 15:07:15.150 5967 DEBUG oslo_concurrency.lockutils [-] Lock "ca12badc-42a2-43f2-8a28-cb2afa3a377c" acquired by "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "53988d8e-75bc-40de-b9af-ce7509f94918" acquired by "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "bc2b5ad1-a7a4-45fc-8d95-e986476c3129" acquired by "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.151 5967 DEBUG oslo_concurrency.lockutils [-] Lock "f2068deb-c12f-4c31-998c-cf48af4440ff" acquired by "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "48577b65-28e2-4858-abed-db00f90c0dc6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "5e6a68fc-b711-4115-bc77-179a224ed5c6" acquired by "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.152 5967 DEBUG oslo_concurrency.lockutils [-] Lock "4f47003a-c056-4aa6-9be2-e5bec0cb4a07" acquired by "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "a1416265-68cb-4e06-a1b9-80c0bd138a09" acquired by "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 15:07:15.153 5967 DEBUG oslo_concurrency.lockutils [-] Lock "96408c82-5951-46e6-b105-bcb21118ed99" acquired by "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" :: waited 0.000s inner /usr/local/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 2022-04-28 17:11:03.055 5967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers timer() File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/timer.py", line 59, in __call__ cb(*args, **kw) File "/usr/local/lib/python3.6/site-packages/eventlet/semaphore.py", line 152, in _do_acquire waiter.switch() greenlet.error: cannot switch to a different thread Environment === Python 3.6.8 eventlet0.30.2 oslo.concurrency4.4.0 oslo.messaging 12.7.1 nova-compute Stack list == Stack for thread 281471352828416 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281473394471424 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281471310557696 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time) File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 77, in wait time.sleep(seconds) Stack for thread 281471838974464 File "/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 365, in run self.wait(sleep_time)
