[Yahoo-eng-team] [Bug 2045950] Re: Security group performance issue for iptables driver due to "stateless feature"

2023-12-26 Thread OpenStack Infra 
Reviewed:  https://review.opendev.org/c/openstack/neutron/+/903707
Committed: 
https://opendev.org/openstack/neutron/commit/6b6abb9698318a0b5db09f0c4d30a47438a94643
Submitter: "Zuul (22348)"
Branch:master

commit 6b6abb9698318a0b5db09f0c4d30a47438a94643
Author: Rodolfo Alonso Hernandez 
Date:   Thu Dec 14 15:45:48 2023 +

Improve the SG RPC callback ``security_group_info_for_ports``

This method populates the SG rules in a dictionary. Each SG rule
inherits the "stateful" value of the SG. Prior to this patch, each
SG rule was isuing a database call to retrieve the SG register.

In this patch, the SG "stateful" retrieval is done in one database
query for all SG. That improves the performance of this method
reducing the database access to only one single call.

This improvement, as commented in the LP bug, affects to
ML2/LinuxBridge. ML2/OVS agent uses a cached RPC implementation
that not requires to perform any RPC call/database query.

Closes-Bug: #2045950
Change-Id: Iafd0419a1d1eeb25d5589edc2570ebf287450957


** Changed in: neutron
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2045950

Title:
  Security group performance issue for iptables driver due to "stateless
  feature"

Status in neutron:
  Fix Released

Bug description:
  There is a huge performance issue with the security groups when using the 
iptables implementation:
  If you have a security group with say 500 rules it will take minutes for the 
RPC server to create the the port configuration.
  You will see this when you restart the neutron-linuxbridge-agent on a compute 
node with an instance with the security group with a lot of rules.
  In the agent log you will see "Preparing filters for devices" and this will 
take minutes for a single port when having a significate amount of rules in the 
security group.

  After some investigation this seems to be cause:

  In the commit below stateful functionality was added for iptables
  implementation:

  
https://opendev.org/openstack/neutron/commit/cbc473e066d#diff-7d7a372d8ed39ad8489a39ff7c3f3d783235218c

  However there is a huge performance impact, in the following function in 
  neutron/db/securitygroups_rpc_base.py
   def security_group_info_for_ports

  For EACH rule in the security group rule in a a group it will do a database 
lookup to check what the setting is on the group:
  stateful = self._is_security_group_stateful(context,
  security_group_id)
  Which will call:

  def _is_security_group_stateful(self, context, sg_id):
  return sg_obj.SecurityGroup.get_sg_by_id(context, sg_id).stateful

  
  So if you have say 500 rules it will go 500 times(!) to the database to check 
the exact same property on the group object which absolutely tanks performance.

  I played around with caching the stateful property for the group
  (since it is not even changeable on a security group if there are
  rules present) and the function went from taking multiple minutes to
  about a second.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2045950/+subscriptions


-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 2047399] [NEW] nova api returns 500 when resizing an instance with memory encryption enabled

2023-12-26 Thread Takashi Kajinami 
Public bug reported:

Description
===
When a user attempts to resize an instance with memory encryption enabled, API 
returns 500 error consistently.
Looking into nova-api.log, it seems the issue is caused by a mechanism similar 
to https://bugs.launchpad.net/nova/+bug/2041511 .

Steps to reproduce
==
* Create an image with hw_mem_encryption=True
 $ openstack image create encrypted ...
 $ openstack image set encrypted --property hw_mem_encryption=True

* Create an instance
 $ openstack server create testinstance --image encrypted --flavor flavor1 ...

* Resize the instance
 $ openstack server resize testinstance --flavor flavor2 

Expected result
===
Instance resize is accepted and processed by nova, without errors

Actual result
=
Nova api returns 500 error and does not accept the request

Environment
===

1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

Ubuntu 22.04 and UCA bobcat.

# dpkg -l | grep nova
ii nova-api 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - API frontend
ii nova-common 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute node 
base
ii nova-compute-kvm 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute 
node (KVM)
ii nova-compute-libvirt 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - 
compute node libvirt support
ii nova-conductor 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - conductor 
service
ii nova-novncproxy 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - virtual 
machine scheduler
ii python3-nova 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute Python 3 
libraries
ii python3-novaclient 2:18.4.0-0ubuntu1~cloud0 all client library for OpenStack 
Compute API - 3.x

2. Which hypervisor did you use?
Libvirt + KVM

3. Which storage type did you use?
LVM

4. Which networking type did you use?
ml2 + ovs

Logs & Configs
==
The following traceback is found in nova-api.log

```
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi [None 
req-20b9b69c-a792-45cd-8520-7e9cd3387c0d 838cd42e04884ddfa8ec4ac11e2f8818 
baf003aa0202430a92edd003f98794a3 - - default default] Unexpected exception in 
API method: NotImplementedError: Cannot load 'id' in the base class
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi Traceback (most 
recent call last):
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi   File 
"/usr/lib/python3/dist-packages/nova/api/openstack/wsgi.py", line 658, in 
wrapped
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi return f(*args, 
**kwargs)
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi   File 
"/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in 
wrapper
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi return 
func(*args, **kwargs)
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi   File 
"/usr/lib/python3/dist-packages/nova/api/openstack/compute/servers.py", line 
1146, in _action_resize
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi 
self._resize(req, id, flavor_ref, **kwargs)
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi   File 
"/usr/lib/python3/dist-packages/nova/api/openstack/compute/servers.py", line 
1060, in _resize
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi 
self.compute_api.resize(context, instance, flavor_id,
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi   File 
"/usr/lib/python3/dist-packages/nova/compute/api.py", line 389, in inner
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi return 
function(self, context, instance, *args, **kwargs)
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi   File 
"/usr/lib/python3/dist-packages/nova/compute/api.py", line 374, in wrapper
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi return 
func(self, context, instance, *args, **kwargs)
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi   File 
"/usr/lib/python3/dist-packages/nova/compute/api.py", line 357, in wrapper
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi return 
func(self, context, instance, *args, **kwargs)
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi   File 
"/usr/lib/python3/dist-packages/nova/compute/api.py", line 242, in inner
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi return 
function(self, context, instance, *args, **kwargs)
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi   File 
"/usr/lib/python3/dist-packages/nova/compute/api.py", line 168, in inner
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi return f(self, 
context, instance, *args, **kw)
2023-12-26 08:02:19.371 30791 ERROR nova.api.openstack.wsgi   File 
"/usr/lib/python3/dist-packages/nova/comput