[Yahoo-eng-team] [Bug 1998317] [NEW] CachedResourceConsumerTracker update triggered for every command
Public bug reported: Some agents don't report resource_versions in agent state dict, since they don't use OVO. But for db.agents_db.AgentDbMixin.is_agent_considered_for_versions (and get_agents_resource_versions) every agent is considered for versions tracking. As a result for each api call api.rpc.callbacks.version_manager.get_resource_versions is called and triggers refresh of agent OVO versions. ** Affects: neutron Importance: Undecided Assignee: Szymon Wróblewski (bluex) Status: New ** Changed in: neutron Assignee: (unassigned) => Szymon Wróblewski (bluex) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1998317 Title: CachedResourceConsumerTracker update triggered for every command Status in neutron: New Bug description: Some agents don't report resource_versions in agent state dict, since they don't use OVO. But for db.agents_db.AgentDbMixin.is_agent_considered_for_versions (and get_agents_resource_versions) every agent is considered for versions tracking. As a result for each api call api.rpc.callbacks.version_manager.get_resource_versions is called and triggers refresh of agent OVO versions. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1998317/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1987780] [NEW] Nova notifications with error response are not retried
Public bug reported: Sometimes Neutron is failing to send notification to Nova due to timeout, refused connection or another HTTP error. Whenever such event occurs, Neutron notifier for Nova will log an error and consider it's job done. Meanwhile, since there is no retry mechanism, Nova will wait till timeout and put VM in error state. ** Affects: neutron Importance: Undecided Assignee: Szymon Wróblewski (bluex) Status: New ** Changed in: neutron Assignee: (unassigned) => Szymon Wróblewski (bluex) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1987780 Title: Nova notifications with error response are not retried Status in neutron: New Bug description: Sometimes Neutron is failing to send notification to Nova due to timeout, refused connection or another HTTP error. Whenever such event occurs, Neutron notifier for Nova will log an error and consider it's job done. Meanwhile, since there is no retry mechanism, Nova will wait till timeout and put VM in error state. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1987780/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1954785] [NEW] Session semantic violated when sending FloatingIP disassociation update during port deletion
context, port_id, do_notify)
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py",
line 233, in wrapped
return method(*args, **kwargs)
File "/opt/stack/neutron/neutron/db/l3_db.py", line 1702, in
disassociate_floatingips
metadata={'association_event': False}))
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/registry.py",
line 60, in publish
_get_callback_manager().publish(resource, event, trigger, payload=payload)
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py",
line 149, in publish
return self.notify(resource, event, trigger, payload=payload)
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/utils.py",
line 102, in _wrapped
return function(*args, **kwargs)
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py",
line 167, in notify
errors = self._notify_loop(resource, event, trigger, **kwargs)
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py",
line 197, in _notify_loop
callback(resource, event, trigger, **kwargs)
File "/opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py", line 95, in
handle_payload_event
payload.context, resource, event):
File "/opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py", line 83, in
_is_session_semantic_violated
stack = traceback.extract_stack()
** Affects: neutron
Importance: Undecided
Assignee: Szymon Wróblewski (bluex)
Status: New
** Changed in: neutron
Assignee: (unassigned) => Szymon Wróblewski (bluex)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1954785
Title:
Session semantic violated when sending FloatingIP disassociation
update during port deletion
Status in neutron:
New
Bug description:
In plugins.ml2.plugin.Ml2Plugin.delete_port we are calling
L3RouterPlugin.disassociate_floatingips
(db.l3_db.L3_NAT_dbonly_mixin.disassociate_floatingips) which at the end calls
neutron_lib.callbacks.registry.publish to send FLOATING_IP AFTER_UPDATE
notification.
Because call to disassociate_floatingips is performed within context of DB
transaction in delete_port,
plugins.ml2.ovo_rpc._ObjectChangeHandler._is_session_semantic_violated check
in plugins.ml2.ovo_rpc._ObjectChangeHandler.handle_event listening to all
AFTER_ notifications, will log a following warning and PREVENT notification
from being sent at all.
WARNING neutron.plugins.ml2.ovo_rpc [req-6588b226-81b6-463d-a376-45488162671f
req-3783a80a-16f1-468e-ae74-f9a9b8db8eca
tempest-ServersWithSpecificFlavorTestJSON-1266243365
tempest-ServersWithSpecificFlavorTestJSON-1266243365] This handler is supposed
to handle AFTER events, as in 'AFTER it's committed', not BEFORE. Offending
resource event: floatingip, after_update. Location:
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/greenthread.py",
line 219, in main
result = function(*args, **kwargs)
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line
789, in process_request
proto.__init__(conn_state, self)
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line
343, in __init__
self.handle()
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line
376, in handle
self.handle_one_request()
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line
450, in handle_one_request
self.handle_one_response()
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line
547, in handle_one_response
result = self.application(self.environ, start_response)
File "/opt/stack/neutron.venv/lib/python3.6/site-packages/paste/urlmap.py",
line 216, in __call__
return app(environ, start_response)
File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py",
line 129, in __call__
resp = self.call_func(req, *args, **kw)
File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py",
line 193, in call_func
return self.func(req, *args, **kwargs)
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_middleware/base.py",
line 131, in __call__
response = req.get_response(self.application)
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line
1314, in send
application, catch_exc_info=False)
File
"/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line
1278, in call_application
app_iter = a
[Yahoo-eng-team] [Bug 1952898] [NEW] Allow modification of max retries in OVSNeutronAgent
Public bug reported: Modifying max device retries in classes inheriting from OVSNeutronAgent is quite hard because it uses module constant. Since our internal agent providing vrack functionality depends heavily on OVSNeutronAgent it would be nice to be able to change that value without having to overwrite all functions using MAX_DEVICE_RETRIES constant. ** Affects: neutron Importance: Undecided Assignee: Szymon Wróblewski (bluex) Status: In Progress ** Changed in: neutron Assignee: (unassigned) => Szymon Wróblewski (bluex) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1952898 Title: Allow modification of max retries in OVSNeutronAgent Status in neutron: In Progress Bug description: Modifying max device retries in classes inheriting from OVSNeutronAgent is quite hard because it uses module constant. Since our internal agent providing vrack functionality depends heavily on OVSNeutronAgent it would be nice to be able to change that value without having to overwrite all functions using MAX_DEVICE_RETRIES constant. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1952898/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1943846] [NEW] Create floating agent gw port for DVR fails
Public bug reported: _create_dvr_floating_gw_port fails due to missing notification payload field: Error during notification for neutron.services.l3_router.l3_router_plugin.DVRResourceOperationHandler._create_dvr_floating_gw_port--9223372036853272501 floatingip, after_create: KeyError: 'fixed_port_id' Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop callback(resource, event, trigger, **kwargs) File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 359, in _create_dvr_floating_gw_port context, fip['fixed_port_id']) KeyError: 'fixed_port_id' ** Affects: neutron Importance: Undecided Assignee: Szymon Wróblewski (bluex) Status: New ** Description changed: - _create_dvr_floating_gw_port fails due to missing notification payload field: - ERROR neutron_lib.callbacks.manager [None req-73c39560-9820-4b08-85d0-709e57835767 tempest-PortsTest-568331032 tempest-PortsTest-568331032] Error during notification for neutron.services.l3_router.l3_router_plugin.DVRResourceOperationHandler._create_dvr_floating_gw_port--9223372036853272501 floatingip, after_create: KeyError: 'fixed_port_id' - ERROR neutron_lib.callbacks.manager Traceback (most recent call last): - ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop - ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) - ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 359, in _create_dvr_floating_gw_port - ERROR neutron_lib.callbacks.manager context, fip['fixed_port_id']) - ERROR neutron_lib.callbacks.manager KeyError: 'fixed_port_id' - ERROR neutron_lib.callbacks.manager + _create_dvr_floating_gw_port fails due to missing notification payload + field: + + Error during notification for neutron.services.l3_router.l3_router_plugin.DVRResourceOperationHandler._create_dvr_floating_gw_port--9223372036853272501 floatingip, after_create: KeyError: 'fixed_port_id' + Traceback (most recent call last): + File "/usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop + callback(resource, event, trigger, **kwargs) + File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 359, in _create_dvr_floating_gw_port + context, fip['fixed_port_id']) + KeyError: 'fixed_port_id' ** Changed in: neutron Assignee: (unassigned) => Szymon Wróblewski (bluex) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1943846 Title: Create floating agent gw port for DVR fails Status in neutron: New Bug description: _create_dvr_floating_gw_port fails due to missing notification payload field: Error during notification for neutron.services.l3_router.l3_router_plugin.DVRResourceOperationHandler._create_dvr_floating_gw_port--9223372036853272501 floatingip, after_create: KeyError: 'fixed_port_id' Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop callback(resource, event, trigger, **kwargs) File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 359, in _create_dvr_floating_gw_port context, fip['fixed_port_id']) KeyError: 'fixed_port_id' To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1943846/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1939125] [NEW] Incorect Auto schedule new network segments notification listner
Public bug reported: auto_schedule_new_network_segments() added in Ic9e64aa4ecdc3d56f00c26204ad931b810db7599 uses new payload notification listener in old stable branches of Neutron that still use old notify syntax. Following branches are affected: stable/stein, stable/rocky, stable/queens ** Affects: neutron Importance: Undecided Assignee: Szymon Wróblewski (bluex) Status: New ** Changed in: neutron Assignee: (unassigned) => Szymon Wróblewski (bluex) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1939125 Title: Incorect Auto schedule new network segments notification listner Status in neutron: New Bug description: auto_schedule_new_network_segments() added in Ic9e64aa4ecdc3d56f00c26204ad931b810db7599 uses new payload notification listener in old stable branches of Neutron that still use old notify syntax. Following branches are affected: stable/stein, stable/rocky, stable/queens To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1939125/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1933502] [NEW] L3 DB FloatingIP callbacks improvements
Public bug reported: FloatingIP change notifications in l3_db use obsolete notify calls (instead of publish) and form that isn't compatible with OVOServerRpcInterface, which makes usage of RemoteResourceCache for FloatingIP by agents quite difficult. ** Affects: neutron Importance: Undecided Assignee: Szymon Wróblewski (bluex) Status: New ** Changed in: neutron Assignee: (unassigned) => Szymon Wróblewski (bluex) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1933502 Title: L3 DB FloatingIP callbacks improvements Status in neutron: New Bug description: FloatingIP change notifications in l3_db use obsolete notify calls (instead of publish) and form that isn't compatible with OVOServerRpcInterface, which makes usage of RemoteResourceCache for FloatingIP by agents quite difficult. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1933502/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1926417] [NEW] ObjectChangeHandler excessive thread usage
Public bug reported: neutron.plugins.ml2.ovo_rpc._ObjectChangeHandler spawns new dispatch_events thread (within futurist.ThreadPoolExecutor) with each OVO change with handle_event. Then dispatch_events is locked with lockutils.synchronized that ensures only one dispatch_events is executed at a time even between different OVO types, all the other threads are waiting doing nothing. ThreadPoolExecutor has default upper limit of threads spawned equal to processor count * 5 for each _ObjectChangeHandler. That means on a busy system with large amount of processors and multiple OVOs monitored we could be spawning: >32 (proc_count) * 5 (ThreadPoolExecutor limit) * 9 (OVOs monitored) * >16 >(Neutron API processes) = >23k threads With that amount of threads spawned it's possible to hit max limit of threads for user and "RuntimeError: can't start new thread" exception starts appearing. ** Affects: neutron Importance: Undecided Assignee: Szymon Wróblewski (bluex) Status: New ** Changed in: neutron Assignee: (unassigned) => Szymon Wróblewski (bluex) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1926417 Title: ObjectChangeHandler excessive thread usage Status in neutron: New Bug description: neutron.plugins.ml2.ovo_rpc._ObjectChangeHandler spawns new dispatch_events thread (within futurist.ThreadPoolExecutor) with each OVO change with handle_event. Then dispatch_events is locked with lockutils.synchronized that ensures only one dispatch_events is executed at a time even between different OVO types, all the other threads are waiting doing nothing. ThreadPoolExecutor has default upper limit of threads spawned equal to processor count * 5 for each _ObjectChangeHandler. That means on a busy system with large amount of processors and multiple OVOs monitored we could be spawning: >32 (proc_count) * 5 (ThreadPoolExecutor limit) * 9 (OVOs monitored) * >16 (Neutron API processes) = >23k threads With that amount of threads spawned it's possible to hit max limit of threads for user and "RuntimeError: can't start new thread" exception starts appearing. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1926417/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1439389] [NEW] Metadata widget multivalue selection support
Public bug reported: Metadata widget don't allow to select more than one value when property type is array. ** Affects: horizon Importance: Undecided Assignee: Szymon Wróblewski (bluex) Status: In Progress ** Changed in: horizon Assignee: (unassigned) => Szymon Wróblewski (bluex) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1439389 Title: Metadata widget multivalue selection support Status in OpenStack Dashboard (Horizon): In Progress Bug description: Metadata widget don't allow to select more than one value when property type is array. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1439389/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1439312] [NEW] [Launch Instance Fix] Glance rest API fix
Public bug reported: Neither horizon/static/horizon/js/angular/services/hz.api.glance.js#GlanceAPI.getNamespace nor openstack_dashboard.api.rest.glance.MetadefsNamespace don't support resource_type retrieving properties without specific prefixes. ** Affects: horizon Importance: Undecided Assignee: Szymon Wróblewski (bluex) Status: New ** Changed in: horizon Assignee: (unassigned) => Szymon Wróblewski (bluex) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1439312 Title: [Launch Instance Fix] Glance rest API fix Status in OpenStack Dashboard (Horizon): New Bug description: Neither horizon/static/horizon/js/angular/services/hz.api.glance.js#GlanceAPI.getNamespace nor openstack_dashboard.api.rest.glance.MetadefsNamespace don't support resource_type retrieving properties without specific prefixes. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1439312/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
