[Bug 1859649] Re: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible
Added upstream to this bug and would like to get their opinion. Minimally this should have been release-noted I'd think. ** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1859649 Title: networking disruption on upgrade from 14.0.0 to 14.0.3 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1859649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1859649] Re: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible
taking a look closer, this could have happened since the neutron-gateway (server side) had an older version that is not aware of 1.5 objects, while neutron-ovs-agent (client side) requested a 1.5 versioned object, the backwards compatibility layer is meant to be used the other way around where the server is aware of newer versions while the client is not, so the server can remove fields from the response to downgrade the object and hand back a compatible one. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1859649 Title: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1859649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1859649] Re: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible
According to the comments in https://review.opendev.org/#/c/669360/ the backport mentioned earlier shouldn't have broken things. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1859649 Title: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1859649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1859649] Re: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible
About "neutron.agent.rpc oslo_messaging.rpc.client.RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5" commit b452c508b62 landed in 14.0.3 which bumped up the Port object's version to 1.5 making it incompatible with older versions -> https://github.com/openstack/neutron/commit/b452c508b62 This commit was backported to fix bug 1834484 ([QoS] qos_plugin._extend_port_resource_request is killing port retrieval performance) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1859649 Title: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1859649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1859649] Re: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: neutron (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1859649 Title: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1859649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1859649] Re: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible
The other in neutron-l3-agent (still on neutron-gateway), but this is
probably due to the upgrade restarting the daemons :
neutron-l3-agent.log:2020-01-14 16:22:11.298 456968 ERROR pyroute2.netns.nslink
[-] forced shutdown procedure, clean up netns manually
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info [-] [Errno 9] Bad file descriptor: OSError: [Errno
9] Bad file descriptor
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info Traceback (most recent call last):
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/neutron/common/utils.py", line 158, in call
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info return func(*args, **kwargs)
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 1186, in
process
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info self._process_internal_ports()
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 584, in
_process_internal_ports
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info self.internal_network_added(p)
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 486, in
internal_network_added
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info mtu=port.get('mtu'))
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 461, in
_internal_network_added
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info prefix=prefix, mtu=mtu)
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/neutron/agent/linux/interface.py", line 265, in
plug
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info namespace=namespace):
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 818, in
device_exists
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info return IPDevice(device_name,
namespace=namespace).exists()
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 318, in
exists
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info return privileged.interface_exists(self.name,
self.namespace)
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/neutron/privileged/agent/linux/ip_lib.py", line
50, in sync_inner
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info return input_func(*args, **kwargs)
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 241, in
_wrap
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info return self.channel.remote_call(name, args,
kwargs)
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info File
"/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 203, in
remote_call
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info raise exc_type(*result[2])
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info OSError: [Errno 9] Bad file descriptor
neutron-l3-agent.log:2020-01-14 16:22:11.322 456743 ERROR
neutron.agent.l3.router_info
neutron-l3-agent.log:2020-01-14 16:22:11.327 456743 ERROR
neutron.agent.l3.agent [-] Failed to process compatible router:
209673dc-a96b-4367-8b48-08bdf8f5ad5e: OSError: [Errno 9] Bad file descriptor
neutron-l3-agent.log:2020-01-14 16:22:11.327 456743 ERROR
neutron.agent.l3.agent Traceback (most recent call last):
neutron-l3-agent.log:2020-01-14 16:22:11.327 456743 ERROR
neutron.agent.l3.agent File
"/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 723, in
_process_routers_if_compatible
neutron-l3-agent.log:2020-01-14 16:22:11.327 456743 ERROR
neutron.agent.l3.agent self._process_router_if_compatible(router)
neutron-l3-agent.log:2020-01-14 16:22:11.327 456743 ERROR
neutron.agent.l3.agent File
[Bug 1859649] Re: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible
All errors appear to be on the neutron gateway. One kind in neutron-ovs-agent : neutron-openvswitch-agent.log:2020-01-14 16:21:31.563 456292 ERROR neutron.agent.rpc neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc [req-c5918d0d-f11b-4de7-b77c-f4628e33774d - - - - -] Failed to get details for device ab683ddb-a1f3-4404-b2b4-f912efe3530a: oslo_messaging.rpc.client.RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5 neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc Traceback (most recent call last): neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/neutron/agent/rpc.py", line 303, in get_devices_details_list_and_failed_devices neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc agent_restarted)) neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/neutron/agent/rpc.py", line 312, in get_device_details neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc resources.PORT, device, agent_restarted) neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/neutron/agent/resource_cache.py", line 61, in get_resource_by_id neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc agent_restarted=agent_restarted) neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/neutron/agent/resource_cache.py", line 79, in _flood_cache_for_query neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc filter_kwargs=filter_kwargs) neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/oslo_log/helpers.py", line 67, in wrapper neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc return method(*args, **kwargs) neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/neutron/api/rpc/handlers/resources_rpc.py", line 114, in bulk_pull neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs) neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/neutron_lib/rpc.py", line 157, in call neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc return self._original_context.call(ctxt, method, **kwargs) neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 178, in call neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc retry=self.retry) neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 128, in _send neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc retry=retry) neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc call_monitor_timeout, retry=retry) neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc raise result neutron-openvswitch-agent.log:2020-01-14 16:21:31.674 456292 ERROR neutron.agent.rpc oslo_messaging.rpc.client.RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1859649 Title: neutron 2:14.0.3-0ubuntu1~cloud0 and 2:14.0.0-0ubuntu1.1~cloud0 not compatible To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1859649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
