[Yahoo-eng-team] [Bug 2060587] [NEW] [ML2][OVS] more precise flow table cleaning
Public bug reported: OVS-agent wants to clean flows table by table during restart, but actually it does not. [1] If one table has same cookie with other tables, all related flows will be clean at once. A bit radical in such style. [1] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py#L186 ** Affects: neutron Importance: Undecided Status: In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2060587 Title: [ML2][OVS] more precise flow table cleaning Status in neutron: In Progress Bug description: OVS-agent wants to clean flows table by table during restart, but actually it does not. [1] If one table has same cookie with other tables, all related flows will be clean at once. A bit radical in such style. [1] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py#L186 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2060587/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2052681] Re: Many stale neutron-keepalived-state-change processes left after upgrade to native pyroute2 state-change
** Changed in: neutron Status: Invalid => New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2052681 Title: Many stale neutron-keepalived-state-change processes left after upgrade to native pyroute2 state-change Status in neutron: New Bug description: Needs a post-upgrade script to remove those stale "ip -o monitor" and traditional "neutron-keepalived-state-change" processes. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2052681/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2052681] [NEW] Many stale neutron-keepalived-state-change processes left after upgrade to native pyroute2 state-change
Public bug reported: Needs a post-upgrade script to remove those stale "ip -o monitor" and traditional "neutron-keepalived-state-change" processes. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2052681 Title: Many stale neutron-keepalived-state-change processes left after upgrade to native pyroute2 state-change Status in neutron: New Bug description: Needs a post-upgrade script to remove those stale "ip -o monitor" and traditional "neutron-keepalived-state-change" processes. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2052681/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2052367] [NEW] neutron-keepalived-state-change run neutron-rootwrap-daemon anyway regardless of config
Public bug reported: neutron-keepalived-state-change run neutron-rootwrap-daemon anyway regardless of config https://review.opendev.org/q/topic:%22bug/1680183%22 It is the series of patches which replace "ip -o monitor" to pyroute2 python native process. But we noticed that the neutron-keepalived-state-change run neutron- rootwrap-daemon anyway. It is not configurable. A example process structure is: root 30474 1 0 10:23 ?00:00:00 /usr/bin/python /bin/neutron-keepalived-state-change --router_id=0a3f589a-40f7-400c-8cb7-e915c255dec1 --namespace=qrouter-0a3f589a-40f7-400c-8cb7-e915c255dec1 --conf_dir=/var/lib/neutron/ha_confs/0a3f589a-40f7-400c-8cb7-e915c255dec1 --log-file=/var/lib/neutron/ha_confs/0a3f589a-40f7-400c-8cb7-e915c255dec1/neutron-keepalived-state-change.log --monitor_interface=ha-1c854712-f1 --monitor_cidr=169.254.0.167/24 --pid_file=/var/lib/neutron/external/pids/0a3f589a-40f7-400c-8cb7-e915c255dec1.monitor.pid.neutron-keepalived-state-change-monitor --state_path=/var/lib/neutron --user=993 --group=990 root 30478 30474 0 10:23 ?00:00:00 sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf root 30482 30478 0 10:23 ?00:00:00 /usr/bin/python /bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf root 30479 30474 0 10:23 ?00:00:00 /usr/bin/python /bin/neutron-keepalived-state-change --router_id=0a3f589a-40f7-400c-8cb7-e915c255dec1 --namespace=qrouter-0a3f589a-40f7-400c-8cb7-e915c255dec1 --conf_dir=/var/lib/neutron/ha_confs/0a3f589a-40f7-400c-8cb7-e915c255dec1 --log-file=/var/lib/neutron/ha_confs/0a3f589a-40f7-400c-8cb7-e915c255dec1/neutron-keepalived-state-change.log --monitor_interface=ha-1c854712-f1 --monitor_cidr=169.254.0.167/24 --pid_file=/var/lib/neutron/external/pids/0a3f589a-40f7-400c-8cb7-e915c255dec1.monitor.pid.neutron-keepalived-state-change-monitor --state_path=/var/lib/neutron --user=993 --group=990 We can see that the neutron-rootwrap-daemon did not spawn the "neutron- keepalived-state-change" sub-process. And that "sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf" will run into a zombie process finally. neutron-keepalived-state-change.log LOGs: 2024-02-04 10:23:21.022 30449 INFO neutron.common.config [-] Logging enabled! 2024-02-04 10:23:21.023 30449 INFO neutron.common.config [-] /bin/neutron-keepalived-state-change version 13.7.2.dev20240202170237 2024-02-04 10:23:21.023 30449 DEBUG neutron.common.config [-] command line: /bin/neutron-keepalived-state-change --router_id=0a3f589a-40f7-400c-8cb7-e915c255dec1 --namespace=qrouter-0a3f589a-40f7-400c-8cb7-e915c255dec1 --conf_dir=/var/lib/neutron/ha_confs/0a3f589a-40f7-400c-8cb7-e915c255dec1 --log-file=/var/lib/neutron/ha_confs/0a3f589a-40f7-400c-8cb7-e915c255dec1/neutron-keepalived-state-change.log --monitor_interface=ha-1c854712-f1 --monitor_cidr=169.254.0.167/24 --pid_file=/var/lib/neutron/external/pids/0a3f589a-40f7-400c-8cb7-e915c255dec1.monitor.pid.neutron-keepalived-state-change-monitor --state_path=/var/lib/neutron --user=993 --group=990 setup_logging /usr/lib/python/site-packages/neutron/common/config.py:106 2024-02-04 10:23:21.033 30474 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-0a3f589a-40f7-400c-8cb7-e915c255dec1', 'ip', 'addr', 'show', 'ha-1c854712-f1'] execute_rootwrap_daemon /usr/lib/python/site-packages/neutron/agent/linux/utils.py:103 2024-02-04 10:23:21.043 30474 DEBUG oslo_rootwrap.client [-] Popen for ['sudo', 'neutron-rootwrap-daemon', '/etc/neutron/rootwrap.conf'] command has been instantiated _initialize /usr/lib/python/site-packages/oslo_rootwrap/client.py:73 2024-02-04 10:23:21.185 30474 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=30478 2024-02-04 10:23:21.232 30474 DEBUG neutron.agent.l3.keepalived_state_change [-] Wrote router 0a3f589a-40f7-400c-8cb7-e915c255dec1 state master write_state_change /usr/lib/python/site-packages/neutron/agent/l3/keepalived_state_change.py:123 2024-02-04 10:23:21.235 30474 DEBUG neutron.agent.l3.keepalived_state_change [-] Notified agent router 0a3f589a-40f7-400c-8cb7-e915c255dec1, state master notify_agent /usr/lib/python/site-packages/neutron/agent/l3/keepalived_state_change.py:137 2024-02-04 10:23:21.236 30474 DEBUG neutron.agent.l3.keepalived_state_change [-] Initial status of router 0a3f589a-40f7-400c-8cb7-e915c255dec1 is master handle_initial_state /usr/lib/python/site-packages/neutron/agent/l3/keepalived_state_change.py:114 So the code of https://github.com/openstack/neutron/blob/master/neutron/agent/l3/ha_router.py#L433-L444 should add some configurable options to decide whether run neutron-rootwrap-daemon the neutron-keepalived-state-change. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/
[Yahoo-eng-team] [Bug 2051863] Re: Race condition during VM creation - could not open network device tapXXX (No such device)
Looks more like a libvirt error, or nova side problem. Neutron does not take responsibilities to create the tap-XXX device. It is plugged by nova-compute. Need to find out why the tap device is not created before TC rules creating. ** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2051863 Title: Race condition during VM creation - could not open network device tapXXX (No such device) Status in neutron: New Status in OpenStack Compute (nova): New Bug description: [High-Level Description] While creating an amphora from the Octavia service, we are encountering a race condition. Nova-compute is unable to add a QC ingress rule because the tap interface does not exist yet. This situation results in the service becoming temporarily unavailable, failing in approximately 50% of cases. [Pre-conditions] Create a simple octavia loadbalancer with environment deployed by kolla-environment. For amphora server, use flavor with qos properties as from example: https://paste.openstack.org/show/bkx3RgEFqUwxCU0Qr8z2/ [Step-by-Step Reproduction] Create a load balancer as part of a Heat stack, following the example provided at: https://github.com/syseleven/heat-examples/tree/master/lbaas-octavia-http Use a qos flavor as shown in example: https://paste.openstack.org/show/bkx3RgEFqUwxCU0Qr8z2/ [Expected Output] Expect 100% successful creation of the amphora and subsequent Nova instances serving as members for the load balancer. [Actual Output] During the creation process, it is observed that nova-compute fails due to: /var/log/kolla/libvirt/libvirtd.log:2024-01-31 09:24:55.852+: 4065628: error : virCommandWait:2748 : internal error: Child process (tc filter add dev tap5f365c23-89 parent : protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists. /var/log/kolla/nova/nova-compute.log:2024-01-31 10:24:56.086 7 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: Child process (tc filter add dev tap5f365c23-89 parent : protocol all u32 match u32 0 0 police rate 64000kbps burst 64000kb mtu 64kb drop flowid :1) unexpected exit status 2: Error: Parent Qdisc doesn't exists. Unfortunately ovs-vswitchd.log file says that tap interface was not created at this point: /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.469Z|00094|bridge|WARN|could not open network device tap5f365c23-89 (No such device) /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.809Z|00095|bridge|INFO|bridge br-int: added interface tap5f365c23-89 on port 27 /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.885Z|00096|bridge|INFO|bridge br-int: deleted interface tap5f365c23-89 on port 27 /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.888Z|00097|bridge|WARN|could not open network device tap5f365c23-89 (No such device) /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.927Z|00098|bridge|WARN|could not open network device tap5f365c23-89 (No such device) /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.934Z|00099|bridge|WARN|could not open network device tap5f365c23-89 (No such device) /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:55.943Z|00100|bridge|WARN|could not open network device tap5f365c23-89 (No such device) /var/log/kolla/openvswitch/ovs-vswitchd.log:2024-01-31T09:24:56.024Z|00101|bridge|WARN|could not open network device tap5f365c23-89 (No such device) [Version] OpenStack Zed, deployed by kolla-ansible with all defaults Ubuntu 22.04 LTS libvirt: 8.0.0-1ubuntu7.7 nova: zed python3-openvswitch: 3.0.3-0ubuntu0.22.10.3~cloud3 neutron: zed To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2051863/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1847747] Re: [RPC] digging RPC timeout for client and server
** Changed in: neutron Status: New => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1847747 Title: [RPC] digging RPC timeout for client and server Status in neutron: Won't Fix Status in oslo.messaging: Fix Released Bug description: RPC timeout can be found frequently, but we have no statistical data for it. A simple log can help. Since all the projects are using oslo.messaging as midlware between services and message queue, we can add a log in it, something like this, a local test: 2019-10-11 19:23:05.703 40774 INFO oslo_messaging.rpc.server [-] Receive incoming message with id cf4ab75b89e64458bc989c709add43cf and method: sync_routers. 2019-10-11 19:29:19.781 40774 INFO oslo_messaging.rpc.server:188 [None req-e33a11a2-64cd-4aea-b801-7689996a8ace - - - - -] Replied message with id cf4ab75b89e64458bc989c709add43cf and method: sync_routers. Time elapsed: 374.078 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1847747/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2048979] [NEW] [ml2][ovs] ports without local vlan tag are processed on openflow security group
Public bug reported:
We recently met an issue during VM live migration:
1. nova starts live migration
2. plug ports on new host
3. neutron-ovs-agent starts to process the port, but the port is in 'added' and
'updated' set at the same time.
4. because nova still not activate the destination port binding, so there is no
local vlan for this port.
Then, ovs-agent met errors:
Error while processing VIF ports: OVSFWTagNotFound: Cannot get tag for port
tap092f38ed-a7 from its other_config: {}
A fix should be added for ``setup_port_filters`` to remove ports of the
"binding_no_activated_devices".
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2048979
Title:
[ml2][ovs] ports without local vlan tag are processed on openflow
security group
Status in neutron:
New
Bug description:
We recently met an issue during VM live migration:
1. nova starts live migration
2. plug ports on new host
3. neutron-ovs-agent starts to process the port, but the port is in 'added'
and 'updated' set at the same time.
4. because nova still not activate the destination port binding, so there is
no local vlan for this port.
Then, ovs-agent met errors:
Error while processing VIF ports: OVSFWTagNotFound: Cannot get tag for port
tap092f38ed-a7 from its other_config: {}
A fix should be added for ``setup_port_filters`` to remove ports of the
"binding_no_activated_devices".
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2048979/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2047100] [NEW] Agent resource cache updates
Public bug reported:
1. Agent resource cache has an infinite growth set: _satisfied_server_queries
https://github.com/openstack/neutron/blob/master/neutron/agent/resource_cache.py#L41
there is no entry removal for this set.
2. Because this set has a non-standard structure, for instance:
set([('Port', ('id', (u'830d035e-5138-49ae-bbe4-324f4096656d',))), ('Network',
('id', ('e04208de-1006-4a6b-881a-83129856afa6',))), ('Network', ('id',
('505155ea-8fbb-42d1-a8c9-cc2c78f8476e',))), ('Port', ('id',
(u'ac825cc9-906a-45db-a77d-4e336fc1c4ea',))), ('Port', ('id',
(u'c3a72a39-dbd5-4737-a68c-120de93b186c',))), ('Network', ('id',
('cd5155df-9777-4487-a730-b5ee533c4f80',))), ('Port', ('id',
(u'340e02f2-fe54-4f31-8858-7f6413fb0010',))), ('Port', ('id',
(u'64fc4d85-04d6-453f-8d20-f4d1308d34fd',))), ('Network', ('id',
('a6201723-a237-433c-b357-82a6a24526e5',))), ('Network', ('id',
('71a80697-1705-4bd0-b65b-0fd7dd616836',))), ('Port', ('security_group_ids',
('48a2ebb8-16ea-4a0a-9d45-eabc6a6b3dcf',))), ('Network', ('id',
('7e83c48a-b246-4a02-bb87-10016ac4b47e',))), ('SecurityGroupRule',
('security_group_id', (u'48a2ebb8-16ea-4a0a-9d45-eabc6a6b3dcf',))), ('Port',
('id', (u'2cc656ba-b07b-4e85-ad56-ee6da4b2e763',))), ('Port', ('id',
(u'89d0aab8-82f7-4e5e-98b1-e009e31498ce',))), ('Port',
('id', (u'd820dbc2-bf4f-463b-9a67-6b704202bee0',))), ('Network', ('id',
('aea5771b-9655-4936-b9f4-f94d482c0b15',))), ('Port', ('id',
(u'68c3e31b-9bf6-45e9-bfbb-3da1cafebcec',)))])
It's hardly to remove all entries for one resource at all, because if
some codes query cache by filter=None, some codes use fitler={"x":y,
"a": b}, the entries are various, especially when the code is not in
Neutron.
3. If the port removed, and added again, because the query is in the
_satisfied_server_queries:
https://github.com/openstack/neutron/blob/master/neutron/agent/resource_cache.py#L75
It may return None or stale resource.
So, it's better to remove such "server_queries" records. Because, if the
resource is in cache, just return it. If it is not, get it from neutron-
server.
** Affects: neutron
Importance: Undecided
Status: New
** Description changed:
- 1. Agent resource cache has a infinite growth set: _satisfied_server_queries
+ 1. Agent resource cache has an infinite growth set: _satisfied_server_queries
https://github.com/openstack/neutron/blob/master/neutron/agent/resource_cache.py#L41
there is no entry removal for this set.
2. Because this set has a non-standard structure, for instance:
set([('Port', ('id', (u'830d035e-5138-49ae-bbe4-324f4096656d',))),
('Network', ('id', ('e04208de-1006-4a6b-881a-83129856afa6',))), ('Network',
('id', ('505155ea-8fbb-42d1-a8c9-cc2c78f8476e',))), ('Port', ('id',
(u'ac825cc9-906a-45db-a77d-4e336fc1c4ea',))), ('Port', ('id',
(u'c3a72a39-dbd5-4737-a68c-120de93b186c',))), ('Network', ('id',
('cd5155df-9777-4487-a730-b5ee533c4f80',))), ('Port', ('id',
(u'340e02f2-fe54-4f31-8858-7f6413fb0010',))), ('Port', ('id',
(u'64fc4d85-04d6-453f-8d20-f4d1308d34fd',))), ('Network', ('id',
('a6201723-a237-433c-b357-82a6a24526e5',))), ('Network', ('id',
('71a80697-1705-4bd0-b65b-0fd7dd616836',))), ('Port', ('security_group_ids',
('48a2ebb8-16ea-4a0a-9d45-eabc6a6b3dcf',))), ('Network', ('id',
('7e83c48a-b246-4a02-bb87-10016ac4b47e',))), ('SecurityGroupRule',
('security_group_id', (u'48a2ebb8-16ea-4a0a-9d45-eabc6a6b3dcf',))), ('Port',
('id', (u'2cc656ba-b07b-4e85-ad56-ee6da4b2e763',))), ('Port', ('id',
(u'89d0aab8-82f7-4e5e-98b1-e009e31498ce',))), ('Port
', ('id', (u'd820dbc2-bf4f-463b-9a67-6b704202bee0',))), ('Network', ('id',
('aea5771b-9655-4936-b9f4-f94d482c0b15',))), ('Port', ('id',
(u'68c3e31b-9bf6-45e9-bfbb-3da1cafebcec',)))])
It's hardly to remove all one ports' entry at all, because if some codes
query cache by filter=None, some codes use fitler={"x":y, "a": b},
especially when the code is not in Neutron.
3. If the port removed, and added again, because the query is in the
_satisfied_server_queries:
https://github.com/openstack/neutron/blob/master/neutron/agent/resource_cache.py#L75
It may return None or stale resource.
So, it's better to remove such "server_queries" records. Because, if the
resource is in cache, just return it. If it is not, get it from neutron-
server.
** Description changed:
1. Agent resource cache has an infinite growth set: _satisfied_server_queries
https://github.com/openstack/neutron/blob/master/neutron/agent/resource_cache.py#L41
there is no entry removal for this set.
2. Because this set has a non-standard structure, for instance:
set([('Port', ('id', (u'830d035e-5138-49ae-bbe4-324f4096656d',))),
('Network', ('id', ('e04208de-1006-4a6b-881a-83129856afa6',))), ('Network',
('id', ('505155ea-8fbb-42d1-a8c9-cc2c78f8476e',))), ('Port', ('id',
(u'ac825cc9-906a-45db-a77d-4e336fc1c4ea',))), ('Port', ('id',
(u'c3a72a39-dbd5-4737-a68c-120de93b186c',))), ('Network', ('id',
('cd5155df-9777-448
[Yahoo-eng-team] [Bug 2046510] [NEW] neutron-servers have high memory usage
Public bug reported: USERPID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND neutron 11686 6.8 7.3 29205516 28869568 ? SNov30 1438:32 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log neutron 11678 5.3 5.9 23972284 23636560 ? SNov30 1117:52 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log neutron 11677 23.3 5.9 23925788 23590656 ? SNov30 4882:29 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log neutron 11694 21.7 5.9 23909852 23574128 ? SNov30 4550:55 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log neutron 11693 16.4 5.9 23624384 23288660 ? SNov30 3431:10 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log neutron 11696 4.2 5.5 22357296 22021572 ? SNov30 878:51 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log You may see the RSS reachs 28869568KB, it's about 20G+ memory. Openstack version: stable/rocky ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2046510 Title: neutron-servers have high memory usage Status in neutron: New Bug description: USERPID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND neutron 11686 6.8 7.3 29205516 28869568 ? SNov30 1438:32 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log neutron 11678 5.3 5.9 23972284 23636560 ? SNov30 1117:52 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log neutron 11677 23.3 5.9 23925788 23590656 ? SNov30 4882:29 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log neutron 11694 21.7 5.9 23909852 23574128 ? SNov30 4550:55 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log neutron 11693 16.4 5.9 23624384 23288660 ? SNov30 3431:10 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d
[Yahoo-eng-team] [Bug 2043761] [NEW] Creating subnet for exsiting external network is time-comsuming
Public bug reported: If an external network has many router on it, then create a subnet will be time-consuming. Neutron will try to update all routers' external gateway ports anyway, ignore the subnet service_type. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2043761 Title: Creating subnet for exsiting external network is time-comsuming Status in neutron: New Bug description: If an external network has many router on it, then create a subnet will be time-consuming. Neutron will try to update all routers' external gateway ports anyway, ignore the subnet service_type. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2043761/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2042411] [NEW] [CI][Fullstack] neutron.tests.fullstack.test_securitygroup cases are missing
Public bug reported: For instance: neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork.test_securitygroup https://f9c28e74a4e26f2c90e8-fa663b3b43bb6eacc0d3184a52007f13.ssl.cf5.rackcdn.com/888098/11/check/neutron- fullstack-with-uwsgi/e34e362/testr_results.html Code: https://github.com/openstack/neutron/blob/master/neutron/tests/fullstack/test_securitygroup.py#L114 These cases are really important for securitygroup functionalities. ** Affects: neutron Importance: Undecided Status: Invalid ** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2042411 Title: [CI][Fullstack] neutron.tests.fullstack.test_securitygroup cases are missing Status in neutron: Invalid Bug description: For instance: neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork.test_securitygroup https://f9c28e74a4e26f2c90e8-fa663b3b43bb6eacc0d3184a52007f13.ssl.cf5.rackcdn.com/888098/11/check/neutron- fullstack-with-uwsgi/e34e362/testr_results.html Code: https://github.com/openstack/neutron/blob/master/neutron/tests/fullstack/test_securitygroup.py#L114 These cases are really important for securitygroup functionalities. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2042411/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2039553] [NEW] dhcp-agent can not interact with neutron-server after neutron-server restarted
Public bug reported: Recently we meet a strange issue between neutron-server and neutron- dhcp-agents. In a long run deployment, we just restart all neutron- servers, then we failed to boot VM. Yes, it is vif-plug-timeout!!! We noticed that the DHCP provisioningblock was not deleted. Our operator gives the recover steps: 1. stop the network's scheduled dhcp-agents 2. delete the related Queues in rabbitMQ of these dhcp-agents 3. start dhcp-agents. Then everythin will works fine. So, do you have any idea about such issue? ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2039553 Title: dhcp-agent can not interact with neutron-server after neutron-server restarted Status in neutron: New Bug description: Recently we meet a strange issue between neutron-server and neutron- dhcp-agents. In a long run deployment, we just restart all neutron- servers, then we failed to boot VM. Yes, it is vif-plug-timeout!!! We noticed that the DHCP provisioningblock was not deleted. Our operator gives the recover steps: 1. stop the network's scheduled dhcp-agents 2. delete the related Queues in rabbitMQ of these dhcp-agents 3. start dhcp-agents. Then everythin will works fine. So, do you have any idea about such issue? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2039553/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2039554] [NEW] dhcp-agent can not interact with neutron-server after neutron-server restarted
Public bug reported: Recently we meet a strange issue between neutron-server and neutron- dhcp-agents. In a long run deployment, we just restart all neutron- servers, then we failed to boot VM. Yes, it is vif-plug-timeout!!! We noticed that the DHCP provisioningblock was not deleted. Our operator gives the recover steps: 1. stop the network's scheduled dhcp-agents 2. delete the related Queues in rabbitMQ of these dhcp-agents 3. start dhcp-agents. Then everythin will works fine. So, do you have any idea about such issue? ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2039554 Title: dhcp-agent can not interact with neutron-server after neutron-server restarted Status in neutron: New Bug description: Recently we meet a strange issue between neutron-server and neutron- dhcp-agents. In a long run deployment, we just restart all neutron- servers, then we failed to boot VM. Yes, it is vif-plug-timeout!!! We noticed that the DHCP provisioningblock was not deleted. Our operator gives the recover steps: 1. stop the network's scheduled dhcp-agents 2. delete the related Queues in rabbitMQ of these dhcp-agents 3. start dhcp-agents. Then everythin will works fine. So, do you have any idea about such issue? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2039554/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2016198] [NEW] [L3][HA] race condition between first two router creations when tenant has no HA network
Public bug reported: When the tenant creates the first HA router-1, neutron will try to create HA network for this project. But during the HA network creation procedure, we assume it has 2 steps: (1) create HA network (2) create subnet for this HA network another router-2 creation API call is comming, while HA network creation is just done the step (1). This router-2 creation API can retrieve the project HA network, and the HA network has no subnet. Because of the "[Open Discussion] neutron can create port from network which has no subnet" https://bugs.launchpad.net/neutron/+bug/2016197, router-2 will create HA ports without fixed IP. So this HA router-2 will not get UP forever in L3 agent side. In our production environment, we see many times of such issue. ** Affects: neutron Importance: Undecided Status: New ** Summary changed: - [L3][HA] race condition amoning router creations when tenant has no HA network + [L3][HA] race condition first two router creations when tenant has no HA network ** Summary changed: - [L3][HA] race condition first two router creations when tenant has no HA network + [L3][HA] race condition between first two router creations when tenant has no HA network ** Description changed: When the tenant creates the first HA router-1, neutron will try to create HA network for this project. But during the HA network creation procedure, we assume it has 2 steps: (1) create HA network (2) create subnet for this HA network - another router-2 creation API call is comming, while HA network creation is just done the step (1), - this API can retrieve the project HA network, and the HA network has no subnet. + another router-2 creation API call is comming, while HA network creation is just done the step (1). + This router-2 creation API can retrieve the project HA network, and the HA network has no subnet. - Because of the + Because of the "[Open Discussion] neutron can create port from network which has no subnet" https://bugs.launchpad.net/neutron/+bug/2016197, router-2 will create HA ports without fixed IP. So this HA router-2 will not get UP forever in L3 agent side. In our production environment, we see many times of such issue. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2016198 Title: [L3][HA] race condition between first two router creations when tenant has no HA network Status in neutron: New Bug description: When the tenant creates the first HA router-1, neutron will try to create HA network for this project. But during the HA network creation procedure, we assume it has 2 steps: (1) create HA network (2) create subnet for this HA network another router-2 creation API call is comming, while HA network creation is just done the step (1). This router-2 creation API can retrieve the project HA network, and the HA network has no subnet. Because of the "[Open Discussion] neutron can create port from network which has no subnet" https://bugs.launchpad.net/neutron/+bug/2016197, router-2 will create HA ports without fixed IP. So this HA router-2 will not get UP forever in L3 agent side. In our production environment, we see many times of such issue. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2016198/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2016197] [NEW] [Open Discussion] neutron can create port from network which has no subnet
Public bug reported: # openstack network create test-network +---+--+ | Field | Value| +---+--+ | admin_state_up| UP | | availability_zone_hints | | | availability_zones| | | created_at| 2023-04-13T07:31:16Z | | description | | | dns_domain| None | | id| d595860d-e576-4899-882a-8428cfe601f2 | | ipv4_address_scope| None | | ipv6_address_scope| None | | is_default| False| | is_vlan_transparent | None | | mtu | 1450 | | name | test-network | | port_security_enabled | True | | project_id| c675870e8b8e48e7a339907b5bcf7c8c | | provider:network_type | vxlan| | provider:physical_network | None | | provider:segmentation_id | 99941| | qos_policy_id | None | | revision_number | 1| | router:external | Internal | | segments | None | | shared| False| | status| ACTIVE | | subnets | | | tags | | | updated_at| 2023-04-13T07:31:16Z | +---+--+ # openstack port create --network d595860d-e576-4899-882a-8428cfe601f2 test-create-port-net-has-no-subnet +---+--+ | Field | Value| +---+--+ | admin_state_up| UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at| 2023-04-13T07:35:46Z | | data_plane_status | None | | description | | | device_id | | | device_owner | | | dns_assignment| None | | dns_domain| None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | | | id| 1f2e55bc-9e8f-45ca-b16d-d8e32c1c4bb5 | | mac_address | fa:16:3e:04:18:10| | name | test-create-port-net-has-no-subnet | | network_id| d595860d-e576-4899-882a-8428cfe601f2 | | port_security_enabled | True | | project_id| c675870e8b8e48e7a339907b5bcf7c8c | | qos_policy_id | None | | revision_number | 2| | security_group_ids| a3d879a7-2e72-4b68-960e-cfa20569c2b0 | | status| DOWN | | tags | | | trunk_details | None | | updated_at| 2023-04-13T07:35:46Z | +---+--+ Port has no fixed ips. Is such behavior is useful? or neutron should disable such creation? ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2016197 Title: [Open Discussion] neutron can create port from network which has no subnet Status in neutron: New Bug description: # openstack networ
[Yahoo-eng-team] [Bug 2000046] [NEW] [ml2][ovs] port flows Unexpectedly deleted by arp_spoofing_protection
Public bug reported: Port arp_spoofing_protection will install flows like this: table=0, priority=9,in_port=2 actions=goto_table:25 table=25, priority=2,in_port=2,dl_src=fa:16:3e:54:f0:71 actions=goto_table:60 For network ports or port_security_enabled = False, those flows will be delete by setup_arp_spoofing_protection in _bind_devices [1][2][3][4]. Besides, the ovs_agent extension handle_port will be run before these actions [5]. So, if any extesnion adds flows in table=0 with "in_port=x". will be delete unexpectedly. [1] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py#L385 [2] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1300 [3] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1307 [4] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1241 [5] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L2038 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/246 Title: [ml2][ovs] port flows Unexpectedly deleted by arp_spoofing_protection Status in neutron: New Bug description: Port arp_spoofing_protection will install flows like this: table=0, priority=9,in_port=2 actions=goto_table:25 table=25, priority=2,in_port=2,dl_src=fa:16:3e:54:f0:71 actions=goto_table:60 For network ports or port_security_enabled = False, those flows will be delete by setup_arp_spoofing_protection in _bind_devices [1][2][3][4]. Besides, the ovs_agent extension handle_port will be run before these actions [5]. So, if any extesnion adds flows in table=0 with "in_port=x". will be delete unexpectedly. [1] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py#L385 [2] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1300 [3] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1307 [4] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1241 [5] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L2038 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/246/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1998751] [NEW] [L3][DVR][vlan] HA VRRP traffic flooding on physical bridges on compute nodes
Public bug reported: For L3 DVR of VLAN networks, the HA VRRP traffic flooding from vlan HA networks will be flooding on physical bridges on compute nodes forever. For compute nodes, the physical bridges can directly drop the multicast packets of CIDR "l3_ha_net_cidr". ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1998751 Title: [L3][DVR][vlan] HA VRRP traffic flooding on physical bridges on compute nodes Status in neutron: New Bug description: For L3 DVR of VLAN networks, the HA VRRP traffic flooding from vlan HA networks will be flooding on physical bridges on compute nodes forever. For compute nodes, the physical bridges can directly drop the multicast packets of CIDR "l3_ha_net_cidr". To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1998751/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1998749] [NEW] [L3][DVR][vlan] east-west traffic flooding on physical bridges
Public bug reported: For L3 DVR of VLAN networks, the east west traffic between router subnets will be flooding on the physical bridge. Assuming we have resources like this: 1. subnet-1 10.10.10.0/24 with gateway-1 10.10.10.1, mac-address-[01] 2. subnet-2 20.20.20.0/24 with gateway-2 20.20.20.1, mac-address-[02] 3. subnet-1 and subnet-2 are all connected to router-1 4. host-1 with dvr_host_mac-[03] 5. host-2 with dvr_host_mac-[04] 6. vm-1 on host-1 with IP 10.10.10.100 and mac-address-[05] 7. vm-2 on host-2 with IP 20.20.20.100 and mac-address-[06] The packet procedure looks like this: 1. VM-1 is trying to ping VM-2 2. VM-1 gets the gateway-1's MAC address, and send ping request out. packet tuple (src_mac=[05], des_mac=[01], src_ip=10.10.10.100, dst_ip=20.20.20.100) 3. packet goes to dvr local router namespace, and routing 4. packet source mac address is changed to mac-address-[02] and destination to mac-addres-[06]. packet tuple (src_mac=[02], des_mac=[06], src_ip=10.10.10.100, dst_ip=20.20.20.100) 5. the source MAC address will be changed to dvr_host_mac-[03] on physical bridge (br-vlan), and hit NORMAL packet tuple (src_mac=[01], des_mac=[06], src_ip=10.10.10.100, dst_ip=20.20.20.100) * flooding happens, because the br-vlan had never learnt the destination mac address of VM-2 mac-address-[06]. Tracing the packets, you will see: NORMAL -> no learned MAC for destination, flooding Typically, the flooding action of fast flows are: actions:set(eth(src=fa:16:3f:xx:xx:03)),push_vlan(vid=X,pcp=0),br-vlan,NIC_PORT 6. On host-2, it hits a dvr_host_mac-[03] flows in table=3 and output to patch-to-int-br, which has a side effect is that the source mac address will not be learnt. packet tuple (src_mac=[01], des_mac=[06], src_ip=10.10.10.100, dst_ip=20.20.20.100) flows: table=3, priority=2,dl_src=fa:16:3f:xx:xx:03 actions=output:"phy- br-vlan" 7. On host-2, VM-2 send reply back, and finally go to br-vlan, it flooding again packet tuple (src_mac=[04], des_mac=[05], src_ip=20.20.20.100, dst_ip=10.10.10.100) 8. ping reply back to host-2, on br-vlan, it hits the output flows which match the dvr_host_mac-[04], then the VM-2's mac address will never be learnt. So, ping request is flooding on br-vlan forever. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1998749 Title: [L3][DVR][vlan] east-west traffic flooding on physical bridges Status in neutron: New Bug description: For L3 DVR of VLAN networks, the east west traffic between router subnets will be flooding on the physical bridge. Assuming we have resources like this: 1. subnet-1 10.10.10.0/24 with gateway-1 10.10.10.1, mac-address-[01] 2. subnet-2 20.20.20.0/24 with gateway-2 20.20.20.1, mac-address-[02] 3. subnet-1 and subnet-2 are all connected to router-1 4. host-1 with dvr_host_mac-[03] 5. host-2 with dvr_host_mac-[04] 6. vm-1 on host-1 with IP 10.10.10.100 and mac-address-[05] 7. vm-2 on host-2 with IP 20.20.20.100 and mac-address-[06] The packet procedure looks like this: 1. VM-1 is trying to ping VM-2 2. VM-1 gets the gateway-1's MAC address, and send ping request out. packet tuple (src_mac=[05], des_mac=[01], src_ip=10.10.10.100, dst_ip=20.20.20.100) 3. packet goes to dvr local router namespace, and routing 4. packet source mac address is changed to mac-address-[02] and destination to mac-addres-[06]. packet tuple (src_mac=[02], des_mac=[06], src_ip=10.10.10.100, dst_ip=20.20.20.100) 5. the source MAC address will be changed to dvr_host_mac-[03] on physical bridge (br-vlan), and hit NORMAL packet tuple (src_mac=[01], des_mac=[06], src_ip=10.10.10.100, dst_ip=20.20.20.100) * flooding happens, because the br-vlan had never learnt the destination mac address of VM-2 mac-address-[06]. Tracing the packets, you will see: NORMAL -> no learned MAC for destination, flooding Typically, the flooding action of fast flows are: actions:set(eth(src=fa:16:3f:xx:xx:03)),push_vlan(vid=X,pcp=0),br-vlan,NIC_PORT 6. On host-2, it hits a dvr_host_mac-[03] flows in table=3 and output to patch-to-int-br, which has a side effect is that the source mac address will not be learnt. packet tuple (src_mac=[01], des_mac=[06], src_ip=10.10.10.100, dst_ip=20.20.20.100) flows: table=3, priority=2,dl_src=fa:16:3f:xx:xx:03 actions=output:"phy-br-vlan" 7. On host-2, VM-2 send reply back, and finally go to br-vlan, it flooding again packet tuple (src_mac=[04], des_mac=[05], src_ip=20.20.20.100, dst_ip=10.10.10.100) 8. ping reply back to host-2, on br-vlan, it hits the output flows which match the dvr_host_mac-[04], then the VM-2's mac address will never be learnt. So, ping request is flooding on br-vlan forever. To manage notification
[Yahoo-eng-team] [Bug 1990840] Re: Log Agent rpc_loop very noisy
*** This bug is a duplicate of bug 1988077 *** https://bugs.launchpad.net/bugs/1988077 ** Changed in: neutron Importance: Undecided => Wishlist ** Changed in: neutron Status: New => Opinion ** This bug has been marked a duplicate of bug 1988077 Noisy neutron-openvswitch-agent service -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1990840 Title: Log Agent rpc_loop very noisy Status in neutron: Opinion Bug description: Hello, since the train release the openvswitch agent is very noisy related to "Agent rpc_loop" logs. I found that this was done with the following Change [1]. This results in at least >=2 messages per second per agent which can be a very big number of logs in a production environment. I wanted to discuss if these messages really need to be in the info logging as for me this is just noise and not useful information. If I had a problem I myself would restart an agent into debug mode to get deeper into the problem. For me at least either those messages should stay in debug mode or maybe be logged if the time it took for the rpc loop exceeded some threshold. Any opinions on this? [1] https://github.com/openstack/neutron/commit/8e73de8bc42067c0a6796df3cca9938d25ae754e To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1990840/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1964342] [NEW] [QoS][OvS] implement QoS bandwidth limitation by leveraging ovs meter
Public bug reported: We are going to implement packet rate limit on ovs bridge by using meter rules [1], at the same time, meter can also be used to limit the bandwidth. os-key(ryu) supports the rule types of OFPMF_KBPS [2]. And usually, some smart-NICs for ovs offloading will support offloading meter rule to NIC to save the QoS cost of CPU on hosts naturally. Since TC rules can be offloaded to smart-NIC as well. So this is an alternative to make the implementation of QoS limitation in a consistent mechanism. thought? [1] https://review.opendev.org/c/openstack/neutron/+/816800/8/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py [2] https://ryu.readthedocs.io/en/latest/ofproto_v1_3_ref.html#ryu.ofproto.ofproto_v1_3_parser.OFPMeterMod ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1964342 Title: [QoS][OvS] implement QoS bandwidth limitation by leveraging ovs meter Status in neutron: New Bug description: We are going to implement packet rate limit on ovs bridge by using meter rules [1], at the same time, meter can also be used to limit the bandwidth. os-key(ryu) supports the rule types of OFPMF_KBPS [2]. And usually, some smart-NICs for ovs offloading will support offloading meter rule to NIC to save the QoS cost of CPU on hosts naturally. Since TC rules can be offloaded to smart-NIC as well. So this is an alternative to make the implementation of QoS limitation in a consistent mechanism. thought? [1] https://review.opendev.org/c/openstack/neutron/+/816800/8/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py [2] https://ryu.readthedocs.io/en/latest/ofproto_v1_3_ref.html#ryu.ofproto.ofproto_v1_3_parser.OFPMeterMod To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1964342/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1961011] [NEW] [RFE][L3][OVS] use controller (ovs-agent) send (packet-out) RA to (VMs) ports
Public bug reported:
Currently neutron l3 router will run radvd to send out RA packets about
the ManagedFlag, LinkMTU and prefix of IPv6 subnet. But rememeber we
have a distributed SDN controller, aka ovs-agent, which can do this work
more naturally and gracefully.
Current radvd config looks like this:
interface qr-8fd65326-c4
{
   AdvSendAdvert on;
   MinRtrAdvInterval 30;
   MaxRtrAdvInterval 100;
   AdvLinkMTU 1500;
   AdvManagedFlag on;
   prefix fda7:a5cc:3460:3::/64
   {
AdvOnLink on;
AdvAutonomous off;
   };
};
While RYU supports Router Advertisement:
https://ryu.readthedocs.io/en/latest/library_packet_ref/packet_icmpv6.html#ryu.lib.packet.icmpv6.nd_router_advert
This can narrow down the burden of L3-agent management and run less
process for a router.
** Affects: neutron
Importance: Undecided
Status: New
** Description changed:
Currently neutron l3 router will run radvd to send out RA packets about
- the ManagedFlag, LinkMTU, prefix of IPv6 subnet. But rememeber we have a
- distributed SND controller, aka ovs-agent, which can do this work more
- naturally and gracefully.
+ the ManagedFlag, LinkMTU and prefix of IPv6 subnet. But rememeber we
+ have a distributed SDN controller, aka ovs-agent, which can do this work
+ more naturally and gracefully.
Current radvd config looks like this:
- radvd.conf
interface qr-8fd65326-c4
{
-AdvSendAdvert on;
-MinRtrAdvInterval 30;
-MaxRtrAdvInterval 100;
-AdvLinkMTU 1500;
-AdvManagedFlag on;
-prefix fda7:a5cc:3460:3::/64
-{
- AdvOnLink on;
- AdvAutonomous off;
-};
+ Â Â Â AdvSendAdvert on;
+ Â Â Â MinRtrAdvInterval 30;
+ Â Â Â MaxRtrAdvInterval 100;
+ Â Â Â AdvLinkMTU 1500;
+ Â Â Â AdvManagedFlag on;
+ Â Â Â prefix fda7:a5cc:3460:3::/64
+ Â Â Â {
+ AdvOnLink on;
+ AdvAutonomous off;
+ Â Â Â };
};
While RYU supports Router Advertisement:
https://ryu.readthedocs.io/en/latest/library_packet_ref/packet_icmpv6.html#ryu.lib.packet.icmpv6.nd_router_advert
- This can narrow down the burden of L3-agent management, run less process
- for a router.
+ This can narrow down the burden of L3-agent management and run less
+ process for a router.
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1961011
Title:
[RFE][L3][OVS] use controller (ovs-agent) send (packet-out) RA to
(VMs) ports
Status in neutron:
New
Bug description:
Currently neutron l3 router will run radvd to send out RA packets
about the ManagedFlag, LinkMTU and prefix of IPv6 subnet. But
rememeber we have a distributed SDN controller, aka ovs-agent, which
can do this work more naturally and gracefully.
Current radvd config looks like this:
interface qr-8fd65326-c4
{
   AdvSendAdvert on;
   MinRtrAdvInterval 30;
   MaxRtrAdvInterval 100;
   AdvLinkMTU 1500;
   AdvManagedFlag on;
   prefix fda7:a5cc:3460:3::/64
   {
AdvOnLink on;
AdvAutonomous off;
   };
};
While RYU supports Router Advertisement:
https://ryu.readthedocs.io/en/latest/library_packet_ref/packet_icmpv6.html#ryu.lib.packet.icmpv6.nd_router_advert
This can narrow down the burden of L3-agent management and run less
process for a router.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1961011/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1952867] [NEW] [ml2][ovs] allow multiple physical networks map to one physical ovs bridge
Public bug reported:
In real cloud production environment, there are many hosts, which can
access external network, which may not. Some have enough NICs to work
for different networks, while some are lack of NICs.
For instance, an external network, provider:network_type is ``vlan``,
provider:physical_network is ``external``, provider:segmentation_id is
``4000``.
While tenant network, provider:network_type is ``vlan``,
provider:physical_network is ``user``, provider:segmentation_id is
``1000-3000``.
So for neutron's limitation, in one single host, you have to add two
ovs-bridges which are mapping external->br-ex, user->br-usr。While br-ex
add physical port eth0, and br-usr and eth1.
But, in real world, these vlans can work in same physical nic, and physical
hosts may be lack of NICs, which means, Neutron should allow set bridge mapping
like this:
{"external": br-vlan, "user": br-vlan}
Then, for those hosts with only one NIC (or one bond NIC), can work for
both physical types of network.
You may say, may be we can set one network with two types of
"provider:physical_network". One single network is currently type uniq.
This will be a bit more complicated than the former solution. This needs
not only neutron-server side DB model changes, but also agent side
change. While the former may only need agent side change to allow set
that mappings.
Any ideas?
** Affects: neutron
Importance: Undecided
Status: New
** Summary changed:
- [ml2][ovs] allow physical map to one physical ovs bridge
+ [ml2][ovs] allow multiple physical networks map to one physical ovs bridge
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1952867
Title:
[ml2][ovs] allow multiple physical networks map to one physical ovs
bridge
Status in neutron:
New
Bug description:
In real cloud production environment, there are many hosts, which can
access external network, which may not. Some have enough NICs to work
for different networks, while some are lack of NICs.
For instance, an external network, provider:network_type is ``vlan``,
provider:physical_network is ``external``, provider:segmentation_id is
``4000``.
While tenant network, provider:network_type is ``vlan``,
provider:physical_network is ``user``, provider:segmentation_id is
``1000-3000``.
So for neutron's limitation, in one single host, you have to add two
ovs-bridges which are mapping external->br-ex, user->br-usr。While br-
ex add physical port eth0, and br-usr and eth1.
But, in real world, these vlans can work in same physical nic, and physical
hosts may be lack of NICs, which means, Neutron should allow set bridge mapping
like this:
{"external": br-vlan, "user": br-vlan}
Then, for those hosts with only one NIC (or one bond NIC), can work
for both physical types of network.
You may say, may be we can set one network with two types of
"provider:physical_network". One single network is currently type
uniq. This will be a bit more complicated than the former solution.
This needs not only neutron-server side DB model changes, but also
agent side change. While the former may only need agent side change
to allow set that mappings.
Any ideas?
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1952867/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1952567] [NEW] [ml2][ovs] ports tag are missing and flood on those
Public bug reported: During some ml2 ovs agent port processing performance test, we noticed that some ports are missing tag before it really done processing. While ovs treats those ports without tag as trunk port, so some packets will be flooded to it. In large scale cloud, if too many port added to the bridge, the ovs-vswitchd will consume a huge amount of CPU cores if ports are not bound in a short time. Another potential problem is openflow security group may not get processed during the first created event. Upstream test failures of waiting too long time to ping some cases, may be related to these problems. ** Affects: neutron Importance: High Assignee: LIU Yulong (dragon889) Status: In Progress ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Status: New => In Progress ** Changed in: neutron Assignee: (unassigned) => LIU Yulong (dragon889) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1952567 Title: [ml2][ovs] ports tag are missing and flood on those Status in neutron: In Progress Bug description: During some ml2 ovs agent port processing performance test, we noticed that some ports are missing tag before it really done processing. While ovs treats those ports without tag as trunk port, so some packets will be flooded to it. In large scale cloud, if too many port added to the bridge, the ovs-vswitchd will consume a huge amount of CPU cores if ports are not bound in a short time. Another potential problem is openflow security group may not get processed during the first created event. Upstream test failures of waiting too long time to ping some cases, may be related to these problems. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1952567/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1940226] [NEW] OVN related functional cases keep getting failed recently
Public bug reported: OVN related cases keep getting failed recently, examples: [1] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d0a/804378/3/check/neutron-functional-with-uwsgi/d0a39b4/testr_results.html [2] https://d94b4cd84b6a46f0b36c-82f789b81dd771ff819fc3b404846d37.ssl.cf2.rackcdn.com/804213/4/check/neutron-functional-with-uwsgi/6b8e4e8/testr_results.html [3] https://af8afce8abda9555a29f-599837b0db2c318a2a67088a328a8d15.ssl.cf2.rackcdn.com/804394/3/check/neutron-functional-with-uwsgi/cc90586/testr_results.html ** Affects: neutron Importance: Undecided Status: New ** Summary changed: - OVN related cases keep getting failed recently + OVN related functional cases keep getting failed recently -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1940226 Title: OVN related functional cases keep getting failed recently Status in neutron: New Bug description: OVN related cases keep getting failed recently, examples: [1] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d0a/804378/3/check/neutron-functional-with-uwsgi/d0a39b4/testr_results.html [2] https://d94b4cd84b6a46f0b36c-82f789b81dd771ff819fc3b404846d37.ssl.cf2.rackcdn.com/804213/4/check/neutron-functional-with-uwsgi/6b8e4e8/testr_results.html [3] https://af8afce8abda9555a29f-599837b0db2c318a2a67088a328a8d15.ssl.cf2.rackcdn.com/804394/3/check/neutron-functional-with-uwsgi/cc90586/testr_results.html To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1940226/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1938966] [NEW] [RFE] pps limitation on neutron ports for openvswitch agent
Public bug reported: Since neutron supports packet rate limit rule [1][2], it's time for us to support real pps limitation in agent side for neutron ports and IPs. So this RFE is for real pps limitation functionality. We are going to implement neutron port's pps limitation with ovs meter first. IPs pps limitaion will be considered in a new approach. Implementation notes [1]: 1. ovs kernel datapath based on kernel version <= 4.14 does not support meter. 2. kernel version >=4.15 + ovs >= 2.10 will support meter. 3. user space ovs data path should use ovs with version >= 2.7 So, this will be only available for kernel >=4.15 and ovs>=2.10. Basic limit pipeline is: Ingress: packets get into br-int table 0, before send to table 60, check the destanation MAC and local_vlan ID, if the dest is resident in this host, do the meter pps action and send to table 60. Egress: match src MAC and in_port, before send to table 60, do the meter pps action and send to table 60. Neutron ovs-agent process workflow: 1. port pluging detected 2. reterieve the port information 3. do the flow installation 4. handle_port in the pps limitation QoS agent extension (or a new pps agent extension) 5. create the meter rule for port based on the binding qos policy's QosPacketRateLimitRule 6. applied the Ingress and Egress ovs flows [1] https://bugs.launchpad.net/neutron/+bug/1912460 [2] https://review.opendev.org/c/openstack/neutron/+/796363 [3] https://github.com/openvswitch/ovs/blob/master/Documentation/faq/releases.rst ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1938966 Title: [RFE] pps limitation on neutron ports for openvswitch agent Status in neutron: New Bug description: Since neutron supports packet rate limit rule [1][2], it's time for us to support real pps limitation in agent side for neutron ports and IPs. So this RFE is for real pps limitation functionality. We are going to implement neutron port's pps limitation with ovs meter first. IPs pps limitaion will be considered in a new approach. Implementation notes [1]: 1. ovs kernel datapath based on kernel version <= 4.14 does not support meter. 2. kernel version >=4.15 + ovs >= 2.10 will support meter. 3. user space ovs data path should use ovs with version >= 2.7 So, this will be only available for kernel >=4.15 and ovs>=2.10. Basic limit pipeline is: Ingress: packets get into br-int table 0, before send to table 60, check the destanation MAC and local_vlan ID, if the dest is resident in this host, do the meter pps action and send to table 60. Egress: match src MAC and in_port, before send to table 60, do the meter pps action and send to table 60. Neutron ovs-agent process workflow: 1. port pluging detected 2. reterieve the port information 3. do the flow installation 4. handle_port in the pps limitation QoS agent extension (or a new pps agent extension) 5. create the meter rule for port based on the binding qos policy's QosPacketRateLimitRule 6. applied the Ingress and Egress ovs flows [1] https://bugs.launchpad.net/neutron/+bug/1912460 [2] https://review.opendev.org/c/openstack/neutron/+/796363 [3] https://github.com/openvswitch/ovs/blob/master/Documentation/faq/releases.rst To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1938966/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1937039] [NEW] dsvm-functional can not run due to some ovs/ovn commands not found
Public bug reported:
The dsvm-functional test can not run even for those test cases not
involved with OVN.
$ tox -e dsvm-functional
neutron.tests.functional.db.test_migrations.TestModelsMigrationsMysql
dsvm-functional develop-inst-noop: /opt/stack/neutron
dsvm-functional installed:
...
...
=
Failures during discovery
=
--- import errors ---
Failed to import test module:
neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_impl_idl
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib64/python3.6/unittest/loader.py", line 369, in
_get_module_from_name
__import__(name)
File
"/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_impl_idl.py",
line 21, in
from ovsdbapp.tests.functional import base
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/tests/functional/base.py",
line 23, in
class FunctionalTestCase(base.TestCase):
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/tests/functional/base.py",
line 28, in FunctionalTestCase
remove=not bool(os.getenv('KEEP_VENV')))
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/venv.py",
line 165, in __init__
[self.SBSCHEMA, self.NBSCHEMA])
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/venv.py",
line 75, in _share_path
", ".join(paths + (str(override),)))
Exception: Invalid directories: /usr/local/share/ovn, /usr/share/ovn,
/usr/local/share/openvswitch, /usr/share/openvswitch, None
Failed to import test module:
neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib64/python3.6/unittest/loader.py", line 369, in
_get_module_from_name
__import__(name)
File
"/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py",
line 24, in
from ovsdbapp.tests.functional import base as ovs_base
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/tests/functional/base.py",
line 23, in
class FunctionalTestCase(base.TestCase):
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/tests/functional/base.py",
line 28, in FunctionalTestCase
remove=not bool(os.getenv('KEEP_VENV')))
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/venv.py",
line 165, in __init__
[self.SBSCHEMA, self.NBSCHEMA])
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/venv.py",
line 75, in _share_path
", ".join(paths + (str(override),)))
Exception: Invalid directories: /usr/local/share/ovn, /usr/share/ovn,
/usr/local/share/openvswitch, /usr/share/openvswitch, None
The above traceback was encountered during test discovery which imports all the
found test modules in the specified test_path.
ERROR: InvocationError for command
/opt/stack/neutron/.tox/dsvm-functional/bin/stestr run
'--group_regex=neutron\.tests\.functional\.db\.test_migrations\.(TestModelsMigrationsPsql|TestModelsMigrationsMysql)'
neutron.tests.functional.db.test_migrations.TestModelsMigrationsMysql (exited
with code 100)
__ summary
__
ERROR: dsvm-functional: commands failed
After remove the cases:
$ git status
On branch packet_rate_limit
Changes not staged for commit:
(use "git add/rm ..." to update what will be committed)
(use "git restore ..." to discard changes in working directory)
deleted:neutron/tests/functional/agent/ovn/__init__.py
deleted:neutron/tests/functional/agent/ovn/metadata/__init__.py
deleted:
neutron/tests/functional/agent/ovn/metadata/test_metadata_agent.py
deleted:neutron/tests/functional/plugins/ml2/drivers/ovn/__init__.py
deleted:
neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/__init__.py
deleted:
neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/__init__.py
deleted:
neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/extensions/__init__.py
deleted:
neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/extensions/test_qos.py
deleted:
neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_impl_idl.py
deleted:
neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py
deleted:
neu
[Yahoo-eng-team] [Bug 1934948] [NEW] [RFE] refactor of L3 resources update procedure
Public bug reported: In the L3 meeting 2021-06-30, I mentioned this topic. https://meetings.opendev.org/meetings/neutron_l3/2021/neutron_l3.2021-06-30-14.00.log.html#l-28 Current L3 resources (floating IPs, router interface, router external gateway) processing procedure is a bit heavy, and sometimes waste of times. For instance, if one floating IP is bind to a port under one router, the steps are: 1. floating IP updated 2. notify L3 agent that this floating IPs router is updated 3. L3 agent sync the router info 4. L3 agent reprocess all router related resources The alternative is we can use resource cache for router related resources, then the procedure for floating IP update can be changed to: 1. floating IP updated. 2. OVO object update event send out. 3. L3 agents which has the related router resident will do the processing of floating IP only, no sync of full router info, no processing of all router resources again! This can be a huge performace improvement for L3 related resources processing. Thoughts? ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1934948 Title: [RFE] refactor of L3 resources update procedure Status in neutron: New Bug description: In the L3 meeting 2021-06-30, I mentioned this topic. https://meetings.opendev.org/meetings/neutron_l3/2021/neutron_l3.2021-06-30-14.00.log.html#l-28 Current L3 resources (floating IPs, router interface, router external gateway) processing procedure is a bit heavy, and sometimes waste of times. For instance, if one floating IP is bind to a port under one router, the steps are: 1. floating IP updated 2. notify L3 agent that this floating IPs router is updated 3. L3 agent sync the router info 4. L3 agent reprocess all router related resources The alternative is we can use resource cache for router related resources, then the procedure for floating IP update can be changed to: 1. floating IP updated. 2. OVO object update event send out. 3. L3 agents which has the related router resident will do the processing of floating IP only, no sync of full router info, no processing of all router resources again! This can be a huge performace improvement for L3 related resources processing. Thoughts? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1934948/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1934646] [NEW] fullstack fails locally after several times run due to shared dhclient lease file
Public bug reported:
Each cases are sharing the common lease path for dhclient,
for instance, in CentOS it is: /var/lib/dhclient/dhclient.leases.
That means all fullstack cases will use this file to store
fake VM's NIC DHCP lease information.
After run several times of fullstack cases, the dhclient will
get failed to set the test fake VM port's IP due to the mess
settings in this file. Errors are:
"""
# ip netns exec test-f00c713e-97df-440a-9bd0-e88a0bc5ab38 dhclient -4 -sf
/opt/stack/neutron/.tox/dsvm-fullstack/bin/fullstack-dhclient-script --no-pid
-d port71fc1d
Internet Systems Consortium DHCP Client 4.2.5
Copyright 2004-2013 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Can't allocate interface portd8lease {
interface .
This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.
Please report for this software via the CentOS Bugs Database:
http://bugs.centos.org/
exiting.
"""
The mess settings looks like this:
}
lease {
interface "portb88816 {
interface "port71fd02";
fixed-address 20.0.0.115;
...
There is a "{" after the port name.
Looks like there is a race condition among different cases, so this file
is rendered with broken settings.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1934646
Title:
fullstack fails locally after several times run due to shared dhclient
lease file
Status in neutron:
New
Bug description:
Each cases are sharing the common lease path for dhclient,
for instance, in CentOS it is: /var/lib/dhclient/dhclient.leases.
That means all fullstack cases will use this file to store
fake VM's NIC DHCP lease information.
After run several times of fullstack cases, the dhclient will
get failed to set the test fake VM port's IP due to the mess
settings in this file. Errors are:
"""
# ip netns exec test-f00c713e-97df-440a-9bd0-e88a0bc5ab38 dhclient -4 -sf
/opt/stack/neutron/.tox/dsvm-fullstack/bin/fullstack-dhclient-script --no-pid
-d port71fc1d
Internet Systems Consortium DHCP Client 4.2.5
Copyright 2004-2013 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Can't allocate interface portd8lease {
interface .
This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.
Please report for this software via the CentOS Bugs Database:
http://bugs.centos.org/
exiting.
"""
The mess settings looks like this:
}
lease {
interface "portb88816 {
interface "port71fd02";
fixed-address 20.0.0.115;
...
There is a "{" after the port name.
Looks like there is a race condition among different cases, so this
file is rendered with broken settings.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1934646/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1934466] [NEW] Devstack install for ML2 OVS
Public bug reported: Since devstack had set OVN as the default backend for Neutron. Then the minimum local.conf [1] for ML2 ovs will not work at all. For some local testing of ML2 OVS, it is not right deployment for users to test the ML2 OVS related cases. Neutron needs to add a sample local.conf for ML2 OVS to install a small all in one environment by using devstack. Then cloud users can use it to have a try for openstack and testing. (This is a bug track for patch https://review.opendev.org/c/openstack/neutron/+/799159) [1] https://docs.openstack.org/devstack/latest/#create-a-local-conf ** Affects: neutron Importance: Undecided Assignee: LIU Yulong (dragon889) Status: In Progress ** Changed in: neutron Assignee: (unassigned) => LIU Yulong (dragon889) ** Changed in: neutron Status: New => In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1934466 Title: Devstack install for ML2 OVS Status in neutron: In Progress Bug description: Since devstack had set OVN as the default backend for Neutron. Then the minimum local.conf [1] for ML2 ovs will not work at all. For some local testing of ML2 OVS, it is not right deployment for users to test the ML2 OVS related cases. Neutron needs to add a sample local.conf for ML2 OVS to install a small all in one environment by using devstack. Then cloud users can use it to have a try for openstack and testing. (This is a bug track for patch https://review.opendev.org/c/openstack/neutron/+/799159) [1] https://docs.openstack.org/devstack/latest/#create-a-local-conf To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1934466/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1933222] [NEW] [RFE] Add distributed datapath for metadata
Public bug reported: When instances are booting, they will try to retrieve metadata from Nova by the path of Neutron virtual switches(bridges), virtual devices, namespaces and metadata-agents. After that, metadata agent has no other functionalities. In large-scale scenarios, a large number of deployed metadata agents will cause a waste of resources for hosts and message queue. Because they will start tons of external processes based on the number of users' resources, report state to Neutron server for heartbeat keepalive. How many metadata-agent should run for a large scale cloud deployment? There is no exact answer to this question. Cloud operators may setup metadata agent to all hosts, something like DHCP agent. Config drive can be an alternative for clouds to supply metadata for VMs. But what if users do not want to add a cd-rom device to VM? So, I'd like to request for implementing an agent extension for Neutron openvswitch agent to make the metadata datapath distributed. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1933222 Title: [RFE] Add distributed datapath for metadata Status in neutron: New Bug description: When instances are booting, they will try to retrieve metadata from Nova by the path of Neutron virtual switches(bridges), virtual devices, namespaces and metadata-agents. After that, metadata agent has no other functionalities. In large-scale scenarios, a large number of deployed metadata agents will cause a waste of resources for hosts and message queue. Because they will start tons of external processes based on the number of users' resources, report state to Neutron server for heartbeat keepalive. How many metadata-agent should run for a large scale cloud deployment? There is no exact answer to this question. Cloud operators may setup metadata agent to all hosts, something like DHCP agent. Config drive can be an alternative for clouds to supply metadata for VMs. But what if users do not want to add a cd-rom device to VM? So, I'd like to request for implementing an agent extension for Neutron openvswitch agent to make the metadata datapath distributed. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1933222/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1932483] [NEW] CI neutron.tests.functional.services.l3_router.test_l3_dvr get failed frequently
Public bug reported: These 4 cases get failed frequently neutron.tests.functional.services.l3_router.test_l3_dvr_ha_router_plugin.L3DvrHATestCase 1) test_agent_gw_port_delete_when_last_gateway_for_ext_net_removed 2) test_delete_agent_gw_port_for_network neutron.tests.functional.services.l3_router.test_l3_dvr_router_plugin.L3DvrTestCase 3) test_agent_gw_port_delete_when_last_gateway_for_ext_net_removed 4) test_delete_agent_gw_port_for_network CI examples: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_152/796404/3/check/neutron-functional-with-uwsgi/1520594/testr_results.html https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_522/796363/5/check/neutron-functional-with-uwsgi/522268d/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3c2/780055/13/check/neutron-functional-with-uwsgi/3c2a086/testr_results.html ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1932483 Title: CI neutron.tests.functional.services.l3_router.test_l3_dvr get failed frequently Status in neutron: New Bug description: These 4 cases get failed frequently neutron.tests.functional.services.l3_router.test_l3_dvr_ha_router_plugin.L3DvrHATestCase 1) test_agent_gw_port_delete_when_last_gateway_for_ext_net_removed 2) test_delete_agent_gw_port_for_network neutron.tests.functional.services.l3_router.test_l3_dvr_router_plugin.L3DvrTestCase 3) test_agent_gw_port_delete_when_last_gateway_for_ext_net_removed 4) test_delete_agent_gw_port_for_network CI examples: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_152/796404/3/check/neutron-functional-with-uwsgi/1520594/testr_results.html https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_522/796363/5/check/neutron-functional-with-uwsgi/522268d/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3c2/780055/13/check/neutron-functional-with-uwsgi/3c2a086/testr_results.html To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1932483/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1931844] Re: Can't ping router with packet size greater than 1476 when ovs datapath_type set to netdev
"ovs datapath_type netdev" should only be used for VM, neutron router
related virtual network devices are not compatible with it, [1] has
those limitations. The only way you can run L3 routers with VMs (using
DPDK) is to run l3-agents and ovs-agents in dedicated nodes with data
patch type system, while VMs in hosts with netdev.
[1] https://docs.openstack.org/neutron/latest/admin/config-ovs-dpdk.html
#known-limitations
** Changed in: neutron
Status: New => Won't Fix
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1931844
Title:
Can't ping router with packet size greater than 1476 when ovs
datapath_type set to netdev
Status in neutron:
Won't Fix
Bug description:
Setup:
  Openstack origin/stable/wallaby deployed by devstack on single node
  [stack@localhost devstack]$ cat /etc/centos-release
  CentOS Linux release 8.4.2105
  [stack@localhost devstack]$ uname -a
  Linux localhost.localdomain 4.18.0-305.3.1.el8.x86_64 #1 SMP Tue Jun 1
16:14:33 UTC 2021 x86_64
  x86_64 x86_64 GNU/Linux
  [stack@localhost devstack]$
  [stack@localhost devstack]$ git status
  HEAD detached at origin/stable/wallaby
  ovs version (builded with DPDK)
  [stack@localhost devstack]$ ovs-vswitchd --version
  ovs-vswitchd (Open vSwitch) 2.13.4
  DPDK 19.11.7
  [stack@localhost devstack]$ cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep
datapath_type
  datapath_type = netdev
  [stack@localhost devstack]$ cat /etc/neutron/plugins/ml2/ml2_conf.ini |grep
mtu
  path_mtu = 9000
  physical_network_mtus = 9000
  [stack@localhost devstack]$ cat /etc/neutron/neutron.conf |grep mtu
  global_physnet_mtu = 9000
Steps:
  1. Setup hugepages, update flavor (hw:mem_page_size='large',
hw_rng:allowed='True')
  2. boot vm and try ping router and vice versa
br-int:
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: netdev
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Port vhua5beaaae-c8
tag: 1
Interface vhua5beaaae-c8
type: dpdkvhostuserclient
options:
{vhost-server-path="/var/run/openvswitch/vhua5beaaae-c8"}
Port qr-c48b27ee-0f
tag: 1
Interface qr-c48b27ee-0f
type: internal
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port tap3b960ab3-6a
tag: 1
Interface tap3b960ab3-6a
type: internal
From VM:
[root@test-mtu ~]# ip a |grep mtu
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1000
2: eth0: mtu 8950 qdisc fq_codel state UP
group default qlen 1000
[root@test-mtu ~]#
[root@test-mtu ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.275 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1035ms
rtt min/avg/max/mdev = 0.167/0.221/0.275/0.054 ms
[root@test-mtu ~]# ping 192.168.1.1 -s 2000
PING 192.168.1.1 (192.168.1.1) 2000(2028) bytes of data.
--- 192.168.1.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3086ms
[root@test-mtu ~]# ping 192.168.1.1 -c 5
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.139 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.154 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.144 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.153 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=0.146 ms
--- 192.168.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 0.139/0.147/0.154/0.005 ms
[root@test-mtu ~]# ping 192.168.1.1 -c 5 -s 2000
PING 192.168.1.1 (192.168.1.1) 2000(2028) bytes of data.
--- 192.168.1.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4084ms
[root@test-mtu ~]#
In ns:
[root@localhost ~]# ip a |grep mtu
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1000
29: qr-c48b27ee-0f: mtu 8950 qdisc
fq_codel state UNKNOWN group default qlen 1000
[root@localhost ~]#
# default packet size:
10:44:41.276140 IP 192.168.1.176 > localhost.localdomain: ICMP echo request,
id 7, seq 1, length 64
10:44:41.276176 IP localhost.localdomain > 192.168.1.176: ICMP echo reply, id
7, seq 1, length 64
10:44:42.323564 IP 192.168.1.176 > localhost.
[Yahoo-eng-team] [Bug 1811352] Re: [RFE] Include neutron CLI floatingip port-forwarding support
** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1811352 Title: [RFE] Include neutron CLI floatingip port-forwarding support Status in neutron: Fix Released Bug description: The floating ip port-forwarding is supported since Rocky by neutron API: https://developer.openstack.org/api-ref/network/v2/index.html?expanded =create-port-forwarding-detail#floating-ips-port-forwarding but the neutron client does not include this option yet. Also floatingip-update method is missing. It should include an option similar to this for floatingip *-create/*-update $ neutron floatingip-create --port-forwarding protocol=tcp,internal_port_id=,internal_port=,external_port= You should be able to repeat --port-forwarding several times Also for floatingip-update with and extra option: --port-forwarding clean To remove the current port-forwarding rules list. * Version: OpenStack Rocky To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1811352/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1930432] [NEW] [L2] provisioning_block should be added to Neutron internal service port? Or should not?
Public bug reported: provisioning_blocks are mostly used for compute port to notify that Neutron has done the networking settings, so VM can go power-on now. But now, Neutron does not check the port's device owner, it adds provisioning_block to all types of ports, even it is used by neutron. For some internal service ports, neutron does not notify anything to neutron itself, but the provisioning_block may cause some timeout for resource setup. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1930432 Title: [L2] provisioning_block should be added to Neutron internal service port? Or should not? Status in neutron: New Bug description: provisioning_blocks are mostly used for compute port to notify that Neutron has done the networking settings, so VM can go power-on now. But now, Neutron does not check the port's device owner, it adds provisioning_block to all types of ports, even it is used by neutron. For some internal service ports, neutron does not notify anything to neutron itself, but the provisioning_block may cause some timeout for resource setup. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1930432/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1926109] [NEW] SSH timeout (wait timeout) due to potential paramiko issue
Public bug reported:
Recently, nothing changed on the test case, but we got this failure:
https://04a9f9fdd9afdf12de4e-f889a65b4dfb1f628c8309e9eb44b225.ssl.cf2.rackcdn.com/787304/5/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/bb9e9a9/testr_results.html
LOG:
Traceback (most recent call last):
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py",
line 160, in test_port_forwarding_editing_and_deleting_tcp_rule
utils.wait_until_true(
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py",
line 79, in wait_until_true
while not predicate():
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py",
line 153, in no_fip_pf_connectivity
return not fip_pf_connectivity(6)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py",
line 146, in fip_pf_connectivity
self.check_servers_hostnames(
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/base.py",
line 537, in check_servers_hostnames
ssh_client.get_hostname())
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py",
line 292, in get_hostname
return self.exec_command('hostname')
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 333, in wrapped_f
return self(f, *args, **kw)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 423, in __call__
do = self.iter(retry_state=retry_state)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 360, in iter
return fut.result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in
__get_result
raise self._exception
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 426, in __call__
result = fn(*args, **kwargs)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py",
line 171, in exec_command
return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 162, in exec_command
channel.exec_command(cmd)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py",
line 72, in _check
return func(self, *args, **kwds)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py",
line 257, in exec_command
self._wait_for_event()
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py",
line 1219, in _wait_for_event
self.event.wait()
File "/usr/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
File "/usr/lib/python3.8/threading.py", line 302, in wait
waiter.acquire()
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/fixtures/_fixtures/timeout.py",
line 52, in signal_handler
raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException
Some search in the neutron LP bug list, there are some old bugs related to this
project "paramiko":
https://bugs.launchpad.net/neutron/+bug/1892861
https://bugs.launchpad.net/neutron/+bug/1571486
https://bugs.launchpad.net/neutron/+bug/1253896
The issue seems to be _wait_for_event is waiting for something, and
finally get timeout.
After some searching in the repo of paramiko in github, I saw this issue:
https://github.com/paramiko/paramiko/pull/1248
With a commit for this:
https://github.com/paramiko/paramiko/pull/1248/commits/b059139ea712a46f632fe20e3c278414f311c21d
But it is not included in any release version.
Some thoughts:
1. if it is possible to not use paramiko?
2. run (ssh) commands directly by some oslo libs?
3. ping paramiko maintainers to release once to see if the test success rate
can get better
** Affects: neutron
Importance: Undecided
Status: New
** Affects: tempest
Importance: Undecided
Status: New
** Also affects: tempest
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1926109
Title:
SSH timeout (wait timeout) due to potential paramiko issue
Status in neutron:
New
Status in tempest:
New
Bug description:
Recently, nothing changed on the test case, but we got this failure:
https://04a9f9fdd9afdf12de4e-f889a65b4dfb1f628c8309e9eb44b225.ssl.cf2.rackcdn.com/787304/5/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/bb9e9a9/testr_results.html
[Yahoo-eng-team] [Bug 1925406] [NEW] [functional] run dsvm-functional locally will need ovn no matter the running cases are
Public bug reported:
Code: neutron master
Env: a new deployment by devstack
$ tox -e dsvm-functional
neutron.tests.functional.agent.test_firewall.FirewallTestCase.test_rule_ordering_correct
=
Failures during discovery
=
--- import errors ---
Failed to import test module:
neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_impl_idl
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib64/python3.6/unittest/loader.py", line 369, in
_get_module_from_name
__import__(name)
File
"/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_impl_idl.py",
line 20, in
from ovsdbapp.tests.functional import base
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/tests/functional/base.py",
line 23, in
class FunctionalTestCase(base.TestCase):
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/tests/functional/base.py",
line 28, in FunctionalTestCase
remove=not bool(os.getenv('KEEP_VENV')))
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/venv.py",
line 165, in __init__
[self.SBSCHEMA, self.NBSCHEMA])
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/venv.py",
line 75, in _share_path
", ".join(paths + (str(override),)))
Exception: Invalid directories: /usr/local/share/ovn, /usr/share/ovn,
/usr/local/share/openvswitch, /usr/share/openvswitch, None
Failed to import test module:
neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib64/python3.6/unittest/loader.py", line 369, in
_get_module_from_name
__import__(name)
File
"/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py",
line 24, in
from ovsdbapp.tests.functional import base as ovs_base
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/tests/functional/base.py",
line 23, in
class FunctionalTestCase(base.TestCase):
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/tests/functional/base.py",
line 28, in FunctionalTestCase
remove=not bool(os.getenv('KEEP_VENV')))
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/venv.py",
line 165, in __init__
[self.SBSCHEMA, self.NBSCHEMA])
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/venv.py",
line 75, in _share_path
", ".join(paths + (str(override),)))
Exception: Invalid directories: /usr/local/share/ovn, /usr/share/ovn,
/usr/local/share/openvswitch, /usr/share/openvswitch, None
The above traceback was encountered during test discovery which imports all the
found test modules in the specified test_path.
ERROR: InvocationError for command
/opt/stack/neutron/.tox/dsvm-functional/bin/stestr run
neutron.tests.functional.agent.test_firewall.FirewallTestCase.test_rule_ordering_correct
(exited with code 100)
__ summary
__
ERROR: dsvm-functional: commands failed
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1925406
Title:
[functional] run dsvm-functional locally will need ovn no matter the
running cases are
Status in neutron:
New
Bug description:
Code: neutron master
Env: a new deployment by devstack
$ tox -e dsvm-functional
neutron.tests.functional.agent.test_firewall.FirewallTestCase.test_rule_ordering_correct
=
Failures during discovery
=
--- import errors ---
Failed to import test module:
neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_impl_idl
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib64/python3.6/unittest/loader.py", line 369, in
_get_module_from_name
__import__(name)
File
"/opt/stack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_impl_idl.py",
line 20, in
from ovsdbapp.tests.functional import base
File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/ovsdbapp/tests/functional/base.py",
line 23, in
class Functional
[Yahoo-eng-team] [Bug 1922653] [NEW] [L3][Port forwarding] multiple floating_ip:port to same internal fixed_ip:port (N-to-1 rule support)
Public bug reported:
Floating ip port forwradings table has constraints:
TABLE_NAME = 'portforwardings'
op.create_unique_constraint(
constraint_name=('uniq_port_forwardings0floatingip_id0'
'external_port0protocol'),
table_name=TABLE_NAME,
columns=['floatingip_id', 'external_port', 'protocol']
)
op.create_unique_constraint(
constraint_name=('uniq_port_forwardings0internal_neutron_port_id0'
'socket0protocol'),
table_name=TABLE_NAME,
columns=['internal_neutron_port_id', 'socket', 'protocol']
)
This allows create port forwardings like:
172.24.4.64:22 -> tcp -> 192.168.111.45:22
It does not support (failed on constraint
uniq_port_forwardings0internal_neutron_port_id0socket0protocol):
172.24.4.64:22 -> tcp -> 192.168.111.45:22
172.24.4.64:122 -> tcp -> 192.168.111.45:22
172.24.4.168:22 -> tcp -> 192.168.111.45:22
With some local tests, IMO, all these rules works fine in L3 agent side:
# ip netns exec snat-b247f145-569a-4d5a-bdd8-31a5213641ea conntrack -L |grep
"192.168.111.45"
conntrack v1.4.4 (conntrack-tools): 9 flow entries have been shown.
tcp 6 431835 ESTABLISHED src=172.24.4.1 dst=172.24.4.64 sport=53774
dport=122 src=192.168.111.45 dst=172.24.4.1 sport=22 dport=53774 [ASSURED]
mark=0 use=1
tcp 6 430336 ESTABLISHED src=172.24.4.1 dst=172.24.4.168 sport=53443
dport=22 src=192.168.111.45 dst=172.24.4.1 sport=22 dport=53443 [ASSURED]
mark=0 use=1
tcp 6 431995 ESTABLISHED src=172.24.4.1 dst=172.24.4.64 sport=53781
dport=22 src=192.168.111.45 dst=172.24.4.1 sport=22 dport=53781 [ASSURED]
mark=0 use=1
All rules can be used to login (ssh) the VM.
So here, I'd like to remove the constraint
uniq_port_forwardings0internal_neutron_port_id0socket0protocol to
support these.
** Affects: neutron
Importance: Undecided
Status: New
** Summary changed:
- [L3][Port forwarding] multiple floating_ips to same internal fixed_ip:port
+ [L3][Port forwarding] multiple floating_ip:port to same internal
fixed_ip:port (N-to-1 rule support)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1922653
Title:
[L3][Port forwarding] multiple floating_ip:port to same internal
fixed_ip:port (N-to-1 rule support)
Status in neutron:
New
Bug description:
Floating ip port forwradings table has constraints:
TABLE_NAME = 'portforwardings'
op.create_unique_constraint(
constraint_name=('uniq_port_forwardings0floatingip_id0'
'external_port0protocol'),
table_name=TABLE_NAME,
columns=['floatingip_id', 'external_port', 'protocol']
)
op.create_unique_constraint(
constraint_name=('uniq_port_forwardings0internal_neutron_port_id0'
'socket0protocol'),
table_name=TABLE_NAME,
columns=['internal_neutron_port_id', 'socket', 'protocol']
)
This allows create port forwardings like:
172.24.4.64:22 -> tcp -> 192.168.111.45:22
It does not support (failed on constraint
uniq_port_forwardings0internal_neutron_port_id0socket0protocol):
172.24.4.64:22 -> tcp -> 192.168.111.45:22
172.24.4.64:122 -> tcp -> 192.168.111.45:22
172.24.4.168:22 -> tcp -> 192.168.111.45:22
With some local tests, IMO, all these rules works fine in L3 agent
side:
# ip netns exec snat-b247f145-569a-4d5a-bdd8-31a5213641ea conntrack -L |grep
"192.168.111.45"
conntrack v1.4.4 (conntrack-tools): 9 flow entries have been shown.
tcp 6 431835 ESTABLISHED src=172.24.4.1 dst=172.24.4.64 sport=53774
dport=122 src=192.168.111.45 dst=172.24.4.1 sport=22 dport=53774 [ASSURED]
mark=0 use=1
tcp 6 430336 ESTABLISHED src=172.24.4.1 dst=172.24.4.168 sport=53443
dport=22 src=192.168.111.45 dst=172.24.4.1 sport=22 dport=53443 [ASSURED]
mark=0 use=1
tcp 6 431995 ESTABLISHED src=172.24.4.1 dst=172.24.4.64 sport=53781
dport=22 src=192.168.111.45 dst=172.24.4.1 sport=22 dport=53781 [ASSURED]
mark=0 use=1
All rules can be used to login (ssh) the VM.
So here, I'd like to remove the constraint
uniq_port_forwardings0internal_neutron_port_id0socket0protocol to
support these.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1922653/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1917409] Re: neutron-l3-agents won't become active
*** This bug is a duplicate of bug 1883089 ***
https://bugs.launchpad.net/bugs/1883089
** This bug has been marked a duplicate of bug 1883089
[L3] floating IP failed to bind due to no agent gateway port(fip-ns)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1917409
Title:
neutron-l3-agents won't become active
Status in neutron:
Fix Released
Status in neutron package in Ubuntu:
New
Bug description:
We have a Ubuntu Ussari cloud deployed on Ubuntu 20.04 using the juju
charms from the 20.08 bundle (planning to upgrade soon).
The problem that is occuring that all l3 agents for routers using a
particular external network show up with their ha_state in standby.
I've tried removing and re-adding, and we never see the state go to
active.
$ neutron l3-agent-list-hosting-router bradm-router
neutron CLI is deprecated and will be removed in the future. Use openstack
CLI instead.
+--+-++---+--+
| id | host| admin_state_up | alive
| ha_state |
+--+-++---+--+
| 09ae92c9-ae8f-4209-b1a8-d593cc6d6602 | oschv1.maas | True | :-)
| standby |
| 4d9fe934-b1f8-4c2b-83ea-04971f827209 | oschv2.maas | True | :-)
| standby |
| 70b8b60e-7fbd-4b3a-80a3-90875ca72ce6 | oschv4.maas | True | :-)
| standby |
+--+-++---+--+
This generates a stack trace:
2021-03-01 02:59:47.344 3675486 ERROR neutron.agent.l3.router_info [-]
'NoneType' object has no attribute 'get'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line
165, in _process_incoming
res = self.dispatcher.dispatch(message)
File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 276, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 196, in _do_dispatch
result = func(ctxt, **new_args)
File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 139, in
wrapped
setattr(e, '_RETRY_EXCEEDED', True)
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 135, in
wrapped
return f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 154, in wrapper
ectxt.value = e.inner_exc
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 142, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 183, in
wrapped
LOG.debug("Retry wrapper got retriable exception: %s", e)
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 179, in
wrapped
return f(*dup_args, **dup_kwargs)
File "/usr/lib/python3/dist-packages/neutron/api/rpc/handlers/l3_rpc.py",
line 306, in get_agent_gateway_port
agent_port = self.l3plugin.create_fip_agent_gw_port_if_not_exists(
File "/usr/lib/python3/dist-packages/neutron/db/l3_dvr_db.py", line 1101,
in create_fip_agent_gw_port_if_not_exists
self._populate_mtu_and_subnets_for_ports(context, [agent_port])
File "/usr/lib/python3/dist-packages/neutron/db/l3_db.py", line 1772, in
_populate_mtu_and_subnets_for_ports
network_ids = [p['network_id']
File "/usr/lib/python3/dist-packages/neutron/db/l3_db.py", line 1772, in
network_ids = [p['network_id']
File "/usr/lib/python3/dist-packages/neutron/db/l3_db.py", line 1720, in
_each_port_having_fixed_ips
fixed_ips = port.get('fixed_ips', [])
This system was running successfully after deployment, and
[Yahoo-eng-team] [Bug 1919107] [NEW] [L3][L2 pop] Is it necessary to enforce enabling the option arp_responder and l2_population?
Public bug reported: Recently I played the ovs-agent with bagpipe_bgpvpn. It has some mechanism which will install the same flows to br-tun by using BGP. But now arp_responder and l2_population are enforced, after this patch https://review.opendev.org/c/openstack/neutron/+/669938. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1919107 Title: [L3][L2 pop] Is it necessary to enforce enabling the option arp_responder and l2_population? Status in neutron: New Bug description: Recently I played the ovs-agent with bagpipe_bgpvpn. It has some mechanism which will install the same flows to br-tun by using BGP. But now arp_responder and l2_population are enforced, after this patch https://review.opendev.org/c/openstack/neutron/+/669938. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1919107/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1917409] Re: neutron-l3-agents won't become active
*** This bug is a duplicate of bug 1883089 ***
https://bugs.launchpad.net/bugs/1883089
** This bug has been marked a duplicate of bug 1883089
[L3] floating IP failed to bind due to no agent gateway port(fip-ns)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1917409
Title:
neutron-l3-agents won't become active
Status in neutron:
New
Bug description:
We have a Ubuntu Ussari cloud deployed on Ubuntu 20.04 using the juju
charms from the 20.08 bundle (planning to upgrade soon).
The problem that is occuring that all l3 agents for routers using a
particular external network show up with their ha_state in standby.
I've tried removing and re-adding, and we never see the state go to
active.
$ neutron l3-agent-list-hosting-router bradm-router
neutron CLI is deprecated and will be removed in the future. Use openstack
CLI instead.
+--+-++---+--+
| id | host| admin_state_up | alive
| ha_state |
+--+-++---+--+
| 09ae92c9-ae8f-4209-b1a8-d593cc6d6602 | oschv1.maas | True | :-)
| standby |
| 4d9fe934-b1f8-4c2b-83ea-04971f827209 | oschv2.maas | True | :-)
| standby |
| 70b8b60e-7fbd-4b3a-80a3-90875ca72ce6 | oschv4.maas | True | :-)
| standby |
+--+-++---+--+
This generates a stack trace:
2021-03-01 02:59:47.344 3675486 ERROR neutron.agent.l3.router_info [-]
'NoneType' object has no attribute 'get'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line
165, in _process_incoming
res = self.dispatcher.dispatch(message)
File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 276, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 196, in _do_dispatch
result = func(ctxt, **new_args)
File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 139, in
wrapped
setattr(e, '_RETRY_EXCEEDED', True)
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 135, in
wrapped
return f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 154, in wrapper
ectxt.value = e.inner_exc
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 142, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 183, in
wrapped
LOG.debug("Retry wrapper got retriable exception: %s", e)
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 179, in
wrapped
return f(*dup_args, **dup_kwargs)
File "/usr/lib/python3/dist-packages/neutron/api/rpc/handlers/l3_rpc.py",
line 306, in get_agent_gateway_port
agent_port = self.l3plugin.create_fip_agent_gw_port_if_not_exists(
File "/usr/lib/python3/dist-packages/neutron/db/l3_dvr_db.py", line 1101,
in create_fip_agent_gw_port_if_not_exists
self._populate_mtu_and_subnets_for_ports(context, [agent_port])
File "/usr/lib/python3/dist-packages/neutron/db/l3_db.py", line 1772, in
_populate_mtu_and_subnets_for_ports
network_ids = [p['network_id']
File "/usr/lib/python3/dist-packages/neutron/db/l3_db.py", line 1772, in
network_ids = [p['network_id']
File "/usr/lib/python3/dist-packages/neutron/db/l3_db.py", line 1720, in
_each_port_having_fixed_ips
fixed_ips = port.get('fixed_ips', [])
This system was running successfully after deployment, and has been
left running for a while and when it was
[Yahoo-eng-team] [Bug 1752903] Re: Floating IPs should not allocate IPv6 addresses
Add service type [1] to your subnets of public network, then it will overcome the 'problem'. It's a bit complicated because there are some deployment and usage situations: 1. None DVR and IPv4 only (1) public network has only one subnet which is serving for floating IP and external gateways. set "network:floatingip", "network:router_gateway" to this subnet. (2) public network two types of subnet, one is for floating IP, one for external gatewy. set "network:floatingip" one set "network:router_gateway" to another 2. DVR and IPv4 only (1) public network has only one subnet which is serving for floating IP and external gateways. set "network:floatingip", "network:router_gateway", "network:floatingip_agent_gateway" to this subnet. (2) public network two types of subnet, one is for floating IP, one for external gatewy. set "network:floatingip" one set "network:router_gateway", "network:floatingip_agent_gateway" to another 3. None DVR and IPv4 + IPv6 4. DVR and IPv4 + IPv6 For 3. and 4., the public network may have one IPv6 subnet, set "network:router_gateway" and "network:floatingip_agent_gateway" to it. But for now, DVR with IPv6 in compute node does not work. [1] https://specs.openstack.org/openstack/neutron-specs/specs/newton /subnet-service-types.html ** Changed in: neutron Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1752903 Title: Floating IPs should not allocate IPv6 addresses Status in neutron: Won't Fix Bug description: When there are both IPv4 and IPv6 subnets on the public network, the port that a floating IP allocates there gets assigned addresses from both subnets. Since a floation IP is a pure IPv4 construct, allocating an IPv6 address for it is completely useless and should be avoided, because it will for example block removing the IPv6 subnet without a good reason. Seen in Pike as well as in master. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1752903/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1917393] [NEW] [L3][Port forwarding] admin state DOWN/UP router will lose all pf-floating-ips and nat rules
Public bug reported: Need to clean cache when router is down, otherwise the port forwarding extension will skip all objects processing due to cache is hitting. ** Affects: neutron Importance: High Status: Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1917393 Title: [L3][Port forwarding] admin state DOWN/UP router will lose all pf- floating-ips and nat rules Status in neutron: Confirmed Bug description: Need to clean cache when router is down, otherwise the port forwarding extension will skip all objects processing due to cache is hitting. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1917393/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1917279] Re: [L3][Port forwarding][DVR] failed to process floating ip port forwarding on 'legacy' mode L3 agent
For DVR router the snat node should be configured with "dvr_snat". For now, "legacy/ha router" can run on a "dvr_snat" node. But, DVR router can not run on "legacy" node, since the RouterInfo instance type is based on the agent mode [1]. [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L401 ** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1917279 Title: [L3][Port forwarding][DVR] failed to process floating ip port forwarding on 'legacy' mode L3 agent Status in neutron: Invalid Bug description: Create dvr router on 'legacy' agent node, then we got AttributeError: 'DvrLocalRouter' object has no attribute 'snat_namespace'. ERROR LOG: .agent.l3.agent [-] Failed to process compatible router: b247f145-569a-4d5a-bdd8-31a5213641ea: AttributeError: 'DvrLocalRouter' object has no attribute 'snat_namespace' .agent.l3.agent Traceback (most recent call last): .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 783, in _process_routers_if_compatible .agent.l3.agent self._process_router_if_compatible(router) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 625, in _process_router_if_compatible .agent.l3.agent self._process_updated_router(router) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 673, in _process_updated_router .agent.l3.agent self.l3_ext_manager.update_router(self.context, router) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/l3_agent_extensions_manager.py", line 54, in update_router .agent.l3.agent extension.obj.update_router(context, data) .agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner .agent.l3.agent return f(*args, **kwargs) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/extensions/port_forwarding.py", line 461, in update_router .agent.l3.agent self.process_port_forwarding(context, data) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/extensions/port_forwarding.py", line 435, in process_port_forwarding .agent.l3.agent if not self._check_if_need_process(ri): .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/extensions/port_forwarding.py", line 279, in _check_if_need_process .agent.l3.agent if is_distributed and not ri.snat_namespace.exists(): .agent.l3.agent AttributeError: 'DvrLocalRouter' object has no attribute 'snat_namespace' To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1917279/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1917279] [NEW] [L3][Port forwarding][DVR] failed to process floating ip port forwarding on 'legacy' mode L3 agent
Public bug reported: Create dvr router on 'legacy' agent node, then we got AttributeError: 'DvrLocalRouter' object has no attribute 'snat_namespace'. ERROR LOG: .agent.l3.agent [-] Failed to process compatible router: b247f145-569a-4d5a-bdd8-31a5213641ea: AttributeError: 'DvrLocalRouter' object has no attribute 'snat_namespace' .agent.l3.agent Traceback (most recent call last): .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 783, in _process_routers_if_compatible .agent.l3.agent self._process_router_if_compatible(router) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 625, in _process_router_if_compatible .agent.l3.agent self._process_updated_router(router) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 673, in _process_updated_router .agent.l3.agent self.l3_ext_manager.update_router(self.context, router) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/l3_agent_extensions_manager.py", line 54, in update_router .agent.l3.agent extension.obj.update_router(context, data) .agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner .agent.l3.agent return f(*args, **kwargs) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/extensions/port_forwarding.py", line 461, in update_router .agent.l3.agent self.process_port_forwarding(context, data) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/extensions/port_forwarding.py", line 435, in process_port_forwarding .agent.l3.agent if not self._check_if_need_process(ri): .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/extensions/port_forwarding.py", line 279, in _check_if_need_process .agent.l3.agent if is_distributed and not ri.snat_namespace.exists(): .agent.l3.agent AttributeError: 'DvrLocalRouter' object has no attribute 'snat_namespace' ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1917279 Title: [L3][Port forwarding][DVR] failed to process floating ip port forwarding on 'legacy' mode L3 agent Status in neutron: New Bug description: Create dvr router on 'legacy' agent node, then we got AttributeError: 'DvrLocalRouter' object has no attribute 'snat_namespace'. ERROR LOG: .agent.l3.agent [-] Failed to process compatible router: b247f145-569a-4d5a-bdd8-31a5213641ea: AttributeError: 'DvrLocalRouter' object has no attribute 'snat_namespace' .agent.l3.agent Traceback (most recent call last): .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 783, in _process_routers_if_compatible .agent.l3.agent self._process_router_if_compatible(router) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 625, in _process_router_if_compatible .agent.l3.agent self._process_updated_router(router) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 673, in _process_updated_router .agent.l3.agent self.l3_ext_manager.update_router(self.context, router) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/l3_agent_extensions_manager.py", line 54, in update_router .agent.l3.agent extension.obj.update_router(context, data) .agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner .agent.l3.agent return f(*args, **kwargs) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/extensions/port_forwarding.py", line 461, in update_router .agent.l3.agent self.process_port_forwarding(context, data) .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/extensions/port_forwarding.py", line 435, in process_port_forwarding .agent.l3.agent if not self._check_if_need_process(ri): .agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/extensions/port_forwarding.py", line 279, in _check_if_need_process .agent.l3.agent if is_distributed and not ri.snat_namespace.exists(): .agent.l3.agent AttributeError: 'DvrLocalRouter' object has no attribute 'snat_namespace' To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1917279/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1916889] [NEW] StaleDataError: DELETE statement on table 'standardattributes' expected to delete 2 row(s); 1 were matched. Please set confirm_deleted_rows=False within the mapper
Public bug reported:
Delete port failed with final DB error:
Feb 25 19:24:34 devstack neutron-server[15279]: DEBUG
neutron.api.rpc.handlers.l3_rpc [None req-a6ccb04c-401f-4e23-bc16-e7fc9cfc9ae6
None None] New status for floating IP f681d60c-edf9-41e9-b8b3-70c7cf3d8d42:
ERROR {{(pid=15361) update_floatingip_statuses
/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py:298}}
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api
[req-8bf1060f-eea9-42f0-9d3a-e16aed5d927b
req-b9e29b5a-9b7a-4838-9df0-80edc3fec7f4 admin admin] DB exceeded retry limit.:
StaleDataError: DELETE statement on table 'standardattributes' expected to
delete 2 row(s); 1 were matched. Please set confirm_deleted_rows=False within
the mapper configuration to prevent this warning.
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api Traceback
(most recent call last):
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api return
f(*args, **kwargs)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api
LOG.debug("Retry wrapper got retriable exception: %s", e)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api
self.force_reraise()
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api
six.reraise(self.type_, self.value, self.tb)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api return
f(*dup_args, **dup_kwargs)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1711, in delete_port
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api
super(Ml2Plugin, self).delete_port(context, id)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/opt/stack/neutron/neutron/db/api.py", line 125, in wrapped
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api return
method(*args, **kwargs)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1439, in delete_port
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api
self.ipam.delete_port(context, id)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 451, in
delete_port
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api
port['fixed_ips'])
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 95, in
_ipam_deallocate_ips
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api "external
system for %s", addresses)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api
self.force_reraise()
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api
six.reraise(self.type_, self.value, self.tb)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 71, in
_ipam_deallocate_ips
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api
ipam_subnet = ipam_driver.get_subnet(ip['subnet_id'])
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 337,
in get_subnet
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api return
NeutronDbSubnet.load(subnet_id, self._context)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 94, in
load
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api ctx,
neutron_subnet_id)
Feb 25 19:24:34 devstack neutron-server[15279]: ERROR oslo_db.api File
"/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 30, in
[Yahoo-eng-team] [Bug 1916572] [NEW] neutron.tests.unit.common.test_utils.TestThrottler test_throttler is failing
Public bug reported:
Example:
ft1.2:
neutron.tests.unit.common.test_utils.TestThrottler.test_throttlertesttools.testresult.real._StringException:
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py",
line 182, in func
return f(self, *args, **kwargs)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/common/test_utils.py",
line 433, in test_throttler
throttled_func()
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py",
line 107, in wrapper
eventlet.sleep(time_to_wait)
File "/usr/lib/python3.6/unittest/mock.py", line 939, in __call__
return _mock_self._mock_call(*args, **kwargs)
File "/usr/lib/python3.6/unittest/mock.py", line 1005, in _mock_call
result = effect(*args, **kwargs)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/common/test_utils.py",
line 429, in sleep_mock
self.assertGreater(threshold, amount_to_sleep)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/py36/lib/python3.6/site-packages/unittest2/case.py",
line 1233, in assertGreater
self.fail(self._formatMessage(msg, standardMsg))
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/py36/lib/python3.6/site-packages/unittest2/case.py",
line 690, in fail
raise self.failureException(msg)
AssertionError: 1 not greater than 1
https://3f074c0cfbd488dcf6af-
941e362261845d27ff2c0dd3e3a521f3.ssl.cf1.rackcdn.com/773283/6/check
/openstack-tox-py36/4e6fea3/testr_results.html
Log search:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%201%20not%20greater%20than%201%5C%22
** Affects: neutron
Importance: Undecided
Assignee: LIU Yulong (dragon889)
Status: New
** Changed in: neutron
Assignee: (unassigned) => LIU Yulong (dragon889)
** Description changed:
Example:
-
https://3f074c0cfbd488dcf6af-941e362261845d27ff2c0dd3e3a521f3.ssl.cf1.rackcdn.com/773283/6/check/openstack-tox-py36/4e6fea3/testr_results.html
+ ft1.2:
neutron.tests.unit.common.test_utils.TestThrottler.test_throttlertesttools.testresult.real._StringException:
Traceback (most recent call last):
+ File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py",
line 182, in func
+ return f(self, *args, **kwargs)
+ File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/common/test_utils.py",
line 433, in test_throttler
+ throttled_func()
+ File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line
107, in wrapper
+ eventlet.sleep(time_to_wait)
+ File "/usr/lib/python3.6/unittest/mock.py", line 939, in __call__
+ return _mock_self._mock_call(*args, **kwargs)
+ File "/usr/lib/python3.6/unittest/mock.py", line 1005, in _mock_call
+ result = effect(*args, **kwargs)
+ File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/common/test_utils.py",
line 429, in sleep_mock
+ self.assertGreater(threshold, amount_to_sleep)
+ File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/py36/lib/python3.6/site-packages/unittest2/case.py",
line 1233, in assertGreater
+ self.fail(self._formatMessage(msg, standardMsg))
+ File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/py36/lib/python3.6/site-packages/unittest2/case.py",
line 690, in fail
+ raise self.failureException(msg)
+ AssertionError: 1 not greater than 1
+
+ https://3f074c0cfbd488dcf6af-
+ 941e362261845d27ff2c0dd3e3a521f3.ssl.cf1.rackcdn.com/773283/6/check
+ /openstack-tox-py36/4e6fea3/testr_results.html
Log search:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%201%20not%20greater%20than%201%5C%22
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1916572
Title:
neutron.tests.unit.common.test_utils.TestThrottler test_throttler is
failing
Status in neutron:
New
Bug description:
Example:
ft1.2:
neutron.tests.unit.common.test_utils.TestThrottler.test_throttlertesttools.testresult.real._StringException:
Traceback (most recent call last):
File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py",
line 182, in func
return f(self, *args, **kwargs)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/common/test_utils.py",
line 433, in test_throttler
throttled_func()
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line
107, in wrapper
eventlet.sleep(time_to_wait)
File "/usr/lib/python3.6/unittest/mock.py", line 939, in __call__
return _mock_self._mock_call(*args, **kwargs)
File "/
[Yahoo-eng-team] [Bug 1916332] [NEW] neutron-specs's zuul job openstack-tox-docs fails
Public bug reported: Examples: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automated-testing openstack-tox-docs https://zuul.opendev.org/t/openstack/build/888eec88174946a89e4725e216f4aef3 : FAILURE in 3m 39s openstack-tox-py36 https://zuul.opendev.org/t/openstack/build/c60905a4ae5b4cd78d084263243b9d71 : SUCCESS in 3m 55s Warning: Job openstack-tox-docs: unable to map line for file comments: stderr: 'fatal: There is no path doc/source/specs/mitaka/address-scopes.rst in the commit' Comments left for invalid file doc/source/specs/mitaka/address-scopes.rst Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automated-testing openstack-tox-docs https://zuul.opendev.org/t/openstack/build/271c8a6b05d349279ba45095dc4814ae : FAILURE in 6m 05s openstack-tox-py36 https://zuul.opendev.org/t/openstack/build/5adae416d6774110bd565022d7749b71 : SUCCESS in 4m 02s Warning: Job openstack-tox-docs: unable to map line for file comments: stderr: 'fatal: There is no path doc/source/specs/mitaka/address-scopes.rst in the commit' Comments left for invalid file doc/source/specs/mitaka/address-scopes.rst ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1916332 Title: neutron-specs's zuul job openstack-tox-docs fails Status in neutron: New Bug description: Examples: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automated-testing openstack-tox-docs https://zuul.opendev.org/t/openstack/build/888eec88174946a89e4725e216f4aef3 : FAILURE in 3m 39s openstack-tox-py36 https://zuul.opendev.org/t/openstack/build/c60905a4ae5b4cd78d084263243b9d71 : SUCCESS in 3m 55s Warning: Job openstack-tox-docs: unable to map line for file comments: stderr: 'fatal: There is no path doc/source/specs/mitaka/address-scopes.rst in the commit' Comments left for invalid file doc/source/specs/mitaka/address-scopes.rst Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automated-testing openstack-tox-docs https://zuul.opendev.org/t/openstack/build/271c8a6b05d349279ba45095dc4814ae : FAILURE in 6m 05s openstack-tox-py36 https://zuul.opendev.org/t/openstack/build/5adae416d6774110bd565022d7749b71 : SUCCESS in 4m 02s Warning: Job openstack-tox-docs: unable to map line for file comments: stderr: 'fatal: There is no path doc/source/specs/mitaka/address-scopes.rst in the commit' Comments left for invalid file doc/source/specs/mitaka/address-scopes.rst To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1916332/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1732067] Re: openvswitch firewall flows cause flooding on integration bridge
** Changed in: neutron Status: In Progress => Fix Committed ** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1732067 Title: openvswitch firewall flows cause flooding on integration bridge Status in neutron: Fix Released Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: Environment: OpenStack Newton Driver: ML2 w/ OVS Firewall: openvswitch In this environment, we have observed OVS flooding network traffic across all ports in a given VLAN on the integration bridge due to the lack of a FDB entry for the destination MAC address. Across the large fleet of 240+ nodes, this is causing a considerable amount of noise on any given node. In this test, we have 3 machines: Client: fa:16:3e:e8:59:00 (10.10.60.2) Server: fa:16:3e:80:cb:0a (10.10.60.9) Bystander: fa:16:3e:a0:ee:02 (10.10.60.10) The server is running a web server using netcat: while true ; do sudo nc -l -p 80 < index.html ; done Client requests page using curl: ip netns exec qdhcp-b07e6cb3-0943-45a2-b5ff-efb7e99e4d3d curl http://10.10.60.9/ We should expect to see the communication limited to the client and server. However, the captures below reflect the server->client responses being broadcast out all tap interfaces connected to br-int in the same local vlan: root@osa-newton-ovs-compute01:~# tcpdump -i tap5f03424d-1c -ne port 80 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tap5f03424d-1c, link-type EN10MB (Ethernet), capture size 262144 bytes 02:20:30.190675 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 74: 10.10.60.2.54796 > 10.10.60.9.80: Flags [S], seq 213484442, win 29200, options [mss 1460,sackOK,TS val 140883559 ecr 0,nop,wscale 7], length 0 02:20:30.191926 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 74: 10.10.60.9.80 > 10.10.60.2.54796: Flags [S.], seq 90006557, ack 213484443, win 14480, options [mss 1460,sackOK,TS val 95716 ecr 140883559,nop,wscale 4], length 0 02:20:30.192837 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 66: 10.10.60.2.54796 > 10.10.60.9.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 140883560 ecr 95716], length 0 02:20:30.192986 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 140: 10.10.60.2.54796 > 10.10.60.9.80: Flags [P.], seq 1:75, ack 1, win 229, options [nop,nop,TS val 140883560 ecr 95716], length 74: HTTP: GET / HTTP/1.1 02:20:30.195806 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 79: 10.10.60.9.80 > 10.10.60.2.54796: Flags [P.], seq 1:14, ack 1, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 13: HTTP 02:20:30.196207 fa:16:3e:e8:59:00 > fa:16:3e:80:cb:0a, ethertype IPv4 (0x0800), length 66: 10.10.60.2.54796 > 10.10.60.9.80: Flags [.], ack 14, win 229, options [nop,nop,TS val 140883561 ecr 95717], length 0 02:20:30.197481 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 66: 10.10.60.9.80 > 10.10.60.2.54796: Flags [.], ack 75, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 0 ^^^ On the server tap we see the bi-directional traffic root@osa-newton-ovs-compute01:/home/ubuntu# tcpdump -i tapb8051da9-60 -ne port 80 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tapb8051da9-60, link-type EN10MB (Ethernet), capture size 262144 bytes 02:20:30.192165 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 74: 10.10.60.9.80 > 10.10.60.2.54796: Flags [S.], seq 90006557, ack 213484443, win 14480, options [mss 1460,sackOK,TS val 95716 ecr 140883559,nop,wscale 4], length 0 02:20:30.195827 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 79: 10.10.60.9.80 > 10.10.60.2.54796: Flags [P.], seq 1:14, ack 1, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 13: HTTP 02:20:30.197500 fa:16:3e:80:cb:0a > fa:16:3e:e8:59:00, ethertype IPv4 (0x0800), length 66: 10.10.60.9.80 > 10.10.60.2.54796: Flags [.], ack 75, win 905, options [nop,nop,TS val 95717 ecr 140883560], length 0 ^^^ On the bystander tap we see the flooded traffic The FDB tables reflect the lack of CAM entry for the client on br-int bridge. I would expect to see the MAC address on the patch uplink: root@osa-newton-ovs-compute01:/home/ubuntu# ovs-appctl fdb/show br-int | grep 'fa:16:3e:e8:59:00' root@osa-newton-ovs-compute01:/home/ubuntu# ovs-appctl fdb/show br-provider | grep 'fa:16:3e:e8:59:00' 2 850 fa:16:3e:e8:59:003 Sources[1] point to the fact that an 'output' action negates the MAC learning mechanism in OVS. Related Table 82 entries are below, and code is here[2]: cooki
[Yahoo-eng-team] [Bug 1913664] [NEW] [CI] neutron multinode jobs does not run neutron_tempest_plugin scenario cases
Public bug reported: This is the job neutron-tempest-plugin-scenario-openvswitch's cases: https://812aefd7f17477a1c0dc-8bc1c0202523f17b73621207314548bd.ssl.cf5.rackcdn.com/772255/6/check/neutron-tempest-plugin-scenario-openvswitch/5221232/testr_results.html This is neutron-tempest-dvr-ha-multinode-full cases: https://87e09d95af4c4ee8cb65-839132c9f2f257823716e8f40ef80a9a.ssl.cf1.rackcdn.com/772255/6/check/neutron-tempest-dvr-ha-multinode-full/0e428cd/testr_results.html IMO, neutron-tempest-*-multinode-full should contain all the neutron- tempest-plugin-scenario-* cases. But it does not. ** Affects: neutron Importance: Undecided Status: New ** Description changed: This is the job neutron-tempest-plugin-scenario-openvswitch's cases: https://812aefd7f17477a1c0dc-8bc1c0202523f17b73621207314548bd.ssl.cf5.rackcdn.com/772255/6/check/neutron-tempest-plugin-scenario-openvswitch/5221232/testr_results.html - This is neutron-tempest-dvr-ha-multinode-full cases: https://87e09d95af4c4ee8cb65-839132c9f2f257823716e8f40ef80a9a.ssl.cf1.rackcdn.com/772255/6/check/neutron-tempest-dvr-ha-multinode-full/0e428cd/testr_results.html - IMO, neutron-tempest-dvr-ha-multinode-full should contain all the - neutron-tempest-plugin-scenario cases. But it does not. + IMO, neutron-tempest-*-multinode-full should contain all the neutron- + tempest-plugin-scenario-* cases. But it does not. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1913664 Title: [CI] neutron multinode jobs does not run neutron_tempest_plugin scenario cases Status in neutron: New Bug description: This is the job neutron-tempest-plugin-scenario-openvswitch's cases: https://812aefd7f17477a1c0dc-8bc1c0202523f17b73621207314548bd.ssl.cf5.rackcdn.com/772255/6/check/neutron-tempest-plugin-scenario-openvswitch/5221232/testr_results.html This is neutron-tempest-dvr-ha-multinode-full cases: https://87e09d95af4c4ee8cb65-839132c9f2f257823716e8f40ef80a9a.ssl.cf1.rackcdn.com/772255/6/check/neutron-tempest-dvr-ha-multinode-full/0e428cd/testr_results.html IMO, neutron-tempest-*-multinode-full should contain all the neutron- tempest-plugin-scenario-* cases. But it does not. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1913664/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1907175] Re: intermittently ALL VM's floating IP connection is disconnected, and can be reconnected after 5-6 minutes
** Changed in: neutron Status: New => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1907175 Title: intermittently ALL VM's floating IP connection is disconnected, and can be reconnected after 5-6 minutes Status in neutron: Won't Fix Bug description: Current configuration: 49node centos 7.8(Kernel version = 3.10.0-1127.el7.x86_64) kolla-ansible 9.2.1 (openvswitch - 2.12.0 / neutron-server 15.1.0) Phenomenon: The floating IP connection is disconnected, and the connection becomes possible again after 5-6 minutes. Occurs by all vm on nodes. The internal ip connection is not disconnected, and if openvswitch_vswitchd is restarted in case of failure, the problem is solved. The public network, physnet1 (172.29.75.0~172.29.84.0), is tied in LACP(Bond_mode =4 ) mode by VLAN, and the TENANT NETWORK is composed of vxlan. (Use DVR) As a result of the ping tcpdump test, the network sends a ping to the node with the vm, but the vm does not respond. === ml2_conf.ini [root@2020c5lut006 neutron-server]# cat ml2_conf.ini [ml2] type_drivers = flat,vlan,vxlan tenant_network_types = vxlan mechanism_drivers = openvswitch,baremetal,l2population extension_drivers = qos,port_security path_mtu = 9000 [ml2_type_vlan] network_vlan_ranges = physnet1 [ml2_type_flat] flat_networks = * [ml2_type_vxlan] vni_ranges = 1:1000 [securitygroup] firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver [agent] tunnel_types = vxlan l2_population = true arp_responder = true enable_distributed_routing = True extensions = qos [ovs] bridge_mappings = physnet1:br-ex,physnet2:br-cephfs,physnet3:br-api datapath_type = system ovsdb_connection = tcp:127.0.0.1:6640 local_ip = 20.21.2.101 ==neutron.conf [root@2020c5lut006 neutron-server]# cat neutron.conf [DEFAULT] debug = False log_dir = /var/log/kolla/neutron use_stderr = False bind_host = 20.21.1.101 bind_port = 9696 api_paste_config = /usr/share/neutron/api-paste.ini endpoint_type = internalURL api_workers = 5 metadata_workers = 5 rpc_workers = 3 rpc_state_report_workers = 3 metadata_proxy_socket = /var/lib/neutron/kolla/metadata_proxy interface_driver = openvswitch allow_overlapping_ips = true core_plugin = ml2 service_plugins = firewall_v2,qos,router dhcp_agents_per_network = 2 l3_ha = true max_l3_agents_per_router = 3 transport_url = rabbit://openstack:mMZl0hvZ5KSGQgfqtAbbBRkpMfEbzIKjDUHu8NSd@20.21.1.101:5672,openstack:mMZl0hvZ5KSGQgfqtAbbBRkpMfEbzIKjDUHu8NSd@20.21.1.102:5672,openstack:mMZl0hvZ5KSGQgfqtAbbBRkpMfEbzIKjDUHu8NSd@20.21.1.103:5672// router_distributed = True ipam_driver = internal global_physnet_mtu = 9000 [nova] auth_url = http://20.21.1.100:35357 auth_type = password project_domain_id = default user_domain_id = default region_name = RegionOne project_name = service username = nova password = rChwHtVHMqLK3AHRkKfZ7rxiQ74Am8EJHWvbEyQt endpoint_type = internal [oslo_middleware] enable_proxy_headers_parsing = True [oslo_concurrency] lock_path = /var/lib/neutron/tmp [agent] root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf [database] connection = mysql+pymysql://neutron:PZl2BQm7LesapA6Ks9lqOuUc6DU4kRHeSWwPNvH1@20.21.1.100:3306/neutron max_retries = -1 [keystone_authtoken] www_authenticate_uri = http://20.21.1.100:5000 auth_url = http://20.21.1.100:35357 auth_type = password project_domain_id = default user_domain_id = default project_name = service username = neutron password = XjxBaFwek0aaKj0rLaqeUXqfp7lrNk5sdkIFGAeE memcache_security_strategy = ENCRYPT memcache_secret_key = w6eOcER3TlZzidSL7wjea2rnbMWGUlV7BiO3ls3J memcached_servers = 20.21.1.101:11211,20.21.1.102:11211,20.21.1.103:11211 [oslo_messaging_notifications] transport_url = rabbit://openstack:mMZl0hvZ5KSGQgfqtAbbBRkpMfEbzIKjDUHu8NSd@20.21.1.101:5672,openstack:mMZl0hvZ5KSGQgfqtAbbBRkpMfEbzIKjDUHu8NSd@20.21.1.102:5672,openstack:mMZl0hvZ5KSGQgfqtAbbBRkpMfEbzIKjDUHu8NSd@20.21.1.103:5672// driver = noop [octavia] base_url = http://20.21.1.100:9876 [placement] auth_type = password auth_url = http://20.21.1.100:35357 username = placement password = s1VxNvJeh8CDOjeqa6hi8eF0QhQdDBp12SJdyfll user_domain_name = Default project_name = service project_domain_name = Default os_region_name = RegionOne os_interface = internal [privsep] helper_command = sudo neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1907
[Yahoo-eng-team] [Bug 1912460] [NEW] [QoS] add qos rule type packet per second (pps)
Public bug reported: For cloud providers, to limit the packet per second (pps) of VM NIC is popular and sometimes essential. Transit large set packets for VM in physical compute hosts will consume the CPU/phy-nic performance. And for small packets, even the bandwidth is low, the pps can still be higher, which can be an attact point inside the cloud while some VMs are becoming hacked. So for neutron, IMO, we should add such type of QoS rule, packet per second (pps). Am I missing something? Thoughts? ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1912460 Title: [QoS] add qos rule type packet per second (pps) Status in neutron: New Bug description: For cloud providers, to limit the packet per second (pps) of VM NIC is popular and sometimes essential. Transit large set packets for VM in physical compute hosts will consume the CPU/phy-nic performance. And for small packets, even the bandwidth is low, the pps can still be higher, which can be an attact point inside the cloud while some VMs are becoming hacked. So for neutron, IMO, we should add such type of QoS rule, packet per second (pps). Am I missing something? Thoughts? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1912460/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1911864] [NEW] [DHCP] AgentBinding for network will be created no matter the state
Public bug reported: Neutron creates N NetworkDhcpAgentBindings (N is equal to dhcp_agents_per_network) for network even if the subnets disabled the dhcp. This means no matter the DHCP state, the dhcp_scheduler will schedule the network anyway. Reproduce steps: $ source demo_rc $ openstack network create 111 $ openstack subnet create --no-dhcp --subnet-range 1.1.1.0/24 --network 871c588b-2582-4a08-bff7-d92e1590eecd 111-subnet $ source admin_rc $ openstack network agent list --network 871c588b-2582-4a08-bff7-d92e1590eecd +--++--+---+---+---++ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--++--+---+---+---++ | c9921687-30ee-470e-813f-6e45247b57f1 | DHCP agent | network2 | nova | :-) | UP| neutron-dhcp-agent | +--++--+---+---+---++ Then this behavior has a result is that HA network for HA routers will also have such bindings. Because the external network will not have VM created normally, so its subnets' enable_dhcp are all disabled. Again, the bindings are created. Good news is that in DHCP agent side dhcp-namespace and dnsmasq are not created. So, basically this is inconsitent between neutron server side and DHCP agent side. So, I have some thoughts about this is to add config option to disable some types of network's dhcp agent binding. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1911864 Title: [DHCP] AgentBinding for network will be created no matter the state Status in neutron: New Bug description: Neutron creates N NetworkDhcpAgentBindings (N is equal to dhcp_agents_per_network) for network even if the subnets disabled the dhcp. This means no matter the DHCP state, the dhcp_scheduler will schedule the network anyway. Reproduce steps: $ source demo_rc $ openstack network create 111 $ openstack subnet create --no-dhcp --subnet-range 1.1.1.0/24 --network 871c588b-2582-4a08-bff7-d92e1590eecd 111-subnet $ source admin_rc $ openstack network agent list --network 871c588b-2582-4a08-bff7-d92e1590eecd +--++--+---+---+---++ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--++--+---+---+---++ | c9921687-30ee-470e-813f-6e45247b57f1 | DHCP agent | network2 | nova | :-) | UP| neutron-dhcp-agent | +--++--+---+---+---++ Then this behavior has a result is that HA network for HA routers will also have such bindings. Because the external network will not have VM created normally, so its subnets' enable_dhcp are all disabled. Again, the bindings are created. Good news is that in DHCP agent side dhcp-namespace and dnsmasq are not created. So, basically this is inconsitent between neutron server side and DHCP agent side. So, I have some thoughts about this is to add config option to disable some types of network's dhcp agent binding. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1911864/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1911126] [NEW] [RFE][L3] add ability to control router SNAT more granularly
Public bug reported: Neutron router now supports SNAT when the attribute ``enable_snat`` of the gateway is set to True. This will enable all the VMs which has no binding floating IP to access the public world. But, generally the DataCenter bandwidths for cloud providers are not free. And some users may want to buy a higher SNAT bandwidth for one of their VMs, a CIDR, or a subnet. So for Neutron, it should support these scenarios: 1. enable/disable SNAT once for all (supported, controlled by ``enable_snat``) 2. enable/disable SNAT for one internal IP (of VM) 3. enable/disable SNAT for a range CIDR of IPs 4. enable/disable SNAT for a subnet For 2., 3. and 4. scenario should have QoS support. So I would like to add a new mechanism for Neutron to support these: 1. An new API extension to add specific SNAT type 2. An new L3 agent extension to install SNAT iptables rules. Ideas? ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1911126 Title: [RFE][L3] add ability to control router SNAT more granularly Status in neutron: New Bug description: Neutron router now supports SNAT when the attribute ``enable_snat`` of the gateway is set to True. This will enable all the VMs which has no binding floating IP to access the public world. But, generally the DataCenter bandwidths for cloud providers are not free. And some users may want to buy a higher SNAT bandwidth for one of their VMs, a CIDR, or a subnet. So for Neutron, it should support these scenarios: 1. enable/disable SNAT once for all (supported, controlled by ``enable_snat``) 2. enable/disable SNAT for one internal IP (of VM) 3. enable/disable SNAT for a range CIDR of IPs 4. enable/disable SNAT for a subnet For 2., 3. and 4. scenario should have QoS support. So I would like to add a new mechanism for Neutron to support these: 1. An new API extension to add specific SNAT type 2. An new L3 agent extension to install SNAT iptables rules. Ideas? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1911126/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1906487] [NEW] [L2][scale] add a trunk size config option for bundle flow install
onnmgr|INFO|br-int<->unix#557: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:51.337Z|01007|connmgr|INFO|br-int<->unix#560: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:51.627Z|01008|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 16 flow_mods in the 7 s starting 27 s ago (12 adds, 4 deletes) 2020-12-02T08:01:51.771Z|01009|connmgr|INFO|br-int<->unix#563: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:52.285Z|01010|connmgr|INFO|br-int<->unix#566: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:52.695Z|01011|connmgr|INFO|br-int<->unix#569: 96 flow_mods in the last 0 s (96 adds) 2. the AGENT_RES_PROCESSING_STEP is 1000 (changed) 1 OpenFlow rules are installed in 6s. 2020-12-02 16:13:41.855 3090 INFO neutron.agent.common.ovs_lib [None req-0eef620d-613e-4485-ad2f-b9380c3fc381 - - - - -] Processing 1 OpenFlow rules. 2020-12-02 16:13:47.846 3090 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-0eef620d-613e-4485-ad2f-b9380c3fc381 - - - - -] process_network_ports - iteration:0 - agent port security group processed in 202.523 2020-12-02T08:13:42.299Z|01310|connmgr|INFO|br-int<->unix#1298: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:42.718Z|01311|connmgr|INFO|br-int<->unix#1301: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:43.174Z|01312|connmgr|INFO|br-int<->unix#1304: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:43.698Z|01313|connmgr|INFO|br-int<->unix#1307: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:44.087Z|01314|connmgr|INFO|br-int<->unix#1310: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:44.554Z|01315|connmgr|INFO|br-int<->unix#1313: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:45.050Z|01316|connmgr|INFO|br-int<->unix#1316: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:45.534Z|01317|connmgr|INFO|br-int<->unix#1319: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:45.939Z|01318|connmgr|INFO|br-int<->unix#1322: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:46.388Z|01319|connmgr|INFO|br-int<->unix#1325: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:46.869Z|01320|connmgr|INFO|br-int<->unix#1328: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:47.323Z|01321|connmgr|INFO|br-int<->unix#1331: 1000 flow_mods in the last 0 s (1000 adds) 2020-12-02T08:13:47.834Z|01322|connmgr|INFO|br-int<->unix#1334: 222 flow_mods in the last 0 s (222 adds) So we can say maybe we should increase the trunk (step) size for the ovs-ofctl bundle installaion. We do not want to add a fixed value because the vswitchd may be in high pressure according to your deployment, the flow install performace may not be as good as you expected. So we could add a config option for users, they can tune the size for their cloud based on the load of the agent, vswitchd and cpu usage and so on. ** Affects: neutron Importance: Undecided Assignee: LIU Yulong (dragon889) Status: New ** Description changed: We recently tested the openvswitch-agent flow installation performance, we have following two results: 1. the AGENT_RES_PROCESSING_STEP is 100 (default) We can see the 4296 OpenFlow rules are installed in 20s. - - 2020-12-02 16:01:32.843 792 INFO neutron.agent.common.ovs_lib [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] Processing 4296 OpenFlow rules. + 2020-12-02 16:01:32.843 792 INFO neutron.agent.common.ovs_lib [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] Processing 4296 OpenFlow rules. 2020-12-02 16:01:52.698 792 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] process_network_ports - iteration:266 - agent port security group processed in 31.044 2020-12-02T08:01:32.303Z|00967|connmgr|INFO|br-int<->unix#440: 1 flow_mods in the last 0 s (1 adds) 2020-12-02T08:01:33.353Z|00968|connmgr|INFO|br-int<->unix#443: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:33.864Z|00969|connmgr|INFO|br-int<->unix#446: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:34.435Z|00970|connmgr|INFO|br-int<->unix#449: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:34.966Z|00971|connmgr|INFO|br-int<->unix#452: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:35.399Z|00972|connmgr|INFO|br-int<->unix#455: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:35.904Z|00973|connmgr|INFO|br-int<->unix#458: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:36.335Z|00974|connmgr|INFO|br-int<->unix#461: 100 flow_mods in the last 0 s (100 adds) 2020-12-02T08:01:36.820Z|00975|connmgr|INFO|br-int<->unix#464: 100 flow_mods in the last 0 s (100 adds) 2020-12-
[Yahoo-eng-team] [Bug 1906381] [NEW] [L2] current binding port get type errors
Public bug reported:
get_binding_levels can return None if the binding host is None. But the
patch [1] did not handle such case. The code will raise a TypeError.
[1] https://review.opendev.org/c/openstack/neutron/+/606827
Bug was first found in stable/rocky, but the master branch has same code.
ERROR logs:
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server
LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server return
f(*dup_args, **dup_kwargs)
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 491, in
_bind_port_if_needed
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server need_notify,
try_again))
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 679, in
_commit_port_binding
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server for level in
levels:
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server TypeError:
'NoneType' object is not iterable
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server
** Affects: neutron
Importance: Low
Assignee: LIU Yulong (dragon889)
Status: In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1906381
Title:
[L2] current binding port get type errors
Status in neutron:
In Progress
Bug description:
get_binding_levels can return None if the binding host is None. But
the patch [1] did not handle such case. The code will raise a
TypeError.
[1] https://review.opendev.org/c/openstack/neutron/+/606827
Bug was first found in stable/rocky, but the master branch has same code.
ERROR logs:
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server
LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server return
f(*dup_args, **dup_kwargs)
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 491, in
_bind_port_if_needed
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server
need_notify, try_again))
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 679, in
_commit_port_binding
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc.server for level
in levels:
2020-11-30 16:46:55.942 62088 ERROR oslo_messaging.rpc
[Yahoo-eng-team] [Bug 1906375] [NEW] [L3] router HA port concurrently deleting
PortNotFound:
Port 3f838c59-e84a-49de-a381-f3328d47a69f could not be found.
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server
2020-12-01 10:53:03.921 62076 ERROR oslo_messaging.rpc.server
[req-3822c366-0cb7-46c3-ad6b-b50081da3dc8 - - - - -] Exception during message
handling: PortNotFound: Port 80911dcd-06be-456a-9054-4bf9f40680aa could not be
found.
** Affects: neutron
Importance: Low
Assignee: LIU Yulong (dragon889)
Status: In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1906375
Title:
[L3] router HA port concurrently deleting
Status in neutron:
In Progress
Bug description:
Router HA port may be deleted concurrently while the plugin is trying to
update. Then a PortNotFound exception raised.
ERROR was found at rocky deployment, but the master branch has the same code.
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server
[req-86012433-ab6b-41f5-bbba-411ec3e1d973 - - - - -] Exception during message
handling: PortNotFound: Port 3f838c59-e84a-49de-a381-f3328d47a69f could not be
found.
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in
_process_incoming
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265,
in dispatch
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194,
in _do_dispatch
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 93,
in update_all_ha_network_port_statuses
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server context, p,
{'port': {'status': constants.PORT_STATUS_DOWN}})
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 632, in inner
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return
f(self, context, *args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return
method(*args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server setattr(e,
'_RETRY_EXCEEDED', True)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server ectxt.value
= e.inner_exc
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-12-01 10:52:46.738 62077 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db
[Yahoo-eng-team] [Bug 1896563] Re: Floating IP port forwarding rule update external_ Port, HTTP return code 500 error
Neutron had covered such issue [1][2][3].
[1]
https://github.com/openstack/neutron/blob/master/neutron/services/portforwarding/pf_plugin.py#L433-L439
[2]
https://github.com/openstack/neutron/blob/master/neutron/db/migration/alembic_migrations/versions/rocky/expand/867d39095bf4_port_forwarding.py#L58
[3]
https://github.com/openstack/neutron/blob/master/neutron/db/migration/alembic_migrations/versions/stein/expand/d72db3e25539_modify_uniq_port_forwarding.py
** Changed in: neutron
Status: In Progress => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1896563
Title:
Floating IP port forwarding rule update external_ Port, HTTP return
code 500 error
Status in neutron:
Invalid
Bug description:
When an existing port forwarding rule is updated on the floating IP, if the
external_port field contained in the updated content is the same as that in
other port forwarding rule entries under the floating IP, the HTTP return code
is 500;
The log of the neutron server is as follows:
2020-09-22 11:01:28.779 24 ERROR oslo_db.api
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource
[req-6d482405-6141-444a-96dc-00dc3377be26 3b68afd339ef43ed97c1e5b145a3961b
4bf85834786143a28c75239bd52add52 - default default] update failed: No details.:
DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry
'65f3ec87-b937-4485-a205-e4946acbf745-77' for key
'uniq_port_forwardings0floatingip_id0external_port'") [SQL: u'UPDATE
portforwardings SET external_port=%(external_port)s WHERE portforwardings.id =
%(portforwardings_id)s'] [parameters: {'external_port': 77,
'portforwardings_id': u'622041f5-981a-4cef-bd27-c5009c419572'}] (Background on
this error at: http://sqlalche.me/e/gkpj)
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource Traceback (most
recent call last):
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/resource.py",
line 98, in resource
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource result =
method(request=request, **args)
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line
626, in update
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource return
self._update(request, id, body, **kwargs)
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron_lib/db/api.py", line
140, in wrapped
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource setattr(e,
'_RETRY_EXCEEDED', True)
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line
220, in __exit__
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource
self.force_reraise()
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line
196, in force_reraise
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource
six.reraise(self.type_, self.value, self.tb)
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron_lib/db/api.py", line
136, in wrapped
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource return f(*args,
**kwargs)
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/api.py", line 154, in
wrapper
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource ectxt.value =
e.inner_exc
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line
220, in __exit__
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource
self.force_reraise()
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line
196, in force_reraise
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource
six.reraise(self.type_, self.value, self.tb)
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/api.py", line 142, in
wrapper
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource return f(*args,
**kwargs)
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/neutron_lib/db/api.py", line
183, in wrapped
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource LOG.debug("Retry
wrapper got retriable exception: %s", e)
2020-09-22 11:01:28.782 24 ERROR neutron.api.v2.resource File
"/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line
220, in __exit__
[Yahoo-eng-team] [Bug 1900934] [NEW] [RFE][DHCP][OVS] flow based DHCP
Public bug reported: Add a new ovs agent extension to support fully distributed DHCP for VMs in compute nodes, especially for large scale cloud. We had some disscussions during Shanghai PTG: https://etherpad.opendev.org/p/Shanghai-Neutron-Planning-restored http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010702.html https://github.com/gotostack/shanghai_ptg/blob/master/shanghai_neutron_ptg_topics_liuyulong.pdf ** Affects: neutron Importance: Undecided Status: New ** Attachment added: "dhcp_ovs.png" https://bugs.launchpad.net/bugs/1900934/+attachment/5425306/+files/dhcp_ovs.png -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1900934 Title: [RFE][DHCP][OVS] flow based DHCP Status in neutron: New Bug description: Add a new ovs agent extension to support fully distributed DHCP for VMs in compute nodes, especially for large scale cloud. We had some disscussions during Shanghai PTG: https://etherpad.opendev.org/p/Shanghai-Neutron-Planning-restored http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010702.html https://github.com/gotostack/shanghai_ptg/blob/master/shanghai_neutron_ptg_topics_liuyulong.pdf To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1900934/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1897423] [NEW] [L3] Let agent extension do delete router first
Public bug reported: For some agent extension implementation, it may need the router_info to do some clean up work. So the L3 agent extensions(s) can move the delete action ahead of the L3 agent cache deleting. ** Affects: neutron Importance: Low Assignee: LIU Yulong (dragon889) Status: In Progress ** Changed in: neutron Importance: Undecided => Low ** Changed in: neutron Assignee: (unassigned) => LIU Yulong (dragon889) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1897423 Title: [L3] Let agent extension do delete router first Status in neutron: In Progress Bug description: For some agent extension implementation, it may need the router_info to do some clean up work. So the L3 agent extensions(s) can move the delete action ahead of the L3 agent cache deleting. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1897423/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1895950] Re: keepalived can't perform failover if the l3 agent is down
OK, let's mark this as "Won't Fix". And move to state change can be a way to new install envrionment. But for running could, the existing keepalived-state-change processes may need to re-create. ** Changed in: neutron Status: New => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1895950 Title: keepalived can't perform failover if the l3 agent is down Status in neutron: Won't Fix Bug description: If the l3 agent is down on the new master node of keepalived all the interfaces remain down so no packets can pass through the new router. The replication scenario is the following: 1. Shut down all the l3 agents 2. Kill the keepalived process on the master node 3. Try to access VM behind the HA router (it will fail) To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1895950/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1895401] [NEW] [L3][IPv6][DVR] missing address scope mark for IPv6 traffic
Public bug reported: There is a address sope mark iptables rule missing for IPv6 traffic of DVR router in compute node. It should be part of the bug #1774463 , but IMO, we can fix it seperately [2]. [1] https://bugs.launchpad.net/neutron/+bug/1774463 [2] https://review.opendev.org/#/c/737261 ** Affects: neutron Importance: Low Assignee: LIU Yulong (dragon889) Status: In Progress ** Changed in: neutron Importance: Undecided => Low ** Changed in: neutron Status: New => In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1895401 Title: [L3][IPv6][DVR] missing address scope mark for IPv6 traffic Status in neutron: In Progress Bug description: There is a address sope mark iptables rule missing for IPv6 traffic of DVR router in compute node. It should be part of the bug #1774463 , but IMO, we can fix it seperately [2]. [1] https://bugs.launchpad.net/neutron/+bug/1774463 [2] https://review.opendev.org/#/c/737261 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1895401/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1891448] [NEW] [RFE] l3 agent mode transition between dvr and dvr_no_external
Public bug reported: For now if the L3 agent change the agent mode (dvr to dvr_no_external, or dvr_no_external to dvr), the floating IP traffic will not recover. There is a mannully workflow to achive the transition, but it will need to shutdown the router during the transition. The backwards is obviously the floating IP under the router will be shutdown for a bit long time (admin state down/up). And the floating IPs not related to this compute host will be influenced as well. So, IMO, neutron should support such agent mode change for varied cloud deployments. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1891448 Title: [RFE] l3 agent mode transition between dvr and dvr_no_external Status in neutron: New Bug description: For now if the L3 agent change the agent mode (dvr to dvr_no_external, or dvr_no_external to dvr), the floating IP traffic will not recover. There is a mannully workflow to achive the transition, but it will need to shutdown the router during the transition. The backwards is obviously the floating IP under the router will be shutdown for a bit long time (admin state down/up). And the floating IPs not related to this compute host will be influenced as well. So, IMO, neutron should support such agent mode change for varied cloud deployments. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1891448/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1888121] Re: L3 agent fails to update routers with onlink gateway
*** This bug is a duplicate of bug 1861674 ***
https://bugs.launchpad.net/bugs/1861674
** Changed in: neutron
Status: New => Incomplete
** Changed in: neutron
Status: Incomplete => Confirmed
** This bug has been marked a duplicate of bug 1861674
Gateway which is not in subnet CIDR is unsupported in ha router
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1888121
Title:
L3 agent fails to update routers with onlink gateway
Status in neutron:
Confirmed
Bug description:
If a router uses an external gateway network with an "onlink" gateway
(= gateway not in subnet range), L3 agents fails to process router
update.
* Versions: currently in Train, since the code did not changed I think
in Ussuri too.
* How to reproduce:
# Create external network
openstack network create public --external
# Create associated subnet with a gateway not in the subnet range. This kind
of gateway should be
# handle as an "onlink" route.
openstack subnet create --network public --subnet-range 192.168.144.0/24
--gateway 192.168.0.1
# Create router and set external gateway
openstack router create external
openstack router set --external-gateway public
# Check l3 agent logs
http://paste.openstack.org/show/796084/
* Current fix:
During gateway setup here
https://github.com/openstack/neutron/blob/stable/train/neutron/agent/linux/ip_lib.py#L604,
adding 'onlink' flag allows pyroute2 to successfully add the onlink
default gateway:
```
def add_gateway(self, gateway, metric=None, table=None, scope='global',
flags=[]):
kwargs = {'flags': ['onlink']}
self.add_route(None, via=gateway, table=table, metric=metric,
scope=scope, **kwargs)
```
Result: http://paste.openstack.org/show/796085/
About the patch, I don't really know the consequences of adding the onlink
flag standards gateway. Maybe we could add a check "if 'gateway not in subnet
cidr' then onlink", this will impact all existing routers otherwise.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1888121/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1883089] [NEW] [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
saging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
f(*dup_args, **dup_kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
348, in get_agent_gateway_port
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server admin_ctx,
network_id, host)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 953, in
create_fip_agent_gw_port_if_not_exists
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
self._populate_mtu_and_subnets_for_ports(context, [agent_port])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1978, in
_populate_mtu_and_subnets_for_ports
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server for p in
self._each_port_having_fixed_ips(ports)]
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1925, in
_each_port_having_fixed_ips
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server fixed_ips =
port.get('fixed_ips', [])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server AttributeError:
'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server
** Affects: neutron
Importance: Medium
Assignee: LIU Yulong (dragon889)
Status: Confirmed
** Changed in: neutron
Assignee: (unassigned) => LIU Yulong (dragon889)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1883089
Title:
[L3] floating IP failed to bind due to no agent gateway port(fip-ns)
Status in neutron:
Confirmed
Bug description:
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.
[1] https://review.opendev.org/#/c/702547/
Exception log:
2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host
host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host
host-compute-1. Probably it was just created by other worker.
create_fip_agent_gw_port_if_not_exists
/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway
port None found for the destination host: host-compute-1
create_fip_agent_gw_port_if_not_exists
/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None
req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message
handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback
(most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in
_process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220,
in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rp
[Yahoo-eng-team] [Bug 1880657] [NEW] [openstack][net] static subnet type does not work
Public bug reported: commit 62bbc262c3c7f633eac1d09ec78c055eef05166a changes the default code branch condition which breaks the existing cloud static network config. [1] https://github.com/canonical/cloud-init/commit/62bbc262c3c7f633eac1d09ec78c055eef05166a#r39437585 ** Affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1880657 Title: [openstack][net] static subnet type does not work Status in cloud-init: New Bug description: commit 62bbc262c3c7f633eac1d09ec78c055eef05166a changes the default code branch condition which breaks the existing cloud static network config. [1] https://github.com/canonical/cloud-init/commit/62bbc262c3c7f633eac1d09ec78c055eef05166a#r39437585 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1880657/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1879215] [NEW] [L3] Unexcepted HA router scheduled instance shows up after manully scheduling
Public bug reported: ENV: stable/queens, but master branch basically has the same code. Unexcepted HA router scheduled instance shows up after manully scheduling and admin-state down/up. Step to reproduce: $ openstack network agent list --router c0f96d58-5521-40fa-9536-205635facc69 --long +--++--+---+---+---+--+--+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | HA State | +--++--+---+---+---+--+--+ | 581cb0b8-10e9-41cf-bc4a-01054e5303f3 | L3 agent | network3 | nova | :-) | UP| neutron-l3-agent | active | | a195dc34-b325-4b5b-8dde-78a261ca5692 | L3 agent | network2 | nova | :-) | UP| neutron-l3-agent | standby | +--++--+---+---+---+--+--+ $ openstack network agent add router --l3 0eb1d073-8b2e-4fa3-bd27-43f2ec684334 c0f96d58-5521-40fa-9536-205635facc69 $ openstack network agent list --router c0f96d58-5521-40fa-9536-205635facc69 --long +--++--+---+---+---+--+--+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | HA State | +--++--+---+---+---+--+--+ | 0eb1d073-8b2e-4fa3-bd27-43f2ec684334 | L3 agent | network1 | nova | :-) | UP| neutron-l3-agent | standby | | 581cb0b8-10e9-41cf-bc4a-01054e5303f3 | L3 agent | network3 | nova | :-) | UP| neutron-l3-agent | active | | a195dc34-b325-4b5b-8dde-78a261ca5692 | L3 agent | network2 | nova | :-) | UP| neutron-l3-agent | standby | +--++--+---+---+---+--+--+ # Now remove router from l3-agent (network2) $ openstack network agent remove router --l3 a195dc34-b325-4b5b-8dde-78a261ca5692 c0f96d58-5521-40fa-9536-205635facc69 $ openstack network agent list --router c0f96d58-5521-40fa-9536-205635facc69 --long +--++--+---+---+---+--+--+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | HA State | +--++--+---+---+---+--+--+ | 0eb1d073-8b2e-4fa3-bd27-43f2ec684334 | L3 agent | network1 | nova | :-) | UP| neutron-l3-agent | standby | | 581cb0b8-10e9-41cf-bc4a-01054e5303f3 | L3 agent | network3 | nova | :-) | UP| neutron-l3-agent | active | +--++--+---+---+---+--+--+ # Admin state down/up the router $ openstack router set --disable c0f96d58-5521-40fa-9536-205635facc69 $ openstack router set --enable c0f96d58-5521-40fa-9536-205635facc69 # Unexcepted scheduled router instance shows up, router was scheduled to network2 $ openstack network agent list --router c0f96d58-5521-40fa-9536-205635facc69 --long +--++--+---+---+---+--+--+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | HA State | +--++--+---+---+---+--+--+ | 0eb1d073-8b2e-4fa3-bd27-43f2ec684334 | L3 agent | network1 | nova | :-) | UP| neutron-l3-agent | standby | | 581cb0b8-10e9-41cf-bc4a-01054e5303f3 | L3 agent | network3 | nova | :-) | UP| neutron-l3-agent | standby | | a195dc34-b325-4b5b-8dde-78a261ca5692 | L3 agent | network2 | nova | :-) | UP| neutron-l3-agent | standby | +--++--+---+---+---+--+--+ # checking the router HA state... $ openstack network agent list --router c0f96d58-5521-40fa-9536-205635facc69 --long +--++--+---+---+---+--+--+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | HA State | +--++--+---+-
[Yahoo-eng-team] [Bug 1871850] [NEW] [L3] existing router resources are partial deleted unexceptedly when MQ is gone
Public bug reported: ENV: meet this issue on our stable/queens deployment, but master branch has the same code logic When the L3 agent get a router update notification, it will try to retrieve the router info from DB server [1]. But at this time, if the message queue is down/unreachable. It will get exceptions related message queue. A resync action will be run then [2]. Sometimes, from my personal experience, rabbitMQ cluster is not so much easy to recover. Long time MQ recover time will cause the router info sync RPC never get successful until it meets the max retry time [3]. So the bad thing happens, L3 agent is trying to remove the router now [4]. It basically shutdown all the existing L3 traffic of this router. [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L705 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L710 [3] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L666 [4] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L671 ** Affects: neutron Importance: Critical Status: Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1871850 Title: [L3] existing router resources are partial deleted unexceptedly when MQ is gone Status in neutron: Confirmed Bug description: ENV: meet this issue on our stable/queens deployment, but master branch has the same code logic When the L3 agent get a router update notification, it will try to retrieve the router info from DB server [1]. But at this time, if the message queue is down/unreachable. It will get exceptions related message queue. A resync action will be run then [2]. Sometimes, from my personal experience, rabbitMQ cluster is not so much easy to recover. Long time MQ recover time will cause the router info sync RPC never get successful until it meets the max retry time [3]. So the bad thing happens, L3 agent is trying to remove the router now [4]. It basically shutdown all the existing L3 traffic of this router. [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L705 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L710 [3] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L666 [4] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L671 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1871850/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1871730] [NEW] [OVN] local conf devstack for a ovn-northd (DB only) node does not work
Public bug reported: Code branch: master Assuming you have 5 nodes to run a multi-node devstack deployment with neutron and OVN. One node for "ovn-northd" DB only. Two chassis for compute, and two for gateway. For the DB only node, if you do not set "ovn-controller" to the enable_services list. The stack.sh of devstack will get failed. Here is an example local.conf: http://paste.openstack.org/show/791811/ Error output: http://paste.openstack.org/show/791832/ ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1871730 Title: [OVN] local conf devstack for a ovn-northd (DB only) node does not work Status in neutron: New Bug description: Code branch: master Assuming you have 5 nodes to run a multi-node devstack deployment with neutron and OVN. One node for "ovn-northd" DB only. Two chassis for compute, and two for gateway. For the DB only node, if you do not set "ovn-controller" to the enable_services list. The stack.sh of devstack will get failed. Here is an example local.conf: http://paste.openstack.org/show/791811/ Error output: http://paste.openstack.org/show/791832/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1871730/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1867119] Re: [security] Add allowed-address-pair 0.0.0.0/0 to one port will open all others' protocol under same security group
** This bug is no longer a duplicate of bug 1793029 adding 0.0.0.0/0 address pair to a port bypasses all other vm security groups -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1867119 Title: [security] Add allowed-address-pair 0.0.0.0/0 to one port will open all others' protocol under same security group Status in neutron: In Progress Bug description: [security] Add allowed-address-pair 0.0.0.0/0 to one port will open all others' protocol under same security group When add allowed-address-pair 0.0.0.0/0 to one port, it will unexpectedly open all others' protocol under same security group. First found in stable/queens, but also confirmed in master branch. IPv6 has the same problem! Devstack test config: [DEFAULT] [l2pop] [ml2] type_drivers = flat,gre,vlan,vxlan tenant_network_types = vxlan extension_drivers = port_security,qos mechanism_drivers = openvswitch,l2population [ml2_type_vxlan] vni_ranges = 1:1 [securitygroup] firewall_driver = openvswitch [ovs] local_ip = 10.0.5.10 [agent] tunnel_types = vxlan l2_population = True arp_responder = True enable_distributed_routing = True extensions = qos Step to reproduce: 1. Assuming you have following VMs | 24231705-ee79-4643-ae5a-9f0f7ff8f8ba | dvr-ha-vm-2 | ACTIVE | dvr-ha=192.168.30.44, 172.16.12.220 | cirros | nano | | 4865d216-9f95-40bf-a6b4-221e3af06798 | dvr-ha-vm-1 | ACTIVE | dvr-ha=192.168.30.64, 172.16.13.52 | cirros | nano | $ nova interface-list 4865d216-9f95-40bf-a6b4-221e3af06798 ++--+--+---+---+-+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | Tag | ++--+--+---+---+-+ | ACTIVE | b333b1ca-bb9a-41fd-a878-b524ffbc6d7a | a9e82560-f1ac-4909-9afa-686b57df62fa | 192.168.30.64 | fa:16:3e:12:66:05 | - | ++--+--+---+---+-+ $ nova interface-list 24231705-ee79-4643-ae5a-9f0f7ff8f8ba ++--+--+---+---+-+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | Tag | ++--+--+---+---+-+ | ACTIVE | 93197f48-3fe4-47f4-9d15-ba8728c00409 | a9e82560-f1ac-4909-9afa-686b57df62fa | 192.168.30.44 | fa:16:3e:14:ff:f1 | - | ++--+--+---+---+-+ 2. Security group rules $ openstack security group rule list 535018b5-7038-46f2-8f0e-2a6e193788aa --long|grep ingress | 01015261-0ca3-49ad-b033-bc2036a58e26 | tcp | IPv4 | 0.0.0.0/0 | 22:22 | ingress | None | | 36441851-7bd2-4680-be43-2f8119b65040 | icmp| IPv4 | 0.0.0.0/0 || ingress | None | | 8326f59e-cf26-4372-9913-30c71c036a2e | None| IPv6 | ::/0 || ingress | 535018b5-7038-46f2-8f0e-2a6e193788aa | | e47c6731-a0f7-42aa-8125-a9810e7b5a17 | None| IPv4 | 0.0.0.0/0 || ingress | 535018b5-7038-46f2-8f0e-2a6e193788aa | 3. Start a nc test server in dvr-ha-vm-2 # nc -l -p 8000 4. Try to curl that dvr-ha-vm-2 port 8000 in the outside world $ curl http://172.16.12.220:8000/index.html curl: (7) Failed connect to 172.16.12.220:8000; Connection timed out 5. Add allowed address pair 0.0.0.0/0 to dvr-ha-vm-1 openstack port set --allowed-address ip-address=0.0.0.0/0 b333b1ca-bb9a-41fd-a878-b524ffbc6d7a 6. Try to curl that dvr-ha-vm-2 port 8000 again It is connected!!! # nc -l -p 8000 GET /index.html HTTP/1.1 User-Agent: curl/7.29.0 Host: 172.16.12.220:8000 Accept: */* asdfasdf asdfasdf To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1867119/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1867119] [NEW] [security] Add allowed-address-pair 0.0.0.0/0 to one port will open all others' protocol under same security group
Public bug reported: [security] Add allowed-address-pair 0.0.0.0/0 to one port will open all others' protocol under same security group When add allowed-address-pair 0.0.0.0/0 to one port, it will unexpectedly open all others' protocol under same security group. First found in stable/queens, but also confirmed in master branch. IPv6 has the same problem! Devstack test config: [DEFAULT] [l2pop] [ml2] type_drivers = flat,gre,vlan,vxlan tenant_network_types = vxlan extension_drivers = port_security,qos mechanism_drivers = openvswitch,l2population [ml2_type_vxlan] vni_ranges = 1:1 [securitygroup] firewall_driver = openvswitch [ovs] local_ip = 10.0.5.10 [agent] tunnel_types = vxlan l2_population = True arp_responder = True enable_distributed_routing = True extensions = qos Step to reproduce: 1. Assuming you have following VMs | 24231705-ee79-4643-ae5a-9f0f7ff8f8ba | dvr-ha-vm-2 | ACTIVE | dvr-ha=192.168.30.44, 172.16.12.220 | cirros | nano | | 4865d216-9f95-40bf-a6b4-221e3af06798 | dvr-ha-vm-1 | ACTIVE | dvr-ha=192.168.30.64, 172.16.13.52 | cirros | nano | $ nova interface-list 4865d216-9f95-40bf-a6b4-221e3af06798 ++--+--+---+---+-+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | Tag | ++--+--+---+---+-+ | ACTIVE | b333b1ca-bb9a-41fd-a878-b524ffbc6d7a | a9e82560-f1ac-4909-9afa-686b57df62fa | 192.168.30.64 | fa:16:3e:12:66:05 | - | ++--+--+---+---+-+ $ nova interface-list 24231705-ee79-4643-ae5a-9f0f7ff8f8ba ++--+--+---+---+-+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | Tag | ++--+--+---+---+-+ | ACTIVE | 93197f48-3fe4-47f4-9d15-ba8728c00409 | a9e82560-f1ac-4909-9afa-686b57df62fa | 192.168.30.44 | fa:16:3e:14:ff:f1 | - | ++--+--+---+---+-+ 2. Security group rules $ openstack security group rule list 535018b5-7038-46f2-8f0e-2a6e193788aa --long|grep ingress | 01015261-0ca3-49ad-b033-bc2036a58e26 | tcp | IPv4 | 0.0.0.0/0 | 22:22 | ingress | None | | 36441851-7bd2-4680-be43-2f8119b65040 | icmp| IPv4 | 0.0.0.0/0 | | ingress | None | | 8326f59e-cf26-4372-9913-30c71c036a2e | None| IPv6 | ::/0 | | ingress | 535018b5-7038-46f2-8f0e-2a6e193788aa | | e47c6731-a0f7-42aa-8125-a9810e7b5a17 | None| IPv4 | 0.0.0.0/0 | | ingress | 535018b5-7038-46f2-8f0e-2a6e193788aa | 3. Start a nc test server in dvr-ha-vm-2 # nc -l -p 8000 4. Try to curl that dvr-ha-vm-2 port 8000 in the outside world $ curl http://172.16.12.220:8000/index.html curl: (7) Failed connect to 172.16.12.220:8000; Connection timed out 5. Add allowed address pair 0.0.0.0/0 to dvr-ha-vm-1 openstack port set --allowed-address ip-address=0.0.0.0/0 b333b1ca-bb9a-41fd-a878-b524ffbc6d7a 6. Try to curl that dvr-ha-vm-2 port 8000 again It is connected!!! # nc -l -p 8000 GET /index.html HTTP/1.1 User-Agent: curl/7.29.0 Host: 172.16.12.220:8000 Accept: */* asdfasdf asdfasdf ** Affects: neutron Importance: Critical Status: Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1867119 Title: [security] Add allowed-address-pair 0.0.0.0/0 to one port will open all others' protocol under same security group Status in neutron: Confirmed Bug description: [security] Add allowed-address-pair 0.0.0.0/0 to one port will open all others' protocol under same security group When add allowed-address-pair 0.0.0.0/0 to one port, it will unexpectedly open all others' protocol under same security group. First found in stable/queens, but also confirmed in master branch. IPv6 has the same problem! Devstack test config: [DEFAULT] [l2pop] [ml2] type_drivers = flat,gre,vlan,vxlan tenant_network_types = vxlan extension_drivers = port_security,qos mechanism_drivers = openvswitch,l2population [ml2_type_vxlan] vni_ranges = 1:1 [securitygroup] firewall_driver = openvswitch [ovs] local_ip = 10.0.5.10 [agent] tunnel_types = vxlan l2_popula
[Yahoo-eng-team] [Bug 1867101] [NEW] Deployment has security group with empty tenant id
Public bug reported: ENV: devstack, master $ openstack security group list +--+-++--+--+ | ID | Name| Description| Project | Tags | +--+-++--+--+ | 2661ae74-f946-4ef9-b676-fe9be4274c1b | default | Default security group | | [] | | 535018b5-7038-46f2-8f0e-2a6e193788aa | default | Default security group | ae05deb7a16c485986c3666b65f71c71 | [] | | c5d1b354-9896-4e2c-aeab-67c8cd20a489 | default | Default security group | 972c01461f1b4441b8d8648691bb89ff | [] | +--+-++--+--+ The empty project id (tenant id) causes some mechanism_driver like ODL failed to init, and log errors. ** Affects: neutron Importance: Low Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1867101 Title: Deployment has security group with empty tenant id Status in neutron: New Bug description: ENV: devstack, master $ openstack security group list +--+-++--+--+ | ID | Name| Description| Project | Tags | +--+-++--+--+ | 2661ae74-f946-4ef9-b676-fe9be4274c1b | default | Default security group | | [] | | 535018b5-7038-46f2-8f0e-2a6e193788aa | default | Default security group | ae05deb7a16c485986c3666b65f71c71 | [] | | c5d1b354-9896-4e2c-aeab-67c8cd20a489 | default | Default security group | 972c01461f1b4441b8d8648691bb89ff | [] | +--+-++--+--+ The empty project id (tenant id) causes some mechanism_driver like ODL failed to init, and log errors. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1867101/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1866445] Re: br-int bridge in one compute can't learn MAC addresses of VMs in other compute nodes
** This bug is no longer a duplicate of bug 1732067 openvswitch firewall flows cause flooding on integration bridge -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1866445 Title: br-int bridge in one compute can't learn MAC addresses of VMs in other compute nodes Status in neutron: Incomplete Bug description: In Openstack Queens release, we noticed a very serious issue, br-int bridge in one compute node can't learn MAC addresses of VMs in other compute nodes, so after launched many VMs, VM-to-VM network performance will decrease linearly, especially ovs will broadcast packets because it doesn't learn target VM MAC address, other VMs in same subnet in same compute node can receive these broadcast packets, therefore, the corresponding vhost kernel threads are receiving these packets and wasting CPU resources. More VMs, more serious the issue, worse the performance, no matter UDP or TCP performance. We have checked several Queens deployments, they have same issues, but Openstack Rocky doesn't have this issue. Here is the flow I dumped: recirc_id(0),in_port(12),eth(src=fa:16:3e:49:26:51,dst=fa:16:3e:a7:0a:3a),eth_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:11012944, bytes:726983412, used:0.000s, flags:SP., actions:push_vlan(vid=1,pcp=0),2,set(tunnel(tun_id=0x49,src=10.3.2.17,dst=10.3.2.16,ttl=64,tp_dst=4789,flags(df|key))),pop_vlan,9,8,11,13,14,15,16,17,18,19 MAC address of target VM wasn't learnt by br-int $ sudo ovs-appctl fdb/show br-int | grep "fa:16:3e:a7:0a:3a" By the way, we used linuxbridge for security group. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1866445/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1866077] [NEW] [L3][IPv6] IPv6 traffic with DVR in compute host
Public bug reported: One question: how to let the IPv6 traffic to the outside world run directly in the compute host? We have a BP before: https://blueprints.launchpad.net/neutron/+spec/ipv6-router-and-dvr And one spec for it: https://review.opendev.org/#/c/136878/ ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1866077 Title: [L3][IPv6] IPv6 traffic with DVR in compute host Status in neutron: New Bug description: One question: how to let the IPv6 traffic to the outside world run directly in the compute host? We have a BP before: https://blueprints.launchpad.net/neutron/+spec/ipv6-router-and-dvr And one spec for it: https://review.opendev.org/#/c/136878/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1866077/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1863830] [NEW] Testing ERROR: "Exit code: 2; Stdin: ; Stdout: ; Stderr: Ncat: Invalid -w timeout (must be greater than 0). QUITTING."
Public bug reported: Branch: neutron master, HEAD: commit 7a0e5185c6cf7b5f8bcfe50576e86798947a7ba7 Exception: File "/home/yulong/github/neutron/neutron/agent/l3/dvr_edge_router.py", line 160, in initialize self._create_snat_namespace() File "/home/yulong/github/neutron/neutron/agent/l3/dvr_edge_router.py", line 181, in _create_snat_namespace self.snat_namespace.create() File "/home/yulong/github/neutron/neutron/agent/l3/dvr_snat_ns.py", line 32, in create super(SnatNamespace, self).create() File "/home/yulong/github/neutron/neutron/agent/l3/namespaces.py", line 95, in create ip_wrapper = self.ip_wrapper_root.ensure_namespace(self.name) File "/home/yulong/github/neutron/neutron/agent/linux/ip_lib.py", line 243, in ensure_namespace ip = self.netns.add(name) File "/home/yulong/github/neutron/neutron/agent/linux/ip_lib.py", line 696, in add 'net.ipv4.conf.all.promote_secondaries=1']) File "/home/yulong/github/neutron/neutron/agent/linux/ip_lib.py", line 718, in execute run_as_root=run_as_root) File "/home/yulong/github/neutron/neutron/agent/linux/utils.py", line 147, in execute returncode=returncode) neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: Ncat: Invalid -w timeout (must be greater than 0). QUITTING. Upstream log search: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Exit%20code%3A%202%3B%20Stdin%3A%20%3B%20Stdout%3A%20%3B%20Stderr%3A%20Ncat%3A%20Invalid%20-w%20timeout%20(must%20be%20greater%20than%200).%20QUITTING.%5C%22 local testing: http://paste.openstack.org/show/789733/ ** Affects: neutron Importance: Undecided Status: New ** Summary changed: - Exit code: 2; Stdin: ; Stdout: ; Stderr: Ncat: Invalid -w timeout (must be greater than 0). QUITTING. + Testing ERROR: "Exit code: 2; Stdin: ; Stdout: ; Stderr: Ncat: Invalid -w timeout (must be greater than 0). QUITTING." -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1863830 Title: Testing ERROR: "Exit code: 2; Stdin: ; Stdout: ; Stderr: Ncat: Invalid -w timeout (must be greater than 0). QUITTING." Status in neutron: New Bug description: Branch: neutron master, HEAD: commit 7a0e5185c6cf7b5f8bcfe50576e86798947a7ba7 Exception: File "/home/yulong/github/neutron/neutron/agent/l3/dvr_edge_router.py", line 160, in initialize self._create_snat_namespace() File "/home/yulong/github/neutron/neutron/agent/l3/dvr_edge_router.py", line 181, in _create_snat_namespace self.snat_namespace.create() File "/home/yulong/github/neutron/neutron/agent/l3/dvr_snat_ns.py", line 32, in create super(SnatNamespace, self).create() File "/home/yulong/github/neutron/neutron/agent/l3/namespaces.py", line 95, in create ip_wrapper = self.ip_wrapper_root.ensure_namespace(self.name) File "/home/yulong/github/neutron/neutron/agent/linux/ip_lib.py", line 243, in ensure_namespace ip = self.netns.add(name) File "/home/yulong/github/neutron/neutron/agent/linux/ip_lib.py", line 696, in add 'net.ipv4.conf.all.promote_secondaries=1']) File "/home/yulong/github/neutron/neutron/agent/linux/ip_lib.py", line 718, in execute run_as_root=run_as_root) File "/home/yulong/github/neutron/neutron/agent/linux/utils.py", line 147, in execute returncode=returncode) neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: Ncat: Invalid -w timeout (must be greater than 0). QUITTING. Upstream log search: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Exit%20code%3A%202%3B%20Stdin%3A%20%3B%20Stdout%3A%20%3B%20Stderr%3A%20Ncat%3A%20Invalid%20-w%20timeout%20(must%20be%20greater%20than%200).%20QUITTING.%5C%22 local testing: http://paste.openstack.org/show/789733/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1863830/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1859638] [NEW] VIP between dvr east-west networks does not work at all
Public bug reported: ENV: devstack with master branch neutron HEAD: ab24a11f13cdfdf623a4b696f469aa621d59405b Reproduce: 1. network1 + subnet1: 192.168.1.0/24 2. network2 + subnet2: 192.168.2.0/24 3. dvr router with attached subnets: subnet1, subnet2 4. VM1 (192.168.1.10) and VM2 (192.168.1.11) created from network1, VM3 (192.168.2.20) from network2 5. create a port from network1 and its fixed-IP-1 (192.168.1.100) will be used as VIP 6. set VM1 & VM2 port allowed-address-pair: openstack port set --allowed-address ip-address=192.168.1.100 VM1-port/VM2-port 7. set the VIP to the NIC inside the VM1 or VM2 8. try to access/ping VIP fixed-IP-1 (192.168.1.100) from VM3 Excepted behavior: pingable, 192.168.2.20 can reach 192.168.1.100 Actual behavior: not reachable ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1859638 Title: VIP between dvr east-west networks does not work at all Status in neutron: New Bug description: ENV: devstack with master branch neutron HEAD: ab24a11f13cdfdf623a4b696f469aa621d59405b Reproduce: 1. network1 + subnet1: 192.168.1.0/24 2. network2 + subnet2: 192.168.2.0/24 3. dvr router with attached subnets: subnet1, subnet2 4. VM1 (192.168.1.10) and VM2 (192.168.1.11) created from network1, VM3 (192.168.2.20) from network2 5. create a port from network1 and its fixed-IP-1 (192.168.1.100) will be used as VIP 6. set VM1 & VM2 port allowed-address-pair: openstack port set --allowed-address ip-address=192.168.1.100 VM1-port/VM2-port 7. set the VIP to the NIC inside the VM1 or VM2 8. try to access/ping VIP fixed-IP-1 (192.168.1.100) from VM3 Excepted behavior: pingable, 192.168.2.20 can reach 192.168.1.100 Actual behavior: not reachable To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1859638/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1858262] Re: "CREATE TABLE ovn_hash_ring" Specified key was too long; max key length is 767 bytes
** Changed in: neutron Status: Invalid => New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1858262 Title: "CREATE TABLE ovn_hash_ring" Specified key was too long; max key length is 767 bytes Status in neutron: New Bug description: During the DB upgrading of neutron, the following error raised: CREATE TABLE ovn_hash_ring ( node_uuid VARCHAR(36) NOT NULL, group_name VARCHAR(256) NOT NULL, hostname VARCHAR(256) NOT NULL, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, PRIMARY KEY (node_uuid, group_name) )ENGINE=InnoDB ERROR: 1071 Specified key was too long; max key length is 767 bytes Execution Time : 00:00:00:000 Transfer Time : 00:00:00:000 Total Time : 00:00:00:000 [1] neutron/db/migration/alembic_migrations/versions/ussuri/expand/f4b9654dd40c_ovn_backend.py To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1858262/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1858262] Re: "CREATE TABLE ovn_hash_ring" Specified key was too long; max key length is 767 bytes
** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1858262 Title: "CREATE TABLE ovn_hash_ring" Specified key was too long; max key length is 767 bytes Status in neutron: Invalid Bug description: During the DB upgrading of neutron, the following error raised: CREATE TABLE ovn_hash_ring ( node_uuid VARCHAR(36) NOT NULL, group_name VARCHAR(256) NOT NULL, hostname VARCHAR(256) NOT NULL, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, PRIMARY KEY (node_uuid, group_name) )ENGINE=InnoDB ERROR: 1071 Specified key was too long; max key length is 767 bytes Execution Time : 00:00:00:000 Transfer Time : 00:00:00:000 Total Time : 00:00:00:000 [1] neutron/db/migration/alembic_migrations/versions/ussuri/expand/f4b9654dd40c_ovn_backend.py To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1858262/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1858262] [NEW] "CREATE TABLE ovn_hash_ring" Specified key was too long; max key length is 767 bytes
Public bug reported: During the DB upgrading of neutron, the following error raised: CREATE TABLE ovn_hash_ring ( node_uuid VARCHAR(36) NOT NULL, group_name VARCHAR(256) NOT NULL, hostname VARCHAR(256) NOT NULL, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, PRIMARY KEY (node_uuid, group_name) )ENGINE=InnoDB ERROR: 1071 Specified key was too long; max key length is 767 bytes Execution Time : 00:00:00:000 Transfer Time : 00:00:00:000 Total Time : 00:00:00:000 [1] neutron/db/migration/alembic_migrations/versions/ussuri/expand/f4b9654dd40c_ovn_backend.py ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1858262 Title: "CREATE TABLE ovn_hash_ring" Specified key was too long; max key length is 767 bytes Status in neutron: New Bug description: During the DB upgrading of neutron, the following error raised: CREATE TABLE ovn_hash_ring ( node_uuid VARCHAR(36) NOT NULL, group_name VARCHAR(256) NOT NULL, hostname VARCHAR(256) NOT NULL, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, PRIMARY KEY (node_uuid, group_name) )ENGINE=InnoDB ERROR: 1071 Specified key was too long; max key length is 767 bytes Execution Time : 00:00:00:000 Transfer Time : 00:00:00:000 Total Time : 00:00:00:000 [1] neutron/db/migration/alembic_migrations/versions/ussuri/expand/f4b9654dd40c_ovn_backend.py To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1858262/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1858260] [NEW] Upstream CI neutron-tempest-plugin-* fails
Public bug reported: neutron-tempest-plugin-dvr-multinode-scenario FAILURE in 1h 22m 34s (non-voting) neutron-tempest-plugin-scenario-linuxbridge FAILURE in 1h 01m 47s neutron-tempest-plugin-scenario-openvswitch FAILURE in 1h 02m 16s neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid FAILURE in 54m 44s Example 1: https://review.opendev.org/#/c/701071/ https://233b44f5e68656358e60-d611ee375f3e2efb503ca51cfa66bd96.ssl.cf1.rackcdn.com/701071/1/check/neutron-tempest-plugin-dvr-multinode-scenario/3443865/testr_results.html.gz https://e0c6de5c870dee31ebc0-781059ce20a289d02f0d95bf10cdd7d3.ssl.cf5.rackcdn.com/701071/1/check/neutron-tempest-plugin-scenario-linuxbridge/ed6625c/testr_results.html.gz https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_de3/701071/1/check/neutron-tempest-plugin-scenario-openvswitch/de3fa6d/testr_results.html.gz https://1e4c070c6d8fc22f3cfe-bf313b0386a54af757ff2672eb4d7d35.ssl.cf1.rackcdn.com/701071/1/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/d002b9e/testr_results.html.gz Example 2: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_138/666991/10/check/neutron-tempest-plugin-dvr-multinode-scenario/1383725/testr_results.html.gz https://4ae644854fb3bf106e9b-6877b85dbe482cd2daa62a6731b06023.ssl.cf1.rackcdn.com/666991/10/check/neutron-tempest-plugin-scenario-linuxbridge/37d350f/testr_results.html.gz https://26b132603a8765e63b5c-445d4465f34d2b24df5d805a76ff9803.ssl.cf5.rackcdn.com/666991/10/check/neutron-tempest-plugin-scenario-openvswitch/928b90f/testr_results.html.gz https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_175/666991/10/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/175961b/testr_results.html.gz Example 3: https://review.opendev.org/#/c/701003/ ** Affects: neutron Importance: Undecided Status: New ** Summary changed: - neutron-tempest-plugin-dvr-multinode-scenario FAILURE in 1h 22m 34s (non-voting) neutron-tempest-plugin-scenario-linuxbridge FAILURE in 1h 01m 47s neutron-tempest-plugin-scenario-openvswitch FAILURE in 1h 02m 16s neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid FAILURE in 54m 44s + Upstream CI neutron-tempest-plugin-* fails -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1858260 Title: Upstream CI neutron-tempest-plugin-* fails Status in neutron: New Bug description: neutron-tempest-plugin-dvr-multinode-scenario FAILURE in 1h 22m 34s (non-voting) neutron-tempest-plugin-scenario-linuxbridge FAILURE in 1h 01m 47s neutron-tempest-plugin-scenario-openvswitch FAILURE in 1h 02m 16s neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid FAILURE in 54m 44s Example 1: https://review.opendev.org/#/c/701071/ https://233b44f5e68656358e60-d611ee375f3e2efb503ca51cfa66bd96.ssl.cf1.rackcdn.com/701071/1/check/neutron-tempest-plugin-dvr-multinode-scenario/3443865/testr_results.html.gz https://e0c6de5c870dee31ebc0-781059ce20a289d02f0d95bf10cdd7d3.ssl.cf5.rackcdn.com/701071/1/check/neutron-tempest-plugin-scenario-linuxbridge/ed6625c/testr_results.html.gz https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_de3/701071/1/check/neutron-tempest-plugin-scenario-openvswitch/de3fa6d/testr_results.html.gz https://1e4c070c6d8fc22f3cfe-bf313b0386a54af757ff2672eb4d7d35.ssl.cf1.rackcdn.com/701071/1/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/d002b9e/testr_results.html.gz Example 2: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_138/666991/10/check/neutron-tempest-plugin-dvr-multinode-scenario/1383725/testr_results.html.gz https://4ae644854fb3bf106e9b-6877b85dbe482cd2daa62a6731b06023.ssl.cf1.rackcdn.com/666991/10/check/neutron-tempest-plugin-scenario-linuxbridge/37d350f/testr_results.html.gz https://26b132603a8765e63b5c-445d4465f34d2b24df5d805a76ff9803.ssl.cf5.rackcdn.com/666991/10/check/neutron-tempest-plugin-scenario-openvswitch/928b90f/testr_results.html.gz https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_175/666991/10/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/175961b/testr_results.html.gz Example 3: https://review.opendev.org/#/c/701003/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1858260/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1856839] [NEW] [L3] router processing time increase if there are large large set ports
Public bug reported: The function "_update_arp_entry" [1] was called under a double loop [2][3], and it has a "device.exists()" check [4]. When there are tons of ports under the router, the router processing time will definitely increase. [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L249 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L288-L290 [3] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L291 [4] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L260 ** Affects: neutron Importance: Medium Assignee: LIU Yulong (dragon889) Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1856839 Title: [L3] router processing time increase if there are large large set ports Status in neutron: New Bug description: The function "_update_arp_entry" [1] was called under a double loop [2][3], and it has a "device.exists()" check [4]. When there are tons of ports under the router, the router processing time will definitely increase. [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L249 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L288-L290 [3] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L291 [4] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L260 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1856839/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840979] Re: [L2] update the port DB status directly in agent-side
** Changed in: neutron Assignee: (unassigned) => LIU Yulong (dragon889) ** Changed in: neutron Status: Won't Fix => New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1840979 Title: [L2] update the port DB status directly in agent-side Status in neutron: New Bug description: When ovs-agent done processing the port, it will call neutron-server to make some DB update. Especially when restart the ovs-agent, all ports in one agent will do such RPC and DB update again to make port status consistent. When a large number of concurrent agent restart happen, neutron-server may not work fine. So how about making the following DB updating locally in neutron agent side directly? It may have some mechanism driver notification, IMO, this can also be done in agent-side. def update_device_down(self, context, device, agent_id, host=None): cctxt = self.client.prepare() return cctxt.call(context, 'update_device_down', device=device, agent_id=agent_id, host=host) def update_device_up(self, context, device, agent_id, host=None): cctxt = self.client.prepare() return cctxt.call(context, 'update_device_up', device=device, agent_id=agent_id, host=host) def update_device_list(self, context, devices_up, devices_down, ret = cctxt.call(context, 'update_device_list', To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1840979/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1854051] [NEW] py36 unit test cases fails
Public bug reported: This should be a NOTE, not a bug in case someone who meets this issue someday, since the minimum support python version of neutron is 3.7 now. Branch: master heads: 2a8b70d Merge "Update security group rule if port range is all ports" fd5e292 Merge "Remove neutron-grenade job from Neutron CI queues" f6aef3c Merge "Switch neutron-tempest-with-os-ken-master job to zuul v3" 2174bb0 Merge "Remove old, legacy experimental CI jobs" 8672029 Merge "HA race condition test for DHCP scheduling" 71e3cb0 Merge "Parameter 'fileds' value is not used in _get_subnets" b5e5082 Merge "Update networking-bgpvpn and networking-bagpipe liuetenants" 3c1139c Merge "Make network support read and write separation" 67b613b Merge "NetcatTester.stop_processes skip "No such process" exception" 185efb3 Update networking-bgpvpn and networking-bagpipe liuetenants 728d8ee NetcatTester.stop_processes skip "No such process" exception Tox env was definitely upgraded to meet the requirements.txt and test-requirements.txt Exceptions: == Failed 2 tests - output below: == neutron.tests.unit.plugins.ml2.drivers.openvswitch.agent.test_ovs_neutron_agent.TestOvsDvrNeutronAgentOSKen.test_get_dvr_mac_address_exception -- Captured traceback: ~~~ b'Traceback (most recent call last):' b' File "/home/yulong/github/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 164, in get_dvr_mac_address' b'self.get_dvr_mac_address_with_retry()' b' File "/home/yulong/github/neutron/.tox/py36/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper' b'result = f(*args, **kwargs)' b' File "/home/yulong/github/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 184, in get_dvr_mac_address_with_retry' b'self.context, self.host)' b' File "/home/yulong/github/neutron/.tox/py36/lib/python3.6/site-packages/mock/mock.py", line 1092, in __call__' b'return _mock_self._mock_call(*args, **kwargs)' b' File "/home/yulong/github/neutron/.tox/py36/lib/python3.6/site-packages/mock/mock.py", line 1143, in _mock_call' b'raise effect' b'oslo_messaging.rpc.client.RemoteError: Remote error: None None' b'None.' b'' b'During handling of the above exception, another exception occurred:' b'' b'Traceback (most recent call last):' b' File "/home/yulong/github/neutron/neutron/tests/base.py", line 182, in func' b'return f(self, *args, **kwargs)' b' File "/home/yulong/github/neutron/neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/test_ovs_neutron_agent.py", line 3614, in test_get_dvr_mac_address_exception' b'self.agent.dvr_agent.get_dvr_mac_address()' b' File "/home/yulong/github/neutron/.tox/py36/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper' b'result = f(*args, **kwargs)' b' File "/home/yulong/github/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py", line 169, in get_dvr_mac_address' b"'message: %s', e)" b' File "/usr/lib64/python3.6/logging/__init__.py", line 1653, in error' b'self.log(ERROR, msg, *args, **kwargs)' b' File "/usr/lib64/python3.6/logging/__init__.py", line 1674, in log' b'self.logger.log(level, msg, *args, **kwargs)' b' File "/usr/lib64/python3.6/logging/__init__.py", line 1374, in log' b'self._log(level, msg, args, **kwargs)' b' File "/usr/lib64/python3.6/logging/__init__.py", line 1443, in _log' b'exc_info, func, extra, sinfo)' b' File "/usr/lib64/python3.6/logging/__init__.py", line 1413, in makeRecord' b'sinfo)' b' File "/usr/lib64/python3.6/logging/__init__.py", line 277, in __init__' b'if (args and len(args) == 1 and isinstance(args[0], collections.Mapping)' b' File "/home/yulong/github/neutron/.tox/py36/lib64/python3.6/abc.py", line 193, in __instancecheck__' b'return cls.__subclasscheck__(subclass)' b' File "/home/yulong/github/neutron/.tox/py36/lib64/python3.6/abc.py", line 228, in __subclasscheck__' b'if issubclass(subclass, scls):' b' File "/home/yulong/github/neutron/.tox/py36/lib64/python3.6/abc.py", line 228, in __subclasscheck__' b'if issubclass(subclass, scls):' b' File "/usr/lib64/python3.6/typing.py", line 1154, in __subclasscheck__' b'return super().__subclasscheck__(cls)' b' File "/home/yulong/github/neutron/.tox/py36/lib64/python3.6/abc.py", line 209, in __subclasscheck__' b'ok = cls.__subclasshook__(subclass)' b' File "/usr/lib64/python3.6/typing.py", line 884, in __extrahook__' b'if issubclass(subclass, scls):' b' File "/usr/lib64/python3.6/typing.py"
[Yahoo-eng-team] [Bug 1850779] [NEW] [L3] snat-ns will be initialized twice for DVR+HA routers during agent restart
Public bug reported: If the DVR+HA router has external gateway, the snat-namespace will be initialized twice during agent restart. And that initialized function will run many [1][2] external resource processing actions which will definitely increase the starting time of agent. https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39 https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108 ** Affects: neutron Importance: Medium Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1850779 Title: [L3] snat-ns will be initialized twice for DVR+HA routers during agent restart Status in neutron: New Bug description: If the DVR+HA router has external gateway, the snat-namespace will be initialized twice during agent restart. And that initialized function will run many [1][2] external resource processing actions which will definitely increase the starting time of agent. https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39 https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1850779/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1849510] [NEW] Multiple IPv6 subnets with SLAAC
Public bug reported: For one network with multiple slaac IPv6 subnets, the created port will have all IPv6 subnets address automatically by default. For some use case, we do not want the port to have all the IPv6 address from all IPv6 subnets, but only one of it. It is a behavior for neutron now. This is the code: https://github.com/openstack/neutron/blob/master/neutron/db/ipam_pluggable_backend.py#L256 How to reproduce: $ openstack network create ipv6-net $ openstack subnet create --network ipv6-net --ip-version 4 --subnet-range 192.168.1.0/24 ipv4-sub-1 $ openstack subnet create --network ipv6-net --ip-version 4 --subnet-range 192.168.2.0/24 ipv4-sub-2 $ openstack subnet create --network ipv6-net --ip-version 6 --subnet-range 2001::/64 --ipv6-ra-mode slaac --ipv6-address-mode slaac ipv6-sub-1 $ openstack subnet create --network ipv6-net --ip-version 6 --subnet-range 2002::/64 --ipv6-ra-mode slaac --ipv6-address-mode slaac ipv6-sub-2 create a port, you will see it will have all IPv6 address from all IPv6 subnets: openstack port create --network ipv6-net ipv6-port-1 | fixed_ips | ip_address='192.168.1.19', subnet_id='8e0d9fc0-de72-47fa-b3fc-384f19f6c0ae' | | | ip_address='2001::f816:3eff:fe49:651b', subnet_id='6352875f-c406-4a7c-9722-857ee9c58455' | | | ip_address='2002::f816:3eff:fe49:651b', subnet_id='193b76ec-8527-4c48-9ad6-7922d0d2e63d' | | id| 1736e437-fb8e-442b-9b04-343eae48a250 | | ip_address| None | | mac_address | fa:16:3e:49:65:1b | ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1849510 Title: Multiple IPv6 subnets with SLAAC Status in neutron: New Bug description: For one network with multiple slaac IPv6 subnets, the created port will have all IPv6 subnets address automatically by default. For some use case, we do not want the port to have all the IPv6 address from all IPv6 subnets, but only one of it. It is a behavior for neutron now. This is the code: https://github.com/openstack/neutron/blob/master/neutron/db/ipam_pluggable_backend.py#L256 How to reproduce: $ openstack network create ipv6-net $ openstack subnet create --network ipv6-net --ip-version 4 --subnet-range 192.168.1.0/24 ipv4-sub-1 $ openstack subnet create --network ipv6-net --ip-version 4 --subnet-range 192.168.2.0/24 ipv4-sub-2 $ openstack subnet create --network ipv6-net --ip-version 6 --subnet-range 2001::/64 --ipv6-ra-mode slaac --ipv6-address-mode slaac ipv6-sub-1 $ openstack subnet create --network ipv6-net --ip-version 6 --subnet-range 2002::/64 --ipv6-ra-mode slaac --ipv6-address-mode slaac ipv6-sub-2 create a port, you will see it will have all IPv6 address from all IPv6 subnets: openstack port create --network ipv6-net ipv6-port-1 | fixed_ips | ip_address='192.168.1.19', subnet_id='8e0d9fc0-de72-47fa-b3fc-384f19f6c0ae' | | | ip_address='2001::f816:3eff:fe49:651b', subnet_id='6352875f-c406-4a7c-9722-857ee9c58455' | | | ip_address='2002::f816:3eff:fe49:651b', subnet_id='193b76ec-8527-4c48-9ad6-7922d0d2e63d' | | id| 1736e437-fb8e-442b-9b04-343eae48a250 | | ip_address| None | | mac_address | fa:16:3e:49:65:1b | To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1849510/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1847747] [NEW] [RPC] digging RPC timeout for client and server
Public bug reported: RPC timeout can be found frequently, but we have no statistical data for it. A simple log can help. Since all the projects are using oslo.messaging as midlware between services and message queue, we can add a log in it, something like this, a local test: 2019-10-11 19:23:05.703 40774 INFO oslo_messaging.rpc.server [-] Receive incoming message with id cf4ab75b89e64458bc989c709add43cf and method: sync_routers. 2019-10-11 19:29:19.781 40774 INFO oslo_messaging.rpc.server:188 [None req-e33a11a2-64cd-4aea-b801-7689996a8ace - - - - -] Replied message with id cf4ab75b89e64458bc989c709add43cf and method: sync_routers. Time elapsed: 374.078 ** Affects: neutron Importance: Undecided Status: New ** Affects: oslo.messaging Importance: Undecided Status: New ** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1847747 Title: [RPC] digging RPC timeout for client and server Status in neutron: New Status in oslo.messaging: New Bug description: RPC timeout can be found frequently, but we have no statistical data for it. A simple log can help. Since all the projects are using oslo.messaging as midlware between services and message queue, we can add a log in it, something like this, a local test: 2019-10-11 19:23:05.703 40774 INFO oslo_messaging.rpc.server [-] Receive incoming message with id cf4ab75b89e64458bc989c709add43cf and method: sync_routers. 2019-10-11 19:29:19.781 40774 INFO oslo_messaging.rpc.server:188 [None req-e33a11a2-64cd-4aea-b801-7689996a8ace - - - - -] Replied message with id cf4ab75b89e64458bc989c709add43cf and method: sync_routers. Time elapsed: 374.078 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1847747/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1845145] [NEW] [L3] add abilitiy for iptables_manager to ensure rule was added only once
Public bug reported: iptables_manager should have abilitiy to ensure rule was added only once. In function [1], it just adds the new rule to the cache list no matter if it is duplicated. And finally, warning LOG [2] will be raised. Sometimes, there will have multiple threads to add rule for one same resource, it may be not easy for users to ensure that their rule generation code was run as expected. So rule will be duplicated in cache. And during the removal procedure, cache has duplicated rules, remove one then there still has same rule remained. As a result, the linux netfilter rule may have nothing changed after user's removal action. [1] https://github.com/openstack/neutron/blob/master/neutron/agent/linux/iptables_manager.py#L205-L225 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/linux/iptables_manager.py#L718-L725 ** Affects: neutron Importance: High Assignee: LIU Yulong (dragon889) Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1845145 Title: [L3] add abilitiy for iptables_manager to ensure rule was added only once Status in neutron: New Bug description: iptables_manager should have abilitiy to ensure rule was added only once. In function [1], it just adds the new rule to the cache list no matter if it is duplicated. And finally, warning LOG [2] will be raised. Sometimes, there will have multiple threads to add rule for one same resource, it may be not easy for users to ensure that their rule generation code was run as expected. So rule will be duplicated in cache. And during the removal procedure, cache has duplicated rules, remove one then there still has same rule remained. As a result, the linux netfilter rule may have nothing changed after user's removal action. [1] https://github.com/openstack/neutron/blob/master/neutron/agent/linux/iptables_manager.py#L205-L225 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/linux/iptables_manager.py#L718-L725 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1845145/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1844168] [NEW] [L3] TooManyExternalNetworks: More than one external network exists.
Public bug reported: Code: master with nothing changed. Exception: Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server [None req-9b3e8e62-b6b3-4506-8950-f73c3e5e2be3 None None] Exception during message handling: TooManyExternalNetworks: More than one external network exists. Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 253, in get_external_network_id Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server net_id = self.plugin.get_external_network_id(context) Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/db/external_net_db.py", line 149, in get_external_network_id Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server raise n_exc.TooManyExternalNetworks() Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server TooManyExternalNetworks: More than one external network exists. Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server Upstream log search: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TooManyExternalNetworks%3A%20More%20than%20one%20external%20network%20exists.%5C%22 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1844168 Title: [L3] TooManyExternalNetworks: More than one external network exists. Status in neutron: New Bug description: Code: master with nothing changed. Exception: Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server [None req-9b3e8e62-b6b3-4506-8950-f73c3e5e2be3 None None] Exception during message handling: TooManyExternalNetworks: More than one external network exists. Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 253, in get_external_network_id Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server net_id = self.plugin.get_external_network_id(context) Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/db/external_net_db.py", line 149, in get_external_network_id Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server raise n_exc.TooManyExternalNetworks() Sep 17 00:41:16 controller neutron-server[10222]: ERROR oslo_messaging.rpc.server TooManyExternalNetworks: More than one external network exists. Sep 17 00:41:16 controller neutron-server[102
[Yahoo-eng-team] [Bug 1841865] [NEW] [L2] stop processing ports twice in ovs-agent
Public bug reported: When port installed to the agent, it will be processed in rpc_loop X as "added". In next X + 1 rpc_loop, it will be processed again as "updated". This is unnecessary. And it can highly probably increase the processing time of new "added" ports in this X+1 loop. We have do something for the first rpc_loop 0 during agent restart, the ports will not be processed again. But for the running loop, we should re-examine such double processing mechanism or bug. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1841865 Title: [L2] stop processing ports twice in ovs-agent Status in neutron: New Bug description: When port installed to the agent, it will be processed in rpc_loop X as "added". In next X + 1 rpc_loop, it will be processed again as "updated". This is unnecessary. And it can highly probably increase the processing time of new "added" ports in this X+1 loop. We have do something for the first rpc_loop 0 during agent restart, the ports will not be processed again. But for the running loop, we should re-examine such double processing mechanism or bug. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1841865/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1841622] [NEW] [L2][OVS] add accepted egress fdb flows
Public bug reported: Bug https://bugs.launchpad.net/neutron/+bug/1732067 has a bad impact on VM traffic. And all the fix has some potenial risk of data-plane down. So we added a new bug for the new solution: It will add a flow table something like a switch FDB table. The accepted egress flows will be take care in that. table=94 will be used to do accept egress traffic classification when enable openflow firewall: 1. the "dest mac" is handled this ovs-agent, direct "output" to that port 2. "ARP request" with enabled L2 pop, packets will still be sent to patch port to tunnel bridge 3. "dest mac" not in this host, vlan or tunnel (gre/vxlan/geneve) unicast will be sent to corresponding patch port of tunnel/physical bridge. 4. other traffic still match the original NORMAL flow A new table=61 will be used to do accept egress traffic classification when not enable openflow firewall: 1. egress packets will be send to table 61, match rule will be of-port which be handled by ovs-agent "in_port=" 2. the "dest mac" is handled this ovs-agent, direct "output" to that port 3. "ARP request" with enabled L2 pop, packets will still be sent to patch port to tunnel bridge 4. "dest mac" not in this host, vlan or tunnel (gre/vxlan/geneve) unicast will be sent to corresponding patch port of tunnel/physical bridge. 5. other traffic still match the original NORMAL flow ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1841622 Title: [L2][OVS] add accepted egress fdb flows Status in neutron: New Bug description: Bug https://bugs.launchpad.net/neutron/+bug/1732067 has a bad impact on VM traffic. And all the fix has some potenial risk of data-plane down. So we added a new bug for the new solution: It will add a flow table something like a switch FDB table. The accepted egress flows will be take care in that. table=94 will be used to do accept egress traffic classification when enable openflow firewall: 1. the "dest mac" is handled this ovs-agent, direct "output" to that port 2. "ARP request" with enabled L2 pop, packets will still be sent to patch port to tunnel bridge 3. "dest mac" not in this host, vlan or tunnel (gre/vxlan/geneve) unicast will be sent to corresponding patch port of tunnel/physical bridge. 4. other traffic still match the original NORMAL flow A new table=61 will be used to do accept egress traffic classification when not enable openflow firewall: 1. egress packets will be send to table 61, match rule will be of-port which be handled by ovs-agent "in_port=" 2. the "dest mac" is handled this ovs-agent, direct "output" to that port 3. "ARP request" with enabled L2 pop, packets will still be sent to patch port to tunnel bridge 4. "dest mac" not in this host, vlan or tunnel (gre/vxlan/geneve) unicast will be sent to corresponding patch port of tunnel/physical bridge. 5. other traffic still match the original NORMAL flow To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1841622/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840979] [NEW] [L2] [opinion] update the port DB status directly in agent-side
Public bug reported: When ovs-agent done processing the port, it will call neutron-server to make some DB update. Especially when restart the ovs-agent, all ports in one agent will do such RPC and DB update again to make port status consistent. When a large number of concurrent agent restart happen, neutron-server may not work fine. So how about making the following DB updating locally in neutron agent side directly? It may have some mechanism driver notification, IMO, this can also be done in agent-side. def update_device_down(self, context, device, agent_id, host=None): cctxt = self.client.prepare() return cctxt.call(context, 'update_device_down', device=device, agent_id=agent_id, host=host) def update_device_up(self, context, device, agent_id, host=None): cctxt = self.client.prepare() return cctxt.call(context, 'update_device_up', device=device, agent_id=agent_id, host=host) def update_device_list(self, context, devices_up, devices_down, ret = cctxt.call(context, 'update_device_list', ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1840979 Title: [L2] [opinion] update the port DB status directly in agent-side Status in neutron: New Bug description: When ovs-agent done processing the port, it will call neutron-server to make some DB update. Especially when restart the ovs-agent, all ports in one agent will do such RPC and DB update again to make port status consistent. When a large number of concurrent agent restart happen, neutron-server may not work fine. So how about making the following DB updating locally in neutron agent side directly? It may have some mechanism driver notification, IMO, this can also be done in agent-side. def update_device_down(self, context, device, agent_id, host=None): cctxt = self.client.prepare() return cctxt.call(context, 'update_device_down', device=device, agent_id=agent_id, host=host) def update_device_up(self, context, device, agent_id, host=None): cctxt = self.client.prepare() return cctxt.call(context, 'update_device_up', device=device, agent_id=agent_id, host=host) def update_device_list(self, context, devices_up, devices_down, ret = cctxt.call(context, 'update_device_list', To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1840979/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840737] Re: Neutron create IPv6 subnet error. Gateway is not valid on subnet.
A really old neutron version.
But anyway, IMO, you met this bug:
https://bugs.launchpad.net/neutron/+bug/1682094
This is the error comes from:
https://github.com/openstack/neutron/blob/master/neutron/db/db_base_plugin_v2.py#L611-L616
According to the exception, I did the following tests:
>>> ip = netaddr.IPAddress("2a0a:4180::2")
>>> net = netaddr.IPNetwork("2a0a:4180::2/127")
>>> (ip in net and (ip == net.network or (net.version == 4 and ip == net[-1])))
True
>>> ip == net.network
True
>>> (ip in net and (net.version == 4 and ip in (net.network, net[-1])))
>>>< current check
False
>>> ip in net
True
So you should update this function to such style:
https://github.com/openstack/neutron/blob/master/neutron/ipam/utils.py#L40-L51
Since this verison 9.1.1 of neutron had been marked as EOL long time
ago, I will marked this bug to invalid.
** Changed in: neutron
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1840737
Title:
Neutron create IPv6 subnet error. Gateway is not valid on subnet.
Status in neutron:
Invalid
Bug description:
Hello!
I try create subnet ipv6, but i got error: Gateway is not valid on subnet.
neutron subnet-create --debug --disable-dhcp --tenant-id
b060ad4bdf256ab02 --ip-version 6 --gateway 2a0a:4180::2 --ipv6
-address-mode slaac --name test-ipv6-subnet test-ipv6-net
2a0a:4180::2/127
Output of this command:
DEBUG: stevedore.extension found extension EntryPoint.parse('v2token =
keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('none =
keystoneauth1.loading._plugins.noauth:NoAuth')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oauth1 =
keystoneauth1.extras.oauth1._loading:V3OAuth1')
DEBUG: stevedore.extension found extension EntryPoint.parse('admin_token =
keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcauthcode =
keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG: stevedore.extension found extension EntryPoint.parse('v2password =
keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3samlpassword =
keystoneauth1.extras._saml2._loading:Saml2Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3password =
keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3adfspassword =
keystoneauth1.extras._saml2._loading:ADFSPassword')
DEBUG: stevedore.extension found extension
EntryPoint.parse('v3oidcaccesstoken =
keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcpassword =
keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3kerberos =
keystoneauth1.extras.kerberos._loading:Kerberos')
DEBUG: stevedore.extension found extension EntryPoint.parse('token =
keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG: stevedore.extension found extension
EntryPoint.parse('v3oidcclientcredentials =
keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3tokenlessauth
= keystoneauth1.loading._plugins.identity.v3:TokenlessAuth')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3token =
keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3totp =
keystoneauth1.loading._plugins.identity.v3:TOTP')
DEBUG: stevedore.extension found extension EntryPoint.parse('password =
keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3fedkerb =
keystoneauth1.extras.kerberos._loading:MappedKerberos')
DEBUG: stevedore.extension found extension EntryPoint.parse('token_endpoint =
openstackclient.api.auth_plugin:TokenEndpoint')
DEBUG: stevedore.extension found extension EntryPoint.parse('table =
cliff.formatters.table:TableFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('json =
cliff.formatters.json_format:JSONFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('shell =
cliff.formatters.shell:ShellFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('value =
cliff.formatters.value:ValueFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('yaml =
cliff.formatters.yaml_format:YAMLFormatter')
DEBUG: neutronclient.neutron.v2_0.subnet.CreateSubnet
run(Namespace(allocation_pools=None, cidr=u'2a0a:4180::2/127', columns=[],
description=None, disable_dhcp=True, dns_nameservers=None, enable_dhcp=Fal
[Yahoo-eng-team] [Bug 1838431] [NEW] [scale issue] ovs-agent port processing time increases linearly and eventually timeouts
Public bug reported:
ENV: stable/queens
But master has basically same code, so the issue may also exist.
Config: L2 ovs-agent with enabled openflow based security group.
Recently I run one extreme test locally, booting 2700 instances for one single
tenant.
The instance will be booted in 2000 networks. But the entire tenant has only
one security group with only 5 rules. (This is the key point to the problem.)
The result is totally unacceptable. Almost 2000+ instances failed to
boot (ERROR), and almost every of them meets the "vif-plug-timeout"
exception.
How to reproduce:
1. create 2700 networks one by one "openstack network create"
2. create one IPv4 subnet and one IPv6 subnet for every network
3. create 2700 router (one single tenant can not create HA router more than
255, because of the VRID range) and connect to these subnets
4. boot instances
for i in {1..100}
do
for i in {1..27}
nova boot --nic net-name="test-network-xxx" ...
done
echo "CLI: boot 27 VMs"
sleep 30s
done
I have some clue of this issue, the linearly processing time increasing is
something like this:
(1) rpc_loop X
5 port added to the ovs-agent, they are processed and will be add to the
updated list due to the local notification
(2) rpc_loop X + 1
another 10 ports are added to the ovs-agent, and 10 updated-port to local
notification.
This loop the processing time is 5 ports update processing time, and 10 added
port processing.
(3) rpc_loop X + 2
another 20 are ports added to ovs-agent,
10 updated + 20 added port processing time
And the worse thing is, when the port number is getting larger, every port
under this one security group will be related. The openflow based security
group processing time is get longer and longer.
Until some instance ports meet the timeout of vif-plug. And the instance get
failed to boot.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1838431
Title:
[scale issue] ovs-agent port processing time increases linearly and
eventually timeouts
Status in neutron:
New
Bug description:
ENV: stable/queens
But master has basically same code, so the issue may also exist.
Config: L2 ovs-agent with enabled openflow based security group.
Recently I run one extreme test locally, booting 2700 instances for one
single tenant.
The instance will be booted in 2000 networks. But the entire tenant has only
one security group with only 5 rules. (This is the key point to the problem.)
The result is totally unacceptable. Almost 2000+ instances failed to
boot (ERROR), and almost every of them meets the "vif-plug-timeout"
exception.
How to reproduce:
1. create 2700 networks one by one "openstack network create"
2. create one IPv4 subnet and one IPv6 subnet for every network
3. create 2700 router (one single tenant can not create HA router more than
255, because of the VRID range) and connect to these subnets
4. boot instances
for i in {1..100}
do
for i in {1..27}
nova boot --nic net-name="test-network-xxx" ...
done
echo "CLI: boot 27 VMs"
sleep 30s
done
I have some clue of this issue, the linearly processing time increasing is
something like this:
(1) rpc_loop X
5 port added to the ovs-agent, they are processed and will be add to the
updated list due to the local notification
(2) rpc_loop X + 1
another 10 ports are added to the ovs-agent, and 10 updated-port to local
notification.
This loop the processing time is 5 ports update processing time, and 10 added
port processing.
(3) rpc_loop X + 2
another 20 are ports added to ovs-agent,
10 updated + 20 added port processing time
And the worse thing is, when the port number is getting larger, every port
under this one security group will be related. The openflow based security
group processing time is get longer and longer.
Until some instance ports meet the timeout of vif-plug. And the instance get
failed to boot.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1838431/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1835663] [NEW] Some L3 RPCs are time-consuming especially get_routers
Public bug reported: Examples: http://logs.openstack.org/11/669111/4/check/neutron-tempest-plugin-dvr-multinode-scenario/dc3af26/controller/logs/screen-q-l3.txt.gz#_Jul_07_04_18_11_791730 http://logs.openstack.org/11/669111/4/check/neutron-tempest-plugin-dvr-multinode-scenario/dc3af26/controller/logs/screen-q-l3.txt.gz#_Jul_07_04_18_45_507205 http://logs.openstack.org/11/669111/4/check/neutron-tempest-plugin-dvr-multinode-scenario/dc3af26/controller/logs/screen-q-l3.txt.gz#_Jul_07_04_20_04_023626 http://logs.openstack.org/11/669111/4/check/neutron-tempest-plugin-dvr-multinode-scenario/dc3af26/controller/logs/screen-q-l3.txt.gz#_Jul_07_04_20_34_698863 Some possible reasons: 1. neutron server is busy while it can not process this RPC 2. DB slow query 3. L3 agent thread (context) switching ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1835663 Title: Some L3 RPCs are time-consuming especially get_routers Status in neutron: New Bug description: Examples: http://logs.openstack.org/11/669111/4/check/neutron-tempest-plugin-dvr-multinode-scenario/dc3af26/controller/logs/screen-q-l3.txt.gz#_Jul_07_04_18_11_791730 http://logs.openstack.org/11/669111/4/check/neutron-tempest-plugin-dvr-multinode-scenario/dc3af26/controller/logs/screen-q-l3.txt.gz#_Jul_07_04_18_45_507205 http://logs.openstack.org/11/669111/4/check/neutron-tempest-plugin-dvr-multinode-scenario/dc3af26/controller/logs/screen-q-l3.txt.gz#_Jul_07_04_20_04_023626 http://logs.openstack.org/11/669111/4/check/neutron-tempest-plugin-dvr-multinode-scenario/dc3af26/controller/logs/screen-q-l3.txt.gz#_Jul_07_04_20_34_698863 Some possible reasons: 1. neutron server is busy while it can not process this RPC 2. DB slow query 3. L3 agent thread (context) switching To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1835663/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1830014] Re: [RFE] add API for neutron debug tool "probe"
** Changed in: neutron Assignee: (unassigned) => LIU Yulong (dragon889) ** Changed in: neutron Status: Opinion => New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1830014 Title: [RFE] add API for neutron debug tool "probe" Status in neutron: New Bug description: Recently, due to this bug: https://bugs.launchpad.net/neutron/+bug/1821912 We noticed that sometimes the guest OS is not fully UP, but test case is trying to login it. A simple idea is to ping it first, then try to login. So we hope to find a way for tempest to verify the neutron port link state. In high probability, the DB resource state is not reliable. We need an independent mechanism to check the VM network status. Because tempest is "blackbox" test, it can run in any host, we can not use the current resources under the existing mechanism, such as qdhcp-namepace or qrouter-namepace to do such check. Then this RFE is up. We have neutron-debug tool which include a "probe" resource in the agent side. https://docs.openstack.org/neutron/latest/cli/neutron-debug.html We could add some API to neutron, and let the proper agent to add such "probe" for us. In agent side, it will be a general agent extension, you can enable it to the ovs-agent, L3-agent or DHCP-agent. Once you have such "probe" resource in the agent side, then you can run any command in it. This will be useful for neutron CI to check the VM link state. So a basic workflow will be: 1. neutron tempest create router and connected to one subnet (network-1) 2. neutron tempest create one VM 3. neutron tempest create one floating IP and bind it to the VM-1 port 4. create a "probe" for network-1 via neutron API 5. ping the VM port until reachable in the "probe" namespace 6. ssh the VM by floating IP 7. do the next step One more thing, we now have set the "neutron-debug" tool as deprecated: https://bugs.launchpad.net/neutron/+bug/1583700 But we can remain that "probe" mechanism. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1830014/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1834308] [NEW] [DVR][DB] too many slow query during agent restart
Public bug reported: ENV: stable/queens In a large scale cloud deployment, when restart neutron agent, especially l3 agents, neutron server side will trigger too many slow DB query. And this will cause the agent restart time to be too long to operate. ** Affects: neutron Importance: Medium Status: Confirmed ** Changed in: neutron Importance: Undecided => Medium ** Changed in: neutron Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1834308 Title: [DVR][DB] too many slow query during agent restart Status in neutron: Confirmed Bug description: ENV: stable/queens In a large scale cloud deployment, when restart neutron agent, especially l3 agents, neutron server side will trigger too many slow DB query. And this will cause the agent restart time to be too long to operate. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1834308/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1832968] [NEW] neutron test fails 100% due to abandoned ubuntu image release
Public bug reported: Failure log: http://logs.openstack.org/17/665517/1/check/neutron-tempest-plugin-dvr-multinode-scenario/95359a1/controller/logs/devstacklog.txt.gz neutron tempest image link: http://cloud-images.ubuntu.com/releases/16.04/release-20180622/ Code: https://github.com/openstack/neutron-tempest-plugin/blob/master/.zuul.yaml#L395 https://github.com/openstack/neutron-tempest-plugin/blob/master/.zuul.yaml#L510 ** Affects: neutron Importance: Critical Assignee: LIU Yulong (dragon889) Status: In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1832968 Title: neutron test fails 100% due to abandoned ubuntu image release Status in neutron: In Progress Bug description: Failure log: http://logs.openstack.org/17/665517/1/check/neutron-tempest-plugin-dvr-multinode-scenario/95359a1/controller/logs/devstacklog.txt.gz neutron tempest image link: http://cloud-images.ubuntu.com/releases/16.04/release-20180622/ Code: https://github.com/openstack/neutron-tempest-plugin/blob/master/.zuul.yaml#L395 https://github.com/openstack/neutron-tempest-plugin/blob/master/.zuul.yaml#L510 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1832968/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1491317] Re: [RFE] Add TCP/UDP port forwarding extension to L3
** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1491317 Title: [RFE] Add TCP/UDP port forwarding extension to L3 Status in neutron: Fix Released Bug description: I have searched and found many past efforts to implement port forwarding in Neutron. I have found two incomplete blueprints [1], [2] and an abandoned patch [3]. There is even a project in Stackforge [4], [5] that claims to implement this, but the L3 parts in it seems older then current master. I have recently came across this requirement for various use cases, one of them is providing feature compliance with Docker port-mapping feature (for Kuryr), and saving floating IP's space. There has been many discussions in the past that require this feature, so i assume there is a demand to make this formal, just a small examples [6], [7], [8], [9] The idea in a nutshell is to support port forwarding (TCP/UDP ports) on the external router leg from the public network to internal ports, so user can use one Floating IP (the external gateway router interface IP) and reach different internal ports depending on the port numbers. This should happen on the network node (and can also be leveraged for security reasons). I think that the POC implementation in the Stackforge project shows that this needs to be implemented inside the L3 parts of the current reference implementation, it will be hard to maintain something like that in an external repository. (I also think that the API/DB extensions should be close to the current L3 reference implementation) I would like to renew the efforts on this feature and propose a spec for this to the next release. And of course if any of the people interested or any of the people that worked on this before want to join the effort, you are more then welcome to join and comment. [1] https://blueprints.launchpad.net/neutron/+spec/router-port-forwarding [2] https://blueprints.launchpad.net/neutron/+spec/fip-portforwarding [3] https://review.openstack.org/#/c/60512/ [4] https://github.com/stackforge/networking-portforwarding [5] https://review.openstack.org/#/q/port+forwarding,n,z [6] https://ask.openstack.org/en/question/75190/neutron-port-forwarding-qrouter-vms/ [7] http://www.gossamer-threads.com/lists/openstack/dev/34307 [8] http://openstack.10931.n7.nabble.com/Neutron-port-forwarding-for-router-td46639.html [9] http://openstack.10931.n7.nabble.com/Neutron-port-forwarding-from-gateway-to-internal-hosts-td32410.html Some more descriptions: https://review.openstack.org/#/c/224727/2/specs/mitaka/port_forwarding.rst https://review.openstack.org/#/q/status:abandoned+project:openstack/neutron+branch:master+topic:bp/router-port-forwarding To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1491317/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1832745] [NEW] _update_network_segmentation_id KeyError: 'provider:network_type'
Public bug reported:
ENV: master
Exception:
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager [None req-1c6689ba-8524-4f93-9387-d5bcce5101dd
None None] Error during notification for
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-1722583
Network, after_update: KeyError: 'provider:network_type'
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager Traceback (most recent call last):
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager File
"/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py", line 197,
in _notify_loop
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs)
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/agent/rpc.py",
line 259, in _legacy_notifier
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager getattr(self._legacy_interface,
method)(context, **payload)
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager File
"/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager result = f(*args, **kwargs)
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager File
"/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 572, in network_update
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager self._update_network_segmentation_id(network)
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager File
"/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 418, in _update_network_segmentation_id
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager if network[provider_net.NETWORK_TYPE] !=
n_const.TYPE_VLAN:
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager KeyError: 'provider:network_type'
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager
For tenant HA network, the provider network type may in such style:
{
"network": {
"updated_at": "2019-02-19T13:25:15Z",
"revision_number": 5,
"id": "ba973781-065b-4c69-9f07-a5c4f9a3b754",
"router:external": false,
"availability_zone_hints": [],
"availability_zones": ["nova"],
"segments": [{
"provider:network_type": "vxlan",
"provider:physical_network": null,
"provider:segmentation_id": 10041
},
{
"provider:network_type": "vxlan",
"provider:physical_network": null,
"provider:segmentation_id": 10066
}],
"ipv4_address_scope": null,
"shared": false,
"project_id": "",
"l2_adjacency": true,
"status": "ACTIVE",
"subnets": ["3e098a8e-9e1f-456b-a429-2ab46493e6b5"],
"description": "",
"tags": [],
"ipv6_address_scope": null,
"qos_policy_id": null,
"name": "HA network tenant ae05deb7a16c485986c3666b65f71c71",
"admin_state_up": true,
"tenant_id": "",
"created_at": "2018-12-19T14:53:20Z",
"mtu": 1450
}
}
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1832745
Title:
_update_network_segmentation_id KeyError: 'provider:network_type'
Status in neutron:
New
Bug description:
ENV: master
Exception:
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager [None req-1c6689ba-8524-4f93-9387-d5bcce5101dd
None None] Error during notification for
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-1722583
Network, after_update: KeyError: 'provider:network_type'
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager Traceback (most recent call last):
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR
neutron_lib.callbacks.manager File
"/usr/lib/python2.7/site-packages/neutron_lib/
[Yahoo-eng-team] [Bug 1832743] [NEW] delete_dvr_dst_mac_for_arp uses wrong table id
RROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent for table
in table_id:
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent TypeError:
'int' object is not iterable
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
** Affects: neutron
Importance: Low
Assignee: LIU Yulong (dragon889)
Status: In Progress
** Changed in: neutron
Importance: Undecided => Low
** Changed in: neutron
Assignee: (unassigned) => LIU Yulong (dragon889)
** Changed in: neutron
Status: New => Confirmed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1832743
Title:
delete_dvr_dst_mac_for_arp uses wrong table id
Status in neutron:
In Progress
Bug description:
ENV: master
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None
req-64e6eccc-0d85-4145-923e-de511adf9946 None None] Error while processing VIF
ports: TypeError: 'int' object is not iterable
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 2398, in rpc_loop
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info,
provisioning_needed)
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result =
f(*args, **kwargs)
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1965, in process_network_ports
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
port_info['removed'])
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result =
f(*args, **kwargs)
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1875, in treat_devices_removed
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.port_unbound(device)
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result =
f(*args, **kwargs)
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
line 1186, in port_unbound
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
self.dvr_agent.unbind_port_from_dvr(vif_port, lvm)
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result =
f(*args, **kwargs)
Jun 14 00:30:49 network2 neutron-openvswitch-agent[8378]: ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neut
[Yahoo-eng-team] [Bug 1831534] [NEW] [l3][dvr] with openflow security group east-west traffic between different vlan networks is broken
Public bug reported: ENV: stable/queens & master This is a long story long time ago [1] [2] [3]. But we recently meet such issue, when dvr router is connected to two different vlan networks, the east-west traffic is not reachable. # ovs-ofctl show br-int 1(int-br-ex): addr:22:32:17:d4:08:6a 2(int-br-vlan): addr:76:ed:47:bf:21:ec 3(patch-tun): addr:9a:56:bf:23:ac:37 ... ... 255(tap321a4669-c2): addr:fe:16:3e:93:31:67 LOCAL(br-int): addr:7a:ae:b6:87:7b:4d # ovs-ofctl dump-flows br-int # this will be applied aways, since it has higher priority, fa:16:3f:93:05:7d is the dvr host mac from request VM's hypervisor cookie=0xb27e128dd9a83dfc, duration=6408639.091s, table=0, n_packets=22187, n_bytes=30725358, idle_age=860, hard_age=65534, priority=4,in_port=2,dl_src=fa:16:3f:93:05:7d actions=resubmit(,2) # this will not get matched cookie=0xb27e128dd9a83dfc, duration=116506.106s, table=0, n_packets=60698, n_bytes=80563747, idle_age=825, hard_age=65534, priority=3,in_port=2,dl_vlan=587 actions=mod_vlan_vid:45,resubmit(,60) cookie=0xb27e128dd9a83dfc, duration=167233.168s, table=2, n_packets=22177, n_bytes=30724518, idle_age=51621, hard_age=65534, priority=4,dl_vlan=587,dl_dst=fa:16:3e:93:31:67 actions=mod_dl_src:fa:16:3e:ca:bf:28,resubmit(,60) cookie=0xb27e128dd9a83dfc, duration=167719.120s, table=60, n_packets=22257, n_bytes=30732678, idle_age=4, hard_age=65534, priority=4,dl_vlan=587,dl_dst=fa:16:3e:93:31:67 actions=strip_vlan,output:255 Since the request packet never go into conntrack table, so the reply packets will be dropped. [1] https://specs.openstack.org/openstack/neutron-specs/specs/kilo/neutron-ovs-dvr-vlan.html [2] https://blueprints.launchpad.net/neutron/+spec/neutron-ovs-dvr-vlan [3] https://review.opendev.org/#/q/topic:bp/neutron-ovs-dvr-vlan ** Affects: neutron Importance: High Status: Confirmed ** Changed in: neutron Importance: Undecided => Critical ** Changed in: neutron Importance: Critical => High ** Changed in: neutron Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1831534 Title: [l3][dvr] with openflow security group east-west traffic between different vlan networks is broken Status in neutron: Confirmed Bug description: ENV: stable/queens & master This is a long story long time ago [1] [2] [3]. But we recently meet such issue, when dvr router is connected to two different vlan networks, the east-west traffic is not reachable. # ovs-ofctl show br-int 1(int-br-ex): addr:22:32:17:d4:08:6a 2(int-br-vlan): addr:76:ed:47:bf:21:ec 3(patch-tun): addr:9a:56:bf:23:ac:37 ... ... 255(tap321a4669-c2): addr:fe:16:3e:93:31:67 LOCAL(br-int): addr:7a:ae:b6:87:7b:4d # ovs-ofctl dump-flows br-int # this will be applied aways, since it has higher priority, fa:16:3f:93:05:7d is the dvr host mac from request VM's hypervisor cookie=0xb27e128dd9a83dfc, duration=6408639.091s, table=0, n_packets=22187, n_bytes=30725358, idle_age=860, hard_age=65534, priority=4,in_port=2,dl_src=fa:16:3f:93:05:7d actions=resubmit(,2) # this will not get matched cookie=0xb27e128dd9a83dfc, duration=116506.106s, table=0, n_packets=60698, n_bytes=80563747, idle_age=825, hard_age=65534, priority=3,in_port=2,dl_vlan=587 actions=mod_vlan_vid:45,resubmit(,60) cookie=0xb27e128dd9a83dfc, duration=167233.168s, table=2, n_packets=22177, n_bytes=30724518, idle_age=51621, hard_age=65534, priority=4,dl_vlan=587,dl_dst=fa:16:3e:93:31:67 actions=mod_dl_src:fa:16:3e:ca:bf:28,resubmit(,60) cookie=0xb27e128dd9a83dfc, duration=167719.120s, table=60, n_packets=22257, n_bytes=30732678, idle_age=4, hard_age=65534, priority=4,dl_vlan=587,dl_dst=fa:16:3e:93:31:67 actions=strip_vlan,output:255 Since the request packet never go into conntrack table, so the reply packets will be dropped. [1] https://specs.openstack.org/openstack/neutron-specs/specs/kilo/neutron-ovs-dvr-vlan.html [2] https://blueprints.launchpad.net/neutron/+spec/neutron-ovs-dvr-vlan [3] https://review.opendev.org/#/q/topic:bp/neutron-ovs-dvr-vlan To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1831534/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
