[Yahoo-eng-team] [Bug 2022321] [NEW] Using Isolated metadata+ipv6 haproxy metadata isn't working becasue haproxy container isn't created in some controlers
Public bug reported: Keys and metadata info isn't loaded in the vms: Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 136, in _get_ssh_connection ssh.connect(self.host, port=self.port, username=self.username, File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 406, in connect t.start_client(timeout=timeout) File "/usr/lib/python3.9/site-packages/paramiko/transport.py", line 699, in start_client raise e File "/usr/lib/python3.9/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/usr/lib/python3.9/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/usr/lib/python3.9/site-packages/paramiko/packet.py", line 303, in read_all raise EOFError() EOFError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper return f(*func_args, **func_kwargs) File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 535, in test_hotplug_nic self._check_public_network_connectivity(should_connect=True) File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 212, in _check_public_network_connectivity self.check_vm_connectivity( File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 964, in check_vm_connectivity self.get_remote_client(ip_address, username, private_key, File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 733, in get_remote_client linux_client.validate_authentication() File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 31, in wrapper return function(self, *args, **kwargs) File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 123, in validate_authentication self.ssh_client.test_connection_auth() File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 245, in test_connection_auth connection = self._get_ssh_connection() File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 155, in _get_ssh_connection raise exceptions.SSHTimeout(host=self.host, tempest.lib.exceptions.SSHTimeout: Connection to the 10.0.0.190 via SSH timed out. User: cirros, Password: None The trigger of the problem is this patch: https://review.opendev.org/c/openstack/neutron/+/876566/13/neutron/agent/metadata/driver.py when Dad ipv6 error is detected haproxy isn't created due to the return in the line 269: .. 'namespace': ns_name, 'network': network_id, 'exception': str(exc)}) try: ip_lib.delete_ip_address(bind_address_v6, bind_interface, namespace=ns_name) except Exception as exc: # do not re-raise a delete failure, just log LOG.info('Address deletion failure: %s', str(exc)) return pm.enable() . The problem needs that Dad error was detected in the controller is reported as metadata source because in this case without haproxy in this controller the metadata is unreachbable: Dad error: 2023-05-31 14:27:40.140 79551 INFO neutron.agent.metadata.driver [req-a76cfcdd-887b-4c36-86d5-a5eb2b87615c - - - - -] DAD failed for address fe80::a9fe:a9fe on interface tapb07b4b7c-3b in namespace qdhcp- abd16487-68bb-4090-8ccb-b6ec8a77cc2c on network abd16487-68bb-4090-8ccb-b6ec8a77cc2c, deleting it. Exception: Failure waiting for address fe80::a9fe:a9fe to become ready: Duplicate address detected haproxy doesn't start: 2023-05-31 14:27:39.461 79551 DEBUG neutron.agent.linux.utils [req-a76cfcdd-887b-4c36-86d5-a5eb2b87615c - - - - -] Unable to access /var/lib/neutron/external/pids/abd16487-68bb-4090-8ccb-b6ec8a77cc2c.pid.haproxy; Error: [Errno 2] No such file or directory: '/var/lib/neutron/external/pids/abd16487-68bb-4090-8ccb-b6ec8a77cc2c.pid.haproxy' get_value_from_file /usr/lib/python3.9/site-packages/neutron/agent/linux/utils.py:252 2023-05-31 14:27:39.462 79551 DEBUG neutron.agent.linux.utils [req-a76cfcdd-887b-4c36-86d5-a5eb2b87615c - - - - -] Unable to access /var/lib/neutron/external/pids/abd16487-68bb-4090-8ccb-b6ec8a77cc2c.pid.haproxy; Error: [Errno 2] No such file or directory: '/var/lib/neutron/external/pids/abd16487-68bb-4090-8ccb-b6ec8a77cc2c.pid.haproxy' get_value_from_file /usr/lib/python3.9/site-packages/neutron/agent/linux/utils.py:252 2023-05-31 14:27:39.463 79551 DEBUG neutron.agent.linux.external_process [req-a76cfcdd-887b-4c36-86d5-a5eb2b87615c - - - - -] No haproxy process started for
[Yahoo-eng-team] [Bug 2018585] [NEW] [SRBAC]New policies change the behavior for check rule type
Public bug reported: Example commandd affected: openstack network qos rule type list Several qos test case are skipped due to this chanmge beahavior because: (Pdb) p cls.os_tempest.network_client │ *** AttributeError: 'Manager' object has no attribute 'network_client' │ (Pdb) ll │ 858 -> @classmethod │ 859 def get_supported_qos_rule_types(cls): │ 860 body = cls.client.list_qos_rule_types() │ 861 return [rule_type['type'] for rule_type in body['rule_types']] │ (Pdb) cls.client.list_qos_rule_types() │ {'rule_types': []} │ (Pdb) old behavior rule Any: policy.DocumentedRuleDefault( name='get_rule_type', check_str=base.ADMIN, scope_types=['project'], description='Get available QoS rule types', operations=[ { 'method': 'GET', 'path': '/qos/rule-types', }, { 'method': 'GET', 'path': '/qos/rule-types/{rule_type}', }, ], deprecated_rule=policy.DeprecatedRule( name='get_rule_type', check_str=neutron_policy.RULE_ANY, deprecated_reason=DEPRECATED_REASON, deprecated_since=versionutils.deprecated.WALLABY) ), New : https://github.com/openstack/neutron/commit/f1541f29152a75df4efc5b5d53f426a362286ff6#diff-d0398e566a536eb5f27118bf5[…]621369660a13c502b8ae934b043R99 initially it was done correctly https://github.com/openstack/neutron/commit/c4618857b0249535eeed28f0c7a0abf5dbdbc9d0#diff-d0398e566a536eb5f27118bf5[…]9e8621369660a13c502b8ae934b043 later it was done for SYSTEM_READER but then we dropped system scope it should be ROLE:READER I guess to match old behaviour ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2018585 Title: [SRBAC]New policies change the behavior for check rule type Status in neutron: New Bug description: Example commandd affected: openstack network qos rule type list Several qos test case are skipped due to this chanmge beahavior because: (Pdb) p cls.os_tempest.network_client │ *** AttributeError: 'Manager' object has no attribute 'network_client' │ (Pdb) ll │ 858 -> @classmethod │ 859 def get_supported_qos_rule_types(cls): │ 860 body = cls.client.list_qos_rule_types() │ 861 return [rule_type['type'] for rule_type in body['rule_types']] │ (Pdb) cls.client.list_qos_rule_types() │ {'rule_types': []} │ (Pdb) old behavior rule Any: policy.DocumentedRuleDefault( name='get_rule_type', check_str=base.ADMIN, scope_types=['project'], description='Get available QoS rule types',
[Yahoo-eng-team] [Bug 1926219] [NEW] neutron command l3-agent-list-hosting-router and dhcp-agent-list-hosting-net don't exist in openstack client
Public bug reported: these two useful neutron cli command aren't in openstack client: (overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router tempest-router-2120132397 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--+---++---+--+ | id | host | admin_state_up | alive | ha_state | +--+---++---+--+ | 1ee45cf7-1aa6-47fc-aa21-4713dd3ae119 | networker-2.redhat.local | True | :-) | standby | | 2baa2189-8d27-4477-89b8-31afb4d7b1ba | networker-0.redhat.local | True | :-) | active | | 39ae6b11-a49a-4d85-9a8f-919416f4931e | controller-1.redhat.local | True | :-) | standby | +--+---++---+--+ (overcloud) [stack@undercloud-0 ~]$ neutron dhcp-agent-list-hosting-net nova neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--+---++---+ | id | host | admin_state_up | alive | +--+---++---+ | 71befd34-b569-4d00-b6b5-d34266e5b33e | controller-0.redhat.local | True | :-) | | 77f6e7ee-1d37-4ab6-a805-455b2dbbf341 | networker-1.redhat.local | True | :-) | | 81638b13-95d0-43c1-be23-eb8422c00ce0 | controller-2.redhat.local | True | :-) | | 95bf9893-79b3-4f56-a6f2-653977e1700d | networker-2.redhat.local | True | :-) | | be3a478d-8ea7-4eca-9430-72b6e1a826d4 | controller-1.redhat.local | True | :-) | | dfd410b2-1fc0-4eb3-a383-583e37e810fb | networker-0.redhat.local | True | :-) | +--+---++---+ (overcloud) [stack@undercloud-0 ~]$ ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1926219 Title: neutron command l3-agent-list-hosting-router and dhcp-agent-list- hosting-net don't exist in openstack client Status in neutron: New Bug description: these two useful neutron cli command aren't in openstack client: (overcloud) [stack@undercloud-0 ~]$ neutron l3-agent-list-hosting-router tempest-router-2120132397 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--+---++---+--+ | id | host | admin_state_up | alive | ha_state | +--+---++---+--+ | 1ee45cf7-1aa6-47fc-aa21-4713dd3ae119 | networker-2.redhat.local | True | :-) | standby | | 2baa2189-8d27-4477-89b8-31afb4d7b1ba | networker-0.redhat.local | True | :-) | active | | 39ae6b11-a49a-4d85-9a8f-919416f4931e | controller-1.redhat.local | True | :-) | standby | +--+---++---+--+ (overcloud) [stack@undercloud-0 ~]$ neutron dhcp-agent-list-hosting-net nova neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--+---++---+ | id | host | admin_state_up | alive | +--+---++---+ | 71befd34-b569-4d00-b6b5-d34266e5b33e | controller-0.redhat.local | True | :-) | | 77f6e7ee-1d37-4ab6-a805-455b2dbbf341 | networker-1.redhat.local | True | :-) | | 81638b13-95d0-43c1-be23-eb8422c00ce0 | controller-2.redhat.local | True | :-) | | 95bf9893-79b3-4f56-a6f2-653977e1700d | networker-2.redhat.local | True | :-) | | be3a478d-8ea7-4eca-9430-72b6e1a826d4 | controller-1.redhat.local | True | :-) | | dfd410b2-1fc0-4eb3-a383-583e37e810fb | networker-0.redhat.local | True | :-) | +--+---++---+ (overcloud) [stack@undercloud-0 ~]$ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1926219/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe :
[Yahoo-eng-team] [Bug 1835044] [NEW] [Rocky]Memory leak in privsep-helper(neutron agents)
Public bug reported: Description of problem: Memory leak in privsep-helper(neutron agents) when create/destroy action are executed. Version-Release number of selected component (if applicable): How reproducible: Rocky and Queens are affected Steps to Reproduce: 1. Deploy Osp 14 2.create the next scripts: (overcloud) [stack@undercloud-0 ~]$ cat create2.sh set -x ips=(0 10.0.0.215 10.0.0.249 10.0.0.223 10.0.0.222 10.0.0.218 10.0.0.247 10.0.0.210 10.0.0.220 10.0.0.246 10.0.0.213 10.0.0.224 10.0.0.212 10.0.0.217 10.0.0.221 10.0.0.216) ips=(0 10.0.0.220 10.0.0.216 10.0.0.235 10.0.0.232 10.0.0.245 10.0.0.226 10.0.0.217 10.0.0.211 10.0.0.221 10.0.0.230 10.0.0.248 10.0.0.228 10.0.0.223 10.0.0.212 10.0.0.225) openstack network create net_$1 openstack subnet create --network net_$1 --dns-nameserver 10.0.0.1 --gateway 10.$1.0.1 --subnet-range 10.$1.0.0/16 net_$1 openstack router create router_$1 openstack router add subnet router_$1 net_$1 #openstack router set router_$1 --external-gateway public #openstack server create --flavor cirros --image cirros --nic net- id=net$1 --security-group test --key-name mykey vm$1 #openstack server add floating ip vm$1 ${ips[$1]} #ping ${ips[$1]} -c 10 (overcloud) [stack@undercloud-0 ~]$ cat delete2.sh #openstack server delete vm$1 openstack router remove subnet router_$1 net_$1 openstack network delete net_$1 openstack router delete router_$1 3. Execute: for j in $(seq 1 10); do for i in $(seq 1 10) ; do ./create2.sh $i ; done ; for i in $(seq 1 10) ; do ./delete2.sh $i ; done ;echo "LOOP $j " ;done Chech the memory usage of the privsep-helper in the controllers: [root@controller-1 heat-admin]# top -c -p 125052 -p 278912 top - 22:37:22 up 9 days, 4:59, 1 user, load average: 4.45, 3.71, 3.77 Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie %Cpu(s): 19.8 us, 4.1 sy, 0.0 ni, 75.2 id, 0.0 wa, 0.0 hi, 0.8 si, 0.0 st KiB Mem : 32779936 total, 3150768 free, 16209200 used, 13419968 buff/cache KiB Swap:0 total,0 free,0 used. 15146444 avail Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 125052 root 20 0 2497576 2.2g 2468 S 0.0 7.2 76:06.44 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-fi+ 278912 root 20 0 1048908 897004 2468 S 0.0 2.7 13:55.34 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-fi+ The important colunm is RES Actual results: Memory usage increases in every loop. Expected results: Memory usage should be stable Additional info: Leak is in pyroute2--> necessary update version to 0.5.2.x More info : https://bugzilla.redhat.com/show_bug.cgi?id=1723597 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1835044 Title: [Rocky]Memory leak in privsep-helper(neutron agents) Status in neutron: New Bug description: Description of problem: Memory leak in privsep-helper(neutron agents) when create/destroy action are executed. Version-Release number of selected component (if applicable): How reproducible: Rocky and Queens are affected Steps to Reproduce: 1. Deploy Osp 14 2.create the next scripts: (overcloud) [stack@undercloud-0 ~]$ cat create2.sh set -x ips=(0 10.0.0.215 10.0.0.249 10.0.0.223 10.0.0.222 10.0.0.218 10.0.0.247 10.0.0.210 10.0.0.220 10.0.0.246 10.0.0.213 10.0.0.224 10.0.0.212 10.0.0.217 10.0.0.221 10.0.0.216) ips=(0 10.0.0.220 10.0.0.216 10.0.0.235 10.0.0.232 10.0.0.245 10.0.0.226 10.0.0.217 10.0.0.211 10.0.0.221 10.0.0.230 10.0.0.248 10.0.0.228 10.0.0.223 10.0.0.212 10.0.0.225) openstack network create net_$1 openstack subnet create --network net_$1 --dns-nameserver 10.0.0.1 --gateway 10.$1.0.1 --subnet-range 10.$1.0.0/16 net_$1 openstack router create router_$1 openstack router add subnet router_$1 net_$1 #openstack router set router_$1 --external-gateway public #openstack server create --flavor cirros --image cirros --nic net- id=net$1 --security-group test --key-name mykey vm$1 #openstack server add floating ip vm$1 ${ips[$1]} #ping ${ips[$1]} -c 10 (overcloud) [stack@undercloud-0 ~]$ cat delete2.sh #openstack server delete vm$1 openstack router remove subnet router_$1 net_$1 openstack network delete net_$1 openstack router delete router_$1 3. Execute: for j in $(seq 1 10); do for i in $(seq 1 10) ; do ./create2.sh $i ; done ; for i in $(seq 1 10) ; do ./delete2.sh $i ; done ;echo "LOOP $j " ;done
[Yahoo-eng-team] [Bug 1821311] [NEW] openstack router remove/add command out without error, when it fails
Public bug reported: the command fails but the failure is not shown if you don't use --debug option: (overcloud) [stack@undercloud-0 ~]$ openstack router remove subnet router selfservice ; echo $? 0 (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ openstack router add subnet router selfservice ; echo $? 0 (overcloud) [stack@undercloud-0 ~]$ openstack router remove subnet router selfservice --debug START with options: [u'router', u'remove', u'subnet', u'router', u'selfservice', u'--debug'] ... RESP: [409] Content-Length: 268 Content-Type: application/json Date: Fri, 22 Mar 2019 09:29:10 GMT X-Openstack-Request-Id: req-e46e3e8b-76d3-4535-8cad-e14eed2c9190 RESP BODY: {"NeutronError": {"message": "Router interface for subnet ca7de33b-98c7-4ff4-9fae-cc2fcb7c41cc on router daa62d34-037d-4188-a37c-ab5d058d5489 cannot be deleted, as it is required by one or more floating IPs.", "type": "RouterInterfaceInUseByFloatingIP", "detail": ""}} PUT call to network for http://10.0.0.107:9696/v2.0/routers/daa62d34-037d-4188-a37c-ab5d058d5489/remove_router_interface used request id req-e46e3e8b-76d3-4535-8cad-e14eed2c9190 Manager unknown ran task network.PUT.routers.remove_router_interface in 1.10061788559s clean_up RemoveSubnetFromRouter: END return value: 0 (overcloud) [stack@undercloud-0 ~]$ Reproduction example: -create a router: (overcloud) [stack@undercloud-0 ~]$ history | grep router 18 openstack router create router 19 openstack router add subnet router selfservice 20 openstack router set router --external-gateway public -associate a floating ip to a vm: 56 openstack server add floating ip provider-instance 10.0.0.216 -try to add/remove the subnet 61 openstack router remove subnet router selfservice --debug 62 openstack router add subnet router selfservice --debug Logs and versio: (overcloud) [stack@undercloud-0 ~]$ yum info openstack-neutron Loaded plugins: search-disabled-repos Available Packages Name: openstack-neutron Arch: noarch Epoch : 1 Version : 13.0.3 Release : 0.20190119134915.886782c.el7ost Size: 28 k Repo: rhelosp-14.0-puddle/x86_64 Summary : OpenStack Networking Service URL : http://launchpad.net/neutron/ License : ASL 2.0 Description : : Neutron is a virtual network service for Openstack. Just like : OpenStack Nova provides an API to dynamically request and configure : virtual servers, Neutron provides an API to dynamically request and : configure virtual networks. These networks connect "interfaces" from : other OpenStack services (e.g., virtual NICs from Nova VMs). The : Neutron API supports extensions to provide advanced network : capabilities (e.g., QoS, ACLs, network monitoring, etc.) (overcloud) [stack@undercloud-0 ~]$ cat /etc/rhosp-release Red Hat OpenStack Platform release 14.0.1 RC (Rocky) (overcloud) [stack@undercloud-0 ~]$ openstack server add floating ip provider-instance 10.0.0.216 (overcloud) [stack@undercloud-0 ~]$ openstack router remove subnet router selfservice (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ openstack router show router +-+---+ | Field | Value | +-+---+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova
[Yahoo-eng-team] [Bug 1821203] [NEW] DVR: When a subnet is removed from a router the qrouter namespaces aren't removed from the compute nodes
Public bug reported: [stack@undercloud-0 ~]$ cat /etc/rhosp-release Red Hat OpenStack Platform release 14.0.1 RC (Rocky) [stack@undercloud-0 ~]$ yum info openstack-neutron Loaded plugins: search-disabled-repos Available Packages Name : openstack-neutron Arch : noarch Epoch : 1 Version : 13.0.3 Release : 0.20190119134915.886782c.el7ost Size : 28 k Repo : rhelosp-14.0-puddle/x86_64 Summary : OpenStack Networking Service URL : http://launchpad.net/neutron/ License : ASL 2.0 Description : : Neutron is a virtual network service for Openstack. Just like : OpenStack Nova provides an API to dynamically request and configure : virtual servers, Neutron provides an API to dynamically request and : configure virtual networks. These networks connect "interfaces" from : other OpenStack services (e.g., virtual NICs from Nova VMs). The : Neutron API supports extensions to provide advanced network : capabilities (e.g., QoS, ACLs, network monitoring, etc.) [stack@undercloud-0 ~]$ yum list | grep neutron puppet-neutron.noarch 13.3.1-0.20181013115834.el7ost python2-neutron-lib.noarch 1.18.0-0.20180816094046.67865c7.el7ost python2-neutronclient.noarch 6.9.1-0.20180925041810.7eba94e.el7ost openstack-neutron.noarch 1:13.0.3-0.20190119134915.886782c.el7ost openstack-neutron-bigswitch-agent.noarch openstack-neutron-bigswitch-lldp.noarch openstack-neutron-common.noarch 1:13.0.3-0.20190119134915.886782c.el7ost openstack-neutron-fwaas.noarch 1:13.0.2-0.20190123183836.90951a5.el7ost openstack-neutron-l2gw-agent.noarch openstack-neutron-lbaas.noarch 1:13.0.1-0.20181017150329.1353bad.el7ost openstack-neutron-lbaas-ui.noarch openstack-neutron-linuxbridge.noarch openstack-neutron-macvtap-agent.noarch openstack-neutron-metering-agent.noarch openstack-neutron-ml2.noarch 1:13.0.3-0.20190119134915.886782c.el7ost openstack-neutron-openvswitch.noarch openstack-neutron-rpc-server.noarch openstack-neutron-sriov-nic-agent.noarch python-neutron.noarch 1:13.0.3-0.20190119134915.886782c.el7ost python-neutron-fwaas.noarch 1:13.0.2-0.20190123183836.90951a5.el7ost python-neutron-fwaas-tests.noarch python-neutron-lbaas.noarch 1:13.0.1-0.20181017150329.1353bad.el7ost python-neutron-lbaas-tests.noarch python2-ironic-neutron-agent.noarch python2-neutron-lib-tests.noarch 1.18.0-0.20180816094046.67865c7.el7ost python2-neutron-tests-tempest.noarch Subnets are deleted from the routers: 51 openstack router remove subnet router 8646bed0-7dfd-43a3-bdb5-ab7368cbbbdb 54 openstack router remove subnet router2 dd8f26ec-b98a-4fe3-8d36-ee54b117dbca (overcloud) [stack@undercloud-0 ~]$ openstack router show router +-++ | Field | Value | +-++ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2019-03-20T09:15:21Z | | description | | | distributed | True | | external_gateway_info | {"network_id": "4eecbefb-7d7a-4210-836e-3b2b3de215db", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "6c9c958d-caf3-4154-b266-6df78c9bd4df", "ip_address": "10.0.0.223"}]} | | flavor_id | None
[Yahoo-eng-team] [Bug 1820321] [NEW] create a router distributed comand over a cluster without dvr fails
Public bug reported: Comand fails: (overcloud) [stack@undercloud-0 ~]$ openstack router create roter3 --distributed --debug START with options: [u'router', u'create', u'roter3', u'--distributed', u'--debug'] options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_type='password', auth_url='http://10.0.0.102:5000//v3', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.37', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='2', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='3', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None) Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.102:5000//v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.37', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '3', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://10.0.0.102:5000//v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True,
[Yahoo-eng-team] [Bug 1818824] [NEW] When a fip is added to a vm with dvr, previous connections loss the connectivity
Public bug reported: The behavior wihout DVR is that the previous connections continue using the snat ip and the new connections go to use the fip. then without dvr: -1 add fip -> no delete conntrack flows -2 delete fip -> delete conntrack flows I don know if 1 is a expected behavior or a bug. Delete the conntrack entries in the 1 and 2 would be a simpler solution(less casuistic). then first we should be sure of the desired behavior when a fip is added, becuase it is not working with DVR. If the decission isn't maintain the old connections then: -with and without DVR the conntrack entries shoud be deleted. If the decission is maintais the old connection then: -the fix only would be necessary for DVR and it consists in create a way for the external traffic without nat(in the qrouter of the compute, the nat is done in the controller qrouter )(*). Related bug: https://bugs.launchpad.net/neutron/+bug/1818805 "Conntrack rules in the qrouter are not deleted when a fip is removed with dvr" (*) With dvr the external traffic is managed using pbr(in the qrouter): without fip: [root@compute-2 heat-admin]# ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: rfp-d01c89b0-c: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 06:09:90:b6:24:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 169.254.106.114/31 scope global rfp-d01c89b0-c valid_lft forever preferred_lft forever inet6 fe80::409:90ff:feb6:2447/64 scope link valid_lft forever preferred_lft forever 43: qr-c47b0417-7d: mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:d9:7d:01 brd ff:ff:ff:ff:ff:ff inet 10.2.0.1/24 brd 10.2.0.255 scope global qr-c47b0417-7d valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fed9:7d01/64 scope link valid_lft forever preferred_lft forever [root@compute-2 heat-admin]# ip r 10.2.0.0/24 dev qr-c47b0417-7d proto kernel scope link src 10.2.0.1 169.254.106.114/31 dev rfp-d01c89b0-c proto kernel scope link src 169.254.106.114 [root@compute-2 heat-admin]# ip rule list 0: from all lookup local 32766: from all lookup main 32767: from all lookup default 167903233: from 10.2.0.1/24 lookup 167903233 [root@compute-2 heat-admin]# ip r show table 167903233 default via 10.2.0.8 dev qr-c47b0417-7d [root@compute-2 heat-admin]# ip route get 8.8.8.8 from 10.2.0.12 iif qr-c47b0417-7d 8.8.8.8 from 10.2.0.12 via 10.2.0.8 dev qr-c47b0417-7d cache iif * The traffic is sent to the qrouter in the controller With fip: [root@compute-1 heat-admin]# ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: rfp-d01c89b0-c@if3: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 06:5e:ac:51:83:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 169.254.106.114/31 scope global rfp-d01c89b0-c valid_lft forever preferred_lft forever inet6 fe80::45e:acff:fe51:837a/64 scope link valid_lft forever preferred_lft forever 23: qr-c47b0417-7d: mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:d9:7d:01 brd ff:ff:ff:ff:ff:ff inet 10.2.0.1/24 brd 10.2.0.255 scope global qr-c47b0417-7d valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fed9:7d01/64 scope link valid_lft forever preferred_lft forever [root@compute-1 heat-admin]# ip r 10.2.0.0/24 dev qr-c47b0417-7d proto kernel scope link src 10.2.0.1 169.254.106.114/31 dev rfp-d01c89b0-c proto kernel scope link src 169.254.106.114 [root@compute-1 heat-admin]# ip rule list 0: from all lookup local 32766: from all lookup main 32767: from all lookup default 57483: from 10.2.0.12 lookup 16 167903233: from 10.2.0.1/24 lookup 167903233 [root@compute-1 heat-admin]# ip r show table 16 default via 169.254.106.115 dev rfp-d01c89b0-c [root@compute-1 heat-admin]# [root@compute-1 heat-admin]# ip route get 8.8.8.8 from 10.2.0.12 iif qr-c47b0417-7d 8.8.8.8 from 10.2.0.12 via 169.254.106.115 dev rfp-d01c89b0-c cache iif * The traffic is sent to the fip namestpace to out directly without pass for the controller. The problem is that the traffic of old connections with contrack entries is sent to the fip name space without snat, but this traffic should be sent to the controller in the same way than in the scenario without fip. traffic before adding fip: [root@compute-1 heat-admin]# [root@compute-1 heat-admin]# tcpdump -nei rfp-d01c89b0-c tcpdump: verbose output
[Yahoo-eng-team] [Bug 1818805] [NEW] Conntrack rules in the qrouter are not deleted when a fip is removed with dvr
Public bug reported: If a fip ip is removed of a network with a distributed router: openstack server remove floating ip X The conntrack rules aren't deleted in the qrouter and the qrouter continues doing nating of the ongoing connections. overcloud) [stack@undercloud-0 ~]$ openstack router show router +-+--+ | Field | Value | +-+--+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2019-02-20T15:46:53Z | | description | | | distributed | True | | external_gateway_info | {"network_id": "15a5c01e-4e42-4890-a850-db4f97bb5834", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "c59ae813-1df7-4a14-9eba-be2e35afa13e", "ip_address": "10.0.0.214"}]} | | flavor_id | None | | ha | False | | id | d01c89b0-c2df-46e2-9c12-8d14b1c8ce9a | | interfaces_info | [{"subnet_id": "5e8ddfa7-d546-4f59-94d1-e2b65e8ecdb6", "ip_address": "10.2.0.8", "port_id": "06c6e9d3-2c6b-40b8-8919-92be6efd0153"}, {"subnet_id": "5e8ddfa7-d546-4f59-94d1-e2b65e8ecdb6", "ip_address": "10.2.0.1", "port_id": "c47b0417-7dbe-4434-8c50-72a78e6335a1"}] | | name| router | | project_id | 9447276fedbf4c4eab15494f8d187d97
[Yahoo-eng-team] [Bug 1804842] [NEW] When kill(sometines doesn't restart) the ovs switch or restart it in the compute nodes vm conectivity is lost
Public bug reported: OSP 14 3 controllers + 3 computes + dvr several vms in one compute with fip. Problem 1 : root@compute-2 heat-admin]# systemctl restart openvswitch fip conectivity with undercloud vm is lost and no recover conectivity with other computes is lost, but it is recovered restarting neutron openvswitch agent container root@compute-2 heat-admin]# systemctl restart openvswitch. Problem 2: After kill -9 "pid ovs switch" Sometimes the ovs switch in not restarted automatically Same problems that in the scenario1 [root@compute-2 heat-admin]# ps -ef | grep ovs root 105587292 0 12:09 ?00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock 42435 46886 46871 0 13:17 ?00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh openvsw+ 49054 1 1 13:20 ?00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach root 49217 17666 0 13:21 pts/000:00:00 grep --color=auto ovs [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# ps -ef | grep ovs root 105587292 0 12:09 ?00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock 42435 46886 46871 0 13:17 ?00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh openvsw+ 49054 1 0 13:20 ?00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach root 49421 17666 0 13:21 pts/000:00:00 grep --color=auto ovs [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# ps -ef | grep ovs root 105587292 0 12:09 ?00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock 42435 46886 46871 0 13:17 ?00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh openvsw+ 49054 1 0 13:20 ?00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach root 49423 17666 0 13:21 pts/000:00:00 grep --color=auto ovs [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# kill -9 49054 [root@compute-2 heat-admin]# ps -ef | grep ovs root 105587292 0 12:09 ?00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock 42435 46886 46871 0 13:17 ?00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh root 49610 17666 0 13:22 pts/000:00:00 grep --color=auto ovs [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# ps -ef | grep ovs root 105587292 0 12:09 ?00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock 42435 46886 46871 0 13:17 ?00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh root 49628 17666 0 13:22 pts/000:00:00 grep --color=auto ovs [root@compute-2 heat-admin]# date Fri Nov 23 13:22:22 UTC 2018 [root@compute-2 heat-admin]# ps -ef | grep ovs root 105587292 0 12:09 ?00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpATdioG/privsep.sock 42435 46886 46871 0 13:17 ?00:00:00 /bin/bash /neutron_ovs_agent_launcher.sh root 49788 17666 0 13:22 pts/000:00:00 grep --color=auto ovs [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# [root@compute-2 heat-admin]# ps -ef | grep ovs root 105587292 0 12:09 ?00:00:01 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file
[Yahoo-eng-team] [Bug 1795432] Re: neutron does not create the necessary iptables rules for dhcp agents when linuxbridge is used
After checking it with Rodolfo help, I can check that it is fixed in the lass queens(13) version. The fix that solve this issue is: https://review.openstack.org/#/c/568907/1 ** Changed in: neutron Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1795432 Title: neutron does not create the necessary iptables rules for dhcp agents when linuxbridge is used Status in neutron: Fix Released Bug description: Reproduction: Create a enviroment with controller and compute in different hosts: controller: [root@controller1 ~]# brctl show bridge name bridge id STP enabled interfaces brq37841a31-d78000.0a7e069299a3 no tap80087b5b-33 tap94526e09-2c vxlan-46 brqbab8fb94-c88000.1275449f51ef no eth3 tap4baecbed-83 tap8924b588-55 [root@controller1 ~]# ip netns qrouter-bcb8c407-ab4c-4916-89a5-d1ba8ac786ae (id: 2) qdhcp-37841a31-d744-4c9f-b084-37cfaafe71ca (id: 1) qdhcp-bab8fb94-c849-4c6c-ada7-98ec9bc33b87 (id: 0) Compute host: [root@compute1 ~]# brctl show bridge name bridge id STP enabled interfaces brq37841a31-d78000.5e530dd5073b no tap171ccdb9-66 vxlan-46 brqbab8fb94-c88000.525400fec4c7 no eth3 tap80b3e489-a6 tapfec914c0-0e virbr08000.525400ed85d9 yes virbr0-nic [root@compute1 ~]# virsh list IdName State 28instance-002f running 39instance-0044 running 41instance-0047 running Then when dhcp namespace and vms are in different hosts, dhcp traffic(in provider and selfservice network mode) is dropped in the controller bridge. Because no rule for permiting that the dhcp reply goes out of the controller: Iptables: -A neutron-filter-top -j neutron-linuxbri-local -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap4baecbed-83 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap80087b5b-33 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap94526e09-2c --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap8924b588-55 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT interfaces: [root@controller1 ~]# ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:d6:e9:8f brd ff:ff:ff:ff:ff:ff 3: eth1: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:7a:23:a5 brd ff:ff:ff:ff:ff:ff 4: eth2: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:5f:07:d9 brd ff:ff:ff:ff:ff:ff 28: eth3: mtu 1500 qdisc pfifo_fast master brqbab8fb94-c8 state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:b2:b7:bc brd ff:ff:ff:ff:ff:ff 30: tap4baecbed-83@if2: mtu 1500 qdisc noqueue master brqbab8fb94-c8 state UP mode DEFAULT group default qlen 1000 link/ether c6:e3:d5:e8:49:78 brd ff:ff:ff:ff:ff:ff link-netnsid 0 31: brqbab8fb94-c8: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 12:75:44:9f:51:ef brd ff:ff:ff:ff:ff:ff 32: tap80087b5b-33@if2: mtu 1450 qdisc noqueue master brq37841a31-d7 state UP mode DEFAULT group default qlen 1000 link/ether 0a:7e:06:92:99:a3 brd ff:ff:ff:ff:ff:ff link-netnsid 1 33: vxlan-46: mtu 1450 qdisc noqueue master brq37841a31-d7 state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 92:6d:dd:cd:ab:43 brd ff:ff:ff:ff:ff:ff 34: brq37841a31-d7: mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 0a:7e:06:92:99:a3 brd ff:ff:ff:ff:ff:ff 35: tap94526e09-2c@if2: mtu 1450 qdisc noqueue master brq37841a31-d7 state UP mode DEFAULT group default qlen 1000 link/ether
[Yahoo-eng-team] [Bug 1795432] [NEW] neutron does not create the necessary iptables rules for dhcp agents when linuxbridge is used
Public bug reported: Reproduction: Create a enviroment with controller and compute in different hosts: controller: [root@controller1 ~]# brctl show bridge name bridge id STP enabled interfaces brq37841a31-d7 8000.0a7e069299a3 no tap80087b5b-33 tap94526e09-2c vxlan-46 brqbab8fb94-c8 8000.1275449f51ef no eth3 tap4baecbed-83 tap8924b588-55 [root@controller1 ~]# ip netns qrouter-bcb8c407-ab4c-4916-89a5-d1ba8ac786ae (id: 2) qdhcp-37841a31-d744-4c9f-b084-37cfaafe71ca (id: 1) qdhcp-bab8fb94-c849-4c6c-ada7-98ec9bc33b87 (id: 0) Compute host: [root@compute1 ~]# brctl show bridge name bridge id STP enabled interfaces brq37841a31-d7 8000.5e530dd5073b no tap171ccdb9-66 vxlan-46 brqbab8fb94-c8 8000.525400fec4c7 no eth3 tap80b3e489-a6 tapfec914c0-0e virbr0 8000.525400ed85d9 yes virbr0-nic [root@compute1 ~]# virsh list IdName State 28instance-002f running 39instance-0044 running 41instance-0047 running Then when dhcp namespace and vms are in different hosts, dhcp traffic(in provider and selfservice network mode) is dropped in the controller bridge. Because no rule for permiting that the dhcp reply goes out of the controller: Iptables: -A neutron-filter-top -j neutron-linuxbri-local -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap4baecbed-83 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap80087b5b-33 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap94526e09-2c --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-out tap8924b588-55 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT interfaces: [root@controller1 ~]# ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:d6:e9:8f brd ff:ff:ff:ff:ff:ff 3: eth1: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:7a:23:a5 brd ff:ff:ff:ff:ff:ff 4: eth2: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:5f:07:d9 brd ff:ff:ff:ff:ff:ff 28: eth3: mtu 1500 qdisc pfifo_fast master brqbab8fb94-c8 state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:b2:b7:bc brd ff:ff:ff:ff:ff:ff 30: tap4baecbed-83@if2: mtu 1500 qdisc noqueue master brqbab8fb94-c8 state UP mode DEFAULT group default qlen 1000 link/ether c6:e3:d5:e8:49:78 brd ff:ff:ff:ff:ff:ff link-netnsid 0 31: brqbab8fb94-c8: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 12:75:44:9f:51:ef brd ff:ff:ff:ff:ff:ff 32: tap80087b5b-33@if2: mtu 1450 qdisc noqueue master brq37841a31-d7 state UP mode DEFAULT group default qlen 1000 link/ether 0a:7e:06:92:99:a3 brd ff:ff:ff:ff:ff:ff link-netnsid 1 33: vxlan-46: mtu 1450 qdisc noqueue master brq37841a31-d7 state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 92:6d:dd:cd:ab:43 brd ff:ff:ff:ff:ff:ff 34: brq37841a31-d7: mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 0a:7e:06:92:99:a3 brd ff:ff:ff:ff:ff:ff 35: tap94526e09-2c@if2: mtu 1450 qdisc noqueue master brq37841a31-d7 state UP mode DEFAULT group default qlen 1000 link/ether fe:a4:58:9e:52:2f brd ff:ff:ff:ff:ff:ff link-netnsid 2 36: tap8924b588-55@if3: mtu 1500 qdisc noqueue master brqbab8fb94-c8 state UP mode DEFAULT group default qlen 1000 link/ether 12:75:44:9f:51:ef brd ff:ff:ff:ff:ff:ff link-netnsid 2 Only rules for the tap ports. It is necessary add rules to permit dhcp traffic between hosts, for example permit dhcp ports as dev-in: -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap4baecbed-83 --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT -A neutron-linuxbri-FORWARD -m physdev --physdev-in tap80087b5b-33 --physdev-is-bridged -m comment --comment "Accept all packets when port