Re: [ovs-discuss] OVN/OVS no ARP response for internal router interface from external port
After investigating further, I believe I am hitting the following issue:
https://bugs.launchpad.net/neutron/+bug/1995078
Essentially the external port and the LRP are being scheduled separately and
without coordination. Because of this, if these ports are scheduled on
different chassis the ARP request is dropped. Will need to build/test this fix
and will follow up with a conclusion.
Juniper Business Use Only
From: Austin Cormier
Date: Thursday, February 1, 2024 at 5:39 PM
To: ovs-discuss@openvswitch.org
Subject: OVN/OVS no ARP response for internal router interface from external
port
Looking for some help troubleshooting why OVS is not generating a response for
my internal router port on a VLAN tenant network. I’ve dug down as far as I am
reasonably able to but need a quick boost here.
The ARP request is coming from an external system which is on the appropriate
VLAN for the tenant network. East/West traffic is working as expected as I am
able to communicate successfully with another VM on that vlan tenant network.
It APPEARS that the flow never gets generated on br-int in the appropriate
controller.
I am going to walk through my debugging steps starting from the NB database to
OVS on the appropriate controller that should be generating the ARP response:
In OVN NB, I have a router with a port. This is my internal gateway interface
of 192.168.5.1
-
router 21cd6ac3-4804-4c68-a683-9bba07d97967
(neutron-5d87debf-cf0b-4fac-ba49-01b7680368aa) (aka vlan_test)
port lrp-8c020ac1-ae54-4aa7-a143-4440067e9f42
mac: "fa:16:3e:37:43:b3"
networks: ["192.168.5.1/24"]
port lrp-e18d09db-19d8-4362-8252-751e6974ef5e
mac: "fa:16:3e:37:67:a3"
networks: ["10.27.14.50/23"]
gateway chassis: [infra-prod-controller-02 infra-prod-controller-01
infra-prod-controller-03]
nat 40605ce2-3f93-4877-ac26-47e4b257fa5f
external ip: "10.27.14.50"
logical ip: "192.168.5.0/24"
type: "snat"
The logical switch for the tenant network is connected to a localnet with tag
1106 and has the external port for my baremetal device and the appropriate
router port.
switch be7c870d-6d9c-471f-8996-e48a551068a0
(neutron-fcc39c5e-d3d8-4400-b5cf-b10f76d0112b) (aka vlan_test)
port provnet-5ddf8b20-c849-484a-ad8f-a86a9856c2b2
type: localnet
tag: 1106
addresses: ["unknown"]
port 1f01c94e-f32f-4e94-b02a-813bb1ad4a47
addresses: ["unknown"]
port 85aa9a1e-cb84-4137-97ce-85958a948390
addresses: ["fa:16:3e:ca:6e:3b 192.168.5.188"]
port 8c020ac1-ae54-4aa7-a143-4440067e9f42
type: router
router-port: lrp-8c020ac1-ae54-4aa7-a143-4440067e9f42
port 19279d0c-7c9a-498e-a3e5-269933c49df6
type: localport
addresses: ["fa:16:3e:4a:14:28 192.168.5.2"]
port 4c9047c4-a6c7-4b27-9cfa-58b5d30ce964
type: external
addresses: ["90:ec:77:32:e6:6e 192.168.5.56"]
In OVN SB, I can see that the external port
(4c9047c4-a6c7-4b27-9cfa-58b5d30ce964) has been scheduled on
infra-prod-controller-02. This is important because the ARP response would
only get generated from a single HA Chassis.
---
Chassis infra-prod-controller-02
hostname: infra-prod-controller-02
Encap geneve
ip: "10.27.12.24"
options: {csum="true"}
Port_Binding cr-lrp-20ba6028-7220-4c8d-a20f-9e4c416da3f7
Port_Binding "71e436bb-7121-473e-a024-e34d4d7f4a4f"
Port_Binding cr-lrp-c03b5dd9-92e1-4046-be1c-a953c0fab238
Port_Binding "f8eb9e30-e65f-44c4-94b6-a67700790880"
Port_Binding cr-lrp-ae2f5dbb-2cd0-44d0-9061-71c8186440be
Port_Binding "4c9047c4-a6c7-4b27-9cfa-58b5d30ce964"
Port_Binding "eb4435ad-37f2-44f9-a786-470b18bb9f0d"
Port_Binding cr-lrp-950eec85-b785-474b-837b-4ecbbcf080c9
Port_Binding "f3bbbe9a-a1a7-44b5-b6bc-b00a351ca1a5"
Port_Binding "79430028-7ae3-448c-bc12-c9d7d44d218b"
---
In OVN SB again, I can issue a trace command to verify that the Logical Flow
exists to generate the ARP response:
---
# ovn-trace neutron-fcc39c5e-d3d8-4400-b5cf-b10f76d0112b 'inport ==
"provnet-5ddf8b20-c849-484a-ad8f-a86a9856c2b2" && eth.src == 90:ec:77:32:e6:6e
&& eth.dst == ff:ff:ff:ff:ff:ff && arp.tpa == 192.168.5.1 && arp.spa ==
192.168.5
.178 && arp.op == 1 && arp.tha == ff:ff:ff:ff:ff:ff && arp.sha ==
90:ec:77:32:e6:6e'
…
ingress(dp="vlan_test", inport="lrp-8c020a")
0. lr_in_admission (northd.c:12885): eth.mcast && inport ==
"lrp-8c020a", priority 50, uuid faac4787
xreg0[0..47] = fa:16:3e:37:43:b3;
next;
1. lr_in_lookup_neighbor (northd.c:13142): inport == "lrp-8c020a" &&
arp.spa == 192.168.5.0/24 && arp.tpa == 192.168.5.1 && arp.op == 1, priority
110, uuid d447a962
reg9[2] = lookup_arp(inport, arp.spa, arp.sha);
/* MAC binding to ff:ff:ff:ff:ff:ff found. */
reg9[3] = 1;
next;
[ovs-discuss] OVN/OVS no ARP response for internal router interface from external port
Looking for some help troubleshooting why OVS is not generating a response for
my internal router port on a VLAN tenant network. I’ve dug down as far as I am
reasonably able to but need a quick boost here.
The ARP request is coming from an external system which is on the appropriate
VLAN for the tenant network. East/West traffic is working as expected as I am
able to communicate successfully with another VM on that vlan tenant network.
It APPEARS that the flow never gets generated on br-int in the appropriate
controller.
I am going to walk through my debugging steps starting from the NB database to
OVS on the appropriate controller that should be generating the ARP response:
In OVN NB, I have a router with a port. This is my internal gateway interface
of 192.168.5.1
-
router 21cd6ac3-4804-4c68-a683-9bba07d97967
(neutron-5d87debf-cf0b-4fac-ba49-01b7680368aa) (aka vlan_test)
port lrp-8c020ac1-ae54-4aa7-a143-4440067e9f42
mac: "fa:16:3e:37:43:b3"
networks: ["192.168.5.1/24"]
port lrp-e18d09db-19d8-4362-8252-751e6974ef5e
mac: "fa:16:3e:37:67:a3"
networks: ["10.27.14.50/23"]
gateway chassis: [infra-prod-controller-02 infra-prod-controller-01
infra-prod-controller-03]
nat 40605ce2-3f93-4877-ac26-47e4b257fa5f
external ip: "10.27.14.50"
logical ip: "192.168.5.0/24"
type: "snat"
The logical switch for the tenant network is connected to a localnet with tag
1106 and has the external port for my baremetal device and the appropriate
router port.
switch be7c870d-6d9c-471f-8996-e48a551068a0
(neutron-fcc39c5e-d3d8-4400-b5cf-b10f76d0112b) (aka vlan_test)
port provnet-5ddf8b20-c849-484a-ad8f-a86a9856c2b2
type: localnet
tag: 1106
addresses: ["unknown"]
port 1f01c94e-f32f-4e94-b02a-813bb1ad4a47
addresses: ["unknown"]
port 85aa9a1e-cb84-4137-97ce-85958a948390
addresses: ["fa:16:3e:ca:6e:3b 192.168.5.188"]
port 8c020ac1-ae54-4aa7-a143-4440067e9f42
type: router
router-port: lrp-8c020ac1-ae54-4aa7-a143-4440067e9f42
port 19279d0c-7c9a-498e-a3e5-269933c49df6
type: localport
addresses: ["fa:16:3e:4a:14:28 192.168.5.2"]
port 4c9047c4-a6c7-4b27-9cfa-58b5d30ce964
type: external
addresses: ["90:ec:77:32:e6:6e 192.168.5.56"]
In OVN SB, I can see that the external port
(4c9047c4-a6c7-4b27-9cfa-58b5d30ce964) has been scheduled on
infra-prod-controller-02. This is important because the ARP response would
only get generated from a single HA Chassis.
---
Chassis infra-prod-controller-02
hostname: infra-prod-controller-02
Encap geneve
ip: "10.27.12.24"
options: {csum="true"}
Port_Binding cr-lrp-20ba6028-7220-4c8d-a20f-9e4c416da3f7
Port_Binding "71e436bb-7121-473e-a024-e34d4d7f4a4f"
Port_Binding cr-lrp-c03b5dd9-92e1-4046-be1c-a953c0fab238
Port_Binding "f8eb9e30-e65f-44c4-94b6-a67700790880"
Port_Binding cr-lrp-ae2f5dbb-2cd0-44d0-9061-71c8186440be
Port_Binding "4c9047c4-a6c7-4b27-9cfa-58b5d30ce964"
Port_Binding "eb4435ad-37f2-44f9-a786-470b18bb9f0d"
Port_Binding cr-lrp-950eec85-b785-474b-837b-4ecbbcf080c9
Port_Binding "f3bbbe9a-a1a7-44b5-b6bc-b00a351ca1a5"
Port_Binding "79430028-7ae3-448c-bc12-c9d7d44d218b"
---
In OVN SB again, I can issue a trace command to verify that the Logical Flow
exists to generate the ARP response:
---
# ovn-trace neutron-fcc39c5e-d3d8-4400-b5cf-b10f76d0112b 'inport ==
"provnet-5ddf8b20-c849-484a-ad8f-a86a9856c2b2" && eth.src == 90:ec:77:32:e6:6e
&& eth.dst == ff:ff:ff:ff:ff:ff && arp.tpa == 192.168.5.1 && arp.spa ==
192.168.5
.178 && arp.op == 1 && arp.tha == ff:ff:ff:ff:ff:ff && arp.sha ==
90:ec:77:32:e6:6e'
…
ingress(dp="vlan_test", inport="lrp-8c020a")
0. lr_in_admission (northd.c:12885): eth.mcast && inport ==
"lrp-8c020a", priority 50, uuid faac4787
xreg0[0..47] = fa:16:3e:37:43:b3;
next;
1. lr_in_lookup_neighbor (northd.c:13142): inport == "lrp-8c020a" &&
arp.spa == 192.168.5.0/24 && arp.tpa == 192.168.5.1 && arp.op == 1, priority
110, uuid d447a962
reg9[2] = lookup_arp(inport, arp.spa, arp.sha);
/* MAC binding to ff:ff:ff:ff:ff:ff found. */
reg9[3] = 1;
next;
2. lr_in_learn_neighbor (northd.c:13078): reg9[2] == 1 || reg9[3] ==
0, priority 100, uuid 6aad6f8d
mac_cache_use;
next;
3. lr_in_ip_input (northd.c:12440): inport == "lrp-8c020a" && arp.op
== 1 && arp.tpa == 192.168.5.1 && arp.spa == 192.168.5.0/24 &&
is_chassis_resident("cr-lrp-e18d09"), priority 90, uuid 6187e537
eth.dst = eth.src;
eth.src = xreg0[0..47];
arp.op = 2;
arp.tha = arp.sha;
arp.sha = xreg0[0..47];
arp.tpa <-> arp.spa;
outport = inport;
flags.loopback =
[ovs-discuss] OVN VTEP service replication mode handing
I am investigating HW VTEP integration with OVN using Juniper QFX 5100/5120. This switch does not support source node replication for BUM traffic (broadcast, unknown unicast, multicast). As such it will use service node replication which sends the packet to one of the other OVN nodes to be replicated there. My confusion here is when a OVN node receives this packets how does it know that it needs to replicate the packet to the other nodes as opposed just consuming it? Is there metadata on the packet which indicates that it should be replicated? Or does it check whether the packet was sent by the VTEP? Any pointers on how this mechanism works would be greatly appreciated. -Austin Juniper Business Use Only ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: DHCP Support for vtep port type
Hi Vladislav,
Thank you for the references. I will investigate this approach. It would be
simpler for us to provide the DHCP this way but will have to dive in to
understand how much effort would be required. Really appreciate it.
-Austin
Juniper Business Use Only
From: Vladislav Odintsov
Date: Thursday, August 31, 2023 at 3:19 PM
To: Austin Cormier
Cc: ovs-discuss@openvswitch.org
Subject: Re: [ovs-discuss] OVN: DHCP Support for vtep port type
[External Email. Be cautious of content]
Hi Austin,
There is a special function in northd.c to handle hairpin traffic -
build_vtep_hairpin().
The mentioned flows are not matched due to flow [1], which by default sends all
traffic received from vtep lport to l2lkp table for local delivery.
As you correctly noticed, there is no instance to handle "special" traffic
coming from VTEP.
But there is a mechanism introduced in [2], [3], where you can attach LS to LR
and set gw chassis for associated LRP to one of your chassis.
ARP responses for LRPs are made with such logic [4].
You can try to add similar logic for DHCP traffic, set loopback flag and
resubmit packet to appropriate table. This can be not enough.
Hope this helps.
1: https://github.com/ovn-org/ovn/blob/0d021216c/northd/northd.c#L8374-L8377
2:
https://github.com/ovn-org/ovn/commit/4e90bcf55c2ef1ab9940836455e4bfe36e57f307
3:
https://github.com/ovn-org/ovn/commit/f20f664bc962094e4679ff2d3a8d834637bff27f
4: https://github.com/ovn-org/ovn/blob/0d021216c/northd/northd.c#L8403-L8405
On 31 Aug 2023, at 21:04, Austin Cormier via discuss
wrote:
I’m trying to understand whether DHCP is supported (or can easily be supported)
for vtep port types. The use case I am driving towards is to support DHCP/PXE
booting for hardware over vxlan overlay network via a TOR switch. I believe
this could work using Neutron DHCP Agent but we’ve hit scaling issues in that
area so we’re attempting to use OVN DHCP server if possible.
In the NB DB, the vtep port is added to the ironic provisioning network with
both the IP/MAC set.
$ ovn-nbctl show
…
switch 99a907e0-7525-4d4e-a029-fddaafeff3e5
(neutron-a2d49759-d8fa-43a3-8620-35201420f72b) (aka Ironic Provisioning)
port provnet-2fd9fb49-7e68-4f86-8690-aec776e9f128
type: localnet
addresses: ["unknown"]
port 2ebf0521-3f7e-44b6-848f-82a0214e3c25
type: localport
addresses: ["fa:16:3e:7f:e9:56 10.90.100.1"]
port 23af2a74-0333-4600-9e11-e99bff860999
type: vtep
addresses: ["0c:c4:7a:ac:75:4e 10.90.100.32"]
In the SB database, I see the port binding under the data-switch-01 chassis:
$ ovn-sbctl show
Chassis compute-test
hostname: compute-test
Encap vxlan
ip: "10.40.0.1"
options: {csum="true"}
Port_Binding "04901637-9467-4f01-b16b-e22de768a1b8"
Chassis data-switch-01
Encap vxlan
ip: "17.17.17.17"
options: {csum="false"}
Port_Binding "23af2a74-0333-4600-9e11-e99bff860999"
Chassis controller-test
hostname: controller-test
Encap geneve
ip: "10.40.0.3"
options: {csum="true"}
Encap vxlan
ip: "10.40.0.3"
options: {csum="true"}
Port_Binding cr-lrp-876f85ad-0a6f-4668-8736-e122fedc32c2
In the logical flow list, it appears that the DHCP responses are added:
table=19(ls_in_arp_rsp ), priority=50 , match=(arp.tpa == 10.90.100.1
&& arp.op == 1), action=(eth.dst = eth.src; eth.src = fa:16:3e:7f:e9:56; arp.op
= 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = fa:16:3e:7f:e9:56; arp.tpa =
arp.spa; arp.spa = 10.90.100.1; ou
tport = inport; flags.loopback = 1; output;)
table=19(ls_in_arp_rsp ), priority=50 , match=(arp.tpa == 10.90.100.32
&& arp.op == 1), action=(eth.dst = eth.src; eth.src = 0c:c4:7a:ac:75:4e; arp.op
= 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = 0c:c4:7a:ac:75:4e; arp.tpa =
arp.spa; arp.spa = 10.90.100.32;
outport = inport; flags.loopback = 1; output;)
table=19(ls_in_arp_rsp ), priority=0, match=(1), action=(next;)
table=20(ls_in_dhcp_options ), priority=100 , match=(inport ==
"23af2a74-0333-4600-9e11-e99bff860999" && eth.src == 0c:c4:7a:ac:75:4e &&
ip4.src == 0.0.0.0 && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst ==
67), action=(reg0[3] = put_dhcp_opts(offerip = 10.90
.100.32, bootfile_name =
http://192.168.3.3:8089/boot.ipxe<https://urldefense.com/v3/__http:/192.168.3.3:8089/boot.ipxe__;!!NEt6yMaO-gk!AfdRg7WKrtczkbjgobERRNKzATR-BOn8U7axVrVO3AJOaGsP2i_FLcnmeHJpqkCBI6ytfXL2Z9A3W4Sz$>,
bootfile_name_alt = "snponly.efi", classless_static_route =
{169.254.169.254/32,10.90.100.1}, dns_server = {127.0.0.53}, lease_time =
43200, mtu = 1500, netmask = 255.255.0.0, next_server = 192.168.3.3, server_
id = 10.90.100.1, tftp_server =
[ovs-discuss] OVN: DHCP Support for vtep port type
I’m trying to understand whether DHCP is supported (or can easily be supported)
for vtep port types. The use case I am driving towards is to support DHCP/PXE
booting for hardware over vxlan overlay network via a TOR switch. I believe
this could work using Neutron DHCP Agent but we’ve hit scaling issues in that
area so we’re attempting to use OVN DHCP server if possible.
In the NB DB, the vtep port is added to the ironic provisioning network with
both the IP/MAC set.
$ ovn-nbctl show
…
switch 99a907e0-7525-4d4e-a029-fddaafeff3e5
(neutron-a2d49759-d8fa-43a3-8620-35201420f72b) (aka Ironic Provisioning)
port provnet-2fd9fb49-7e68-4f86-8690-aec776e9f128
type: localnet
addresses: ["unknown"]
port 2ebf0521-3f7e-44b6-848f-82a0214e3c25
type: localport
addresses: ["fa:16:3e:7f:e9:56 10.90.100.1"]
port 23af2a74-0333-4600-9e11-e99bff860999
type: vtep
addresses: ["0c:c4:7a:ac:75:4e 10.90.100.32"]
In the SB database, I see the port binding under the data-switch-01 chassis:
$ ovn-sbctl show
Chassis compute-test
hostname: compute-test
Encap vxlan
ip: "10.40.0.1"
options: {csum="true"}
Port_Binding "04901637-9467-4f01-b16b-e22de768a1b8"
Chassis data-switch-01
Encap vxlan
ip: "17.17.17.17"
options: {csum="false"}
Port_Binding "23af2a74-0333-4600-9e11-e99bff860999"
Chassis controller-test
hostname: controller-test
Encap geneve
ip: "10.40.0.3"
options: {csum="true"}
Encap vxlan
ip: "10.40.0.3"
options: {csum="true"}
Port_Binding cr-lrp-876f85ad-0a6f-4668-8736-e122fedc32c2
In the logical flow list, it appears that the DHCP responses are added:
table=19(ls_in_arp_rsp ), priority=50 , match=(arp.tpa == 10.90.100.1
&& arp.op == 1), action=(eth.dst = eth.src; eth.src = fa:16:3e:7f:e9:56; arp.op
= 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = fa:16:3e:7f:e9:56; arp.tpa =
arp.spa; arp.spa = 10.90.100.1; ou
tport = inport; flags.loopback = 1; output;)
table=19(ls_in_arp_rsp ), priority=50 , match=(arp.tpa == 10.90.100.32
&& arp.op == 1), action=(eth.dst = eth.src; eth.src = 0c:c4:7a:ac:75:4e; arp.op
= 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = 0c:c4:7a:ac:75:4e; arp.tpa =
arp.spa; arp.spa = 10.90.100.32;
outport = inport; flags.loopback = 1; output;)
table=19(ls_in_arp_rsp ), priority=0, match=(1), action=(next;)
table=20(ls_in_dhcp_options ), priority=100 , match=(inport ==
"23af2a74-0333-4600-9e11-e99bff860999" && eth.src == 0c:c4:7a:ac:75:4e &&
ip4.src == 0.0.0.0 && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst ==
67), action=(reg0[3] = put_dhcp_opts(offerip = 10.90
.100.32, bootfile_name = http://192.168.3.3:8089/boot.ipxe, bootfile_name_alt =
"snponly.efi", classless_static_route = {169.254.169.254/32,10.90.100.1},
dns_server = {127.0.0.53}, lease_time = 43200, mtu = 1500, netmask =
255.255.0.0, next_server = 192.168.3.3, server_
id = 10.90.100.1, tftp_server = "192.168.3.3", tftp_server_address =
192.168.3.3); next;)
table=20(ls_in_dhcp_options ), priority=100 , match=(inport ==
"23af2a74-0333-4600-9e11-e99bff860999" && eth.src == 0c:c4:7a:ac:75:4e &&
ip4.src == 10.90.100.32 && ip4.dst == {10.90.100.1, 255.255.255.255} && udp.src
== 68 && udp.dst == 67), action=(reg0[3] = put_dhcp_
opts(offerip = 10.90.100.32, bootfile_name = http://192.168.3.3:8089/boot.ipxe,
bootfile_name_alt = "snponly.efi", classless_static_route =
{169.254.169.254/32,10.90.100.1}, dns_server = {127.0.0.53}, lease_time =
43200, mtu = 1500, netmask = 255.255.0.0, next_server =
192.168.3.3, server_id = 10.90.100.1, tftp_server = "192.168.3.3",
tftp_server_address = 192.168.3.3); next;)
table=20(ls_in_dhcp_options ), priority=0, match=(1), action=(next;)
table=21(ls_in_dhcp_response), priority=100 , match=(inport ==
"23af2a74-0333-4
But when I packet capture I see the DHCP requests but no responses are sent.
Using OVN-trace, it doesn’t appear that we’re hitting the appropriate flow to
generate the DHCP response:
---
# ovn-trace 99a907e0-7525-4d4e-a029-fddaafeff3e5 'inport ==
"23af2a74-0333-4600-9e11-e99bff860999" && eth.src == 0c:c4:7a:ac:75:4e &&
ip4.src == 0.0.0.0 && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst ==
67'
#
udp,reg14=0x3,vlan_tci=0x,dl_src=0c:c4:7a:ac:75:4e,dl_dst=00:00:00:00:00:00,nw_src=0.0.0.0,nw_dst=255.255.255.255,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no,tp_src=68,tp_dst=67
ingress(dp="Ironic Provisioning", inport="23af2a")
--
0. ls_in_check_port_sec (northd.c:5680): inport == "23af2a", priority 70, uuid
aaeb17c1
reg0[14] = 1;
next(16);
16. ls_in_hairpin (northd.c:7637): reg0[14] == 1, priority 1000, uuid bee04e33
next(25);
25. ls_in_l2_lkup (northd.c:8326): 1, priority 0, uuid f7842991
outport = get_fdb(eth.dst);
next;
26. ls_in_l2_unknown (northd.c:8330): outport == "none", priority 50, uuid
da5c
