UDP stateful ACL not working? The logical representation of My
network is shown bellow
('ovn-nbctl show' shown towards the end). I have a Port Group
(pg_vcn3_net1_sl3) that has
two ports in it, the VM port on switch(ls_vcn3_net1) and
lsb_vcn4_stgw-lr_vcn3_stgw switch
(ls_vcn3_backbone) asshown below ((o)).
I do a 'showmount -e 192.16.1.106' in the VM, I see the pkt go out
from the VM get to the NFS
server on the underlay, see the reply on the underlay and then I see
my PG ACL drop the pkt.
The ACLs are:
Egress From VM - Ingress to switch
-----------------------------------
from-lport 32767 (inport == @pg_vcn3_net1_sl3 && (arp || udp.dst == 67
|| udp.dst == 68)) allow-related
from-lport 27000 (inport == @pg_vcn3_net1_sl3 && ip4.dst ==
192.16.1.0/24 && udp.dst == 111) allow-related
from-lport 0 (inport == @pg_vcn3_net1_sl3) drop
log(name=fss-8,severity=debug) <------- Drops the return pkt
Ingress TO VM - Egress from switch
------------------------------------
to-lport 32767 (outport == @pg_vcn3_net1_sl3 && (arp || udp.dst ==
67 || udp.dst == 68)) allow-related
to-lport 27000 (outport == @pg_vcn3_net1_sl3 && ip4.src ==
192.16.1.0/24 && tcp.dst == 111) allow-related
to-lport 27000 (outport == @pg_vcn3_net1_sl3 && ip4.src ==
192.16.1.0/24 && tcp.dst == 20048) allow-related
* to-lport 27000 (outport == @pg_vcn3_net1_sl3 && ip4.src ==
192.16.1.0/24 && udp.dst == 111) allow-related <------- But this should *
to-lport 0 (outport == @pg_vcn3_net1_sl3) drop
log(name=fss-17,severity=debug) have
allowed the
return pkt
+----------------+
| VM |
| 192.16.1.6 |
+-----((O))------+
| *284195d2-9280-4334-900e-571ecd00327a *in
PG*pg_vcn3_net1_sl3*
+---------------------+
| ls_vcn3_net1 |
+---------------------+
| ls_vcn3_net1-lr_vcn3_net1 (proxy ARP for 192.16.1.106)
|
|
| lr_vcn3_net1-ls_vcn3_net1 (192.16.1.1/24)
/------------\
( lr_vcn3_net1 )
\------------/
| lr_vcn3_net1-lsb_vcn3_net1 (253.255.25.1/25)
|
|
| lsb_vcn3_net1-lr_vcn3_net1
+------------------------+
| ls_vcn3_backbone |
+--------((O))-----------+
| *lsb_vcn4_stgw-lr_vcn3_stgw *in PG*pg_vcn3_net1_sl3*
|
|
| lr_vcn3_stgw-lsb_vcn3_stgw (253.255.25.10/25)
/------------\
( lr_vcn3_stgw ) SNAT 192.16.1.6 to 253.255.80.8
\------------/
| lr_vcn3_stgw-ls_vcn3_external_stgw (253.255.80.20/16)
|
|
| ls_vcn3_external_stgw-lr_vcn3_stgw
+-----------------------+
| ls_vcn3_external_stgw |
+-----------------------+
| ln-ls_vcn3_external_stgw
| (localnet)
|
+---------+
| br-ext | Physical OVS on chassis
+---------+
| Egress : Change dst 192.16.1.106 to dst 253.255.0.2
| Ingress: Change src 253.255.0.2 to 192.16.1.106
253.255.0.0/16 |
|
+---------------+
| NFS server |
| 253.255.0.2 |
+---------------+
When I do a trace of the out going pkt, it looks like to me that there
is no conntrack
established in the ls_vcn3_backbone so it does not recognize the
return pkt as a return
but the 'allow-related' should have established that. See Below
ovn-trace --detailed ls_vcn3_net1 'inport ==
"284195d2-9280-4334-900e-571ecd00327a" && eth.dst == 40:44:00:00:00:90
&& eth.src == 52:54:00:02:55:96 && ip4.src == 192.16.1.6 && ip4.dst ==
192.16.1.106 && ip.ttl == 64 && udp.dst == 111'
#
udp,reg14=0x1,vlan_tci=0x0000,dl_src=52:54:00:02:55:96,dl_dst=40:44:00:00:00:90,nw_src=192.16.1.6,nw_dst=192.16.1.106,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=111
ingress(dp="ls_vcn3_net1", inport="284195")
-------------------------------------------
0. ls_in_port_sec_l2 (northd.c:5493): inport == "284195", priority
50, uuid 0b9563aa
next;
5. ls_in_pre_acl (northd.c:5753): ip, priority 100, uuid fad1d4d2
reg0[0] = 1;
next;
7. ls_in_pre_stateful (northd.c:5937): reg0[0] == 1, priority 100,
uuid 0bb84a55
ct_next;
ct_next(ct_state=est|trk /* default (use --ct to customize) */)
*<----- Looks like we set up conntrack on the ls_vcn3_net1*
---------------------------------------------------------------
8. ls_in_acl_hint (northd.c:6019): !ct.new && ct.est && !ct.rpl &&
ct_label.blocked == 0, priority 4, uuid 2712c6a2
reg0[8] = 1;
reg0[10] = 1;
next;
9. ls_in_acl (northd.c:6244): reg0[8] == 1 && (inport ==
@pg_vcn3_net1_sl3 && ip4.dst == 192.16.1.0/24 && udp.dst == 111),
priority 28000, uuid aff2df9c
next;
22. ls_in_l2_lkup (northd.c:8218): eth.dst == 40:44:00:00:00:90,
priority 50, uuid 9b0ee70d
outport = "ls_vcn3_net1-lr_vcn3_net1";
output;
egress(dp="ls_vcn3_net1", inport="284195",
outport="ls_vcn3_net1-lr_vcn3_net1")
-------------------------------------------------------------------------------
0. ls_out_pre_lb (northd.c:5643): ip && outport ==
"ls_vcn3_net1-lr_vcn3_net1", priority 110, uuid d8ef0aac
next;
1. ls_out_pre_acl (northd.c:5643): ip && outport ==
"ls_vcn3_net1-lr_vcn3_net1", priority 110, uuid d5cd7965
next;
3. ls_out_acl_hint (northd.c:6019): !ct.new && ct.est && !ct.rpl &&
ct_label.blocked == 0, priority 4, uuid 930df986
reg0[8] = 1;
reg0[10] = 1;
next;
9. ls_out_port_sec_l2 (northd.c:5591): outport ==
"ls_vcn3_net1-lr_vcn3_net1", priority 50, uuid 56148a7c
output;
/* output to "ls_vcn3_net1-lr_vcn3_net1", type "patch" */
ingress(dp="lr_vcn3_net1", inport="lr_vcn3_net1-ls_vcn3_net1")
--------------------------------------------------------------
0. lr_in_admission (northd.c:10519): eth.dst == 40:44:00:00:00:90 &&
inport == "lr_vcn3_net1-ls_vcn3_net1", priority 50, uuid 08eee924
xreg0[0..47] = 40:44:00:00:00:90;
next;
1. lr_in_lookup_neighbor (northd.c:10662): 1, priority 0, uuid d3159b2b
reg9[2] = 1;
next;
2. lr_in_learn_neighbor (northd.c:10671): reg9[2] == 1, priority 100,
uuid bef4eddc
next;
10. lr_in_ip_routing_pre (northd.c:10905): 1, priority 0, uuid c7269d9a
reg7 = 0;
next;
11. lr_in_ip_routing (northd.c:9435): ip4.dst == 192.16.1.0/24,
priority 74, uuid a571523c
ip.ttl--;
reg8[0..15] = 0;
reg0 = ip4.dst;
reg1 = 192.16.1.1;
eth.src = 40:44:00:00:00:90;
outport = "lr_vcn3_net1-ls_vcn3_net1";
flags.loopback = 1;
next;
12. lr_in_ip_routing_ecmp (northd.c:10980): reg8[0..15] == 0, priority
150, uuid 5aa40905
next;
13. lr_in_policy (northd.c:8667): ip4.dst == 192.16.1.106, priority
100, uuid fe728a63
reg0 = 253.255.25.10;
reg1 = 253.255.25.1;
eth.src = 40:44:00:00:00:a0;
outport = "lr_vcn3_net1-lsb_vcn3_net1";
flags.loopback = 1;
reg8[0..15] = 0;
next;
14. lr_in_policy_ecmp (northd.c:11115): reg8[0..15] == 0, priority
150, uuid 3f7c2e78
next;
15. lr_in_arp_resolve (northd.c:11503): outport ==
"lr_vcn3_net1-lsb_vcn3_net1" && reg0 == 253.255.25.10, priority 100,
uuid 8274aa30
eth.dst = 40:44:00:00:05:00;
next;
19. lr_in_arp_request (northd.c:11795): 1, priority 0, uuid 27c79d22
output;
egress(dp="lr_vcn3_net1", inport="lr_vcn3_net1-ls_vcn3_net1",
outport="lr_vcn3_net1-lsb_vcn3_net1")
---------------------------------------------------------------------------------------------------
0. lr_out_chk_dnat_local (northd.c:13021): 1, priority 0, uuid 60605710
reg9[4] = 0;
next;
6. lr_out_delivery (northd.c:11843): outport ==
"lr_vcn3_net1-lsb_vcn3_net1", priority 100, uuid 0ca6195d
output;
/* output to "lr_vcn3_net1-lsb_vcn3_net1", type "patch" */
ingress(dp="ls_vcn3_backbone", inport="lsb_vcn3_net1-lr_vcn3_net1")
-------------------------------------------------------------------
0. ls_in_port_sec_l2 (northd.c:5493): inport ==
"lsb_vcn3_net1-lr_vcn3_net1", priority 50, uuid b32ddca1
next;
5. ls_in_pre_acl (northd.c:5640): ip && inport ==
"lsb_vcn3_net1-lr_vcn3_net1", priority 110, uuid ba4adcd0
next;
6. ls_in_pre_lb (northd.c:5640): ip && inport ==
"lsb_vcn3_net1-lr_vcn3_net1", priority 110, uuid d1a8a9c5
next;
8. ls_in_acl_hint (northd.c:6019): !ct.new && ct.est && !ct.rpl &&
ct_label.blocked == 0, priority 4, uuid 2712c6a2
reg0[8] = 1;
reg0[10] = 1;
next;
22. ls_in_l2_lkup (northd.c:8289): eth.dst == 40:44:00:00:05:00,
priority 50, uuid cc11ddc9
outport = "lsb_vcn3_stgw-lr_vcn3_stgw";
output;
egress(dp="ls_vcn3_backbone", inport="lsb_vcn3_net1-lr_vcn3_net1",
outport="lsb_vcn3_stgw-lr_vcn3_stgw") *<--- Does not look like any
conntrack estblished on the switch.*
--------------------------------------------------------------------------------------------------------
0. ls_out_pre_lb (northd.c:5643): ip && outport ==
"lsb_vcn3_stgw-lr_vcn3_stgw", priority 110, uuid e2f6ec8c
next;
1. ls_out_pre_acl (northd.c:5643): ip && outport ==
"lsb_vcn3_stgw-lr_vcn3_stgw", priority 110, uuid a79ed7f1
next;
3. ls_out_acl_hint (northd.c:6019): !ct.new && ct.est && !ct.rpl &&
ct_label.blocked == 0, priority 4, uuid 930df986
reg0[8] = 1;
reg0[10] = 1;
next;
4. ls_out_acl (northd.c:6244): reg0[8] == 1 && (outport ==
@pg_vcn3_net1_sl3 && ip4.src == 192.16.1.0/24 && udp.dst == 111),
priority 28000, uuid 159ff341
next;
9. ls_out_port_sec_l2 (northd.c:5591): outport ==
"lsb_vcn3_stgw-lr_vcn3_stgw", priority 50, uuid b6d24444
output;
/* output to "lsb_vcn3_stgw-lr_vcn3_stgw", type "patch" */
ingress(dp="lr_vcn3_stgw", inport="lr_vcn3_stgw-lsb_vcn3_stgw")
---------------------------------------------------------------
0. lr_in_admission (northd.c:10519): eth.dst == 40:44:00:00:05:00 &&
inport == "lr_vcn3_stgw-lsb_vcn3_stgw", priority 50, uuid 6bbbb2f9
xreg0[0..47] = 40:44:00:00:05:00;
next;
1. lr_in_lookup_neighbor (northd.c:10662): 1, priority 0, uuid d3159b2b
reg9[2] = 1;
next;
2. lr_in_learn_neighbor (northd.c:10671): reg9[2] == 1, priority 100,
uuid bef4eddc
next;
10. lr_in_ip_routing_pre (northd.c:10905): 1, priority 0, uuid c7269d9a
reg7 = 0;
next;
11. lr_in_ip_routing (northd.c:9435): reg7 == 0 && ip4.dst ==
192.16.0.0/16, priority 49, uuid 4bb6bae4
ip.ttl--;
reg8[0..15] = 0;
reg0 = 253.255.25.1;
reg1 = 253.255.25.10;
eth.src = 40:44:00:00:05:00;
outport = "lr_vcn3_stgw-lsb_vcn3_stgw";
flags.loopback = 1;
next;
12. lr_in_ip_routing_ecmp (northd.c:10980): reg8[0..15] == 0, priority
150, uuid 5aa40905
next;
13. lr_in_policy (northd.c:8667): ip4.dst == 192.16.1.106, priority
100, uuid 1fbe521c
reg0 = 253.255.0.2;
reg1 = 253.255.80.20;
eth.src = 40:44:00:00:05:01;
outport = "lr_vcn3_stgw-ls_vcn3_external_stgw";
flags.loopback = 1;
reg8[0..15] = 0;
next;
14. lr_in_policy_ecmp (northd.c:11115): reg8[0..15] == 0, priority
150, uuid 3f7c2e78
next;
15. lr_in_arp_resolve (northd.c:11149): ip4, priority 0, uuid 8070025f
get_arp(outport, reg0);
/* MAC binding to 98:03:9b:59:af:24. */
next;
18. lr_in_gw_redirect (northd.c:11716): outport ==
"lr_vcn3_stgw-ls_vcn3_external_stgw", priority 50, uuid 7a07c64e
outport = "cr-lr_vcn3_stgw-ls_vcn3_external_stgw";
next;
19. lr_in_arp_request (northd.c:11795): 1, priority 0, uuid 27c79d22
output;
/* Replacing type "chassisredirect" outport
"cr-lr_vcn3_stgw-ls_vcn3_external_stgw" with distributed port
"lr_vcn3_stgw-ls_vcn3_external_stgw". */
egress(dp="lr_vcn3_stgw", inport="lr_vcn3_stgw-lsb_vcn3_stgw",
outport="lr_vcn3_stgw-ls_vcn3_external_stgw")
------------------------------------------------------------------------------------------------------------
0. lr_out_chk_dnat_local (northd.c:13021): 1, priority 0, uuid 60605710
reg9[4] = 0;
next;
3. lr_out_snat (northd.c:12774): ip && ip4.src == 192.16.1.6 &&
outport == "lr_vcn3_stgw-ls_vcn3_external_stgw" &&
is_chassis_resident("cr-lr_vcn3_stgw-ls_vcn3_external_stgw"), priority
161, uuid a6cc7e3c
ct_snat_in_czone(253.255.80.8);
ct_snatin_czone(ip4.src=253.255.80.8)
-------------------------------------
6. lr_out_delivery (northd.c:11843): outport ==
"lr_vcn3_stgw-ls_vcn3_external_stgw", priority 100, uuid 5b43ad83
output;
/* output to "lr_vcn3_stgw-ls_vcn3_external_stgw", type "patch" */
ingress(dp="ls_vcn3_external_stgw",
inport="ls_vcn3_external_stgw-lr_vcn3_stgw")
--------------------------------------------------------------------------------
0. ls_in_port_sec_l2 (northd.c:5493): inport ==
"ls_vcn3_external_stgw-lr_vcn3_stgw", priority 50, uuid 311f5f1a
next;
6. ls_in_pre_lb (northd.c:5640): ip && inport ==
"ls_vcn3_external_stgw-lr_vcn3_stgw", priority 110, uuid f122a239
next;
22. ls_in_l2_lkup (northd.c:7494): 1, priority 0, uuid 0432699a
outport = get_fdb(eth.dst);
next;
23. ls_in_l2_unknown (northd.c:7499): outport == "none", priority 50,
uuid c66c5cf1
outport = "_MC_unknown";
output;
multicast(dp="ls_vcn3_external_stgw", mcgroup="_MC_unknown")
------------------------------------------------------------
egress(dp="ls_vcn3_external_stgw",
inport="ls_vcn3_external_stgw-lr_vcn3_stgw",
outport="ln-ls_vcn3_external_stgw")
-------------------------------------------------------------------------------------------------------------------
0. ls_out_pre_lb (northd.c:5643): ip && outport ==
"ln-ls_vcn3_external_stgw", priority 110, uuid 9e2e1e50
next;
9. ls_out_port_sec_l2 (northd.c:5591): outport ==
"ln-ls_vcn3_external_stgw", priority 50, uuid 33e53686
output;
/* output to "ln-ls_vcn3_external_stgw", type "localnet" */
More Details on the config
============================
============================
lr_vcn3_stgw
=============
Routing Tables
---------------
192.16.0.0/16 253.255.25.1 dst-ip lr_vcn3_stgw-lsb_vcn3_stgw
0.0.0.0/0 253.255.0.2 dst-ip
lr_vcn3_stgw-ls_vcn3_external_stgw
Policy Routing
--------------
100 ip4.dst == 192.16.1.106 reroute 253.255.0.2
lr_vcn3_net1
============
Policy Routing
--------------
100 ip4.dst == 192.16.1.106 reroute 253.255.25.10
br-ext flows
=============
priority=1013,ip,in_port=216,nw_dst=192.16.1.106
actions=mod_nw_dst:253.255.0.2,output:1
priority=913,ip,in_port=1,nw_src=253.255.0.2,nw_dst=253.255.80.8
actions=mod_nw_src:192.16.1.106,output:216
Failing ACLs - that Should Work
=================================
ovn-nbctl list Port_Group
_uuid : a4de7036-896e-4e54-a466-8a44f9a87960
acls : [1d2e36f9-47ca-4e8a-a0c6-587df26f0f3e,
44a4c403-ac29-4579-a771-0963d1f1fc4a,
bad8f976-a87d-47e4-805b-a3bf8057bed8,
c7655e9a-9eb8-48fc-8d25-6ca17d303e28,
ce002242-457b-48e8-b40f-77419cee43fb,
dd9bbafd-2e91-42bd-b1ad-df11b71850fe,
fc345e12-bfaf-498b-a011-bc4eeba31670,
fc6b6c71-6171-43d1-ba97-3e5e1a43065c]
external_ids : {}
name : pg_vcn3_net1_sl3
ports : [/306e0eab-6b23-405c-a38c-918c1b2e795d/,
*f8819747-84cf-4019-9028-690dff014bc1*]
ovn-nbctl lsp-list ls_vcn3_backbone | grep stgw
306e0eab-6b23-405c-a38c-918c1b2e795d (lsb_vcn3_stgw-lr_vcn3_stgw)
ovn-nbctl lsp-list ls_vcn3_net1
*f8819747-84cf-4019-9028-690dff014bc1*
(284195d2-9280-4334-900e-571ecd00327a)
/295f1e55-a9e5-42f9-bb8c-a679bcfa084d/ (ls_vcn3_net1-lr_vcn3_net1)
ovn-nbctl acl-list pg_vcn3_net1_sl3
=====================================
Egress From VM - Ingress to switch
-----------------------------------
from-lport 32767 (inport == @pg_vcn3_net1_sl3 && (arp || udp.dst == 67
|| udp.dst == 68)) allow-related
from-lport 27000 (inport == @pg_vcn3_net1_sl3 && ip4.dst ==
192.16.1.0/24 && udp.dst == 111) allow-related
from-lport 0 (inport == @pg_vcn3_net1_sl3) drop
log(name=fss-8,severity=debug) <------- Drops
Ingress TO VM - Egress from switch
------------------------------------
to-lport 32767 (outport == @pg_vcn3_net1_sl3 && (arp || udp.dst ==
67 || udp.dst == 68)) allow-related
to-lport 27000 (outport == @pg_vcn3_net1_sl3 && ip4.src ==
192.16.1.0/24 && tcp.dst == 111) allow-related
to-lport 27000 (outport == @pg_vcn3_net1_sl3 && ip4.src ==
192.16.1.0/24 && tcp.dst == 20048) allow-related
to-lport 27000 (outport == @pg_vcn3_net1_sl3 && ip4.src ==
192.16.1.0/24 && udp.dst == 111) allow-related <------- But this should
to-lport 0 (outport == @pg_vcn3_net1_sl3) drop
log(name=fss-17,severity=debug) have allowed the
&n
bsp; return pkt
Drop Message
=============
*name="fss-8", verdict=drop,
udp,dl_src=40:44:00:00:05:00,dl_dst=40:44:00:00:00:a0,nw_src=192.16.1.106,nw_dst=192.16.1.6,tp_src=111,tp_dst=755*
tcpdump on VM
--------------
52:54:00:02:55:96 > 40:44:00:00:00:90, ethertype IPv4, proto UDP (17)
192.16.1.6.755 > 192.16.1.106.111
tcpdump on underlay NFS Server
------------------------------
40:44:00:00:05:01 > 98:03:9b:59:af:24, ethertype IPv4, proto UDP (17)
253.255.80.8.755 > 253.255.0.2.111:
98:03:9b:59:af:24 > 40:44:00:00:05:01, ethertype IPv4, proto UDP (17)
253.255.0.2.111 > 253.255.80.8.755:
ovn-nbctl show
===============
switch 18e9d8e6-1cbc-4d4f-afc1-caec9aecf569 (ls_vcn3_net1)
port 284195d2-9280-4334-900e-571ecd00327a
addresses: ["52:54:00:02:55:96 192.16.1.6"]
port ls_vcn3_net1-lr_vcn3_net1
type: router
addresses: ["40:44:00:00:00:90"]
router-port: lr_vcn3_net1-ls_vcn3_net1
switch 7f43d2e4-f146-40d9-8be5-a7b75b319e75 (ls_vcn3_backbone)
port lsb_vcn3_net1-lr_vcn3_net1
type: router
router-port: lr_vcn3_net1-lsb_vcn3_net1
port lsb_vcn3_stgw-lr_vcn3_stgw
type: router
router-port: lr_vcn3_stgw-lsb_vcn3_stgw
switch a8e0c8fe-872d-4bfc-bdd9-90063a196603 (ls_vcn3_external_stgw)
port ls_vcn3_external_stgw-lr_vcn3_stgw
type: router
router-port: lr_vcn3_stgw-ls_vcn3_external_stgw
port ln-ls_vcn3_external_stgw
type: localnet
addresses: ["unknown"]
router 10f1c098-e5cc-4f85-b8a1-33c49356fd1c (lr_vcn3_net1)
port lr_vcn3_net1-ls_vcn3_net1
mac: "40:44:00:00:00:90"
networks: ["192.16.1.1/24"]
port lr_vcn3_net1-lsb_vcn3_net1
mac: "40:44:00:00:00:a0"
networks: ["253.255.25.1/25"]
router a63b3879-b694-45c5-ac97-75df53b5ca66 (lr_vcn3_stgw)
port lr_vcn3_stgw-lsb_vcn3_stgw
mac: "40:44:00:00:05:00"
networks: ["253.255.25.10/25"]
port lr_vcn3_stgw-ls_vcn3_external_stgw
mac: "40:44:00:00:05:01"
networks: ["253.255.80.20/16", "253.255.80.8/16"]
gateway chassis: [sca15-rain06 sca15-rain17 sca15-rain05]
nat 909bf813-7f7e-4eea-b460-5261529fcd5d
external ip: "253.255.80.8"
logical ip: "192.16.1.6"
type: "snat"
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!JRVNTziGt4XNeZ2SOqeVpPAJBumMpBrFghfvmDgxloqJYkHdd5yJ-4J7MI8VHal0HkV4VYbXcNk4gUrOZOdyEZ_q3NY$
