Summary
In an OpenStack deployment using Neutron with OVN as the backend, I
observed that enabling sampling on ACLs appears to create duplicated
sampling actions for certain OpenFlow datapath flows, resulting in
duplicated IPFIX records. This may indicate that ovn-controller is
installing redundant sampled flows.
Test Topology
vm_a ---- network1 ---- router ---- network2 ---- vm_b
-
OpenStack Neutron with OVN backend.
-
Testing with ICMP traffic (but TCP shows the same behavior).
-
Two ACLs: one allowing related outbound IPv4 traffic and one allowing
related inbound ICMP traffic.
-
Security groups (containing the two ACLs) applied symmetrically to both
VMs.
-
ACL correctness has been verified separately.
Problem Description
-
When sampling is enabled on both the inbound ICMP rule and the outbound
IPv4 rule, the same OpenFlow flow (0x3d62e572) appears twice, leading to
duplicated IPFIX records.
-
Dumping flows with ovs-dpctl dump-flows system@ovs-system shows
duplicated flow_sample actions on some flows.
*"recirc_id(0x6b9d),in_port(9),ct_state(-new+est-rel+rpl-inv+trk),ct_mark(0x20020/0xff0031),ct_label(0x1c8000000000000000000000000),eth(src=fa:16:3e:01:f9:9f,dst=fa:16:3e:05:9c:9a),eth_type(0x0800),ipv4(src=10.2.1.171,dst=10.2.2.88,proto=1,ttl=64,frag=no),
packets:3298, bytes:323204, used:0.277s,
actions:userspace(pid=4294967295,flow_sample(probability=65535,collector_set_id=2,obs_domain_id=33554449,obs_point_id=456,output_port=4294967295)),userspace(pid=4294967295,flow_sample(probability=65535,collector_set_id=2,obs_domain_id=33554449,obs_point_id=456,output_port=4294967295)),ct_clear,set(eth(src=fa:16:3e:37:38:47,dst=fa:16:3e:b0:69:d3)),set(ipv4(ttl=63)),ct(zone=23),recirc(0x6b9e)"*
-
This behavior does not occur when sampling is applied to only one
direction (either inbound or outbound).
Verification Steps
1.
*Sampling only inbound ICMP ACL*
→ Flows sampled once as expected.
2.
*Sampling both inbound ICMP and outbound IPv4 ACLs*
→ Certain OpenFlow flows are installed twice with sampling, suggesting
duplication.
3.
*Sampling only outbound IPv4 ACL*
→ No duplication observed.
(Example OpenFlow outputs and logical_flow outputs included below.)
Observations
-
The same cookie (e.g., 0x3d62e572) shows up multiple times when both
sampling rules are active.
-
It seems sampling instructions are redundantly applied at both ingress
and egress pipelines.
-
This could result in inaccurate flow export behavior (duplicated
samples).
Additional Information
-
*OVN Version:* 24.09.0
-
*Open vSwitch Version:* 3.4.0
-
*SB Schema:* 20.37.0
-
Outputs of ovn-sbctl list logical_flow, ovn-nbctl list sample, ovn-nbctl
list sample_collector, ovn-nbctl list sampling_app attached below for
full context.
-
Full OpenFlow dump samples included.
Request for Clarification
Based on the above, I would like to ask:
-
Is this duplicated sampling behavior expected when both inbound and
outbound ACLs sample?
-
If not expected, could this indicate a bug in how ovn-controller
generates OpenFlow rules under overlapping ACL sampling?
-
Are there any known workarounds?
Thank you very much for your time and support. I greatly appreciate your
guidance to better understand OVN’s sampling design here.
*Best regards,* Oscar
This is my TOPOLOGY: vm_a ---- network1 ---- router ---- network2 ---- vm_b
- Firstly, I sample only on the inbound icmp rule, these are the
openflow-flows related to sample that I can observe:
=============================== SAMPLE INBOUND ICMP
===============================
cookie=0xca44181d, duration=349282.691s, table=17, n_packets=345188,
n_bytes=40248274, idle_age=0, hard_age=60718,
priority=1000,ct_state=+est+rpl+trk,ct_mark=0x20000/0xff0000,ct_label=0/0xffffffffffffffffffffffff,ip,metadata=0x11
actions=sample(probability=65535,collector_set_id=2,obs_domain_id=33554449,obs_point_id=NXM_NX_CT_LABEL[96..127]),resubmit(,18)
cookie=0x666fbb74, duration=349282.694s, table=50, n_packets=343513,
n_bytes=33611396, idle_age=0, hard_age=60718,
priority=1000,ct_state=+est-rpl+trk,ct_mark=0x20020/0xff0030,ct_label=0/0xffffffffffffffffffffffff,ip,metadata=0x11
actions=sample(probability=65535,collector_set_id=2,obs_domain_id=33554449,obs_point_id=NXM_NX_CT_LABEL[96..127]),resubmit(,51)
=============================== SAMPLE INBOUND ICMP
===============================
- Secondly, I sample on both the inbound icmp rule and the outbound ipv4
rule, these are the openflow-flows:
=============================== SAMPLE INBOUND ICMP && OUTBOUND ALL
===============================
cookie=0xca44181d, duration=349682.075s, table=17, n_packets=345586,
n_bytes=40287278, idle_age=0, hard_age=61117,
priority=1000,ct_state=+est+rpl+trk,ct_mark=0x20000/0xff0000,ct_label=0/0xffffffffffffffffffffffff,ip,metadata=0x11
actions=sample(probability=65535,collector_set_id=2,obs_domain_id=33554449,obs_point_id=NXM_NX_CT_LABEL[96..127]),resubmit(,18)
cookie=0x4ca1e118, duration=6.136s, table=17, n_packets=2, n_bytes=196,
idle_age=0,
priority=1000,ct_state=+est-rpl+trk,ct_mark=0x20000/0xff0030,ct_label=0/0xffffffffffffffffffffffff,ip,metadata=0x12
actions=sample(probability=65535,collector_set_id=2,obs_domain_id=33554450,obs_point_id=NXM_NX_CT_LABEL[96..127]),resubmit(,18)
cookie=0x666fbb74, duration=349682.078s, table=50, n_packets=343910,
n_bytes=33650302, idle_age=0, hard_age=61117,
priority=1000,ct_state=+est-rpl+trk,ct_mark=0x20020/0xff0030,ct_label=0/0xffffffffffffffffffffffff,ip,metadata=0x11
actions=sample(probability=65535,collector_set_id=2,obs_domain_id=33554449,obs_point_id=NXM_NX_CT_LABEL[96..127]),resubmit(,51)
cookie=0x3d62e572, duration=6.139s, table=50, n_packets=5, n_bytes=490,
idle_age=0,
priority=1000,ct_state=+est+rpl+trk,ct_mark=0x20000/0xff0000,ct_label=0/0xffffffffffffffffffffffff,ip,metadata=0x11
actions=sample(probability=65535,collector_set_id=2,obs_domain_id=33554449,obs_point_id=NXM_NX_CT_LABEL[96..127]),resubmit(,51)
cookie=0x3d62e572, duration=6.139s, table=50, n_packets=3, n_bytes=294,
idle_age=0,
priority=1000,ct_state=+est+rpl+trk,ct_mark=0x20000/0xff0000,ct_label=0/0xffffffffffffffffffffffff,ip,metadata=0x12
actions=sample(probability=65535,collector_set_id=2,obs_domain_id=33554450,obs_point_id=NXM_NX_CT_LABEL[96..127]),resubmit(,51)
=============================== SAMPLE INBOUND ICMP && OUTBOUND ALL
===============================
(At this point, the openflow-flows 0x3d62e572 appeared twice)
- Thirdly, I removed the sample in inbound icmp rule, only sample in
outbound ipv4 rule, these are the openflow-flows:
=============================== SAMPLE OUTBOUND IPv4
===============================
cookie=0x4ca1e118, duration=6.136s, table=17, n_packets=57723,
n_bytes=5655704, idle_age=1,
priority=1000,ct_state=+est-rpl+trk,ct_mark=0x20000/0xff0030,ct_label=0/0xffffffffffffffffffffffff,ip,metadata=0x12
actions=sample(probability=65535,collector_set_id=2,obs_domain_id=33554450,obs_point_id=NXM_NX_CT_LABEL[96..127]),resubmit(,18)
cookie=0x3d62e572, duration=6.136s, table=50, n_packets=57744,
n_bytes=5837838, idle_age=1,
priority=1000,ct_state=+est+rpl+trk,ct_mark=0x20000/0xff0000,ct_label=0/0xffffffffffffffffffffffff,ip,metadata=0x12
actions=sample(probability=65535,collector_set_id=2,obs_domain_id=33554450,obs_point_id=NXM_NX_CT_LABEL[96..127]),resubmit(,51)
=============================== SAMPLE OUTBOUND IPv4
===============================
(The openflow-flows 0x3d62e572 appeared only once. It should still be only
once when I enable the sample on the inbound icmp rule, isn't it? But when ever
I tried to enable the sample on the inbound icmp rule, 0x3d62e572 doubles)
### ADDITIONAL INFORMATION ###
1. In case you need more information on the Logical Flows:
(ovn-sb-db)[root@site2-osp-controller-01-2024 /]# ovn-sbctl list logical_flow
4ca1e118
_uuid : 4ca1e118-5830-4bf7-aa6e-52121d86463f
actions :
"sample(probability=65535,collector_set=2,obs_domain=2,obs_point=ct_label.obs_point_id);
next;"
controller_meter : []
external_ids : {source="northd.c:6924", stage-name=ls_in_acl_sample}
flow_desc : []
logical_datapath : []
logical_dp_group : f9e97961-08af-493f-9056-06b3f0b96b94
match : "ip && ct.trk && (ct.est || ct.rel) &&
ct_label.obs_unused == 0 && !ct.rpl && ct_mark.obs_collector_id == 2 &&
ct_mark.obs_stage == 0"
pipeline : ingress
priority : 1000
table_id : 9
tags : {}
hash : 0
(ovn-sb-db)[root@site2-osp-controller-01-2024 /]# ovn-sbctl list logical_flow
ca44181d
_uuid : ca44181d-46d8-4531-b1b2-80b5c681d3ab
actions :
"sample(probability=65535,collector_set=2,obs_domain=2,obs_point=ct_label.obs_point_id);
next;"
controller_meter : []
external_ids : {source="northd.c:6935", stage-name=ls_in_acl_sample}
flow_desc : []
logical_datapath : []
logical_dp_group : f9e97961-08af-493f-9056-06b3f0b96b94
match : "ip && ct.trk && (ct.est || ct.rel) &&
ct_label.obs_unused == 0 && ct.rpl && ct_mark.obs_collector_id == 2"
pipeline : ingress
priority : 1000
table_id : 9
tags : {}
hash : 0
(ovn-sb-db)[root@site2-osp-controller-01-2024 /]# ovn-sbctl list logical_flow
666fbb74
_uuid : 666fbb74-0483-4a64-b8fb-8346ab300689
actions :
"sample(probability=65535,collector_set=2,obs_domain=2,obs_point=ct_label.obs_point_id);
next;"
controller_meter : []
external_ids : {source="northd.c:6924", stage-name=ls_out_acl_sample}
flow_desc : []
logical_datapath : []
logical_dp_group : f9e97961-08af-493f-9056-06b3f0b96b94
match : "ip && ct.trk && (ct.est || ct.rel) &&
ct_label.obs_unused == 0 && !ct.rpl && ct_mark.obs_collector_id == 2 &&
ct_mark.obs_stage == 2"
pipeline : egress
priority : 1000
table_id : 5
tags : {}
hash : 0
(ovn-sb-db)[root@site2-osp-controller-01-2024 /]# ovn-sbctl list logical_flow
3d62e572
_uuid : 3d62e572-fbf3-4882-9145-5b8b946ac7be
actions :
"sample(probability=65535,collector_set=2,obs_domain=2,obs_point=ct_label.obs_point_id);
next;"
controller_meter : []
external_ids : {source="northd.c:6935", stage-name=ls_out_acl_sample}
flow_desc : []
logical_datapath : []
logical_dp_group : f9e97961-08af-493f-9056-06b3f0b96b94
match : "ip && ct.trk && (ct.est || ct.rel) &&
ct_label.obs_unused == 0 && ct.rpl && ct_mark.obs_collector_id == 2"
pipeline : egress
priority : 1000
table_id : 5
tags : {}
hash : 0
2. The versions
(ovn-controller)[root@site2-osp-compute-02-2024 /]# ovn-controller --version
ovn-controller 24.09.0
Open vSwitch Library 3.4.0
OpenFlow versions 0x6:0x6
SB DB Schema 20.37.0
(ovn-northd)[root@site2-osp-controller-01-2024 /]# ovn-northd --version
ovn-northd 24.09.0
Open vSwitch Library 3.4.0
3. Related OVN Info:
_uuid : 145084fe-feb3-4967-822d-2d3104015fbf
action : allow-related
direction : from-lport
external_ids :
{"neutron:security_group_rule_id"="38e48a61-7e12-4f72-ab15-669e9b163876"}
label : 0
log : false
match : "inport == @pg_ad579651_c29c_462a_a4ac_7bce9082b645 &&
ip4"
meter : []
name : []
options : {}
priority : 1002
sample_est : 49d84040-1081-430a-be1b-a8dfbcbf6fe6
sample_new : []
severity : []
tier : 0
_uuid : 960fba5a-160f-4c89-a397-32459faae38d
action : allow-related
direction : to-lport
external_ids :
{"neutron:security_group_rule_id"="8138e214-f7c0-487a-b91b-0824c26b2002"}
label : 0
log : false
match : "outport == @pg_ad579651_c29c_462a_a4ac_7bce9082b645 &&
ip4 && ip4.src == 0.0.0.0/0 && icmp4"
meter : []
name : []
options : {}
priority : 1002
sample_est : 49d84040-1081-430a-be1b-a8dfbcbf6fe6
sample_new : []
severity : []
tier : 0
(ovn-nb-db)[root@site2-osp-controller-01-2024 /]# ovn-nbctl --no-leader-only
list sample
_uuid : 49d84040-1081-430a-be1b-a8dfbcbf6fe6
collectors : [60407201-e6db-4719-9908-ea0dc53c64e4]
metadata : 456
(ovn-nb-db)[root@site2-osp-controller-01-2024 /]# ovn-nbctl --no-leader-only
list sample_collector
_uuid : 60407201-e6db-4719-9908-ea0dc53c64e4
external_ids : {}
id : 2
name : prob-100-sample-collector
probability : 65535
set_id : 2
(ovn-nb-db)[root@site2-osp-controller-01-2024 /]# ovn-nbctl --no-leader-only
list sampling_app
_uuid : f5bcf2ad-76f9-4708-819b-1326f1857899
external_ids : {}
id : 1
type : acl-new
_uuid : 3f140431-20d5-4656-bb27-752b38444b20
external_ids : {}
id : 3
type : drop
_uuid : cf01a57b-16e7-445e-a863-5cb5161b399b
external_ids : {}
id : 2
type : acl-est
_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
